Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Greenplum
DeepSpeed
提交
9d541a63
D
DeepSpeed
项目概览
Greenplum
/
DeepSpeed
上一次同步 大约 1 年
通知
10
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DeepSpeed
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
9d541a63
编写于
9月 12, 2022
作者:
A
Arash Bakhtiari
提交者:
GitHub
9月 12, 2022
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add unit tests for residual_add kernels (#2307)
上级
efa8aded
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
106 addition
and
0 deletion
+106
-0
tests/unit/ops/transformer/inference/test_residual_add.py
tests/unit/ops/transformer/inference/test_residual_add.py
+106
-0
未找到文件。
tests/unit/ops/transformer/inference/test_residual_add.py
0 → 100644
浏览文件 @
9d541a63
"""
Copyright 2022 The Microsoft DeepSpeed Team
"""
import
pytest
import
torch
import
deepspeed
from
deepspeed.ops.op_builder
import
InferenceBuilder
if
not
deepspeed
.
ops
.
__compatible_ops__
[
InferenceBuilder
.
NAME
]:
pytest
.
skip
(
"Inference ops are not available on this system"
,
allow_module_level
=
True
)
def
allclose
(
x
,
y
):
assert
x
.
dtype
==
y
.
dtype
rtol
,
atol
=
{
torch
.
float32
:
(
5e-4
,
5e-5
),
torch
.
float16
:
(
3e-2
,
2e-2
)}[
x
.
dtype
]
return
torch
.
allclose
(
x
,
y
,
rtol
=
rtol
,
atol
=
atol
)
@
pytest
.
fixture
def
inference_module
():
return
InferenceBuilder
().
load
()
def
run_residual_add_reference
(
hidden_state
,
residual
,
attention_output
,
final_bias
,
attention_output_bias
,
mlp_after_attn
,
add_bias
,
mp_size
=
1
):
residual_scaled
=
residual
/
mp_size
final_bias_scaled
=
final_bias
/
mp_size
attention_output_scaled
=
attention_output
/
mp_size
attention_output_bias_scaled
=
attention_output_bias
/
mp_size
hidden_state
=
hidden_state
+
residual_scaled
+
final_bias_scaled
# in case that mlp_after_attn = True, we additionally need to scale attention_output as well
if
mlp_after_attn
:
hidden_state
+=
attention_output_scaled
else
:
hidden_state
+=
attention_output
# TODO: The `add_bias` flag is used only for `launch_gptj_residual_add` kernel (aka, mlp_after_attn is False).
# This is a hack to get the parametarized add_bias to work. We need to fix this after refactoring the kernels.
add_bias
=
True
if
mlp_after_attn
else
add_bias
if
add_bias
:
hidden_state
=
hidden_state
+
attention_output_bias_scaled
return
hidden_state
@
pytest
.
mark
.
inference
@
pytest
.
mark
.
parametrize
(
"batch"
,
[
1
,
2
])
@
pytest
.
mark
.
parametrize
(
"sequence"
,
[
1
,
128
,
255
])
@
pytest
.
mark
.
parametrize
(
"hidden_dim"
,
[
512
,
1232
,
4096
])
@
pytest
.
mark
.
parametrize
(
"dtype"
,
[
torch
.
float16
,
torch
.
float32
])
@
pytest
.
mark
.
parametrize
(
"mlp_after_attn"
,
[
True
,
False
])
@
pytest
.
mark
.
parametrize
(
"add_bias"
,
[
True
,
False
])
@
pytest
.
mark
.
parametrize
(
"mp_size"
,
[
1
,
2
])
# @pytest.mark.parametrize("preln", [True]) # TODO: add support for preln
def
test_residual_add
(
inference_module
,
batch
,
sequence
,
hidden_dim
,
dtype
,
mlp_after_attn
,
add_bias
,
mp_size
):
preln
=
True
ds_out
=
torch
.
randn
((
batch
,
sequence
,
hidden_dim
),
dtype
=
dtype
,
device
=
'cuda'
)
residual
=
torch
.
randn
((
batch
,
sequence
,
hidden_dim
),
dtype
=
dtype
,
device
=
'cuda'
)
attention_output
=
torch
.
randn
((
batch
,
sequence
,
hidden_dim
),
dtype
=
dtype
,
device
=
'cuda'
)
final_bias
=
torch
.
randn
((
hidden_dim
),
dtype
=
dtype
,
device
=
'cuda'
)
attention_output_bias
=
torch
.
randn
((
hidden_dim
),
dtype
=
dtype
,
device
=
'cuda'
)
ref_out
=
ds_out
.
clone
()
ref_out
=
run_residual_add_reference
(
ref_out
,
residual
,
attention_output
,
final_bias
,
attention_output_bias
,
mlp_after_attn
,
add_bias
,
mp_size
)
inference_module
.
residual_add
(
ds_out
,
# in-place update of ds_out. Needs reafactoring to be consistent with other kernels.
residual
,
attention_output
,
final_bias
,
attention_output_bias
,
mp_size
,
mlp_after_attn
,
add_bias
,
preln
)
assert
(
allclose
(
ds_out
,
ref_out
))
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录