Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Greenplum
DeepSpeed
提交
17a479dd
D
DeepSpeed
项目概览
Greenplum
/
DeepSpeed
上一次同步 大约 1 年
通知
10
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DeepSpeed
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
17a479dd
编写于
10月 21, 2021
作者:
C
Conglong Li
提交者:
GitHub
10月 21, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix pipeline engine (#1473)
* fix pp * better fix
上级
58a8e13c
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
21 addition
and
3 deletion
+21
-3
deepspeed/runtime/pipe/engine.py
deepspeed/runtime/pipe/engine.py
+21
-3
未找到文件。
deepspeed/runtime/pipe/engine.py
浏览文件 @
17a479dd
...
...
@@ -1060,7 +1060,7 @@ class PipelineEngine(DeepSpeedEngine):
local_part
=
outputs
[
1
],
group
=
self
.
grid
.
get_slice_parallel_group
())
outputs
[
0
].
data
=
part_output
.
full
()
outputs
=
(
[
outputs
[
0
],
*
outputs
[
2
:]
])
outputs
=
(
outputs
[
0
],
*
outputs
[
2
:
])
# save for backward
self
.
pipe_buffers
[
'outputs'
][
buffer_id
]
=
outputs
...
...
@@ -1072,8 +1072,26 @@ class PipelineEngine(DeepSpeedEngine):
dtype
=
outputs
.
dtype
,
num_buffers
=
1
)[
0
]
else
:
sizes_and_dtypes
=
[(
list
(
t
.
size
()),
t
.
dtype
)
for
t
in
outputs
if
t
.
is_floating_point
()]
# XXX This is a HACK
# When we exchange activations/gradients, the two pipe stages
# need to issue the send/recv with the same buffer sizes or
# else there is a deadlock. The is_floating_point() filter is
# used to avoid sending gradients for tensors that do not
# produce gradients. When TP>1, we partition the first
# activations/gradients across TP ranks to save communication
# volume and memory. That partitioned tensor is represented as
# two tensors: a 1/TPth chunk of the original data and also a
# small LongTensor storing the metadata used to reconstruct on
# the other side. When combined, the floating point filter also
# filtered out the metadata tensor. A quick (hacky) fix just
# branches on is_grad_partitioned so we don't filter out the
# metadata tensor.
if
self
.
is_grad_partitioned
:
sizes_and_dtypes
=
[(
list
(
t
.
size
()),
t
.
dtype
)
for
t
in
outputs
]
else
:
sizes_and_dtypes
=
[(
list
(
t
.
size
()),
t
.
dtype
)
for
t
in
outputs
if
t
.
is_floating_point
()]
self
.
grad_layer
=
self
.
_allocate_buffers
(
sizes_and_dtypes
,
num_buffers
=
1
)[
0
]
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录