Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Greenplum
DeepSpeed
提交
55ed1057
D
DeepSpeed
项目概览
Greenplum
/
DeepSpeed
上一次同步 大约 1 年
通知
10
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DeepSpeed
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
55ed1057
编写于
9月 15, 2020
作者:
J
Jeff Rasley
提交者:
GitHub
9月 15, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix bug related to stitching reduced grads across communication partitions (#318)
上级
91b4a93d
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
13 addition
and
21 deletion
+13
-21
deepspeed/runtime/zero/stage1.py
deepspeed/runtime/zero/stage1.py
+13
-21
未找到文件。
deepspeed/runtime/zero/stage1.py
浏览文件 @
55ed1057
...
...
@@ -249,8 +249,7 @@ class FP16_DeepSpeedZeroOptimizer_Stage1(object):
# RS: divide up the sub-partitions and keep track of offsets for each param
# partition_size = len(self.fp16_groups_flat[i]) / dist.get_world_size(group=self.dp_process_group)
params_in_rank_sub_partition
,
params_in_rank_sub_partitions_offsets
,
\
params_not_local
=
self
.
get_all_sub_partition_info
(
params_in_rank_sub_partition
,
params_in_rank_sub_partitions_offsets
,
params_not_local
=
self
.
get_all_sub_partition_info
(
tensor_list
=
self
.
fp16_groups
[
i
],
all_element_intervals
=
element_intervals
,
local_rank
=
local_rank
,
...
...
@@ -591,28 +590,20 @@ class FP16_DeepSpeedZeroOptimizer_Stage1(object):
all_comm_partitions
.
append
(
single_comm_all_partitions
)
for
p
in
my_params
:
partitions
=
param_partition_map
[
p
]
parts
=
[]
for
part
in
partitions
:
params
,
offsets
=
partition_param_map
[
part
]
found
=
False
for
p_idx
,
_p
in
enumerate
(
params
):
if
p
.
__hash__
()
==
_p
.
__hash__
():
found
=
True
if
offsets
[
p_idx
][
0
]
is
not
None
:
my_part
=
part
.
narrow
(
0
,
offsets
[
p_idx
][
0
],
offsets
[
p_idx
][
1
])
parts
.
append
(
my_part
)
assert
found
if
p
is
not
None
:
updated_grad
=
_unflatten_dense_tensors
(
torch
.
cat
(
parts
),
[
p
])
p
.
grad
.
copy_
(
updated_grad
[
0
])
# stitch together all rank sub partitions for each comm idx
flat_comm_grads
=
[]
for
comm_idx
,
rank_partitions
in
enumerate
(
all_comm_partitions
):
flat_comm_grads
.
append
(
torch
.
cat
(
rank_partitions
))
flat_all_grads
=
torch
.
cat
(
flat_comm_grads
)
# copy back reduced gradients but only those needed for this local rank
for
param
,
updated_grad
in
zip
(
self
.
fp16_groups
[
i
],
_unflatten_dense_tensors
(
flat_all_grads
,
self
.
fp16_groups
[
i
])):
if
param
in
my_params
:
param
.
grad
.
copy_
(
updated_grad
)
def
step
(
self
,
closure
=
None
):
# First compute norm for all group so we know if there is overflow
self
.
overflow
=
self
.
overflow_checker
.
check
()
prev_scale
=
self
.
loss_scale
...
...
@@ -649,6 +640,7 @@ class FP16_DeepSpeedZeroOptimizer_Stage1(object):
#)
#TODO RS: can we safely use dtype of the first sub-partition? i think so
# create flat gradient partitions for parameters updated by this process
local_grad_sub_partitions
=
self
.
get_flat_sub_partitions
(
comm_tensor_list
=
self
.
params_in_rank_sub_partitions
[
i
][
partition_id
],
comm_param_offsets
=
self
.
params_in_rank_sub_partitions_offsets
[
i
]
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录