Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
xuexixianjinzhishi
Paddle
提交
20b9be65
P
Paddle
项目概览
xuexixianjinzhishi
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
20b9be65
编写于
5月 26, 2021
作者:
J
JZ-LIANG
提交者:
GitHub
5月 26, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[Tensor Parallelism] split fix bug (#33015)
上级
a2a45d8d
变更
6
隐藏空白更改
内联
并排
Showing
6 changed file
with
20 addition
and
5 deletion
+20
-5
python/paddle/distributed/collective.py
python/paddle/distributed/collective.py
+11
-0
python/paddle/distributed/fleet/base/distributed_strategy.py
python/paddle/distributed/fleet/base/distributed_strategy.py
+1
-1
python/paddle/fluid/contrib/mixed_precision/fp16_lists.py
python/paddle/fluid/contrib/mixed_precision/fp16_lists.py
+1
-0
python/paddle/fluid/tests/unittests/column_parallel_linear_api.py
...addle/fluid/tests/unittests/column_parallel_linear_api.py
+1
-1
python/paddle/fluid/tests/unittests/row_parallel_linear_api.py
...n/paddle/fluid/tests/unittests/row_parallel_linear_api.py
+2
-2
python/paddle/fluid/tests/unittests/test_collective_api_base.py
.../paddle/fluid/tests/unittests/test_collective_api_base.py
+4
-1
未找到文件。
python/paddle/distributed/collective.py
浏览文件 @
20b9be65
...
...
@@ -977,6 +977,11 @@ def _parallel_linear(x,
group
=
None
):
"""
Parallel Linear
axis the dimension of the parameter of linear layer.
axis = 0: the row dimension
axid = 1: the col dimension
"""
if
group
is
not
None
and
not
group
.
is_member
():
return
...
...
@@ -1008,6 +1013,12 @@ def _parallel_linear(x,
main_block
=
paddle
.
static
.
default_main_program
().
global_block
()
startup_block
.
vars
[
linear
.
weight
.
name
].
is_distributed
=
True
main_block
.
vars
[
linear
.
weight
.
name
].
is_distributed
=
True
# set is_distributed for splited bias
# if a linear layer is splited by row, each rank would hold a complete bias and they should be the same in each rank.
# if a linear layer is splited by col, the bias would also be split into each rank as its weight
if
axis
==
1
and
linear
.
_bias_attr
!=
False
:
startup_block
.
vars
[
linear
.
bias
.
name
].
is_distributed
=
True
main_block
.
vars
[
linear
.
bias
.
name
].
is_distributed
=
True
if
not
gather_out
:
return
linear_out
...
...
python/paddle/distributed/fleet/base/distributed_strategy.py
100755 → 100644
浏览文件 @
20b9be65
...
...
@@ -814,7 +814,7 @@ class DistributedStrategy(object):
"sharding_segment_strategy": "segment_broadcast_MB",
"segment_broadcast_MB": 32,
"sharding_degree": 8,
"
sharding
_degree": 2,
"
dp
_degree": 2,
"gradient_merge_acc_step": 4,
}
"""
...
...
python/paddle/fluid/contrib/mixed_precision/fp16_lists.py
浏览文件 @
20b9be65
...
...
@@ -145,6 +145,7 @@ gray_list = {
'sign'
,
'cast'
,
'fused_bn_add_activation'
,
'c_identity'
,
}
# The set of ops that don't support fp16 calculation
...
...
python/paddle/fluid/tests/unittests/column_parallel_linear_api.py
浏览文件 @
20b9be65
...
...
@@ -69,7 +69,7 @@ class TestColumnParallelLinearAPI(TestCollectiveAPIRunnerBase):
axis
=
1
,
num_partitions
=
2
,
weight_attr
=
param_attr
,
bias_attr
=
Fals
e
,
)
bias_attr
=
Tru
e
,
)
return
[
linear_out
]
...
...
python/paddle/fluid/tests/unittests/row_parallel_linear_api.py
浏览文件 @
20b9be65
...
...
@@ -65,12 +65,12 @@ class TestRowParallelLinearAPI(TestCollectiveAPIRunnerBase):
linear_out
=
paddle
.
distributed
.
split
(
data
,
size
=
(
1000
,
8
),
size
=
(
1000
,
16
),
operation
=
'linear'
,
axis
=
0
,
num_partitions
=
2
,
weight_attr
=
param_attr
,
bias_attr
=
Fals
e
,
)
bias_attr
=
Tru
e
,
)
return
[
linear_out
]
...
...
python/paddle/fluid/tests/unittests/test_collective_api_base.py
浏览文件 @
20b9be65
...
...
@@ -154,7 +154,10 @@ class TestDistBase(unittest.TestCase):
#update environment
env0
.
update
(
envs
)
env1
.
update
(
envs
)
tr_cmd
=
"%s %s"
if
os
.
getenv
(
'WITH_COVERAGE'
,
'OFF'
)
==
'ON'
:
tr_cmd
=
"%s -m coverage run --branch -p %s"
else
:
tr_cmd
=
"%s %s"
tr0_cmd
=
tr_cmd
%
(
self
.
_python_interp
,
model_file
)
tr1_cmd
=
tr_cmd
%
(
self
.
_python_interp
,
model_file
)
tr0_pipe
=
open
(
"/tmp/tr0_err_%d.log"
%
os
.
getpid
(),
"w"
)
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录