Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
OPTHREE
Paddle
提交
a2dbb0c2
P
Paddle
项目概览
OPTHREE
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
1
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
1
Issue
1
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
a2dbb0c2
编写于
9月 08, 2021
作者:
W
WangXi
提交者:
GitHub
9月 08, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[hybrid] check pipeline persist var which changed in forward and used in backward (#35453)
上级
b95c5ae0
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
36 addition
and
0 deletion
+36
-0
python/paddle/fluid/optimizer.py
python/paddle/fluid/optimizer.py
+34
-0
python/paddle/fluid/tests/unittests/test_fleet_pipeline_meta_optimizer.py
...uid/tests/unittests/test_fleet_pipeline_meta_optimizer.py
+2
-0
未找到文件。
python/paddle/fluid/optimizer.py
浏览文件 @
a2dbb0c2
...
...
@@ -5715,6 +5715,35 @@ class PipelineOptimizer(object):
backward_insert_index
+=
1
block
.
_sync_with_cpp
()
def
_check_pipeline_persist_var
(
self
,
program
):
"""
Pipeline may need multiple forward before
"""
block
=
program
.
global_block
()
persist_output
=
set
()
used_in_backward
=
set
()
for
op
in
block
.
ops
:
if
self
.
_is_forward_op
(
op
):
for
var_name
in
op
.
output_arg_names
:
var
=
block
.
vars
[
var_name
]
if
var
.
persistable
:
persist_output
.
add
(
var_name
)
elif
self
.
_is_backward_op
(
op
):
for
var_name
in
op
.
input_arg_names
:
if
var_name
in
persist_output
:
used_in_backward
.
add
(
var_name
)
if
len
(
used_in_backward
)
==
0
:
return
warnings
.
warn
(
"The pipeline requires multiple forward calculations before backward, "
"so when the persistable var is changed in the forward, it may cause "
"errors in the backward calculation who using this persistable var. "
"However, some backward op don't need this var(NoNeedBufferVars), "
"there will be no error at this time.
\n
"
"So please check these persistable vars which changed in "
"forward and used in backward:
\n
{}"
.
format
(
used_in_backward
))
def
minimize
(
self
,
loss
,
startup_program
=
None
,
...
...
@@ -5831,6 +5860,11 @@ class PipelineOptimizer(object):
# A pass to move the recv op to the beginning of
# the forward/backward phase
self
.
_mv_head_recv
(
program_list
[
self
.
local_rank
])
# A pass to check pipeline persist var which changed in
# forward and used in backward
self
.
_check_pipeline_persist_var
(
program_list
[
self
.
local_rank
])
main_program
.
_pipeline_opt
=
{
"trainer"
:
"PipelineTrainer"
,
"device_worker"
:
"Section"
,
...
...
python/paddle/fluid/tests/unittests/test_fleet_pipeline_meta_optimizer.py
浏览文件 @
a2dbb0c2
...
...
@@ -45,6 +45,8 @@ class TestFleetMetaOptimizer(unittest.TestCase):
with
static
.
device_guard
(
"gpu:1"
):
fc_2
=
paddle
.
fluid
.
layers
.
fc
(
input
=
fc_1
,
size
=
64
,
act
=
'tanh'
)
# for pipeline check_pipeline_persist_var coverage
fc_2
.
persistable
=
True
fc_2
=
fc_2
*
input_z
prediction
=
paddle
.
fluid
.
layers
.
fc
(
input
=
[
fc_2
],
size
=
2
,
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录