多机启动pserver 带内存优化报错
Created by: ccmeteorljh
启动脚本:
python fluid_benchmark.py --model resnet --data_set flowers --use_reader_op --data_path recordio --memory_optimize --device GPU --gpus 1 --batch_size 64 --pass_num 1
pserver报错如下:
Traceback (most recent call last):
File "fluid_benchmark.py", line 364, in <module>
main()
File "fluid_benchmark.py", line 328, in main
startup_prog)
File "fluid_benchmark.py", line 96, in dist_transpile
pserver_program = t.get_pserver_program(current_endpoint)
File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/transpiler/distribute_transpiler.py", line 558, in get_pserver_program
merged_var, lr_ops)
File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/transpiler/distribute_transpiler.py", line 489, in __append_optimize_op__
self.origin_program, merged_var)
File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/transpiler/distribute_transpiler.py", line 1360, in _append_pserver_ops
attrs=opt_op.all_attrs())
File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/framework.py", line 1134, in append_op
op = Operator(block=self, desc=op_desc, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/framework.py", line 520, in __init__
in_proto.name)
AssertionError: Input Grad not found
关闭memory_optimize参数后即可正常运行