训练到第一轮就没有输出了,看日志可能是有问题,但不清楚是为什么
Created by: zhaoyang1708
使用mpi训练到第一轮就没有输出了,看日志可能是有问题,但不清楚是为什么。求帮助
Mon Jul 22 10:23:43 2019[1,16]:Traceback (most recent call last): Mon Jul 22 10:23:43 2019[1,16]: File "train.py", line 726, in Mon Jul 22 10:23:43 2019[1,24]:use_parallel_executor=bool(args.use_parallel_exe) Mon Jul 22 10:23:43 2019[1,14]:Traceback (most recent call last): Mon Jul 22 10:23:43 2019[1,16]: use_parallel_executor=bool(args.use_parallel_exe) Mon Jul 22 10:23:43 2019[1,16]: File "train.py", line 700, in train Mon Jul 22 10:23:43 2019[1,16]: train_loop(main_program, trainer_id) Mon Jul 22 10:23:43 2019[1,16]: File "train.py", line 613, in train_loop Mon Jul 22 10:23:43 2019[1,16]: auc_var.name, auc_var_cos.name, cur_auc.name, cur_auc_cos.name]) Mon Jul 22 10:23:43 2019[1,16]: File "/home/disk1/normandy/maybach/app-user-20190719212232-10626/workspace/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/parallel_executor.py", line 205, in run