paddlecloud训练模型,cost为nan
Created by: yabinggoing
序列标注问题 模型为BI-LSTM + SOFTMAX 单机小数据量测试没有问题 paddlecloud训练模型 出现cost为nan的情况,以下为paddle cloud日志情况
I0730 02:42:43.908999 20 Util.cpp:166] commandline: --num_gradient_servers=8 --p orts_num_for_sparse=1 --use_gpu=1 --trainer_id=1 --pservers=10.1.3.10 --trainer_coun t=1 --num_passes=1 --ports_num=1 --port=7164 10 I0730 02:42:53.613766 20 GradientMachine.cpp:85] Initing parameters.. 11 I0730 02:43:08.587280 20 GradientMachine.cpp:92] Init parameters done. 12 .................................................................................... .................................................................................... .................................................................................... .................................................................................... .................................................................................... ................................................................................Pass 0, Batch 500, Cost nan, time 98.470099926 s 13 .................................................................................... .................................................................................... .................................................................................... .................................................................................... .................................................................................... ...............................................................................Pass 0, Batch 1000, Cost nan, time 192.780102015 s