内部用户, Floating point exception when trainning with GPU
Created by: fsfszongming256
日志::
I0313 19:00:05.295281 8472 Stat.cpp:140] ======= BarrierStatSet status ======
I0313 19:00:05.295287 8472 Stat.cpp:153] --------------------------------------------------
Thread [139773011044416] Backwarding lstm_1, lstm_transform___lstm_1__, embedding_1, seq_pooling_0, lstm_0, lstm_trans
form___lstm_0__, embedding_0, t_word, q_word,
* Aborted at 1489402805 (unix time) try "date -d @1489402805" if you are using GNU date *
PC: @ 0x7b3859 paddle::GpuVectorT<>::getAbsMax()
* SIGFPE (@0x7b3859) received by PID 8472 (TID 0x7f1f70ac2840) from PID 8075353; stack trace: *
@ 0x7f1f7069d160 (unknown)
@ 0x7b3859 paddle::GpuVectorT<>::getAbsMax()
@ 0x74c2ef ZNSt17_Function_handlerIFvPN6paddle9ParameterEEZNS0_15TrainerInternal13trainOneBatchElRKNS0_9DataBatchEPSt6v
ectorINS0_8ArgumentESaIS9_EEEUlS2_E_E9_M_invokeERKSt9_Any_dataS2
@ 0x8da203 paddle::Parameter::incUpdate()
@ 0x5f808c paddle::LstmLayer::backward()
@ 0x6add1f paddle::NeuralNetwork::backward()
@ 0x74c3e3 paddle::TrainerInternal::forwardBackwardBatch()
@ 0x74d2ed paddle::TrainerInternal::trainOneBatch()
@ 0x74ac06 paddle::Trainer::trainOneDataBatch()
@ 0x74b0f9 paddle::Trainer::trainOnePass()
@ 0x74bcc5 paddle::Trainer::train()
@ 0x5b90ca main
@ 0x7f1f6f2babd5 __libc_start_main
@ 0x5ce555 (unknown)
/home/img/zongming/paddle_env/paddle_internal_release_tools/idl/paddle/output/bin/paddle_local: line 109: 8472 Floating point excepti
on(core dumped) ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}