Created by: hiahiahu

gpu集群任务 http://yq01-idl-gpu-offline92.yq01.baidu.com:8880/look/overview/file?id=4312&file=/job-4312/log/train.log&offset=318

F1229 11:45:09.434870 8013 hl_cuda_device.cc:646] Check failed: cudaSuccess == cudaStat (0 vs. 77) Cuda Error: an illegal memory access was encountered

* Check failure stack trace: *

@ 0xffab28 google::LogMessage::Fail() @ 0xffaa80 google::LogMessage::SendToLog() @ 0xffa515 google::LogMessage::Flush() @ 0xffd2d6 google::LogMessageFatal::~LogMessageFatal() @ 0x9bddd8 hl_stream_synchronize() @ 0x9dbc67 hl_max_sequence_backward() @ 0x7b0db8 paddle::GpuMatrix::maxSequenceBackward() @ 0x6cdc32 paddle::MaxLayer::backward() @ 0x745ca0 paddle::NeuralNetwork::backward() @ 0x73d78c paddle::TrainerThread::backward() @ 0x73d8b4 paddle::TrainerThread::computeThread() @ 0x7fd73c5408a0 execute_native_thread_routine @ 0x7fd73d6081c3 start_thread @ 0x7fd73bcb112d __clone

相关配置：

谢谢！