Runtime Error with CUDA
Created by: wondervictor
I installed the paddle with GPU in one node, but I trained my network with choosing --use_gpu=false, but an error occurred.
Paddle release a new version 0.9.0, you can get the install package in http://www.paddlepaddle.org
I0227 21:38:53.200770 2632 Util.cpp:160] commandline: /usr/local/bin/../opt/paddle/bin/paddle_trainer --config=trainer_config.py --save_dir=./output --trainer_count=32 --log_period=1000 --dot_period=100 --num_passes=1000 --use_gpu=false --show_parameter_stats_period=3000
[INFO 2017-02-27 21:38:53,363 networks.py:1472] The input order is [data_0, data_1, data_2, data_3, data_4, data_5, data_6, label]
[INFO 2017-02-27 21:38:53,363 networks.py:1478] The output order is [__cost_0__]
I0227 21:38:53.371265 2632 Trainer.cpp:165] trainer mode: Normal
I0227 21:38:53.467387 2632 PyDataProvider2.cpp:243] loading dataprovider data_provider::process
I0227 21:38:53.469506 2632 PyDataProvider2.cpp:243] loading dataprovider data_provider::process
I0227 21:38:53.469774 2632 GradientMachine.cpp:86] Initing parameters..
I0227 21:38:53.474694 2632 GradientMachine.cpp:93] Init parameters done.
*** Check failure stack trace: ***
F0227 21:38:54.077792 2661 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077803 2642 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detected
*** Check failure stack trace: ***
F0227 21:38:54.077816 2647 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detected
*** Check failure stack trace: ***
F0227 21:38:54.077792 2661 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077803 2642 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:3F:540227F0778190270000 2655 hl_cuda_device.cc:38566] :38:cudaSuccess == cudaStat (0 vs. 38)5Cuda Error: noF077807 000002 7 133: 2651 .] e077829 566 ] 2 27_ uda_device.ccc566Success no CUDA-c13:38:54.Check failed: cudaSu cCuda Error: no CUDA-capable device is detected
637 hl_cuda_device.cc:566] FCheck failed: 2Cuda Error: = cno CUDA-capable device is detected21:38:54.077857 2649 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 F210227 8:21:38:54. 077882 2657 Check failed: .cc:566] CcudaSuccess == cudaStat (0 vs. 38)Cuda Error: Cuda Error: F0227 21:38:54.077883 2643 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2660 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077833 2663 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2648 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2659 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077848 2633 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077857 2665 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detected
*** Check failure stack trace: ***
F0227 21:38:54.077792 2661 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077803 2642 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:3F:540227F0778190270000 2655 hl_cuda_device.cc:38566] :38:cudaSuccess == cudaStat (0 vs. 38)5Cuda Error: noF077807 000002 7 133: 2651 .] e077829 566 ] 2 27_ uda_device.ccc566Success no CUDA-c13:38:54.Check failed: cudaSu cCuda Error: no CUDA-capable device is detected
637 hl_cuda_device.cc:566] FCheck failed: 2Cuda Error: = cno CUDA-capable device is detected21:38:54.077857 2649 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 F210227 8:21:38:54. 077882 2657 Check failed: .cc:566] CcudaSuccess == cudaStat (0 vs. 38)Cuda Error: Cuda Error: F0227 21:38:54.077883 2643 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2660 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077833 2663 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2648 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2659 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077848 2633 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077857 2665 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detected
*** Check failure stack trace: ***
*** Check failure stack trace: ***
F0227 21:38:54.077792 2661 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077803 2642 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:3F:540227F0778190270000 2655 hl_cuda_device.cc:38566] :38:cudaSuccess == cudaStat (0 vs. 38)5Cuda Error: noF077807 000002 7 133: 2651 .] e077829 566 ] 2 27_ uda_device.ccc566Success no CUDA-c13:38:54.Check failed: cudaSu cCuda Error: no CUDA-capable device is detected
637 hl_cuda_device.cc:566] FCheck failed: 2Cuda Error: = cno CUDA-capable device is detected21:38:54.077857 2649 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 F210227 8:21:38:54. 077882 2657 Check failed: .cc:566] CcudaSuccess == cudaStat (0 vs. 38)Cuda Error: Cuda Error: F0227 21:38:54.077883 2643 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2660 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077833 2663 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2648 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077846 2659 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077848 2633 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077857 2665 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077857 2653 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detectedF0227 21:38:54.077858 2640 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 38) Cuda Error: no CUDA-capable device is detected
*** Check failure stack trace: ***
F0227 21:38:54.077816 2644 logging.cc:1360] RAW: Check data_->num_chars_to_log_ > 0 && data_->message_text_[data_->num_chars_to_log_-1] == '\n' failed:
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93cb00 google::LogMessage::Fail()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x943d74 google::RawLog__()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x93ca5c google::LogMessage::SendToLog()
@ 0x93c594 google::LogMessage::SendToLog()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93c3e0 google::LogMessage::Flush()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x90e327 hl_stream_synchronize()
@ 0x90e327 hl_stream_synchronize()
@ 0x90e327 hl_stream_synchronize()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x93f487 google::LogMessageFatal::~LogMessageFatal()
@ 0x90e327 hl_stream_synchronize()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x90e327 hl_stream_synchronize()
@ 0x90e327 hl_stream_synchronize()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x90e327 hl_stream_synchronize()
@ 0x90e327 hl_stream_synchronize()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x88dcee paddle::Argument::resizeAndCopyFrom()
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bc9f2 paddle::TrainerThread::copyInArgs()
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x7f8742deb230 (unknown)
@ 0x7f8742deb230 (unknown)
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x7f8742deb230 (unknown)
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x6bcd9d paddle::TrainerThread::forward()
@ 0x7f8742deb230 (unknown)
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x7f8743c86dc5 start_thread
@ 0x7f8743c86dc5 start_thread
@ 0x7f8743c86dc5 start_thread
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x6bf385 paddle::TrainerThread::computeThread()
@ 0x7f8743c86dc5 start_thread
@ 0x7f874255473d __clone
@ 0x7f874255473d __clone
@ 0x7f8742deb230 (unknown)
@ 0x7f874255473d __clone
@ 0x7f8742deb230 (unknown)
@ 0x7f8742deb230 (unknown)
@ 0x7f8742deb230 (unknown)
@ 0x7f874255473d __clone
@ (nil) (unknown)
@ (nil) (unknown)