训练开始出现Cuda Error: out of memory
Created by: 930083287
I1019 09:20:42.064462 82681 Util.cpp:166] commandline: --use_gpu=True --trainer_count=1 [INFO 2018-10-19 09:20:44,765 layers.py:2714] output for conv_0: c = 64, h = 500, w = 500, size = 16000000 [INFO 2018-10-19 09:20:44,767 layers.py:2714] output for conv_1: c = 64, h = 500, w = 500, size = 16000000 [INFO 2018-10-19 09:20:44,767 layers.py:2856] output for pool_0: c = 64, h = 250, w = 250, size = 4000000 [INFO 2018-10-19 09:20:44,768 layers.py:2714] output for conv_2: c = 128, h = 250, w = 250, size = 8000000 [INFO 2018-10-19 09:20:44,769 layers.py:2714] output for conv_3: c = 128, h = 250, w = 250, size = 8000000 [INFO 2018-10-19 09:20:44,770 layers.py:2856] output for pool_1: c = 128, h = 125, w = 125, size = 2000000 [INFO 2018-10-19 09:20:44,770 layers.py:2714] output for conv_4: c = 256, h = 125, w = 125, size = 4000000 [INFO 2018-10-19 09:20:44,771 layers.py:2714] output for conv_5: c = 256, h = 125, w = 125, size = 4000000 [INFO 2018-10-19 09:20:44,772 layers.py:2714] output for conv_6: c = 256, h = 125, w = 125, size = 4000000 [INFO 2018-10-19 09:20:44,773 layers.py:2856] output for pool_2: c = 256, h = 63, w = 63, size = 1016064 [INFO 2018-10-19 09:20:44,773 layers.py:2714] output for conv_7: c = 512, h = 63, w = 63, size = 2032128 [INFO 2018-10-19 09:20:44,774 layers.py:2714] output for conv_8: c = 512, h = 63, w = 63, size = 2032128 [INFO 2018-10-19 09:20:44,775 layers.py:2714] output for conv_9: c = 512, h = 63, w = 63, size = 2032128 [INFO 2018-10-19 09:20:44,776 layers.py:2856] output for pool_3: c = 512, h = 32, w = 32, size = 524288 [INFO 2018-10-19 09:20:44,820 layers.py:2714] output for conv_10: c = 512, h = 32, w = 32, size = 524288 [INFO 2018-10-19 09:20:44,821 layers.py:2714] output for conv_11: c = 512, h = 32, w = 32, size = 524288 [INFO 2018-10-19 09:20:44,822 layers.py:2714] output for conv_12: c = 512, h = 32, w = 32, size = 524288 [INFO 2018-10-19 09:20:44,823 layers.py:2856] output for pool_4: c = 512, h = 16, w = 16, size = 131072 cost F1019 09:20:44.874554 82681 hl_cuda_device.cc:273] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory *** Check failure stack trace: *** @ 0x7fb708336bcd google::LogMessage::Fail() @ 0x7fb70833a67c google::LogMessage::SendToLog() @ 0x7fb7083366f3 google::LogMessage::Flush() @ 0x7fb70833bb8e google::LogMessageFatal::~LogMessageFatal() @ 0x7fb7082efcd4 hl_malloc_device() @ 0x7fb708138646 paddle::GpuAllocator::alloc() @ 0x7fb70816cdaf paddle::PoolAllocator::alloc() @ 0x7fb708125504 paddle::GpuMemoryHandle::GpuMemoryHandle() @ 0x7fb70813078e paddle::GpuVectorT<>::GpuVectorT() @ 0x7fb708130fb8 paddle::VectorT<>::create() @ 0x7fb708131369 paddle::VectorT<>::createParallelVector() @ 0x7fb70803e267 paddle::Parameter::enableType() @ 0x7fb70803c2ce paddle::NeuralNetwork::init() @ 0x7fb70806558f paddle::GradientMachine::create() @ 0x7fb708313365 GradientMachine::createFromPaddleModelPtr() @ 0x7fb70831354f GradientMachine::createByConfigProtoStr() @ 0x7fb707ebe007 _wrap_GradientMachine_createByConfigProtoStr @ 0x7fb75ba00990 PyEval_EvalFrameEx @ 0x7fb75ba034e9 PyEval_EvalCodeEx
为什么刚刚开始就报显卡不足呀?