Check failed: ptr Fail to allocate GPU memory 2048 bytes
Created by: Ayle3125
root@833fa085bd51:/home/code/traffic_prediction# sh train.sh
I0221 20:11:44.045990 62 Util.cpp:155] commandline: /usr/local/bin/../opt/paddle/bin/paddle_trainer --config=trainer_config.py --save_dir=./output --trainer_count=1 --log_period=1000 --dot_period=10 --num_passes=10 --use_gpu=true --show_parameter_stats_period=3000
I0221 20:11:44.046171 62 Util.cpp:130] Calling runInitFunctions
I0221 20:11:44.046438 62 Util.cpp:143] Call runInitFunctions done.
[WARNING 2017-02-21 20:11:44,148 networks.py:1438] outputs
routine try to calculate network's inputs and outputs order. It might not work well.Please see follow log carefully.
[INFO 2017-02-21 20:11:44,148 networks.py:1466] The input order is [link_encode, label_5min, label_10min, label_15min, label_20min, label_25min, label_30min, label_35min, label_40min, label_45min, label_50min, label_55min, label_60min, label_65min, label_70min, label_75min, label_80min, label_85min, label_90min, label_95min, label_100min, label_105min, label_110min, label_115min, label_120min]
[INFO 2017-02-21 20:11:44,148 networks.py:1472] The output order is [cost_5min, cost_10min, cost_15min, cost_20min, cost_25min, cost_30min, cost_35min, cost_40min, cost_45min, cost_50min, cost_55min, cost_60min, cost_65min, cost_70min, cost_75min, cost_80min, cost_85min, cost_90min, cost_95min, cost_100min, cost_105min, cost_110min, cost_115min, cost_120min]
I0221 20:11:44.162466 62 Trainer.cpp:170] trainer mode: Normal
F0221 20:11:44.162664 62 Allocator.h:84] Check failed: ptr Fail to allocate GPU memory 2048 bytes
*** Check failure stack trace: ***
@ 0x7f40fcb02daa (unknown)
@ 0x7f40fcb02ce4 (unknown)
@ 0x7f40fcb026e6 (unknown)
@ 0x7f40fcb05687 (unknown)
@ 0x75f3de paddle::GpuAllocator::alloc()
@ 0x75e496 paddle::PoolAllocator::alloc()
@ 0x75c5bf paddle::GpuMemoryHandle::GpuMemoryHandle()
@ 0x7570fe paddle::GpuVectorT<>::GpuVectorT()
@ 0x7572b8 paddle::VectorT<>::create()
@ 0x757369 paddle::VectorT<>::createParallelVector()
@ 0x53ad96 paddle::Parameter::enableType()
@ 0x5376dc paddle::parameterInitNN()
@ 0x539923 paddle::NeuralNetwork::init()
@ 0x53f093 paddle::GradientMachine::create()
@ 0x67bc98 paddle::TrainerInternal::init()
@ 0x6783ee paddle::Trainer::init()
@ 0x5132a9 main
@ 0x7f40fbd0ef45 (unknown)
@ 0x51f2a5 (unknown)
@ (nil) (unknown)
/usr/local/bin/paddle: line 109: 62 Aborted (core dumped) ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}
root@833fa085bd51:/home/code/traffic_pre
plz