Created by: blood0708

I1228 01:28:25.300649 173 Util.cpp:166] commandline: --use_gpu=1 --rnn_use_batch=True --log_clipping=True --trainer_count=1 F1228 01:28:25.392861 173 hl_cuda_device.cc:399] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory

* Check failure stack trace: *

@ 0x7febaa61804d google::LogMessage::Fail() @ 0x7febaa61a398 google::LogMessage::SendToLog() @ 0x7febaa617b5b google::LogMessage::Flush() @ 0x7febaa61b26e google::LogMessageFatal::~LogMessageFatal() @ 0x7febaa5c39f1 hl_create_global_resources() @ 0x7febaa5c41a4 hl_specify_devices_start() @ 0x7febaa5c447d hl_start() @ 0x7febaa545e7e paddle::initMain() @ 0x7febaa5fe621 initPaddle() @ 0x7febaa09a347 _wrap_initPaddle @ 0x4c468a PyEval_EvalFrameEx @ 0x4c9d8f PyEval_EvalFrameEx @ 0x4c2765 PyEval_EvalCodeEx @ 0x4de6fe (unknown) @ 0x4b0cb3 PyObject_Call @ 0x4c6ad1 PyEval_EvalFrameEx @ 0x4c2765 PyEval_EvalCodeEx @ 0x4ca8d1 PyEval_EvalFrameEx @ 0x4c2765 PyEval_EvalCodeEx @ 0x4ca8d1 PyEval_EvalFrameEx @ 0x4c2765 PyEval_EvalCodeEx @ 0x4c2509 PyEval_EvalCode @ 0x4f1def (unknown) @ 0x4ec652 PyRun_FileExFlags @ 0x4eae31 PyRun_SimpleFileExFlags @ 0x49e14a Py_Main @ 0x7febd7056830 __libc_start_main @ 0x49d9d9 _start @ (nil) (unknown) ./run_train.sh: line 33: 173 Aborted (core dumped) CUDA_VISIBLE_DEVICES=0 python -u train.py --batch_size=1 --trainer_count=1 --num_passes=20 --num_proc_data=1 --num_conv_layers=2 --num_rnn_layers=3 --rnn_layer_size=2048 --num_iter_print=100 --learning_rate=1e-5 --max_duration=27.0 --min_duration=0.0 --test_off=False --use_sortagrad=True --use_gru=False --use_gpu=True --is_local=True --share_rnn_weights=True --train_manifest='data/tiny/manifest.tiny' --dev_manifest='data/tiny/manifest.tiny' --mean_std_path='data/tiny/mean_std.npz' --vocab_path='data/tiny/vocab.txt' --output_model_dir='./checkpoints/tiny' --augment_conf_path='conf/augmentation.config' --specgram_type='linear' --shuffle_method='batch_shuffle_clipped' Fail in training!