Created by: lightsailpro

Trying to use the generate_sequence_by_rnn_lm model for sequence prediction. But I have large vocabulary size about half million (non word). The GPU card has 12GB ram. I got memory error even with small batch size 32. Can this model handle large dictionary size - max_word_num = 500000?

INFO 2017-08-15 12:57:05,733 train.py:95] dictionay size = 461546
I0815 12:57:05.739673  4302 Util.cpp:166] commandline:  --use_gpu=True --trainer_count=1
I0815 12:57:07.165385  4302 GradientMachine.cpp:85] Initing parameters..
I0815 12:57:25.333416  4302 GradientMachine.cpp:92] Init parameters done.
[INFO 2017-08-15 12:57:25,333 train.py:78] start training...
[INFO 2017-08-15 12:57:28,632 train.py:58] Pass 0, Batch 0, Cost 257.150330, {'classification_error_evaluator': 1.0}
F0815 12:57:49.568514  4302 hl_cuda_device.cc:273] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory
*** Check failure stack trace: ***
    @     0x7f96cf30c8bd  google::LogMessage::Fail()
    @     0x7f96cf310415  google::LogMessage::SendToLog()
    @     0x7f96cf30c3b3  google::LogMessage::Flush()
    @     0x7f96cf31192e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f96cf2ce79c  hl_malloc_device()
    @     0x7f96cf18fd06  paddle::GpuAllocator::alloc()
    @     0x7f96cf260666  paddle::PoolAllocator::alloc()
    @     0x7f96cf163614  paddle::GpuMemoryHandle::GpuMemoryHandle()
    @     0x7f96cf16d094  paddle::GpuMatrix::resize()
    @     0x7f96cf17db22  paddle::Matrix::resizeOrCreate()
    @     0x7f96cf09a1ad  paddle::Layer::resetSpecifyOutput()
    @     0x7f96cf09a311  paddle::Layer::reserveOutput()
    @     0x7f96cf00e63f  paddle::FullyConnectedLayer::forward()
    @     0x7f96cefdba10  paddle::NeuralNetwork::forward()
    @     0x7f96cefc9033  paddle::GradientMachine::forwardBackward()
    @     0x7f96cf2e7caa  GradientMachine::forwardBackward()
    @     0x7f96cef609ee  _wrap_GradientMachine_forwardBackward
    @           0x52714b  PyEval_EvalFrameEx
    @           0x555551  PyEval_EvalCodeEx
    @           0x525560  PyEval_EvalFrameEx
    @           0x555551  PyEval_EvalCodeEx
    @           0x524338  PyEval_EvalFrameEx
    @           0x555551  PyEval_EvalCodeEx
    @           0x524338  PyEval_EvalFrameEx
    @           0x555551  PyEval_EvalCodeEx
    @           0x524338  PyEval_EvalFrameEx
    @           0x5247ea  PyEval_EvalFrameEx
    @           0x567d14  (unknown)
    @           0x465bf4  PyRun_FileExFlags
    @           0x46612d  PyRun_SimpleFileExFlags
    @           0x466d92  Py_Main
    @     0x7f96d0ceaf45  __libc_start_main
Aborted (core dumped)