Memory leaking of deepspeech2
Created by: tensor-tang
Here is my command:
python train.py --use_gpu=False --num_passes=1 --trainer_count=1 --batch_size=512 --num_iter_print=1 --num_proc_data=4
I did not train with GPU, and only give 1trainer_count
and 4 num_proc_data
, which should makes the thing easy.
Total available memory of my system is 62.6G.
When stat to train, I found the memory percent from ~50% increasing to 99%, then killed by system.
Is this possible that some memory leaking ?
Pass: 0, Batch: 1, TrainCost: 221.743195 I0907 17:58:52.262146 50789 Thread.h:271] SyncThreadPool worker thread 0
Pass: 0, Batch: 2, TrainCost: 745.176697
Pass: 0, Batch: 3, TrainCost: 217.676758 Killed
PaddlePaddle version:
commit a072ab9e74227361421e6b1fdc167b6709fcdf87 Date: Thu Sep 7 11:10:47 2017 +0800
DeepSpeech2 version:
commit acea40b6 Date: Wed Sep 6 20:32:20 2017 +0800