Created by: jiayifang1

gpu单卡运行，11G，能够正常测试预发布模型，但是在运行训练代码过程中： export CUDA_VISIBLE_DEVICES=0 python -u train.py --batch_size=3 --pretrained_model=vgg_ilsvrc_16_fc_reduced 执行这两个步骤之后，也是正常进入训练的，但是训练过程中是如下这样显示的！出现".......MemoryError"这样的问题，并卡在最后的训练处，不再继续进行，希望能得到大家的帮助！！！

Pass 0, batch 1890, face loss 3.608721, head loss 3.827300, time 0.82712 Pass 0, batch 1900, face loss 6.004067, head loss 6.459546, time 0.81461 Pass 0, batch 1910, face loss 4.193470, head loss 3.602981, time 0.82136 Process Process-7: Traceback (most recent call last): File "/home/jyf/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/jyf/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/jyf/anaconda3/lib/python3.6/site-packages/paddle/reader/decorator.py", line 393, in _read_into_queue for sample in reader(): File "/home/jyf/JYF/Pyramidbox/models/PaddleCV/face_detection/reader.py", line 267, in reader image_path) File "/home/jyf/JYF/Pyramidbox/models/PaddleCV/face_detection/reader.py", line 116, in preprocess settings.min_face_size) File "/home/jyf/JYF/Pyramidbox/models/PaddleCV/face_detection/image_util.py", line 424, in crop_image_sampling sample_img = np.zeros((height, width, 3)) MemoryError Pass 0, batch 1920, face loss 4.207895, head loss 3.901461, time 0.81387 Pass 0, batch 1930, face loss 4.981308, head loss 4.160357, time 0.82922 Pass 0, batch 1940, face loss 4.242156, head loss 3.531482, time 0.89638 Pass 0, batch 1950, face loss 5.884556, head loss 3.824725, time 0.81456 Pass 0, batch 1960, face loss 3.307667, head loss 2.812590, time 0.81372 Pass 0, batch 1970, face loss 4.135998, head loss 4.142908, time 0.81373 Pass 0, batch 1980, face loss 3.992738, head loss 2.869507, time 0.81205 Pass 0, batch 1990, face loss 3.689611, head loss 3.299284, time 0.81052 Pass 0, batch 2000, face loss 2.499412, head loss 2.287693, time 0.81455 Pass 0, batch 2010, face loss 1.459440, head loss 1.841028, time 0.81367 Pass 0, batch 2020, face loss 2.988102, head loss 2.815126, time 0.83127 Pass 0, batch 2030, face loss 4.024381, head loss 3.740697, time 0.82407 Pass 0, batch 2040, face loss 3.497251, head loss 3.646049, time 0.80784 Pass 0, batch 2050, face loss 3.888634, head loss 3.634361, time 0.88443 Pass 0, batch 2060, face loss 4.123483, head loss 3.784822, time 0.81228 Pass 0, batch 2070, face loss 3.865510, head loss 3.366984, time 0.80212 Process Process-3: Traceback (most recent call last): File "/home/jyf/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/jyf/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/jyf/anaconda3/lib/python3.6/site-packages/paddle/reader/decorator.py", line 393, in _read_into_queue for sample in reader(): File "/home/jyf/JYF/Pyramidbox/models/PaddleCV/face_detection/reader.py", line 267, in reader image_path) File "/home/jyf/JYF/Pyramidbox/models/PaddleCV/face_detection/reader.py", line 116, in preprocess settings.min_face_size) File "/home/jyf/JYF/Pyramidbox/models/PaddleCV/face_detection/image_util.py", line 424, in crop_image_sampling sample_img = np.zeros((height, width, 3)) MemoryError Pass 0, batch 2080, face loss 4.575073, head loss 3.918978, time 1.04506 Pass 0, batch 2090, face loss 4.212999, head loss 3.976611, time 0.81284 Pass 0, batch 2100, face loss 4.210123, head loss 3.808105, time 0.80688 Pass 0, batch 2110, face loss 3.437980, head loss 2.679230, time 0.81080 Pass 0, batch 2120, face loss 2.723385, head loss 2.337259, time 0.80899 Pass 0, batch 2130, face loss 5.665165, head loss 5.790230, time 0.81662 Pass 0, batch 2140, face loss 5.166775, head loss 5.815830, time 0.81377 Pass 0, batch 2150, face loss 4.529456, head loss 4.198637, time 0.81539 Pass 0, batch 2160, face loss 1.446540, head loss 1.389654, time 0.85053 Pass 0, batch 2170, face loss 1.424659, head loss 1.341642, time 0.81604