Created by: imistyrain

运行一段时间就会报错，训练过程中可见内存占用持续上涨

2020-06-05 19:25:09,136-INFO: Total number of images: 60, inference time: 63.622197896099365 fps.
2020-06-05 19:25:09,136-INFO: Start evaluate...
2020-06-05 19:25:09,161-INFO: Accumulating evaluatation results...
2020-06-05 19:25:09,163-INFO: mAP(0.50, 11point) = 63.47
2020-06-05 19:25:09,163-INFO: Best test box ap: 68.54552720244386, in iter: 10800
2020-06-05 19:25:10,583-INFO: iter: 12020, lr: 0.000010, 'loss': '4.558191', time: 0.287, eta: 66 days, 9:32:59
2020-06-05 19:25:11,996-INFO: iter: 12040, lr: 0.000010, 'loss': '5.984420', time: 0.071, eta: 16 days, 8:14:02
2020-06-05 19:25:13,385-INFO: iter: 12060, lr: 0.000010, 'loss': '5.201263', time: 0.070, eta: 16 days, 1:53:01
2020-06-05 19:25:14,790-INFO: iter: 12080, lr: 0.000010, 'loss': '3.418216', time: 0.069, eta: 15 days, 21:57:56
2020-06-05 19:25:16,153-INFO: iter: 12100, lr: 0.000010, 'loss': '4.669427', time: 0.070, eta: 16 days, 2:12:23
2020-06-05 19:25:17,634-INFO: iter: 12120, lr: 0.000010, 'loss': '5.519851', time: 0.072, eta: 16 days, 14:26:03
2020-06-05 19:25:19,031-INFO: iter: 12140, lr: 0.000010, 'loss': '4.321500', time: 0.071, eta: 16 days, 11:17:47
2020-06-05 19:25:20,356-INFO: iter: 12160, lr: 0.000010, 'loss': '5.364328', time: 0.067, eta: 15 days, 12:12:26
2020-06-05 19:25:21,707-INFO: iter: 12180, lr: 0.000010, 'loss': '3.863911', time: 0.068, eta: 15 days, 15:37:45
2020-06-05 19:25:23,086-INFO: iter: 12200, lr: 0.000010, 'loss': '4.006394', time: 0.068, eta: 15 days, 17:46:28
2020-06-05 19:25:23,086-INFO: Save model to output/yolov3_mobilenet_v1_fruit/12200.
2020-06-05 19:25:25,326-WARNING: fail to map op [ExpandImage_2f41a4] with error:  and stack:
Traceback (most recent call last):
  File "/home/ar/gongyanhe/PaddleDetection/ppdet/data/reader.py", line 45, in __call__
    data = f(data, ctx)
  File "/home/ar/gongyanhe/PaddleDetection/ppdet/data/transform/operators.py", line 731, in __call__
    expand_im = np.uint8(expand_im * np.squeeze(self.mean))
MemoryError