Created by: yhzhangyu

能正常跑一会;然后就出现如下错误;

配置信息： architecture: YOLOv3 train_feed: YoloTrainFeed eval_feed: YoloEvalFeed test_feed: YoloTestFeed use_gpu: true max_iters: 20000 log_smooth_window: 20 save_dir: output snapshot_iter: 200 metric: VOC map_type: 11point pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/yolov3_mobilenet_v1.tar weights: output/yolov3_mobilenet_v1_fruit/best_model num_classes: 3 finetune_exclude_pretrained_params: ['yolo_output']

YOLOv3: backbone: MobileNet yolo_head: YOLOv3Head

MobileNet: norm_type: sync_bn norm_decay: 0. conv_group_scale: 1 with_extra_blocks: false

YOLOv3Head: anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]] anchors: [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45], [59, 119], [116, 90], [156, 198], [373, 326]] norm_decay: 0. ignore_thresh: 0.7 label_smooth: true nms: background_label: -1 keep_top_k: 100 nms_threshold: 0.45 nms_top_k: 1000 normalized: false score_threshold: 0.01

LearningRate: base_lr: 0.000002 schedulers:

• !PiecewiseDecay gamma: 0.1 milestones:
  • 15000
  • 18000
• !LinearWarmup start_factor: 0. steps: 100

OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0005 type: L2

YoloTrainFeed: batch_size: 1 dataset: dataset_dir: dataset/fruit/fruit-detection annotation: train.txt use_default_label: false num_workers: 16 bufsize: 128 use_process: true mixup_epoch: -1 sample_transforms:

• !DecodeImage to_rgb: true with_mixup: false
• !NormalizeBox {}
• !ExpandImage max_ratio: 4.0 mean: [123.675, 116.28, 103.53] prob: 0.5
• !RandomInterpImage max_size: 0 target_size: 608
• !RandomFlipImage is_mask_flip: false is_normalized: true prob: 0.5
• !NormalizeImage is_channel_first: false is_scale: true mean:
  • 0.485
  • 0.456
  • 0.406 std:
  • 0.229
  • 0.224
  • 0.225
• !Permute channel_first: true to_bgr: false batch_transforms:
• !RandomShape sizes: [608] with_background: false

YoloEvalFeed: batch_size: 1 image_shape: [3, 608, 608] dataset: dataset_dir: dataset/fruit/fruit-detection annotation: val.txt use_default_label: false

YoloTestFeed: batch_size: 1 image_shape: [3, 608, 608] dataset: dataset_dir: dataset/fruit use_default_label: false

报错信息如下：

I1120 17:06:49.690585 9759 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True I1120 17:06:49.924094 9759 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 2019-11-20 17:06:50,362-INFO: iter: 0, lr: 0.000000, 'loss': '16984.570312', time: 0.163, eta: 0:54:11 2019-11-20 17:06:52,587-INFO: iter: 20, lr: 0.000000, 'loss': '9977.352539', time: 0.190, eta: 1:03:09 2019-11-20 17:06:54,451-INFO: iter: 40, lr: 0.000001, 'loss': '673.993042', time: 0.094, eta: 0:31:10 2019-11-20 17:06:56,082-INFO: iter: 60, lr: 0.000001, 'loss': '166.233780', time: 0.082, eta: 0:27:06 2019-11-20 17:06:57,709-INFO: iter: 80, lr: 0.000002, 'loss': '103.996521', time: 0.081, eta: 0:26:59 2019-11-20 17:06:59,353-INFO: iter: 100, lr: 0.000002, 'loss': '72.225800', time: 0.082, eta: 0:27:10 2019-11-20 17:07:00,999-INFO: iter: 120, lr: 0.000002, 'loss': '57.962166', time: 0.083, eta: 0:27:22 2019-11-20 17:07:02,637-INFO: iter: 140, lr: 0.000002, 'loss': '52.966858', time: 0.082, eta: 0:27:05 2019-11-20 17:07:04,310-INFO: iter: 160, lr: 0.000002, 'loss': '51.043819', time: 0.083, eta: 0:27:31 2019-11-20 17:07:06,543-INFO: iter: 180, lr: 0.000002, 'loss': '44.794296', time: 0.107, eta: 0:35:12 2019-11-20 17:07:09,575-INFO: iter: 200, lr: 0.000002, 'loss': '38.660149', time: 0.153, eta: 0:50:26 2019-11-20 17:07:09,703-INFO: Save model to output/yolov3_mobilenet_v1_fruit/200. 2019-11-20 17:07:14,636-INFO: iter: 220, lr: 0.000002, 'loss': '36.993340', time: 0.256, eta: 1:24:30 2019-11-20 17:07:17,816-WARNING: fail to map op [ExpandImage_909a9c] with error: Unable to allocate array with shape (10016, 10016, 3) and data type float64 and stack: Traceback (most recent call last): File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 44, in _proxy_method return func(*args, **kwargs) File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/dataset.py", line 48, in reset (self.class.name)) NotImplementedError: MappedDataset.reset not available

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 44, in _proxy_method return func(*args, **kwargs) File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/dataset.py", line 48, in reset (self.class.name)) NotImplementedError: BatchedDataset.reset not available

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/init.py", line 77, in _mapper out = f(sample, ctx) File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/operators.py", line 611, in call expand_im = np.uint8(expand_im * np.squeeze(self.mean)) MemoryError: Unable to allocate array with shape (10016, 10016, 3) and data type float64

2019-11-20 17:07:17,905-INFO: iter: 240, lr: 0.000002, 'loss': '35.379051', time: 0.164, eta: 0:54:03 2019-11-20 17:07:19,741-INFO: iter: 260, lr: 0.000002, 'loss': '41.599369', time: 0.092, eta: 0:30:16 2019-11-20 17:07:21,477-INFO: iter: 280, lr: 0.000002, 'loss': '28.459013', time: 0.087, eta: 0:28:27 2019-11-20 17:07:23,173-INFO: iter: 300, lr: 0.000002, 'loss': '29.857658', time: 0.085, eta: 0:27:54 2019-11-20 17:07:24,858-INFO: iter: 320, lr: 0.000002, 'loss': '31.095598', time: 0.084, eta: 0:27:36 2019-11-20 17:07:26,532-INFO: iter: 340, lr: 0.000002, 'loss': '25.751993', time: 0.084, eta: 0:27:25 2019-11-20 17:07:28,204-INFO: iter: 360, lr: 0.000002, 'loss': '34.456688', time: 0.084, eta: 0:27:21 2019-11-20 17:07:28,962-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64 2019-11-20 17:07:29,879-INFO: iter: 380, lr: 0.000002, 'loss': '23.786648', time: 0.084, eta: 0:27:23 2019-11-20 17:07:31,552-INFO: iter: 400, lr: 0.000002, 'loss': '23.337822', time: 0.084, eta: 0:27:18 2019-11-20 17:07:31,569-INFO: Save model to output/yolov3_mobilenet_v1_fruit/400. 2019-11-20 17:07:32,793-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits] 2019-11-20 17:07:32,794-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits] 2019-11-20 17:07:32,797-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits] 2019-11-20 17:07:32,798-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits] 2019-11-20 17:07:32,800-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits] 2019-11-20 17:07:32,802-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits] 2019-11-20 17:07:32,803-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits] 2019-11-20 17:07:32,805-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits] 2019-11-20 17:07:32,806-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits] 2019-11-20 17:07:32,807-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits][consumer[consumer-1f9_14] exits] 2019-11-20 17:07:32,823-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits][consumer[consumer-1f9_14] exits][consumer[consumer-1f9_13] exits] 2019-11-20 17:07:32,825-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits][consumer[consumer-1f9_14] exits][consumer[consumer-1f9_13] exits][consumer[consumer-1f9_15] exits] 2019-11-20 17:07:32,843-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits][consumer[consumer-1f9_14] exits][consumer[consumer-1f9_13] exits][consumer[consumer-1f9_15] exits][consumer[consumer-1f9_11] exits] 2019-11-20 17:07:32,845-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits][consumer[consumer-1f9_14] exits][consumer[consumer-1f9_13] exits][consumer[consumer-1f9_15] exits][consumer[consumer-1f9_11] exits][consumer[consumer-1f9_7] exits] 2019-11-20 17:07:32,856-WARNING: consumer failed with error: failed to map consumer[%s], error: Unable to allocate array with shape (10016, 10016, 3) and data type float64[consumer[consumer-1f9_2] exits][consumer[consumer-1f9_12] exits][consumer[consumer-1f9_5] exits][consumer[consumer-1f9_3] exits][consumer[consumer-1f9_4] exits][consumer[consumer-1f9_10] exits][consumer[consumer-1f9_6] exits][consumer[consumer-1f9_9] exits][consumer[consumer-1f9_0] exits][consumer[consumer-1f9_14] exits][consumer[consumer-1f9_13] exits][consumer[consumer-1f9_15] exits][consumer[consumer-1f9_11] exits][consumer[consumer-1f9_7] exits][consumer[consumer-1f9_1] exits] 2019-11-20 17:07:32,857-WARNING: Your reader has raised an exception! Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/reader.py", line 488, in thread_main six.reraise(*sys.exc_info()) File "/home/zhangyu/.local/lib/python3.5/site-packages/six.py", line 696, in reraise raise value File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/reader.py", line 468, in thread_main for tensors in self._tensor_reader(): File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/reader.py", line 542, in tensor_reader_impl for slots in paddle_reader(): File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/data_feeder.py", line 454, in reader_creator for item in reader(): File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/reader.py", line 103, in _reader for _batch in batched_ds: File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/dataset.py", line 30, in next return self.next() File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 44, in _proxy_method return func(*args, **kwargs) File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 57, in next sample = self._ds.next() File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 44, in _proxy_method return func(*args, **kwargs) File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 99, in next out = self._ds.next() File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/transformer.py", line 44, in _proxy_method return func(*args, **kwargs) File "/home/zhangyu/Paddle/models/PaddleCV/PaddleDetection/ppdet/data/transform/parallel_map.py", line 187, in next raise ValueError("all consumers exited, no more samples") ValueError: all consumers exited, no more samples

2019-11-20 17:07:33,468-INFO: iter: 420, lr: 0.000002, 'loss': '33.105843', time: 0.096, eta: 0:31:16 2019-11-20 17:07:35,167-INFO: iter: 440, lr: 0.000002, 'loss': '21.036983', time: 0.085, eta: 0:27:38 2019-11-20 17:07:36,835-INFO: iter: 460, lr: 0.000002, 'loss': '21.888847', time: 0.084, eta: 0:27:15 Traceback (most recent call last): File "tools/train.py", line 358, in main() File "tools/train.py", line 245, in main outs = exe.run(compiled_train_prog, fetch_list=train_values) File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/executor.py", line 775, in run six.reraise(*sys.exc_info()) File "/home/zhangyu/.local/lib/python3.5/site-packages/six.py", line 696, in reraise raise value File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/executor.py", line 770, in run use_program_cache=use_program_cache) File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/executor.py", line 829, in _run_impl return_numpy=return_numpy) File "/home/zhangyu/.local/lib/python3.5/site-packages/paddle/fluid/executor.py", line 669, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EOFException: There is no next data. at [/paddle/paddle/fluid/operators/reader/read_op.cc:90]