Created by: wchange

启动命令： CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python3.7 tools/train.py -c configs/ppyolo/ppyolo_2x_gf2.yml --eval

日志：

2020-08-26 02:15:01,572-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000500] in Optimizer will not take effect, and it will only be applied to other Parameters!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
2020-08-26 02:15:07,757-INFO: places would be ommited when DataLoader is not iterable
W0826 02:15:08.004554 15971 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.2, Runtime API Version: 10.0
W0826 02:15:08.022433 15971 device_context.cc:260] device: 0, cuDNN Version: 7.6.
2020-08-26 02:15:12,193-WARNING: /root/.cache/paddle/weights/ResNet50_vd_ssld_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
/usr/local/lib/python3.7/site-packages/paddle/fluid/io.py:1998: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc_0.b_0 fc_0.w_0
  format(" ".join(unused_para_list)))
loading annotations into memory...
Done (t=0.04s)
creating index...
index created!
2020-08-26 02:15:13,240-WARNING: Found an invalid bbox in annotations: im_id: 53, area: 0.0 x1: 0, y1: 195, x2: 0, y2: 360.
W0826 02:15:31.991529 15971 init.cc:226] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0826 02:15:31.991603 15971 init.cc:228] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0826 02:15:31.991616 15971 init.cc:231] The detail failure signal is:

W0826 02:15:31.991628 15971 init.cc:234] *** Aborted at 1598408131 (unix time) try "date -d @1598408131" if you are using GNU date ***
W0826 02:15:31.993551 15971 init.cc:234] PC: @                0x0 (unknown)
W0826 02:15:31.993789 15971 init.cc:234] *** SIGBUS (@0x7f1b91277000) received by PID 15971 (TID 0x7f1dcde7e700) from PID 18446744071849865216; stack trace: ***
W0826 02:15:31.995285 15971 init.cc:234]     @     0x7f1dcd506390 (unknown)
W0826 02:15:31.996726 15971 init.cc:234]     @     0x7f1dcd29daf8 (unknown)
W0826 02:15:31.997678 15971 init.cc:234]     @     0x7f1d87192e40 ffi_call_unix64
W0826 02:15:31.998585 15971 init.cc:234]     @     0x7f1d871928ab ffi_call
W0826 02:15:31.999513 15971 init.cc:234]     @     0x7f1d873a6f4f _ctypes_callproc
W0826 02:15:32.000423 15971 init.cc:234]     @     0x7f1d8739dca3 PyCFuncPtr_call
W0826 02:15:32.002307 15971 init.cc:234]     @     0x7f1dcd7ac024 _PyObject_FastCallKeywords
W0826 02:15:32.003989 15971 init.cc:234]     @     0x7f1dcd7810db _PyEval_EvalFrameDefault
W0826 02:15:32.005344 15971 init.cc:234]     @     0x7f1dcd77a8c0 function_code_fastcall
W0826 02:15:32.006947 15971 init.cc:234]     @     0x7f1dcd782ae0 _PyEval_EvalFrameDefault
W0826 02:15:32.008316 15971 init.cc:234]     @     0x7f1dcd77a8c0 function_code_fastcall
W0826 02:15:32.009905 15971 init.cc:234]     @     0x7f1dcd782ae0 _PyEval_EvalFrameDefault
W0826 02:15:32.011255 15971 init.cc:234]     @     0x7f1dcd77a8c0 function_code_fastcall
W0826 02:15:32.012847 15971 init.cc:234]     @     0x7f1dcd78306c _PyEval_EvalFrameDefault
W0826 02:15:32.014436 15971 init.cc:234]     @     0x7f1dcd8904e4 _PyEval_EvalCodeWithName
W0826 02:15:32.016003 15971 init.cc:234]     @     0x7f1dcd7ab7c0 _PyFunction_FastCallDict
W0826 02:15:32.017597 15971 init.cc:234]     @     0x7f1dcd7aca2d _PyObject_Call_Prepend
W0826 02:15:32.018985 15971 init.cc:234]     @     0x7f1dcd80fc01 slot_tp_init
W0826 02:15:32.020416 15971 init.cc:234]     @     0x7f1dcd80a2d3 type_call
W0826 02:15:32.021998 15971 init.cc:234]     @     0x7f1dcd7ac024 _PyObject_FastCallKeywords
W0826 02:15:32.023636 15971 init.cc:234]     @     0x7f1dcd77feb1 _PyEval_EvalFrameDefault
W0826 02:15:32.025221 15971 init.cc:234]     @     0x7f1dcd8904e4 _PyEval_EvalCodeWithName
W0826 02:15:32.026798 15971 init.cc:234]     @     0x7f1dcd7ab7c0 _PyFunction_FastCallDict
W0826 02:15:32.028409 15971 init.cc:234]     @     0x7f1dcd7aca2d _PyObject_Call_Prepend
W0826 02:15:32.029791 15971 init.cc:234]     @     0x7f1dcd80fc01 slot_tp_init
W0826 02:15:32.031170 15971 init.cc:234]     @     0x7f1dcd80a2d3 type_call
W0826 02:15:32.032748 15971 init.cc:234]     @     0x7f1dcd7ac024 _PyObject_FastCallKeywords
W0826 02:15:32.034349 15971 init.cc:234]     @     0x7f1dcd77feb1 _PyEval_EvalFrameDefault
W0826 02:15:32.035683 15971 init.cc:234]     @     0x7f1dcd77a8c0 function_code_fastcall
W0826 02:15:32.037284 15971 init.cc:234]     @     0x7f1dcd78306c _PyEval_EvalFrameDefault
W0826 02:15:32.038868 15971 init.cc:234]     @     0x7f1dcd8904e4 _PyEval_EvalCodeWithName
W0826 02:15:32.040446 15971 init.cc:234]     @     0x7f1dcd7ab6b7 _PyFunction_FastCallDict
Bus error

configs/ppyolo/ppyolo_2x_gf2.yml：

architecture: YOLOv3
use_gpu: true
max_iters: 500000
log_smooth_window: 100
log_iter: 100
save_dir: output
snapshot_iter: 10000
metric: COCO
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_ssld_pretrained.tar
weights: output/ppyolo/model_final
num_classes: 80
use_fine_grained_loss: true
use_ema: true
ema_decay: 0.9998

YOLOv3:
  backbone: ResNet
  yolo_head: YOLOv3Head
  use_fine_grained_loss: true

ResNet:
  norm_type: sync_bn
  freeze_at: 0
  freeze_norm: false
  norm_decay: 0.
  depth: 50
  feature_maps: [3, 4, 5]
  variant: d
  dcn_v2_stages: [5]

YOLOv3Head:
  anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
  anchors: [[10, 13], [16, 30], [33, 23],
            [30, 61], [62, 45], [59, 119],
            [116, 90], [156, 198], [373, 326]]
  norm_decay: 0.
  coord_conv: true
  iou_aware: true
  iou_aware_factor: 0.4
  scale_x_y: 1.05
  spp: true
  yolo_loss: YOLOv3Loss
  nms: MatrixNMS
  drop_block: true

YOLOv3Loss:
  batch_size: 24
  ignore_thresh: 0.7
  scale_x_y: 1.05
  label_smooth: false
  use_fine_grained_loss: true
  iou_loss: IouLoss
  iou_aware_loss: IouAwareLoss

IouLoss:
  loss_weight: 2.5
  max_height: 608
  max_width: 608

IouAwareLoss:
  loss_weight: 1.0
  max_height: 608
  max_width: 608

MatrixNMS:
    background_label: -1
    keep_top_k: 100
    normalized: false
    score_threshold: 0.01
    post_threshold: 0.01

LearningRate:
  base_lr: 0.01
  schedulers:
  - !PiecewiseDecay
    gamma: 0.1
    milestones:
    - 400000
    - 450000
  - !LinearWarmup
    start_factor: 0.
    steps: 4000

OptimizerBuilder:
  optimizer:
    momentum: 0.9
    type: Momentum
  regularizer:
    factor: 0.0005
    type: L2

_READER_: 'ppyolo_reader_gf2.yml'

configs/ppyolo/ppyolo_reader_gf2.yml：

TrainReader:
  inputs_def:
    fields: ['image', 'gt_bbox', 'gt_class', 'gt_score']
    num_max_boxes: 50
  dataset:
    !COCODataSet
      image_dir: train
      anno_path: annotations/train.json
      dataset_dir: dataset/gf2
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
      with_mixup: True
    - !MixupImage
      alpha: 1.5
      beta: 1.5
    - !ColorDistort {}
    - !RandomExpand
      fill_value: [123.675, 116.28, 103.53]
    - !RandomCrop {}
    - !RandomFlipImage
      is_normalized: false
    - !NormalizeBox {}
    - !PadBox
      num_max_boxes: 50
    - !BboxXYXY2XYWH {}
  batch_transforms:
  - !RandomShape
    sizes: [320, 352, 384, 416, 448, 480, 512, 544, 576, 608]
    random_inter: True
  - !NormalizeImage
    mean: [0.485, 0.456, 0.406]
    std: [0.229, 0.224, 0.225]
    is_scale: True
    is_channel_first: false
  - !Permute
    to_bgr: false
    channel_first: True
  # Gt2YoloTarget is only used when use_fine_grained_loss set as true,
  # this operator will be deleted automatically if use_fine_grained_loss
  # is set as false
  - !Gt2YoloTarget
    anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
    anchors: [[10, 13], [16, 30], [33, 23],
              [30, 61], [62, 45], [59, 119],
              [116, 90], [156, 198], [373, 326]]
    downsample_ratios: [32, 16, 8]
  batch_size: 8
  shuffle: true
  mixup_epoch: 25000
  drop_last: true
  worker_num: 8
  bufsize: 4
  use_process: true

EvalReader:
  inputs_def:
    fields: ['image', 'im_size', 'im_id']
    num_max_boxes: 50
  dataset:
    !COCODataSet
      image_dir: val
      anno_path: annotations/val.json
      dataset_dir: dataset/gf2
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
    - !ResizeImage
      target_size: 608
      interp: 2
    - !NormalizeImage
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]
      is_scale: True
      is_channel_first: false
    - !PadBox
      num_max_boxes: 50
    - !Permute
      to_bgr: false
      channel_first: True
  batch_size: 8
  drop_empty: false
  worker_num: 8
  bufsize: 4

TestReader:
  inputs_def:
    image_shape: [3, 608, 608]
    fields: ['image', 'im_size', 'im_id']
  dataset:
    !ImageFolder
      anno_path: annotations/instances_val2017.json
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
    - !ResizeImage
      target_size: 608
      interp: 2
    - !NormalizeImage
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]
      is_scale: True
      is_channel_first: false
    - !Permute
      to_bgr: false
      channel_first: True
  batch_size: 1