Created by: lurenlym

ubuntu18.04 release0.4最新 GPU V100 自己数据集，四类

运行命令 CUDA_VISIBLE_DEVICES=1,2 python tools/train.py -c configs/ppyolo/ppyolo_grddc.yml --eval

错误

2020-09-21 15:48:39,420-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000500] in Optimizer will not take effect, and it will only be applied to other Parameters!
2020-09-21 15:48:46,059-INFO: places would be ommited when DataLoader is not iterable
W0921 15:48:46.160423  2666 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 10.2, Runtime API Version: 10.0
W0921 15:48:46.163522  2666 device_context.cc:260] device: 0, cuDNN Version: 8.0.
2020-09-21 15:48:48,595-WARNING: /root/.cache/paddle/weights/ResNet50_vd_ssld_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
/opt/conda/lib/python3.6/site-packages/paddle/fluid/io.py:1998: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc_0.w_0 fc_0.b_0
  format(" ".join(unused_para_list)))
2020-09-21 15:48:52,352-INFO: places would be ommited when DataLoader is not iterable
I0921 15:48:55.301230  2666 build_strategy.cc:361] set enable_sequential_execution:1
W0921 15:48:56.606209  2666 fuse_all_reduce_op_pass.cc:74] Find all_reduce operators: 240. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 171.
W0921 15:49:18.407598  2739 init.cc:226] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0921 15:49:18.407640  2739 init.cc:228] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0921 15:49:18.407650  2739 init.cc:231] The detail failure signal is:

W0921 15:49:18.407660  2739 init.cc:234] *** Aborted at 1600703358 (unix time) try "date -d @1600703358" if you are using GNU date ***
W0921 15:49:18.409673  2739 init.cc:234] PC: @                0x0 (unknown)
W0921 15:49:18.409899  2739 init.cc:234] *** SIGSEGV (@0x0) received by PID 2666 (TID 0x7f43e0ffd700) from PID 0; stack trace: ***
W0921 15:49:18.411737  2739 init.cc:234]     @     0x7f44ed38f8a0 (unknown)
W0921 15:49:18.413237  2739 init.cc:234]     @                0x0 (unknown)
Segmentation fault (core dumped)

配置文件
architecture: YOLOv3
use_gpu: true
max_iters: 250000
log_smooth_window: 100
log_iter: 100
save_dir: output
snapshot_iter: 10000
metric: VOC
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_ssld_pretrained.tar
weights: output/ppyolo/model_final
num_classes: 4
use_fine_grained_loss: true
use_ema: true
ema_decay: 0.9998

YOLOv3:
  backbone: ResNet
  yolo_head: YOLOv3Head
  use_fine_grained_loss: true

ResNet:
  norm_type: sync_bn
  freeze_at: 0
  freeze_norm: false
  norm_decay: 0.
  depth: 50
  feature_maps: [3, 4, 5]
  variant: d
  dcn_v2_stages: [5]

YOLOv3Head:
  anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
  anchors: [[10, 13], [16, 30], [33, 23],
            [30, 61], [62, 45], [59, 119],
            [116, 90], [156, 198], [373, 326]]
  norm_decay: 0.
  coord_conv: true
  iou_aware: true
  iou_aware_factor: 0.4
  scale_x_y: 1.05
  spp: true
  yolo_loss: YOLOv3Loss
  nms: MatrixNMS
  drop_block: true

YOLOv3Loss:
  ignore_thresh: 0.7
  scale_x_y: 1.05
  label_smooth: false
  use_fine_grained_loss: true
  iou_loss: IouLoss
  iou_aware_loss: IouAwareLoss

IouLoss:
  loss_weight: 2.5
  max_height: 608
  max_width: 608

IouAwareLoss:
  loss_weight: 1.0
  max_height: 608
  max_width: 608

MatrixNMS:
    background_label: -1
    keep_top_k: 100
    normalized: false
    score_threshold: 0.01
    post_threshold: 0.01

LearningRate:
  base_lr: 0.0025
  schedulers:
  - !PiecewiseDecay
    gamma: 0.1
    milestones:
    - 150000
    - 200000
  - !LinearWarmup
    start_factor: 0.
    steps: 4000

OptimizerBuilder:
  optimizer:
    momentum: 0.9
    type: Momentum
  regularizer:
    factor: 0.0005
    type: L2

_READER_: 'ppyolo_reader_grddc.yml'


TrainReader:
  inputs_def:
    fields: ['image', 'gt_bbox', 'gt_class', 'gt_score']
    num_max_boxes: 50
  dataset:
    !VOCDataSet
      anno_path: trainval.txt  
      dataset_dir: dataset/voc
      use_default_label: false   
      with_background: false  
  sample_transforms:
    - !DecodeImage
      to_rgb: True
      with_mixup: True
    - !MixupImage
      alpha: 1.5
      beta: 1.5
    - !ColorDistort {}
    - !RandomExpand
      fill_value: [123.675, 116.28, 103.53]
    - !RandomCrop {}
    - !RandomFlipImage
      is_normalized: false
    - !NormalizeBox {}
    - !PadBox
      num_max_boxes: 50
    - !BboxXYXY2XYWH {}
  batch_transforms:
  - !RandomShape
    sizes: [320, 352, 384, 416, 448, 480, 512, 544, 576, 608]
    random_inter: True
  - !NormalizeImage
    mean: [0.485, 0.456, 0.406]
    std: [0.229, 0.224, 0.225]
    is_scale: True
    is_channel_first: false
  - !Permute
    to_bgr: false
    channel_first: True
  # Gt2YoloTarget is only used when use_fine_grained_loss set as true,
  # this operator will be deleted automatically if use_fine_grained_loss
  # is set as false
  - !Gt2YoloTarget
    anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
    anchors: [[10, 13], [16, 30], [33, 23],
              [30, 61], [62, 45], [59, 119],
              [116, 90], [156, 198], [373, 326]]
    downsample_ratios: [32, 16, 8]
  batch_size: 12
  shuffle: true
  mixup_epoch: 25000
  drop_last: true
  worker_num: 1
  bufsize: 4
  use_process: false

EvalReader:
  inputs_def:
    fields: ['image', 'im_size', 'im_id']
    num_max_boxes: 50
  dataset:
    !VOCDataSet
      anno_path: test.txt
      dataset_dir: dataset/voc
      use_default_label: false
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
    - !ResizeImage
      target_size: 608
      interp: 2
    - !NormalizeImage
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]
      is_scale: True
      is_channel_first: false
    - !PadBox
      num_max_boxes: 50
    - !Permute
      to_bgr: false
      channel_first: True
  batch_size: 8
  drop_empty: false
  worker_num: 1
  bufsize: 4

TestReader:
  inputs_def:
    image_shape: [3, 608, 608]
    fields: ['image', 'im_size', 'im_id']
  dataset:
    !ImageFolder
      anno_path: annotations/instances_val2017.json
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
    - !ResizeImage
      target_size: 608
      interp: 2
    - !NormalizeImage
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]
      is_scale: True
      is_channel_first: false
    - !Permute
      to_bgr: false
      channel_first: True
  batch_size: 1

此外数据集存在空gt的情况，存在没有difficult标签的情况，因此更改如下文件：

PaddleDetection/ppdet/data/source/voc.py line145

#_difficult = int(obj.find('difficult').text)
_difficult = 0

PaddleDetection/ppdet/data/transform/operators.py Line1272

    gt_class1 = sample['gt_class'].reshape((-1, 1))
    gt_class2 = sample['mixup']['gt_class'].reshape((-1, 1))
    gt_class = np.concatenate((gt_class1, gt_class2), axis=0)

    gt_score1 = sample['gt_score'].reshape((-1, 1))
    gt_score2 = sample['mixup']['gt_score'].reshape((-1, 1))
    gt_score = np.concatenate(
        (gt_score1 * factor, gt_score2 * (1. - factor)), axis=0)

    is_crowd1 = sample['is_crowd'].reshape((-1, 1))
    is_crowd2 = sample['mixup']['is_crowd'].reshape((-1, 1))
    is_crowd = np.concatenate((is_crowd1, is_crowd2), axis=0)