使用faster_rcnn_r50_fpn_1x训练自己的数据,iter在240后loss=nan,报错RuntimeWarning: Invalid value encountered in median
Created by: jionyii
这是训练时的信息 BBoxAssigner: batch_size_per_im: 512 bbox_reg_weights:
- 0.1
- 0.1
- 0.2
- 0.2
bg_thresh_hi: 0.5
bg_thresh_lo: 0.0
fg_fraction: 0.25
fg_thresh: 0.5
num_classes: 81
shuffle_before_sample: true
BBoxHead:
[32mhead[0m: TwoFCHead
[32mnms[0m:
keep_top_k: 100
nms_threshold: 0.5
score_threshold: 0.05
box_coder:
axis: 1
box_normalized: false
code_type: decode_center_size
prior_box_var:
- 0.1
- 0.1
- 0.2
- 0.2 num_classes: 81 FPN: freeze_norm: false has_extra_convs: false max_level: 6 min_level: 2 norm_type: null num_chan: 256 spatial_scale:
- 0.03125
- 0.0625
- 0.125
- 0.25
FPNRPNHead:
[32mrpn_target_assign[0m:
rpn_batch_size_per_im: 256
rpn_fg_fraction: 0.5
rpn_negative_overlap: 0.3
rpn_positive_overlap: 0.7
rpn_straddle_thresh: 0.0
[32mtest_proposal[0m:
min_size: 0.0
nms_thresh: 0.7
post_nms_top_n: 1000
pre_nms_top_n: 1000
[32mtrain_proposal[0m:
min_size: 0.0
nms_thresh: 0.7
post_nms_top_n: 2000
pre_nms_top_n: 2000
anchor_generator:
anchor_sizes:
- 32
- 64
- 128
- 256
- 512 aspect_ratios:
- 0.5
- 1.0
- 2.0 stride:
- 16.0
- 16.0 variance:
- 1.0
- 1.0
- 1.0
- 1.0 anchor_start_size: 32 max_level: 6 min_level: 2 num_chan: 256 num_classes: 1 FPNRoIAlign: [32msampling_ratio[0m: 2 box_resolution: 7 canconical_level: 4 canonical_size: 224 mask_resolution: 14 max_level: 5 min_level: 2 FasterRCNN: [32mbackbone[0m: ResNet [32mfpn[0m: FPN [32mroi_extractor[0m: FPNRoIAlign [32mrpn_head[0m: FPNRPNHead bbox_assigner: BBoxAssigner bbox_head: BBoxHead rpn_only: false FasterRCNNEvalFeed: [32mbatch_size[0m: 2 [32mbatch_transforms[0m:
- !PadBatch pad_to_stride: 32 [32mdataset[0m: annotation: ImageSets/val.txt dataset_dir: dataset/setp drop_last: false enable_aug_flip: false enable_multiscale: false fields:
- image
- im_info
- im_id
- im_shape
- gt_box
- gt_label
- is_difficult image_shape:
- null
- 3
- null
- null num_scale: 1 num_workers: 2 sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !NormalizeImage
is_channel_first: false
is_scale: true
mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false samples: -1 shuffle: false use_padded_im_info: true FasterRCNNTestFeed: [32mbatch_transforms[0m:
- !PadBatch pad_to_stride: 32 [32mdataset[0m: annotation: ImageSets/label_list.txt batch_size: 1 drop_last: false fields:
- image
- im_info
- im_id
- im_shape image_shape:
- null
- 3
- null
- null num_workers: 2 sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !NormalizeImage
is_channel_first: false
is_scale: true
mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false samples: -1 shuffle: false use_padded_im_info: true FasterRCNNTrainFeed: [32mbatch_size[0m: 2 [32mbatch_transforms[0m:
- !PadBatch pad_to_stride: 32 [32mdataset[0m: annotation: ImageSets/train.txt dataset_dir: dataset/setp bufsize: 10 class_aware_sampling: false drop_last: false fields:
- image
- im_info
- im_id
- gt_box
- gt_label
- is_crowd image_shape:
- null
- 3
- null
- null memsize: null num_workers: 2 sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !RandomFlipImage is_mask_flip: false is_normalized: false prob: 0.5
- !NormalizeImage
is_channel_first: false
is_scale: true
mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false samples: -1 shuffle: true use_process: false LearningRate: [32mbase_lr[0m: 0.0025 [32mschedulers[0m:
- !PiecewiseDecay
gamma: 0.1
milestones:
- 60000
- 80000 values: null
- !LinearWarmup start_factor: 0.3333333333333333 steps: 500 OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2 ResNet: [32mnorm_type[0m: bn dcn_v2_stages: [] depth: 50 feature_maps:
- 2
- 3
- 4
- 5 freeze_at: 2 freeze_norm: true nonlocal_stages: [] norm_decay: 0.0 variant: b weight_prefix_name: '' TwoFCHead: mlp_dim: 1024 architecture: FasterRCNN eval_feed: FasterRCNNEvalFeed log_smooth_window: 20 max_iters: 90000 metric: VOC num_classes: 1 pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_cos_pretrained.tar save_dir: output snapshot_iter: 10000 test_feed: FasterRCNNTestFeed train_feed: FasterRCNNTrainFeed use_gpu: true weights: output/faster_rcnn_r50_fpn_1x/model_final
2020-07-22 22:36:56,388-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000100] in Optimizer will not take effect, and it will only be applied to other Parameters! W0722 22:36:56.505952 298 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0 W0722 22:36:56.510417 298 device_context.cc:260] device: 0, cuDNN Version: 7.3. 2020-07-22 22:36:59,392-INFO: Found /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained 2020-07-22 22:36:59,392-INFO: Loading parameters from /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained... 2020-07-22 22:36:59,597-INFO: places would be ommited when DataLoader is not iterable I0722 22:36:59.644739 298 graph_pattern_detector.cc:101] --- detected 23 subgraphs I0722 22:36:59.659315 298 graph_pattern_detector.cc:101] --- detected 20 subgraphs 2020-07-22 22:37:01,277-INFO: iter: 0, lr: 0.000833, 'loss_cls': '0.000000', 'loss_bbox': '0.000001', 'loss_rpn_cls': '0.697717', 'loss_rpn_bbox': '0.041035', 'loss': '0.738753', time: 0.000, eta: 0:00:03 2020-07-22 22:37:05,299-INFO: iter: 20, lr: 0.000900, 'loss_cls': '0.000000', 'loss_bbox': '0.000000', 'loss_rpn_cls': '0.670895', 'loss_rpn_bbox': '0.008974', 'loss': '0.694092', time: 0.276, eta: 6:53:33 2020-07-22 22:37:12,498-INFO: iter: 40, lr: 0.000967, 'loss_cls': '0.000000', 'loss_bbox': '0.000001', 'loss_rpn_cls': '0.427321', 'loss_rpn_bbox': '0.014830', 'loss': '0.438422', time: 0.360, eta: 9:00:10 2020-07-22 22:37:16,637-INFO: iter: 60, lr: 0.001033, 'loss_cls': '0.000000', 'loss_bbox': '0.000001', 'loss_rpn_cls': '0.123900', 'loss_rpn_bbox': '0.018455', 'loss': '0.145454', time: 0.203, eta: 5:04:16 2020-07-22 22:37:22,073-INFO: iter: 80, lr: 0.001100, 'loss_cls': '0.000000', 'loss_bbox': '0.000002', 'loss_rpn_cls': '0.110554', 'loss_rpn_bbox': '0.026210', 'loss': '0.138858', time: 0.272, eta: 6:47:56 2020-07-22 22:37:27,188-INFO: iter: 100, lr: 0.001167, 'loss_cls': '0.000000', 'loss_bbox': '0.000002', 'loss_rpn_cls': '0.078616', 'loss_rpn_bbox': '0.015112', 'loss': '0.092849', time: 0.256, eta: 6:23:25 2020-07-22 22:37:31,662-INFO: iter: 120, lr: 0.001233, 'loss_cls': '0.000000', 'loss_bbox': '0.009060', 'loss_rpn_cls': '0.064810', 'loss_rpn_bbox': '0.013977', 'loss': '0.093647', time: 0.226, eta: 5:38:44 2020-07-22 22:37:36,884-INFO: iter: 140, lr: 0.001300, 'loss_cls': '0.000000', 'loss_bbox': '0.010353', 'loss_rpn_cls': '0.047988', 'loss_rpn_bbox': '0.008759', 'loss': '0.071452', time: 0.252, eta: 6:17:06 2020-07-22 22:37:42,083-INFO: iter: 160, lr: 0.001367, 'loss_cls': '0.000000', 'loss_bbox': '0.016858', 'loss_rpn_cls': '0.050457', 'loss_rpn_bbox': '0.009337', 'loss': '0.088492', time: 0.270, eta: 6:43:53 2020-07-22 22:37:46,829-INFO: iter: 180, lr: 0.001433, 'loss_cls': '0.000000', 'loss_bbox': '0.026308', 'loss_rpn_cls': '0.046593', 'loss_rpn_bbox': '0.008388', 'loss': '0.081561', time: 0.237, eta: 5:55:04 2020-07-22 22:37:52,103-INFO: iter: 200, lr: 0.001500, 'loss_cls': '0.000000', 'loss_bbox': '0.018197', 'loss_rpn_cls': '0.038879', 'loss_rpn_bbox': '0.007777', 'loss': '0.106401', time: 0.265, eta: 6:35:53 2020-07-22 22:37:57,043-INFO: iter: 220, lr: 0.001567, 'loss_cls': '0.000000', 'loss_bbox': '0.014204', 'loss_rpn_cls': '0.041513', 'loss_rpn_bbox': '0.011074', 'loss': '0.071006', time: 0.247, eta: 6:09:21 2020-07-22 22:38:02,651-INFO: iter: 240, lr: 0.001633, 'loss_cls': '0.000000', 'loss_bbox': '0.036474', 'loss_rpn_cls': '0.056716', 'loss_rpn_bbox': '0.013674', 'loss': '0.130377', time: 0.277, eta: 6:55:04 /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median r = func(a, **kwargs) 2020-07-22 22:38:08,538-INFO: iter: 260, lr: 0.001700, 'loss_cls': '0.000000', 'loss_bbox': 'nan', 'loss_rpn_cls': 'nan', 'loss_rpn_bbox': '0.319314', 'loss': 'nan', time: 0.297, eta: 7:24:51 2020-07-22 22:38:14,925-INFO: iter: 280, lr: 0.001767, 'loss_cls': '0.000000', 'loss_bbox': 'nan', 'loss_rpn_cls': 'nan', 'loss_rpn_bbox': 'nan', 'loss': 'nan', time: 0.319, eta: 7:56:28 ^C