Skip to content

  • 体验新版
    • 正在加载...
  • 登录
  • PaddlePaddle
  • PaddleDetection
  • Issue
  • #1090

P
PaddleDetection
  • 项目概览

PaddlePaddle / PaddleDetection
大约 2 年 前同步成功

通知 708
Star 11112
Fork 2696
  • 代码
    • 文件
    • 提交
    • 分支
    • Tags
    • 贡献者
    • 分支图
    • Diff
  • Issue 184
    • 列表
    • 看板
    • 标记
    • 里程碑
  • 合并请求 40
  • Wiki 0
    • Wiki
  • 分析
    • 仓库
    • DevOps
  • 项目成员
  • Pages
P
PaddleDetection
  • 项目概览
    • 项目概览
    • 详情
    • 发布
  • 仓库
    • 仓库
    • 文件
    • 提交
    • 分支
    • 标签
    • 贡献者
    • 分支图
    • 比较
  • Issue 184
    • Issue 184
    • 列表
    • 看板
    • 标记
    • 里程碑
  • 合并请求 40
    • 合并请求 40
  • Pages
  • 分析
    • 分析
    • 仓库分析
    • DevOps
  • Wiki 0
    • Wiki
  • 成员
    • 成员
  • 收起侧边栏
  • 动态
  • 分支图
  • 创建新Issue
  • 提交
  • Issue看板
已关闭
开放中
Opened 7月 22, 2020 by saxon_zh@saxon_zhGuest

使用faster_rcnn_r50_fpn_1x训练自己的数据,iter在240后loss=nan,报错RuntimeWarning: Invalid value encountered in median

Created by: jionyii

这是训练时的信息 BBoxAssigner: batch_size_per_im: 512 bbox_reg_weights:

  • 0.1
  • 0.1
  • 0.2
  • 0.2 bg_thresh_hi: 0.5 bg_thresh_lo: 0.0 fg_fraction: 0.25 fg_thresh: 0.5 num_classes: 81 shuffle_before_sample: true BBoxHead: [32mhead[0m: TwoFCHead [32mnms[0m: keep_top_k: 100 nms_threshold: 0.5 score_threshold: 0.05 box_coder: axis: 1 box_normalized: false code_type: decode_center_size prior_box_var:
    • 0.1
    • 0.1
    • 0.2
    • 0.2 num_classes: 81 FPN: freeze_norm: false has_extra_convs: false max_level: 6 min_level: 2 norm_type: null num_chan: 256 spatial_scale:
  • 0.03125
  • 0.0625
  • 0.125
  • 0.25 FPNRPNHead: [32mrpn_target_assign[0m: rpn_batch_size_per_im: 256 rpn_fg_fraction: 0.5 rpn_negative_overlap: 0.3 rpn_positive_overlap: 0.7 rpn_straddle_thresh: 0.0 [32mtest_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 1000 pre_nms_top_n: 1000 [32mtrain_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 2000 pre_nms_top_n: 2000 anchor_generator: anchor_sizes:
    • 32
    • 64
    • 128
    • 256
    • 512 aspect_ratios:
    • 0.5
    • 1.0
    • 2.0 stride:
    • 16.0
    • 16.0 variance:
    • 1.0
    • 1.0
    • 1.0
    • 1.0 anchor_start_size: 32 max_level: 6 min_level: 2 num_chan: 256 num_classes: 1 FPNRoIAlign: [32msampling_ratio[0m: 2 box_resolution: 7 canconical_level: 4 canonical_size: 224 mask_resolution: 14 max_level: 5 min_level: 2 FasterRCNN: [32mbackbone[0m: ResNet [32mfpn[0m: FPN [32mroi_extractor[0m: FPNRoIAlign [32mrpn_head[0m: FPNRPNHead bbox_assigner: BBoxAssigner bbox_head: BBoxHead rpn_only: false FasterRCNNEvalFeed: [32mbatch_size[0m: 2 [32mbatch_transforms[0m:
  • !PadBatch pad_to_stride: 32 [32mdataset[0m: annotation: ImageSets/val.txt dataset_dir: dataset/setp drop_last: false enable_aug_flip: false enable_multiscale: false fields:
  • image
  • im_info
  • im_id
  • im_shape
  • gt_box
  • gt_label
  • is_difficult image_shape:
  • null
  • 3
  • null
  • null num_scale: 1 num_workers: 2 sample_transforms:
  • !DecodeImage to_rgb: true with_mixup: false
  • !NormalizeImage is_channel_first: false is_scale: true mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
  • !Permute channel_first: true to_bgr: false samples: -1 shuffle: false use_padded_im_info: true FasterRCNNTestFeed: [32mbatch_transforms[0m:
  • !PadBatch pad_to_stride: 32 [32mdataset[0m: annotation: ImageSets/label_list.txt batch_size: 1 drop_last: false fields:
  • image
  • im_info
  • im_id
  • im_shape image_shape:
  • null
  • 3
  • null
  • null num_workers: 2 sample_transforms:
  • !DecodeImage to_rgb: true with_mixup: false
  • !NormalizeImage is_channel_first: false is_scale: true mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
  • !Permute channel_first: true to_bgr: false samples: -1 shuffle: false use_padded_im_info: true FasterRCNNTrainFeed: [32mbatch_size[0m: 2 [32mbatch_transforms[0m:
  • !PadBatch pad_to_stride: 32 [32mdataset[0m: annotation: ImageSets/train.txt dataset_dir: dataset/setp bufsize: 10 class_aware_sampling: false drop_last: false fields:
  • image
  • im_info
  • im_id
  • gt_box
  • gt_label
  • is_crowd image_shape:
  • null
  • 3
  • null
  • null memsize: null num_workers: 2 sample_transforms:
  • !DecodeImage to_rgb: true with_mixup: false
  • !RandomFlipImage is_mask_flip: false is_normalized: false prob: 0.5
  • !NormalizeImage is_channel_first: false is_scale: true mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
  • !Permute channel_first: true to_bgr: false samples: -1 shuffle: true use_process: false LearningRate: [32mbase_lr[0m: 0.0025 [32mschedulers[0m:
  • !PiecewiseDecay gamma: 0.1 milestones:
    • 60000
    • 80000 values: null
  • !LinearWarmup start_factor: 0.3333333333333333 steps: 500 OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2 ResNet: [32mnorm_type[0m: bn dcn_v2_stages: [] depth: 50 feature_maps:
  • 2
  • 3
  • 4
  • 5 freeze_at: 2 freeze_norm: true nonlocal_stages: [] norm_decay: 0.0 variant: b weight_prefix_name: '' TwoFCHead: mlp_dim: 1024 architecture: FasterRCNN eval_feed: FasterRCNNEvalFeed log_smooth_window: 20 max_iters: 90000 metric: VOC num_classes: 1 pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_cos_pretrained.tar save_dir: output snapshot_iter: 10000 test_feed: FasterRCNNTestFeed train_feed: FasterRCNNTrainFeed use_gpu: true weights: output/faster_rcnn_r50_fpn_1x/model_final

2020-07-22 22:36:56,388-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000100] in Optimizer will not take effect, and it will only be applied to other Parameters! W0722 22:36:56.505952 298 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0 W0722 22:36:56.510417 298 device_context.cc:260] device: 0, cuDNN Version: 7.3. 2020-07-22 22:36:59,392-INFO: Found /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained 2020-07-22 22:36:59,392-INFO: Loading parameters from /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained... 2020-07-22 22:36:59,597-INFO: places would be ommited when DataLoader is not iterable I0722 22:36:59.644739 298 graph_pattern_detector.cc:101] --- detected 23 subgraphs I0722 22:36:59.659315 298 graph_pattern_detector.cc:101] --- detected 20 subgraphs 2020-07-22 22:37:01,277-INFO: iter: 0, lr: 0.000833, 'loss_cls': '0.000000', 'loss_bbox': '0.000001', 'loss_rpn_cls': '0.697717', 'loss_rpn_bbox': '0.041035', 'loss': '0.738753', time: 0.000, eta: 0:00:03 2020-07-22 22:37:05,299-INFO: iter: 20, lr: 0.000900, 'loss_cls': '0.000000', 'loss_bbox': '0.000000', 'loss_rpn_cls': '0.670895', 'loss_rpn_bbox': '0.008974', 'loss': '0.694092', time: 0.276, eta: 6:53:33 2020-07-22 22:37:12,498-INFO: iter: 40, lr: 0.000967, 'loss_cls': '0.000000', 'loss_bbox': '0.000001', 'loss_rpn_cls': '0.427321', 'loss_rpn_bbox': '0.014830', 'loss': '0.438422', time: 0.360, eta: 9:00:10 2020-07-22 22:37:16,637-INFO: iter: 60, lr: 0.001033, 'loss_cls': '0.000000', 'loss_bbox': '0.000001', 'loss_rpn_cls': '0.123900', 'loss_rpn_bbox': '0.018455', 'loss': '0.145454', time: 0.203, eta: 5:04:16 2020-07-22 22:37:22,073-INFO: iter: 80, lr: 0.001100, 'loss_cls': '0.000000', 'loss_bbox': '0.000002', 'loss_rpn_cls': '0.110554', 'loss_rpn_bbox': '0.026210', 'loss': '0.138858', time: 0.272, eta: 6:47:56 2020-07-22 22:37:27,188-INFO: iter: 100, lr: 0.001167, 'loss_cls': '0.000000', 'loss_bbox': '0.000002', 'loss_rpn_cls': '0.078616', 'loss_rpn_bbox': '0.015112', 'loss': '0.092849', time: 0.256, eta: 6:23:25 2020-07-22 22:37:31,662-INFO: iter: 120, lr: 0.001233, 'loss_cls': '0.000000', 'loss_bbox': '0.009060', 'loss_rpn_cls': '0.064810', 'loss_rpn_bbox': '0.013977', 'loss': '0.093647', time: 0.226, eta: 5:38:44 2020-07-22 22:37:36,884-INFO: iter: 140, lr: 0.001300, 'loss_cls': '0.000000', 'loss_bbox': '0.010353', 'loss_rpn_cls': '0.047988', 'loss_rpn_bbox': '0.008759', 'loss': '0.071452', time: 0.252, eta: 6:17:06 2020-07-22 22:37:42,083-INFO: iter: 160, lr: 0.001367, 'loss_cls': '0.000000', 'loss_bbox': '0.016858', 'loss_rpn_cls': '0.050457', 'loss_rpn_bbox': '0.009337', 'loss': '0.088492', time: 0.270, eta: 6:43:53 2020-07-22 22:37:46,829-INFO: iter: 180, lr: 0.001433, 'loss_cls': '0.000000', 'loss_bbox': '0.026308', 'loss_rpn_cls': '0.046593', 'loss_rpn_bbox': '0.008388', 'loss': '0.081561', time: 0.237, eta: 5:55:04 2020-07-22 22:37:52,103-INFO: iter: 200, lr: 0.001500, 'loss_cls': '0.000000', 'loss_bbox': '0.018197', 'loss_rpn_cls': '0.038879', 'loss_rpn_bbox': '0.007777', 'loss': '0.106401', time: 0.265, eta: 6:35:53 2020-07-22 22:37:57,043-INFO: iter: 220, lr: 0.001567, 'loss_cls': '0.000000', 'loss_bbox': '0.014204', 'loss_rpn_cls': '0.041513', 'loss_rpn_bbox': '0.011074', 'loss': '0.071006', time: 0.247, eta: 6:09:21 2020-07-22 22:38:02,651-INFO: iter: 240, lr: 0.001633, 'loss_cls': '0.000000', 'loss_bbox': '0.036474', 'loss_rpn_cls': '0.056716', 'loss_rpn_bbox': '0.013674', 'loss': '0.130377', time: 0.277, eta: 6:55:04 /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median r = func(a, **kwargs) 2020-07-22 22:38:08,538-INFO: iter: 260, lr: 0.001700, 'loss_cls': '0.000000', 'loss_bbox': 'nan', 'loss_rpn_cls': 'nan', 'loss_rpn_bbox': '0.319314', 'loss': 'nan', time: 0.297, eta: 7:24:51 2020-07-22 22:38:14,925-INFO: iter: 280, lr: 0.001767, 'loss_cls': '0.000000', 'loss_bbox': 'nan', 'loss_rpn_cls': 'nan', 'loss_rpn_bbox': 'nan', 'loss': 'nan', time: 0.319, eta: 7:56:28 ^C

指派人
分配到
无
里程碑
无
分配里程碑
工时统计
无
截止日期
无
标识: paddlepaddle/PaddleDetection#1090
渝ICP备2023009037号

京公网安备11010502055752号

网络110报警服务 Powered by GitLab CE v13.7
开源知识
Git 入门 Pro Git 电子书 在线学 Git
Markdown 基础入门 IT 技术知识开源图谱
帮助
使用手册 反馈建议 博客
《GitCode 隐私声明》 《GitCode 服务条款》 关于GitCode
Powered by GitLab CE v13.7