Created by: learning-boy

训练自定义coco数据集，先通过labelme产生单个json文件，再通过label2coco.py，转换成整个json，训练时报错，但奇怪的是有的时候会训练一次 训练内容如下： /home/aistudio/PaddleDetection-release-0.2 BBoxAssigner: batch_size_per_im: 512 bbox_reg_weights:

• 0.1
• 0.1
• 0.2
• 0.2 bg_thresh_hi: 0.5 bg_thresh_lo: 0.0 fg_fraction: 0.25 fg_thresh: 0.5 num_classes: 81 shuffle_before_sample: true BBoxHead: [32mhead[0m: TwoFCHead [32mnms[0m: keep_top_k: 100 nms_threshold: 0.5 score_threshold: 0.05 bbox_loss: sigma: 1.0 box_coder: axis: 1 box_normalized: false code_type: decode_center_size prior_box_var:
  • 0.1
  • 0.1
  • 0.2
  • 0.2 num_classes: 81 EvalReader: batch_size: 1 batch_transforms:
• !PadBatch pad_to_stride: 32 use_padded_im_info: true dataset: !COCODataSet anno_path: /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_val.json dataset_dir: dataset/coco2 image_dir: val2017/JPEGImages sample_num: -1 with_background: true drop_empty: false drop_last: false inputs_def: fields:
  • image
  • im_info
  • im_id
  • im_shape sample_transforms:
• !DecodeImage to_rgb: true with_mixup: false
• !NormalizeImage is_channel_first: false is_scale: true mean:
  • 0.485
  • 0.456
  • 0.406 std:
  • 0.229
  • 0.224
  • 0.225
• !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
• !Permute channel_first: true to_bgr: false shuffle: false worker_num: 2 FPN: freeze_norm: false has_extra_convs: false max_level: 6 min_level: 2 norm_type: null num_chan: 256 spatial_scale:
• 0.03125
• 0.0625
• 0.125
• 0.25 FPNRPNHead: [32manchor_generator[0m: aspect_ratios:
  • 0.5
  • 1.0
  • 2.0 variance:
  • 1.0
  • 1.0
  • 1.0
  • 1.0 [32mrpn_target_assign[0m: rpn_batch_size_per_im: 256 rpn_fg_fraction: 0.5 rpn_negative_overlap: 0.3 rpn_positive_overlap: 0.7 rpn_straddle_thresh: 0.0 [32mtest_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 1000 pre_nms_top_n: 1000 [32mtrain_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 2000 pre_nms_top_n: 2000 anchor_start_size: 32 max_level: 6 min_level: 2 num_chan: 256 num_classes: 1 FPNRoIAlign: [32msampling_ratio[0m: 2 box_resolution: 7 canconical_level: 4 canonical_size: 224 mask_resolution: 14 max_level: 5 min_level: 2 LearningRate: [32mschedulers[0m:
• !PiecewiseDecay gamma: 0.1 milestones:
  • 120000
  • 160000 values: null
• !LinearWarmup start_factor: 0.3333333333333333 steps: 500 base_lr: 0.01 MaskAssigner: [32mresolution[0m: 28 num_classes: 81 MaskHead: [32mnum_convs[0m: 4 [32mresolution[0m: 28 conv_dim: 256 dilation: 1 norm_type: null num_classes: 81 MaskRCNN: [32mbackbone[0m: ResNet [32mfpn[0m: FPN [32mroi_extractor[0m: FPNRoIAlign [32mrpn_head[0m: FPNRPNHead bbox_assigner: BBoxAssigner bbox_head: BBoxHead mask_assigner: MaskAssigner mask_head: MaskHead rpn_only: false OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2 ResNet: [32mnorm_type[0m: bn dcn_v2_stages: [] depth: 50 feature_maps:
• 2
• 3
• 4
• 5 freeze_at: 2 freeze_norm: true gcb_params: {} gcb_stages: [] nonlocal_stages: [] norm_decay: 0.0 variant: b weight_prefix_name: '' TestReader: batch_size: 1 batch_transforms:
• !PadBatch pad_to_stride: 32 use_padded_im_info: true dataset: !ImageFolder anno_path: /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_val.json dataset_dir: '' image_dir: '' sample_num: -1 use_default_label: null with_background: true drop_last: false inputs_def: fields:
  • image
  • im_info
  • im_id
  • im_shape sample_transforms:
• !DecodeImage to_rgb: true with_mixup: false
• !NormalizeImage is_channel_first: false is_scale: true mean:
  • 0.485
  • 0.456
  • 0.406 std:
  • 0.229
  • 0.224
  • 0.225
• !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
• !Permute channel_first: true to_bgr: false shuffle: false TrainReader: batch_size: 1 batch_transforms:
• !PadBatch pad_to_stride: 32 use_padded_im_info: false dataset: !COCODataSet anno_path: /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_train.json dataset_dir: dataset/coco2 image_dir: train2017/JPEGImages sample_num: -1 with_background: true drop_last: false inputs_def: fields:
  • image
  • im_info
  • im_id
  • gt_bbox
  • gt_class
  • is_crowd
  • gt_mask sample_transforms:
• !DecodeImage to_rgb: true with_mixup: false
• !RandomFlipImage is_mask_flip: true is_normalized: false prob: 0.5
• !NormalizeImage is_channel_first: false is_scale: true mean:
  • 0.485
  • 0.456
  • 0.406 std:
  • 0.229
  • 0.224
  • 0.225
• !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
• !Permute channel_first: true to_bgr: false shuffle: true use_process: false worker_num: 2 TwoFCHead: mlp_dim: 1024 architecture: MaskRCNN finetune_exclude_pretrained_params:
• cls_score
• bbox_pred
• mask_fcn_logits log_smooth_window: 20 max_iters: 10000 metric: COCO num_classes: 2 pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/mask_rcnn_r50_fpn_1x.tar save_dir: output snapshot_iter: 200 use_gpu: true weights: output/mask_rcnn_r50_fpn_1x/best_model

loading annotations into memory... Done (t=0.00s) creating index... index created! 2020-04-13 01:45:34,575-INFO: 3 samples in file /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_val.json 2020-04-13 01:45:34,576-INFO: places would be ommited when DataLoader is not iterable W0413 01:45:35.495064 7896 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0 W0413 01:45:35.499096 7896 device_context.cc:245] device: 0, cuDNN Version: 7.3. 2020-04-13 01:45:37,157-INFO: Found /home/aistudio/.cache/paddle/weights/mask_rcnn_r50_fpn_1x 2020-04-13 01:45:37,158-INFO: Loading parameters from /home/aistudio/.cache/paddle/weights/mask_rcnn_r50_fpn_1x... 2020-04-13 01:45:37,435-WARNING: /tmp/tmp3011l3n2/mask_rcnn_r50_fpn_1x.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-04-13 01:45:37,435-WARNING: /tmp/tmp3011l3n2/mask_rcnn_r50_fpn_1x.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] loading annotations into memory... Done (t=0.00s) creating index... index created! 2020-04-13 01:45:37,870-INFO: 24 samples in file /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_train.json 2020-04-13 01:45:37,870-INFO: places would be ommited when DataLoader is not iterable I0413 01:45:37.892241 7896 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0413 01:45:37.933853 7896 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0413 01:45:38.026324 7896 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0413 01:45:38.055179 7896 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 2020-04-13 01:45:38,436-INFO: iter: 0, lr: 0.003333, 'loss_cls': '0.793345', 'loss_bbox': '0.228089', 'loss_rpn_cls': '0.004561', 'loss_rpn_bbox': '0.002970', 'loss_mask': '7.051753', 'loss': '8.080717', time: 0.000, eta: 0:00:00 W0413 01:45:38.484573 7961 init.cc:209] Warning: PaddlePaddle catches a failure signal, it may not work properly W0413 01:45:38.484616 7961 init.cc:211] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0413 01:45:38.484622 7961 init.cc:214] The detail failure signal is:

W0413 01:45:38.484628 7961 init.cc:217] *** Aborted at 1586713538 (unix time) try "date -d @1586713538" if you are using GNU date *** W0413 01:45:38.487787 7961 init.cc:217] PC: @ 0x0 (unknown) W0413 01:45:38.488077 7961 init.cc:217] *** SIGSEGV (@0x100000029) received by PID 7896 (TID 0x7f66c1408700) from PID 41; stack trace: *** W0413 01:45:38.490900 7961 init.cc:217] @ 0x7f67852b1390 (unknown) W0413 01:45:38.495342 7961 init.cc:217] @ 0x7f6721258908 paddle::memory::detail::MemoryBlock::Split() W0413 01:45:38.501958 7961 init.cc:217] @ 0x7f6721256e32 _ZN6paddle6memory6detail14BuddyAllocator12SplitToAllocESt23_Rb_tree_const_iteratorISt5tupleIJmmPvEEEm W0413 01:45:38.509239 7961 init.cc:217] @ 0x7f6721257315 paddle::memory::detail::BuddyAllocator::Alloc() W0413 01:45:38.515251 7961 init.cc:217] @ 0x7f6721246cb5 paddle::memory::legacy::Alloc<>() W0413 01:45:38.522933 7961 init.cc:217] @ 0x7f6721247c55 paddle::memory::allocation::NaiveBestFitAllocator::AllocateImpl() W0413 01:45:38.527529 7961 init.cc:217] @ 0x7f6721240903 paddle::memory::allocation::AllocatorFacade::Alloc() W0413 01:45:38.533730 7961 init.cc:217] @ 0x7f6721240b9e paddle::memory::allocation::AllocatorFacade::AllocShared() W0413 01:45:38.537961 7961 init.cc:217] @ 0x7f672124031c paddle::memory::AllocShared() W0413 01:45:38.542625 7961 init.cc:217] @ 0x7f672122d2b2 paddle::framework::Tensor::mutable_data() W0413 01:45:38.549592 7961 init.cc:217] @ 0x7f6720a2c53e paddle::operators::SampleRpnFgBgGt<>() W0413 01:45:38.556818 7961 init.cc:217] @ 0x7f6720a31c76 paddle::operators::RpnTargetAssignKernel<>::Compute() W0413 01:45:38.565286 7961 init.cc:217] @ 0x7f6720a33393 ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform8CPUPlaceELb0ELm0EJNS0_9operators21RpnTargetAssignKernelIfEENSA_IdEEEEclEPKcSF_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4 W0413 01:45:38.572489 7961 init.cc:217] @ 0x7f67211a5616 paddle::framework::OperatorWithKernel::RunImpl() W0413 01:45:38.582528 7961 init.cc:217] @ 0x7f67211a5de1 paddle::framework::OperatorWithKernel::RunImpl() W0413 01:45:38.588263 7961 init.cc:217] @ 0x7f672119ef30 paddle::framework::OperatorBase::Run() W0413 01:45:38.596307 7961 init.cc:217] @ 0x7f6720f23c26 paddle::framework::details::ComputationOpHandle::RunImpl() W0413 01:45:38.603711 7961 init.cc:217] @ 0x7f6720edade1 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync() W0413 01:45:38.611949 7961 init.cc:217] @ 0x7f6720ed9b4f paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp() W0413 01:45:38.614733 7961 init.cc:217] @ 0x7f6720ed9e14 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data W0413 01:45:38.623734 7961 init.cc:217] @ 0x7f671e9c0513 std::_Function_handler<>::_M_invoke() W0413 01:45:38.631021 7961 init.cc:217] @ 0x7f671e750717 std::__future_base::_State_base::_M_do_set() W0413 01:45:38.633886 7961 init.cc:217] @ 0x7f67852aea99 __pthread_once_slow W0413 01:45:38.635687 7961 init.cc:217] @ 0x7f6720ed5302 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv W0413 01:45:38.642803 7961 init.cc:217] @ 0x7f671e752944 _ZZN10ThreadPoolC1EmENKUlvE_clEv W0413 01:45:38.644412 7961 init.cc:217] @ 0x7f6746598421 execute_native_thread_routine_compat W0413 01:45:38.647086 7961 init.cc:217] @ 0x7f67852a76ba start_thread W0413 01:45:38.649739 7961 init.cc:217] @ 0x7f6784fdd41d clone W0413 01:45:38.652365 7961 init.cc:217] @ 0x0 (unknown) Segmentation fault (core dumped)