训练自定义coco数据集,是通过labelme产生单个json文件,再通过label2coco.py,转换成json
Created by: learning-boy
训练自定义coco数据集,先通过labelme产生单个json文件,再通过label2coco.py,转换成整个json,训练时报错,但奇怪的是有的时候会训练一次 训练内容如下: /home/aistudio/PaddleDetection-release-0.2 BBoxAssigner: batch_size_per_im: 512 bbox_reg_weights:
- 0.1
- 0.1
- 0.2
- 0.2
bg_thresh_hi: 0.5
bg_thresh_lo: 0.0
fg_fraction: 0.25
fg_thresh: 0.5
num_classes: 81
shuffle_before_sample: true
BBoxHead:
[32mhead[0m: TwoFCHead
[32mnms[0m:
keep_top_k: 100
nms_threshold: 0.5
score_threshold: 0.05
bbox_loss:
sigma: 1.0
box_coder:
axis: 1
box_normalized: false
code_type: decode_center_size
prior_box_var:
- 0.1
- 0.1
- 0.2
- 0.2 num_classes: 81 EvalReader: batch_size: 1 batch_transforms:
- !PadBatch
pad_to_stride: 32
use_padded_im_info: true
dataset: !COCODataSet
anno_path: /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_val.json
dataset_dir: dataset/coco2
image_dir: val2017/JPEGImages
sample_num: -1
with_background: true
drop_empty: false
drop_last: false
inputs_def:
fields:
- image
- im_info
- im_id
- im_shape sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !NormalizeImage
is_channel_first: false
is_scale: true
mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false shuffle: false worker_num: 2 FPN: freeze_norm: false has_extra_convs: false max_level: 6 min_level: 2 norm_type: null num_chan: 256 spatial_scale:
- 0.03125
- 0.0625
- 0.125
- 0.25
FPNRPNHead:
[32manchor_generator[0m:
aspect_ratios:
- 0.5
- 1.0
- 2.0 variance:
- 1.0
- 1.0
- 1.0
- 1.0 [32mrpn_target_assign[0m: rpn_batch_size_per_im: 256 rpn_fg_fraction: 0.5 rpn_negative_overlap: 0.3 rpn_positive_overlap: 0.7 rpn_straddle_thresh: 0.0 [32mtest_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 1000 pre_nms_top_n: 1000 [32mtrain_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 2000 pre_nms_top_n: 2000 anchor_start_size: 32 max_level: 6 min_level: 2 num_chan: 256 num_classes: 1 FPNRoIAlign: [32msampling_ratio[0m: 2 box_resolution: 7 canconical_level: 4 canonical_size: 224 mask_resolution: 14 max_level: 5 min_level: 2 LearningRate: [32mschedulers[0m:
- !PiecewiseDecay
gamma: 0.1
milestones:
- 120000
- 160000 values: null
- !LinearWarmup start_factor: 0.3333333333333333 steps: 500 base_lr: 0.01 MaskAssigner: [32mresolution[0m: 28 num_classes: 81 MaskHead: [32mnum_convs[0m: 4 [32mresolution[0m: 28 conv_dim: 256 dilation: 1 norm_type: null num_classes: 81 MaskRCNN: [32mbackbone[0m: ResNet [32mfpn[0m: FPN [32mroi_extractor[0m: FPNRoIAlign [32mrpn_head[0m: FPNRPNHead bbox_assigner: BBoxAssigner bbox_head: BBoxHead mask_assigner: MaskAssigner mask_head: MaskHead rpn_only: false OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2 ResNet: [32mnorm_type[0m: bn dcn_v2_stages: [] depth: 50 feature_maps:
- 2
- 3
- 4
- 5 freeze_at: 2 freeze_norm: true gcb_params: {} gcb_stages: [] nonlocal_stages: [] norm_decay: 0.0 variant: b weight_prefix_name: '' TestReader: batch_size: 1 batch_transforms:
- !PadBatch
pad_to_stride: 32
use_padded_im_info: true
dataset: !ImageFolder
anno_path: /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_val.json
dataset_dir: ''
image_dir: ''
sample_num: -1
use_default_label: null
with_background: true
drop_last: false
inputs_def:
fields:
- image
- im_info
- im_id
- im_shape sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !NormalizeImage
is_channel_first: false
is_scale: true
mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false shuffle: false TrainReader: batch_size: 1 batch_transforms:
- !PadBatch
pad_to_stride: 32
use_padded_im_info: false
dataset: !COCODataSet
anno_path: /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_train.json
dataset_dir: dataset/coco2
image_dir: train2017/JPEGImages
sample_num: -1
with_background: true
drop_last: false
inputs_def:
fields:
- image
- im_info
- im_id
- gt_bbox
- gt_class
- is_crowd
- gt_mask sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !RandomFlipImage is_mask_flip: true is_normalized: false prob: 0.5
- !NormalizeImage
is_channel_first: false
is_scale: true
mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false shuffle: true use_process: false worker_num: 2 TwoFCHead: mlp_dim: 1024 architecture: MaskRCNN finetune_exclude_pretrained_params:
- cls_score
- bbox_pred
- mask_fcn_logits log_smooth_window: 20 max_iters: 10000 metric: COCO num_classes: 2 pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/mask_rcnn_r50_fpn_1x.tar save_dir: output snapshot_iter: 200 use_gpu: true weights: output/mask_rcnn_r50_fpn_1x/best_model
loading annotations into memory... Done (t=0.00s) creating index... index created! 2020-04-13 01:45:34,575-INFO: 3 samples in file /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_val.json 2020-04-13 01:45:34,576-INFO: places would be ommited when DataLoader is not iterable W0413 01:45:35.495064 7896 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0 W0413 01:45:35.499096 7896 device_context.cc:245] device: 0, cuDNN Version: 7.3. 2020-04-13 01:45:37,157-INFO: Found /home/aistudio/.cache/paddle/weights/mask_rcnn_r50_fpn_1x 2020-04-13 01:45:37,158-INFO: Loading parameters from /home/aistudio/.cache/paddle/weights/mask_rcnn_r50_fpn_1x... 2020-04-13 01:45:37,435-WARNING: /tmp/tmp3011l3n2/mask_rcnn_r50_fpn_1x.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-04-13 01:45:37,435-WARNING: /tmp/tmp3011l3n2/mask_rcnn_r50_fpn_1x.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] loading annotations into memory... Done (t=0.00s) creating index... index created! 2020-04-13 01:45:37,870-INFO: 24 samples in file /home/aistudio/PaddleDetection-release-0.2/dataset/coco2/annotations/instances_train.json 2020-04-13 01:45:37,870-INFO: places would be ommited when DataLoader is not iterable I0413 01:45:37.892241 7896 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0413 01:45:37.933853 7896 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0413 01:45:38.026324 7896 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0413 01:45:38.055179 7896 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 2020-04-13 01:45:38,436-INFO: iter: 0, lr: 0.003333, 'loss_cls': '0.793345', 'loss_bbox': '0.228089', 'loss_rpn_cls': '0.004561', 'loss_rpn_bbox': '0.002970', 'loss_mask': '7.051753', 'loss': '8.080717', time: 0.000, eta: 0:00:00 W0413 01:45:38.484573 7961 init.cc:209] Warning: PaddlePaddle catches a failure signal, it may not work properly W0413 01:45:38.484616 7961 init.cc:211] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0413 01:45:38.484622 7961 init.cc:214] The detail failure signal is:
W0413 01:45:38.484628 7961 init.cc:217] *** Aborted at 1586713538 (unix time) try "date -d @1586713538" if you are using GNU date *** W0413 01:45:38.487787 7961 init.cc:217] PC: @ 0x0 (unknown) W0413 01:45:38.488077 7961 init.cc:217] *** SIGSEGV (@0x100000029) received by PID 7896 (TID 0x7f66c1408700) from PID 41; stack trace: *** W0413 01:45:38.490900 7961 init.cc:217] @ 0x7f67852b1390 (unknown) W0413 01:45:38.495342 7961 init.cc:217] @ 0x7f6721258908 paddle::memory::detail::MemoryBlock::Split() W0413 01:45:38.501958 7961 init.cc:217] @ 0x7f6721256e32 _ZN6paddle6memory6detail14BuddyAllocator12SplitToAllocESt23_Rb_tree_const_iteratorISt5tupleIJmmPvEEEm W0413 01:45:38.509239 7961 init.cc:217] @ 0x7f6721257315 paddle::memory::detail::BuddyAllocator::Alloc() W0413 01:45:38.515251 7961 init.cc:217] @ 0x7f6721246cb5 paddle::memory::legacy::Alloc<>() W0413 01:45:38.522933 7961 init.cc:217] @ 0x7f6721247c55 paddle::memory::allocation::NaiveBestFitAllocator::AllocateImpl() W0413 01:45:38.527529 7961 init.cc:217] @ 0x7f6721240903 paddle::memory::allocation::AllocatorFacade::Alloc() W0413 01:45:38.533730 7961 init.cc:217] @ 0x7f6721240b9e paddle::memory::allocation::AllocatorFacade::AllocShared() W0413 01:45:38.537961 7961 init.cc:217] @ 0x7f672124031c paddle::memory::AllocShared() W0413 01:45:38.542625 7961 init.cc:217] @ 0x7f672122d2b2 paddle::framework::Tensor::mutable_data() W0413 01:45:38.549592 7961 init.cc:217] @ 0x7f6720a2c53e paddle::operators::SampleRpnFgBgGt<>() W0413 01:45:38.556818 7961 init.cc:217] @ 0x7f6720a31c76 paddle::operators::RpnTargetAssignKernel<>::Compute() W0413 01:45:38.565286 7961 init.cc:217] @ 0x7f6720a33393 ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform8CPUPlaceELb0ELm0EJNS0_9operators21RpnTargetAssignKernelIfEENSA_IdEEEEclEPKcSF_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4 W0413 01:45:38.572489 7961 init.cc:217] @ 0x7f67211a5616 paddle::framework::OperatorWithKernel::RunImpl() W0413 01:45:38.582528 7961 init.cc:217] @ 0x7f67211a5de1 paddle::framework::OperatorWithKernel::RunImpl() W0413 01:45:38.588263 7961 init.cc:217] @ 0x7f672119ef30 paddle::framework::OperatorBase::Run() W0413 01:45:38.596307 7961 init.cc:217] @ 0x7f6720f23c26 paddle::framework::details::ComputationOpHandle::RunImpl() W0413 01:45:38.603711 7961 init.cc:217] @ 0x7f6720edade1 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync() W0413 01:45:38.611949 7961 init.cc:217] @ 0x7f6720ed9b4f paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp() W0413 01:45:38.614733 7961 init.cc:217] @ 0x7f6720ed9e14 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data W0413 01:45:38.623734 7961 init.cc:217] @ 0x7f671e9c0513 std::_Function_handler<>::_M_invoke() W0413 01:45:38.631021 7961 init.cc:217] @ 0x7f671e750717 std::__future_base::_State_base::_M_do_set() W0413 01:45:38.633886 7961 init.cc:217] @ 0x7f67852aea99 __pthread_once_slow W0413 01:45:38.635687 7961 init.cc:217] @ 0x7f6720ed5302 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv W0413 01:45:38.642803 7961 init.cc:217] @ 0x7f671e752944 _ZZN10ThreadPoolC1EmENKUlvE_clEv W0413 01:45:38.644412 7961 init.cc:217] @ 0x7f6746598421 execute_native_thread_routine_compat W0413 01:45:38.647086 7961 init.cc:217] @ 0x7f67852a76ba start_thread W0413 01:45:38.649739 7961 init.cc:217] @ 0x7f6784fdd41d clone W0413 01:45:38.652365 7961 init.cc:217] @ 0x0 (unknown) Segmentation fault (core dumped)