object365数据集 Invalid segm type: <class 'NoneType'>
Created by: ash12358
object365数据集训练，按照 https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/featured_model/CACascadeRCNN.md 要求准备好数据，执行命令为CUDA_VISIBLE_DEVICES=1 python tools/train.py -c configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml，结果报错。
我自己初步检查发现是因为
ppdet/data/source/coco.py第102行，instances = coco.loadAnns(ins_anno_ids)，其中instances长度为0，导致导致下面119行num_bbox = len(bboxes)的num_bbox为0，最终导致138行的 coco_rec = { 'im_file': im_fname, 'im_id': np.array([img_id]), 'h': im_h, 'w': im_w, 'is_crowd': is_crowd, 'gt_class': gt_class, 'gt_bbox': gt_bbox, 'gt_score': gt_score, 'gt_poly': gt_poly, } 中的gt_class,gt_bbox,gt_score全部为None或空集。
举个例子，
比如有张图片obj365_train_000000136386.jpg，它在train.json中images里的记录是{'file_name': 'obj365_train_000000136386.jpg', 'id': 136386, 'width': 640, 'height': 480}， 在annotations里的记录是{'area': 29727.047166903096, 'category_id': 3, 'image_id': 136386, 'id': 2088, 'bbox': [-0.004699712, 263.249328624, 639.927185088, 46.453796400000044], 'iscrowd': 1}， 这里是有一个标注框的，并且是valid的，但是coco.py读取出来的instances却是[]，不知道是哪一步读取出问题了。
具体报错日志如下，请大家帮忙看一看，谢谢。
(paddle) w*****@i*****:/data/ssh/PaddleDetection$ CUDA_VISIBLE_DEVICES=1 python tools/train.py -c configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml CascadeBBoxAssigner: [32mbatch_size_per_im[0m: 1024 bbox_reg_weights:
- 10
- 20
- 30 bg_thresh_hi:
- 0.5
- 0.6
- 0.7 bg_thresh_lo:
- 0.0
- 0.0
- 0.0 class_aware: false fg_fraction: 0.25 fg_thresh:
- 0.5
- 0.6
- 0.7 num_classes: 81 shuffle_before_sample: true CascadeBBoxHead: [32mhead[0m: CascadeXConvNormHead [32mnms[0m: keep_top_k: 100 nms_threshold: 0.5 score_threshold: 0.05 num_classes: 81 CascadeRCNN: [32mbackbone[0m: SENet [32mrpn_head[0m: FPNRPNHead bbox_assigner: CascadeBBoxAssigner bbox_head: CascadeBBoxHead fpn: FPN roi_extractor: FPNRoIAlign rpn_only: false CascadeTwoFCHead: [32mmlp_dim[0m: 1024 CascadeXConvNormHead: [32mnorm_type[0m: gn conv_dim: 256 freeze_norm: false mlp_dim: 1024 num_conv: 4 EvalReader: batch_size: 1 batch_transforms:
- !PadBatch
pad_to_stride: 32
use_padded_im_info: true
dataset: !COCODataSet
anno_path: annotations/val.json
dataset_dir: dataset/objects365
image_dir: val
sample_num: -1
with_background: true
drop_empty: false
inputs_def:
fields:
- image
- im_info
- im_id
- im_shape sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !NormalizeImage
is_channel_first: false
is_scale: false
mean:
- 102.9801
- 115.9465
- 122.7717 std:
- 1.0
- 1.0
- 1.0
- !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
- !Permute channel_first: true to_bgr: false worker_num: 2 FPN: [32mfreeze_norm[0m: false [32mnorm_type[0m: gn has_extra_convs: false max_level: 6 min_level: 2 num_chan: 256 spatial_scale:
- 0.03125
- 0.0625
- 0.125
- 0.25
FPNRPNHead:
[32mrpn_target_assign[0m:
rpn_batch_size_per_im: 256
rpn_fg_fraction: 0.5
rpn_negative_overlap: 0.3
rpn_positive_overlap: 0.7
rpn_straddle_thresh: 0.0
[32mtest_proposal[0m:
min_size: 0.0
nms_thresh: 0.7
post_nms_top_n: 1000
pre_nms_top_n: 1000
[32mtrain_proposal[0m:
min_size: 0.0
nms_thresh: 0.7
post_nms_top_n: 2000
pre_nms_top_n: 2000
anchor_generator:
anchor_sizes:
- 32
- 64
- 128
- 256
- 512 aspect_ratios:
- 0.5
- 1.0
- 2.0 stride:
- 16.0
- 16.0 variance:
- 1.0
- 1.0
- 1.0
- 1.0 anchor_start_size: 32 max_level: 6 min_level: 2 num_chan: 256 num_classes: 1 FPNRoIAlign: [32msampling_ratio[0m: 2 box_resolution: 7 canconical_level: 4 canonical_size: 224 mask_resolution: 14 max_level: 5 min_level: 2 LearningRate: [32mschedulers[0m:
- !PiecewiseDecay
gamma: 0.1
milestones:
- 400000
- 460000 values: null
- !LinearWarmup start_factor: 0.01 steps: 2000 base_lr: 0.01 OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2 SENet: [32mdcn_v2_stages[0m:
- 3
- 4
- 5 [32mdepth[0m: 152 [32mfreeze_norm[0m: true [32mnorm_type[0m: bn [32mstd_senet[0m: true feature_maps:
- 2
- 3
- 4
- 5 freeze_at: 2 group_width: 4 groups: 64 norm_decay: 0.0 variant: d weight_prefix_name: '' TestReader: batch_size: 1 batch_transforms:
- !PadBatch pad_to_stride: 32 use_padded_im_info: true dataset: !ImageFolder anno_path: dataset/obj365/annotations/val.json dataset_dir: '' image_dir: '' sample_num: -1 use_default_label: null with_background: true sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !NormalizeImage
is_channel_first: false
is_scale: false
mean:
- 102.9801
- 115.9465
- 122.7717 std:
- 1.0
- 1.0
- 1.0
- !Permute channel_first: true to_bgr: false worker_num: 2 TrainReader: batch_size: 1 batch_transforms:
- !PadBatch
pad_to_stride: 32
use_padded_im_info: true
class_aware_sampling: true
dataset: !COCODataSet
anno_path: annotations/train.json
dataset_dir: dataset/objects365
image_dir: train
sample_num: -1
with_background: true
inputs_def:
fields:
- image
- im_info
- im_id
- gt_bbox
- gt_class
- is_crowd sample_transforms:
- !DecodeImage to_rgb: true with_mixup: false
- !RandomFlipImage is_mask_flip: true is_normalized: false prob: 0.5
- !NormalizeImage
is_channel_first: false
is_scale: false
mean:
- 102.9801
- 115.9465
- 122.7717 std:
- 1.0
- 1.0
- 1.0
- !ResizeImage
interp: 1
max_size: 1600
target_size:
- 416
- 448
- 480
- 512
- 544
- 576
- 608
- 640
- 672
- 704
- 736
- 768
- 800
- 832
- 864
- 896
- 928
- 960
- 992
- 1024
- 1056
- 1088
- 1120
- 1152
- 1184
- 1216
- 1248
- 1280
- 1312
- 1344
- 1376
- 1408 use_cv2: true
- !Permute channel_first: true to_bgr: false shuffle: true use_process: false worker_num: 4 architecture: CascadeRCNN log_iter: 20 log_smooth_window: 20 max_iters: 500000 metric: COCO num_classes: 366 pretrain_weights: https://paddlemodels.bj.bcebos.com/object_detection/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.tar save_dir: output snapshot_iter: 10000 use_gpu: true weights: output/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas/model_final
W0423 11:05:14.742431 9435 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.0, Runtime API Version: 10.0 W0423 11:05:14.748662 9435 device_context.cc:245] device: 0, cuDNN Version: 7.6. 2020-04-23 11:05:18,194-INFO: Found /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained 2020-04-23 11:05:18,195-INFO: Loading parameters from /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained... 2020-04-23 11:05:18,195-WARNING: /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-04-23 11:05:18,195-WARNING: /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-04-23 11:05:18,298-WARNING: variable file [ /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_weights /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_weights /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_w /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_weights /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_b /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_b /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_w /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_weights ] not used 2020-04-23 11:05:18,298-WARNING: variable file [ /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_weights /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_weights /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_w /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_weights /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_scale /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_offset /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_b /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_b /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_w /home/w*****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_weights ] not used loading annotations into memory... Done (t=112.57s) creating index... index created! 2020-04-23 11:07:50,700-WARNING: Found an invalid bbox in annotations: im_id: 608537, area: 0.0 x1: 269.571228032, y1: 248.73785399999997, x2: 269.651184064, y2: 248.73785399999997. 2020-04-23 11:07:54,409-WARNING: Found an invalid bbox in annotations: im_id: 549169, area: 0.0 x1: 742.3126221, y1: 8.545593250000001, x2: 742.3126221, y2: 10.2289734. 2020-04-23 11:07:54,923-WARNING: Found an invalid bbox in annotations: im_id: 91534, area: 0.0 x1: 433.56384276160003, y1: 420.4645385728, x2: 438.66027832960003, y2: 420.4645385728. 2020-04-23 11:07:55,166-WARNING: Found an invalid bbox in annotations: im_id: 555363, area: 0.0 x1: 337.9105372966, y1: 345.2622006784, x2: 337.9105372966, y2: 348.4704679424. 2020-04-23 11:07:57,040-WARNING: Found an invalid bbox in annotations: im_id: 536966, area: 0.0 x1: 554.6245116984, y1: 299.9646606336, x2: 554.6245116984, y2: 314.069702144. 2020-04-23 11:07:57,997-WARNING: Found an invalid bbox in annotations: im_id: 548210, area: 0.0 x1: 193.9760131584, y1: 273.4229126144, x2: 193.9760131584, y2: 273.5556640768. 2020-04-23 11:07:59,855-WARNING: Found an invalid bbox in annotations: im_id: 580684, area: 0.0 x1: 92.80249024999999, y1: 246.2189331, x2: 92.80249024999999, y2: 251.5336914125. 2020-04-23 11:08:01,507-WARNING: Found an invalid bbox in annotations: im_id: 545995, area: 12.73863515740429 x1: 776.3353271618, y1: 663.263916032, x2: 682.0, y2: 511.0. 2020-04-23 11:08:04,240-WARNING: Found an invalid bbox in annotations: im_id: 691979, area: 7.077892501177361 x1: 840.7648925851, y1: 316.9513549824, x2: 682.0, y2: 318.6118163968. 2020-04-23 11:08:07,744-WARNING: Found an invalid bbox in annotations: im_id: 652823, area: 1.154319683283666 x1: 703.4766845439999, y1: 513.236083968, x2: 639.0, y2: 479.0. 2020-04-23 11:08:11,566-WARNING: Found an invalid bbox in annotations: im_id: 535871, area: 0.0 x1: 444.405273457, y1: 427.4860839936, x2: 444.405273457, y2: 430.7103271424. 2020-04-23 11:08:15,927-WARNING: Found an invalid bbox in annotations: im_id: 359829, area: 46.71179995933256 x1: 613.3458251776, y1: 662.7961425408, x2: 511.0, y2: 671.4617920256001. 2020-04-23 11:08:21,938-WARNING: Found an invalid bbox in annotations: im_id: 533112, area: 0.0 x1: 652.6395257820001, y1: 150.0715779584, x2: 652.6395257820001, y2: 151.5980086272. 2020-04-23 11:08:26,161-WARNING: Found an invalid bbox in annotations: im_id: 280484, area: 4.1640145666077 x1: 683.2222900679001, y1: 426.990234368, x2: 682.0, y2: 428.3464965632. 2020-04-23 11:08:27,337-WARNING: Found an invalid bbox in annotations: im_id: 335604, area: 0.0 x1: 132.0651185036, y1: 490.995319296, x2: 135.152460152, y2: 490.995319296. 2020-04-23 11:09:06,249-WARNING: Found an invalid bbox in annotations: im_id: 545724, area: 3.1565472338744165 x1: 408.56115725259997, y1: 512.0473632768, x2: 411.9129638641, y2: 511.0. 2020-04-23 11:09:08,154-WARNING: Found an invalid bbox in annotations: im_id: 646677, area: 4.7767199789934365 x1: 300.07025145, y1: 390.55169676, x2: 299.0, y2: 391.50653076000003. 2020-04-23 11:09:13,348-WARNING: Found an invalid bbox in annotations: im_id: 574787, area: 0.0 x1: 633.690185536, y1: 370.5393676572, x2: 634.97998048, y2: 370.5393676572. 2020-04-23 11:09:14,792-WARNING: Found an invalid bbox in annotations: im_id: 521927, area: 0.0 x1: 55.575744628100004, y1: 270.0220947456, x2: 57.9324340495, y2: 270.0220947456. 2020-04-23 11:09:17,220-WARNING: Found an invalid bbox in annotations: im_id: 527817, area: 657.4470979715645 x1: 515.3116454912, y1: 674.2288818061, x2: 511.0, y2: 910.0. 2020-04-23 11:09:19,779-WARNING: Found an invalid bbox in annotations: im_id: 145983, area: 0.0 x1: 211.64685056, y1: 203.173278816, x2: 211.64685056, y2: 203.356811504. 2020-04-23 11:09:20,553-WARNING: Found an invalid bbox in annotations: im_id: 599992, area: 50.02852134771837 x1: 718.86511232, y1: 441.29223632120005, x2: 639.0, y2: 435.0. 2020-04-23 11:09:22,498-WARNING: Found an invalid bbox in annotations: im_id: 365040, area: 2.20002733606966 x1: 536.6248779263999, y1: 511.0604858368, x2: 536.7738036992, y2: 511.0. 2020-04-23 11:09:26,303-WARNING: Found an invalid bbox in annotations: im_id: 657457, area: 8.41608131001895 x1: 479.75616456, y1: 0.028564480000000003, x2: 479.0, y2: 531.41967776. 2020-04-23 11:09:28,668-WARNING: Found an invalid bbox in annotations: im_id: 560944, area: 0.0 x1: 281.2375488, y1: 306.20666505599996, x2: 281.417053248, y2: 306.20666505599996. 2020-04-23 11:09:32,803-WARNING: Found an invalid bbox in annotations: im_id: 650302, area: 27.592423196214177 x1: 0, y1: 653.989868159, x2: 4.872924800000002, y2: 477.0. 2020-04-23 11:09:45,668-WARNING: Found an invalid bbox in annotations: im_id: 344510, area: 0.0 x1: 878.1358642939999, y1: 360.5763549696, x2: 771.0, y2: 360.5763549696. 2020-04-23 11:09:48,981-WARNING: Found an invalid bbox in annotations: im_id: 547644, area: 0.0 x1: 432.0398170408, y1: 374.5224859648, x2: 432.0398170408, y2: 377.7821602816. 2020-04-23 11:09:52,356-INFO: 608606 samples in file dataset/objects365/annotations/train.json 2020-04-23 11:11:05,879-INFO: places would be ommited when DataLoader is not iterable I0423 11:11:06.125291 9435 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0423 11:11:06.342186 9435 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0423 11:11:08.811236 9435 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0423 11:11:09.054006 9435 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 2020-04-23 11:11:09,563-INFO: fail to map op [RandomFlipImage_e2210c] with error: Invalid segm type: <class 'NoneType'> and stack: Traceback (most recent call last): File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 45, in call data = f(data, ctx) File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 431, in call height, width) File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 376, in flip_segms if is_poly(segm): File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 371, in is_poly "Invalid segm type: {}".format(type(segm)) AssertionError: Invalid segm type: <class 'NoneType'>
2020-04-23 11:11:09,563-WARNING: recv endsignal from outq with errmsg[consumer[consumer-cf8-3] failed to map with error:[Invalid segm type: <class 'NoneType'>]] 2020-04-23 11:11:10,032-INFO: fail to map op [RandomFlipImage_e2210c] with error: Invalid segm type: <class 'NoneType'> and stack: Traceback (most recent call last): File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 45, in call data = f(data, ctx) File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 431, in call height, width) File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 376, in flip_segms if is_poly(segm): File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 371, in is_poly "Invalid segm type: {}".format(type(segm)) AssertionError: Invalid segm type: <class 'NoneType'>
2020-04-23 11:11:10,033-WARNING: recv endsignal from outq with errmsg[consumer[consumer-cf8-2] failed to map with error:[Invalid segm type: <class 'NoneType'>]] 2020-04-23 11:11:10,035-INFO: fail to map op [RandomFlipImage_e2210c] with error: Invalid segm type: <class 'NoneType'> and stack: Traceback (most recent call last): File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 45, in call data = f(data, ctx) File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 431, in call height, width) File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 376, in flip_segms if is_poly(segm): File "/data/ssh/PaddleDetection/ppdet/data/transform/operators.py", line 371, in is_poly "Invalid segm type: {}".format(type(segm)) AssertionError: Invalid segm type: <class 'NoneType'>
W0423 11:11:20.876386 22791 operator.cc:181] deformable_conv raises an exception paddle::memory::allocation::BadAlloc,
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackStringstd::string(std::string&&, char const*, int) 1 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) 2 paddle::memory::allocation::AlignedAllocator::AllocateImpl(unsigned long) 3 paddle::memory::allocation::AutoGrowthBestFitAllocator::AllocateImpl(unsigned long) 4 paddle::memory::allocation::Allocator::Allocate(unsigned long) 5 paddle::memory::allocation::RetryAllocator::AllocateImpl(unsigned long) 6 paddle::memory::allocation::AllocatorFacade::Alloc(paddle::platform::Place const&, unsigned long) 7 paddle::memory::Alloc(paddle::platform::Place const&, unsigned long) 8 paddle::memory::Alloc(paddle::platform::DeviceContext const&, unsigned long) 9 paddle::framework::Tensor paddle::framework::ExecutionContext::AllocateTmpTensor<float, paddle::platform::CUDADeviceContext>(paddle::framework::DDim const&, paddle::platform::CUDADeviceContext const&) const 10 paddle::operators::DeformableConvCUDAKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 11 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::DeformableConvCUDAKernel<paddle::platform::CUDADeviceContext, float> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 12 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 13 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 14 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 15 paddle::framework::details::ComputationOpHandle::RunImpl() 16 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) 17 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 18 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 19 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 20 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const
Error Message Summary:
ResourceExhaustedError:
Out of memory error on GPU 0. Cannot allocate 119.531494MB memory on GPU 0, available memory is only 27.625000MB.
Please check whether there is any other process using GPU 0.
- If yes, please stop them, or start PaddlePaddle on another GPU.
- If no, please decrease the batch size of your model.
at (/paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:69) F0423 11:11:20.877792 22791 exception_holder.h:37] std::exception caught,
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackStringstd::string(std::string&&, char const*, int) 1 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) 2 paddle::memory::allocation::AlignedAllocator::AllocateImpl(unsigned long) 3 paddle::memory::allocation::AutoGrowthBestFitAllocator::AllocateImpl(unsigned long) 4 paddle::memory::allocation::Allocator::Allocate(unsigned long) 5 paddle::memory::allocation::RetryAllocator::AllocateImpl(unsigned long) 6 paddle::memory::allocation::AllocatorFacade::Alloc(paddle::platform::Place const&, unsigned long) 7 paddle::memory::Alloc(paddle::platform::Place const&, unsigned long) 8 paddle::memory::Alloc(paddle::platform::DeviceContext const&, unsigned long) 9 paddle::framework::Tensor paddle::framework::ExecutionContext::AllocateTmpTensor<float, paddle::platform::CUDADeviceContext>(paddle::framework::DDim const&, paddle::platform::CUDADeviceContext const&) const 10 paddle::operators::DeformableConvCUDAKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 11 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::DeformableConvCUDAKernel<paddle::platform::CUDADeviceContext, float> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 12 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 13 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 14 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 15 paddle::framework::details::ComputationOpHandle::RunImpl() 16 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) 17 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 18 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 19 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 20 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const
Error Message Summary:
ResourceExhaustedError:
Out of memory error on GPU 0. Cannot allocate 119.531494MB memory on GPU 0, available memory is only 27.625000MB.
Please check whether there is any other process using GPU 0.
- If yes, please stop them, or start PaddlePaddle on another GPU.
- If no, please decrease the batch size of your model.
at (/paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:69) *** Check failure stack trace: *** @ 0x7f73f3ba2c2d google::LogMessage::Fail() @ 0x7f73f3ba66dc google::LogMessage::SendToLog() @ 0x7f73f3ba2753 google::LogMessage::Flush() @ 0x7f73f3ba7bee google::LogMessageFatal::~LogMessageFatal() @ 0x7f73f617c9b8 paddle::framework::details::ExceptionHolder::Catch() @ 0x7f73f622868e paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync() @ 0x7f73f622729f paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp() @ 0x7f73f6227564 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data @ 0x7f73f3bfb983 std::_Function_handler<>::_M_invoke() @ 0x7f73f3989c37 std::__future_base::_State_base::_M_do_set() @ 0x7f7535d2fa99 __pthread_once_slow @ 0x7f73f6222a52 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv @ 0x7f73f398be64 _ZZN10ThreadPoolC1EmENKUlvE_clEv @ 0x7f744b6db421 execute_native_thread_routine_compat @ 0x7f7535d286ba start_thread @ 0x7f7535a5e41d clone @ (nil) (unknown) 已放弃 (核心已转储) (paddle) w*****@i*****:/data/ssh/PaddleDetection$