Skip to content

P
PaddleDetection

PaddlePaddle / PaddleDetection
大约 2 年 前同步成功

通知 708
Star 11112
Fork 2696
P
PaddleDetection
已关闭
Opened by saxon_zh@saxon_zhGuest

数据读取问题

Created by: ljx111

数据的读取是自己用load_roidb_and_cname2cid重写的，格式是VOC格式的，在yolov3能跑得了，但是cascadercnn却报错，提示assert 'gt_poly' in samples，但是我也没有设置gt_poly字段，会不会是配置的问题

数据格式：

            ply_rec = {
                'im_file': img_file,
                'im_id': img_id,
                'h': im_h,
                'w': im_w,
                'is_crowd': is_crowd,
                'gt_class': gt_class,
                'gt_bbox': gt_bbox,
                'gt_score': gt_score,
                #'gt_poly': gt_poly,
                'difficult': difficult
            }

以下为训练报错 CascadeBBoxAssigner: batch_size_per_im: 512 bbox_reg_weights:

  • 10
  • 20
  • 30 bg_thresh_hi:
  • 0.5
  • 0.6
  • 0.7 bg_thresh_lo:
  • 0.0
  • 0.0
  • 0.0 class_aware: false fg_fraction: 0.25 fg_thresh:
  • 0.5
  • 0.6
  • 0.7 num_classes: 81 shuffle_before_sample: true CascadeBBoxHead: [32mhead[0m: CascadeTwoFCHead [32mnms[0m: keep_top_k: 100 nms_threshold: 0.5 score_threshold: 0.05 num_classes: 81 CascadeMaskRCNN: [32mbackbone[0m: ResNet [32mrpn_head[0m: FPNRPNHead bbox_assigner: CascadeBBoxAssigner bbox_head: CascadeBBoxHead fpn: FPN mask_assigner: MaskAssigner mask_head: MaskHead roi_extractor: FPNRoIAlign rpn_only: false CascadeTwoFCHead: [32mmlp_dim[0m: 1024 EvalReader: batch_size: 1 batch_transforms:
  • !PadBatch pad_to_stride: 32 use_padded_im_info: true dataset: !PolypsDataSet anno_path: label_val__.txt dataset_dir: dataset/polyps/data image_dir: image_train sample_num: -1 with_background: true drop_empty: false drop_last: false inputs_def: fields:
    • image
    • im_info
    • im_id
    • im_shape sample_transforms:
  • !DecodeImage to_rgb: true with_mixup: false
  • !NormalizeImage is_channel_first: false is_scale: true mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
  • !Permute channel_first: true to_bgr: false shuffle: false worker_num: 2 FPN: freeze_norm: false has_extra_convs: false max_level: 6 min_level: 2 norm_type: null num_chan: 256 spatial_scale:
  • 0.03125
  • 0.0625
  • 0.125
  • 0.25 FPNRPNHead: [32manchor_generator[0m: aspect_ratios:
    • 0.5
    • 1.0
    • 2.0 variance:
    • 1.0
    • 1.0
    • 1.0
    • 1.0 [32mrpn_target_assign[0m: rpn_batch_size_per_im: 256 rpn_fg_fraction: 0.5 rpn_negative_overlap: 0.3 rpn_positive_overlap: 0.7 rpn_straddle_thresh: 0.0 [32mtest_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 1000 pre_nms_top_n: 1000 [32mtrain_proposal[0m: min_size: 0.0 nms_thresh: 0.7 post_nms_top_n: 2000 pre_nms_top_n: 2000 anchor_start_size: 32 max_level: 6 min_level: 2 num_chan: 256 num_classes: 1 FPNRoIAlign: [32msampling_ratio[0m: 2 box_resolution: 7 canconical_level: 4 canonical_size: 224 mask_resolution: 14 max_level: 5 min_level: 2 LearningRate: [32mschedulers[0m:
  • !PiecewiseDecay gamma: 0.1 milestones:
    • 120000
    • 160000 values: null
  • !LinearWarmup start_factor: 0.3333333333333333 steps: 500 base_lr: 0.01 MaskAssigner: [32mresolution[0m: 28 num_classes: 81 MaskHead: [32mnum_convs[0m: 4 [32mresolution[0m: 28 conv_dim: 256 dilation: 1 norm_type: null num_classes: 81 OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2 ResNet: [32mnorm_type[0m: affine_channel dcn_v2_stages: [] depth: 50 feature_maps:
  • 2
  • 3
  • 4
  • 5 freeze_at: 2 freeze_norm: true gcb_params: {} gcb_stages: [] nonlocal_stages: [] norm_decay: 0.0 variant: b weight_prefix_name: '' TestReader: batch_size: 1 batch_transforms:
  • !PadBatch pad_to_stride: 32 use_padded_im_info: true dataset: !ImageFolder anno_path: label_test__.txt dataset_dir: '' image_dir: '' sample_num: -1 use_default_label: null with_background: true drop_last: false inputs_def: fields:
    • image
    • im_info
    • im_id
    • im_shape sample_transforms:
  • !DecodeImage to_rgb: true with_mixup: false
  • !NormalizeImage is_channel_first: false is_scale: true mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
  • !Permute channel_first: true to_bgr: false shuffle: false TrainReader: batch_size: 1 batch_transforms:
  • !PadBatch pad_to_stride: 32 use_padded_im_info: false dataset: !PolypsDataSet anno_path: label_train__.txt dataset_dir: dataset/polyps/data image_dir: image_train sample_num: -1 with_background: true drop_last: false inputs_def: fields:
    • image
    • im_info
    • im_id
    • gt_bbox
    • gt_class
    • is_crowd
    • gt_mask sample_transforms:
  • !DecodeImage to_rgb: true with_mixup: false
  • !RandomFlipImage is_mask_flip: false is_normalized: false prob: 0.5
  • !NormalizeImage is_channel_first: false is_scale: true mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
  • !Permute channel_first: true to_bgr: false shuffle: true use_process: false architecture: CascadeMaskRCNN log_smooth_window: 20 max_iters: 500 metric: COCO num_classes: 2 pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_cos_pretrained.tar save_dir: output snapshot_iter: 100 use_gpu: true weights: output/cascade_mask_rcnn_r50_fpn_1x/model_final

W0519 11:06:51.913882 735 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0 W0519 11:06:51.918042 735 device_context.cc:245] device: 0, cuDNN Version: 7.3. 2020-05-19 11:06:53,238-INFO: Load model and fuse batch norm if have from https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_cos_pretrained.tar... 2020-05-19 11:06:53,238-INFO: Found /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained 2020-05-19 11:06:53,243-INFO: Loading parameters from /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained... 2020-05-19 11:06:53,243-WARNING: /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-05-19 11:06:53,243-WARNING: /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-05-19 11:06:53,250-WARNING: variable file [ /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.w_0 /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.b_0 ] not used 2020-05-19 11:06:53,250-WARNING: variable file [ /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.w_0 /home/aistudio/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.b_0 ] not used 2020-05-19 11:06:53,432-INFO: places would be ommited when DataLoader is not iterable 2020-05-19 11:06:53,433-WARNING: Your reader has raised an exception! I0519 11:06:53.449985 735 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0519 11:06:53.482686 735 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0519 11:06:53.565059 735 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0519 11:06:53.589509 735 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 Exception in thread Thread-1: Traceback (most recent call last): File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 805, in thread_main six.reraise(*sys.exc_info()) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 785, in thread_main for tensors in self._tensor_reader(): File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 853, in tensor_reader_impl for slots in paddle_reader(): File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 488, in reader_creator for item in reader(): File "/home/aistudio/work/PaddleDetection/ppdet/data/reader.py", line 415, in _reader for _batch in reader: File "/home/aistudio/work/PaddleDetection/ppdet/data/reader.py", line 301, in next return self.next() File "/home/aistudio/work/PaddleDetection/ppdet/data/reader.py", line 308, in next batch = self._load_batch() File "/home/aistudio/work/PaddleDetection/ppdet/data/reader.py", line 327, in _load_batch if _has_empty(_segm(sample)): File "/home/aistudio/work/PaddleDetection/ppdet/data/reader.py", line 98, in _segm assert 'gt_poly' in samples AssertionError

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "tools/train.py", line 333, in main() File "tools/train.py", line 237, in main outs = exe.run(compiled_train_prog, fetch_list=train_values) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 783, in run six.reraise(*sys.exc_info()) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 778, in run use_program_cache=use_program_cache) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 843, in _run_impl return_numpy=return_numpy) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 677, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >) 3 paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >) 4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&) 5 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 6 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

Python Call Stacks (More useful to users):

File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2525, in append_op attrs=kwargs.get("attrs", None)) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 733, in _init_non_iterable outputs={'Out': self._feed_list}) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 646, in init self._init_non_iterable() File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 280, in from_generator iterable, return_list) File "/home/aistudio/work/PaddleDetection/ppdet/modeling/architectures/cascade_mask_rcnn.py", line 426, in build_inputs iterable=iterable) if use_dataloader else None File "tools/train.py", line 115, in main feed_vars, train_loader = model.build_inputs(**inputs_def) File "tools/train.py", line 333, in main()

Error Message Summary:

Error: Blocking queue is killed because the data reader raises an exception [Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141) [operator < read > error]