object365数据集训练报错
Created by: ash12358
按照https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/featured_model/CACascadeRCNN.md 里的说明，把object365数据集按照要求放在了dataset下， 因为PaddleDetection的configs/obj365/下没有上面网址里说的configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn.yml这个文件，只有cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms.yml和cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml这两个文件，所有就执行了CUDA_VISIBLE_DEVICES=2 python tools/train.py -c configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml，
然后报了如下错误，请问是哪里出错了呢：
(paddle) w****@i****:/data/ssh/PaddleDetection$ CUDA_VISIBLE_DEVICES=2 python tools/train.py -c configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml
...
W0422 09:23:00.328461 51055 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.0, Runtime API Version: 10.0
W0422 09:23:00.336308 51055 device_context.cc:245] device: 0, cuDNN Version: 7.6.
2020-04-22 09:23:02,648-INFO: Found /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained
2020-04-22 09:23:02,649-INFO: Loading parameters from /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained...
2020-04-22 09:23:02,649-WARNING: /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
2020-04-22 09:23:02,649-WARNING: /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
2020-04-22 09:23:02,686-WARNING: variable file [ /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_b /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_b ] not used
2020-04-22 09:23:02,686-WARNING: variable file [ /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_b /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_b ] not used
loading annotations into memory...
Done (t=76.26s)
creating index...
index created!
2020-04-22 09:24:41,942-WARNING: Found an invalid bbox in annotations: im_id: 608537, area: 0.0 x1: 269.571228032, y1: 248.73785399999997, x2: 269.651184064, y2: 248.73785399999997.
2020-04-22 09:24:43,904-WARNING: Found an invalid bbox in annotations: im_id: 549169, area: 0.0 x1: 742.3126221, y1: 8.545593250000001, x2: 742.3126221, y2: 10.2289734.
2020-04-22 09:24:44,153-WARNING: Found an invalid bbox in annotations: im_id: 91534, area: 0.0 x1: 433.56384276160003, y1: 420.4645385728, x2: 438.66027832960003, y2: 420.4645385728.
2020-04-22 09:24:44,400-WARNING: Found an invalid bbox in annotations: im_id: 555363, area: 0.0 x1: 337.9105372966, y1: 345.2622006784, x2: 337.9105372966, y2: 348.4704679424.
2020-04-22 09:24:45,600-WARNING: Found an invalid bbox in annotations: im_id: 536966, area: 0.0 x1: 554.6245116984, y1: 299.9646606336, x2: 554.6245116984, y2: 314.069702144.
2020-04-22 09:24:46,032-WARNING: Found an invalid bbox in annotations: im_id: 548210, area: 0.0 x1: 193.9760131584, y1: 273.4229126144, x2: 193.9760131584, y2: 273.5556640768.
2020-04-22 09:24:47,112-WARNING: Found an invalid bbox in annotations: im_id: 580684, area: 0.0 x1: 92.80249024999999, y1: 246.2189331, x2: 92.80249024999999, y2: 251.5336914125.
2020-04-22 09:24:48,239-WARNING: Found an invalid bbox in annotations: im_id: 545995, area: 12.73863515740429 x1: 776.3353271618, y1: 663.263916032, x2: 682.0, y2: 511.0.
2020-04-22 09:24:49,919-WARNING: Found an invalid bbox in annotations: im_id: 691979, area: 7.077892501177361 x1: 840.7648925851, y1: 316.9513549824, x2: 682.0, y2: 318.6118163968.
2020-04-22 09:24:51,937-WARNING: Found an invalid bbox in annotations: im_id: 652823, area: 1.154319683283666 x1: 703.4766845439999, y1: 513.236083968, x2: 639.0, y2: 479.0.
2020-04-22 09:24:54,307-WARNING: Found an invalid bbox in annotations: im_id: 535871, area: 0.0 x1: 444.405273457, y1: 427.4860839936, x2: 444.405273457, y2: 430.7103271424.
2020-04-22 09:24:56,769-WARNING: Found an invalid bbox in annotations: im_id: 359829, area: 46.71179995933256 x1: 613.3458251776, y1: 662.7961425408, x2: 511.0, y2: 671.4617920256001.
2020-04-22 09:25:00,411-WARNING: Found an invalid bbox in annotations: im_id: 533112, area: 0.0 x1: 652.6395257820001, y1: 150.0715779584, x2: 652.6395257820001, y2: 151.5980086272.
2020-04-22 09:25:02,764-WARNING: Found an invalid bbox in annotations: im_id: 280484, area: 4.1640145666077 x1: 683.2222900679001, y1: 426.990234368, x2: 682.0, y2: 428.3464965632.
2020-04-22 09:25:03,322-WARNING: Found an invalid bbox in annotations: im_id: 335604, area: 0.0 x1: 132.0651185036, y1: 490.995319296, x2: 135.152460152, y2: 490.995319296.
2020-04-22 09:25:26,498-WARNING: Found an invalid bbox in annotations: im_id: 545724, area: 3.1565472338744165 x1: 408.56115725259997, y1: 512.0473632768, x2: 411.9129638641, y2: 511.0.
2020-04-22 09:25:27,482-WARNING: Found an invalid bbox in annotations: im_id: 646677, area: 4.7767199789934365 x1: 300.07025145, y1: 390.55169676, x2: 299.0, y2: 391.50653076000003.
2020-04-22 09:25:30,321-WARNING: Found an invalid bbox in annotations: im_id: 574787, area: 0.0 x1: 633.690185536, y1: 370.5393676572, x2: 634.97998048, y2: 370.5393676572.
2020-04-22 09:25:31,066-WARNING: Found an invalid bbox in annotations: im_id: 521927, area: 0.0 x1: 55.575744628100004, y1: 270.0220947456, x2: 57.9324340495, y2: 270.0220947456.
2020-04-22 09:25:32,386-WARNING: Found an invalid bbox in annotations: im_id: 527817, area: 657.4470979715645 x1: 515.3116454912, y1: 674.2288818061, x2: 511.0, y2: 910.0.
2020-04-22 09:25:33,726-WARNING: Found an invalid bbox in annotations: im_id: 145983, area: 0.0 x1: 211.64685056, y1: 203.173278816, x2: 211.64685056, y2: 203.356811504.
2020-04-22 09:25:34,154-WARNING: Found an invalid bbox in annotations: im_id: 599992, area: 50.02852134771837 x1: 718.86511232, y1: 441.29223632120005, x2: 639.0, y2: 435.0.
2020-04-22 09:25:35,601-WARNING: Found an invalid bbox in annotations: im_id: 365040, area: 2.20002733606966 x1: 536.6248779263999, y1: 511.0604858368, x2: 536.7738036992, y2: 511.0.
2020-04-22 09:25:37,722-WARNING: Found an invalid bbox in annotations: im_id: 657457, area: 8.41608131001895 x1: 479.75616456, y1: 0.028564480000000003, x2: 479.0, y2: 531.41967776.
2020-04-22 09:25:38,899-WARNING: Found an invalid bbox in annotations: im_id: 560944, area: 0.0 x1: 281.2375488, y1: 306.20666505599996, x2: 281.417053248, y2: 306.20666505599996.
2020-04-22 09:25:41,188-WARNING: Found an invalid bbox in annotations: im_id: 650302, area: 27.592423196214177 x1: 0, y1: 653.989868159, x2: 4.872924800000002, y2: 477.0.
2020-04-22 09:25:48,904-WARNING: Found an invalid bbox in annotations: im_id: 344510, area: 0.0 x1: 878.1358642939999, y1: 360.5763549696, x2: 771.0, y2: 360.5763549696.
2020-04-22 09:25:51,076-WARNING: Found an invalid bbox in annotations: im_id: 547644, area: 0.0 x1: 432.0398170408, y1: 374.5224859648, x2: 432.0398170408, y2: 377.7821602816.
2020-04-22 09:25:52,973-INFO: 608606 samples in file dataset/objects365/annotations/train.json
2020-04-22 09:26:34,802-INFO: places would be ommited when DataLoader is not iterable
I0422 09:26:34.912900 51055 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel.
I0422 09:26:35.236132 51055 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1
I0422 09:26:37.039860 51055 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I0422 09:26:37.289964 51055 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0
2020-04-22 09:26:37,364-WARNING: Your reader has raised an exception!
Exception in thread Thread-6:
Traceback (most recent call last):
File "/opt/Anaconda3/envs/paddle/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/opt/Anaconda3/envs/paddle/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 805, in __thread_main__
six.reraise(*sys.exc_info())
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 785, in __thread_main__
for tensors in self._tensor_reader():
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 853, in __tensor_reader_impl__
for slots in paddle_reader():
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 488, in __reader_creator__
for item in reader():
File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 415, in _reader
for _batch in reader:
File "/data/ssh/PaddleDetection/ppdet/data/parallel_map.py", line 93, in __next__
return self.next()
File "/data/ssh/PaddleDetection/ppdet/data/parallel_map.py", line 226, in next
self.reset()
File "/data/ssh/PaddleDetection/ppdet/data/parallel_map.py", line 282, in reset
self._source.reset()
File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 283, in reset
p=self.img_weights)
File "mtrand.pyx", line 953, in numpy.random.mtrand.RandomState.choice
ValueError: Fewer non-zero entries in p than size
/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")
Traceback (most recent call last):
File "tools/train.py", line 331, in <module>
main()
File "tools/train.py", line 235, in main
outs = exe.run(compiled_train_prog, fetch_list=train_values)
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 783, in run
six.reraise(*sys.exc_info())
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 778, in run
use_program_cache=use_program_cache)
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 843, in _run_impl
return_numpy=return_numpy)
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 677, in _run_parallel
tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:
--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2 paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> >*)
3 paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> >*)
4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<unsigned long>, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&)
5 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
6 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const
------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2525, in append_op
attrs=kwargs.get("attrs", None))
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 733, in _init_non_iterable
outputs={'Out': self._feed_list})
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 646, in __init__
self._init_non_iterable()
File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 280, in from_generator
iterable, return_list)
File "/data/ssh/PaddleDetection/ppdet/modeling/architectures/cascade_rcnn.py", line 324, in build_inputs
iterable=iterable) if use_dataloader else None
File "tools/train.py", line 115, in main
feed_vars, train_loader = model.build_inputs(**inputs_def)
File "tools/train.py", line 331, in <module>
main()
----------------------
Error Message Summary:
----------------------
Error: Blocking queue is killed because the data reader raises an exception
[Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141)
[operator < read > error]
(paddle) w****@i****:/data/ssh/PaddleDetection$