Skip to content

  • 体验新版
    • 正在加载...
  • 登录
  • PaddlePaddle
  • PaddleDetection
  • Issue
  • #538

P
PaddleDetection
  • 项目概览

PaddlePaddle / PaddleDetection
大约 2 年 前同步成功

通知 708
Star 11112
Fork 2696
  • 代码
    • 文件
    • 提交
    • 分支
    • Tags
    • 贡献者
    • 分支图
    • Diff
  • Issue 184
    • 列表
    • 看板
    • 标记
    • 里程碑
  • 合并请求 40
  • Wiki 0
    • Wiki
  • 分析
    • 仓库
    • DevOps
  • 项目成员
  • Pages
P
PaddleDetection
  • 项目概览
    • 项目概览
    • 详情
    • 发布
  • 仓库
    • 仓库
    • 文件
    • 提交
    • 分支
    • 标签
    • 贡献者
    • 分支图
    • 比较
  • Issue 184
    • Issue 184
    • 列表
    • 看板
    • 标记
    • 里程碑
  • 合并请求 40
    • 合并请求 40
  • Pages
  • 分析
    • 分析
    • 仓库分析
    • DevOps
  • Wiki 0
    • Wiki
  • 成员
    • 成员
  • 收起侧边栏
  • 动态
  • 分支图
  • 创建新Issue
  • 提交
  • Issue看板
已关闭
开放中
Opened 4月 22, 2020 by saxon_zh@saxon_zhGuest

object365数据集训练报错

Created by: ash12358

按照https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/featured_model/CACascadeRCNN.md 里的说明,把object365数据集按照要求放在了dataset下, 因为PaddleDetection的configs/obj365/下没有上面网址里说的configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn.yml这个文件,只有cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms.yml和cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml这两个文件,所有就执行了CUDA_VISIBLE_DEVICES=2 python tools/train.py -c configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml,

然后报了如下错误,请问是哪里出错了呢:

(paddle) w****@i****:/data/ssh/PaddleDetection$ CUDA_VISIBLE_DEVICES=2 python tools/train.py -c configs/obj365/cascade_rcnn_dcnv2_se154_vd_fpn_gn_cas.yml
...

W0422 09:23:00.328461 51055 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.0, Runtime API Version: 10.0
W0422 09:23:00.336308 51055 device_context.cc:245] device: 0, cuDNN Version: 7.6.
2020-04-22 09:23:02,648-INFO: Found /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained
2020-04-22 09:23:02,649-INFO: Loading parameters from /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained...
2020-04-22 09:23:02,649-WARNING: /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
2020-04-22 09:23:02,649-WARNING: /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
2020-04-22 09:23:02,686-WARNING: variable file [ /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_b /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_b ] not used
2020-04-22 09:23:02,686-WARNING: variable file [ /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_b /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_offset /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/conv5_mask_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_4_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_3_weights /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_1_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_w /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_inter_feat_2_scale /home/w****/.cache/paddle/weights/cascade_mask_rcnn_dcnv2_se154_vd_fpn_gn_coco_pretrained/mask_fcn_logits_b ] not used
loading annotations into memory...
Done (t=76.26s)
creating index...
index created!
2020-04-22 09:24:41,942-WARNING: Found an invalid bbox in annotations: im_id: 608537, area: 0.0 x1: 269.571228032, y1: 248.73785399999997, x2: 269.651184064, y2: 248.73785399999997.
2020-04-22 09:24:43,904-WARNING: Found an invalid bbox in annotations: im_id: 549169, area: 0.0 x1: 742.3126221, y1: 8.545593250000001, x2: 742.3126221, y2: 10.2289734.
2020-04-22 09:24:44,153-WARNING: Found an invalid bbox in annotations: im_id: 91534, area: 0.0 x1: 433.56384276160003, y1: 420.4645385728, x2: 438.66027832960003, y2: 420.4645385728.
2020-04-22 09:24:44,400-WARNING: Found an invalid bbox in annotations: im_id: 555363, area: 0.0 x1: 337.9105372966, y1: 345.2622006784, x2: 337.9105372966, y2: 348.4704679424.
2020-04-22 09:24:45,600-WARNING: Found an invalid bbox in annotations: im_id: 536966, area: 0.0 x1: 554.6245116984, y1: 299.9646606336, x2: 554.6245116984, y2: 314.069702144.
2020-04-22 09:24:46,032-WARNING: Found an invalid bbox in annotations: im_id: 548210, area: 0.0 x1: 193.9760131584, y1: 273.4229126144, x2: 193.9760131584, y2: 273.5556640768.
2020-04-22 09:24:47,112-WARNING: Found an invalid bbox in annotations: im_id: 580684, area: 0.0 x1: 92.80249024999999, y1: 246.2189331, x2: 92.80249024999999, y2: 251.5336914125.
2020-04-22 09:24:48,239-WARNING: Found an invalid bbox in annotations: im_id: 545995, area: 12.73863515740429 x1: 776.3353271618, y1: 663.263916032, x2: 682.0, y2: 511.0.
2020-04-22 09:24:49,919-WARNING: Found an invalid bbox in annotations: im_id: 691979, area: 7.077892501177361 x1: 840.7648925851, y1: 316.9513549824, x2: 682.0, y2: 318.6118163968.
2020-04-22 09:24:51,937-WARNING: Found an invalid bbox in annotations: im_id: 652823, area: 1.154319683283666 x1: 703.4766845439999, y1: 513.236083968, x2: 639.0, y2: 479.0.
2020-04-22 09:24:54,307-WARNING: Found an invalid bbox in annotations: im_id: 535871, area: 0.0 x1: 444.405273457, y1: 427.4860839936, x2: 444.405273457, y2: 430.7103271424.
2020-04-22 09:24:56,769-WARNING: Found an invalid bbox in annotations: im_id: 359829, area: 46.71179995933256 x1: 613.3458251776, y1: 662.7961425408, x2: 511.0, y2: 671.4617920256001.
2020-04-22 09:25:00,411-WARNING: Found an invalid bbox in annotations: im_id: 533112, area: 0.0 x1: 652.6395257820001, y1: 150.0715779584, x2: 652.6395257820001, y2: 151.5980086272.
2020-04-22 09:25:02,764-WARNING: Found an invalid bbox in annotations: im_id: 280484, area: 4.1640145666077 x1: 683.2222900679001, y1: 426.990234368, x2: 682.0, y2: 428.3464965632.
2020-04-22 09:25:03,322-WARNING: Found an invalid bbox in annotations: im_id: 335604, area: 0.0 x1: 132.0651185036, y1: 490.995319296, x2: 135.152460152, y2: 490.995319296.
2020-04-22 09:25:26,498-WARNING: Found an invalid bbox in annotations: im_id: 545724, area: 3.1565472338744165 x1: 408.56115725259997, y1: 512.0473632768, x2: 411.9129638641, y2: 511.0.
2020-04-22 09:25:27,482-WARNING: Found an invalid bbox in annotations: im_id: 646677, area: 4.7767199789934365 x1: 300.07025145, y1: 390.55169676, x2: 299.0, y2: 391.50653076000003.
2020-04-22 09:25:30,321-WARNING: Found an invalid bbox in annotations: im_id: 574787, area: 0.0 x1: 633.690185536, y1: 370.5393676572, x2: 634.97998048, y2: 370.5393676572.
2020-04-22 09:25:31,066-WARNING: Found an invalid bbox in annotations: im_id: 521927, area: 0.0 x1: 55.575744628100004, y1: 270.0220947456, x2: 57.9324340495, y2: 270.0220947456.
2020-04-22 09:25:32,386-WARNING: Found an invalid bbox in annotations: im_id: 527817, area: 657.4470979715645 x1: 515.3116454912, y1: 674.2288818061, x2: 511.0, y2: 910.0.
2020-04-22 09:25:33,726-WARNING: Found an invalid bbox in annotations: im_id: 145983, area: 0.0 x1: 211.64685056, y1: 203.173278816, x2: 211.64685056, y2: 203.356811504.
2020-04-22 09:25:34,154-WARNING: Found an invalid bbox in annotations: im_id: 599992, area: 50.02852134771837 x1: 718.86511232, y1: 441.29223632120005, x2: 639.0, y2: 435.0.
2020-04-22 09:25:35,601-WARNING: Found an invalid bbox in annotations: im_id: 365040, area: 2.20002733606966 x1: 536.6248779263999, y1: 511.0604858368, x2: 536.7738036992, y2: 511.0.
2020-04-22 09:25:37,722-WARNING: Found an invalid bbox in annotations: im_id: 657457, area: 8.41608131001895 x1: 479.75616456, y1: 0.028564480000000003, x2: 479.0, y2: 531.41967776.
2020-04-22 09:25:38,899-WARNING: Found an invalid bbox in annotations: im_id: 560944, area: 0.0 x1: 281.2375488, y1: 306.20666505599996, x2: 281.417053248, y2: 306.20666505599996.
2020-04-22 09:25:41,188-WARNING: Found an invalid bbox in annotations: im_id: 650302, area: 27.592423196214177 x1: 0, y1: 653.989868159, x2: 4.872924800000002, y2: 477.0.
2020-04-22 09:25:48,904-WARNING: Found an invalid bbox in annotations: im_id: 344510, area: 0.0 x1: 878.1358642939999, y1: 360.5763549696, x2: 771.0, y2: 360.5763549696.
2020-04-22 09:25:51,076-WARNING: Found an invalid bbox in annotations: im_id: 547644, area: 0.0 x1: 432.0398170408, y1: 374.5224859648, x2: 432.0398170408, y2: 377.7821602816.
2020-04-22 09:25:52,973-INFO: 608606 samples in file dataset/objects365/annotations/train.json
2020-04-22 09:26:34,802-INFO: places would be ommited when DataLoader is not iterable
I0422 09:26:34.912900 51055 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel.
I0422 09:26:35.236132 51055 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1
I0422 09:26:37.039860 51055 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I0422 09:26:37.289964 51055 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0
2020-04-22 09:26:37,364-WARNING: Your reader has raised an exception!
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 805, in __thread_main__
    six.reraise(*sys.exc_info())
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 785, in __thread_main__
    for tensors in self._tensor_reader():
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 853, in __tensor_reader_impl__
    for slots in paddle_reader():
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 488, in __reader_creator__
    for item in reader():
  File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 415, in _reader
    for _batch in reader:
  File "/data/ssh/PaddleDetection/ppdet/data/parallel_map.py", line 93, in __next__
    return self.next()
  File "/data/ssh/PaddleDetection/ppdet/data/parallel_map.py", line 226, in next
    self.reset()
  File "/data/ssh/PaddleDetection/ppdet/data/parallel_map.py", line 282, in reset
    self._source.reset()
  File "/data/ssh/PaddleDetection/ppdet/data/reader.py", line 283, in reset
    p=self.img_weights)
  File "mtrand.pyx", line 953, in numpy.random.mtrand.RandomState.choice
ValueError: Fewer non-zero entries in p than size

/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception.
  "The following exception is not an EOF exception.")
Traceback (most recent call last):
  File "tools/train.py", line 331, in <module>
    main()
  File "tools/train.py", line 235, in main
    outs = exe.run(compiled_train_prog, fetch_list=train_values)
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 783, in run
    six.reraise(*sys.exc_info())
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 778, in run
    use_program_cache=use_program_cache)
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 843, in _run_impl
    return_numpy=return_numpy)
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 677, in _run_parallel
    tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2   paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> >*)
3   paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> >*)
4   std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<unsigned long>, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&)
5   std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
6   ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2525, in append_op
    attrs=kwargs.get("attrs", None))
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 733, in _init_non_iterable
    outputs={'Out': self._feed_list})
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 646, in __init__
    self._init_non_iterable()
  File "/opt/Anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/reader.py", line 280, in from_generator
    iterable, return_list)
  File "/data/ssh/PaddleDetection/ppdet/modeling/architectures/cascade_rcnn.py", line 324, in build_inputs
    iterable=iterable) if use_dataloader else None
  File "tools/train.py", line 115, in main
    feed_vars, train_loader = model.build_inputs(**inputs_def)
  File "tools/train.py", line 331, in <module>
    main()

----------------------
Error Message Summary:
----------------------
Error: Blocking queue is killed because the data reader raises an exception
  [Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141)
  [operator < read > error]
(paddle) w****@i****:/data/ssh/PaddleDetection$
指派人
分配到
无
里程碑
无
分配里程碑
工时统计
无
截止日期
无
标识: paddlepaddle/PaddleDetection#538
渝ICP备2023009037号

京公网安备11010502055752号

网络110报警服务 Powered by GitLab CE v13.7
开源知识
Git 入门 Pro Git 电子书 在线学 Git
Markdown 基础入门 IT 技术知识开源图谱
帮助
使用手册 反馈建议 博客
《GitCode 隐私声明》 《GitCode 服务条款》 关于GitCode
Powered by GitLab CE v13.7