generate_proposal报错
Created by: littletomatodonkey
训练cascade_rcnn_cls_aware_r101_vd_fpn模型时，遇到这个问题
Traceback (most recent call last):
File "tools/train.py", line 340, in <module>
main()
File "tools/train.py", line 246, in main
outs = exe.run(compiled_train_prog, fetch_list=train_values)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 774, in run
six.reraise(*sys.exc_info())
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 769, in run
use_program_cache=use_program_cache)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 828, in _run_impl
return_numpy=return_numpy)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 668, in _run_parallel
tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:
--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&&&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2 void paddle::operators::GPUGather<float, int>(paddle::platform::DeviceContext const&, paddle::framework::Tensor const&, paddle::framework::Tensor const&, paddle::framework::Tensor*)
3 paddle::operators::CUDAGenerateProposalsKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
4 _ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform9CUDAPlaceELb0ELm0EJNS0_9operators27CUDAGenerateProposalsKernelINS7_17CUDADeviceContextEfEEEEclEPKcSF_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4_
5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const
6 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
7 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
8 paddle::framework::details::ComputationOpHandle::RunImpl()
9 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*)
10 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue<unsigned long> > const&, unsigned long*)
11 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&)
12 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()()>&, bool&)
13 _ZZN10ThreadPoolC1EmENKUlvE_clEv
------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/framework.py", line 2508, in append_op
attrs=kwargs.get("attrs", None))
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/layers/detection.py", line 2810, in generate_proposals
'RpnRoiProbs': rpn_roi_probs})
File "/ssd1/xiege/PaddleDetection/ppdet/core/workspace.py", line 113, in partial_apply
return op(*args, **kwargs_)
File "/ssd1/xiege/PaddleDetection/ppdet/modeling/anchor_heads/rpn_head.py", line 438, in _get_single_proposals
variances=self.anchor_var)
File "/ssd1/xiege/PaddleDetection/ppdet/modeling/anchor_heads/rpn_head.py", line 462, in get_proposals
fpn_feat, im_info, lvl, mode)
File "/ssd1/xiege/PaddleDetection/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 95, in build
rpn_rois = self.rpn_head.get_proposals(body_feats, im_info, mode=mode)
File "/ssd1/xiege/PaddleDetection/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 178, in train
return self.build(feed_vars, 'train')
File "tools/train.py", line 128, in main
train_fetches = model.train(feed_vars)
File "tools/train.py", line 340, in <module>
main()
----------------------
Error Message Summary:
----------------------
Error: The index of gather_op should not be empty when the index's rank is 1.
[Hint: Expected index.dims()[0] > 0, but received index.dims()[0]:0 <= 0:0.] at (/ssd3/xiege/paddle/Paddle/paddle/fluid/operators/gather.cu.h:82)
[operator < generate_proposals > error]
terminate called without an active exception
W1202 12:22:18.564357 5013 init.cc:205] *** Aborted at 1575289338 (unix time) try "date -d @1575289338" if you are using GNU date ***
W1202 12:22:18.566856 5013 init.cc:205] PC: @ 0x0 (unknown)
W1202 12:22:18.567122 5013 init.cc:205] *** SIGABRT (@0x12c2) received by PID 4802 (TID 0x7f25b6bfd700) from PID 4802; stack trace: ***
W1202 12:22:18.569178 5013 init.cc:205] @ 0x7f2c6a0647e0 (unknown)
W1202 12:22:18.571429 5013 init.cc:205] @ 0x7f2c694604f5 __GI_raise
W1202 12:22:18.573428 5013 init.cc:205] @ 0x7f2c69461cd5 __GI_abort
W1202 12:22:18.574733 5013 init.cc:205] @ 0x7f2bd822aa8d __gnu_cxx::__verbose_terminate_handler()
W1202 12:22:18.575976 5013 init.cc:205] @ 0x7f2bd8228be6 (unknown)
W1202 12:22:18.577111 5013 init.cc:205] @ 0x7f2bd8228c13 std::terminate()
W1202 12:22:18.578280 5013 init.cc:205] @ 0x7f2bd82288a6 __gxx_personality_v0
W1202 12:22:18.579454 5013 init.cc:205] @ 0x7f2c154bb1ce (unknown)
W1202 12:22:18.580602 5013 init.cc:205] @ 0x7f2c154bb2b4 _Unwind_ForcedUnwind
W1202 12:22:18.582444 5013 init.cc:205] @ 0x7f2c6a062f60 __GI___pthread_unwind
W1202 12:22:18.584290 5013 init.cc:205] @ 0x7f2c6a05d175 __pthread_exit
W1202 12:22:18.586236 5013 init.cc:205] @ 0x7f2c6a426c5f PyThread_exit_thread
W1202 12:22:18.588135 5013 init.cc:205] @ 0x7f2c6a3d32aa PyEval_RestoreThread
W1202 12:22:18.590368 5013 init.cc:205] @ 0x7f2baac61dd9 pybind11::gil_scoped_release::~gil_scoped_release()
W1202 12:22:18.591140 5013 init.cc:205] @ 0x7f2baac14643 _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9operators6reader22LoDTensorBlockingQueueERKSt6vectorINS2_9framework9LoDTensorESaISC_EEE58_bIS9_SG_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESY_
W1202 12:22:18.593189 5013 init.cc:205] @ 0x7f2baac83fb1 pybind11::cpp_function::dispatcher()
W1202 12:22:18.595247 5013 init.cc:205] @ 0x7f2c6a33cfbd _PyCFunction_FastCallDict
W1202 12:22:18.597225 5013 init.cc:205] @ 0x7f2c6a3d4138 (unknown)
W1202 12:22:18.599215 5013 init.cc:205] @ 0x7f2c6a3d7acc _PyEval_EvalFrameDefault
W1202 12:22:18.601167 5013 init.cc:205] @ 0x7f2c6a3d3fce (unknown)
W1202 12:22:18.603116 5013 init.cc:205] @ 0x7f2c6a3d45f3 PyEval_EvalCodeEx
W1202 12:22:18.605052 5013 init.cc:205] @ 0x7f2c6a3163f3 (unknown)
W1202 12:22:18.606988 5013 init.cc:205] @ 0x7f2c6a2e434a PyObject_Call
W1202 12:22:18.608937 5013 init.cc:205] @ 0x7f2c6a3d7e55 _PyEval_EvalFrameDefault
W1202 12:22:18.610893 5013 init.cc:205] @ 0x7f2c6a3d3660 (unknown)
W1202 12:22:18.612833 5013 init.cc:205] @ 0x7f2c6a3d4584 (unknown)
W1202 12:22:18.614792 5013 init.cc:205] @ 0x7f2c6a3d7acc _PyEval_EvalFrameDefault
W1202 12:22:18.616705 5013 init.cc:205] @ 0x7f2c6a3d3660 (unknown)
W1202 12:22:18.618620 5013 init.cc:205] @ 0x7f2c6a3d4584 (unknown)
W1202 12:22:18.620504 5013 init.cc:205] @ 0x7f2c6a3d7acc _PyEval_EvalFrameDefault
W1202 12:22:18.622380 5013 init.cc:205] @ 0x7f2c6a3d3660 (unknown)
W1202 12:22:18.624182 5013 init.cc:205] @ 0x7f2c6a3dcc66 _PyFunction_FastCallDict我试了10次，遇到了1次，不知道是否是generate_proposal op的问题~