Created by: flishwang

使用cascade_rcnn_r200训练的模型，进行推理时，如果test_proposals的参数min_size设为非0，在个别图片上会报错误。

请问这个bug应该在这里反馈，还是在PaddlePaddle的issues里反馈？

具体报错的日志如下：

2020-06-28 09:55:50,943-INFO: loading roidb 2012_debug 2020-06-28 09:57:29,972-INFO: load roidb from scope 2012_debug 2020-06-28 09:57:29,972-INFO: finish loading roidbs, total num = 500 2020-06-28 09:57:29,972-INFO: set max batches to 0 2020-06-28 09:57:29,973-INFO: places would be ommited when DataLoader is not iterable W0628 09:57:30.245823 4165090 device_context.cc:252] Please NOTE: device: 4, CUDA Capability: 70, Driver API Version: 10.1, Runtime API Version: 10.0 W0628 09:57:31.638833 4165090 device_context.cc:260] device: 4, cuDNN Version: 7.6. 2020-06-28 10:00:02,147-INFO: Test iter 0 2020-06-28 10:02:24,172-INFO: Test iter 100 2020-06-28 10:04:39,244-INFO: Test iter 200 /home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py:1070: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "tools/eval.py", line 243, in main() File "tools/eval.py", line 180, in main sub_eval_prog, sub_keys, sub_values, resolution) File "/home/bwang/projects/sniper-paddle/ppdet/utils/eval_utils.py", line 134, in eval_run return_merged=False) File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 1071, in run six.reraise(*sys.exc_info()) File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 1066, in run return_merged=return_merged) File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 1167, in _run_impl return_merged=return_merged) File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 879, in _run_parallel tensors = exe.run(fetch_var_names, return_merged)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 void paddle::operators::GPUGather<float, int>(paddle::platform::DeviceContext const&, paddle::framework::Tensor const&, paddle::framework::Tensor const&, paddle::framework::Tensor*) 3 paddle::operators::CUDAGenerateProposalsKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 4 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::CUDAGenerateProposalsKernel<paddle::platform::CUDADeviceContext, float> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 6 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 7 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 8 paddle::framework::details::ComputationOpHandle::RunImpl() 9 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) 10 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 11 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 12 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 13 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

Python Call Stacks (More useful to users):

File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2610, in append_op attrs=kwargs.get("attrs", None)) File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/home/bwang/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layers/detection.py", line 2844, in generate_proposals 'RpnRoisLod': rpn_rois_lod File "/home/bwang/projects/sniper-paddle/ppdet/core/workspace.py", line 150, in partial_apply return op(*args, **kwargs_) File "/home/bwang/projects/sniper-paddle/ppdet/modeling/anchor_heads/rpn_head.py", line 438, in _get_single_proposals variances=self.anchor_var) File "/home/bwang/projects/sniper-paddle/ppdet/modeling/anchor_heads/rpn_head.py", line 462, in get_proposals fpn_feat, im_info, lvl, mode) File "/home/bwang/projects/sniper-paddle/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 197, in build_multi_scale body_feats, im_info, mode="test") File "/home/bwang/projects/sniper-paddle/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 330, in eval return self.build_multi_scale(feed_vars) File "tools/eval.py", line 105, in main fetches = model.eval(feed_vars, multi_scale_test) File "tools/eval.py", line 243, in main()

Error Message Summary:

Error: The index of gather_op should not be empty when the index's rank is 1. [Hint: Expected index.dims()[0] > 0, but received index.dims()[0]:0 <= 0:0.] at (/paddle/paddle/fluid/operators/gather.cu.h:82) [operator < generate_proposals > error] terminate called without an active exception W0628 10:05:48.434573 4165181 init.cc:216] Warning: PaddlePaddle catches a failure signal, it may not work properly W0628 10:05:48.434607 4165181 init.cc:218] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0628 10:05:48.434615 4165181 init.cc:221] The detail failure signal is:

W0628 10:05:48.434626 4165181 init.cc:224] *** Aborted at 1593309948 (unix time) try "date -d @1593309948" if you are using GNU date *** W0628 10:05:48.439401 4165181 init.cc:224] PC: @ 0x0 (unknown) W0628 10:05:48.560981 4165181 init.cc:224] *** SIGABRT (@0x2716003f8de2) received by PID 4165090 (TID 0x7fe905fb8700) from PID 4165090; stack trace: *** W0628 10:05:48.565740 4165181 init.cc:224] @ 0x7fe96d5c4390 (unknown) W0628 10:05:48.570439 4165181 init.cc:224] @ 0x7fe96d21e428 gsignal W0628 10:05:48.575121 4165181 init.cc:224] @ 0x7fe96d22002a abort W0628 10:05:48.909902 4165181 init.cc:224] @ 0x7fe8bdcd084a __gnu_cxx::__verbose_terminate_handler() W0628 10:05:48.912384 4165181 init.cc:224] @ 0x7fe8bdccef47 __cxxabiv1::__terminate() W0628 10:05:48.920951 4165181 init.cc:224] @ 0x7fe8bdccef7d std::terminate() W0628 10:05:48.923445 4165181 init.cc:224] @ 0x7fe8bdccec5a __gxx_personality_v0 W0628 10:05:48.927270 4165181 init.cc:224] @ 0x7fe96a790b97 _Unwind_ForcedUnwind_Phase2 W0628 10:05:48.931037 4165181 init.cc:224] @ 0x7fe96a790e7d _Unwind_ForcedUnwind W0628 10:05:48.934270 4165181 init.cc:224] @ 0x7fe96d5c3070 __GI___pthread_unwind W0628 10:05:48.937424 4165181 init.cc:224] @ 0x7fe96d5bb845 __pthread_exit W0628 10:05:49.322953 4165181 init.cc:224] @ 0x55eadd5ea059 PyThread_exit_thread W0628 10:05:49.323395 4165181 init.cc:224] @ 0x55eadd46fc10 PyEval_RestoreThread.cold.799 W0628 10:05:49.345948 4165181 init.cc:224] @ 0x7fe88283f5c9 pybind11::gil_scoped_release::~gil_scoped_release()