Created by: zyxcambridge

drop_block: false keep_prob: 0.9 norm_decay: 0.0 num_classes: 80 weight_prefix_name: '' yolo_loss: YOLOv3Loss YOLOv3Loss: batch_size: 8 ignore_thresh: 0.7 iou_loss: null label_smooth: true use_fine_grained_loss: false architecture: YOLOv3 log_smooth_window: 20 max_iters: 500200 metric: COCO num_classes: 80 pretrain_weights: http://paddle-imagenet-models-name.bj.bcebos.com/MobileNetV1_pretrained.tar save_dir: output snapshot_iter: 2000 use_gpu: true weights: output/yolov3_mobilenet_v1/model_final

W0325 01:33:53.457494 30362 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.1, Runtime API Version: 10.0 W0325 01:33:53.459408 30362 device_context.cc:245] device: 0, cuDNN Version: 7.3. W0325 01:33:53.459417 30362 device_context.cc:271] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.6, but CUDNN version in your machine is 7.3, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version. 2020-03-25 01:33:54,444-INFO: Found /root/.cache/paddle/weights/MobileNetV1_pretrained 2020-03-25 01:33:54,444-INFO: Loading parameters from /root/.cache/paddle/weights/MobileNetV1_pretrained... 2020-03-25 01:33:54,444-WARNING: /root/.cache/paddle/weights/MobileNetV1_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-03-25 01:33:54,444-WARNING: /root/.cache/paddle/weights/MobileNetV1_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-03-25 01:33:54,448-WARNING: variable file [ /root/.cache/paddle/weights/MobileNetV1_pretrained/fc7_offset /root/.cache/paddle/weights/MobileNetV1_pretrained/fc7_weights ] not used 2020-03-25 01:33:54,448-WARNING: variable file [ /root/.cache/paddle/weights/MobileNetV1_pretrained/fc7_offset /root/.cache/paddle/weights/MobileNetV1_pretrained/fc7_weights ] not used 2020-03-25 01:33:54,485-INFO: pruned params: ['yolo_block.0.0.0.conv.weights', 'yolo_block.0.0.1.conv.weights', 'yolo_block.0.1.0.conv.weights'] 2020-03-25 01:33:54,485-INFO: pruned ratios: [0.2, 0.3, 0.4] loading annotations into memory... Done (t=9.57s) creating index... index created! 2020-03-25 01:34:06,753-WARNING: Found an invalid bbox in annotations: im_id: 200365, area: 0.0 x1: 296.65, y1: 388.33, x2: 296.67999999999995, y2: 388.33. 2020-03-25 01:34:09,501-WARNING: Found an invalid bbox in annotations: im_id: 550395, area: 0.0 x1: 9.98, y1: 188.56, x2: 14.52, y2: 188.56. 2020-03-25 01:34:11,114-INFO: 118287 samples in file dataset/coco/annotations/instances_train2017.json 2020-03-25 01:34:13,448-INFO: places would be ommited when DataLoader is not iterable I0325 01:34:13.457126 30362 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0325 01:34:13.499372 30362 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0325 01:34:13.524021 30362 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0325 01:34:13.533594 30362 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 /root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "/root/Documents/code/PaddleDetection/slim/prune/prune.py", line 392, in main() File "/root/Documents/code/PaddleDetection/slim/prune/prune.py", line 279, in main outs = exe.run(compiled_train_prog, fetch_list=train_values) File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 783, in run six.reraise(*sys.exc_info()) File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 778, in run use_program_cache=use_program_cache) File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 843, in _run_impl return_numpy=return_numpy) File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/executor.py", line 677, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 3 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 4 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 6 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 7 paddle::framework::details::ComputationOpHandle::RunImpl() 8 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) 9 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 10 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 11 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 12 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

Python Call Stacks (More useful to users):

File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2525, in append_op attrs=kwargs.get("attrs", None)) File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/root/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 2663, in batch_norm type="batch_norm", inputs=inputs, outputs=outputs, attrs=attrs) File "/root/Documents/code/PaddleDetection/ppdet/modeling/backbones/mobilenet.py", line 98, in _conv_norm moving_variance_name=bn_name + '_variance') File "/root/Documents/code/PaddleDetection/ppdet/modeling/backbones/mobilenet.py", line 159, in call input, 3, int(32 * scale), 2, 1, name=self.prefix_name + "conv1") File "/root/Documents/code/PaddleDetection/ppdet/modeling/architectures/yolov3.py", line 61, in build body_feats = self.backbone(im) File "/root/Documents/code/PaddleDetection/ppdet/modeling/architectures/yolov3.py", line 156, in train return self.build(feed_vars, mode='train') File "/root/Documents/code/PaddleDetection/slim/prune/prune.py", line 102, in main train_fetches = model.train(feed_vars) File "/root/Documents/code/PaddleDetection/slim/prune/prune.py", line 392, in main()

Error Message Summary:

Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.

• New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
• Recommended issue content: all error stack information [Hint: CUDNN_STATUS_BAD_PARAM] at (/paddle/paddle/fluid/operators/batch_norm_op.cu:287) [operator < batch_norm > error]

Process finished with exit code 1