How to solve the problem when I deploy the test of PaddleSeg?
Created by: ljx111
the deploy order is: python infer.py --conf=/path/to/deploy.yaml --input_dir/path/to/images_directory --use_pr=False, reference link: https://github.com/PaddlePaddle/PaddleSeg/tree/release/v0.4.0/deploy/python
The wrong information is following:
I0409 19:09:09.204787 13298 analysis_predictor.cc:84] Profiler is deactivated, and no profiling report will be generated. I0409 19:09:09.218232 13298 analysis_predictor.cc:833] MODEL VERSION: 1.6.1 I0409 19:09:09.218256 13298 analysis_predictor.cc:835] PREDICTOR VERSION: 1.7.1 --- Running analysis [ir_graph_build_pass] --- Running analysis [ir_graph_clean_pass] --- Running analysis [ir_analysis_pass] --- Running IR pass [is_test_pass] --- Running IR pass [simplify_with_basic_ops_pass] --- Running IR pass [conv_affine_channel_fuse_pass] --- Running IR pass [conv_eltwiseadd_affine_channel_fuse_pass] --- Running IR pass [conv_bn_fuse_pass] I0409 19:09:09.338311 13298 graph_pattern_detector.cc:101] --- detected 35 subgraphs --- Running IR pass [conv_eltwiseadd_bn_fuse_pass] --- Running IR pass [multihead_matmul_fuse_pass] --- Running IR pass [fc_fuse_pass] --- Running IR pass [fc_elementwise_layernorm_fuse_pass] --- Running IR pass [conv_elementwise_add_act_fuse_pass] --- Running IR pass [conv_elementwise_add2_act_fuse_pass] --- Running IR pass [conv_elementwise_add_fuse_pass] I0409 19:09:09.420740 13298 graph_pattern_detector.cc:101] --- detected 36 subgraphs --- Running IR pass [transpose_flatten_concat_fuse_pass] --- Running IR pass [runtime_context_cache_pass] --- Running analysis [ir_params_sync_among_devices_pass] I0409 19:09:09.428835 13298 ir_params_sync_among_devices_pass.cc:41] Sync params from CPU to GPU --- Running analysis [adjust_cudnn_workspace_size_pass] --- Running analysis [inference_op_replace_pass] --- Running analysis [memory_optimize_pass] I0409 19:09:09.439870 13298 memory_optimize_pass.cc:223] Cluster name : batch_norm_12.tmp_2 size: 540800 I0409 19:09:09.439889 13298 memory_optimize_pass.cc:223] Cluster name : batch_norm_42.tmp_2 size: 696960 I0409 19:09:09.439903 13298 memory_optimize_pass.cc:223] Cluster name : image size: 3158028 I0409 19:09:09.439906 13298 memory_optimize_pass.cc:223] Cluster name : batch_norm_2.tmp_2 size: 8454272 I0409 19:09:09.439913 13298 memory_optimize_pass.cc:223] Cluster name : batch_norm_4.tmp_2 size: 25362816 I0409 19:09:09.439929 13298 memory_optimize_pass.cc:223] Cluster name : relu6_3.tmp_0 size: 25362816 --- Running analysis [ir_graph_to_program_pass] I0409 19:09:09.455893 13298 analysis_predictor.cc:462] ======= optimize end ======= W0409 19:09:09.820061 13298 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.2, Runtime API Version: 10.0 W0409 19:09:09.822490 13298 device_context.cc:245] device: 0, cuDNN Version: 7.3. W0409 19:09:09.822511 13298 device_context.cc:271] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.6, but CUDNN version in your machine is 7.3, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version. W0409 19:09:10.381486 13298 naive_executor.cc:45] The NaiveExecutor can not work properly if the cmake flag ON_INFER is not set. W0409 19:09:10.381516 13298 naive_executor.cc:47] Unlike the training phase, all the scopes and variables will be reused to save the allocation overhead. W0409 19:09:10.381520 13298 naive_executor.cc:50] Please re-compile the inference library by setting the cmake flag ON_INFER=ON if you are running Paddle Inference Traceback (most recent call last): File "infer.py", line 319, in run(gflags.FLAGS.conf, gflags.FLAGS.input_dir, gflags.FLAGS.ext) File "infer.py", line 301, in run seg_predictor.predict(imgs) File "infer.py", line 272, in predict output_data = self.predictor.run(input_data)[0] paddle.fluid.core_avx.EnforceNotMet:
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 3 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::BatchNormKernel<paddle::platform::CUDADeviceContext, paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1 (closed)}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 4 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 6 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 7 paddle::framework::NaiveExecutor::Run() 8 paddle::AnalysisPredictor::Run(std::vector<paddle::PaddleTensor, std::allocatorpaddle::PaddleTensor > const&, std::vector<paddle::PaddleTensor, std::allocatorpaddle::PaddleTensor >*, int)
Python Call Stacks (More useful to users):
File "/home/chenguowei01/anaconda3/envs/GPU-Paddle/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2459, in append_op attrs=kwargs.get("attrs", None)) File "/home/chenguowei01/anaconda3/envs/GPU-Paddle/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/home/chenguowei01/anaconda3/envs/GPU-Paddle/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 4331, in batch_norm "use_global_stats": use_global_stats File "/home/chenguowei01/github/PaddleSeg/pdseg/models/backbone/mobilenet_v2.py", line 191, in conv_bn_layer moving_variance_name=bn_name + '_variance') File "/home/chenguowei01/github/PaddleSeg/pdseg/models/backbone/mobilenet_v2.py", line 231, in inverted_residual_unit use_cudnn=False) File "/home/chenguowei01/github/PaddleSeg/pdseg/models/backbone/mobilenet_v2.py", line 261, in invresi_blocks name=name + '_1') File "/home/chenguowei01/github/PaddleSeg/pdseg/models/backbone/mobilenet_v2.py", line 128, in net name='conv' + str(i)) File "/home/chenguowei01/github/PaddleSeg/pdseg/models/modeling/deeplab.py", line 199, in mobilenetv2 input, end_points=end_points, decode_points=decode_point) File "/home/chenguowei01/github/PaddleSeg/pdseg/models/modeling/deeplab.py", line 235, in deeplabv3p data, decode_shortcut = mobilenetv2(img) File "/home/chenguowei01/github/PaddleSeg/pdseg/models/model_builder.py", line 173, in build_model logits = model_func(image, class_num) File "pdseg/export_model.py", line 87, in export_inference_model infer_prog, startup_prog, phase=ModelPhase.PREDICT) File "pdseg/export_model.py", line 123, in main export_inference_model(args) File "pdseg/export_model.py", line 127, in main()
Error Message Summary:
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
- New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
- Recommended issue content: all error stack information [Hint: CUDNN_STATUS_BAD_PARAM] at (/paddle/paddle/fluid/operators/batch_norm_op.cu:166) [operator < batch_norm > error]