paddle检测跑evaluate在开头会偶发SIGSEGV错误
Created by: chenqiuhui
paddle版本 conda install --yes paddlepaddle-gpu=1.6.2=py36_gpu_cuda9.0_many_linux
开始会偶发SIGSEGV错误 [INFO 2020-01-1~~~~8 12:23:37,190 PID:2820 voc_eval.py:get_category_info:136] Load categories from ckpt_dir/model_best/label_list.txt 2020-01-18 12:23:37,831-INFO: places would be ommited when DataLoader is not iterable [INFO 2020-01-18 12:23:37,831 PID:2820 reader.py:set_batch_generator:565] places would be ommited when DataLoader is not iterable W0118 12:23:38.818670 2820 device_context.cc:236] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.0, Runtime API Version: 9.0 W0118 12:23:38.823060 2820 device_context.cc:244] device: 0, cuDNN Version: 7.3. 2020-01-18 12:23:39,749-INFO: Loading parameters from ckpt_dir/model_best... [INFO 2020-01-18 12:23:39,749 PID:2820 checkpoint.py:load_params:100] Loading parameters from ckpt_dir/model_best... I0118 12:23:40.311836 2820 parallel_executor.cc:421] The number of CUDAPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies I0118 12:23:40.325747 2820 build_strategy.cc:363] SeqOnlyAllReduceOps:0, num_trainers:1 I0118 12:23:40.338858 2820 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0118 12:23:40.349560 2820 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 W0118 12:23:40.418445 2981 init.cc:206] *** Aborted at 1579321420 (unix time) try "date -d @1579321420" if you are using GNU date *** W0118 12:23:40.422102 2981 init.cc:206] PC: @ 0x0 (unknown) W0118 12:23:40.422422 2981 init.cc:206] *** SIGSEGV (@0x5e380c) received by PID 2820 (TID 0x7fa4beb8f700) from PID 6174732; stack trace: *** W0118 12:23:40.425122 2981 init.cc:206] @ 0x7fa4f4a56390 (unknown) W0118 12:23:40.427670 2981 init.cc:206] @ 0x7fa4f46ff512 cfree W0118 12:23:40.429601 2981 init.cc:206] @ 0x7fa4a4b05fd9 (unknown) W0118 12:23:40.431500 2981 init.cc:206] @ 0x7fa4a48f6318 (unknown) W0118 12:23:40.434484 2981 init.cc:206] @ 0x7fa4f4f9bc54 _PyCFunction_FastCallDict W0118 12:23:40.437188 2981 init.cc:206] @ 0x7fa4f5023abc call_function W0118 12:23:40.440032 2981 init.cc:206] @ 0x7fa4f504675a _PyEval_EvalFrameDefault W0118 12:23:40.442719 2981 init.cc:206] @ 0x7fa4f501ce66 _PyEval_EvalCodeWithName W0118 12:23:40.445508 2981 init.cc:206] @ 0x7fa4f501e37e _PyFunction_FastCallDict W0118 12:23:40.448176 2981 init.cc:206] @ 0x7fa4f4f9c01f _PyObject_FastCallDict W0118 12:23:40.450846 2981 init.cc:206] @ 0x7fa4f4fa0aa3 _PyObject_Call_Prepend W0118 12:23:40.453565 2981 init.cc:206] @ 0x7fa4f4f9ba5e PyObject_Call W0118 12:23:40.456017 2981 init.cc:206] @ 0x7fa4f4ff5371 slot_tp_call W0118 12:23:40.458570 2981 init.cc:206] @ 0x7fa4f4f9be3b _PyObject_FastCallDict W0118 12:23:40.461006 2981 init.cc:206] @ 0x7fa4f5023c0e call_function W0118 12:23:40.463598 2981 init.cc:206] @ 0x7fa4f504675a _PyEval_EvalFrameDefault W0118 12:23:40.466158 2981 init.cc:206] @ 0x7fa4f501e2db _PyFunction_FastCallDict W0118 12:23:40.468714 2981 init.cc:206] @ 0x7fa4f4f9c01f _PyObject_FastCallDict W0118 12:23:40.471263 2981 init.cc:206] @ 0x7fa4f4fa0aa3 _PyObject_Call_Prepend W0118 12:23:40.473860 2981 init.cc:206] @ 0x7fa4f4f9ba5e PyObject_Call W0118 12:23:40.476251 2981 init.cc:206] @ 0x7fa4f4ff5371 slot_tp_call W0118 12:23:40.478822 2981 init.cc:206] @ 0x7fa4f4f9be3b _PyObject_FastCallDict W0118 12:23:40.481257 2981 init.cc:206] @ 0x7fa4f5023c0e call_function W0118 12:23:40.483852 2981 init.cc:206] @ 0x7fa4f504675a _PyEval_EvalFrameDefault W0118 12:23:40.486279 2981 init.cc:206] @ 0x7fa4f501ce66 _PyEval_EvalCodeWithName W0118 12:23:40.488840 2981 init.cc:206] @ 0x7fa4f501e37e _PyFunction_FastCallDict W0118 12:23:40.491394 2981 init.cc:206] @ 0x7fa4f4f9c01f _PyObject_FastCallDict W0118 12:23:40.493947 2981 init.cc:206] @ 0x7fa4f4fa0aa3 _PyObject_Call_Prepend W0118 12:23:40.496541 2981 init.cc:206] @ 0x7fa4f4f9ba5e PyObject_Call W0118 12:23:40.498962 2981 init.cc:206] @ 0x7fa4f508f592 partial_call W0118 12:23:40.501513 2981 init.cc:206] @ 0x7fa4f4f9be3b _PyObject_FastCallDict W0118 12:23:40.503947 2981 init.cc:206] @ 0x7fa4f5023c0e call_function #目前跑评估结束会偶发SIGABRT错误 [INFO 2020-01-18 11:17:57,662 PID:10701 evaluate.py:evaluate:340] evaluate metrics is : [{'mAP(integral)': 0.06508312584639582}] W0118 11:17:57.795034 10953 init.cc:206] *** Aborted at 1579317477 (unix time) try "date -d @1579317477" if you are using GNU date *** W0118 11:17:57.799403 10953 init.cc:206] PC: @ 0x0 (unknown) W0118 11:17:57.799712 10953 init.cc:206] *** SIGABRT (@0x29cd) received by PID 10701 (TID 0x7f4ead860700) from PID 10701; stack trace: *** W0118 11:17:57.804036 10953 init.cc:206] @ 0x7f4ed474d390 (unknown) W0118 11:17:57.808117 10953 init.cc:206] @ 0x7f4ed43a7428 gsignal W0118 11:17:57.811921 10953 init.cc:206] @ 0x7f4ed43a902a abort W0118 11:17:57.815824 10953 init.cc:206] @ 0x7f4ed43e97ea (unknown) W0118 11:17:57.819382 10953 init.cc:206] @ 0x7f4ed43e980e __libc_fatal W0118 11:17:57.822870 10953 init.cc:206] @ 0x7f4ed474c030 unwind_cleanup W0118 11:17:57.826090 10953 init.cc:206] @ 0x7f4eb94f6713 cv::Mat::create() W0118 11:17:57.829103 10953 init.cc:206] @ 0x7f4eb956bf45 cv::_OutputArray::create() W0118 11:17:57.832098 10953 init.cc:206] @ 0x7f4eb9c294de cv::resize() W0118 11:17:57.835079 10953 init.cc:206] @ 0x7f4eccdf77a4 pyopencv_cv_resize() W0118 11:17:57.838836 10953 init.cc:206] @ 0x7f4ed4c92c54 _PyCFunction_FastCallDict W0118 11:17:57.842471 10953 init.cc:206] @ 0x7f4ed4cbf690 _PyCFunction_FastCallKeywords W0118 11:17:57.845892 10953 init.cc:206] @ 0x7f4ed4d1aabc call_function W0118 11:17:57.849356 10953 init.cc:206] @ 0x7f4ed4d3e51c _PyEval_EvalFrameDefault W0118 11:17:57.852572 10953 init.cc:206] @ 0x7f4ed4d13e66 _PyEval_EvalCodeWithName W0118 11:17:57.855959 10953 init.cc:206] @ 0x7f4ed4d1537e _PyFunction_FastCallDict W0118 11:17:57.859200 10953 init.cc:206] @ 0x7f4ed4c9301f _PyObject_FastCallDict W0118 11:17:57.862426 10953 init.cc:206] @ 0x7f4ed4c97aa3 _PyObject_Call_Prepend W0118 11:17:57.865718 10953 init.cc:206] @ 0x7f4ed4c92a5e PyObject_Call W0118 11:17:57.868629 10953 init.cc:206] @ 0x7f4ed4cec371 slot_tp_call W0118 11:17:57.871692 10953 init.cc:206] @ 0x7f4ed4c92e3b _PyObject_FastCallDict W0118 11:17:57.874684 10953 init.cc:206] @ 0x7f4ed4d1ac0e call_function W0118 11:17:57.877710 10953 init.cc:206] @ 0x7f4ed4d3d75a _PyEval_EvalFrameDefault W0118 11:17:57.880630 10953 init.cc:206] @ 0x7f4ed4d152db _PyFunction_FastCallDict W0118 11:17:57.883533 10953 init.cc:206] @ 0x7f4ed4c9301f _PyObject_FastCallDict W0118 11:17:57.886440 10953 init.cc:206] @ 0x7f4ed4c97aa3 _PyObject_Call_Prepend W0118 11:17:57.889334 10953 init.cc:206] @ 0x7f4ed4c92a5e PyObject_Call W0118 11:17:57.891978 10953 init.cc:206] @ 0x7f4ed4cec371 slot_tp_call W0118 11:17:57.894749 10953 init.cc:206] @ 0x7f4ed4c92e3b _PyObject_FastCallDict W0118 11:17:57.897379 10953 init.cc:206] @ 0x7f4ed4d1ac0e call_function W0118 11:17:57.900142 10953 init.cc:206] @ 0x7f4ed4d3d75a _PyEval_EvalFrameDefault W0118 11:17:57.903038 10953 init.cc:206] @ 0x7f4ed4d13e66 _PyEval_EvalCodeWithName