PaddlePaddle catches a failure signal, it may not work properly
Created by: imistyrain
之前训练的好好的，后来提示共享内存不足就kill掉几个任务后，然后就再也跑不起来了
2020-06-05 17:43:05,394-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000500] in Optimizer will not take effect, and it will only be applied to other Parameters!
2020-06-05 17:43:06,357-INFO: Dataset /home/ar/gongyanhe/PaddleDetection/dataset/fruit is not valid for reason above, try searching /home/ar/.cache/paddle/dataset or downloading dataset...
2020-06-05 17:43:06,358-INFO: Found /home/ar/.cache/paddle/dataset/fruit/Annotations
2020-06-05 17:43:06,358-INFO: Found /home/ar/.cache/paddle/dataset/fruit/JPEGImages
2020-06-05 17:43:06,372-INFO: places would be ommited when DataLoader is not iterable
W0605 17:43:06.412271 25153 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 9.0, Runtime API Version: 9.0
W0605 17:43:06.442119 25153 device_context.cc:260] device: 0, cuDNN Version: 7.3.
2020-06-05 17:43:11,761-WARNING: /home/ar/.cache/paddle/weights/yolov3_mobilenet_v1.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
2020-06-05 17:43:12,629-WARNING: variable yolo_output.2.conv.weights not used
2020-06-05 17:43:12,629-WARNING: variable yolo_output.1.conv.weights not used
2020-06-05 17:43:12,629-WARNING: variable yolo_output.0.conv.bias not used
2020-06-05 17:43:12,629-WARNING: variable yolo_output.1.conv.bias not used
2020-06-05 17:43:12,630-WARNING: variable yolo_output.0.conv.weights not used
2020-06-05 17:43:12,630-WARNING: variable yolo_output.2.conv.bias not used
2020-06-05 17:43:12,858-INFO: Dataset /home/ar/gongyanhe/PaddleDetection/dataset/fruit is not valid for reason above, try searching /home/ar/.cache/paddle/dataset or downloading dataset...
2020-06-05 17:43:12,858-INFO: Found /home/ar/.cache/paddle/dataset/fruit/Annotations
2020-06-05 17:43:12,858-INFO: Found /home/ar/.cache/paddle/dataset/fruit/JPEGImages
2020-06-05 17:43:12,932-INFO: places would be ommited when DataLoader is not iterable
W0605 17:43:19.655546 25153 init.cc:216] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0605 17:43:19.655645 25153 init.cc:218] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0605 17:43:19.655681 25153 init.cc:221] The detail failure signal is:
W0605 17:43:19.655707 25153 init.cc:224] *** Aborted at 1591350199 (unix time) try "date -d @1591350199" if you are using GNU date ***
W0605 17:43:19.661401 25153 init.cc:224] PC: @ 0x0 (unknown)
W0605 17:43:19.661814 25153 init.cc:224] *** SIGSEGV (@0x0) received by PID 25153 (TID 0x7fc1e3133740) from PID 0; stack trace: ***
W0605 17:43:19.662206 25153 init.cc:224] @ 0x38c040f130 (unknown)
W0605 17:43:19.665493 25153 init.cc:224] @ 0x0 (unknown)
./train.sh: line 7: 25153 Segmentation fault python3 -u tools/train.py -c ${CFG} --eval --use_vdl=True --vdl_log_dir=vdl_${DATASET}_dir/scalar