Created by: ff3141044702

Ubuntu 16.04 + cuda 9.0 + cudnn 7.6.1 + paddlepaddle 1.6.1 + python 3.5.2,使用从github上下载的代码，加载自定义的数据进行图像分类训练报错： root@P510:/home/work/models/PaddleCV/image_classification# python3 train.py --model=ResNet50 --batch_size=4 --total_images=8178 --class_dim=2 --image_shape=3,224,224 --model_save_dir=output/ --lr_strategy=piecewise_decay --lr=0.1 --data_dir=/home/work/Data --use_gpu=true ------------- Configuration Arguments ------------- batch_size : 4 checkpoint : None class_dim : 2 crop_size : 224 data_dir : /home/work/Data decay_epochs : 2.4 decay_rate : 0.97 drop_connect_rate : 0.2 ema_decay : 0.9999 image_mean : [0.485, 0.456, 0.406] image_shape : 3,224,224 image_std : [0.229, 0.224, 0.225] interpolation : None is_profiler : 0 l2_decay : 0.0001 label_smoothing_epsilon : 0.1 lower_ratio : 0.75 lower_scale : 0.08 lr : 0.1 lr_strategy : piecewise_decay max_iter : 0 mixup_alpha : 0.2 model : ResNet50 model_save_dir : output/ momentum_rate : 0.9 num_epochs : 120 padding_type : SAME pretrained_model : None print_step : 10 profiler_path : ./ random_seed : None reader_buf_size : 2048 reader_thread : 8 resize_short_size : 256 save_step : 1 step_epochs : [30, 60, 90] test_batch_size : 16 total_images : 8178 upper_ratio : 1.3333333333333333 use_aa : False use_ema : False use_gpu : 1 use_label_smoothing : False use_mixup : False use_se : True validate : 1 warm_up_epochs : 5.0

W1122 18:14:27.471949 3272 device_context.cc:235] Please NOTE: device: 0, CUDA Capability: 35, Driver API Version: 9.0, Runtime API Version: 9.0 W1122 18:14:27.475286 3272 device_context.cc:243] device: 0, cuDNN Version: 7.6. I1122 18:14:28.221356 3272 parallel_executor.cc:421] The number of CUDAPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies I1122 18:14:28.250690 3272 build_strategy.cc:363] SeqOnlyAllReduceOps:0, num_trainers:1 I1122 18:14:28.286288 3272 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True I1122 18:14:28.305943 3272 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 /usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py:774: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "train.py", line 258, in main() File "train.py", line 254, in main train(args) File "train.py", line 208, in train fetch_list=train_fetch_list) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 775, in run six.reraise(*sys.exc_info()) File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise raise value File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 770, in run use_program_cache=use_program_cache) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 829, in _run_impl return_numpy=return_numpy) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/executor.py", line 669, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 paddle::operators::TopkOp::InferShape(paddle::framework::InferShapeContext*) const 3 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const 4 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const 5 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) 6 paddle::framework::details::ComputationOpHandle::RunImpl() 7 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) 8 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 9 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 10 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 11 ThreadPool::ThreadPool(unsigned long)::{lambda()#1 (closed)}::operator()() const

Python Call Stacks (More useful to users):

File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/framework.py", line 2459, in append_op attrs=kwargs.get("attrs", None)) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/layers/nn.py", line 7127, in topk attrs=attrs) File "/usr/local/lib/python3.5/dist-packages/paddle/fluid/layers/metric_op.py", line 87, in accuracy topk_out, topk_indices = nn.topk(input, k=k) File "/home/work/models/PaddleCV/image_classification/build_model.py", line 51, in _basic_model acc_top5 = fluid.layers.accuracy(input=softmax_out, label=label, k=5) File "/home/work/models/PaddleCV/image_classification/build_model.py", line 118, in create_model loss_out = _basic_model(data, model, args, is_train) File "train.py", line 77, in build_program data_loader, loss_out = create_model(model, args, is_train) File "train.py", line 151, in train args=args) File "train.py", line 254, in main train(args) File "train.py", line 258, in main()

Error Message Summary:

PaddleCheckError: Expected input_dims[input_dims.size() - 1] >= k, but received input_dims[input_dims.size() - 1]:2 < k:5. input must have >= k columns at [/paddle/paddle/fluid/operators/top_k_op.cc:40] [operator < top_k > error] terminate called without an active exception W1122 18:14:28.673591 3314 init.cc:205] *** Aborted at 1574417668 (unix time) try "date -d @1574417668" if you are using GNU date *** W1122 18:14:28.676206 3314 init.cc:205] PC: @ 0x0 (unknown) W1122 18:14:28.676353 3314 init.cc:205] *** SIGABRT (@0xcc8) received by PID 3272 (TID 0x7fa1b77fe700) from PID 3272; stack trace: *** W1122 18:14:28.678809 3314 init.cc:205] @ 0x7fa290b4d390 (unknown) W1122 18:14:28.681226 3314 init.cc:205] @ 0x7fa2907a7428 gsignal W1122 18:14:28.682956 3314 init.cc:205] @ 0x7fa2907a902a abort W1122 18:14:28.684118 3314 init.cc:205] @ 0x7fa27fe1a84d __gnu_cxx::__verbose_terminate_handler() W1122 18:14:28.685487 3314 init.cc:205] @ 0x7fa27fe186b6 (unknown) W1122 18:14:28.686635 3314 init.cc:205] @ 0x7fa27fe18701 std::terminate() W1122 18:14:28.687683 3314 init.cc:205] @ 0x7fa27fe182e0 __gxx_personality_v0 W1122 18:14:28.688696 3314 init.cc:205] @ 0x7fa28033b059 (unknown) W1122 18:14:28.689709 3314 init.cc:205] @ 0x7fa28033b3b4 _Unwind_ForcedUnwind W1122 18:14:28.691023 3314 init.cc:205] @ 0x7fa290b4c070 __GI___pthread_unwind W1122 18:14:28.692456 3314 init.cc:205] @ 0x7fa290b44845 __pthread_exit W1122 18:14:28.692644 3314 init.cc:205] @ 0x623f35 PyThread_exit_thread W1122 18:14:28.692802 3314 init.cc:205] @ 0x530264 PyEval_RestoreThread W1122 18:14:28.693331 3314 init.cc:205] @ 0x7fa271edc7ba (unknown) W1122 18:14:28.693490 3314 init.cc:205] @ 0x4e1307 PyCFunction_Call W1122 18:14:28.693657 3314 init.cc:205] @ 0x530b94 PyEval_EvalFrameEx W1122 18:14:28.693820 3314 init.cc:205] @ 0x5350e4 PyEval_EvalFrameEx W1122 18:14:28.693984 3314 init.cc:205] @ 0x53a81b PyEval_EvalCodeEx W1122 18:14:28.694154 3314 init.cc:205] @ 0x4e3537 (unknown) W1122 18:14:28.694278 3314 init.cc:205] @ 0x5c3bd7 PyObject_Call W1122 18:14:28.694414 3314 init.cc:205] @ 0x5f7968 (unknown) W1122 18:14:28.694522 3314 init.cc:205] @ 0x5c3bd7 PyObject_Call W1122 18:14:28.694653 3314 init.cc:205] @ 0x5354af PyEval_EvalFrameEx W1122 18:14:28.694779 3314 init.cc:205] @ 0x53af6a PyEval_EvalCodeEx W1122 18:14:28.694914 3314 init.cc:205] @ 0x4e3537 (unknown) W1122 18:14:28.695022 3314 init.cc:205] @ 0x5c3bd7 PyObject_Call W1122 18:14:28.695153 3314 init.cc:205] @ 0x532a22 PyEval_EvalFrameEx W1122 18:14:28.695284 3314 init.cc:205] @ 0x5350e4 PyEval_EvalFrameEx W1122 18:14:28.695415 3314 init.cc:205] @ 0x5350e4 PyEval_EvalFrameEx W1122 18:14:28.695538 3314 init.cc:205] @ 0x53a81b PyEval_EvalCodeEx W1122 18:14:28.695672 3314 init.cc:205] @ 0x4e3423 (unknown) W1122 18:14:28.695799 3314 init.cc:205] @ 0x5c3bd7 PyObject_Call 已放弃 (核心已转储) root@P510:/home/work/models/PaddleCV/image_classification#