Created by: littletomatodonkey

程序运行时报错,yolov3+darknet的实验,paddle1.5.0.

WARNING: 07-25 22:05:29: compiler.py:239 * 140666825238272 
You can try our memory optimize feature to save your memory usage:
# create a build_strategy variable to set memory optimize option
build_strategy = compiler.BuildStrategy()
build_strategy.enable_inplace = True
build_strategy.memory_optimize = True

# pass the build_strategy to with_data_parallel API
compiled_prog = compiler.CompiledProgram(main).with_data_parallel(
loss_name=loss.name, build_strategy=build_strategy)

!!! Memory optimize is our experimental feature !!!
some variables may be removed/reused internal to save memory usage, 
in order to fetch the right value of the fetch_list, please set the 
persistable property to true for each variable in fetch_list

# Sample
conv1 = fluid.layers.conv2d(data, 4, 5, 1, act=None) 
# if you need to fetch conv1, then:
conv1.persistable = True


W0725 22:05:30.738020 25347 device_context.cc:259] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.1, Runtime API Version: 9.0
W0725 22:05:30.740799 25347 device_context.cc:267] device: 0, cuDNN Version: 7.0.
W0725 22:05:30.740815 25347 device_context.cc:293] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.3, but CUDNN version in your machine is 7.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
INFO: 07-25 22:05:35: checkpoint.py:58 * 140666825238272 Loading pretrained model from ./thirdparty/pretrain_models/ppdet/DarkNet53_pretrained/...
I0725 22:05:36.072645 25347 parallel_executor.cc:329] The number of CUDAPlace, which is used in ParallelExecutor, is 8. And the Program will be copied 8 copies
I0725 22:05:39.439227 25347 build_strategy.cc:329] set enable_sequential_execution:1
I0725 22:05:39.817229 25347 build_strategy.cc:340] SeqOnlyAllReduceOps:0, num_trainers:1
go into aiflow reader.py...
use visreader as now!!!
INFO: 07-25 22:05:47: reader.py:131 * 140485943351040 connect to filesystem[afs://tianqi.afs.baidu.com:9902]
Traceback (most recent call last):
File "train.py", line 63, in <module>
trainmain()
File "train.py", line 58, in trainmain
train.main()
File "thirdparty/paddlemodels/PaddleDetection/tools/train_aiflow_yolov3.py", line 217, in main
trainmain(FLAGS)
File "thirdparty/paddlemodels/PaddleDetection/tools/train_aiflow_yolov3.py", line 182, in trainmain
outs = exe.run(train_compile_program, fetch_list=train_values)
File "/home/slurm/job/tmp/job-12703/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/executor.py", line 665, in run
return_numpy=return_numpy)
File "/home/slurm/job/tmp/job-12703/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/executor.py", line 527, in _run_parallel
exe.run(fetch_var_names, fetch_var_name)
paddle.fluid.core_avx.EnforceNotMet: Invoke operator conv2d error.
Python Callstacks: 
File "/home/slurm/job/tmp/job-12703/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1748, in append_op
attrs=kwargs.get("attrs", None))
File "/home/slurm/job/tmp/job-12703/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/home/slurm/job/tmp/job-12703/python27-gcc482/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 2171, in conv2d
'fuse_relu_before_depthwise_conv': False
File "thirdparty/paddlemodels/PaddleDetection/ppdet/modeling/backbones/darknet.py", line 63, in _conv_norm
bias_attr=False)
File "thirdparty/paddlemodels/PaddleDetection/ppdet/modeling/backbones/darknet.py", line 145, in __call__
name="yolo_input")
File "thirdparty/paddlemodels/PaddleDetection/ppdet/modeling/architectures/yolov3.py", line 46, in build
body_feats = self.backbone(im)
File "thirdparty/paddlemodels/PaddleDetection/ppdet/modeling/architectures/yolov3.py", line 66, in train
return self.build(feed_vars, mode='train')
File "thirdparty/paddlemodels/PaddleDetection/tools/train_aiflow_yolov3.py", line 109, in trainmain
train_fetches = model.train(feed_vars)
File "thirdparty/paddlemodels/PaddleDetection/tools/train_aiflow_yolov3.py", line 217, in main
trainmain(FLAGS)
File "train.py", line 58, in trainmain
train.main()
File "train.py", line 63, in <module>
trainmain()
C++ Callstacks: 
CUDNN_STATUS_INTERNAL_ERROR at [/paddle/paddle/fluid/platform/device_context.cc:217]
PaddlePaddle Call Stacks: 
0 0x7fef2855a6a0p void paddle::platform::EnforceNotMet::Init<char const*>(char const*, char const*, int) + 352
1 0x7fef2855aa19p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 137
2 0x7fef2a52dd14p paddle::platform::CudnnHolder::CudnnHolder(CUstream_st* const*, paddle::platform::CUDAPlace const&) + 996
3 0x7fef2a52dfe0p
4 0x7fef8b832973p pthread_once + 83
5 0x7fef2a52d25bp paddle::platform::CUDADeviceContext::cudnn_holder() const + 91
6 0x7fef2a52d289p paddle::platform::CUDADeviceContext::cudnn_handle() const + 9
7 0x7fef2897519cp paddle::platform::CanCUDNNBeUsed(paddle::framework::ExecutionContext const&) + 204
8 0x7fef2897111bp paddle::operators::ConvOp::GetExpectedKernelType(paddle::framework::ExecutionContext const&) const + 219
9 0x7fef2a4b58fbp paddle::framework::OperatorWithKernel::ChooseKernel(paddle::framework::RuntimeContext const&, paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 235
10 0x7fef2a4b7a68p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 728
11 0x7fef2a4b7ce1p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529
12 0x7fef2a4b52dcp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332
13 0x7fef2a2b1a2ap paddle::framework::details::ComputationOpHandle::RunImpl() + 250
14 0x7fef2a2a43d0p paddle::framework::details::OpHandleBase::Run(bool) + 160
15 0x7fef2a285746p paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) + 310
16 0x7fef2a2843afp paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue<unsigned long> > const&, unsigned long*) + 47
17 0x7fef2a28476fp
18 0x7fef2878eb43p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35
19 0x7fef28625787p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39
20 0x7fef8b832973p pthread_once + 83
21 0x7fef2a27fdf2p
22 0x7fef28626d04p ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const + 404
23 0x7fef7a4b88a0p
24 0x7fef8b82d1c3p
25 0x7fef8ae5512dp clone + 109