PaddleRec/gnn模型cpu训练出错
Created by: zhengya01
python 2.7 cuda 9.0 cudnn7.0
python -u train.py --use_cuda 0 --epoch_num 1
paddle fluid1.2报错
2019-04-04 17:10:35,841 - INFO - load data complete 2019-04-04 17:10:37,749 - INFO - begin train Traceback (most recent call last): File "train.py", line 131, in train() File "train.py", line 113, in train fetch_list=[loss.name, acc.name]) File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/parallel_executor.py", line 301, in run self.executor.run(fetch_list, fetch_var_name) paddle.fluid.core.EnforceNotMet: Enforce failed. Expected output_shape[unk_dim_idx] * capacity == -in_size, but received output_shape[unk_dim_idx] * capacity:0 != -in_size:-6000. Invalid shape is given. at [/paddle/paddle/fluid/operators/reshape_op.cc:98] PaddlePaddle Call Stacks: 0 0x7f94250d76edp void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const*, int) + 365 1 0x7f94250d7a37p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 87 2 0x7f9425486729p paddle::operators::ReshapeOp::ValidateShape(std::vector<int, std::allocator >, paddle::framework::DDim const&) + 2249 3 0x7f9425487e91p paddle::operators::ReshapeOp::InferShape(paddle::framework::InferShapeContext*) const + 689 4 0x7f94254885a9p paddle::operators::Reshape2Op::InferShape(paddle::framework::InferShapeContext*) const + 521 5 0x7f9426b9815fp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 607 6 0x7f9426b962b5p paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 341 7 0x7f9426a0c7c9p 8 0x7f9426a04c71p paddle::framework::details::OpHandleBase::RunAndRecordEvent(std::function<void ()> const&) + 769 9 0x7f9426a0c45cp paddle::framework::details::ComputationOpHandle::RunImpl() + 124 10 0x7f9426a05bb6p paddle::framework::details::OpHandleBase::Run(bool) + 118 11 0x7f942699dfbdp 12 0x7f9425d5abe3p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35 13 0x7f9425d1df07p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39 14 0x7f945c49fbe0p pthread_once + 80 15 0x7f942699cca2p 16 0x7f9425d1f334p ThreadPool::ThreadPool(unsigned long)::{lambda()#1 (closed)}::operator()() const + 404 17 0x7f94421eb678p 18 0x7f945c49adf3p 19 0x7f945babf2cdp clone + 109
paddle develop 报错
2019-04-04 17:19:39,193-INFO: load data complete
ParallelExecutor is deprecated. Please use CompiledProgram and Executor. CompiledProgram is a central place for optimization and Executor is the unified executor. Example can be found in compiler.py.
W0404 17:19:39.320032 16823 graph.h:204] WARN: After a series of passes, the current graph can be quite different from OriginProgram. So, please avoid using the OriginProgram()
method!
I0404 17:19:41.131080 16823 build_strategy.cc:289] SeqOnlyAllReduceOps:0, num_trainers:1
2019-04-04 17:19:41,270-INFO: begin train
Traceback (most recent call last):
File "train.py", line 131, in
train()
File "train.py", line 113, in train
fetch_list=[loss.name, acc.name])
File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/parallel_executor.py", line 199, in run
return_numpy=return_numpy)
File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 544, in run
return_numpy=return_numpy)
File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 445, in run_parallel
exe.run(fetch_var_names, fetch_var_name)
paddle.fluid.core.EnforceNotMet: Invoke operator reshape2 error.
Python Callstacks:
File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1663, in append_op
attrs=kwargs.get("attrs", None))
File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(args, kwargs)
File "/home/paddle/anaconda2/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 6409, in reshape
"XShape": x_shape})
File "/home/paddle/zhengya01/paddle-ce/local_model/gnn/network.py", line 72, in network
x=pre_state, shape=[batch_size, -1, hidden_size])
File "train.py", line 66, in train
args.step)
File "train.py", line 131, in
train()
C++ Callstacks:
Enforce failed. Expected output_shape[unk_dim_idx] * capacity == -in_size, but received output_shape[unk_dim_idx] * capacity:0 != -in_size:-7400.
Invalid shape is given. at [/paddle/paddle/fluid/operators/reshape_op.cc:101]
PaddlePaddle Call Stacks:
0 0x7fd373541338p void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const, int) + 360
1 0x7fd373541687p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const, int) + 87
2 0x7fd373af1334p paddle::operators::ReshapeOp::ValidateShape(std::vector<int, std::allocator >, paddle::framework::DDim const&) + 1956
3 0x7fd373af3131p paddle::operators::ReshapeOp::InferShape(paddle::framework::InferShapeContext) const + 689
4 0x7fd373af3849p paddle::operators::Reshape2Op::InferShape(paddle::framework::InferShapeContext*) const + 521
5 0x7fd3751e7365p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 613
6 0x7fd3751e7b24p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 292
7 0x7fd3751e543cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332
8 0x7fd374ff8639p
9 0x7fd374feb0f5p paddle::framework::details::OpHandleBase::RunAndRecordEvent(std::function<void ()> const&) + 773
10 0x7fd374ff82ccp paddle::framework::details::ComputationOpHandle::RunImpl() + 124
11 0x7fd374feb400p paddle::framework::details::OpHandleBase::Run(bool) + 160
12 0x7fd374f5398dp
13 0x7fd3742a73c3p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35
14 0x7fd37360f997p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39
15 0x7fd3ab82ebe0p pthread_once + 80
16 0x7fd374f53522p
17 0x7fd373610f14p ThreadPool::ThreadPool(unsigned long)::{lambda()#1 (closed)}::operator()() const + 404
18 0x7fd39157a678p
19 0x7fd3ab829df3p
20 0x7fd3aae4e2cdp clone + 109