Error: Cannot find original variable with name ifelse_inputifelse_3_7@GRAD
Created by: AoZhang
-
环境 1)PaddlePaddle版本:1.6.2 2)CPU:Intel(R) Xeon(R) CPU E5-2620 v2 3)GPU:no 4)系统环境:CentOS 6.3 ,Python版本 3.6.5
-
问题描述 程序直接 core 掉了,报了个“Cannot find original variable with name ifelse_inputifelse_3_7@GRAD”的错,哪里梯度传导出了问题?没有显式的打印“Python Call Stacks (More useful to users)”
I0116 15:18:22.781353 1479 parallel_executor.cc:421] The number of CPUPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies
I0116 15:18:22.813120 1479 build_strategy.cc:363] SeqOnlyAllReduceOps:0, num_trainers:1
I0116 15:18:22.855751 1479 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I0116 15:18:22.879025 1479 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 1
/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py:779: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")
Traceback (most recent call last):
File "script/text2sql_train.py", line 45, in <module>
trainer.train_and_eval()
File "/home/xxx/text2sql/textone/training/trainer.py", line 249, in train_and_eval
metrics_tensor_value = self.train_exe.run(fetch_list=self.train_metrics_tensor_name)
File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/parallel_executor.py", line 311, in run
return_numpy=return_numpy)
File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 780, in run
six.reraise(*sys.exc_info())
File "/home/xxx/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 775, in run
use_program_cache=use_program_cache)
File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 834, in _run_impl
return_numpy=return_numpy)
File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 674, in _run_parallel
tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:
--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2 paddle::framework::Scope::RenameInternal(std::string const&, std::string const&) const
3 paddle::framework::Scope::Rename(std::string const&) const
4 paddle::operators::ConditionalBlockGradOp::AssignLocalGradientToGlobal(paddle::platform::Place const&, paddle::framework::Scope const&, std::string const*, unsigned long, std::vector<std::string, std::allocator<std::string> > const&)
const
5 paddle::operators::ConditionalBlockGradOp::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
6 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
7 paddle::framework::details::ComputationOpHandle::RunImpl()
8 paddle::framework::details::OpHandleBase::Run(bool)
9 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*)
10 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue<unsigned long> > const&, unsigned long*)
11 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&)
12 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
13 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const
----------------------
Error Message Summary:
----------------------
Error: Cannot find original variable with name ifelse_inputifelse_3_7@GRAD at (/paddle/paddle/fluid/framework/scope.cc:181)