NotFoundError: Input variable(tmp_163) cannot be found in scope for operator 'Fetch'.
Created by: dingsiyu
- 版本、环境信息: 1)PaddlePaddle版本:2.0-alpha 2)CPU: 3)GPU:p40, cuda 9和cudnn 7 4)系统环境:centos6、 python2
INFO: 07-30 16:21:59: ernie_mrc_dataset_reader.py:341 * 139717780932352 set data_generator and start....... /home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py:1101: UserWarning: Now you are using default_main_program, but there are no operators in the program to be executed. Please ensure you create model correctly or you can pass the Program or the CompiledProgram manually. warnings.warn(error_info) /home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py:1078: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") 2020-07-30 16:21:59,900-ERROR: traceback.format_exc():Traceback (most recent call last): File "/ssd3/dingsiyu/ernie2.1/release/textone_migration/finetuning_textone/sc_code_submit_adv_k8s/sc_code_submit_random_adv_recompute_k8s/textone/training/mrc_trainer.py", line 62, in train_and_eval metrics_tensor_value = self.run(InstanceName.TRAINING, need_fetch=True) File "/ssd3/dingsiyu/ernie2.1/release/textone_migration/finetuning_textone/sc_code_submit_adv_k8s/sc_code_submit_random_adv_recompute_k8s/textone/training/base_trainer.py", line 229, in run fetch_output = self.train_exe.run(fetch_list=self.fetch_list_train, program=self.train_program) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1079, in run six.reraise(*sys.exc_info()) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1074, in run return_merged=return_merged) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1162, in _run_impl use_program_cache=use_program_cache) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1237, in _run_program fetch_var_name) EnforceNotMet:
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 paddle::operators::FetchOp::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 3 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 4 paddle::framework::Executor::RunPartialPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, long, long, bool, bool, bool) 5 paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) 6 paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool, bool)
Python Call Stacks (More useful to users):
File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/framework.py", line 2800, in append_op attrs=kwargs.get("attrs", None)) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 632, in _add_feed_fetch_ops attrs={'col': i}) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1232, in _run_program fetch_var_name=fetch_var_name) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1162, in _run_impl use_program_cache=use_program_cache) File "/home/dingsiyu/bin/anaconda3/envs/python2/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1074, in run return_merged=return_merged) File "/ssd3/dingsiyu/ernie2.1/release/textone_migration/finetuning_textone/sc_code_submit_adv_k8s/sc_code_submit_random_adv_recompute_k8s/textone/training/base_trainer.py", line 229, in run fetch_output = self.train_exe.run(fetch_list=self.fetch_list_train, program=self.train_program) File "/ssd3/dingsiyu/ernie2.1/release/textone_migration/finetuning_textone/sc_code_submit_adv_k8s/sc_code_submit_random_adv_recompute_k8s/textone/training/mrc_trainer.py", line 62, in train_and_eval metrics_tensor_value = self.run(InstanceName.TRAINING, need_fetch=True) File "./run_with_json.py", line 107, in trainer.train_and_eval()
Error Message Summary:
NotFoundError: Input variable(tmp_163) cannot be found in scope for operator 'Fetch'.Confirm that you have used the fetch Variable
format instead of the string literal('tmp_163') in fetch_list
parameter when using executor.run
method. In other words, the format of executor.run(fetch_list=[fetch_var])
(fetch_var is a Variable) is recommended.
[Hint: fetch_var should not be null.] at (/paddle/paddle/fluid/operators/controlflow/fetch_op.cc:82)
[operator < fetch > error]