Created by: kolinwei

18:23:55]W: [Step 3/3] I0803 18:23:55.831578 27089 build_strategy.cc:308] SeqOnlyAllReduceOps:0, num_trainers:1 [18:23:55]W: [Step 3/3] I0803 18:23:55.935079 27089 parallel_executor.cc:331] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 [18:24:25]W: [Step 3/3] Traceback (most recent call last): [18:24:25]W: [Step 3/3] File "train.py", line 444, in [18:24:25]W: [Step 3/3] main(args) [18:24:25]W: [Step 3/3] File "train.py", line 299, in main [18:24:25]W: [Step 3/3] outputs = train_exe.run(fetch_list=fetch_list) [18:24:25]W: [Step 3/3] File "/opt/python/cp27-cp27mu/lib/python2.7/site-packages/paddle/fluid/parallel_executor.py", line 280, in run [18:24:25]W: [Step 3/3] return_numpy=return_numpy) [18:24:25]W: [Step 3/3] File "/opt/python/cp27-cp27mu/lib/python2.7/site-packages/paddle/fluid/executor.py", line 614, in run [18:24:25]W: [Step 3/3] raise e [18:24:25]W: [Step 3/3] paddle.fluid.core_avx.EnforceNotMet: Invoke operator lookup_table error. [18:24:25]W: [Step 3/3] Python Callstacks: [18:24:25]W: [Step 3/3] File "/opt/python/cp27-cp27mu/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1776, in append_op [18:24:25]W: [Step 3/3] attrs=kwargs.get("attrs", None)) [18:24:25]W: [Step 3/3] File "/opt/python/cp27-cp27mu/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op [18:24:25]W: [Step 3/3] return self.main_program.current_block().append_op(args, kwargs) [18:24:25]W: [Step 3/3] File "/opt/python/cp27-cp27mu/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 507, in embedding [18:24:25]W: [Step 3/3] 'padding_idx': padding_idx [18:24:25]W: [Step 3/3] File "../../models/dialogue_model_toolkit/dialogue_general_understanding/bert.py", line 96, in _build_model [18:24:25]W: [Step 3/3] is_sparse=False) [18:24:25]W: [Step 3/3] File "../../models/dialogue_model_toolkit/dialogue_general_understanding/bert.py", line 86, in init [18:24:25]W: [Step 3/3] self.build_model(src_ids, position_ids, sentence_ids, input_mask) [18:24:25]W: [Step 3/3] File "../../models/dialogue_model_toolkit/dialogue_general_understanding/create_model.py", line 59, in create_model [18:24:25]W: [Step 3/3] use_fp16=args.use_fp16) [18:24:25]W: [Step 3/3] File "train.py", line 160, in main [18:24:25]W: [Step 3/3] paradigm_inst=paradigm_inst) [18:24:25]W: [Step 3/3] File "train.py", line 444, in [18:24:25]W: [Step 3/3] main(args) [18:24:25]W: [Step 3/3] C++ Callstacks: [18:24:25]W: [Step 3/3] holder should not be null [18:24:25]W: [Step 3/3] Tensor holds no memory. Call Tensor::mutable_data first. at [/workspace/paddle/fluid/framework/tensor.cc:23] [18:24:25]W: [Step 3/3] PaddlePaddle Call Stacks: [18:24:25]W: [Step 3/3] 0 0x7f4169087b58p void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const, int) + 360 [18:24:25]W: [Step 3/3] 1 0x7f4169087ea7p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const, int) + 87 [18:24:25]W: [Step 3/3] 2 0x7f416b1b2529p paddle::framework::Tensor::check_memory_size() const + 185 [18:24:25]W: [Step 3/3] 3 0x7f41690924d9p long const paddle::framework::Tensor::data() const + 25 [18:24:25]W: [Step 3/3] 4 0x7f4169b3c23ep paddle::operators::LookupTableCUDAKernel::Compute(paddle::framework::ExecutionContext const&) const + 1134 [18:24:25]W: [Step 3/3] 5 0x7f4169b3c943p ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform9CUDAPlaceELb0ELm0EINS0_9operators21LookupTableCUDAKernelIfEENSA_IdEENSA_INS7_7float16EEEEEclEPKcSH_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4 + 35 [18:24:25]W: [Step 3/3] 6 0x7f416b15eacbp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 411 [18:24:25]W: [Step 3/3] 7 0x7f416b15f0c1p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529 [18:24:25]W: [Step 3/3] 8 0x7f416b15c43cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332 [18:24:25]W: [Step 3/3] 9 0x7f416af5ea26p paddle::framework::details::ComputationOpHandle::RunImpl() + 166 [18:24:25]W: [Step 3/3] 10 0x7f416ae9b5bcp paddle::framework::details::ThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) + 316 [18:24:25]W: [Step 3/3] 11 0x7f416ae963b5p [18:24:25]W: [Step 3/3] 12 0x7f41692c5c13p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35 [18:24:25]W: [Step 3/3] 13 0x7f4169152be7p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()()>&, bool&) + 39 [18:24:25]W: [Step 3/3] 14 0x7f41f5a48e03p pthread_once + 83 [18:24:25]W: [Step 3/3] 15 0x7f416ae95ea2p [18:24:25]W: [Step 3/3] 16 0x7f4169154164p _ZZN10ThreadPoolC1EmENKUlvE_clEv + 404 [18:24:25]W: [Step 3/3] 17 0x7f4188f00470p [18:24:25]W: [Step 3/3] 18 0x7f41f5a43aa1p [18:24:25]W: [Step 3/3] 19 0x7f41f5105bcdp clone + 109