Fluid dist training failed with one trainer instance
Created by: Yancey1989
Related #9351 (closed) , the dist training failed with 1trainer + 1pserver, error logs:
I0330 05:59:49.965085 18618 operator.cc:509] expected_kernel_key:data_type[float32]:data_layout[ANY_LAYOUT]:place[CPUPlace]:library_type[PLAIN]
E0330 05:59:49.981782 18618 listen_and_serv_op.cc:68] run sub program error holder_ should not be null
Tensor holds no memory. Call Tensor::mutable_data first. at [/paddle/paddle/fluid/framework/tensor_impl.h:66]
PaddlePaddle Call Stacks:
0 0x7f76693aaf3cp paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 572
1 0x7f76693afc4fp paddle::framework::Tensor::check_memory_size() const + 303
2 0x7f76694e5014p float const* paddle::framework::Tensor::data<float>() const + 36
3 0x7f7669adecc6p paddle::operators::SGDOpKernel<float>::Compute(paddle::framework::ExecutionContext const&) const + 582
4 0x7f7669ba5fa4p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 1524
5 0x7f7669ba3398p paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 72
6 0x7f766945e446p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool) + 1750
7 0x7f766945f357p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 103
8 0x7f7669b1e903p
9 0x7f7669a2ba0ep std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) + 46
10 0x7f76b4aa2a99p