grpc_client.cc:504] SendRPC name:[fc_0.b_0.opti.trainer_0], ep:[127.0.0.1:8181], status:[-1] meets grpc error, error_code:4 error_message:Deadline Exceeded error_details: *** Check failure stack trace: ***
Created by: SaviorD7
Hello. I am trying to run Secure Aggregation strategy in Paddle FL. I have 1 server and 2 trainers like in example ( paddle_fl >> examples >> sec_agg) I have my x.data and when I running training with my dataset (x.data) on two trainers it getting well. I divided x.data in two parts: 1/2 = y.data , 1/2 = z.data, so x .data = z.data + y.data. But now I am trying to start my training and get this
2020-08-30 13:17:25 Epoch 1 start train, step: 100, accuracy: 0.9375
F0830 13:20:26.802268 1993 grpc_client.cc:504] SendRPC name:[fc_0.b_0.opti.trainer_0], ep:[127.0.0.1:8181], status:[-1] meets grpc error, error_code:4 error_message:Deadline Exceeded error_details:
*** Check failure stack trace: ***
@ 0x7fd823d33c4d google::LogMessage::Fail()
@ 0x7fd823d376fc google::LogMessage::SendToLog()
@ 0x7fd823d33773 google::LogMessage::Flush()
@ 0x7fd823d38c0e google::LogMessageFatal::~LogMessageFatal()
@ 0x7fd826c140a2 paddle::operators::distributed::GRPCClient::Proceed()
@ 0x7fd86a54ec5c execute_native_thread_routine_compat
@ 0x7fd877b946ba start_thread
@ 0x7fd8771ba4dd clone
@ (nil) (unknown)
What I need to fix it? Why I getting this? I am running project in AI Studio on single machine.
Thanks for any help.