训练一轮后任务失败status:[-1] meets grpc error, error_code:4 error_message:Deadline Exceeded
Created by: webary
多分类(68类)简单全连接分类任务训练一轮后任务失败,失败提示信息: F1222 19:03:46.287559 19021 grpc_client.cc:414] SendRPC name:[softmax_b@GRAD.trainer_0], ep:[10.90.145.38:62004], status:[-1] meets grpc error, error_code:4 error_message:Deadline Exceeded error_details:
* Check failure stack trace: *
@ 0x7fca1817efbd google::LogMessage::Fail() @ 0x7fca18182a6c google::LogMessage::SendToLog() @ 0x7fca1817eae3 google::LogMessage::Flush() @ 0x7fca18183f7e google::LogMessageFatal::~LogMessageFatal() @ 0x7fca18ee14fe paddle::operators::distributed::GRPCClient::Proceed() @ 0x7fca255b38a0 execute_native_thread_routine @ 0x7fca988dd1c3 start_thread @ 0x7fca97f0512d __clone @ (nil) (unknown)具体是什么原因导致的错误?应该如何修正来让任务正常训练下去?