se-resnet模型训练imagent数据,2ps、4trainer,训练完1个pass后train连不上ps
Created by: kolinwei
ps实际上还在运行。 Pass 0,testbatch 3100,loss 3.85814905167, acc1 0.1875,acc5 0.4375,time 0.10 sec Pass 0,testbatch 3110,loss 4.93263053894, acc1 0.0,acc5 0.25,time 0.11 sec Pass 0,testbatch 3120,loss 4.12771701813, acc1 0.125,acc5 0.5,time 0.11 sec End pass 0, train_loss 5.50010538101, train_acc1 0.0622033625841, train_acc5 0.166548013687, test_loss 4.27425519508, test_acc1 0.16096, test_acc5 0.36604 Pass 1, trainbatch 0, loss 4.62829589844, acc1 0.1640625, acc5 0.31640625 time 2.08 sec E0615 07:36:18.786679 3867 grpc_client.cc:248] var: name:[fc_18.w_0.block1] ep:[192.168.16.26:5002] grpc error:Deadline Exceeded E0615 07:36:18.787700 3867 grpc_client.cc:248] var: name:[fc_15.b_0] ep:[192.168.16.26:5002] grpc error:Deadline Exceeded E0615 07:36:18.787750 3867 grpc_client.cc:248] var: name:[fc_6.b_0] ep:[192.168.16.26:5002] grpc error:Deadline Exceeded E0615 07:36:18.787788 3867 grpc_client.cc:248] var: name:[fc_30.b_0] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.787829 3867 grpc_client.cc:248] var: name:[fc_9.w_0.block1] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.787865 3867 grpc_client.cc:248] var: name:[conv2d_24.w_0.block0] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.787900 3867 grpc_client.cc:248] var: name:[fc_27.w_0.block1] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.787935 3867 grpc_client.cc:248] var: name:[batch_norm_13.w_0] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.787969 3867 grpc_client.cc:248] var: name:[fc_18.w_0.block0] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.788002 3867 grpc_client.cc:248] var: name:[fc_19.b_0] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.788034 3867 grpc_client.cc:248] var: name:[conv2d_26.w_0.block1] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded E0615 07:36:18.788066 3867 grpc_client.cc:248] var: name:[conv2d_50.w_0.block1] ep:[192.168.16.24:5002] grpc error:Deadline Exceeded