第一轮训练快结束时,时延变大
Created by: lexy0093
组网:
label = fluid.layers.data(name='label', shape=[1], dtype='int64', lod_level=0)
index = fluid.layers.data(name='index', shape=[1], dtype='int64', lod_level=1)
feature = fluid.layers.data(name='feature', shape=[1], dtype='float32', lod_level=1)
droped = fluid.layers.dropout(x=feature, dropout_prob=self.dropout_prob)
emb = fluid.layers.embedding(input=index, size=[self.vocab_dim, self.emb_dim], is_sparse=True)
stk = fluid.layers.expand(feature, expand_times = [1, 256])
mm = fluid.layers.elementwise_mul(x=emb, y=stk)
mm_fc = fluid.layers.sequence_pool(input=mm, pool_type='sum')
fc1 = fluid.layers.relu(x=mm_fc)
fc2 = fluid.layers.fc(input=fc1, size=self.hid_dim, act='relu')
pred = fluid.layers.fc(input=[fc1, fc2], size=self.class_dim, act='softmax')
特征维度:80w, batch_size=1024,每轮训练包括:train data 和test data,训练开始每个batch执行大概稳定25s,快结束时batch运行时延增加,并且在test结束后,第二轮batch时延也加大,严重影响运行效率。 日志: http://10.73.173.35:8910/fileview.html?type=logsdir&path=/&instance=0.app-user-20190716103040-38241--mcdnn-fluid_paddlecloud