Created by: lxk1990727

多卡训练过程中报错paddle.fluid.core_avx.EnforceNotMet: Invoke operator mul_grad error

Traceback (most recent call last):
328   File "train.py", line 165, in <module>
329     train(args)
330   File "train.py", line 134, in train
331     infer_outs = exe.run(compiler_prog, fetch_list=fetch_list)
332   File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 666, in run
333     return_numpy=return_numpy)
334   File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 528, in _run_p    arallel
335     exe.run(fetch_var_names, fetch_var_name)
336 paddle.fluid.core_avx.EnforceNotMet: Invoke operator mul_grad error.
337 Python Callstacks:
338   File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1771, in appe    nd_op
339     attrs=kwargs.get("attrs", None))
340   File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in app    end_op
341     return self.main_program.current_block().append_op(*args, **kwargs)
342   File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 334, in fc
343     "y_num_col_dims": 1})
344   File "/home/work/lixiaokang04/data/ernie/vvt_ernie_embs/models/video_text/tsn_res_model.py", line 158, in net
345     size=output_dim, bias_attr=False)
346   File "/home/work/lixiaokang04/data/ernie/vvt_ernie_embs/models/video_text/video_text.py", line 157, in build_model
347     self.video_emb_neg = videomodel.net(input = self.feature_input[7], output_dim=cfg['tsn_output_size'])
348   File "train.py", line 99, in train
349     train_model.build_model()
350   File "train.py", line 165, in <module>
351     train(args)
352 C++ Callstacks:
353 The places of matrices must be same at [/paddle/paddle/fluid/operators/math/blas_impl.h:392]
354 PaddlePaddle Call Stacks:
355 0       0x7f953cc9aad0p void paddle::platform::EnforceNotMet::Init<char const*>(char const*, char const*, int) + 352
356 1       0x7f953cc9ae49p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 137
357 2       0x7f953d42a46cp void paddle::operators::math::Blas<paddle::platform::CUDADeviceContext>::MatMul<float>(paddle::framework::Tensor co    nst&, bool, paddle::framework::Tensor const&, bool, float, paddle::framework::Tensor*, float) const + 412