paddle.fluid.core_avx.EnforceNotMet: Invoke operator mul_grad error
Created by: lxk1990727
多卡训练过程中报错paddle.fluid.core_avx.EnforceNotMet: Invoke operator mul_grad error
Traceback (most recent call last):
328 File "train.py", line 165, in <module>
329 train(args)
330 File "train.py", line 134, in train
331 infer_outs = exe.run(compiler_prog, fetch_list=fetch_list)
332 File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 666, in run
333 return_numpy=return_numpy)
334 File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 528, in _run_p arallel
335 exe.run(fetch_var_names, fetch_var_name)
336 paddle.fluid.core_avx.EnforceNotMet: Invoke operator mul_grad error.
337 Python Callstacks:
338 File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1771, in appe nd_op
339 attrs=kwargs.get("attrs", None))
340 File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in app end_op
341 return self.main_program.current_block().append_op(*args, **kwargs)
342 File "/home/work/lixiaokang04/tools/paddle_release_home/python/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 334, in fc
343 "y_num_col_dims": 1})
344 File "/home/work/lixiaokang04/data/ernie/vvt_ernie_embs/models/video_text/tsn_res_model.py", line 158, in net
345 size=output_dim, bias_attr=False)
346 File "/home/work/lixiaokang04/data/ernie/vvt_ernie_embs/models/video_text/video_text.py", line 157, in build_model
347 self.video_emb_neg = videomodel.net(input = self.feature_input[7], output_dim=cfg['tsn_output_size'])
348 File "train.py", line 99, in train
349 train_model.build_model()
350 File "train.py", line 165, in <module>
351 train(args)
352 C++ Callstacks:
353 The places of matrices must be same at [/paddle/paddle/fluid/operators/math/blas_impl.h:392]
354 PaddlePaddle Call Stacks:
355 0 0x7f953cc9aad0p void paddle::platform::EnforceNotMet::Init<char const*>(char const*, char const*, int) + 352
356 1 0x7f953cc9ae49p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 137
357 2 0x7f953d42a46cp void paddle::operators::math::Blas<paddle::platform::CUDADeviceContext>::MatMul<float>(paddle::framework::Tensor co nst&, bool, paddle::framework::Tensor const&, bool, float, paddle::framework::Tensor*, float) const + 412