ValueError: Variable vemb@GRAD.trainer_0 has been created before. The previous type is VarType.SELECTED_ROWS; the new type is VarType.LOD_TENSOR. They are not matched
Created by: xymyeah
基于最新的代码commit 2a3b9ff1 (HEAD -> develop, origin/develop, origin/HEAD) 出现了如下异常
Traceback (most recent call last):
File "trainer.py", line 207, in
main()
File "trainer.py", line 154, in main
trainers=int(k8s_trainers_count))
File "/usr/local/lib/python2.7/site-packages/paddle/fluid/distribute_transpiler.py", line 200, in transpile
grad_var_mapping = self._append_split_op(program, grad_blocks)
File "/usr/local/lib/python2.7/site-packages/paddle/fluid/distribute_transpiler.py", line 460, in _append_split_op
program, gradblocks, add_trainer_suffix=True)
File "/usr/local/lib/python2.7/site-packages/paddle/fluid/distribute_transpiler.py", line 410, in _create_vars_from_blocklist
program.global_block().rename_var(varname, new_var_name)
File "/usr/local/lib/python2.7/site-packages/paddle/fluid/framework.py", line 789, in rename_var
stop_gradient=stop_gradient)
File "/usr/local/lib/python2.7/site-packages/paddle/fluid/framework.py", line 179, in init
self.desc.type(), type))
ValueError: Variable vemb@GRAD.trainer_0 has been created before. The previous type is VarType.SELECTED_ROWS; the new type is VarType.LOD_TENSOR. They are not matched