Problems in current backward implementation
Created by: JiayiFeng
Current backward implementation is faced with several problems.
-
The design of current backward didn't think much about nested blocks. To make it working on
WhileOp
andIfElesOp
, we have to add many tricks into it, which makes our code extremely hard to read. -
Incomplete implementation of backward pruning. We are simply using a
set
to record names of variables which do not have gradients. This strategy can not disgust variables in different blocks with the same name and causes bugs in some special cases. -
In some cases, such as error clipping, we need to insert some other operators(
ClipOp
) between all backward operators. It's not easy to implement these functions in current C++ code.