Created by: Xreki

resolve #3926 (closed)

• fix SegFault during checking gradient
  • In python, when we define an Operator, all the inputs and outputs, including those AsIntermediate outputs, should be explicitly specified. In TestFCGradOp, operator should be defined as:

     op = Operator(
                   "fc",
                   X="X",
                   W="W",
                   b="b",
                   Out="Out",
                   mul_out="mul_out",
                   add_out="add_out",
                   activation="sigmoid")

• support multiple inputs and weights

  • Do not try to call operators with inplace arguments, such as Y = X + Y. In my first implementation, I construct FCOp as a series of operators as follows:

    1. mul_out = X[0] * W[0]
    2. add_out = X[1] * W[1]
    3. mul_out = mul_out + add_out
    4. add_out = add_out + b
    5. Y = Act(add_out)

  • The unittest failed when constructed an add operator during the backward process. I found two add operators were automatically created and the DebugString are:

Op(add), inputs:{X[mul_out@GRAD@RENAME@1@0, mul_out@GRAD@RENAME@1@1], outputs:{Out[mul_out@GRAD]}}
Op(add), inputs:{X[add_out@GRAD@RENAME@1@0, add_out@GRAD@RENAME@1@1], outputs:{Out[add_out@GRAD]}}

Thus I changed the implementation of FCOp to: 1. mul_out[0] = X[0] * W[0] 2. mul_out[1] = X[1] * W[1] 3. sum_out = mul_out[0] + mul_out[1] 4. add_out = sum_out + b 5. Y = Act(add_out)

• support input tensor of multiple dimensions
  • The attributes of type std::vector cannot be set default value, see #4077 (closed) - solved.