Created by: lcy-seso

We are going to support popular NMT models on Fluid, including but not limited to RNN search, ConvS2S, and Transformer.

I think the first important thing for us is to understand and figure out the problems.

We choose Google' Transformer as our starting point. Here I list some questions should be answered:

A tensor2tensor implementation: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py

• About the Transformer:

  • motivation: What is the problem proposed to solve.
  • how the problem is solved.
  • model architecture:
    • Are there any challenges or functional requirements for a DL framework to implement the transformer architecture?
    • Are there any tricks or key points require our framework to support a function we have not considered yet?
    • Are there any operators we have not implemented yet?

• About Fluid (This part is relatively open currently)

  • How encoder-decoder architecture is implemented in Fluid, run it and follow its execution/codes, do you have any question and problem about it (it is ok just from the perspective of a user)
  • RNN and dynamic RNN. The transformer does not need any recurrent layer and CNN layer. But how Fluid process sequence is still crucial.
  • propose your questions about Fluid, we will collect them.

At the end of this step, we will share our notes with everyone, both about Transformer and Fluid, we can try to make it part of the document. all of us should:

1. have an overview picture of the architecture of Transformer; (from top to bottom, not the reverse.)
2. carry out a clear plan about how we are going to implement Transformer:
   • any special requirements for Fluid?
   • what mechanisms of fluid are we going to make use of? for example: while loop, dynamic RNN, or any other things?
3. a checklist of operators needed: The list can be directly changed to an action list.
   • what do we already have?
   • what has not been implemented yet?

Related issue: https://github.com/PaddlePaddle/Paddle/issues/6821