From fff380b54e1d8da774e0c183ad206ffbac5f9b4b Mon Sep 17 00:00:00 2001
From: Yi Wang <yiwang01@baidu.com>
Date: Sat, 2 Sep 2017 17:03:45 -0700
Subject: [PATCH] Add layers and operators design doc

---
 doc/design/layers_operators.md | 71 ++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 doc/design/layers_operators.md

diff --git a/doc/design/layers_operators.md b/doc/design/layers_operators.md
new file mode 100644
index 000000000..4cb590c85
--- /dev/null
+++ b/doc/design/layers_operators.md
@@ -0,0 +1,71 @@
+In a DL system, we can compose one or more fine grained operators into a coarse grained one.  For example, the FC layer can be composed of a multiplication operator and an add operator.
+
+Historically, some fine grained operations are known as operators, and some coarse level ones are known as layers.  But we need a well-defined separation.
+
+In general, operators are those very fine grained operations, e.g., mul and add. In the implementation, we can write them as C++ functions:
+
+```c++
+template <typename T> T add(T x, T y) { return x + y; }
+template <typename T> T mul(T x, T y) { return x * y; }
+```
+
+Then we can wrap them into operators which are C++ classes and can be created from Python bindings by name.  A C macro can do this. For example, the following macro invocation
+
+```c++
+#define MAKE_FUNCTION_OPERATOR(mul);
+```
+
+generates
+
+```c++
+class mulOp : public OperatorBase {...};
+REGISTER_OP(mulOp, "mul");
+```
+
+so that in Python we can create operator mul by:
+
+```python
+X1 = Var()
+X2 = Var()
+Y = Var()
+paddle.cpp.create_operator("mul", input=[X1, X2], output=Y)
+```
+
+Also, at the same time, we can compose a coarse level C++ operator class by composing functions `mul` and `add`:
+
+```c++
+class FCOp : public OperatorBase {
+ public:
+  void Run(...) {
+    add(mul(Input("X"), Input("W")), Input("b");
+  }
+};
+REGISTER_OP(FCOp, "fc");
+```
+
+We need to support such composition in Python as well.  To do so, we need a higher level Python wrapping of operator creation than `paddle.cpp.create_operator`.  This higher level operator API should be compatible with the layer API.
+
+Let's explain using an example.  Suppose that we are going to compose the FC using mul and add in Python, we'd like to have Python functions `mul` and `add` defined in module `operator`:
+
+```python
+def mul(X1, X2):
+    O = Var
+    paddle.cpp.create_operator("mul", input={X1, Y1], output=O)
+    return O
+
+def add(X1, X2):
+    O = Var
+    paddle.cpp.create_operator("add", input={X1, X2], output=O)
+    return O
+```
+
+so that we can define
+
+```python
+def layer.fc(X):
+    W = Var()
+    b = Var()
+    return operator.add(operator.mul(X, W), b)
+```
+
+We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`.  This is how we differentiate layer and operators in PaddlePaddle.
-- 
GitLab