functions_operators_layers.md 3.5 KB
Newer Older
Y
Yi Wang 已提交
1
# Design Doc: Functions, Operators, and Layers
Y
Yi Wang 已提交
2

Y
Yi Wang 已提交
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
In a DL system, we can compose one or more fine grained operators into a coarse grained one.  For example, the FC layer can be composed of a multiplication operator and an add operator.

Historically, some fine grained operations are known as operators, and some coarse level ones are known as layers.  But we need a well-defined separation.

In general, operators are those very fine grained operations, e.g., mul and add. In the implementation, we can write them as C++ functions:

```c++
template <typename T> T add(T x, T y) { return x + y; }
template <typename T> T mul(T x, T y) { return x * y; }
```

Then we can wrap them into operators which are C++ classes and can be created from Python bindings by name.  A C macro can do this. For example, the following macro invocation

```c++
#define MAKE_FUNCTION_OPERATOR(mul);
```

generates

```c++
Y
Yi Wang 已提交
23 24
template <typename T> class mulOp : public OperatorBase {...};
REGISTER_OP(mulOp<float32>, "mul");
Y
Yi Wang 已提交
25 26 27 28 29 30 31 32 33 34 35 36 37 38
```

so that in Python we can create operator mul by:

```python
X1 = Var()
X2 = Var()
Y = Var()
paddle.cpp.create_operator("mul", input=[X1, X2], output=Y)
```

Also, at the same time, we can compose a coarse level C++ operator class by composing functions `mul` and `add`:

```c++
Y
Yi Wang 已提交
39
template <typename T>
Y
Yi Wang 已提交
40 41 42
class FCOp : public OperatorBase {
 public:
  void Run(...) {
Y
Yi Wang 已提交
43
    add(mul(Input<T>("X"), Input<T>("W")), Input<T>("b");
Y
Yi Wang 已提交
44
  }
Y
Yi Wang 已提交
45
};
Y
Yi Wang 已提交
46 47 48 49 50 51 52 53
REGISTER_OP(FCOp, "fc");
```

We need to support such composition in Python as well.  To do so, we need a higher level Python wrapping of operator creation than `paddle.cpp.create_operator`.  This higher level operator API should be compatible with the layer API.

Let's explain using an example.  Suppose that we are going to compose the FC using mul and add in Python, we'd like to have Python functions `mul` and `add` defined in module `operator`:

```python
Y
Yi Wang 已提交
54
def operator.mul(X1, X2):
Y
Yi Wang 已提交
55
    O = Var()
Y
Yi Wang 已提交
56 57 58
    paddle.cpp.create_operator("mul", input={X1, Y1], output=O)
    return O

Y
Yi Wang 已提交
59
def operator.add(X1, X2):
Y
Yi Wang 已提交
60
    O = Var()
Y
Yi Wang 已提交
61 62 63 64
    paddle.cpp.create_operator("add", input={X1, X2], output=O)
    return O
```

Y
Yi Wang 已提交
65
Above code snippets are automatically generated.  Given them, users can define
Y
Yi Wang 已提交
66 67 68 69 70 71 72 73

```python
def layer.fc(X):
    W = Var()
    b = Var()
    return operator.add(operator.mul(X, W), b)
```

Y
Yi Wang 已提交
74 75 76 77 78 79 80 81 82 83 84 85 86
If we don't have `operator.mul` and `operator.add`, the definiton of `layer.fc` would be complicated:

```python
def layer.fc(X):
    W = Var()
    b = Var()
    O1 = Var()
    paddle.cpp.create_operator("mul", input=[X, W], output=O1)
    O2 = Var()
    paddle.cpp.create_operator("add", input=[O1, b], output=O2)
    return O2
```

Y
Yi Wang 已提交
87 88
We'd like to have Python bindings to operators in package `paddle.operator`, and Python compositions of operators in package `paddle.layer`.  So we have the following concepts in above illustrative example:

Y
Yi Wang 已提交
89

Y
Yi Wang 已提交
90
| C++ functions/functors | mul          | add          |             |          |
Y
Yi Wang 已提交
91
|------------------------|--------------|--------------|-------------|----------|
Y
Yi Wang 已提交
92 93 94
| C++ operator class     | mulOp        | addOp        | FCOp        |          |
| Python binding         | operator.mul | operator.add | operator.fc |          |
| Python function        |              |              |             | layer.fc |
Y
Yi Wang 已提交
95

Y
Yi Wang 已提交
96 97 98 99 100

This is how we differentiate layer and operators in PaddlePaddle:

- those defined in C++ and have a lightweighted Python wrapper in module `operators` are operators; whereas
- those who don't have C++ implementations but a Python implementation that compose C++ operators are known as layers.