In our current operator registration mechanism, for each operator, the programmer should register a *gradient operator creator* function, which takes a C++ operator instance, and returns the corresponding gradient instance.
However, as we decided to separate the *compilation* and *execution* of DL models, we need to reshape the creator to take a protobuf `OpDesc` message, and returns a corresponding message.
More than that, the new registration mechanism need to support the fact that an operators' gradient computation might be a composition of operators.
## Current Implementation
OpInfos store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is
The mapping relationship between an operator and its gradient operators is a function. The interface of that function is:
```cpp
// (OpDesc) --> vector<OpDesc>
using GradOpDescMaker = std::function<std::vector<OpDesc>(const OpDesc&)>;
```
The function take a `OpDesc` of the forward operator and return one or many gradient operator descriptions.
The `GradOpDescMaker` will be registered in `OpInfo`, to replace `grad_op_type_` field. The `OpInfo` should be
```cpp
struct OpInfo {
GradOpDescMaker grad_op_maker_;
...
};
```
The `grad_op_maker_ ` is `nullptr` if the operator does not have associated gradient operators.
We should chagne register macros at the same time. In the current solution, there is no difference between forwarding operators and backward operators. So `REGISTER_OP` just register one operator. If the `REGISTER_OPERATOR ` contains `OpProtoAndCheckerMaker` and `GradOpDescMaker`, we just list them in the same macro. It can be done by a macro contains `__VA_ARGS__`.
// Developers can still manually implement gradient operator.
REGISTER_OPERATOR(minus_grad, MinusGradOp);
```
The interface of current `REGISTER_OP` macro could not be changed. In `REGISTER_OP`, it will invoke `REGISTER_OPERATOR` two times and generate GradOpDescMaker inside.
<liclass="toctree-l2"><aclass="reference internal"href="../getstarted/build_and_install/index_en.html">Install and Build</a><ul>
<liclass="toctree-l3"><aclass="reference internal"href="../getstarted/build_and_install/docker_install_en.html">PaddlePaddle in Docker Containers</a></li>
<liclass="toctree-l3"><aclass="reference internal"href="../getstarted/build_and_install/build_from_source_en.html">Installing from Sources</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/usage/k8s/k8s_en.html">Paddle On Kubernetes</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/usage/k8s/k8s_aws_en.html">Distributed PaddlePaddle Training on AWS with Kubernetes</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/dev/build_en.html">Build PaddlePaddle from Source Code and Run Unit Test</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/dev/new_layer_en.html">Write New Layers</a></li>
<spanid="design-doc-gradient-operators-registration"></span><h1>Design Doc: Gradient Operators Registration<aclass="headerlink"href="#design-doc-gradient-operators-registration"title="Permalink to this headline">¶</a></h1>
<divclass="section"id="the-problem-posed">
<spanid="the-problem-posed"></span><h2>The Problem Posed<aclass="headerlink"href="#the-problem-posed"title="Permalink to this headline">¶</a></h2>
<p>In our current operator registration mechanism, for each operator, the programmer should register a <em>gradient operator creator</em> function, which takes a C++ operator instance, and returns the corresponding gradient instance.</p>
<p>However, as we decided to separate the <em>compilation</em> and <em>execution</em> of DL models, we need to reshape the creator to take a protobuf <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> message, and returns a corresponding message.</p>
<p>More than that, the new registration mechanism need to support the fact that an operators’ gradient computation might be a composition of operators.</p>
</div>
<divclass="section"id="current-implementation">
<spanid="current-implementation"></span><h2>Current Implementation<aclass="headerlink"href="#current-implementation"title="Permalink to this headline">¶</a></h2>
<p>OpInfos store in a association map which key is the operator type. The <codeclass="docutils literal"><spanclass="pre">grad_op_type</span></code> indicate associated gradient operator type. Operator can create gradient operator by <codeclass="docutils literal"><spanclass="pre">OpInfo::creator_</span></code> of gradient. The pseudo code is</p>
<p>The function take a <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> of the forward operator and return one or many gradient operator descriptions.</p>
<p>The <codeclass="docutils literal"><spanclass="pre">GradOpDescMaker</span></code> will be registered in <codeclass="docutils literal"><spanclass="pre">OpInfo</span></code>, to replace <codeclass="docutils literal"><spanclass="pre">grad_op_type_</span></code> field. The <codeclass="docutils literal"><spanclass="pre">OpInfo</span></code> should be</p>
<p>The <codeclass="docutils literal"><spanclass="pre">grad_op_maker_</span></code> is <codeclass="docutils literal"><spanclass="pre">nullptr</span></code> if the operator does not have associated gradient operators.</p>
<p>We should chagne register macros at the same time. In the current solution, there is no difference between forwarding operators and backward operators. So <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code> just register one operator. If the <codeclass="docutils literal"><spanclass="pre">REGISTER_OPERATOR</span></code> contains <codeclass="docutils literal"><spanclass="pre">OpProtoAndCheckerMaker</span></code> and <codeclass="docutils literal"><spanclass="pre">GradOpDescMaker</span></code>, we just list them in the same macro. It can be done by a macro contains <codeclass="docutils literal"><spanclass="pre">__VA_ARGS__</span></code>.</p>
<p>The interface of current <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code> macro could not be changed. In <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code>, it will invoke <codeclass="docutils literal"><spanclass="pre">REGISTER_OPERATOR</span></code> two times and generate GradOpDescMaker inside.</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
In our current operator registration mechanism, for each operator, the programmer should register a *gradient operator creator* function, which takes a C++ operator instance, and returns the corresponding gradient instance.
However, as we decided to separate the *compilation* and *execution* of DL models, we need to reshape the creator to take a protobuf `OpDesc` message, and returns a corresponding message.
More than that, the new registration mechanism need to support the fact that an operators' gradient computation might be a composition of operators.
## Current Implementation
OpInfos store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is
The mapping relationship between an operator and its gradient operators is a function. The interface of that function is:
```cpp
// (OpDesc) --> vector<OpDesc>
using GradOpDescMaker = std::function<std::vector<OpDesc>(const OpDesc&)>;
```
The function take a `OpDesc` of the forward operator and return one or many gradient operator descriptions.
The `GradOpDescMaker` will be registered in `OpInfo`, to replace `grad_op_type_` field. The `OpInfo` should be
```cpp
struct OpInfo {
GradOpDescMaker grad_op_maker_;
...
};
```
The `grad_op_maker_ ` is `nullptr` if the operator does not have associated gradient operators.
We should chagne register macros at the same time. In the current solution, there is no difference between forwarding operators and backward operators. So `REGISTER_OP` just register one operator. If the `REGISTER_OPERATOR ` contains `OpProtoAndCheckerMaker` and `GradOpDescMaker`, we just list them in the same macro. It can be done by a macro contains `__VA_ARGS__`.
// Developers can still manually implement gradient operator.
REGISTER_OPERATOR(minus_grad, MinusGradOp);
```
The interface of current `REGISTER_OP` macro could not be changed. In `REGISTER_OP`, it will invoke `REGISTER_OPERATOR` two times and generate GradOpDescMaker inside.
<spanid="the-problem-posed"></span><h2>The Problem Posed<aclass="headerlink"href="#the-problem-posed"title="永久链接至标题">¶</a></h2>
<p>In our current operator registration mechanism, for each operator, the programmer should register a <em>gradient operator creator</em> function, which takes a C++ operator instance, and returns the corresponding gradient instance.</p>
<p>However, as we decided to separate the <em>compilation</em> and <em>execution</em> of DL models, we need to reshape the creator to take a protobuf <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> message, and returns a corresponding message.</p>
<p>More than that, the new registration mechanism need to support the fact that an operators’ gradient computation might be a composition of operators.</p>
<p>OpInfos store in a association map which key is the operator type. The <codeclass="docutils literal"><spanclass="pre">grad_op_type</span></code> indicate associated gradient operator type. Operator can create gradient operator by <codeclass="docutils literal"><spanclass="pre">OpInfo::creator_</span></code> of gradient. The pseudo code is</p>
<p>The function take a <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> of the forward operator and return one or many gradient operator descriptions.</p>
<p>The <codeclass="docutils literal"><spanclass="pre">GradOpDescMaker</span></code> will be registered in <codeclass="docutils literal"><spanclass="pre">OpInfo</span></code>, to replace <codeclass="docutils literal"><spanclass="pre">grad_op_type_</span></code> field. The <codeclass="docutils literal"><spanclass="pre">OpInfo</span></code> should be</p>
<p>The <codeclass="docutils literal"><spanclass="pre">grad_op_maker_</span></code> is <codeclass="docutils literal"><spanclass="pre">nullptr</span></code> if the operator does not have associated gradient operators.</p>
<p>We should chagne register macros at the same time. In the current solution, there is no difference between forwarding operators and backward operators. So <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code> just register one operator. If the <codeclass="docutils literal"><spanclass="pre">REGISTER_OPERATOR</span></code> contains <codeclass="docutils literal"><spanclass="pre">OpProtoAndCheckerMaker</span></code> and <codeclass="docutils literal"><spanclass="pre">GradOpDescMaker</span></code>, we just list them in the same macro. It can be done by a macro contains <codeclass="docutils literal"><spanclass="pre">__VA_ARGS__</span></code>.</p>
<p>The interface of current <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code> macro could not be changed. In <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code>, it will invoke <codeclass="docutils literal"><spanclass="pre">REGISTER_OPERATOR</span></code> two times and generate GradOpDescMaker inside.</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
