In our current operator registration mechanism, for each operator, the programmer should register a *gradient operator creator* function, which takes a C++ operator instance, and returns the corresponding gradient instance.
Currently, for each C++ operator class definition, there registers a *gradient operator creator* function, which takes a C++ operator instance and returns the corresponding gradient operator instance.
However, as we decided to separate the *compilation* and *execution* of DL models, we need to reshape the creator to take a protobuf `OpDesc` message, and returns a corresponding message.
However, we noticed two problems with the current deisgn:
More than that, the new registration mechanism need to support the fact that an operators' gradient computation might be a composition of operators.
1. As we decided to separate the *compilation* and *execution* phases, we need to change the creator to take an `OpDesc` protobuf message in a `ProgramDesc` and inserts corresponding `OpDesc` messages into the `ProgramDesc` message.
## Current Implementation
1. Some operator's gradient computation requires more than one gradient operators. For example, the gradient of *minus* consists of two operators -- an identity operaotr and a scale operator. So we need to make the registration mechanism to support the mapping from an operator to a set of operators for gradient computation.
OpInfos store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is
## The Current Implementation
The C++ class `OpInfos` store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is
<spanid="design-doc-gradient-operators-registration"></span><h1>Design Doc: Gradient Operators Registration<aclass="headerlink"href="#design-doc-gradient-operators-registration"title="Permalink to this headline">¶</a></h1>
<divclass="section"id="the-problem-posed">
<spanid="the-problem-posed"></span><h2>The Problem Posed<aclass="headerlink"href="#the-problem-posed"title="Permalink to this headline">¶</a></h2>
<p>In our current operator registration mechanism, for each operator, the programmer should register a <em>gradient operator creator</em> function, which takes a C++ operator instance, and returns the corresponding gradient instance.</p>
<p>However, as we decided to separate the <em>compilation</em> and <em>execution</em> of DL models, we need to reshape the creator to take a protobuf <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> message, and returns a corresponding message.</p>
<p>More than that, the new registration mechanism need to support the fact that an operators’ gradient computation might be a composition of operators.</p>
<p>Currently, for each C++ operator class definition, there registers a <em>gradient operator creator</em> function, which takes a C++ operator instance and returns the corresponding gradient operator instance.</p>
<p>However, we noticed two problems with the current deisgn:</p>
<olclass="simple">
<li>As we decided to separate the <em>compilation</em> and <em>execution</em> phases, we need to change the creator to take an <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> protobuf message in a <codeclass="docutils literal"><spanclass="pre">ProgramDesc</span></code> and inserts corresponding <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> messages into the <codeclass="docutils literal"><spanclass="pre">ProgramDesc</span></code> message.</li>
<li>Some operator’s gradient computation requires more than one gradient operators. For example, the gradient of <em>minus</em> consists of two operators – an identity operaotr and a scale operator. So we need to make the registration mechanism to support the mapping from an operator to a set of operators for gradient computation.</li>
</ol>
</div>
<divclass="section"id="current-implementation">
<spanid="current-implementation"></span><h2>Current Implementation<aclass="headerlink"href="#current-implementation"title="Permalink to this headline">¶</a></h2>
<p>OpInfos store in a association map which key is the operator type. The <codeclass="docutils literal"><spanclass="pre">grad_op_type</span></code> indicate associated gradient operator type. Operator can create gradient operator by <codeclass="docutils literal"><spanclass="pre">OpInfo::creator_</span></code> of gradient. The pseudo code is</p>
<spanid="the-current-implementation"></span><h2>The Current Implementation<aclass="headerlink"href="#the-current-implementation"title="Permalink to this headline">¶</a></h2>
<p>The C++ class <codeclass="docutils literal"><spanclass="pre">OpInfos</span></code> store in a association map which key is the operator type. The <codeclass="docutils literal"><spanclass="pre">grad_op_type</span></code> indicate associated gradient operator type. Operator can create gradient operator by <codeclass="docutils literal"><spanclass="pre">OpInfo::creator_</span></code> of gradient. The pseudo code is</p>
In our current operator registration mechanism, for each operator, the programmer should register a *gradient operator creator* function, which takes a C++ operator instance, and returns the corresponding gradient instance.
Currently, for each C++ operator class definition, there registers a *gradient operator creator* function, which takes a C++ operator instance and returns the corresponding gradient operator instance.
However, as we decided to separate the *compilation* and *execution* of DL models, we need to reshape the creator to take a protobuf `OpDesc` message, and returns a corresponding message.
However, we noticed two problems with the current deisgn:
More than that, the new registration mechanism need to support the fact that an operators' gradient computation might be a composition of operators.
1. As we decided to separate the *compilation* and *execution* phases, we need to change the creator to take an `OpDesc` protobuf message in a `ProgramDesc` and inserts corresponding `OpDesc` messages into the `ProgramDesc` message.
## Current Implementation
1. Some operator's gradient computation requires more than one gradient operators. For example, the gradient of *minus* consists of two operators -- an identity operaotr and a scale operator. So we need to make the registration mechanism to support the mapping from an operator to a set of operators for gradient computation.
OpInfos store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is
## The Current Implementation
The C++ class `OpInfos` store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is
<spanid="the-problem-posed"></span><h2>The Problem Posed<aclass="headerlink"href="#the-problem-posed"title="永久链接至标题">¶</a></h2>
<p>In our current operator registration mechanism, for each operator, the programmer should register a <em>gradient operator creator</em> function, which takes a C++ operator instance, and returns the corresponding gradient instance.</p>
<p>However, as we decided to separate the <em>compilation</em> and <em>execution</em> of DL models, we need to reshape the creator to take a protobuf <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> message, and returns a corresponding message.</p>
<p>More than that, the new registration mechanism need to support the fact that an operators’ gradient computation might be a composition of operators.</p>
<p>Currently, for each C++ operator class definition, there registers a <em>gradient operator creator</em> function, which takes a C++ operator instance and returns the corresponding gradient operator instance.</p>
<p>However, we noticed two problems with the current deisgn:</p>
<olclass="simple">
<li>As we decided to separate the <em>compilation</em> and <em>execution</em> phases, we need to change the creator to take an <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> protobuf message in a <codeclass="docutils literal"><spanclass="pre">ProgramDesc</span></code> and inserts corresponding <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> messages into the <codeclass="docutils literal"><spanclass="pre">ProgramDesc</span></code> message.</li>
<li>Some operator’s gradient computation requires more than one gradient operators. For example, the gradient of <em>minus</em> consists of two operators – an identity operaotr and a scale operator. So we need to make the registration mechanism to support the mapping from an operator to a set of operators for gradient computation.</li>
<p>OpInfos store in a association map which key is the operator type. The <codeclass="docutils literal"><spanclass="pre">grad_op_type</span></code> indicate associated gradient operator type. Operator can create gradient operator by <codeclass="docutils literal"><spanclass="pre">OpInfo::creator_</span></code> of gradient. The pseudo code is</p>
<spanid="the-current-implementation"></span><h2>The Current Implementation<aclass="headerlink"href="#the-current-implementation"title="永久链接至标题">¶</a></h2>
<p>The C++ class <codeclass="docutils literal"><spanclass="pre">OpInfos</span></code> store in a association map which key is the operator type. The <codeclass="docutils literal"><spanclass="pre">grad_op_type</span></code> indicate associated gradient operator type. Operator can create gradient operator by <codeclass="docutils literal"><spanclass="pre">OpInfo::creator_</span></code> of gradient. The pseudo code is</p>
