- Operator forward computing is easy to check if the result is right because it has a clear definition. **But** backpropagation is a notoriously difficult algorithm to debug and get right:
- 1. you should get the right backpropagation formula according to the forward computation.
- 2. you should implement it right in CPP.
- 3. it's difficult to prepare test data.
- Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:
1. you should get the right backpropagation formula according to the forward computation.
2. you should implement it right in CPP.
3. it's difficult to prepare test data.
- Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
- 1. numeric gradient checker only need forward operator.
- 2. user only need to prepare the input data for forward Operator.
- Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
1. numerical gradient checker only need forward operator.
2. user only need to prepare the input data for forward Operator.
## Mathematical Theory
The following two document from stanford has a detailed explanation of how to get numeric gradient and why it's useful.
The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it's useful.
- [Gradient checking and advanced optimization(en)](http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization)
- [Gradient checking and advanced optimization(cn)](http://ufldl.stanford.edu/wiki/index.php/%E6%A2%AF%E5%BA%A6%E6%A3%80%E9%AA%8C%E4%B8%8E%E9%AB%98%E7%BA%A7%E4%BC%98%E5%8C%96)
...
...
@@ -20,7 +20,7 @@ The following two document from stanford has a detailed explanation of how to ge
## Numeric Gradient Implementation
### Python Interface
```python
def get_numeric_gradient(op,
def get_numerical_gradient(op,
input_values,
output_name,
input_to_check,
...
...
@@ -30,13 +30,13 @@ def get_numeric_gradient(op,
Get Numeric Gradient for an operator's input.
:param op: C++ operator instance, could be an network
:param input_values: The input variables. Should be an dictionary, key is
variable name. Value is numpy array.
:param input_values: The input variables. Should be an dictionary, whose key is
variable name, and value is numpy array.
:param output_name: The final output variable name.
:param input_to_check: The input variable need to get gradient.
:param input_to_check: The input variable with respect to which to compute the gradient.
:param delta: The perturbation value for numeric gradient method. The
smaller delta is, the more accurate result will get. But if that delta is
too small, it could occur numerical stability problem.
too small, it will suffer from numerical stability problem.
:param local_scope: The local scope used for get_numeric_gradient.
:return: The gradient array in numpy format.
"""
...
...
@@ -45,28 +45,28 @@ def get_numeric_gradient(op,
### Explaination:
- Why need `output_name`
- One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.
- An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.
- Why need `input_to_check`
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
### Core Algorithm Implementation
```python
# we only compute gradient of one element each time.
# we use a for loop to compute the gradient of every element.
# we only compute gradient of one element a time.
# we use a for loop to compute the gradient of each element.
for i in xrange(tensor_size):
# get one input element throw it's index i.
# get one input element by its index i.
origin = tensor_to_check.get_float_element(i)
# add delta to it, run op and then get the sum of the result tensor.
# add delta to it, run op and then get the new value of the result tensor.
x_pos = origin + delta
tensor_to_check.set_float_element(i, x_pos)
y_pos = get_output()
# plus delta to this element, run op and get the sum of the result tensor.
# plus delta to this element, run op and get the new value of the result tensor.
x_neg = origin - delta
tensor_to_check.set_float_element(i, x_neg)
y_neg = get_output()
...
...
@@ -85,15 +85,15 @@ def get_numeric_gradient(op,
Each Operator Kernel has three kinds of Gradient:
- 1. Numeric Gradient
- 2. CPU Operator Gradient
- 3. GPU Operator Gradient(if supported)
1. Numerical gradient
2. CPU kernel gradient
3. GPU kernel gradient (if supported)
Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.
The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:
- 1. calculate the numeric gradient.
- 2. calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.
- 3. calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)
1. calculate the numerical gradient
2. calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient
3. calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)
#### Python Interface
...
...
@@ -110,8 +110,8 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
:param forward_op: used to create backward_op
:param input_vars: numpy value of input variable. The following
computation will use these variables.
:param inputs_to_check: inputs var names that should check gradient.
:param output_name: output name that used to
:param inputs_to_check: the input variable with respect to which to compute the gradient.
:param output_name: The final output variable name.
:param max_relative_error: The relative tolerance parameter.
:param no_grad_set: used when create backward ops
:param only_cpu: only compute and check gradient on cpu kernel.
...
...
@@ -120,24 +120,24 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
```
### How to check if two numpy array is close enough?
if `abs_numeric_grad` is nearly zero, then use abs error for numeric_grad, not relative
if `abs_numerical_grad` is nearly zero, then use abs error for numerical_grad
@@ -56,7 +56,7 @@ For each parameter, like W and b created by `layer.fc`, marked as double circles
## Block and Graph
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block](https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
A Block keeps operators in an array `BlockDesc::ops`
...
...
@@ -67,4 +67,4 @@ message BlockDesc {
}
```
in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
`Parameters` is a concept we designed in Paddle V2 API. `Parameters` is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
`Parameters` is a concept we designed in PaddlePaddle V2 API. `Parameters` is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:
We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:
* We just use `memcpy` to share Parameters between topologies, but this is very inefficient.
* We did not implement share Parameters while training. We just trigger `memcpy` when start training.
* We did not support sharing Parameters while training. We just trigger `memcpy` when start training.
It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with `Parameters`:
It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with `Parameters`:
1. `paddle::Parameter`. A `Parameters` is a container for `paddle::Parameter`.
It is evident that we should use `paddle::Parameter` when developing `Parameters`.
However, the `Parameter` class contains many functions and does not have a clear interface.
It contains `create/store Parameter`, `serialize/deserialize`, `optimize(i.e SGD)`, `randomize/zero`.
When we developing `Parameters`, we only use `create/store Parameter` functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.
2. `paddle::GradientMachine` and its sub-classes, e.g., `paddle::MultiGradientMachine`, `paddle::NeuralNetwork`.
We should pass `Parameters` to `paddle::GradientMachine` when `forward/backward` to avoid `memcpy` between topologies.
...
...
@@ -24,7 +24,7 @@ Also, we should handle multi-GPU/CPU training, because `forward` and `backward`
So `Parameters` should be used by `paddle::ParameterUpdater`, and `paddle::ParameterUpdater` should optimize `Parameters` (by SGD).
The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.
The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.
1. Clean `paddle::Parameter` interface. Extract the functionalities of `paddle::Parameter` to prepare for the implementation of Parameters.
@@ -17,7 +17,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope only contains a map of a name to variable.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.
1. Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.
...
...
@@ -32,7 +32,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope should destruct all Variables inside it when itself is destructed. User can never store `Variable` pointer somewhere else.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be a invalid pointer when associated `Scope` is destroyed.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be an invalid pointer when associated `Scope` is destroyed.
```cpp
class Scope {
...
...
@@ -50,7 +50,7 @@ class Scope {
Just like [scope](https://en.wikipedia.org/wiki/Scope_(computer_science)) in programming languages, `Scope` in the neural network can also be a local scope. There are two attributes about local scope.
1. We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.
1. We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.
2. Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, `scope` will keep searching from its parent, until the variable is found or there is no parent.
```cpp
...
...
@@ -121,4 +121,4 @@ Also, as the parent scope is a `shared_ptr`, we can only `Create()` a scope shar
## Orthogonal interface
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return a `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return an `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
@@ -6,9 +6,9 @@ The Interaction between Python and C++ can be simplified as two steps:
1. C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.
### Message form C++ to Python
### Message from C++ to Python
We define a Protobuf message class `OpProto` to hold message needed in the first step. What should an `OpProto` contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”
<spanid="backgraound"></span><h1>Backgraound：<aclass="headerlink"href="#backgraound"title="Permalink to this headline">¶</a></h1>
<ulclass="simple">
<li>Operator forward computing is easy to check if the result is right because it has a clear definition. <strong>But</strong> backpropagation is a notoriously difficult algorithm to debug and get right:<ul>
<li><olclass="first">
<li>Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:<ol>
<li>you should get the right backpropagation formula according to the forward computation.</li>
</ol>
</li>
<li><olclass="first">
<li>you should implement it right in CPP.</li>
</ol>
</li>
<li><olclass="first">
<li>it’s difficult to prepare test data.</li>
</ol>
</li>
</ul>
</li>
<li>Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator’s result. It has several advantages:<ul>
<li><olclass="first">
<li>numeric gradient checker only need forward operator.</li>
</ol>
</li>
<li><olclass="first">
<li>Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator’s result. It has several advantages:<ol>
<li>numerical gradient checker only need forward operator.</li>
<li>user only need to prepare the input data for forward Operator.</li>
</ol>
</li>
</ul>
</li>
</ul>
</div>
<divclass="section"id="mathematical-theory">
<spanid="mathematical-theory"></span><h1>Mathematical Theory<aclass="headerlink"href="#mathematical-theory"title="Permalink to this headline">¶</a></h1>
<p>The following two document from stanford has a detailed explanation of how to get numeric gradient and why it’s useful.</p>
<p>The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it’s useful.</p>
<divclass="toctree-wrapper compound">
<ul>
<liclass="toctree-l1"><aclass="reference external"href="http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization">Gradient checking and advanced optimization(en)</a></li>
...
...
@@ -225,7 +210,7 @@
<spanid="numeric-gradient-implementation"></span><h1>Numeric Gradient Implementation<aclass="headerlink"href="#numeric-gradient-implementation"title="Permalink to this headline">¶</a></h1>
<divclass="section"id="python-interface">
<spanid="python-interface"></span><h2>Python Interface<aclass="headerlink"href="#python-interface"title="Permalink to this headline">¶</a></h2>
<spanclass="sd"> Get Numeric Gradient for an operator's input.</span>
<spanclass="sd"> :param op: C++ operator instance, could be an network</span>
<spanclass="sd"> :param input_values: The input variables. Should be an dictionary, key is</span>
<spanclass="sd"> variable name. Value is numpy array.</span>
<spanclass="sd"> :param input_values: The input variables. Should be an dictionary, whose key is</span>
<spanclass="sd"> variable name, and value is numpy array.</span>
<spanclass="sd"> :param output_name: The final output variable name.</span>
<spanclass="sd"> :param input_to_check: The input variable need to get gradient.</span>
<spanclass="sd"> :param input_to_check: The input variable with respect to which to compute the gradient.</span>
<spanclass="sd"> :param delta: The perturbation value for numeric gradient method. The</span>
<spanclass="sd"> smaller delta is, the more accurate result will get. But if that delta is</span>
<spanclass="sd"> too small, it could occur numerical stability problem.</span>
<spanclass="sd"> too small, it will suffer from numerical stability problem.</span>
<spanclass="sd"> :param local_scope: The local scope used for get_numeric_gradient.</span>
<spanclass="sd"> :return: The gradient array in numpy format.</span>
<spanclass="sd">"""</span>
...
...
@@ -252,29 +237,29 @@
<spanid="explaination"></span><h2>Explaination:<aclass="headerlink"href="#explaination"title="Permalink to this headline">¶</a></h2>
<ulclass="simple">
<li>Why need <codeclass="docutils literal"><spanclass="pre">output_name</span></code><ul>
<li>One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.</li>
<li>An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.</li>
</ul>
</li>
<li>Why need <codeclass="docutils literal"><spanclass="pre">input_to_check</span></code><ul>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> multiple times.</li>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> multiple times.</li>
<spanid="core-algorithm-implementation"></span><h2>Core Algorithm Implementation<aclass="headerlink"href="#core-algorithm-implementation"title="Permalink to this headline">¶</a></h2>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="c1"># we only compute gradient of one element each time.</span>
<spanclass="c1"># we use a for loop to compute the gradient of every element.</span>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="c1"># we only compute gradient of one element a time.</span>
<spanclass="c1"># we use a for loop to compute the gradient of each element.</span>
<spanid="auto-graident-checker-framework"></span><h1>Auto Graident Checker Framework<aclass="headerlink"href="#auto-graident-checker-framework"title="Permalink to this headline">¶</a></h1>
<p>Each Operator Kernel has three kinds of Gradient:</p>
<ulclass="simple">
<li><olclass="first">
<li>Numeric Gradient</li>
</ol>
</li>
<li><olclass="first">
<li>CPU Operator Gradient</li>
</ol>
</li>
<li><olclass="first">
<li>GPU Operator Gradient(if supported)</li>
</ol>
</li>
</ul>
<p>Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.</p>
<ulclass="simple">
<li><olclass="first">
<li>calculate the numeric gradient.</li>
</ol>
</li>
<li><olclass="first">
<li>calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.</li>
<olclass="simple">
<li>Numerical gradient</li>
<li>CPU kernel gradient</li>
<li>GPU kernel gradient (if supported)</li>
</ol>
</li>
<li><olclass="first">
<li>calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)</li>
<p>The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:</p>
<olclass="simple">
<li>calculate the numerical gradient</li>
<li>calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient</li>
<li>calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)</li>
</ol>
</li>
</ul>
<divclass="section"id="python-interface">
<spanid="id1"></span><h2>Python Interface<aclass="headerlink"href="#python-interface"title="Permalink to this headline">¶</a></h2>
<spanid="how-to-check-if-two-numpy-array-is-close-enough"></span><h2>How to check if two numpy array is close enough?<aclass="headerlink"href="#how-to-check-if-two-numpy-array-is-close-enough"title="Permalink to this headline">¶</a></h2>
<p>if <codeclass="docutils literal"><spanclass="pre">abs_numeric_grad</span></code> is nearly zero, then use abs error for numeric_grad, not relative</p>
<p>Let’s explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we’d like to have Python functions <codeclass="docutils literal"><spanclass="pre">mul</span></code> and <codeclass="docutils literal"><spanclass="pre">add</span></code> defined in module <codeclass="docutils literal"><spanclass="pre">operator</span></code>:</p>
<spanid="block-and-graph"></span><h2>Block and Graph<aclass="headerlink"href="#block-and-graph"title="Permalink to this headline">¶</a></h2>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/pull/3708">Block</a> is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>A Block keeps operators in an array <codeclass="docutils literal"><spanclass="pre">BlockDesc::ops</span></code></p>
<p>in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in <codeclass="docutils literal"><spanclass="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
<p>in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in <codeclass="docutils literal"><spanclass="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
<spanid="design-doc-the-c-class-parameters"></span><h1>Design Doc: The C++ Class <codeclass="docutils literal"><spanclass="pre">Parameters</span></code><aclass="headerlink"href="#design-doc-the-c-class-parameters"title="Permalink to this headline">¶</a></h1>
<p><codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a concept we designed in Paddle V2 API. <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of <codeclass="docutils literal"><spanclass="pre">Parameter</span></code> in <aclass="reference internal"href="api.html"><spanclass="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:</p>
<p><codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a concept we designed in PaddlePaddle V2 API. <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of <codeclass="docutils literal"><spanclass="pre">Parameter</span></code> in <aclass="reference internal"href="api.html"><spanclass="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:</p>
<ulclass="simple">
<li>We just use <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> to share Parameters between topologies, but this is very inefficient.</li>
<li>We did not implement share Parameters while training. We just trigger <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> when start training.</li>
<li>We did not support sharing Parameters while training. We just trigger <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> when start training.</li>
</ul>
<p>It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>:</p>
<p>It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>:</p>
<olclass="simple">
<li><codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code>. A <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a container for <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code>.
It is evident that we should use <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> when developing <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>.
However, the <codeclass="docutils literal"><spanclass="pre">Parameter</span></code> class contains many functions and does not have a clear interface.
It contains <codeclass="docutils literal"><spanclass="pre">create/store</span><spanclass="pre">Parameter</span></code>, <codeclass="docutils literal"><spanclass="pre">serialize/deserialize</span></code>, <codeclass="docutils literal"><spanclass="pre">optimize(i.e</span><spanclass="pre">SGD)</span></code>, <codeclass="docutils literal"><spanclass="pre">randomize/zero</span></code>.
When we developing <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>, we only use <codeclass="docutils literal"><spanclass="pre">create/store</span><spanclass="pre">Parameter</span></code> functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.</li>
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.</li>
<li><codeclass="docutils literal"><spanclass="pre">paddle::GradientMachine</span></code> and its sub-classes, e.g., <codeclass="docutils literal"><spanclass="pre">paddle::MultiGradientMachine</span></code>, <codeclass="docutils literal"><spanclass="pre">paddle::NeuralNetwork</span></code>.
We should pass <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> to <codeclass="docutils literal"><spanclass="pre">paddle::GradientMachine</span></code> when <codeclass="docutils literal"><spanclass="pre">forward/backward</span></code> to avoid <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> between topologies.
Also, we should handle multi-GPU/CPU training, because <codeclass="docutils literal"><spanclass="pre">forward</span></code> and <codeclass="docutils literal"><spanclass="pre">backward</span></code> would perform on multi-GPUs and multi-CPUs.
...
...
@@ -200,7 +200,7 @@ Also, we should handle multi-GPU/CPU training, because <code class="docutils lit
<li><codeclass="docutils literal"><spanclass="pre">paddle::ParameterUpdater</span></code>. The ParameterUpdater is used to update parameters in Paddle.
So <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> should be used by <codeclass="docutils literal"><spanclass="pre">paddle::ParameterUpdater</span></code>, and <codeclass="docutils literal"><spanclass="pre">paddle::ParameterUpdater</span></code> should optimize <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> (by SGD).</li>
</ol>
<p>The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.</p>
<p>The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.</p>
<olclass="simple">
<li>Clean <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> interface. Extract the functionalities of <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> to prepare for the implementation of Parameters.</li>
<li>Implementation a <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> class. It just stores the <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> inside. Make <codeclass="docutils literal"><spanclass="pre">GradientMachine</span></code> uses <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> as a class member.</li>
<li>PaddlePaddle represent the computation, training and inference of DL models, by computation graphs.</li>
<li>Please dig into <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a solid example.</li>
<li>Users write Python programs to describe the graphs and run it (locally or remotely).</li>
<li>A graph is composed of <em>variabels</em> and <em>operators</em>.</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be able to be serialized/deserialized, so it<ol>
<li>could to be sent to the cloud for distributed execution, and</li>
<li>be sent to clients for mobile or enterprise deployment.</li>
...
...
@@ -346,7 +346,7 @@
</ul>
</li>
<li>Hand-writing <codeclass="docutils literal"><spanclass="pre">GPUKernel</span></code> and <codeclass="docutils literal"><spanclass="pre">CPU</span></code> code<ul>
<li>Do not write <codeclass="docutils literal"><spanclass="pre">.h</span></code>. CPU Kernel should be in <codeclass="docutils literal"><spanclass="pre">.cc</span></code>. CPU kernel should be in <codeclass="docutils literal"><spanclass="pre">.cu</span></code>. (<codeclass="docutils literal"><spanclass="pre">GCC</span></code> cannot compile GPU code.)</li>
<li>Do not write <codeclass="docutils literal"><spanclass="pre">.h</span></code>. CPU Kernel should be in <codeclass="docutils literal"><spanclass="pre">.cc</span></code>. GPU kernel should be in <codeclass="docutils literal"><spanclass="pre">.cu</span></code>. (<codeclass="docutils literal"><spanclass="pre">GCC</span></code> cannot compile GPU code.)</li>
<spanid="id2"></span><h1>PaddlePaddle回归测试列表<aclass="headerlink"href="#paddlepaddle"title="Permalink to this headline">¶</a></h1>
<p>本列表说明PaddlePaddle发版之前需要测试的功能点。</p>
<divclass="section"id="paddlepaddle-book">
<spanid="paddlepaddle-book"></span><h2>PaddlePaddle Book中所有章节<aclass="headerlink"href="#paddlepaddle-book"title="Permalink to this headline">¶</a></h2>
<p>Scope is an association of a name to variable. All variables belong to <codeclass="docutils literal"><spanclass="pre">Scope</span></code>. You need to specify a scope to run a Net, i.e., <codeclass="docutils literal"><spanclass="pre">net.Run(&scope)</span></code>. One net can run in different scopes and update different variable in the scope.</p>
<ol>
<li><pclass="first">Scope only contains a map of a name to variable.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.</p>
</li>
<li><pclass="first">Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.</p>
</li>
...
...
@@ -208,7 +208,7 @@
<p>Variable can not belong to many scopes. If you want to use variables from parent scope, you can use <codeclass="docutils literal"><spanclass="pre">parent</span><spanclass="pre">scope</span></code>.</p>
</li>
<li><pclass="first">Scope should destruct all Variables inside it when itself is destructed. User can never store <codeclass="docutils literal"><spanclass="pre">Variable</span></code> pointer somewhere else.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <codeclass="docutils literal"><spanclass="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be a invalid pointer when associated <codeclass="docutils literal"><spanclass="pre">Scope</span></code> is destroyed.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <codeclass="docutils literal"><spanclass="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be an invalid pointer when associated <codeclass="docutils literal"><spanclass="pre">Scope</span></code> is destroyed.</p>
<spanid="parent-scope-and-local-scope"></span><h2>Parent scope and local scope<aclass="headerlink"href="#parent-scope-and-local-scope"title="Permalink to this headline">¶</a></h2>
<p>Just like <aclass="reference external"href="https://en.wikipedia.org/wiki/Scope_(computer_science)">scope</a> in programming languages, <codeclass="docutils literal"><spanclass="pre">Scope</span></code> in the neural network can also be a local scope. There are two attributes about local scope.</p>
<olclass="simple">
<li>We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.</li>
<li>We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.</li>
<li>Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, <codeclass="docutils literal"><spanclass="pre">scope</span></code> will keep searching from its parent, until the variable is found or there is no parent.</li>
<spanid="orthogonal-interface"></span><h2>Orthogonal interface<aclass="headerlink"href="#orthogonal-interface"title="Permalink to this headline">¶</a></h2>
<p><codeclass="docutils literal"><spanclass="pre">FindVar</span></code> will return <codeclass="docutils literal"><spanclass="pre">nullptr</span></code> when <codeclass="docutils literal"><spanclass="pre">name</span></code> is not found. It can be used as <codeclass="docutils literal"><spanclass="pre">Contains</span></code> method. <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> will return a <codeclass="docutils literal"><spanclass="pre">Error</span></code> when there is a name conflict locally. Combine <codeclass="docutils literal"><spanclass="pre">FindVar</span></code> and <codeclass="docutils literal"><spanclass="pre">NewVar</span></code>, we can implement <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> easily.</p>
<p><codeclass="docutils literal"><spanclass="pre">FindVar</span></code> will return <codeclass="docutils literal"><spanclass="pre">nullptr</span></code> when <codeclass="docutils literal"><spanclass="pre">name</span></code> is not found. It can be used as <codeclass="docutils literal"><spanclass="pre">Contains</span></code> method. <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> will return an<codeclass="docutils literal"><spanclass="pre">Error</span></code> when there is a name conflict locally. Combine <codeclass="docutils literal"><spanclass="pre">FindVar</span></code> and <codeclass="docutils literal"><spanclass="pre">NewVar</span></code>, we can implement <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> easily.</p>
<p>The Interaction between Python and C++ can be simplified as two steps:</p>
<olclass="simple">
<li>C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.</li>
</ol>
<divclass="section"id="message-form-c-to-python">
<spanid="message-form-c-to-python"></span><h2>Message form C++ to Python<aclass="headerlink"href="#message-form-c-to-python"title="Permalink to this headline">¶</a></h2>
<divclass="section"id="message-from-c-to-python">
<spanid="message-from-c-to-python"></span><h2>Message from C++ to Python<aclass="headerlink"href="#message-from-c-to-python"title="Permalink to this headline">¶</a></h2>
<p>We define a Protobuf message class <codeclass="docutils literal"><spanclass="pre">OpProto</span></code> to hold message needed in the first step. What should an <codeclass="docutils literal"><spanclass="pre">OpProto</span></code> contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”</p>
- Operator forward computing is easy to check if the result is right because it has a clear definition. **But** backpropagation is a notoriously difficult algorithm to debug and get right:
- 1. you should get the right backpropagation formula according to the forward computation.
- 2. you should implement it right in CPP.
- 3. it's difficult to prepare test data.
- Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:
1. you should get the right backpropagation formula according to the forward computation.
2. you should implement it right in CPP.
3. it's difficult to prepare test data.
- Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
- 1. numeric gradient checker only need forward operator.
- 2. user only need to prepare the input data for forward Operator.
- Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
1. numerical gradient checker only need forward operator.
2. user only need to prepare the input data for forward Operator.
## Mathematical Theory
The following two document from stanford has a detailed explanation of how to get numeric gradient and why it's useful.
The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it's useful.
- [Gradient checking and advanced optimization(en)](http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization)
- [Gradient checking and advanced optimization(cn)](http://ufldl.stanford.edu/wiki/index.php/%E6%A2%AF%E5%BA%A6%E6%A3%80%E9%AA%8C%E4%B8%8E%E9%AB%98%E7%BA%A7%E4%BC%98%E5%8C%96)
...
...
@@ -20,7 +20,7 @@ The following two document from stanford has a detailed explanation of how to ge
## Numeric Gradient Implementation
### Python Interface
```python
def get_numeric_gradient(op,
def get_numerical_gradient(op,
input_values,
output_name,
input_to_check,
...
...
@@ -30,13 +30,13 @@ def get_numeric_gradient(op,
Get Numeric Gradient for an operator's input.
:param op: C++ operator instance, could be an network
:param input_values: The input variables. Should be an dictionary, key is
variable name. Value is numpy array.
:param input_values: The input variables. Should be an dictionary, whose key is
variable name, and value is numpy array.
:param output_name: The final output variable name.
:param input_to_check: The input variable need to get gradient.
:param input_to_check: The input variable with respect to which to compute the gradient.
:param delta: The perturbation value for numeric gradient method. The
smaller delta is, the more accurate result will get. But if that delta is
too small, it could occur numerical stability problem.
too small, it will suffer from numerical stability problem.
:param local_scope: The local scope used for get_numeric_gradient.
:return: The gradient array in numpy format.
"""
...
...
@@ -45,28 +45,28 @@ def get_numeric_gradient(op,
### Explaination:
- Why need `output_name`
- One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.
- An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.
- Why need `input_to_check`
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
### Core Algorithm Implementation
```python
# we only compute gradient of one element each time.
# we use a for loop to compute the gradient of every element.
# we only compute gradient of one element a time.
# we use a for loop to compute the gradient of each element.
for i in xrange(tensor_size):
# get one input element throw it's index i.
# get one input element by its index i.
origin = tensor_to_check.get_float_element(i)
# add delta to it, run op and then get the sum of the result tensor.
# add delta to it, run op and then get the new value of the result tensor.
x_pos = origin + delta
tensor_to_check.set_float_element(i, x_pos)
y_pos = get_output()
# plus delta to this element, run op and get the sum of the result tensor.
# plus delta to this element, run op and get the new value of the result tensor.
x_neg = origin - delta
tensor_to_check.set_float_element(i, x_neg)
y_neg = get_output()
...
...
@@ -85,15 +85,15 @@ def get_numeric_gradient(op,
Each Operator Kernel has three kinds of Gradient:
- 1. Numeric Gradient
- 2. CPU Operator Gradient
- 3. GPU Operator Gradient(if supported)
1. Numerical gradient
2. CPU kernel gradient
3. GPU kernel gradient (if supported)
Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.
The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:
- 1. calculate the numeric gradient.
- 2. calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.
- 3. calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)
1. calculate the numerical gradient
2. calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient
3. calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)
#### Python Interface
...
...
@@ -110,8 +110,8 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
:param forward_op: used to create backward_op
:param input_vars: numpy value of input variable. The following
computation will use these variables.
:param inputs_to_check: inputs var names that should check gradient.
:param output_name: output name that used to
:param inputs_to_check: the input variable with respect to which to compute the gradient.
:param output_name: The final output variable name.
:param max_relative_error: The relative tolerance parameter.
:param no_grad_set: used when create backward ops
:param only_cpu: only compute and check gradient on cpu kernel.
...
...
@@ -120,24 +120,24 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
```
### How to check if two numpy array is close enough?
if `abs_numeric_grad` is nearly zero, then use abs error for numeric_grad, not relative
if `abs_numerical_grad` is nearly zero, then use abs error for numerical_grad
@@ -56,7 +56,7 @@ For each parameter, like W and b created by `layer.fc`, marked as double circles
## Block and Graph
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block](https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
A Block keeps operators in an array `BlockDesc::ops`
...
...
@@ -67,4 +67,4 @@ message BlockDesc {
}
```
in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
`Parameters` is a concept we designed in Paddle V2 API. `Parameters` is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
`Parameters` is a concept we designed in PaddlePaddle V2 API. `Parameters` is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:
We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:
* We just use `memcpy` to share Parameters between topologies, but this is very inefficient.
* We did not implement share Parameters while training. We just trigger `memcpy` when start training.
* We did not support sharing Parameters while training. We just trigger `memcpy` when start training.
It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with `Parameters`:
It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with `Parameters`:
1. `paddle::Parameter`. A `Parameters` is a container for `paddle::Parameter`.
It is evident that we should use `paddle::Parameter` when developing `Parameters`.
However, the `Parameter` class contains many functions and does not have a clear interface.
It contains `create/store Parameter`, `serialize/deserialize`, `optimize(i.e SGD)`, `randomize/zero`.
When we developing `Parameters`, we only use `create/store Parameter` functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.
2. `paddle::GradientMachine` and its sub-classes, e.g., `paddle::MultiGradientMachine`, `paddle::NeuralNetwork`.
We should pass `Parameters` to `paddle::GradientMachine` when `forward/backward` to avoid `memcpy` between topologies.
...
...
@@ -24,7 +24,7 @@ Also, we should handle multi-GPU/CPU training, because `forward` and `backward`
So `Parameters` should be used by `paddle::ParameterUpdater`, and `paddle::ParameterUpdater` should optimize `Parameters` (by SGD).
The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.
The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.
1. Clean `paddle::Parameter` interface. Extract the functionalities of `paddle::Parameter` to prepare for the implementation of Parameters.
@@ -17,7 +17,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope only contains a map of a name to variable.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.
1. Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.
...
...
@@ -32,7 +32,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope should destruct all Variables inside it when itself is destructed. User can never store `Variable` pointer somewhere else.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be a invalid pointer when associated `Scope` is destroyed.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be an invalid pointer when associated `Scope` is destroyed.
```cpp
class Scope {
...
...
@@ -50,7 +50,7 @@ class Scope {
Just like [scope](https://en.wikipedia.org/wiki/Scope_(computer_science)) in programming languages, `Scope` in the neural network can also be a local scope. There are two attributes about local scope.
1. We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.
1. We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.
2. Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, `scope` will keep searching from its parent, until the variable is found or there is no parent.
```cpp
...
...
@@ -121,4 +121,4 @@ Also, as the parent scope is a `shared_ptr`, we can only `Create()` a scope shar
## Orthogonal interface
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return a `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return an `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
@@ -6,9 +6,9 @@ The Interaction between Python and C++ can be simplified as two steps:
1. C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.
### Message form C++ to Python
### Message from C++ to Python
We define a Protobuf message class `OpProto` to hold message needed in the first step. What should an `OpProto` contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”
<li>Operator forward computing is easy to check if the result is right because it has a clear definition. <strong>But</strong> backpropagation is a notoriously difficult algorithm to debug and get right:<ul>
<li><olclass="first">
<li>Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:<ol>
<li>you should get the right backpropagation formula according to the forward computation.</li>
</ol>
</li>
<li><olclass="first">
<li>you should implement it right in CPP.</li>
</ol>
</li>
<li><olclass="first">
<li>it’s difficult to prepare test data.</li>
</ol>
</li>
</ul>
</li>
<li>Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator’s result. It has several advantages:<ul>
<li><olclass="first">
<li>numeric gradient checker only need forward operator.</li>
</ol>
</li>
<li><olclass="first">
<li>Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator’s result. It has several advantages:<ol>
<li>numerical gradient checker only need forward operator.</li>
<li>user only need to prepare the input data for forward Operator.</li>
<li>Why need <codeclass="docutils literal"><spanclass="pre">output_name</span></code><ul>
<li>One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.</li>
<li>An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.</li>
</ul>
</li>
<li>Why need <codeclass="docutils literal"><spanclass="pre">input_to_check</span></code><ul>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> multiple times.</li>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <codeclass="docutils literal"><spanclass="pre">get_numeric_gradient</span></code> multiple times.</li>
<p>Each Operator Kernel has three kinds of Gradient:</p>
<ulclass="simple">
<li><olclass="first">
<li>Numeric Gradient</li>
</ol>
</li>
<li><olclass="first">
<li>CPU Operator Gradient</li>
</ol>
</li>
<li><olclass="first">
<li>GPU Operator Gradient(if supported)</li>
</ol>
</li>
</ul>
<p>Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.</p>
<ulclass="simple">
<li><olclass="first">
<li>calculate the numeric gradient.</li>
</ol>
</li>
<li><olclass="first">
<li>calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.</li>
<olclass="simple">
<li>Numerical gradient</li>
<li>CPU kernel gradient</li>
<li>GPU kernel gradient (if supported)</li>
</ol>
</li>
<li><olclass="first">
<li>calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)</li>
<p>The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:</p>
<olclass="simple">
<li>calculate the numerical gradient</li>
<li>calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient</li>
<li>calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)</li>
<spanid="how-to-check-if-two-numpy-array-is-close-enough"></span><h2>How to check if two numpy array is close enough?<aclass="headerlink"href="#how-to-check-if-two-numpy-array-is-close-enough"title="永久链接至标题">¶</a></h2>
<p>if <codeclass="docutils literal"><spanclass="pre">abs_numeric_grad</span></code> is nearly zero, then use abs error for numeric_grad, not relative</p>
<p>Let’s explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we’d like to have Python functions <codeclass="docutils literal"><spanclass="pre">mul</span></code> and <codeclass="docutils literal"><spanclass="pre">add</span></code> defined in module <codeclass="docutils literal"><spanclass="pre">operator</span></code>:</p>
<spanid="block-and-graph"></span><h2>Block and Graph<aclass="headerlink"href="#block-and-graph"title="永久链接至标题">¶</a></h2>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/pull/3708">Block</a> is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>A Block keeps operators in an array <codeclass="docutils literal"><spanclass="pre">BlockDesc::ops</span></code></p>
<p>in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in <codeclass="docutils literal"><spanclass="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
<p>in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in <codeclass="docutils literal"><spanclass="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
<spanid="design-doc-the-c-class-parameters"></span><h1>Design Doc: The C++ Class <codeclass="docutils literal"><spanclass="pre">Parameters</span></code><aclass="headerlink"href="#design-doc-the-c-class-parameters"title="永久链接至标题">¶</a></h1>
<p><codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a concept we designed in Paddle V2 API. <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of <codeclass="docutils literal"><spanclass="pre">Parameter</span></code> in <aclass="reference internal"href="api.html"><spanclass="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:</p>
<p><codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a concept we designed in PaddlePaddle V2 API. <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of <codeclass="docutils literal"><spanclass="pre">Parameter</span></code> in <aclass="reference internal"href="api.html"><spanclass="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:</p>
<ulclass="simple">
<li>We just use <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> to share Parameters between topologies, but this is very inefficient.</li>
<li>We did not implement share Parameters while training. We just trigger <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> when start training.</li>
<li>We did not support sharing Parameters while training. We just trigger <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> when start training.</li>
</ul>
<p>It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>:</p>
<p>It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>:</p>
<olclass="simple">
<li><codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code>. A <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> is a container for <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code>.
It is evident that we should use <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> when developing <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>.
However, the <codeclass="docutils literal"><spanclass="pre">Parameter</span></code> class contains many functions and does not have a clear interface.
It contains <codeclass="docutils literal"><spanclass="pre">create/store</span><spanclass="pre">Parameter</span></code>, <codeclass="docutils literal"><spanclass="pre">serialize/deserialize</span></code>, <codeclass="docutils literal"><spanclass="pre">optimize(i.e</span><spanclass="pre">SGD)</span></code>, <codeclass="docutils literal"><spanclass="pre">randomize/zero</span></code>.
When we developing <codeclass="docutils literal"><spanclass="pre">Parameters</span></code>, we only use <codeclass="docutils literal"><spanclass="pre">create/store</span><spanclass="pre">Parameter</span></code> functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.</li>
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.</li>
<li><codeclass="docutils literal"><spanclass="pre">paddle::GradientMachine</span></code> and its sub-classes, e.g., <codeclass="docutils literal"><spanclass="pre">paddle::MultiGradientMachine</span></code>, <codeclass="docutils literal"><spanclass="pre">paddle::NeuralNetwork</span></code>.
We should pass <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> to <codeclass="docutils literal"><spanclass="pre">paddle::GradientMachine</span></code> when <codeclass="docutils literal"><spanclass="pre">forward/backward</span></code> to avoid <codeclass="docutils literal"><spanclass="pre">memcpy</span></code> between topologies.
Also, we should handle multi-GPU/CPU training, because <codeclass="docutils literal"><spanclass="pre">forward</span></code> and <codeclass="docutils literal"><spanclass="pre">backward</span></code> would perform on multi-GPUs and multi-CPUs.
...
...
@@ -207,7 +207,7 @@ Also, we should handle multi-GPU/CPU training, because <code class="docutils lit
<li><codeclass="docutils literal"><spanclass="pre">paddle::ParameterUpdater</span></code>. The ParameterUpdater is used to update parameters in Paddle.
So <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> should be used by <codeclass="docutils literal"><spanclass="pre">paddle::ParameterUpdater</span></code>, and <codeclass="docutils literal"><spanclass="pre">paddle::ParameterUpdater</span></code> should optimize <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> (by SGD).</li>
</ol>
<p>The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.</p>
<p>The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.</p>
<olclass="simple">
<li>Clean <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> interface. Extract the functionalities of <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> to prepare for the implementation of Parameters.</li>
<li>Implementation a <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> class. It just stores the <codeclass="docutils literal"><spanclass="pre">paddle::Parameter</span></code> inside. Make <codeclass="docutils literal"><spanclass="pre">GradientMachine</span></code> uses <codeclass="docutils literal"><spanclass="pre">Parameters</span></code> as a class member.</li>
<li>PaddlePaddle represent the computation, training and inference of DL models, by computation graphs.</li>
<li>Please dig into <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a solid example.</li>
<li>Users write Python programs to describe the graphs and run it (locally or remotely).</li>
<li>A graph is composed of <em>variabels</em> and <em>operators</em>.</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be able to be serialized/deserialized, so it<ol>
<li>could to be sent to the cloud for distributed execution, and</li>
<li>be sent to clients for mobile or enterprise deployment.</li>
...
...
@@ -353,7 +353,7 @@
</ul>
</li>
<li>Hand-writing <codeclass="docutils literal"><spanclass="pre">GPUKernel</span></code> and <codeclass="docutils literal"><spanclass="pre">CPU</span></code> code<ul>
<li>Do not write <codeclass="docutils literal"><spanclass="pre">.h</span></code>. CPU Kernel should be in <codeclass="docutils literal"><spanclass="pre">.cc</span></code>. CPU kernel should be in <codeclass="docutils literal"><spanclass="pre">.cu</span></code>. (<codeclass="docutils literal"><spanclass="pre">GCC</span></code> cannot compile GPU code.)</li>
<li>Do not write <codeclass="docutils literal"><spanclass="pre">.h</span></code>. CPU Kernel should be in <codeclass="docutils literal"><spanclass="pre">.cc</span></code>. GPU kernel should be in <codeclass="docutils literal"><spanclass="pre">.cu</span></code>. (<codeclass="docutils literal"><spanclass="pre">GCC</span></code> cannot compile GPU code.)</li>
<p>Scope is an association of a name to variable. All variables belong to <codeclass="docutils literal"><spanclass="pre">Scope</span></code>. You need to specify a scope to run a Net, i.e., <codeclass="docutils literal"><spanclass="pre">net.Run(&scope)</span></code>. One net can run in different scopes and update different variable in the scope.</p>
<ol>
<li><pclass="first">Scope only contains a map of a name to variable.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.</p>
</li>
<li><pclass="first">Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.</p>
</li>
...
...
@@ -215,7 +215,7 @@
<p>Variable can not belong to many scopes. If you want to use variables from parent scope, you can use <codeclass="docutils literal"><spanclass="pre">parent</span><spanclass="pre">scope</span></code>.</p>
</li>
<li><pclass="first">Scope should destruct all Variables inside it when itself is destructed. User can never store <codeclass="docutils literal"><spanclass="pre">Variable</span></code> pointer somewhere else.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <codeclass="docutils literal"><spanclass="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be a invalid pointer when associated <codeclass="docutils literal"><spanclass="pre">Scope</span></code> is destroyed.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <codeclass="docutils literal"><spanclass="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be an invalid pointer when associated <codeclass="docutils literal"><spanclass="pre">Scope</span></code> is destroyed.</p>
<spanid="parent-scope-and-local-scope"></span><h2>Parent scope and local scope<aclass="headerlink"href="#parent-scope-and-local-scope"title="永久链接至标题">¶</a></h2>
<p>Just like <aclass="reference external"href="https://en.wikipedia.org/wiki/Scope_(computer_science)">scope</a> in programming languages, <codeclass="docutils literal"><spanclass="pre">Scope</span></code> in the neural network can also be a local scope. There are two attributes about local scope.</p>
<olclass="simple">
<li>We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.</li>
<li>We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.</li>
<li>Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, <codeclass="docutils literal"><spanclass="pre">scope</span></code> will keep searching from its parent, until the variable is found or there is no parent.</li>
<p><codeclass="docutils literal"><spanclass="pre">FindVar</span></code> will return <codeclass="docutils literal"><spanclass="pre">nullptr</span></code> when <codeclass="docutils literal"><spanclass="pre">name</span></code> is not found. It can be used as <codeclass="docutils literal"><spanclass="pre">Contains</span></code> method. <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> will return a <codeclass="docutils literal"><spanclass="pre">Error</span></code> when there is a name conflict locally. Combine <codeclass="docutils literal"><spanclass="pre">FindVar</span></code> and <codeclass="docutils literal"><spanclass="pre">NewVar</span></code>, we can implement <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> easily.</p>
<p><codeclass="docutils literal"><spanclass="pre">FindVar</span></code> will return <codeclass="docutils literal"><spanclass="pre">nullptr</span></code> when <codeclass="docutils literal"><spanclass="pre">name</span></code> is not found. It can be used as <codeclass="docutils literal"><spanclass="pre">Contains</span></code> method. <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> will return an<codeclass="docutils literal"><spanclass="pre">Error</span></code> when there is a name conflict locally. Combine <codeclass="docutils literal"><spanclass="pre">FindVar</span></code> and <codeclass="docutils literal"><spanclass="pre">NewVar</span></code>, we can implement <codeclass="docutils literal"><spanclass="pre">NewVar</span></code> easily.</p>
<p>The Interaction between Python and C++ can be simplified as two steps:</p>
<olclass="simple">
<li>C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.</li>
</ol>
<divclass="section"id="message-form-c-to-python">
<spanid="message-form-c-to-python"></span><h2>Message form C++ to Python<aclass="headerlink"href="#message-form-c-to-python"title="永久链接至标题">¶</a></h2>
<divclass="section"id="message-from-c-to-python">
<spanid="message-from-c-to-python"></span><h2>Message from C++ to Python<aclass="headerlink"href="#message-from-c-to-python"title="永久链接至标题">¶</a></h2>
<p>We define a Protobuf message class <codeclass="docutils literal"><spanclass="pre">OpProto</span></code> to hold message needed in the first step. What should an <codeclass="docutils literal"><spanclass="pre">OpProto</span></code> contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”</p>
