Skip to content

P
Paddle

机器未来 / Paddle
与 Fork 源项目一致

Fork自 PaddlePaddle / Paddle

通知 1
Star 1
Fork 0
P
Paddle
“7d6263e6e547c2a8cbcc5886129f8a8df246ac97”上不存在“test/legacy_test/test_optimizer_in_control_flow.py”
提交 477e97f1 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: 63bc2ff8

上级 6cdb184c
显示空白变更内容
内联 并排
Showing with 294 addition and 360 deletion
develop/doc/_sources/design/api.md.txt
浏览文件 @ 477e97f1
......@@ -3,7 +3,7 @@
## Ingredients
As our design principle is starting from the essence: how could we
allow users to express and solve their problems at neural networks.
allow users to express and solve their problems as neural networks.
Some essential concepts that our API have to provide include:
1. A *topology* is an expression of *layers*.
......@@ -233,7 +233,7 @@ paddle.dist_train(model,
num_parameter_servers=15)
```
The pseudo code if `paddle.dist_train` is as follows:
The pseudo code of `paddle.dist_train` is as follows:
```python
def dist_train(topology, parameters, trainer, reader, ...):
......
develop/doc/_sources/design/auto_gradient_check.md.txt
浏览文件 @ 477e97f1
## Auto Gradient Checker Design
## Backgraound：
- Operator forward computing is easy to check if the result is right because it has a clear definition. **But** backpropagation is a notoriously difficult algorithm to debug and get right:
- 1. you should get the right backpropagation formula according to the forward computation.
- 2. you should implement it right in CPP.
- 3. it's difficult to prepare test data.
- Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:
1. you should get the right backpropagation formula according to the forward computation.
2. you should implement it right in CPP.
3. it's difficult to prepare test data.
- Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
- 1. numeric gradient checker only need forward operator.
- 2. user only need to prepare the input data for forward Operator.
- Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
1. numerical gradient checker only need forward operator.
2. user only need to prepare the input data for forward Operator.
## Mathematical Theory
The following two document from stanford has a detailed explanation of how to get numeric gradient and why it's useful.
The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it's useful.
- [Gradient checking and advanced optimization(en)](http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization)
- [Gradient checking and advanced optimization(cn)](http://ufldl.stanford.edu/wiki/index.php/%E6%A2%AF%E5%BA%A6%E6%A3%80%E9%AA%8C%E4%B8%8E%E9%AB%98%E7%BA%A7%E4%BC%98%E5%8C%96)
......@@ -20,7 +20,7 @@ The following two document from stanford has a detailed explanation of how to ge
## Numeric Gradient Implementation
### Python Interface
```python
def get_numeric_gradient(op,
def get_numerical_gradient(op,
input_values,
output_name,
input_to_check,
......@@ -30,13 +30,13 @@ def get_numeric_gradient(op,
Get Numeric Gradient for an operator's input.
:param op: C++ operator instance, could be an network
:param input_values: The input variables. Should be an dictionary, key is
variable name. Value is numpy array.
:param input_values: The input variables. Should be an dictionary, whose key is
variable name, and value is numpy array.
:param output_name: The final output variable name.
:param input_to_check: The input variable need to get gradient.
:param input_to_check: The input variable with respect to which to compute the gradient.
:param delta: The perturbation value for numeric gradient method. The
smaller delta is, the more accurate result will get. But if that delta is
too small, it could occur numerical stability problem.
too small, it will suffer from numerical stability problem.
:param local_scope: The local scope used for get_numeric_gradient.
:return: The gradient array in numpy format.
"""
......@@ -45,28 +45,28 @@ def get_numeric_gradient(op,
### Explaination:
- Why need `output_name`
- One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.
- An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.
- Why need `input_to_check`
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
### Core Algorithm Implementation
```python
# we only compute gradient of one element each time.
# we use a for loop to compute the gradient of every element.
# we only compute gradient of one element a time.
# we use a for loop to compute the gradient of each element.
for i in xrange(tensor_size):
# get one input element throw it's index i.
# get one input element by its index i.
origin = tensor_to_check.get_float_element(i)
# add delta to it, run op and then get the sum of the result tensor.
# add delta to it, run op and then get the new value of the result tensor.
x_pos = origin + delta
tensor_to_check.set_float_element(i, x_pos)
y_pos = get_output()
# plus delta to this element, run op and get the sum of the result tensor.
# plus delta to this element, run op and get the new value of the result tensor.
x_neg = origin - delta
tensor_to_check.set_float_element(i, x_neg)
y_neg = get_output()
......@@ -85,15 +85,15 @@ def get_numeric_gradient(op,
Each Operator Kernel has three kinds of Gradient:
- 1. Numeric Gradient
- 2. CPU Operator Gradient
- 3. GPU Operator Gradient(if supported)
1. Numerical gradient
2. CPU kernel gradient
3. GPU kernel gradient (if supported)
Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.
The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:
- 1. calculate the numeric gradient.
- 2. calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.
- 3. calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)
1. calculate the numerical gradient
2. calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient
3. calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)
#### Python Interface
......@@ -110,8 +110,8 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
:param forward_op: used to create backward_op
:param input_vars: numpy value of input variable. The following
computation will use these variables.
:param inputs_to_check: inputs var names that should check gradient.
:param output_name: output name that used to
:param inputs_to_check: the input variable with respect to which to compute the gradient.
:param output_name: The final output variable name.
:param max_relative_error: The relative tolerance parameter.
:param no_grad_set: used when create backward ops
:param only_cpu: only compute and check gradient on cpu kernel.
......@@ -120,24 +120,24 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
```
### How to check if two numpy array is close enough?
if `abs_numeric_grad` is nearly zero, then use abs error for numeric_grad, not relative
if `abs_numerical_grad` is nearly zero, then use abs error for numerical_grad
```python
numeric_grad = ...
numerical_grad = ...
operator_grad = numpy.array(scope.find_var(grad_var_name(name)).get_tensor())
abs_numeric_grad = numpy.abs(numeric_grad)
# if abs_numeric_grad is nearly zero, then use abs error for numeric_grad, not relative
abs_numerical_grad = numpy.abs(numerical_grad)
# if abs_numerical_grad is nearly zero, then use abs error for numeric_grad, not relative
# error.
abs_numeric_grad[abs_numeric_grad < 1e-3] = 1
abs_numerical_grad[abs_numerical_grad < 1e-3] = 1
diff_mat = numpy.abs(abs_numeric_grad - operator_grad) / abs_numeric_grad
diff_mat = numpy.abs(abs_numerical_grad - operator_grad) / abs_numerical_grad
max_diff = numpy.max(diff_mat)
```
#### Notes：
1，The Input data for auto gradient checker should be reasonable to avoid numeric problem.
The Input data for auto gradient checker should be reasonable to avoid numerical stability problem.
#### Refs:
......
develop/doc/_sources/design/functions_operators_layers.md.txt
浏览文件 @ 477e97f1
......@@ -53,12 +53,12 @@ Let's explain using an example. Suppose that we are going to compose the FC usi
```python
def operator.mul(X1, X2):
O = Var()
paddle.cpp.create_operator("mul", input={X1, Y1], output=O)
paddle.cpp.create_operator("mul", input={X1, Y1}, output=O)
return O
def operator.add(X1, X2):
O = Var()
paddle.cpp.create_operator("add", input={X1, X2], output=O)
paddle.cpp.create_operator("add", input={X1, X2}, output=O)
return O
```
......
develop/doc/_sources/design/graph.md.txt
浏览文件 @ 477e97f1
......@@ -56,7 +56,7 @@ For each parameter, like W and b created by `layer.fc`, marked as double circles
## Block and Graph
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block](https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
A Block keeps operators in an array `BlockDesc::ops`
......@@ -67,4 +67,4 @@ message BlockDesc {
}
```
in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
develop/doc/_sources/design/parameters_in_cpp.md.txt
浏览文件 @ 477e97f1
# Design Doc: The C++ Class `Parameters`
`Parameters` is a concept we designed in Paddle V2 API. `Parameters` is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
`Parameters` is a concept we designed in PaddlePaddle V2 API. `Parameters` is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:
We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:
* We just use `memcpy` to share Parameters between topologies, but this is very inefficient.
* We did not implement share Parameters while training. We just trigger `memcpy` when start training.
* We did not support sharing Parameters while training. We just trigger `memcpy` when start training.
It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with `Parameters`:
It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with `Parameters`:
1. `paddle::Parameter`. A `Parameters` is a container for `paddle::Parameter`.
It is evident that we should use `paddle::Parameter` when developing `Parameters`.
However, the `Parameter` class contains many functions and does not have a clear interface.
It contains `create/store Parameter`, `serialize/deserialize`, `optimize(i.e SGD)`, `randomize/zero`.
When we developing `Parameters`, we only use `create/store Parameter` functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.
2. `paddle::GradientMachine` and its sub-classes, e.g., `paddle::MultiGradientMachine`, `paddle::NeuralNetwork`.
We should pass `Parameters` to `paddle::GradientMachine` when `forward/backward` to avoid `memcpy` between topologies.
......@@ -24,7 +24,7 @@ Also, we should handle multi-GPU/CPU training, because `forward` and `backward`
So `Parameters` should be used by `paddle::ParameterUpdater`, and `paddle::ParameterUpdater` should optimize `Parameters` (by SGD).
The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.
The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.
1. Clean `paddle::Parameter` interface. Extract the functionalities of `paddle::Parameter` to prepare for the implementation of Parameters.
......
develop/doc/_sources/design/reader/README.md.txt
浏览文件 @ 477e97f1
......@@ -52,7 +52,7 @@ Here are valid outputs:
# a mini batch of three data items, each data item is a list (single column).
[([1,1,1],),
([2,2,2],),
([3,3,3],),
([3,3,3],)]
```
Please note that each item inside the list must be a tuple, below is an invalid output:
......
develop/doc/_sources/design/refactorization.md.txt
浏览文件 @ 477e97f1
......@@ -15,7 +15,7 @@ The goal of refactorizaiton include:
1. Users write Python programs to describe the graphs and run it (locally or remotely).
1. A graph is composed of *variabels* and *operators*.
1. A graph is composed of *variables* and *operators*.
1. The description of graphs must be able to be serialized/deserialized, so it
......@@ -140,7 +140,7 @@ Compile Time -> IR -> Runtime
* `thrust` has the same API as C++ standard library. Using `transform` can quickly implement a customized elementwise kernel.
* `thrust` has more complex API, like `scan`, `reduce`, `reduce_by_key`.
* Hand-writing `GPUKernel` and `CPU` code
* Do not write `.h`. CPU Kernel should be in `.cc`. CPU kernel should be in `.cu`. (`GCC` cannot compile GPU code.)
* Do not write `.h`. CPU Kernel should be in `.cc`. GPU kernel should be in `.cu`. (`GCC` cannot compile GPU code.)
---
# Operator Register
......
develop/doc/_sources/design/releasing_process.md.txt
浏览文件 @ 477e97f1
# Paddle发行规范
# PaddlePaddle发行规范
Paddle使用git-flow branching model做分支管理，使用[Semantic Versioning](http://semver.org/)标准表示Paddle版本号。
PaddlePaddle使用git-flow branching model做分支管理，使用[Semantic Versioning](http://semver.org/)标准表示PaddlePaddle版本号。
Paddle每次发新的版本，遵循以下流程:
PaddlePaddle每次发新的版本，遵循以下流程:
1. 从`develop`分支派生出新的分支，分支名为`release/版本号`。例如，`release/0.10.0`
2. 将新分支的版本打上tag，tag为`版本号rc.Patch号`。第一个tag为`0.10.0rc1`，第二个为`0.10.0rc2`，依次类推。
......@@ -27,14 +27,14 @@ Paddle每次发新的版本，遵循以下流程:
需要注意的是:
* `release/版本号`分支一旦建立，一般不允许再从`develop`分支合入`release/版本号`。这样保证`release/版本号`分支功能的封闭，方便测试人员测试Paddle的行为。
* `release/版本号`分支一旦建立，一般不允许再从`develop`分支合入`release/版本号`。这样保证`release/版本号`分支功能的封闭，方便测试人员测试PaddlePaddle的行为。
* 在`release/版本号`分支存在的时候，如果有bugfix的行为，需要将bugfix的分支同时merge到`master`, `develop`和`release/版本号`这三个分支。
# Paddle 分支规范
# PaddlePaddle 分支规范
Paddle开发过程使用[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范，并适应github的特性做了一些区别。
PaddlePaddle开发过程使用[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范，并适应github的特性做了一些区别。
* Paddle的主版本库遵循[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范。其中:
* PaddlePaddle的主版本库遵循[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范。其中:
* `master`分支为稳定(stable branch)版本分支。每一个`master`分支的版本都是经过单元测试和回归测试的版本。
* `develop`分支为开发(develop branch)版本分支。每一个`develop`分支的版本都经过单元测试，但并没有经过回归测试。
* `release/版本号`分支为每一次Release时建立的临时分支。在这个阶段的代码正在经历回归测试。
......@@ -42,18 +42,18 @@ Paddle开发过程使用[git-flow](http://nvie.com/posts/a-successful-git-branch
* 其他用户的fork版本库并不需要严格遵守[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范，但所有fork的版本库的所有分支都相当于特性分支。
* 建议，开发者fork的版本库使用`develop`分支同步主版本库的`develop`分支
* 建议，开发者fork的版本库中，再基于`develop`版本fork出自己的功能分支。
* 当功能分支开发完毕后，向Paddle的主版本库提交`Pull Reuqest`，进而进行代码评审。
* 当功能分支开发完毕后，向PaddlePaddle的主版本库提交`Pull Reuqest`，进而进行代码评审。
* 在评审过程中，开发者修改自己的代码，可以继续在自己的功能分支提交代码。
* BugFix分支也是在开发者自己的fork版本库维护，与功能分支不同的是，BugFix分支需要分别给主版本库的`master`、`develop`与可能有的`release/版本号`分支，同时提起`Pull Request`。
# Paddle回归测试列表
# PaddlePaddle回归测试列表
本列表说明Paddle发版之前需要测试的功能点。
本列表说明PaddlePaddle发版之前需要测试的功能点。
## Paddle Book中所有章节
## PaddlePaddle Book中所有章节
Paddle每次发版本首先要保证Paddle Book中所有章节功能的正确性。功能的正确性包括验证Paddle目前的`paddle_trainer`训练和纯使用`Python`训练模型正确性。
PaddlePaddle每次发版本首先要保证PaddlePaddle Book中所有章节功能的正确性。功能的正确性包括验证PaddlePaddle目前的`paddle_trainer`训练和纯使用`Python`训练模型正确性。
| | 新手入门章节 | 识别数字 | 图像分类 | 词向量 | 情感分析 | 语意角色标注 | 机器翻译 | 个性化推荐 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
......
develop/doc/_sources/design/scope.md.txt
浏览文件 @ 477e97f1
......@@ -17,7 +17,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope only contains a map of a name to variable.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state (momentum) etc.
1. Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.
......@@ -32,7 +32,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope should destruct all Variables inside it when itself is destructed. User can never store `Variable` pointer somewhere else.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be a invalid pointer when associated `Scope` is destroyed.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be an invalid pointer when associated `Scope` is destroyed.
```cpp
class Scope {
......@@ -50,7 +50,7 @@ class Scope {
Just like [scope](https://en.wikipedia.org/wiki/Scope_(computer_science)) in programming languages, `Scope` in the neural network can also be a local scope. There are two attributes about local scope.
1. We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.
1. We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.
2. Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, `scope` will keep searching from its parent, until the variable is found or there is no parent.
```cpp
......@@ -121,4 +121,4 @@ Also, as the parent scope is a `shared_ptr`, we can only `Create()` a scope shar
## Orthogonal interface
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return a `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return an `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
develop/doc/_sources/design/simple_op_design.md.txt
浏览文件 @ 477e97f1
......@@ -6,9 +6,9 @@ The Interaction between Python and C++ can be simplified as two steps:
1. C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.
### Message form C++ to Python
### Message from C++ to Python
We define a Protobuf message class `OpProto` to hold message needed in the first step. What should an `OpProto` contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”
......
develop/doc/_sources/design/var_desc.md.txt
浏览文件 @ 477e97f1
## Background
PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.
PaddlePaddle use proto message to describe compile time graph for
PaddlePaddle use proto message to describe compile time graph because
1. Computation graph should be able to be saved to a file.
1. In distributed training, the graph will be serialized and send to multiple workers.
......
develop/doc/design/api.html
浏览文件 @ 477e97f1
......@@ -182,7 +182,7 @@
<div class="section" id="ingredients">
<span id="ingredients"></span><h2>Ingredients<a class="headerlink" href="#ingredients" title="Permalink to this headline"></a></h2>
<p>As our design principle is starting from the essence: how could we
allow users to express and solve their problems at neural networks.
allow users to express and solve their problems as neural networks.
Some essential concepts that our API have to provide include:</p>
<ol class="simple">
<li>A <em>topology</em> is an expression of <em>layers</em>.</li>
......@@ -385,7 +385,7 @@ access a Kubernetes cluster, s/he should be able to call</p>
<span class="n">num_parameter_servers</span><span class="o">=</span><span class="mi">15</span><span class="p">)</span>
</pre></div>
</div>
<p>The pseudo code if <code class="docutils literal"><span class="pre">paddle.dist_train</span></code> is as follows:</p>
<p>The pseudo code of <code class="docutils literal"><span class="pre">paddle.dist_train</span></code> is as follows:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">dist_train</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">trainer</span><span class="p">,</span> <span class="n">reader</span><span class="p">,</span> <span class="o">...</span><span class="p">):</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;KUBERNETES_SERVICE_HOST&quot;</span><span class="p">)</span> <span class="o">==</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">image_name</span> <span class="o">=</span> <span class="n">k8s_user</span> <span class="o">+</span> <span class="s1">&#39;/&#39;</span> <span class="o">+</span> <span class="n">k8s_job</span>
......
develop/doc/design/auto_gradient_check.html
浏览文件 @ 477e97f1
......@@ -183,37 +183,22 @@
<div class="section" id="backgraound">
<span id="backgraound"></span><h1>Backgraound：<a class="headerlink" href="#backgraound" title="Permalink to this headline"></a></h1>
<ul class="simple">
<li>Operator forward computing is easy to check if the result is right because it has a clear definition. <strong>But</strong> backpropagation is a notoriously difficult algorithm to debug and get right:<ul>
<li><ol class="first">
<li>Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:<ol>
<li>you should get the right backpropagation formula according to the forward computation.</li>
</ol>
</li>
<li><ol class="first">
<li>you should implement it right in CPP.</li>
</ol>
</li>
<li><ol class="first">
<li>it&#8217;s difficult to prepare test data.</li>
</ol>
</li>
</ul>
</li>
<li>Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator&#8217;s result. It has several advantages:<ul>
<li><ol class="first">
<li>numeric gradient checker only need forward operator.</li>
</ol>
</li>
<li><ol class="first">
<li>Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator&#8217;s result. It has several advantages:<ol>
<li>numerical gradient checker only need forward operator.</li>
<li>user only need to prepare the input data for forward Operator.</li>
</ol>
</li>
</ul>
</li>
</ul>
</div>
<div class="section" id="mathematical-theory">
<span id="mathematical-theory"></span><h1>Mathematical Theory<a class="headerlink" href="#mathematical-theory" title="Permalink to this headline"></a></h1>
<p>The following two document from stanford has a detailed explanation of how to get numeric gradient and why it&#8217;s useful.</p>
<p>The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it&#8217;s useful.</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference external" href="http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization">Gradient checking and advanced optimization(en)</a></li>
......@@ -225,7 +210,7 @@
<span id="numeric-gradient-implementation"></span><h1>Numeric Gradient Implementation<a class="headerlink" href="#numeric-gradient-implementation" title="Permalink to this headline"></a></h1>
<div class="section" id="python-interface">
<span id="python-interface"></span><h2>Python Interface<a class="headerlink" href="#python-interface" title="Permalink to this headline"></a></h2>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_numeric_gradient</span><span class="p">(</span><span class="n">op</span><span class="p">,</span>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_numerical_gradient</span><span class="p">(</span><span class="n">op</span><span class="p">,</span>
<span class="n">input_values</span><span class="p">,</span>
<span class="n">output_name</span><span class="p">,</span>
<span class="n">input_to_check</span><span class="p">,</span>
......@@ -235,13 +220,13 @@
<span class="sd"> Get Numeric Gradient for an operator&#39;s input.</span>
<span class="sd"> :param op: C++ operator instance, could be an network</span>
<span class="sd"> :param input_values: The input variables. Should be an dictionary, key is</span>
<span class="sd"> variable name. Value is numpy array.</span>
<span class="sd"> :param input_values: The input variables. Should be an dictionary, whose key is</span>
<span class="sd"> variable name, and value is numpy array.</span>
<span class="sd"> :param output_name: The final output variable name.</span>
<span class="sd"> :param input_to_check: The input variable need to get gradient.</span>
<span class="sd"> :param input_to_check: The input variable with respect to which to compute the gradient.</span>
<span class="sd"> :param delta: The perturbation value for numeric gradient method. The</span>
<span class="sd"> smaller delta is, the more accurate result will get. But if that delta is</span>
<span class="sd"> too small, it could occur numerical stability problem.</span>
<span class="sd"> too small, it will suffer from numerical stability problem.</span>
<span class="sd"> :param local_scope: The local scope used for get_numeric_gradient.</span>
<span class="sd"> :return: The gradient array in numpy format.</span>
<span class="sd"> &quot;&quot;&quot;</span>
......@@ -252,29 +237,29 @@
<span id="explaination"></span><h2>Explaination:<a class="headerlink" href="#explaination" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li>Why need <code class="docutils literal"><span class="pre">output_name</span></code><ul>
<li>One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.</li>
<li>An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.</li>
</ul>
</li>
<li>Why need <code class="docutils literal"><span class="pre">input_to_check</span></code><ul>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> multiple times.</li>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> multiple times.</li>
</ul>
</li>
</ul>
</div>
<div class="section" id="core-algorithm-implementation">
<span id="core-algorithm-implementation"></span><h2>Core Algorithm Implementation<a class="headerlink" href="#core-algorithm-implementation" title="Permalink to this headline"></a></h2>
<div class="highlight-python"><div class="highlight"><pre><span></span> <span class="c1"># we only compute gradient of one element each time.</span>
<span class="c1"># we use a for loop to compute the gradient of every element.</span>
<div class="highlight-python"><div class="highlight"><pre><span></span> <span class="c1"># we only compute gradient of one element a time.</span>
<span class="c1"># we use a for loop to compute the gradient of each element.</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">tensor_size</span><span class="p">):</span>
<span class="c1"># get one input element throw it&#39;s index i.</span>
<span class="c1"># get one input element by its index i.</span>
<span class="n">origin</span> <span class="o">=</span> <span class="n">tensor_to_check</span><span class="o">.</span><span class="n">get_float_element</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="c1"># add delta to it, run op and then get the sum of the result tensor.</span>
<span class="c1"># add delta to it, run op and then get the new value of the result tensor.</span>
<span class="n">x_pos</span> <span class="o">=</span> <span class="n">origin</span> <span class="o">+</span> <span class="n">delta</span>
<span class="n">tensor_to_check</span><span class="o">.</span><span class="n">set_float_element</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">x_pos</span><span class="p">)</span>
<span class="n">y_pos</span> <span class="o">=</span> <span class="n">get_output</span><span class="p">()</span>
<span class="c1"># plus delta to this element, run op and get the sum of the result tensor.</span>
<span class="c1"># plus delta to this element, run op and get the new value of the result tensor.</span>
<span class="n">x_neg</span> <span class="o">=</span> <span class="n">origin</span> <span class="o">-</span> <span class="n">delta</span>
<span class="n">tensor_to_check</span><span class="o">.</span><span class="n">set_float_element</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">x_neg</span><span class="p">)</span>
<span class="n">y_neg</span> <span class="o">=</span> <span class="n">get_output</span><span class="p">()</span>
......@@ -294,35 +279,17 @@
<div class="section" id="auto-graident-checker-framework">
<span id="auto-graident-checker-framework"></span><h1>Auto Graident Checker Framework<a class="headerlink" href="#auto-graident-checker-framework" title="Permalink to this headline"></a></h1>
<p>Each Operator Kernel has three kinds of Gradient:</p>
<ul class="simple">
<li><ol class="first">
<li>Numeric Gradient</li>
<ol class="simple">
<li>Numerical gradient</li>
<li>CPU kernel gradient</li>
<li>GPU kernel gradient (if supported)</li>
</ol>
</li>
<li><ol class="first">
<li>CPU Operator Gradient</li>
<p>The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:</p>
<ol class="simple">
<li>calculate the numerical gradient</li>
<li>calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient</li>
<li>calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)</li>
</ol>
</li>
<li><ol class="first">
<li>GPU Operator Gradient(if supported)</li>
</ol>
</li>
</ul>
<p>Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.</p>
<ul class="simple">
<li><ol class="first">
<li>calculate the numeric gradient.</li>
</ol>
</li>
<li><ol class="first">
<li>calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.</li>
</ol>
</li>
<li><ol class="first">
<li>calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)</li>
</ol>
</li>
</ul>
<div class="section" id="python-interface">
<span id="id1"></span><h2>Python Interface<a class="headerlink" href="#python-interface" title="Permalink to this headline"></a></h2>
<div class="highlight-python"><div class="highlight"><pre><span></span> <span class="k">def</span> <span class="nf">check_grad</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
......@@ -337,8 +304,8 @@
<span class="sd"> :param forward_op: used to create backward_op</span>
<span class="sd"> :param input_vars: numpy value of input variable. The following</span>
<span class="sd"> computation will use these variables.</span>
<span class="sd"> :param inputs_to_check: inputs var names that should check gradient.</span>
<span class="sd"> :param output_name: output name that used to</span>
<span class="sd"> :param inputs_to_check: the input variable with respect to which to compute the gradient.</span>
<span class="sd"> :param output_name: The final output variable name.</span>
<span class="sd"> :param max_relative_error: The relative tolerance parameter.</span>
<span class="sd"> :param no_grad_set: used when create backward ops</span>
<span class="sd"> :param only_cpu: only compute and check gradient on cpu kernel.</span>
......@@ -349,22 +316,22 @@
</div>
<div class="section" id="how-to-check-if-two-numpy-array-is-close-enough">
<span id="how-to-check-if-two-numpy-array-is-close-enough"></span><h2>How to check if two numpy array is close enough?<a class="headerlink" href="#how-to-check-if-two-numpy-array-is-close-enough" title="Permalink to this headline"></a></h2>
<p>if <code class="docutils literal"><span class="pre">abs_numeric_grad</span></code> is nearly zero, then use abs error for numeric_grad, not relative</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">numeric_grad</span> <span class="o">=</span> <span class="o">...</span>
<p>if <code class="docutils literal"><span class="pre">abs_numerical_grad</span></code> is nearly zero, then use abs error for numerical_grad</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">numerical_grad</span> <span class="o">=</span> <span class="o">...</span>
<span class="n">operator_grad</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">find_var</span><span class="p">(</span><span class="n">grad_var_name</span><span class="p">(</span><span class="n">name</span><span class="p">))</span><span class="o">.</span><span class="n">get_tensor</span><span class="p">())</span>
<span class="n">abs_numeric_grad</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">numeric_grad</span><span class="p">)</span>
<span class="c1"># if abs_numeric_grad is nearly zero, then use abs error for numeric_grad, not relative</span>
<span class="n">abs_numerical_grad</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">numerical_grad</span><span class="p">)</span>
<span class="c1"># if abs_numerical_grad is nearly zero, then use abs error for numeric_grad, not relative</span>
<span class="c1"># error.</span>
<span class="n">abs_numeric_grad</span><span class="p">[</span><span class="n">abs_numeric_grad</span> <span class="o">&lt;</span> <span class="mf">1e-3</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">abs_numerical_grad</span><span class="p">[</span><span class="n">abs_numerical_grad</span> <span class="o">&lt;</span> <span class="mf">1e-3</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">diff_mat</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">abs_numeric_grad</span> <span class="o">-</span> <span class="n">operator_grad</span><span class="p">)</span> <span class="o">/</span> <span class="n">abs_numeric_grad</span>
<span class="n">diff_mat</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">abs_numerical_grad</span> <span class="o">-</span> <span class="n">operator_grad</span><span class="p">)</span> <span class="o">/</span> <span class="n">abs_numerical_grad</span>
<span class="n">max_diff</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">diff_mat</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="notes">
<span id="notes"></span><h3>Notes：<a class="headerlink" href="#notes" title="Permalink to this headline"></a></h3>
<p>1，The Input data for auto gradient checker should be reasonable to avoid numeric problem.</p>
<p>The Input data for auto gradient checker should be reasonable to avoid numerical stability problem.</p>
</div>
<div class="section" id="refs">
<span id="refs"></span><h3>Refs:<a class="headerlink" href="#refs" title="Permalink to this headline"></a></h3>
......
develop/doc/design/functions_operators_layers.html
浏览文件 @ 477e97f1
......@@ -217,12 +217,12 @@
<p>Let&#8217;s explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we&#8217;d like to have Python functions <code class="docutils literal"><span class="pre">mul</span></code> and <code class="docutils literal"><span class="pre">add</span></code> defined in module <code class="docutils literal"><span class="pre">operator</span></code>:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">operator</span><span class="o">.</span><span class="n">mul</span><span class="p">(</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">):</span>
<span class="n">O</span> <span class="o">=</span> <span class="n">Var</span><span class="p">()</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;mul&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">Y1</span><span class="p">],</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;mul&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">Y1</span><span class="p">},</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="k">return</span> <span class="n">O</span>
<span class="k">def</span> <span class="nf">operator</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">):</span>
<span class="n">O</span> <span class="o">=</span> <span class="n">Var</span><span class="p">()</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;add&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">],</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;add&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">},</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="k">return</span> <span class="n">O</span>
</pre></div>
</div>
......
develop/doc/design/graph.html
浏览文件 @ 477e97f1
......@@ -226,7 +226,7 @@
</div>
<div class="section" id="block-and-graph">
<span id="block-and-graph"></span><h2>Block and Graph<a class="headerlink" href="#block-and-graph" title="Permalink to this headline"></a></h2>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/pull/3708">Block</a> is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>A Block keeps operators in an array <code class="docutils literal"><span class="pre">BlockDesc::ops</span></code></p>
<div class="highlight-protobuf"><div class="highlight"><pre><span></span><span class="kd">message</span> <span class="nc">BlockDesc</span> <span class="p">{</span>
<span class="k">repeated</span> <span class="n">OpDesc</span> <span class="na">ops</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
......@@ -234,7 +234,7 @@
<span class="p">}</span>
</pre></div>
</div>
<p>in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in <code class="docutils literal"><span class="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
<p>in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in <code class="docutils literal"><span class="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
</div>
</div>
......
develop/doc/design/parameters_in_cpp.html
浏览文件 @ 477e97f1
......@@ -179,20 +179,20 @@
<div class="section" id="design-doc-the-c-class-parameters">
<span id="design-doc-the-c-class-parameters"></span><h1>Design Doc: The C++ Class <code class="docutils literal"><span class="pre">Parameters</span></code><a class="headerlink" href="#design-doc-the-c-class-parameters" title="Permalink to this headline"></a></h1>
<p><code class="docutils literal"><span class="pre">Parameters</span></code> is a concept we designed in Paddle V2 API. <code class="docutils literal"><span class="pre">Parameters</span></code> is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of <code class="docutils literal"><span class="pre">Parameter</span></code> in <a class="reference internal" href="api.html"><span class="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:</p>
<p><code class="docutils literal"><span class="pre">Parameters</span></code> is a concept we designed in PaddlePaddle V2 API. <code class="docutils literal"><span class="pre">Parameters</span></code> is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of <code class="docutils literal"><span class="pre">Parameter</span></code> in <a class="reference internal" href="api.html"><span class="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:</p>
<ul class="simple">
<li>We just use <code class="docutils literal"><span class="pre">memcpy</span></code> to share Parameters between topologies, but this is very inefficient.</li>
<li>We did not implement share Parameters while training. We just trigger <code class="docutils literal"><span class="pre">memcpy</span></code> when start training.</li>
<li>We did not support sharing Parameters while training. We just trigger <code class="docutils literal"><span class="pre">memcpy</span></code> when start training.</li>
</ul>
<p>It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with <code class="docutils literal"><span class="pre">Parameters</span></code>:</p>
<p>It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with <code class="docutils literal"><span class="pre">Parameters</span></code>:</p>
<ol class="simple">
<li><code class="docutils literal"><span class="pre">paddle::Parameter</span></code>. A <code class="docutils literal"><span class="pre">Parameters</span></code> is a container for <code class="docutils literal"><span class="pre">paddle::Parameter</span></code>.
It is evident that we should use <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> when developing <code class="docutils literal"><span class="pre">Parameters</span></code>.
However, the <code class="docutils literal"><span class="pre">Parameter</span></code> class contains many functions and does not have a clear interface.
It contains <code class="docutils literal"><span class="pre">create/store</span> <span class="pre">Parameter</span></code>, <code class="docutils literal"><span class="pre">serialize/deserialize</span></code>, <code class="docutils literal"><span class="pre">optimize(i.e</span> <span class="pre">SGD)</span></code>, <code class="docutils literal"><span class="pre">randomize/zero</span></code>.
When we developing <code class="docutils literal"><span class="pre">Parameters</span></code>, we only use <code class="docutils literal"><span class="pre">create/store</span> <span class="pre">Parameter</span></code> functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.</li>
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.</li>
<li><code class="docutils literal"><span class="pre">paddle::GradientMachine</span></code> and its sub-classes, e.g., <code class="docutils literal"><span class="pre">paddle::MultiGradientMachine</span></code>, <code class="docutils literal"><span class="pre">paddle::NeuralNetwork</span></code>.
We should pass <code class="docutils literal"><span class="pre">Parameters</span></code> to <code class="docutils literal"><span class="pre">paddle::GradientMachine</span></code> when <code class="docutils literal"><span class="pre">forward/backward</span></code> to avoid <code class="docutils literal"><span class="pre">memcpy</span></code> between topologies.
Also, we should handle multi-GPU/CPU training, because <code class="docutils literal"><span class="pre">forward</span></code> and <code class="docutils literal"><span class="pre">backward</span></code> would perform on multi-GPUs and multi-CPUs.
......@@ -200,7 +200,7 @@ Also, we should handle multi-GPU/CPU training, because <code class="docutils lit
<li><code class="docutils literal"><span class="pre">paddle::ParameterUpdater</span></code>. The ParameterUpdater is used to update parameters in Paddle.
So <code class="docutils literal"><span class="pre">Parameters</span></code> should be used by <code class="docutils literal"><span class="pre">paddle::ParameterUpdater</span></code>, and <code class="docutils literal"><span class="pre">paddle::ParameterUpdater</span></code> should optimize <code class="docutils literal"><span class="pre">Parameters</span></code> (by SGD).</li>
</ol>
<p>The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.</p>
<p>The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.</p>
<ol class="simple">
<li>Clean <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> interface. Extract the functionalities of <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> to prepare for the implementation of Parameters.</li>
<li>Implementation a <code class="docutils literal"><span class="pre">Parameters</span></code> class. It just stores the <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> inside. Make <code class="docutils literal"><span class="pre">GradientMachine</span></code> uses <code class="docutils literal"><span class="pre">Parameters</span></code> as a class member.</li>
......
develop/doc/design/reader/README.html
浏览文件 @ 477e97f1
......@@ -223,7 +223,7 @@
<span class="c1"># a mini batch of three data items, each data item is a list (single column).</span>
<span class="p">[([</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],),</span>
<span class="p">([</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">],),</span>
<span class="p">([</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">],),</span>
<span class="p">([</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">],)]</span>
</pre></div>
</div>
<p>Please note that each item inside the list must be a tuple, below is an invalid output:</p>
......
develop/doc/design/refactorization.html
浏览文件 @ 477e97f1
......@@ -192,7 +192,7 @@
<li>PaddlePaddle represent the computation, training and inference of DL models, by computation graphs.</li>
<li>Please dig into <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a solid example.</li>
<li>Users write Python programs to describe the graphs and run it (locally or remotely).</li>
<li>A graph is composed of <em>variabels</em> and <em>operators</em>.</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be able to be serialized/deserialized, so it<ol>
<li>could to be sent to the cloud for distributed execution, and</li>
<li>be sent to clients for mobile or enterprise deployment.</li>
......@@ -346,7 +346,7 @@
</ul>
</li>
<li>Hand-writing <code class="docutils literal"><span class="pre">GPUKernel</span></code> and <code class="docutils literal"><span class="pre">CPU</span></code> code<ul>
<li>Do not write <code class="docutils literal"><span class="pre">.h</span></code>. CPU Kernel should be in <code class="docutils literal"><span class="pre">.cc</span></code>. CPU kernel should be in <code class="docutils literal"><span class="pre">.cu</span></code>. (<code class="docutils literal"><span class="pre">GCC</span></code> cannot compile GPU code.)</li>
<li>Do not write <code class="docutils literal"><span class="pre">.h</span></code>. CPU Kernel should be in <code class="docutils literal"><span class="pre">.cc</span></code>. GPU kernel should be in <code class="docutils literal"><span class="pre">.cu</span></code>. (<code class="docutils literal"><span class="pre">GCC</span></code> cannot compile GPU code.)</li>
</ul>
</li>
</ul>
......
develop/doc/design/releasing_process.html
浏览文件 @ 477e97f1
......@@ -8,7 +8,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Paddle发行规范 &mdash; PaddlePaddle documentation</title>
<title>PaddlePaddle发行规范 &mdash; PaddlePaddle documentation</title>
......@@ -168,7 +168,7 @@
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li>Paddle发行规范</li>
<li>PaddlePaddle发行规范</li>
</ul>
</div>
......@@ -177,10 +177,10 @@
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="paddle">
<span id="paddle"></span><h1>Paddle发行规范<a class="headerlink" href="#paddle" title="Permalink to this headline"></a></h1>
<p>Paddle使用git-flow branching model做分支管理，使用<a class="reference external" href="http://semver.org/">Semantic Versioning</a>标准表示Paddle版本号。</p>
<p>Paddle每次发新的版本，遵循以下流程:</p>
<div class="section" id="paddlepaddle">
<span id="paddlepaddle"></span><h1>PaddlePaddle发行规范<a class="headerlink" href="#paddlepaddle" title="Permalink to this headline"></a></h1>
<p>PaddlePaddle使用git-flow branching model做分支管理，使用<a class="reference external" href="http://semver.org/">Semantic Versioning</a>标准表示PaddlePaddle版本号。</p>
<p>PaddlePaddle每次发新的版本，遵循以下流程:</p>
<ol>
<li><p class="first">从<code class="docutils literal"><span class="pre">develop</span></code>分支派生出新的分支，分支名为<code class="docutils literal"><span class="pre">release/版本号</span></code>。例如，<code class="docutils literal"><span class="pre">release/0.10.0</span></code></p>
</li>
......@@ -223,15 +223,15 @@
</ol>
<p>需要注意的是:</p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">release/版本号</span></code>分支一旦建立，一般不允许再从<code class="docutils literal"><span class="pre">develop</span></code>分支合入<code class="docutils literal"><span class="pre">release/版本号</span></code>。这样保证<code class="docutils literal"><span class="pre">release/版本号</span></code>分支功能的封闭，方便测试人员测试Paddle的行为。</li>
<li><code class="docutils literal"><span class="pre">release/版本号</span></code>分支一旦建立，一般不允许再从<code class="docutils literal"><span class="pre">develop</span></code>分支合入<code class="docutils literal"><span class="pre">release/版本号</span></code>。这样保证<code class="docutils literal"><span class="pre">release/版本号</span></code>分支功能的封闭，方便测试人员测试PaddlePaddle的行为。</li>
<li>在<code class="docutils literal"><span class="pre">release/版本号</span></code>分支存在的时候，如果有bugfix的行为，需要将bugfix的分支同时merge到<code class="docutils literal"><span class="pre">master</span></code>, <code class="docutils literal"><span class="pre">develop</span></code>和<code class="docutils literal"><span class="pre">release/版本号</span></code>这三个分支。</li>
</ul>
</div>
<div class="section" id="paddle">
<span id="id1"></span><h1>Paddle 分支规范<a class="headerlink" href="#paddle" title="Permalink to this headline"></a></h1>
<p>Paddle开发过程使用<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范，并适应github的特性做了一些区别。</p>
<div class="section" id="paddlepaddle">
<span id="id1"></span><h1>PaddlePaddle 分支规范<a class="headerlink" href="#paddlepaddle" title="Permalink to this headline"></a></h1>
<p>PaddlePaddle开发过程使用<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范，并适应github的特性做了一些区别。</p>
<ul class="simple">
<li>Paddle的主版本库遵循<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范。其中:<ul>
<li>PaddlePaddle的主版本库遵循<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范。其中:<ul>
<li><code class="docutils literal"><span class="pre">master</span></code>分支为稳定(stable branch)版本分支。每一个<code class="docutils literal"><span class="pre">master</span></code>分支的版本都是经过单元测试和回归测试的版本。</li>
<li><code class="docutils literal"><span class="pre">develop</span></code>分支为开发(develop branch)版本分支。每一个<code class="docutils literal"><span class="pre">develop</span></code>分支的版本都经过单元测试，但并没有经过回归测试。</li>
<li><code class="docutils literal"><span class="pre">release/版本号</span></code>分支为每一次Release时建立的临时分支。在这个阶段的代码正在经历回归测试。</li>
......@@ -240,7 +240,7 @@
<li>其他用户的fork版本库并不需要严格遵守<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范，但所有fork的版本库的所有分支都相当于特性分支。<ul>
<li>建议，开发者fork的版本库使用<code class="docutils literal"><span class="pre">develop</span></code>分支同步主版本库的<code class="docutils literal"><span class="pre">develop</span></code>分支</li>
<li>建议，开发者fork的版本库中，再基于<code class="docutils literal"><span class="pre">develop</span></code>版本fork出自己的功能分支。</li>
<li>当功能分支开发完毕后，向Paddle的主版本库提交<code class="docutils literal"><span class="pre">Pull</span> <span class="pre">Reuqest</span></code>，进而进行代码评审。<ul>
<li>当功能分支开发完毕后，向PaddlePaddle的主版本库提交<code class="docutils literal"><span class="pre">Pull</span> <span class="pre">Reuqest</span></code>，进而进行代码评审。<ul>
<li>在评审过程中，开发者修改自己的代码，可以继续在自己的功能分支提交代码。</li>
</ul>
</li>
......@@ -249,12 +249,12 @@
<li>BugFix分支也是在开发者自己的fork版本库维护，与功能分支不同的是，BugFix分支需要分别给主版本库的<code class="docutils literal"><span class="pre">master</span></code>、<code class="docutils literal"><span class="pre">develop</span></code>与可能有的<code class="docutils literal"><span class="pre">release/版本号</span></code>分支，同时提起<code class="docutils literal"><span class="pre">Pull</span> <span class="pre">Request</span></code>。</li>
</ul>
</div>
<div class="section" id="paddle">
<span id="id2"></span><h1>Paddle回归测试列表<a class="headerlink" href="#paddle" title="Permalink to this headline"></a></h1>
<p>本列表说明Paddle发版之前需要测试的功能点。</p>
<div class="section" id="paddle-book">
<span id="paddle-book"></span><h2>Paddle Book中所有章节<a class="headerlink" href="#paddle-book" title="Permalink to this headline"></a></h2>
<p>Paddle每次发版本首先要保证Paddle Book中所有章节功能的正确性。功能的正确性包括验证Paddle目前的<code class="docutils literal"><span class="pre">paddle_trainer</span></code>训练和纯使用<code class="docutils literal"><span class="pre">Python</span></code>训练模型正确性。</p>
<div class="section" id="paddlepaddle">
<span id="id2"></span><h1>PaddlePaddle回归测试列表<a class="headerlink" href="#paddlepaddle" title="Permalink to this headline"></a></h1>
<p>本列表说明PaddlePaddle发版之前需要测试的功能点。</p>
<div class="section" id="paddlepaddle-book">
<span id="paddlepaddle-book"></span><h2>PaddlePaddle Book中所有章节<a class="headerlink" href="#paddlepaddle-book" title="Permalink to this headline"></a></h2>
<p>PaddlePaddle每次发版本首先要保证PaddlePaddle Book中所有章节功能的正确性。功能的正确性包括验证PaddlePaddle目前的<code class="docutils literal"><span class="pre">paddle_trainer</span></code>训练和纯使用<code class="docutils literal"><span class="pre">Python</span></code>训练模型正确性。</p>
<p>| | 新手入门章节 | 识别数字 | 图像分类 | 词向量 | 情感分析 | 语意角色标注 | 机器翻译 | 个性化推荐 |
| &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; |
| API.V2 + Docker + GPU | | | | | | | | |
......
develop/doc/design/scope.html
浏览文件 @ 477e97f1
......@@ -193,7 +193,7 @@
<p>Scope is an association of a name to variable. All variables belong to <code class="docutils literal"><span class="pre">Scope</span></code>. You need to specify a scope to run a Net, i.e., <code class="docutils literal"><span class="pre">net.Run(&amp;scope)</span></code>. One net can run in different scopes and update different variable in the scope.</p>
<ol>
<li><p class="first">Scope only contains a map of a name to variable.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state (momentum) etc.</p>
</li>
<li><p class="first">Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.</p>
</li>
......@@ -208,7 +208,7 @@
<p>Variable can not belong to many scopes. If you want to use variables from parent scope, you can use <code class="docutils literal"><span class="pre">parent</span> <span class="pre">scope</span></code>.</p>
</li>
<li><p class="first">Scope should destruct all Variables inside it when itself is destructed. User can never store <code class="docutils literal"><span class="pre">Variable</span></code> pointer somewhere else.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <code class="docutils literal"><span class="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be a invalid pointer when associated <code class="docutils literal"><span class="pre">Scope</span></code> is destroyed.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <code class="docutils literal"><span class="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be an invalid pointer when associated <code class="docutils literal"><span class="pre">Scope</span></code> is destroyed.</p>
</li>
</ol>
<div class="highlight-cpp"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Scope</span> <span class="p">{</span>
......@@ -226,7 +226,7 @@
<span id="parent-scope-and-local-scope"></span><h2>Parent scope and local scope<a class="headerlink" href="#parent-scope-and-local-scope" title="Permalink to this headline"></a></h2>
<p>Just like <a class="reference external" href="https://en.wikipedia.org/wiki/Scope_(computer_science)">scope</a> in programming languages, <code class="docutils literal"><span class="pre">Scope</span></code> in the neural network can also be a local scope. There are two attributes about local scope.</p>
<ol class="simple">
<li>We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.</li>
<li>We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.</li>
<li>Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, <code class="docutils literal"><span class="pre">scope</span></code> will keep searching from its parent, until the variable is found or there is no parent.</li>
</ol>
<div class="highlight-cpp"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Scope</span> <span class="p">{</span>
......@@ -295,7 +295,7 @@
</div>
<div class="section" id="orthogonal-interface">
<span id="orthogonal-interface"></span><h2>Orthogonal interface<a class="headerlink" href="#orthogonal-interface" title="Permalink to this headline"></a></h2>
<p><code class="docutils literal"><span class="pre">FindVar</span></code> will return <code class="docutils literal"><span class="pre">nullptr</span></code> when <code class="docutils literal"><span class="pre">name</span></code> is not found. It can be used as <code class="docutils literal"><span class="pre">Contains</span></code> method. <code class="docutils literal"><span class="pre">NewVar</span></code> will return a <code class="docutils literal"><span class="pre">Error</span></code> when there is a name conflict locally. Combine <code class="docutils literal"><span class="pre">FindVar</span></code> and <code class="docutils literal"><span class="pre">NewVar</span></code>, we can implement <code class="docutils literal"><span class="pre">NewVar</span></code> easily.</p>
<p><code class="docutils literal"><span class="pre">FindVar</span></code> will return <code class="docutils literal"><span class="pre">nullptr</span></code> when <code class="docutils literal"><span class="pre">name</span></code> is not found. It can be used as <code class="docutils literal"><span class="pre">Contains</span></code> method. <code class="docutils literal"><span class="pre">NewVar</span></code> will return an <code class="docutils literal"><span class="pre">Error</span></code> when there is a name conflict locally. Combine <code class="docutils literal"><span class="pre">FindVar</span></code> and <code class="docutils literal"><span class="pre">NewVar</span></code>, we can implement <code class="docutils literal"><span class="pre">NewVar</span></code> easily.</p>
</div>
</div>
......
develop/doc/design/simple_op_design.html
浏览文件 @ 477e97f1
......@@ -183,10 +183,10 @@
<p>The Interaction between Python and C++ can be simplified as two steps:</p>
<ol class="simple">
<li>C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.</li>
</ol>
<div class="section" id="message-form-c-to-python">
<span id="message-form-c-to-python"></span><h2>Message form C++ to Python<a class="headerlink" href="#message-form-c-to-python" title="Permalink to this headline"></a></h2>
<div class="section" id="message-from-c-to-python">
<span id="message-from-c-to-python"></span><h2>Message from C++ to Python<a class="headerlink" href="#message-from-c-to-python" title="Permalink to this headline"></a></h2>
<p>We define a Protobuf message class <code class="docutils literal"><span class="pre">OpProto</span></code> to hold message needed in the first step. What should an <code class="docutils literal"><span class="pre">OpProto</span></code> contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”</p>
<p>Following message are necessary:</p>
<ol class="simple">
......
develop/doc/design/var_desc.html
浏览文件 @ 477e97f1
......@@ -180,7 +180,7 @@
<div class="section" id="background">
<span id="background"></span><h1>Background<a class="headerlink" href="#background" title="Permalink to this headline"></a></h1>
<p>PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.</p>
<p>PaddlePaddle use proto message to describe compile time graph for</p>
<p>PaddlePaddle use proto message to describe compile time graph because</p>
<ol class="simple">
<li>Computation graph should be able to be saved to a file.</li>
<li>In distributed training, the graph will be serialized and send to multiple workers.</li>
......
develop/doc/searchindex.js
浏览文件 @ 477e97f1
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc_cn/_sources/design/api.md.txt
浏览文件 @ 477e97f1
......@@ -3,7 +3,7 @@
## Ingredients
As our design principle is starting from the essence: how could we
allow users to express and solve their problems at neural networks.
allow users to express and solve their problems as neural networks.
Some essential concepts that our API have to provide include:
1. A *topology* is an expression of *layers*.
......@@ -233,7 +233,7 @@ paddle.dist_train(model,
num_parameter_servers=15)
```
The pseudo code if `paddle.dist_train` is as follows:
The pseudo code of `paddle.dist_train` is as follows:
```python
def dist_train(topology, parameters, trainer, reader, ...):
......
develop/doc_cn/_sources/design/auto_gradient_check.md.txt
浏览文件 @ 477e97f1
## Auto Gradient Checker Design
## Backgraound：
- Operator forward computing is easy to check if the result is right because it has a clear definition. **But** backpropagation is a notoriously difficult algorithm to debug and get right:
- 1. you should get the right backpropagation formula according to the forward computation.
- 2. you should implement it right in CPP.
- 3. it's difficult to prepare test data.
- Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:
1. you should get the right backpropagation formula according to the forward computation.
2. you should implement it right in CPP.
3. it's difficult to prepare test data.
- Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
- 1. numeric gradient checker only need forward operator.
- 2. user only need to prepare the input data for forward Operator.
- Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
1. numerical gradient checker only need forward operator.
2. user only need to prepare the input data for forward Operator.
## Mathematical Theory
The following two document from stanford has a detailed explanation of how to get numeric gradient and why it's useful.
The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it's useful.
- [Gradient checking and advanced optimization(en)](http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization)
- [Gradient checking and advanced optimization(cn)](http://ufldl.stanford.edu/wiki/index.php/%E6%A2%AF%E5%BA%A6%E6%A3%80%E9%AA%8C%E4%B8%8E%E9%AB%98%E7%BA%A7%E4%BC%98%E5%8C%96)
......@@ -20,7 +20,7 @@ The following two document from stanford has a detailed explanation of how to ge
## Numeric Gradient Implementation
### Python Interface
```python
def get_numeric_gradient(op,
def get_numerical_gradient(op,
input_values,
output_name,
input_to_check,
......@@ -30,13 +30,13 @@ def get_numeric_gradient(op,
Get Numeric Gradient for an operator's input.
:param op: C++ operator instance, could be an network
:param input_values: The input variables. Should be an dictionary, key is
variable name. Value is numpy array.
:param input_values: The input variables. Should be an dictionary, whose key is
variable name, and value is numpy array.
:param output_name: The final output variable name.
:param input_to_check: The input variable need to get gradient.
:param input_to_check: The input variable with respect to which to compute the gradient.
:param delta: The perturbation value for numeric gradient method. The
smaller delta is, the more accurate result will get. But if that delta is
too small, it could occur numerical stability problem.
too small, it will suffer from numerical stability problem.
:param local_scope: The local scope used for get_numeric_gradient.
:return: The gradient array in numpy format.
"""
......@@ -45,28 +45,28 @@ def get_numeric_gradient(op,
### Explaination:
- Why need `output_name`
- One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.
- An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.
- Why need `input_to_check`
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
### Core Algorithm Implementation
```python
# we only compute gradient of one element each time.
# we use a for loop to compute the gradient of every element.
# we only compute gradient of one element a time.
# we use a for loop to compute the gradient of each element.
for i in xrange(tensor_size):
# get one input element throw it's index i.
# get one input element by its index i.
origin = tensor_to_check.get_float_element(i)
# add delta to it, run op and then get the sum of the result tensor.
# add delta to it, run op and then get the new value of the result tensor.
x_pos = origin + delta
tensor_to_check.set_float_element(i, x_pos)
y_pos = get_output()
# plus delta to this element, run op and get the sum of the result tensor.
# plus delta to this element, run op and get the new value of the result tensor.
x_neg = origin - delta
tensor_to_check.set_float_element(i, x_neg)
y_neg = get_output()
......@@ -85,15 +85,15 @@ def get_numeric_gradient(op,
Each Operator Kernel has three kinds of Gradient:
- 1. Numeric Gradient
- 2. CPU Operator Gradient
- 3. GPU Operator Gradient(if supported)
1. Numerical gradient
2. CPU kernel gradient
3. GPU kernel gradient (if supported)
Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.
The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:
- 1. calculate the numeric gradient.
- 2. calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.
- 3. calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)
1. calculate the numerical gradient
2. calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient
3. calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)
#### Python Interface
......@@ -110,8 +110,8 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
:param forward_op: used to create backward_op
:param input_vars: numpy value of input variable. The following
computation will use these variables.
:param inputs_to_check: inputs var names that should check gradient.
:param output_name: output name that used to
:param inputs_to_check: the input variable with respect to which to compute the gradient.
:param output_name: The final output variable name.
:param max_relative_error: The relative tolerance parameter.
:param no_grad_set: used when create backward ops
:param only_cpu: only compute and check gradient on cpu kernel.
......@@ -120,24 +120,24 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
```
### How to check if two numpy array is close enough?
if `abs_numeric_grad` is nearly zero, then use abs error for numeric_grad, not relative
if `abs_numerical_grad` is nearly zero, then use abs error for numerical_grad
```python
numeric_grad = ...
numerical_grad = ...
operator_grad = numpy.array(scope.find_var(grad_var_name(name)).get_tensor())
abs_numeric_grad = numpy.abs(numeric_grad)
# if abs_numeric_grad is nearly zero, then use abs error for numeric_grad, not relative
abs_numerical_grad = numpy.abs(numerical_grad)
# if abs_numerical_grad is nearly zero, then use abs error for numeric_grad, not relative
# error.
abs_numeric_grad[abs_numeric_grad < 1e-3] = 1
abs_numerical_grad[abs_numerical_grad < 1e-3] = 1
diff_mat = numpy.abs(abs_numeric_grad - operator_grad) / abs_numeric_grad
diff_mat = numpy.abs(abs_numerical_grad - operator_grad) / abs_numerical_grad
max_diff = numpy.max(diff_mat)
```
#### Notes：
1，The Input data for auto gradient checker should be reasonable to avoid numeric problem.
The Input data for auto gradient checker should be reasonable to avoid numerical stability problem.
#### Refs:
......
develop/doc_cn/_sources/design/functions_operators_layers.md.txt
浏览文件 @ 477e97f1
......@@ -53,12 +53,12 @@ Let's explain using an example. Suppose that we are going to compose the FC usi
```python
def operator.mul(X1, X2):
O = Var()
paddle.cpp.create_operator("mul", input={X1, Y1], output=O)
paddle.cpp.create_operator("mul", input={X1, Y1}, output=O)
return O
def operator.add(X1, X2):
O = Var()
paddle.cpp.create_operator("add", input={X1, X2], output=O)
paddle.cpp.create_operator("add", input={X1, X2}, output=O)
return O
```
......
develop/doc_cn/_sources/design/graph.md.txt
浏览文件 @ 477e97f1
......@@ -56,7 +56,7 @@ For each parameter, like W and b created by `layer.fc`, marked as double circles
## Block and Graph
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block](https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
A Block keeps operators in an array `BlockDesc::ops`
......@@ -67,4 +67,4 @@ message BlockDesc {
}
```
in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
develop/doc_cn/_sources/design/parameters_in_cpp.md.txt
浏览文件 @ 477e97f1
# Design Doc: The C++ Class `Parameters`
`Parameters` is a concept we designed in Paddle V2 API. `Parameters` is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
`Parameters` is a concept we designed in PaddlePaddle V2 API. `Parameters` is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:
We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:
* We just use `memcpy` to share Parameters between topologies, but this is very inefficient.
* We did not implement share Parameters while training. We just trigger `memcpy` when start training.
* We did not support sharing Parameters while training. We just trigger `memcpy` when start training.
It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with `Parameters`:
It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with `Parameters`:
1. `paddle::Parameter`. A `Parameters` is a container for `paddle::Parameter`.
It is evident that we should use `paddle::Parameter` when developing `Parameters`.
However, the `Parameter` class contains many functions and does not have a clear interface.
It contains `create/store Parameter`, `serialize/deserialize`, `optimize(i.e SGD)`, `randomize/zero`.
When we developing `Parameters`, we only use `create/store Parameter` functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.
2. `paddle::GradientMachine` and its sub-classes, e.g., `paddle::MultiGradientMachine`, `paddle::NeuralNetwork`.
We should pass `Parameters` to `paddle::GradientMachine` when `forward/backward` to avoid `memcpy` between topologies.
......@@ -24,7 +24,7 @@ Also, we should handle multi-GPU/CPU training, because `forward` and `backward`
So `Parameters` should be used by `paddle::ParameterUpdater`, and `paddle::ParameterUpdater` should optimize `Parameters` (by SGD).
The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.
The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.
1. Clean `paddle::Parameter` interface. Extract the functionalities of `paddle::Parameter` to prepare for the implementation of Parameters.
......
develop/doc_cn/_sources/design/reader/README.md.txt
浏览文件 @ 477e97f1
......@@ -52,7 +52,7 @@ Here are valid outputs:
# a mini batch of three data items, each data item is a list (single column).
[([1,1,1],),
([2,2,2],),
([3,3,3],),
([3,3,3],)]
```
Please note that each item inside the list must be a tuple, below is an invalid output:
......
develop/doc_cn/_sources/design/refactorization.md.txt
浏览文件 @ 477e97f1
......@@ -15,7 +15,7 @@ The goal of refactorizaiton include:
1. Users write Python programs to describe the graphs and run it (locally or remotely).
1. A graph is composed of *variabels* and *operators*.
1. A graph is composed of *variables* and *operators*.
1. The description of graphs must be able to be serialized/deserialized, so it
......@@ -140,7 +140,7 @@ Compile Time -> IR -> Runtime
* `thrust` has the same API as C++ standard library. Using `transform` can quickly implement a customized elementwise kernel.
* `thrust` has more complex API, like `scan`, `reduce`, `reduce_by_key`.
* Hand-writing `GPUKernel` and `CPU` code
* Do not write `.h`. CPU Kernel should be in `.cc`. CPU kernel should be in `.cu`. (`GCC` cannot compile GPU code.)
* Do not write `.h`. CPU Kernel should be in `.cc`. GPU kernel should be in `.cu`. (`GCC` cannot compile GPU code.)
---
# Operator Register
......
develop/doc_cn/_sources/design/releasing_process.md.txt
浏览文件 @ 477e97f1
# Paddle发行规范
# PaddlePaddle发行规范
Paddle使用git-flow branching model做分支管理，使用[Semantic Versioning](http://semver.org/)标准表示Paddle版本号。
PaddlePaddle使用git-flow branching model做分支管理，使用[Semantic Versioning](http://semver.org/)标准表示PaddlePaddle版本号。
Paddle每次发新的版本，遵循以下流程:
PaddlePaddle每次发新的版本，遵循以下流程:
1. 从`develop`分支派生出新的分支，分支名为`release/版本号`。例如，`release/0.10.0`
2. 将新分支的版本打上tag，tag为`版本号rc.Patch号`。第一个tag为`0.10.0rc1`，第二个为`0.10.0rc2`，依次类推。
......@@ -27,14 +27,14 @@ Paddle每次发新的版本，遵循以下流程:
需要注意的是:
* `release/版本号`分支一旦建立，一般不允许再从`develop`分支合入`release/版本号`。这样保证`release/版本号`分支功能的封闭，方便测试人员测试Paddle的行为。
* `release/版本号`分支一旦建立，一般不允许再从`develop`分支合入`release/版本号`。这样保证`release/版本号`分支功能的封闭，方便测试人员测试PaddlePaddle的行为。
* 在`release/版本号`分支存在的时候，如果有bugfix的行为，需要将bugfix的分支同时merge到`master`, `develop`和`release/版本号`这三个分支。
# Paddle 分支规范
# PaddlePaddle 分支规范
Paddle开发过程使用[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范，并适应github的特性做了一些区别。
PaddlePaddle开发过程使用[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范，并适应github的特性做了一些区别。
* Paddle的主版本库遵循[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范。其中:
* PaddlePaddle的主版本库遵循[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范。其中:
* `master`分支为稳定(stable branch)版本分支。每一个`master`分支的版本都是经过单元测试和回归测试的版本。
* `develop`分支为开发(develop branch)版本分支。每一个`develop`分支的版本都经过单元测试，但并没有经过回归测试。
* `release/版本号`分支为每一次Release时建立的临时分支。在这个阶段的代码正在经历回归测试。
......@@ -42,18 +42,18 @@ Paddle开发过程使用[git-flow](http://nvie.com/posts/a-successful-git-branch
* 其他用户的fork版本库并不需要严格遵守[git-flow](http://nvie.com/posts/a-successful-git-branching-model/)分支规范，但所有fork的版本库的所有分支都相当于特性分支。
* 建议，开发者fork的版本库使用`develop`分支同步主版本库的`develop`分支
* 建议，开发者fork的版本库中，再基于`develop`版本fork出自己的功能分支。
* 当功能分支开发完毕后，向Paddle的主版本库提交`Pull Reuqest`，进而进行代码评审。
* 当功能分支开发完毕后，向PaddlePaddle的主版本库提交`Pull Reuqest`，进而进行代码评审。
* 在评审过程中，开发者修改自己的代码，可以继续在自己的功能分支提交代码。
* BugFix分支也是在开发者自己的fork版本库维护，与功能分支不同的是，BugFix分支需要分别给主版本库的`master`、`develop`与可能有的`release/版本号`分支，同时提起`Pull Request`。
# Paddle回归测试列表
# PaddlePaddle回归测试列表
本列表说明Paddle发版之前需要测试的功能点。
本列表说明PaddlePaddle发版之前需要测试的功能点。
## Paddle Book中所有章节
## PaddlePaddle Book中所有章节
Paddle每次发版本首先要保证Paddle Book中所有章节功能的正确性。功能的正确性包括验证Paddle目前的`paddle_trainer`训练和纯使用`Python`训练模型正确性。
PaddlePaddle每次发版本首先要保证PaddlePaddle Book中所有章节功能的正确性。功能的正确性包括验证PaddlePaddle目前的`paddle_trainer`训练和纯使用`Python`训练模型正确性。
| | 新手入门章节 | 识别数字 | 图像分类 | 词向量 | 情感分析 | 语意角色标注 | 机器翻译 | 个性化推荐 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
......
develop/doc_cn/_sources/design/scope.md.txt
浏览文件 @ 477e97f1
......@@ -17,7 +17,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope only contains a map of a name to variable.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.
All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state (momentum) etc.
1. Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.
......@@ -32,7 +32,7 @@ Scope is an association of a name to variable. All variables belong to `Scope`.
1. Scope should destruct all Variables inside it when itself is destructed. User can never store `Variable` pointer somewhere else.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be a invalid pointer when associated `Scope` is destroyed.
Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store `Variable` pointer to private data member or some global variable, the pointer will be an invalid pointer when associated `Scope` is destroyed.
```cpp
class Scope {
......@@ -50,7 +50,7 @@ class Scope {
Just like [scope](https://en.wikipedia.org/wiki/Scope_(computer_science)) in programming languages, `Scope` in the neural network can also be a local scope. There are two attributes about local scope.
1. We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.
1. We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.
2. Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, `scope` will keep searching from its parent, until the variable is found or there is no parent.
```cpp
......@@ -121,4 +121,4 @@ Also, as the parent scope is a `shared_ptr`, we can only `Create()` a scope shar
## Orthogonal interface
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return a `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
`FindVar` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `NewVar` will return an `Error` when there is a name conflict locally. Combine `FindVar` and `NewVar`, we can implement `NewVar` easily.
develop/doc_cn/_sources/design/simple_op_design.md.txt
浏览文件 @ 477e97f1
......@@ -6,9 +6,9 @@ The Interaction between Python and C++ can be simplified as two steps:
1. C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.
2. Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.
### Message form C++ to Python
### Message from C++ to Python
We define a Protobuf message class `OpProto` to hold message needed in the first step. What should an `OpProto` contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”
......
develop/doc_cn/_sources/design/var_desc.md.txt
浏览文件 @ 477e97f1
## Background
PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.
PaddlePaddle use proto message to describe compile time graph for
PaddlePaddle use proto message to describe compile time graph because
1. Computation graph should be able to be saved to a file.
1. In distributed training, the graph will be serialized and send to multiple workers.
......
develop/doc_cn/design/api.html
浏览文件 @ 477e97f1
......@@ -189,7 +189,7 @@
<div class="section" id="ingredients">
<span id="ingredients"></span><h2>Ingredients<a class="headerlink" href="#ingredients" title="永久链接至标题"></a></h2>
<p>As our design principle is starting from the essence: how could we
allow users to express and solve their problems at neural networks.
allow users to express and solve their problems as neural networks.
Some essential concepts that our API have to provide include:</p>
<ol class="simple">
<li>A <em>topology</em> is an expression of <em>layers</em>.</li>
......@@ -392,7 +392,7 @@ access a Kubernetes cluster, s/he should be able to call</p>
<span class="n">num_parameter_servers</span><span class="o">=</span><span class="mi">15</span><span class="p">)</span>
</pre></div>
</div>
<p>The pseudo code if <code class="docutils literal"><span class="pre">paddle.dist_train</span></code> is as follows:</p>
<p>The pseudo code of <code class="docutils literal"><span class="pre">paddle.dist_train</span></code> is as follows:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">dist_train</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">trainer</span><span class="p">,</span> <span class="n">reader</span><span class="p">,</span> <span class="o">...</span><span class="p">):</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;KUBERNETES_SERVICE_HOST&quot;</span><span class="p">)</span> <span class="o">==</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">image_name</span> <span class="o">=</span> <span class="n">k8s_user</span> <span class="o">+</span> <span class="s1">&#39;/&#39;</span> <span class="o">+</span> <span class="n">k8s_job</span>
......
develop/doc_cn/design/auto_gradient_check.html
浏览文件 @ 477e97f1
......@@ -190,37 +190,22 @@
<div class="section" id="backgraound">
<span id="backgraound"></span><h1>Backgraound：<a class="headerlink" href="#backgraound" title="永久链接至标题"></a></h1>
<ul class="simple">
<li>Operator forward computing is easy to check if the result is right because it has a clear definition. <strong>But</strong> backpropagation is a notoriously difficult algorithm to debug and get right:<ul>
<li><ol class="first">
<li>Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:<ol>
<li>you should get the right backpropagation formula according to the forward computation.</li>
</ol>
</li>
<li><ol class="first">
<li>you should implement it right in CPP.</li>
</ol>
</li>
<li><ol class="first">
<li>it&#8217;s difficult to prepare test data.</li>
</ol>
</li>
</ul>
</li>
<li>Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator&#8217;s result. It has several advantages:<ul>
<li><ol class="first">
<li>numeric gradient checker only need forward operator.</li>
</ol>
</li>
<li><ol class="first">
<li>Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator&#8217;s result. It has several advantages:<ol>
<li>numerical gradient checker only need forward operator.</li>
<li>user only need to prepare the input data for forward Operator.</li>
</ol>
</li>
</ul>
</li>
</ul>
</div>
<div class="section" id="mathematical-theory">
<span id="mathematical-theory"></span><h1>Mathematical Theory<a class="headerlink" href="#mathematical-theory" title="永久链接至标题"></a></h1>
<p>The following two document from stanford has a detailed explanation of how to get numeric gradient and why it&#8217;s useful.</p>
<p>The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it&#8217;s useful.</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference external" href="http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization">Gradient checking and advanced optimization(en)</a></li>
......@@ -232,7 +217,7 @@
<span id="numeric-gradient-implementation"></span><h1>Numeric Gradient Implementation<a class="headerlink" href="#numeric-gradient-implementation" title="永久链接至标题"></a></h1>
<div class="section" id="python-interface">
<span id="python-interface"></span><h2>Python Interface<a class="headerlink" href="#python-interface" title="永久链接至标题"></a></h2>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_numeric_gradient</span><span class="p">(</span><span class="n">op</span><span class="p">,</span>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_numerical_gradient</span><span class="p">(</span><span class="n">op</span><span class="p">,</span>
<span class="n">input_values</span><span class="p">,</span>
<span class="n">output_name</span><span class="p">,</span>
<span class="n">input_to_check</span><span class="p">,</span>
......@@ -242,13 +227,13 @@
<span class="sd"> Get Numeric Gradient for an operator&#39;s input.</span>
<span class="sd"> :param op: C++ operator instance, could be an network</span>
<span class="sd"> :param input_values: The input variables. Should be an dictionary, key is</span>
<span class="sd"> variable name. Value is numpy array.</span>
<span class="sd"> :param input_values: The input variables. Should be an dictionary, whose key is</span>
<span class="sd"> variable name, and value is numpy array.</span>
<span class="sd"> :param output_name: The final output variable name.</span>
<span class="sd"> :param input_to_check: The input variable need to get gradient.</span>
<span class="sd"> :param input_to_check: The input variable with respect to which to compute the gradient.</span>
<span class="sd"> :param delta: The perturbation value for numeric gradient method. The</span>
<span class="sd"> smaller delta is, the more accurate result will get. But if that delta is</span>
<span class="sd"> too small, it could occur numerical stability problem.</span>
<span class="sd"> too small, it will suffer from numerical stability problem.</span>
<span class="sd"> :param local_scope: The local scope used for get_numeric_gradient.</span>
<span class="sd"> :return: The gradient array in numpy format.</span>
<span class="sd"> &quot;&quot;&quot;</span>
......@@ -259,29 +244,29 @@
<span id="explaination"></span><h2>Explaination:<a class="headerlink" href="#explaination" title="永久链接至标题"></a></h2>
<ul class="simple">
<li>Why need <code class="docutils literal"><span class="pre">output_name</span></code><ul>
<li>One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.</li>
<li>An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.</li>
</ul>
</li>
<li>Why need <code class="docutils literal"><span class="pre">input_to_check</span></code><ul>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> multiple times.</li>
<li>One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call <code class="docutils literal"><span class="pre">get_numeric_gradient</span></code> multiple times.</li>
</ul>
</li>
</ul>
</div>
<div class="section" id="core-algorithm-implementation">
<span id="core-algorithm-implementation"></span><h2>Core Algorithm Implementation<a class="headerlink" href="#core-algorithm-implementation" title="永久链接至标题"></a></h2>
<div class="highlight-python"><div class="highlight"><pre><span></span> <span class="c1"># we only compute gradient of one element each time.</span>
<span class="c1"># we use a for loop to compute the gradient of every element.</span>
<div class="highlight-python"><div class="highlight"><pre><span></span> <span class="c1"># we only compute gradient of one element a time.</span>
<span class="c1"># we use a for loop to compute the gradient of each element.</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">tensor_size</span><span class="p">):</span>
<span class="c1"># get one input element throw it&#39;s index i.</span>
<span class="c1"># get one input element by its index i.</span>
<span class="n">origin</span> <span class="o">=</span> <span class="n">tensor_to_check</span><span class="o">.</span><span class="n">get_float_element</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="c1"># add delta to it, run op and then get the sum of the result tensor.</span>
<span class="c1"># add delta to it, run op and then get the new value of the result tensor.</span>
<span class="n">x_pos</span> <span class="o">=</span> <span class="n">origin</span> <span class="o">+</span> <span class="n">delta</span>
<span class="n">tensor_to_check</span><span class="o">.</span><span class="n">set_float_element</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">x_pos</span><span class="p">)</span>
<span class="n">y_pos</span> <span class="o">=</span> <span class="n">get_output</span><span class="p">()</span>
<span class="c1"># plus delta to this element, run op and get the sum of the result tensor.</span>
<span class="c1"># plus delta to this element, run op and get the new value of the result tensor.</span>
<span class="n">x_neg</span> <span class="o">=</span> <span class="n">origin</span> <span class="o">-</span> <span class="n">delta</span>
<span class="n">tensor_to_check</span><span class="o">.</span><span class="n">set_float_element</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">x_neg</span><span class="p">)</span>
<span class="n">y_neg</span> <span class="o">=</span> <span class="n">get_output</span><span class="p">()</span>
......@@ -301,35 +286,17 @@
<div class="section" id="auto-graident-checker-framework">
<span id="auto-graident-checker-framework"></span><h1>Auto Graident Checker Framework<a class="headerlink" href="#auto-graident-checker-framework" title="永久链接至标题"></a></h1>
<p>Each Operator Kernel has three kinds of Gradient:</p>
<ul class="simple">
<li><ol class="first">
<li>Numeric Gradient</li>
<ol class="simple">
<li>Numerical gradient</li>
<li>CPU kernel gradient</li>
<li>GPU kernel gradient (if supported)</li>
</ol>
</li>
<li><ol class="first">
<li>CPU Operator Gradient</li>
<p>The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:</p>
<ol class="simple">
<li>calculate the numerical gradient</li>
<li>calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient</li>
<li>calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient (if supported)</li>
</ol>
</li>
<li><ol class="first">
<li>GPU Operator Gradient(if supported)</li>
</ol>
</li>
</ul>
<p>Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.</p>
<ul class="simple">
<li><ol class="first">
<li>calculate the numeric gradient.</li>
</ol>
</li>
<li><ol class="first">
<li>calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.</li>
</ol>
</li>
<li><ol class="first">
<li>calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)</li>
</ol>
</li>
</ul>
<div class="section" id="python-interface">
<span id="id1"></span><h2>Python Interface<a class="headerlink" href="#python-interface" title="永久链接至标题"></a></h2>
<div class="highlight-python"><div class="highlight"><pre><span></span> <span class="k">def</span> <span class="nf">check_grad</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
......@@ -344,8 +311,8 @@
<span class="sd"> :param forward_op: used to create backward_op</span>
<span class="sd"> :param input_vars: numpy value of input variable. The following</span>
<span class="sd"> computation will use these variables.</span>
<span class="sd"> :param inputs_to_check: inputs var names that should check gradient.</span>
<span class="sd"> :param output_name: output name that used to</span>
<span class="sd"> :param inputs_to_check: the input variable with respect to which to compute the gradient.</span>
<span class="sd"> :param output_name: The final output variable name.</span>
<span class="sd"> :param max_relative_error: The relative tolerance parameter.</span>
<span class="sd"> :param no_grad_set: used when create backward ops</span>
<span class="sd"> :param only_cpu: only compute and check gradient on cpu kernel.</span>
......@@ -356,22 +323,22 @@
</div>
<div class="section" id="how-to-check-if-two-numpy-array-is-close-enough">
<span id="how-to-check-if-two-numpy-array-is-close-enough"></span><h2>How to check if two numpy array is close enough?<a class="headerlink" href="#how-to-check-if-two-numpy-array-is-close-enough" title="永久链接至标题"></a></h2>
<p>if <code class="docutils literal"><span class="pre">abs_numeric_grad</span></code> is nearly zero, then use abs error for numeric_grad, not relative</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">numeric_grad</span> <span class="o">=</span> <span class="o">...</span>
<p>if <code class="docutils literal"><span class="pre">abs_numerical_grad</span></code> is nearly zero, then use abs error for numerical_grad</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">numerical_grad</span> <span class="o">=</span> <span class="o">...</span>
<span class="n">operator_grad</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">find_var</span><span class="p">(</span><span class="n">grad_var_name</span><span class="p">(</span><span class="n">name</span><span class="p">))</span><span class="o">.</span><span class="n">get_tensor</span><span class="p">())</span>
<span class="n">abs_numeric_grad</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">numeric_grad</span><span class="p">)</span>
<span class="c1"># if abs_numeric_grad is nearly zero, then use abs error for numeric_grad, not relative</span>
<span class="n">abs_numerical_grad</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">numerical_grad</span><span class="p">)</span>
<span class="c1"># if abs_numerical_grad is nearly zero, then use abs error for numeric_grad, not relative</span>
<span class="c1"># error.</span>
<span class="n">abs_numeric_grad</span><span class="p">[</span><span class="n">abs_numeric_grad</span> <span class="o">&lt;</span> <span class="mf">1e-3</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">abs_numerical_grad</span><span class="p">[</span><span class="n">abs_numerical_grad</span> <span class="o">&lt;</span> <span class="mf">1e-3</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">diff_mat</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">abs_numeric_grad</span> <span class="o">-</span> <span class="n">operator_grad</span><span class="p">)</span> <span class="o">/</span> <span class="n">abs_numeric_grad</span>
<span class="n">diff_mat</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">abs_numerical_grad</span> <span class="o">-</span> <span class="n">operator_grad</span><span class="p">)</span> <span class="o">/</span> <span class="n">abs_numerical_grad</span>
<span class="n">max_diff</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">diff_mat</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="notes">
<span id="notes"></span><h3>Notes：<a class="headerlink" href="#notes" title="永久链接至标题"></a></h3>
<p>1，The Input data for auto gradient checker should be reasonable to avoid numeric problem.</p>
<p>The Input data for auto gradient checker should be reasonable to avoid numerical stability problem.</p>
</div>
<div class="section" id="refs">
<span id="refs"></span><h3>Refs:<a class="headerlink" href="#refs" title="永久链接至标题"></a></h3>
......
develop/doc_cn/design/functions_operators_layers.html
浏览文件 @ 477e97f1
......@@ -224,12 +224,12 @@
<p>Let&#8217;s explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we&#8217;d like to have Python functions <code class="docutils literal"><span class="pre">mul</span></code> and <code class="docutils literal"><span class="pre">add</span></code> defined in module <code class="docutils literal"><span class="pre">operator</span></code>:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">operator</span><span class="o">.</span><span class="n">mul</span><span class="p">(</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">):</span>
<span class="n">O</span> <span class="o">=</span> <span class="n">Var</span><span class="p">()</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;mul&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">Y1</span><span class="p">],</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;mul&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">Y1</span><span class="p">},</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="k">return</span> <span class="n">O</span>
<span class="k">def</span> <span class="nf">operator</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">):</span>
<span class="n">O</span> <span class="o">=</span> <span class="n">Var</span><span class="p">()</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;add&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">],</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">cpp</span><span class="o">.</span><span class="n">create_operator</span><span class="p">(</span><span class="s2">&quot;add&quot;</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">{</span><span class="n">X1</span><span class="p">,</span> <span class="n">X2</span><span class="p">},</span> <span class="n">output</span><span class="o">=</span><span class="n">O</span><span class="p">)</span>
<span class="k">return</span> <span class="n">O</span>
</pre></div>
</div>
......
develop/doc_cn/design/graph.html
浏览文件 @ 477e97f1
......@@ -233,7 +233,7 @@
</div>
<div class="section" id="block-and-graph">
<span id="block-and-graph"></span><h2>Block and Graph<a class="headerlink" href="#block-and-graph" title="永久链接至标题"></a></h2>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>The word block and graph are interchangable in the desgin of PaddlePaddle. A <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/pull/3708">Block</a> is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.</p>
<p>A Block keeps operators in an array <code class="docutils literal"><span class="pre">BlockDesc::ops</span></code></p>
<div class="highlight-protobuf"><div class="highlight"><pre><span></span><span class="kd">message</span> <span class="nc">BlockDesc</span> <span class="p">{</span>
<span class="k">repeated</span> <span class="n">OpDesc</span> <span class="na">ops</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
......@@ -241,7 +241,7 @@
<span class="p">}</span>
</pre></div>
</div>
<p>in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in <code class="docutils literal"><span class="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
<p>in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in <code class="docutils literal"><span class="pre">ops</span></code>, we have some forward operators, followed by some gradient operators, and then some optimization operators.</p>
</div>
</div>
......
develop/doc_cn/design/parameters_in_cpp.html
浏览文件 @ 477e97f1
......@@ -186,20 +186,20 @@
<div class="section" id="design-doc-the-c-class-parameters">
<span id="design-doc-the-c-class-parameters"></span><h1>Design Doc: The C++ Class <code class="docutils literal"><span class="pre">Parameters</span></code><a class="headerlink" href="#design-doc-the-c-class-parameters" title="永久链接至标题"></a></h1>
<p><code class="docutils literal"><span class="pre">Parameters</span></code> is a concept we designed in Paddle V2 API. <code class="docutils literal"><span class="pre">Parameters</span></code> is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of <code class="docutils literal"><span class="pre">Parameter</span></code> in <a class="reference internal" href="api.html"><span class="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:</p>
<p><code class="docutils literal"><span class="pre">Parameters</span></code> is a concept we designed in PaddlePaddle V2 API. <code class="docutils literal"><span class="pre">Parameters</span></code> is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of <code class="docutils literal"><span class="pre">Parameter</span></code> in <a class="reference internal" href="api.html"><span class="doc">api.md</span></a>.</p>
<p>We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:</p>
<ul class="simple">
<li>We just use <code class="docutils literal"><span class="pre">memcpy</span></code> to share Parameters between topologies, but this is very inefficient.</li>
<li>We did not implement share Parameters while training. We just trigger <code class="docutils literal"><span class="pre">memcpy</span></code> when start training.</li>
<li>We did not support sharing Parameters while training. We just trigger <code class="docutils literal"><span class="pre">memcpy</span></code> when start training.</li>
</ul>
<p>It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with <code class="docutils literal"><span class="pre">Parameters</span></code>:</p>
<p>It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with <code class="docutils literal"><span class="pre">Parameters</span></code>:</p>
<ol class="simple">
<li><code class="docutils literal"><span class="pre">paddle::Parameter</span></code>. A <code class="docutils literal"><span class="pre">Parameters</span></code> is a container for <code class="docutils literal"><span class="pre">paddle::Parameter</span></code>.
It is evident that we should use <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> when developing <code class="docutils literal"><span class="pre">Parameters</span></code>.
However, the <code class="docutils literal"><span class="pre">Parameter</span></code> class contains many functions and does not have a clear interface.
It contains <code class="docutils literal"><span class="pre">create/store</span> <span class="pre">Parameter</span></code>, <code class="docutils literal"><span class="pre">serialize/deserialize</span></code>, <code class="docutils literal"><span class="pre">optimize(i.e</span> <span class="pre">SGD)</span></code>, <code class="docutils literal"><span class="pre">randomize/zero</span></code>.
When we developing <code class="docutils literal"><span class="pre">Parameters</span></code>, we only use <code class="docutils literal"><span class="pre">create/store</span> <span class="pre">Parameter</span></code> functionality.
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.</li>
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.</li>
<li><code class="docutils literal"><span class="pre">paddle::GradientMachine</span></code> and its sub-classes, e.g., <code class="docutils literal"><span class="pre">paddle::MultiGradientMachine</span></code>, <code class="docutils literal"><span class="pre">paddle::NeuralNetwork</span></code>.
We should pass <code class="docutils literal"><span class="pre">Parameters</span></code> to <code class="docutils literal"><span class="pre">paddle::GradientMachine</span></code> when <code class="docutils literal"><span class="pre">forward/backward</span></code> to avoid <code class="docutils literal"><span class="pre">memcpy</span></code> between topologies.
Also, we should handle multi-GPU/CPU training, because <code class="docutils literal"><span class="pre">forward</span></code> and <code class="docutils literal"><span class="pre">backward</span></code> would perform on multi-GPUs and multi-CPUs.
......@@ -207,7 +207,7 @@ Also, we should handle multi-GPU/CPU training, because <code class="docutils lit
<li><code class="docutils literal"><span class="pre">paddle::ParameterUpdater</span></code>. The ParameterUpdater is used to update parameters in Paddle.
So <code class="docutils literal"><span class="pre">Parameters</span></code> should be used by <code class="docutils literal"><span class="pre">paddle::ParameterUpdater</span></code>, and <code class="docutils literal"><span class="pre">paddle::ParameterUpdater</span></code> should optimize <code class="docutils literal"><span class="pre">Parameters</span></code> (by SGD).</li>
</ol>
<p>The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.</p>
<p>The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.</p>
<ol class="simple">
<li>Clean <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> interface. Extract the functionalities of <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> to prepare for the implementation of Parameters.</li>
<li>Implementation a <code class="docutils literal"><span class="pre">Parameters</span></code> class. It just stores the <code class="docutils literal"><span class="pre">paddle::Parameter</span></code> inside. Make <code class="docutils literal"><span class="pre">GradientMachine</span></code> uses <code class="docutils literal"><span class="pre">Parameters</span></code> as a class member.</li>
......
develop/doc_cn/design/reader/README.html
浏览文件 @ 477e97f1
......@@ -230,7 +230,7 @@
<span class="c1"># a mini batch of three data items, each data item is a list (single column).</span>
<span class="p">[([</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],),</span>
<span class="p">([</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">],),</span>
<span class="p">([</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">],),</span>
<span class="p">([</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">],)]</span>
</pre></div>
</div>
<p>Please note that each item inside the list must be a tuple, below is an invalid output:</p>
......
develop/doc_cn/design/refactorization.html
浏览文件 @ 477e97f1
......@@ -199,7 +199,7 @@
<li>PaddlePaddle represent the computation, training and inference of DL models, by computation graphs.</li>
<li>Please dig into <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a solid example.</li>
<li>Users write Python programs to describe the graphs and run it (locally or remotely).</li>
<li>A graph is composed of <em>variabels</em> and <em>operators</em>.</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be able to be serialized/deserialized, so it<ol>
<li>could to be sent to the cloud for distributed execution, and</li>
<li>be sent to clients for mobile or enterprise deployment.</li>
......@@ -353,7 +353,7 @@
</ul>
</li>
<li>Hand-writing <code class="docutils literal"><span class="pre">GPUKernel</span></code> and <code class="docutils literal"><span class="pre">CPU</span></code> code<ul>
<li>Do not write <code class="docutils literal"><span class="pre">.h</span></code>. CPU Kernel should be in <code class="docutils literal"><span class="pre">.cc</span></code>. CPU kernel should be in <code class="docutils literal"><span class="pre">.cu</span></code>. (<code class="docutils literal"><span class="pre">GCC</span></code> cannot compile GPU code.)</li>
<li>Do not write <code class="docutils literal"><span class="pre">.h</span></code>. CPU Kernel should be in <code class="docutils literal"><span class="pre">.cc</span></code>. GPU kernel should be in <code class="docutils literal"><span class="pre">.cu</span></code>. (<code class="docutils literal"><span class="pre">GCC</span></code> cannot compile GPU code.)</li>
</ul>
</li>
</ul>
......
develop/doc_cn/design/releasing_process.html
浏览文件 @ 477e97f1
......@@ -8,7 +8,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Paddle发行规范 &mdash; PaddlePaddle 文档</title>
<title>PaddlePaddle发行规范 &mdash; PaddlePaddle 文档</title>
......@@ -175,7 +175,7 @@
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li>Paddle发行规范</li>
<li>PaddlePaddle发行规范</li>
</ul>
</div>
......@@ -184,10 +184,10 @@
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="paddle">
<span id="paddle"></span><h1>Paddle发行规范<a class="headerlink" href="#paddle" title="永久链接至标题"></a></h1>
<p>Paddle使用git-flow branching model做分支管理，使用<a class="reference external" href="http://semver.org/">Semantic Versioning</a>标准表示Paddle版本号。</p>
<p>Paddle每次发新的版本，遵循以下流程:</p>
<div class="section" id="paddlepaddle">
<span id="paddlepaddle"></span><h1>PaddlePaddle发行规范<a class="headerlink" href="#paddlepaddle" title="永久链接至标题"></a></h1>
<p>PaddlePaddle使用git-flow branching model做分支管理，使用<a class="reference external" href="http://semver.org/">Semantic Versioning</a>标准表示PaddlePaddle版本号。</p>
<p>PaddlePaddle每次发新的版本，遵循以下流程:</p>
<ol>
<li><p class="first">从<code class="docutils literal"><span class="pre">develop</span></code>分支派生出新的分支，分支名为<code class="docutils literal"><span class="pre">release/版本号</span></code>。例如，<code class="docutils literal"><span class="pre">release/0.10.0</span></code></p>
</li>
......@@ -230,15 +230,15 @@
</ol>
<p>需要注意的是:</p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">release/版本号</span></code>分支一旦建立，一般不允许再从<code class="docutils literal"><span class="pre">develop</span></code>分支合入<code class="docutils literal"><span class="pre">release/版本号</span></code>。这样保证<code class="docutils literal"><span class="pre">release/版本号</span></code>分支功能的封闭，方便测试人员测试Paddle的行为。</li>
<li><code class="docutils literal"><span class="pre">release/版本号</span></code>分支一旦建立，一般不允许再从<code class="docutils literal"><span class="pre">develop</span></code>分支合入<code class="docutils literal"><span class="pre">release/版本号</span></code>。这样保证<code class="docutils literal"><span class="pre">release/版本号</span></code>分支功能的封闭，方便测试人员测试PaddlePaddle的行为。</li>
<li>在<code class="docutils literal"><span class="pre">release/版本号</span></code>分支存在的时候，如果有bugfix的行为，需要将bugfix的分支同时merge到<code class="docutils literal"><span class="pre">master</span></code>, <code class="docutils literal"><span class="pre">develop</span></code>和<code class="docutils literal"><span class="pre">release/版本号</span></code>这三个分支。</li>
</ul>
</div>
<div class="section" id="paddle">
<span id="id1"></span><h1>Paddle 分支规范<a class="headerlink" href="#paddle" title="永久链接至标题"></a></h1>
<p>Paddle开发过程使用<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范，并适应github的特性做了一些区别。</p>
<div class="section" id="paddlepaddle">
<span id="id1"></span><h1>PaddlePaddle 分支规范<a class="headerlink" href="#paddlepaddle" title="永久链接至标题"></a></h1>
<p>PaddlePaddle开发过程使用<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范，并适应github的特性做了一些区别。</p>
<ul class="simple">
<li>Paddle的主版本库遵循<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范。其中:<ul>
<li>PaddlePaddle的主版本库遵循<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范。其中:<ul>
<li><code class="docutils literal"><span class="pre">master</span></code>分支为稳定(stable branch)版本分支。每一个<code class="docutils literal"><span class="pre">master</span></code>分支的版本都是经过单元测试和回归测试的版本。</li>
<li><code class="docutils literal"><span class="pre">develop</span></code>分支为开发(develop branch)版本分支。每一个<code class="docutils literal"><span class="pre">develop</span></code>分支的版本都经过单元测试，但并没有经过回归测试。</li>
<li><code class="docutils literal"><span class="pre">release/版本号</span></code>分支为每一次Release时建立的临时分支。在这个阶段的代码正在经历回归测试。</li>
......@@ -247,7 +247,7 @@
<li>其他用户的fork版本库并不需要严格遵守<a class="reference external" href="http://nvie.com/posts/a-successful-git-branching-model/">git-flow</a>分支规范，但所有fork的版本库的所有分支都相当于特性分支。<ul>
<li>建议，开发者fork的版本库使用<code class="docutils literal"><span class="pre">develop</span></code>分支同步主版本库的<code class="docutils literal"><span class="pre">develop</span></code>分支</li>
<li>建议，开发者fork的版本库中，再基于<code class="docutils literal"><span class="pre">develop</span></code>版本fork出自己的功能分支。</li>
<li>当功能分支开发完毕后，向Paddle的主版本库提交<code class="docutils literal"><span class="pre">Pull</span> <span class="pre">Reuqest</span></code>，进而进行代码评审。<ul>
<li>当功能分支开发完毕后，向PaddlePaddle的主版本库提交<code class="docutils literal"><span class="pre">Pull</span> <span class="pre">Reuqest</span></code>，进而进行代码评审。<ul>
<li>在评审过程中，开发者修改自己的代码，可以继续在自己的功能分支提交代码。</li>
</ul>
</li>
......@@ -256,12 +256,12 @@
<li>BugFix分支也是在开发者自己的fork版本库维护，与功能分支不同的是，BugFix分支需要分别给主版本库的<code class="docutils literal"><span class="pre">master</span></code>、<code class="docutils literal"><span class="pre">develop</span></code>与可能有的<code class="docutils literal"><span class="pre">release/版本号</span></code>分支，同时提起<code class="docutils literal"><span class="pre">Pull</span> <span class="pre">Request</span></code>。</li>
</ul>
</div>
<div class="section" id="paddle">
<span id="id2"></span><h1>Paddle回归测试列表<a class="headerlink" href="#paddle" title="永久链接至标题"></a></h1>
<p>本列表说明Paddle发版之前需要测试的功能点。</p>
<div class="section" id="paddle-book">
<span id="paddle-book"></span><h2>Paddle Book中所有章节<a class="headerlink" href="#paddle-book" title="永久链接至标题"></a></h2>
<p>Paddle每次发版本首先要保证Paddle Book中所有章节功能的正确性。功能的正确性包括验证Paddle目前的<code class="docutils literal"><span class="pre">paddle_trainer</span></code>训练和纯使用<code class="docutils literal"><span class="pre">Python</span></code>训练模型正确性。</p>
<div class="section" id="paddlepaddle">
<span id="id2"></span><h1>PaddlePaddle回归测试列表<a class="headerlink" href="#paddlepaddle" title="永久链接至标题"></a></h1>
<p>本列表说明PaddlePaddle发版之前需要测试的功能点。</p>
<div class="section" id="paddlepaddle-book">
<span id="paddlepaddle-book"></span><h2>PaddlePaddle Book中所有章节<a class="headerlink" href="#paddlepaddle-book" title="永久链接至标题"></a></h2>
<p>PaddlePaddle每次发版本首先要保证PaddlePaddle Book中所有章节功能的正确性。功能的正确性包括验证PaddlePaddle目前的<code class="docutils literal"><span class="pre">paddle_trainer</span></code>训练和纯使用<code class="docutils literal"><span class="pre">Python</span></code>训练模型正确性。</p>
<p>| | 新手入门章节 | 识别数字 | 图像分类 | 词向量 | 情感分析 | 语意角色标注 | 机器翻译 | 个性化推荐 |
| &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; | &#8212; |
| API.V2 + Docker + GPU | | | | | | | | |
......
develop/doc_cn/design/scope.html
浏览文件 @ 477e97f1
......@@ -200,7 +200,7 @@
<p>Scope is an association of a name to variable. All variables belong to <code class="docutils literal"><span class="pre">Scope</span></code>. You need to specify a scope to run a Net, i.e., <code class="docutils literal"><span class="pre">net.Run(&amp;scope)</span></code>. One net can run in different scopes and update different variable in the scope.</p>
<ol>
<li><p class="first">Scope only contains a map of a name to variable.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state(momentum) etc.</p>
<p>All parameters, data, states in a Net should be variables and stored inside a scope. Each op should get inputs and outputs to do computation from a scope, such as data buffer, state (momentum) etc.</p>
</li>
<li><p class="first">Variable can only be created by Scope and a variable can only be got from Scope. User cannot create or get a variable outside a scope. This is a constraints of our framework, and will keep our framework simple and clear.</p>
</li>
......@@ -215,7 +215,7 @@
<p>Variable can not belong to many scopes. If you want to use variables from parent scope, you can use <code class="docutils literal"><span class="pre">parent</span> <span class="pre">scope</span></code>.</p>
</li>
<li><p class="first">Scope should destruct all Variables inside it when itself is destructed. User can never store <code class="docutils literal"><span class="pre">Variable</span></code> pointer somewhere else.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <code class="docutils literal"><span class="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be a invalid pointer when associated <code class="docutils literal"><span class="pre">Scope</span></code> is destroyed.</p>
<p>Because Variable can only be got from Scope. When destroying Scope, we also need to destroy all the Variables in it. If user store <code class="docutils literal"><span class="pre">Variable</span></code> pointer to private data member or some global variable, the pointer will be an invalid pointer when associated <code class="docutils literal"><span class="pre">Scope</span></code> is destroyed.</p>
</li>
</ol>
<div class="highlight-cpp"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Scope</span> <span class="p">{</span>
......@@ -233,7 +233,7 @@
<span id="parent-scope-and-local-scope"></span><h2>Parent scope and local scope<a class="headerlink" href="#parent-scope-and-local-scope" title="永久链接至标题"></a></h2>
<p>Just like <a class="reference external" href="https://en.wikipedia.org/wiki/Scope_(computer_science)">scope</a> in programming languages, <code class="docutils literal"><span class="pre">Scope</span></code> in the neural network can also be a local scope. There are two attributes about local scope.</p>
<ol class="simple">
<li>We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.</li>
<li>We can create local variables in a local scope. When that local scope is destroyed, all local variables should also be destroyed.</li>
<li>Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, <code class="docutils literal"><span class="pre">scope</span></code> will keep searching from its parent, until the variable is found or there is no parent.</li>
</ol>
<div class="highlight-cpp"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Scope</span> <span class="p">{</span>
......@@ -302,7 +302,7 @@
</div>
<div class="section" id="orthogonal-interface">
<span id="orthogonal-interface"></span><h2>Orthogonal interface<a class="headerlink" href="#orthogonal-interface" title="永久链接至标题"></a></h2>
<p><code class="docutils literal"><span class="pre">FindVar</span></code> will return <code class="docutils literal"><span class="pre">nullptr</span></code> when <code class="docutils literal"><span class="pre">name</span></code> is not found. It can be used as <code class="docutils literal"><span class="pre">Contains</span></code> method. <code class="docutils literal"><span class="pre">NewVar</span></code> will return a <code class="docutils literal"><span class="pre">Error</span></code> when there is a name conflict locally. Combine <code class="docutils literal"><span class="pre">FindVar</span></code> and <code class="docutils literal"><span class="pre">NewVar</span></code>, we can implement <code class="docutils literal"><span class="pre">NewVar</span></code> easily.</p>
<p><code class="docutils literal"><span class="pre">FindVar</span></code> will return <code class="docutils literal"><span class="pre">nullptr</span></code> when <code class="docutils literal"><span class="pre">name</span></code> is not found. It can be used as <code class="docutils literal"><span class="pre">Contains</span></code> method. <code class="docutils literal"><span class="pre">NewVar</span></code> will return an <code class="docutils literal"><span class="pre">Error</span></code> when there is a name conflict locally. Combine <code class="docutils literal"><span class="pre">FindVar</span></code> and <code class="docutils literal"><span class="pre">NewVar</span></code>, we can implement <code class="docutils literal"><span class="pre">NewVar</span></code> easily.</p>
</div>
</div>
......
develop/doc_cn/design/simple_op_design.html
浏览文件 @ 477e97f1
......@@ -190,10 +190,10 @@
<p>The Interaction between Python and C++ can be simplified as two steps:</p>
<ol class="simple">
<li>C++ tells Python how many Ops there are, and what parameter do users need to offer to initialize a new Op. Python then builds API for each Op at compile time.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ fo finish Op construction task.</li>
<li>Users invoke APIs built by Python and provide necessary parameters. These parameters will be sent to C++ for finishing the Op construction task.</li>
</ol>
<div class="section" id="message-form-c-to-python">
<span id="message-form-c-to-python"></span><h2>Message form C++ to Python<a class="headerlink" href="#message-form-c-to-python" title="永久链接至标题"></a></h2>
<div class="section" id="message-from-c-to-python">
<span id="message-from-c-to-python"></span><h2>Message from C++ to Python<a class="headerlink" href="#message-from-c-to-python" title="永久链接至标题"></a></h2>
<p>We define a Protobuf message class <code class="docutils literal"><span class="pre">OpProto</span></code> to hold message needed in the first step. What should an <code class="docutils literal"><span class="pre">OpProto</span></code> contain? This question is equivalent to “What message do we need to offer, to build a Python API which is legal and user oriented and can use to describe a whole Op.”</p>
<p>Following message are necessary:</p>
<ol class="simple">
......
develop/doc_cn/design/var_desc.html
浏览文件 @ 477e97f1
......@@ -187,7 +187,7 @@
<div class="section" id="background">
<span id="background"></span><h1>Background<a class="headerlink" href="#background" title="永久链接至标题"></a></h1>
<p>PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.</p>
<p>PaddlePaddle use proto message to describe compile time graph for</p>
<p>PaddlePaddle use proto message to describe compile time graph because</p>
<ol class="simple">
<li>Computation graph should be able to be saved to a file.</li>
<li>In distributed training, the graph will be serialized and send to multiple workers.</li>
......
develop/doc_cn/searchindex.js
浏览文件 @ 477e97f1
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或