@@ -17,22 +17,22 @@ The goals of refactoring include:
1. A graph is composed of *variables* and *operators*.
1. The description of graphs must be capable of being serialized/deserialized, so that:
1. The description of graphs must be serializable/deserializable, so that:
1. It can to be sent to the cloud for distributed execution, and
1. It can be sent to the cloud for distributed execution, and
1. It can be sent to clients for mobile or enterprise deployment.
1. The Python program does the following steps
1. The Python program does two things
1. *compilation*: run a Python program to generate a protobuf message representation of the graph and send it to
1. *Compilation* runs a Python program to generate a protobuf message representation of the graph and send it to
1. the C++ library `libpaddle.so` for local execution,
1. the master process of a distributed training job for training, or
1. the server process of a Kubernetes serving job for distributed serving.
1. *execution*: execute the graph by constructing instances of class [`Variable`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24) and [`OperatorBase`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70), according to the protobuf message.
1. *Execution* executes the graph by constructing instances of class [`Variable`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24) and [`OperatorBase`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70), according to the protobuf message.
## Description and Realization of Computation Graph
At compile time, the Python program generates a protobuf message representation of the graph, or the description of the graph.
At compile time, the Python program generates a protobuf message representation of the graph, or a description of the graph.
At runtime, the C++ program realizes the graph and runs it.
...
...
@@ -42,11 +42,11 @@ At runtime, the C++ program realizes the graph and runs it.
The word *graph* is interchangeable with *block* in this document. A graph represents computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(`{` and `}`).
The word *graph* is interchangeable with *block* in this document. A graph consists of computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(`{` and `}`).
## Compilation and Execution
1. Run an application Python program to describe the graph. In particular, the Python application program does the following:
1. Run a Python program to describe the graph. In particular, the Python application program does the following:
1. Create `VarDesc` to represent local/intermediate variables,
1. Create operators and set attributes,
...
...
@@ -54,10 +54,10 @@ The word *graph* is interchangeable with *block* in this document. A graph repr
1. Infer the type and the shape of variables,
1. Plan memory-reuse for variables,
1. Generate the backward graph
1. Optimize the computation graph.
1. Potentially, split the graph for distributed training.
1. Add optimization operators to the computation graph.
1. Optionally, split the graph for distributed training.
1. The invocation of `train` or [`infer`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108) methods in the application Python program does the following:
1. The invocation of `train` or [`infer`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108) methods in the Python program does the following:
1. Create a new Scope instance in the [scope hierarchy](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/scope.md) for each run of a block,
1. realize local variables defined in the BlockDesc message in the new scope,
* `Operator` is the fundamental building block of the user interface.
* Operator stores input/output variable names, and attributes.
* The `InferShape` interface is used to infer the shape of the output variable shapes based on the shapes of the input variables.
* Operator stores input/output variable names and attributes.
* The `InferShape` interface is used to infer the shape of the output variables based on the shapes of the input variables.
* Use `Run` to compute the `output` variables from the `input` variables.
---
...
...
@@ -139,7 +139,7 @@ Compile Time -> IR -> Runtime
* Limit the number of `tensor.device(dev) = ` in your code.
* `thrust::transform` and `std::transform`.
* `thrust` has the same API as C++ standard library. Using `transform`, one can quickly implement customized element-wise kernels.
* `thrust` also has more complex APIs, like `scan`, `reduce`, `reduce_by_key`.
* `thrust`, in addition, supports more complex APIs, like `scan`, `reduce`, `reduce_by_key`.
* Hand-writing `GPUKernel` and `CPU` code
* Do not write in header (`.h`) files. CPU Kernel should be in cpp source (`.cc`) and GPU kernels should be in cuda (`.cu`) files. (GCC cannot compile GPU code.)
---
...
...
@@ -185,10 +185,10 @@ Make sure the registration process is executed and linked.
1. Write an Op class and its gradient Op class, if required.
2. Write an Op maker class. In the constructor of this class, describe the inputs, outputs and attributes of the operator.
3. Invoke the macro `REGISTER_OP`. This macro will
1. Call maker class to complete the `proto` and the `checker`
1. Call maker class to complete `proto` and `checker`
2. Using the completed `proto` and `checker`, it will add a new key-value pair to the `OpInfoMap`
4. Invoke the `USE` macro in which the Op is used, to make sure that it is linked.
4. Invoke the `USE` macro in which the Op is used to make sure that it is linked.
---
# Backward Module (1/2)
...
...
@@ -199,13 +199,14 @@ Make sure the registration process is executed and linked.
---
# Backward Module (2/2)
### Build Backward Network
- **Input**: graph of forward operators
- **Output**: graph of backward operators
- **Input**: a graph of forward operators
- **Output**: a graph of backward operators
- **Corner cases in construction**
- Shared Variables => insert an `Add` operator to combine gradients
- No Gradient => insert a `fill_zero_grad` operator
- Recursive NetOp => call `Backward` recursively
- RNN Op => recursively call `Backward` on stepnet
- RNN Op => recursively call `Backward` on stepnet
---
...
...
@@ -215,10 +216,10 @@ Make sure the registration process is executed and linked.
* Only dims and data pointers are stored in `Tensor`.
* All operations on `Tensor` are written in `Operator` or global functions.
<li>Please refer to <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a concrete example.</li>
<li>Users write Python programs to describe the graphs and run them (locally or remotely).</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be capable of being serialized/deserialized, so that:<ol>
<li>It can to be sent to the cloud for distributed execution, and</li>
<li>The description of graphs must be serializable/deserializable, so that:<ol>
<li>It can be sent to the cloud for distributed execution, and</li>
<li>It can be sent to clients for mobile or enterprise deployment.</li>
</ol>
</li>
<li>The Python program does the following steps<ol>
<li><em>compilation</em>: run a Python program to generate a protobuf message representation of the graph and send it to<ol>
<li>The Python program does two things<ol>
<li><em>Compilation</em> runs a Python program to generate a protobuf message representation of the graph and send it to<ol>
<li>the C++ library <codeclass="docutils literal"><spanclass="pre">libpaddle.so</span></code> for local execution,</li>
<li>the master process of a distributed training job for training, or</li>
<li>the server process of a Kubernetes serving job for distributed serving.</li>
</ol>
</li>
<li><em>execution</em>: execute the graph by constructing instances of class <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24"><codeclass="docutils literal"><spanclass="pre">Variable</span></code></a> and <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70"><codeclass="docutils literal"><spanclass="pre">OperatorBase</span></code></a>, according to the protobuf message.</li>
<li><em>Execution</em> executes the graph by constructing instances of class <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24"><codeclass="docutils literal"><spanclass="pre">Variable</span></code></a> and <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70"><codeclass="docutils literal"><spanclass="pre">OperatorBase</span></code></a>, according to the protobuf message.</li>
<spanid="description-and-realization-of-computation-graph"></span><h2>Description and Realization of Computation Graph<aclass="headerlink"href="#description-and-realization-of-computation-graph"title="Permalink to this headline">¶</a></h2>
<p>At compile time, the Python program generates a protobuf message representation of the graph, or the description of the graph.</p>
<p>At compile time, the Python program generates a protobuf message representation of the graph, or a description of the graph.</p>
<p>At runtime, the C++ program realizes the graph and runs it.</p>
<p>The word <em>graph</em> is interchangeable with <em>block</em> in this document. A graph represents computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(<codeclass="docutils literal"><spanclass="pre">{</span></code> and <codeclass="docutils literal"><spanclass="pre">}</span></code>).</p>
<p>The word <em>graph</em> is interchangeable with <em>block</em> in this document. A graph consists of computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(<codeclass="docutils literal"><spanclass="pre">{</span></code> and <codeclass="docutils literal"><spanclass="pre">}</span></code>).</p>
<spanid="compilation-and-execution"></span><h2>Compilation and Execution<aclass="headerlink"href="#compilation-and-execution"title="Permalink to this headline">¶</a></h2>
<olclass="simple">
<li>Run an application Python program to describe the graph. In particular, the Python application program does the following:<ol>
<li>Run a Python program to describe the graph. In particular, the Python application program does the following:<ol>
<li>Create <codeclass="docutils literal"><spanclass="pre">VarDesc</span></code> to represent local/intermediate variables,</li>
<li>Create operators and set attributes,</li>
<li>Validate attribute values,</li>
<li>Infer the type and the shape of variables,</li>
<li>Plan memory-reuse for variables,</li>
<li>Generate the backward graph</li>
<li>Optimize the computation graph.</li>
<li>Potentially, split the graph for distributed training.</li>
<li>Add optimization operators to the computation graph.</li>
<li>Optionally, split the graph for distributed training.</li>
</ol>
</li>
<li>The invocation of <codeclass="docutils literal"><spanclass="pre">train</span></code> or <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108"><codeclass="docutils literal"><spanclass="pre">infer</span></code></a> methods in the application Python program does the following:<ol>
<li>The invocation of <codeclass="docutils literal"><spanclass="pre">train</span></code> or <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108"><codeclass="docutils literal"><spanclass="pre">infer</span></code></a> methods in the Python program does the following:<ol>
<li>Create a new Scope instance in the <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/scope.md">scope hierarchy</a> for each run of a block,<ol>
<li>realize local variables defined in the BlockDesc message in the new scope,</li>
<li>a scope is similar to the stack frame in programming languages,</li>
<li><codeclass="docutils literal"><spanclass="pre">Operator</span></code> is the fundamental building block of the user interface.<ul>
<li>Operator stores input/output variable names, and attributes.</li>
<li>The <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> interface is used to infer the shape of the output variable shapes based on the shapes of the input variables.</li>
<li>Operator stores input/output variable names and attributes.</li>
<li>The <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> interface is used to infer the shape of the output variables based on the shapes of the input variables.</li>
<li>Use <codeclass="docutils literal"><spanclass="pre">Run</span></code> to compute the <codeclass="docutils literal"><spanclass="pre">output</span></code> variables from the <codeclass="docutils literal"><spanclass="pre">input</span></code> variables.</li>
</ul>
</li>
...
...
@@ -343,7 +343,7 @@
</li>
<li><codeclass="docutils literal"><spanclass="pre">thrust::transform</span></code> and <codeclass="docutils literal"><spanclass="pre">std::transform</span></code>.<ul>
<li><codeclass="docutils literal"><spanclass="pre">thrust</span></code> has the same API as C++ standard library. Using <codeclass="docutils literal"><spanclass="pre">transform</span></code>, one can quickly implement customized element-wise kernels.</li>
<li><codeclass="docutils literal"><spanclass="pre">thrust</span></code> also has more complex APIs, like <codeclass="docutils literal"><spanclass="pre">scan</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce_by_key</span></code>.</li>
<li><codeclass="docutils literal"><spanclass="pre">thrust</span></code>, in addition, supports more complex APIs, like <codeclass="docutils literal"><spanclass="pre">scan</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce_by_key</span></code>.</li>
</ul>
</li>
<li>Hand-writing <codeclass="docutils literal"><spanclass="pre">GPUKernel</span></code> and <codeclass="docutils literal"><spanclass="pre">CPU</span></code> code<ul>
...
...
@@ -405,11 +405,11 @@
<li>Write an Op class and its gradient Op class, if required.</li>
<li>Write an Op maker class. In the constructor of this class, describe the inputs, outputs and attributes of the operator.</li>
<li>Invoke the macro <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code>. This macro will<ol>
<li>Call maker class to complete the <codeclass="docutils literal"><spanclass="pre">proto</span></code> and the<codeclass="docutils literal"><spanclass="pre">checker</span></code></li>
<li>Call maker class to complete <codeclass="docutils literal"><spanclass="pre">proto</span></code> and<codeclass="docutils literal"><spanclass="pre">checker</span></code></li>
<li>Using the completed <codeclass="docutils literal"><spanclass="pre">proto</span></code> and <codeclass="docutils literal"><spanclass="pre">checker</span></code>, it will add a new key-value pair to the <codeclass="docutils literal"><spanclass="pre">OpInfoMap</span></code></li>
</ol>
</li>
<li>Invoke the <codeclass="docutils literal"><spanclass="pre">USE</span></code> macro in which the Op is used, to make sure that it is linked.</li>
<li>Invoke the <codeclass="docutils literal"><spanclass="pre">USE</span></code> macro in which the Op is used to make sure that it is linked.</li>
</ol>
</div>
<hrclass="docutils"/>
...
...
@@ -429,13 +429,14 @@
<divclass="section"id="build-backward-network">
<spanid="build-backward-network"></span><h2>Build Backward Network<aclass="headerlink"href="#build-backward-network"title="Permalink to this headline">¶</a></h2>
<ulclass="simple">
<li><strong>Input</strong>: graph of forward operators</li>
<li><strong>Output</strong>: graph of backward operators</li>
<li><strong>Input</strong>: a graph of forward operators</li>
<li><strong>Output</strong>: a graph of backward operators</li>
<li><strong>Corner cases in construction</strong><ul>
<li>Shared Variables => insert an <codeclass="docutils literal"><spanclass="pre">Add</span></code> operator to combine gradients</li>
<li>No Gradient => insert a <codeclass="docutils literal"><spanclass="pre">fill_zero_grad</span></code> operator</li>
<li><codeclass="docutils literal"><spanclass="pre">Variable</span></code> instances are the inputs and the outputs of an operator. Not just <codeclass="docutils literal"><spanclass="pre">Tensor</span></code>.<ul>
<li><codeclass="docutils literal"><spanclass="pre">Variable</span></code> instances are the inputs and the outputs of an operator, not just <codeclass="docutils literal"><spanclass="pre">Tensor</span></code>.<ul>
<li><codeclass="docutils literal"><spanclass="pre">step_scopes</span></code> in RNN is a variable and not a tensor.</li>
</ul>
</li>
<li><codeclass="docutils literal"><spanclass="pre">Scope</span></code> is where variables are stores.<ul>
<li><codeclass="docutils literal"><spanclass="pre">Scope</span></code> has a hierarchical structure. The local scope can get variables from its parent scope.</li>
</ul>
</li>
...
...
@@ -503,7 +504,7 @@
<spanid="control-the-migration-quality"></span><h1>Control the migration quality<aclass="headerlink"href="#control-the-migration-quality"title="Permalink to this headline">¶</a></h1>
<ulclass="simple">
<li>Compare the performance of migrated models with old ones.</li>
<li>Follow the google C++ style</li>
<li>Follow the google C++ style guide.</li>
<li>Build the automatic workflow of generating Python/C++ documentations.<ul>
<li>The documentation of layers and ops should be written inside the code.</li>
<li>Take the documentation quality into account when submitting pull requests.</li>
@@ -17,22 +17,22 @@ The goals of refactoring include:
1. A graph is composed of *variables* and *operators*.
1. The description of graphs must be capable of being serialized/deserialized, so that:
1. The description of graphs must be serializable/deserializable, so that:
1. It can to be sent to the cloud for distributed execution, and
1. It can be sent to the cloud for distributed execution, and
1. It can be sent to clients for mobile or enterprise deployment.
1. The Python program does the following steps
1. The Python program does two things
1. *compilation*: run a Python program to generate a protobuf message representation of the graph and send it to
1. *Compilation* runs a Python program to generate a protobuf message representation of the graph and send it to
1. the C++ library `libpaddle.so` for local execution,
1. the master process of a distributed training job for training, or
1. the server process of a Kubernetes serving job for distributed serving.
1. *execution*: execute the graph by constructing instances of class [`Variable`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24) and [`OperatorBase`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70), according to the protobuf message.
1. *Execution* executes the graph by constructing instances of class [`Variable`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24) and [`OperatorBase`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70), according to the protobuf message.
## Description and Realization of Computation Graph
At compile time, the Python program generates a protobuf message representation of the graph, or the description of the graph.
At compile time, the Python program generates a protobuf message representation of the graph, or a description of the graph.
At runtime, the C++ program realizes the graph and runs it.
...
...
@@ -42,11 +42,11 @@ At runtime, the C++ program realizes the graph and runs it.
The word *graph* is interchangeable with *block* in this document. A graph represents computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(`{` and `}`).
The word *graph* is interchangeable with *block* in this document. A graph consists of computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(`{` and `}`).
## Compilation and Execution
1. Run an application Python program to describe the graph. In particular, the Python application program does the following:
1. Run a Python program to describe the graph. In particular, the Python application program does the following:
1. Create `VarDesc` to represent local/intermediate variables,
1. Create operators and set attributes,
...
...
@@ -54,10 +54,10 @@ The word *graph* is interchangeable with *block* in this document. A graph repr
1. Infer the type and the shape of variables,
1. Plan memory-reuse for variables,
1. Generate the backward graph
1. Optimize the computation graph.
1. Potentially, split the graph for distributed training.
1. Add optimization operators to the computation graph.
1. Optionally, split the graph for distributed training.
1. The invocation of `train` or [`infer`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108) methods in the application Python program does the following:
1. The invocation of `train` or [`infer`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108) methods in the Python program does the following:
1. Create a new Scope instance in the [scope hierarchy](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/scope.md) for each run of a block,
1. realize local variables defined in the BlockDesc message in the new scope,
* `Operator` is the fundamental building block of the user interface.
* Operator stores input/output variable names, and attributes.
* The `InferShape` interface is used to infer the shape of the output variable shapes based on the shapes of the input variables.
* Operator stores input/output variable names and attributes.
* The `InferShape` interface is used to infer the shape of the output variables based on the shapes of the input variables.
* Use `Run` to compute the `output` variables from the `input` variables.
---
...
...
@@ -139,7 +139,7 @@ Compile Time -> IR -> Runtime
* Limit the number of `tensor.device(dev) = ` in your code.
* `thrust::transform` and `std::transform`.
* `thrust` has the same API as C++ standard library. Using `transform`, one can quickly implement customized element-wise kernels.
* `thrust` also has more complex APIs, like `scan`, `reduce`, `reduce_by_key`.
* `thrust`, in addition, supports more complex APIs, like `scan`, `reduce`, `reduce_by_key`.
* Hand-writing `GPUKernel` and `CPU` code
* Do not write in header (`.h`) files. CPU Kernel should be in cpp source (`.cc`) and GPU kernels should be in cuda (`.cu`) files. (GCC cannot compile GPU code.)
---
...
...
@@ -185,10 +185,10 @@ Make sure the registration process is executed and linked.
1. Write an Op class and its gradient Op class, if required.
2. Write an Op maker class. In the constructor of this class, describe the inputs, outputs and attributes of the operator.
3. Invoke the macro `REGISTER_OP`. This macro will
1. Call maker class to complete the `proto` and the `checker`
1. Call maker class to complete `proto` and `checker`
2. Using the completed `proto` and `checker`, it will add a new key-value pair to the `OpInfoMap`
4. Invoke the `USE` macro in which the Op is used, to make sure that it is linked.
4. Invoke the `USE` macro in which the Op is used to make sure that it is linked.
---
# Backward Module (1/2)
...
...
@@ -199,13 +199,14 @@ Make sure the registration process is executed and linked.
---
# Backward Module (2/2)
### Build Backward Network
- **Input**: graph of forward operators
- **Output**: graph of backward operators
- **Input**: a graph of forward operators
- **Output**: a graph of backward operators
- **Corner cases in construction**
- Shared Variables => insert an `Add` operator to combine gradients
- No Gradient => insert a `fill_zero_grad` operator
- Recursive NetOp => call `Backward` recursively
- RNN Op => recursively call `Backward` on stepnet
- RNN Op => recursively call `Backward` on stepnet
---
...
...
@@ -215,10 +216,10 @@ Make sure the registration process is executed and linked.
* Only dims and data pointers are stored in `Tensor`.
* All operations on `Tensor` are written in `Operator` or global functions.
<li>Please refer to <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a concrete example.</li>
<li>Users write Python programs to describe the graphs and run them (locally or remotely).</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be capable of being serialized/deserialized, so that:<ol>
<li>It can to be sent to the cloud for distributed execution, and</li>
<li>The description of graphs must be serializable/deserializable, so that:<ol>
<li>It can be sent to the cloud for distributed execution, and</li>
<li>It can be sent to clients for mobile or enterprise deployment.</li>
</ol>
</li>
<li>The Python program does the following steps<ol>
<li><em>compilation</em>: run a Python program to generate a protobuf message representation of the graph and send it to<ol>
<li>The Python program does two things<ol>
<li><em>Compilation</em> runs a Python program to generate a protobuf message representation of the graph and send it to<ol>
<li>the C++ library <codeclass="docutils literal"><spanclass="pre">libpaddle.so</span></code> for local execution,</li>
<li>the master process of a distributed training job for training, or</li>
<li>the server process of a Kubernetes serving job for distributed serving.</li>
</ol>
</li>
<li><em>execution</em>: execute the graph by constructing instances of class <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24"><codeclass="docutils literal"><spanclass="pre">Variable</span></code></a> and <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70"><codeclass="docutils literal"><spanclass="pre">OperatorBase</span></code></a>, according to the protobuf message.</li>
<li><em>Execution</em> executes the graph by constructing instances of class <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24"><codeclass="docutils literal"><spanclass="pre">Variable</span></code></a> and <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70"><codeclass="docutils literal"><spanclass="pre">OperatorBase</span></code></a>, according to the protobuf message.</li>
<spanid="description-and-realization-of-computation-graph"></span><h2>Description and Realization of Computation Graph<aclass="headerlink"href="#description-and-realization-of-computation-graph"title="永久链接至标题">¶</a></h2>
<p>At compile time, the Python program generates a protobuf message representation of the graph, or the description of the graph.</p>
<p>At compile time, the Python program generates a protobuf message representation of the graph, or a description of the graph.</p>
<p>At runtime, the C++ program realizes the graph and runs it.</p>
<p>The word <em>graph</em> is interchangeable with <em>block</em> in this document. A graph represents computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(<codeclass="docutils literal"><spanclass="pre">{</span></code> and <codeclass="docutils literal"><spanclass="pre">}</span></code>).</p>
<p>The word <em>graph</em> is interchangeable with <em>block</em> in this document. A graph consists of computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(<codeclass="docutils literal"><spanclass="pre">{</span></code> and <codeclass="docutils literal"><spanclass="pre">}</span></code>).</p>
<spanid="compilation-and-execution"></span><h2>Compilation and Execution<aclass="headerlink"href="#compilation-and-execution"title="永久链接至标题">¶</a></h2>
<olclass="simple">
<li>Run an application Python program to describe the graph. In particular, the Python application program does the following:<ol>
<li>Run a Python program to describe the graph. In particular, the Python application program does the following:<ol>
<li>Create <codeclass="docutils literal"><spanclass="pre">VarDesc</span></code> to represent local/intermediate variables,</li>
<li>Create operators and set attributes,</li>
<li>Validate attribute values,</li>
<li>Infer the type and the shape of variables,</li>
<li>Plan memory-reuse for variables,</li>
<li>Generate the backward graph</li>
<li>Optimize the computation graph.</li>
<li>Potentially, split the graph for distributed training.</li>
<li>Add optimization operators to the computation graph.</li>
<li>Optionally, split the graph for distributed training.</li>
</ol>
</li>
<li>The invocation of <codeclass="docutils literal"><spanclass="pre">train</span></code> or <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108"><codeclass="docutils literal"><spanclass="pre">infer</span></code></a> methods in the application Python program does the following:<ol>
<li>The invocation of <codeclass="docutils literal"><spanclass="pre">train</span></code> or <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108"><codeclass="docutils literal"><spanclass="pre">infer</span></code></a> methods in the Python program does the following:<ol>
<li>Create a new Scope instance in the <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/scope.md">scope hierarchy</a> for each run of a block,<ol>
<li>realize local variables defined in the BlockDesc message in the new scope,</li>
<li>a scope is similar to the stack frame in programming languages,</li>
<li><codeclass="docutils literal"><spanclass="pre">Operator</span></code> is the fundamental building block of the user interface.<ul>
<li>Operator stores input/output variable names, and attributes.</li>
<li>The <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> interface is used to infer the shape of the output variable shapes based on the shapes of the input variables.</li>
<li>Operator stores input/output variable names and attributes.</li>
<li>The <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> interface is used to infer the shape of the output variables based on the shapes of the input variables.</li>
<li>Use <codeclass="docutils literal"><spanclass="pre">Run</span></code> to compute the <codeclass="docutils literal"><spanclass="pre">output</span></code> variables from the <codeclass="docutils literal"><spanclass="pre">input</span></code> variables.</li>
</ul>
</li>
...
...
@@ -357,7 +357,7 @@
</li>
<li><codeclass="docutils literal"><spanclass="pre">thrust::transform</span></code> and <codeclass="docutils literal"><spanclass="pre">std::transform</span></code>.<ul>
<li><codeclass="docutils literal"><spanclass="pre">thrust</span></code> has the same API as C++ standard library. Using <codeclass="docutils literal"><spanclass="pre">transform</span></code>, one can quickly implement customized element-wise kernels.</li>
<li><codeclass="docutils literal"><spanclass="pre">thrust</span></code> also has more complex APIs, like <codeclass="docutils literal"><spanclass="pre">scan</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce_by_key</span></code>.</li>
<li><codeclass="docutils literal"><spanclass="pre">thrust</span></code>, in addition, supports more complex APIs, like <codeclass="docutils literal"><spanclass="pre">scan</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce</span></code>, <codeclass="docutils literal"><spanclass="pre">reduce_by_key</span></code>.</li>
</ul>
</li>
<li>Hand-writing <codeclass="docutils literal"><spanclass="pre">GPUKernel</span></code> and <codeclass="docutils literal"><spanclass="pre">CPU</span></code> code<ul>
...
...
@@ -419,11 +419,11 @@
<li>Write an Op class and its gradient Op class, if required.</li>
<li>Write an Op maker class. In the constructor of this class, describe the inputs, outputs and attributes of the operator.</li>
<li>Invoke the macro <codeclass="docutils literal"><spanclass="pre">REGISTER_OP</span></code>. This macro will<ol>
<li>Call maker class to complete the <codeclass="docutils literal"><spanclass="pre">proto</span></code> and the<codeclass="docutils literal"><spanclass="pre">checker</span></code></li>
<li>Call maker class to complete <codeclass="docutils literal"><spanclass="pre">proto</span></code> and<codeclass="docutils literal"><spanclass="pre">checker</span></code></li>
<li>Using the completed <codeclass="docutils literal"><spanclass="pre">proto</span></code> and <codeclass="docutils literal"><spanclass="pre">checker</span></code>, it will add a new key-value pair to the <codeclass="docutils literal"><spanclass="pre">OpInfoMap</span></code></li>
</ol>
</li>
<li>Invoke the <codeclass="docutils literal"><spanclass="pre">USE</span></code> macro in which the Op is used, to make sure that it is linked.</li>
<li>Invoke the <codeclass="docutils literal"><spanclass="pre">USE</span></code> macro in which the Op is used to make sure that it is linked.</li>
<li><codeclass="docutils literal"><spanclass="pre">Variable</span></code> instances are the inputs and the outputs of an operator. Not just <codeclass="docutils literal"><spanclass="pre">Tensor</span></code>.<ul>
<li><codeclass="docutils literal"><spanclass="pre">Variable</span></code> instances are the inputs and the outputs of an operator, not just <codeclass="docutils literal"><spanclass="pre">Tensor</span></code>.<ul>
<li><codeclass="docutils literal"><spanclass="pre">step_scopes</span></code> in RNN is a variable and not a tensor.</li>
</ul>
</li>
<li><codeclass="docutils literal"><spanclass="pre">Scope</span></code> is where variables are stores.<ul>
<li><codeclass="docutils literal"><spanclass="pre">Scope</span></code> has a hierarchical structure. The local scope can get variables from its parent scope.</li>
</ul>
</li>
...
...
@@ -517,7 +518,7 @@
<spanid="control-the-migration-quality"></span><h1>Control the migration quality<aclass="headerlink"href="#control-the-migration-quality"title="永久链接至标题">¶</a></h1>
<ulclass="simple">
<li>Compare the performance of migrated models with old ones.</li>
<li>Follow the google C++ style</li>
<li>Follow the google C++ style guide.</li>
<li>Build the automatic workflow of generating Python/C++ documentations.<ul>
<li>The documentation of layers and ops should be written inside the code.</li>
<li>Take the documentation quality into account when submitting pull requests.</li>
