Both deep learning systems and programming languages help users describe computation procedures. These systems use various representations of computation:
- Caffe, Torch, and Paddle: sequences of layers.
- TensorFlow, Caffe2, Mxnet: graphs of operators.
- TensorFlow, Caffe2, Mxnet: graph of operators.
- PaddlePaddle: nested blocks, like C++ and Java programs.
## Block in Programming Languages and Deep Learning
In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions, or operators.
In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions or operators.
Blocks work with control flow structures like `if`, `else`, and `for`, which have equivalents in deep learning:
...
...
@@ -24,14 +24,14 @@ A key difference is that a C++ program describes a one pass computation, whereas
## Stack Frames and the Scope Hierarchy
The existence of the backward makes the execution of a block of traditional programs and PaddlePaddle different to each other:
The existence of the backward pass makes the execution of a block of PaddlePaddle different from traditional programs:
| push at entering block| push at entering block |
| pop at leaving block | destroy when minibatch completes|
1. In traditional programs:
...
...
@@ -42,9 +42,9 @@ The existence of the backward makes the execution of a block of traditional prog
1. In PaddlePaddle
- When the execution enters a block, PaddlePaddle adds a new scope, where it realizes variables.
- PaddlePaddle doesn't pop a scope after the execution of the block because variables therein are to be used by the backward pass. So it has a stack forest known as a *scope hierarchy*.
- PaddlePaddle doesn't pop a scope after the execution of the block because variables therein are used by the backward pass. So it has a stack forest known as a *scope hierarchy*.
- The height of the highest tree is the maximum depth of nested blocks.
- After the process of a minibatch, PaddlePaddle destroys the scope hierarchy.
- After the processing of a minibatch, PaddlePaddle destroys the scope hierarchy.
## Use Blocks in C++ and PaddlePaddle Programs
...
...
@@ -94,14 +94,14 @@ with ie.false_block():
o1, o2 = ie(cond)
```
In both examples, the left branch computes `x+y` and `softmax(x+y)`, the right branch computes `x+1` and `fc(x)`.
In both examples, the left branch computes `x+y` and `softmax(x+y)`, the right branch computes `fc(x)` and `x+1` .
A difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances. The `ie.input(true, 0)` invocation returns instances in the 0-th input, `x`, that corresponds to true values in `cond` as the local variable `x`, where `ie.input(false, 0)` returns instances corresponding to false values.
The difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances.
### Blocks with `for` and `RNNOp`
The following RNN model from the [RNN design doc](./rnn.md)
The following RNN model in PaddlePaddle from the [RNN design doc](./rnn.md) :
```python
x = sequence([10, 20, 30]) # shape=[None, 1]
...
...
@@ -112,9 +112,9 @@ U = var(0.375, param=true) # shape=[1]
rnn = pd.rnn()
with rnn.step():
h = rnn.memory(init = m)
hh = rnn.previous_memory(h)
h_prev = rnn.previous_memory(h)
a = layer.fc(W, x)
b = layer.fc(U, hh)
b = layer.fc(U, h_prev)
s = pd.add(a, b)
act = pd.sigmoid(s)
rnn.update_memory(h, act)
...
...
@@ -147,9 +147,9 @@ for (int i = 1; i <= sizeof(x)/sizeof(x[0]); ++i) {
## Compilation and Execution
Like TensorFlow programs, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest part executes the message for training or inference.
Like TensorFlow, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest executes the message for training or inference.
The generation of this protobuf message is like what a compiler generates a binary executable file. The execution of the message that the OS executes the binary file.
The generation of this protobuf message is similar to how a compiler generates a binary executable file. The execution of the message is similar to how the OS executes the binary file.
## The "Binary Executable File Format"
...
...
@@ -186,8 +186,8 @@ Also, the RNN operator in above example is serialized into a protobuf message of
```
OpDesc {
inputs = {0} // the index of x
outputs = {5, 3} // indices of act and hidden_out
inputs = {0} // the index of x in vars of BlockDesc above
outputs = {5, 3} // indices of act and hidden_out in vars of BlockDesc above
attrs {
"memories" : {1} // the index of h
"step_net" : <above step net>
...
...
@@ -203,14 +203,14 @@ This `OpDesc` value is in the `ops` field of the `BlockDesc` value representing
During the generation of the Protobuf message, the Block should store VarDesc (the Protobuf message which describes Variable) and OpDesc (the Protobuf message which describes Operator).
VarDesc in a block should have its name scope to avoid local variables affect parent block's name scope.
Child block's name scopes should inherit the parent's so that OpDesc in child block can reference a VarDesc that stored in parent block. For example
Child block's name scopes should inherit the parent's so that OpDesc in child block can reference a VarDesc that stored in parent block. For example:
```python
a = pd.Varaible(shape=[20, 20])
a = pd.Variable(shape=[20, 20])
b = pd.fc(a, params=["fc.w", "fc.b"])
rnn = pd.create_rnn()
with rnn.stepnet()
with rnn.stepnet():
x = a.as_step_input()
# reuse fc's parameter
fc_without_b = pd.get_variable("fc.w")
...
...
@@ -218,17 +218,17 @@ with rnn.stepnet()
out = rnn()
```
the method `pd.get_variable` can help retrieve a Variable by a name, a Variable may store in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.
The method `pd.get_variable` can help retrieve a Variable by the name. The Variable may be stored in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.
In compiler design, the symbol table is a data structure created and maintained by compilers to store information about the occurrence of various entities such as variable names, function names, classes, etc.
To store the definition of variables and operators, we define a C++ class `SymbolTable`, like the one used in compilers.
`SymbolTable` can do the following stuff:
`SymbolTable` can do the following:
- store the definitions (some names and attributes) of variables and operators,
- to verify if a variable was declared,
- to make it possible to implement type checking (offer Protobuf message pointers to `InferShape` handlers).
- verify if a variable was declared,
- make it possible to implement type checking (offer Protobuf message pointers to `InferShape` handlers).
```c++
...
...
@@ -240,19 +240,18 @@ class SymbolTable {
OpDesc* NewOp(const string& name="");
// TODO determine whether name is generated by python or C++
// currently assume that a unique name will be generated by C++ if the
// argument name left default.
// TODO determine whether name is generated by python or C++.
// Currently assume that a unique name will be generated by C++ if the
// argument name is left default.
VarDesc* NewVar(const string& name="");
// find a VarDesc by name, if recursive true, find parent's SymbolTable
// find a VarDesc by name, if recursive is true, find parent's SymbolTable
// recursively.
// this interface is introduced to support InferShape, find protobuf messages
// of variables and operators, pass pointers into InferShape.
// operator
//
// NOTE maybe some C++ classes such as VarDescBuilder and OpDescBuilder should
// be proposed and embedded into pybind to enable python operate on C++ pointers.
// be proposed and embedded into pybind to enable python operation on C++ pointers.
// some other necessary interfaces of NetOp are list below
// some other necessary interfaces of NetOp are listed below
// ...
private:
...
...
@@ -316,15 +315,14 @@ private:
Block inherits from OperatorBase, which has a Run method.
Block's Run method will run its operators sequentially.
There is another important interface called `Eval`, which take some arguments called targets, and generate a minimal graph which takes targets as the end points and creates a new Block,
after `Run`, `Eval` will get the latest value and return the targets.
There is another important interface called `Eval`, which takes some arguments called targets and generates a minimal graph which treats targets as the end points and creates a new Block. After `Run`, `Eval` will get the latest value and return the targets.
The definition of Eval is as follows:
```c++
// clean a block description by targets using the corresponding dependency graph.
// return a new BlockDesc with minimal number of operators.
// NOTE not return a Block but the block's description so that this can be distributed
// NOTE: The return type is not a Block but the block's description so that this can be distributed
<p>Both deep learning systems and programming languages help users describe computation procedures. These systems use various representations of computation:</p>
<ulclass="simple">
<li>Caffe, Torch, and Paddle: sequences of layers.</li>
<li>TensorFlow, Caffe2, Mxnet: graphs of operators.</li>
<li>TensorFlow, Caffe2, Mxnet: graph of operators.</li>
<li>PaddlePaddle: nested blocks, like C++ and Java programs.</li>
<spanid="block-in-programming-languages-and-deep-learning"></span><h2>Block in Programming Languages and Deep Learning<aclass="headerlink"href="#block-in-programming-languages-and-deep-learning"title="Permalink to this headline">¶</a></h2>
<p>In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions, or operators.</p>
<p>In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions or operators.</p>
<p>Blocks work with control flow structures like <codeclass="docutils literal"><spanclass="pre">if</span></code>, <codeclass="docutils literal"><spanclass="pre">else</span></code>, and <codeclass="docutils literal"><spanclass="pre">for</span></code>, which have equivalents in deep learning:</p>
<spanid="stack-frames-and-the-scope-hierarchy"></span><h2>Stack Frames and the Scope Hierarchy<aclass="headerlink"href="#stack-frames-and-the-scope-hierarchy"title="Permalink to this headline">¶</a></h2>
<p>The existence of the backward makes the execution of a block of traditional programs and PaddlePaddle different to each other:</p>
| push at entering block| push at entering block |
| pop at leaving block | destroy when minibatch completes|</p>
<olclass="simple">
<li>In traditional programs:<ul>
<li>When the execution enters the left curly brace of a block, the runtime pushes a frame into the stack, where it realizes local variables.</li>
...
...
@@ -217,9 +217,9 @@
</li>
<li>In PaddlePaddle<ul>
<li>When the execution enters a block, PaddlePaddle adds a new scope, where it realizes variables.</li>
<li>PaddlePaddle doesn’t pop a scope after the execution of the block because variables therein are to be used by the backward pass. So it has a stack forest known as a <em>scope hierarchy</em>.</li>
<li>PaddlePaddle doesn’t pop a scope after the execution of the block because variables therein are used by the backward pass. So it has a stack forest known as a <em>scope hierarchy</em>.</li>
<li>The height of the highest tree is the maximum depth of nested blocks.</li>
<li>After the process of a minibatch, PaddlePaddle destroys the scope hierarchy.</li>
<li>After the processing of a minibatch, PaddlePaddle destroys the scope hierarchy.</li>
<p>In both examples, the left branch computes <codeclass="docutils literal"><spanclass="pre">x+y</span></code> and <codeclass="docutils literal"><spanclass="pre">softmax(x+y)</span></code>, the right branch computes <codeclass="docutils literal"><spanclass="pre">x+1</span></code> and <codeclass="docutils literal"><spanclass="pre">fc(x)</span></code>.</p>
<p>A difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances. The <codeclass="docutils literal"><spanclass="pre">ie.input(true,</span><spanclass="pre">0)</span></code> invocation returns instances in the 0-th input, <codeclass="docutils literal"><spanclass="pre">x</span></code>, that corresponds to true values in <codeclass="docutils literal"><spanclass="pre">cond</span></code> as the local variable <codeclass="docutils literal"><spanclass="pre">x</span></code>, where <codeclass="docutils literal"><spanclass="pre">ie.input(false,</span><spanclass="pre">0)</span></code> returns instances corresponding to false values.</p>
<p>In both examples, the left branch computes <codeclass="docutils literal"><spanclass="pre">x+y</span></code> and <codeclass="docutils literal"><spanclass="pre">softmax(x+y)</span></code>, the right branch computes <codeclass="docutils literal"><spanclass="pre">fc(x)</span></code> and <codeclass="docutils literal"><spanclass="pre">x+1</span></code>.</p>
<p>The difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances.</p>
<spanid="blocks-with-for-and-rnnop"></span><h3>Blocks with <codeclass="docutils literal"><spanclass="pre">for</span></code> and <codeclass="docutils literal"><spanclass="pre">RNNOp</span></code><aclass="headerlink"href="#blocks-with-for-and-rnnop"title="Permalink to this headline">¶</a></h3>
<p>The following RNN model from the <aclass="reference external"href="design/rnn.md">RNN design doc</a></p>
<p>The following RNN model in PaddlePaddle from the <aclass="reference external"href="design/rnn.md">RNN design doc</a> :</p>
<spanid="compilation-and-execution"></span><h2>Compilation and Execution<aclass="headerlink"href="#compilation-and-execution"title="Permalink to this headline">¶</a></h2>
<p>Like TensorFlow programs, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest part executes the message for training or inference.</p>
<p>The generation of this protobuf message is like what a compiler generates a binary executable file. The execution of the message that the OS executes the binary file.</p>
<p>Like TensorFlow, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest executes the message for training or inference.</p>
<p>The generation of this protobuf message is similar to how a compiler generates a binary executable file. The execution of the message is similar to how the OS executes the binary file.</p>
<spanid="the-binary-executable-file-format"></span><h2>The “Binary Executable File Format”<aclass="headerlink"href="#the-binary-executable-file-format"title="Permalink to this headline">¶</a></h2>
...
...
@@ -348,8 +348,8 @@
</div>
<p>Also, the RNN operator in above example is serialized into a protobuf message of type <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> and would look like:</p>
<spanid="the-compilation-of-blocks"></span><h2>The Compilation of Blocks<aclass="headerlink"href="#the-compilation-of-blocks"title="Permalink to this headline">¶</a></h2>
<p>During the generation of the Protobuf message, the Block should store VarDesc (the Protobuf message which describes Variable) and OpDesc (the Protobuf message which describes Operator).</p>
<p>VarDesc in a block should have its name scope to avoid local variables affect parent block’s name scope.
Child block’s name scopes should inherit the parent’s so that OpDesc in child block can reference a VarDesc that stored in parent block. For example</p>
Child block’s name scopes should inherit the parent’s so that OpDesc in child block can reference a VarDesc that stored in parent block. For example:</p>
<p>the method <codeclass="docutils literal"><spanclass="pre">pd.get_variable</span></code> can help retrieve a Variable by a name, a Variable may store in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.</p>
<p>The method <codeclass="docutils literal"><spanclass="pre">pd.get_variable</span></code> can help retrieve a Variable by the name. The Variable may be stored in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.</p>
<p>In compiler design, the symbol table is a data structure created and maintained by compilers to store information about the occurrence of various entities such as variable names, function names, classes, etc.</p>
<p>To store the definition of variables and operators, we define a C++ class <codeclass="docutils literal"><spanclass="pre">SymbolTable</span></code>, like the one used in compilers.</p>
<p><codeclass="docutils literal"><spanclass="pre">SymbolTable</span></code> can do the following stuff:</p>
<p><codeclass="docutils literal"><spanclass="pre">SymbolTable</span></code> can do the following:</p>
<ulclass="simple">
<li>store the definitions (some names and attributes) of variables and operators,</li>
<li>to verify if a variable was declared,</li>
<li>to make it possible to implement type checking (offer Protobuf message pointers to <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> handlers).</li>
<li>verify if a variable was declared,</li>
<li>make it possible to implement type checking (offer Protobuf message pointers to <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> handlers).</li>
</ul>
<divclass="highlight-c++"><divclass="highlight"><pre><span></span><spanclass="c1">// Information in SymbolTable is enough to trace the dependency graph. So maybe</span>
<spanclass="c1">// the Eval() interface takes a SymbolTable is enough.</span>
...
...
@@ -394,19 +394,18 @@ Child block’s name scopes should inherit the parent’s so that OpDesc
@@ -423,7 +422,7 @@ Child block’s name scopes should inherit the parent’s so that OpDesc
</div>
<p>After all the description of variables and operators is added into SymbolTable,
the block has enough information to run.</p>
<p>The <codeclass="docutils literal"><spanclass="pre">Block</span></code> class takes a <codeclass="docutils literal"><spanclass="pre">BlockDesc</span></code> as input, and provide <codeclass="docutils literal"><spanclass="pre">Run</span></code> and <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> functions.</p>
<p>The <codeclass="docutils literal"><spanclass="pre">Block</span></code> class takes a <codeclass="docutils literal"><spanclass="pre">BlockDesc</span></code> as input, and provides<codeclass="docutils literal"><spanclass="pre">Run</span></code> and <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> functions.</p>
@@ -466,12 +465,11 @@ the block has enough information to run.</p>
<spanid="the-execution-of-blocks"></span><h2>The Execution of Blocks<aclass="headerlink"href="#the-execution-of-blocks"title="Permalink to this headline">¶</a></h2>
<p>Block inherits from OperatorBase, which has a Run method.
Block’s Run method will run its operators sequentially.</p>
<p>There is another important interface called <codeclass="docutils literal"><spanclass="pre">Eval</span></code>, which take some arguments called targets, and generate a minimal graph which takes targets as the end points and creates a new Block,
after <codeclass="docutils literal"><spanclass="pre">Run</span></code>, <codeclass="docutils literal"><spanclass="pre">Eval</span></code> will get the latest value and return the targets.</p>
<p>There is another important interface called <codeclass="docutils literal"><spanclass="pre">Eval</span></code>, which takes some arguments called targets and generates a minimal graph which treats targets as the end points and creates a new Block. After <codeclass="docutils literal"><spanclass="pre">Run</span></code>, <codeclass="docutils literal"><spanclass="pre">Eval</span></code> will get the latest value and return the targets.</p>
<p>The definition of Eval is as follows:</p>
<divclass="highlight-c++"><divclass="highlight"><pre><span></span><spanclass="c1">// clean a block description by targets using the corresponding dependency graph.</span>
<spanclass="c1">// return a new BlockDesc with minimal number of operators.</span>
<spanclass="c1">// NOTE not return a Block but the block's description so that this can be distributed</span>
<spanclass="c1">// NOTE: The return type is not a Block but the block's description so that this can be distributed</span>
Both deep learning systems and programming languages help users describe computation procedures. These systems use various representations of computation:
- Caffe, Torch, and Paddle: sequences of layers.
- TensorFlow, Caffe2, Mxnet: graphs of operators.
- TensorFlow, Caffe2, Mxnet: graph of operators.
- PaddlePaddle: nested blocks, like C++ and Java programs.
## Block in Programming Languages and Deep Learning
In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions, or operators.
In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions or operators.
Blocks work with control flow structures like `if`, `else`, and `for`, which have equivalents in deep learning:
...
...
@@ -24,14 +24,14 @@ A key difference is that a C++ program describes a one pass computation, whereas
## Stack Frames and the Scope Hierarchy
The existence of the backward makes the execution of a block of traditional programs and PaddlePaddle different to each other:
The existence of the backward pass makes the execution of a block of PaddlePaddle different from traditional programs:
| push at entering block| push at entering block |
| pop at leaving block | destroy when minibatch completes|
1. In traditional programs:
...
...
@@ -42,9 +42,9 @@ The existence of the backward makes the execution of a block of traditional prog
1. In PaddlePaddle
- When the execution enters a block, PaddlePaddle adds a new scope, where it realizes variables.
- PaddlePaddle doesn't pop a scope after the execution of the block because variables therein are to be used by the backward pass. So it has a stack forest known as a *scope hierarchy*.
- PaddlePaddle doesn't pop a scope after the execution of the block because variables therein are used by the backward pass. So it has a stack forest known as a *scope hierarchy*.
- The height of the highest tree is the maximum depth of nested blocks.
- After the process of a minibatch, PaddlePaddle destroys the scope hierarchy.
- After the processing of a minibatch, PaddlePaddle destroys the scope hierarchy.
## Use Blocks in C++ and PaddlePaddle Programs
...
...
@@ -94,14 +94,14 @@ with ie.false_block():
o1, o2 = ie(cond)
```
In both examples, the left branch computes `x+y` and `softmax(x+y)`, the right branch computes `x+1` and `fc(x)`.
In both examples, the left branch computes `x+y` and `softmax(x+y)`, the right branch computes `fc(x)` and `x+1` .
A difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances. The `ie.input(true, 0)` invocation returns instances in the 0-th input, `x`, that corresponds to true values in `cond` as the local variable `x`, where `ie.input(false, 0)` returns instances corresponding to false values.
The difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances.
### Blocks with `for` and `RNNOp`
The following RNN model from the [RNN design doc](./rnn.md)
The following RNN model in PaddlePaddle from the [RNN design doc](./rnn.md) :
```python
x = sequence([10, 20, 30]) # shape=[None, 1]
...
...
@@ -112,9 +112,9 @@ U = var(0.375, param=true) # shape=[1]
rnn = pd.rnn()
with rnn.step():
h = rnn.memory(init = m)
hh = rnn.previous_memory(h)
h_prev = rnn.previous_memory(h)
a = layer.fc(W, x)
b = layer.fc(U, hh)
b = layer.fc(U, h_prev)
s = pd.add(a, b)
act = pd.sigmoid(s)
rnn.update_memory(h, act)
...
...
@@ -147,9 +147,9 @@ for (int i = 1; i <= sizeof(x)/sizeof(x[0]); ++i) {
## Compilation and Execution
Like TensorFlow programs, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest part executes the message for training or inference.
Like TensorFlow, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest executes the message for training or inference.
The generation of this protobuf message is like what a compiler generates a binary executable file. The execution of the message that the OS executes the binary file.
The generation of this protobuf message is similar to how a compiler generates a binary executable file. The execution of the message is similar to how the OS executes the binary file.
## The "Binary Executable File Format"
...
...
@@ -186,8 +186,8 @@ Also, the RNN operator in above example is serialized into a protobuf message of
```
OpDesc {
inputs = {0} // the index of x
outputs = {5, 3} // indices of act and hidden_out
inputs = {0} // the index of x in vars of BlockDesc above
outputs = {5, 3} // indices of act and hidden_out in vars of BlockDesc above
attrs {
"memories" : {1} // the index of h
"step_net" : <above step net>
...
...
@@ -203,14 +203,14 @@ This `OpDesc` value is in the `ops` field of the `BlockDesc` value representing
During the generation of the Protobuf message, the Block should store VarDesc (the Protobuf message which describes Variable) and OpDesc (the Protobuf message which describes Operator).
VarDesc in a block should have its name scope to avoid local variables affect parent block's name scope.
Child block's name scopes should inherit the parent's so that OpDesc in child block can reference a VarDesc that stored in parent block. For example
Child block's name scopes should inherit the parent's so that OpDesc in child block can reference a VarDesc that stored in parent block. For example:
```python
a = pd.Varaible(shape=[20, 20])
a = pd.Variable(shape=[20, 20])
b = pd.fc(a, params=["fc.w", "fc.b"])
rnn = pd.create_rnn()
with rnn.stepnet()
with rnn.stepnet():
x = a.as_step_input()
# reuse fc's parameter
fc_without_b = pd.get_variable("fc.w")
...
...
@@ -218,17 +218,17 @@ with rnn.stepnet()
out = rnn()
```
the method `pd.get_variable` can help retrieve a Variable by a name, a Variable may store in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.
The method `pd.get_variable` can help retrieve a Variable by the name. The Variable may be stored in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.
In compiler design, the symbol table is a data structure created and maintained by compilers to store information about the occurrence of various entities such as variable names, function names, classes, etc.
To store the definition of variables and operators, we define a C++ class `SymbolTable`, like the one used in compilers.
`SymbolTable` can do the following stuff:
`SymbolTable` can do the following:
- store the definitions (some names and attributes) of variables and operators,
- to verify if a variable was declared,
- to make it possible to implement type checking (offer Protobuf message pointers to `InferShape` handlers).
- verify if a variable was declared,
- make it possible to implement type checking (offer Protobuf message pointers to `InferShape` handlers).
```c++
...
...
@@ -240,19 +240,18 @@ class SymbolTable {
OpDesc* NewOp(const string& name="");
// TODO determine whether name is generated by python or C++
// currently assume that a unique name will be generated by C++ if the
// argument name left default.
// TODO determine whether name is generated by python or C++.
// Currently assume that a unique name will be generated by C++ if the
// argument name is left default.
VarDesc* NewVar(const string& name="");
// find a VarDesc by name, if recursive true, find parent's SymbolTable
// find a VarDesc by name, if recursive is true, find parent's SymbolTable
// recursively.
// this interface is introduced to support InferShape, find protobuf messages
// of variables and operators, pass pointers into InferShape.
// operator
//
// NOTE maybe some C++ classes such as VarDescBuilder and OpDescBuilder should
// be proposed and embedded into pybind to enable python operate on C++ pointers.
// be proposed and embedded into pybind to enable python operation on C++ pointers.
// some other necessary interfaces of NetOp are list below
// some other necessary interfaces of NetOp are listed below
// ...
private:
...
...
@@ -316,15 +315,14 @@ private:
Block inherits from OperatorBase, which has a Run method.
Block's Run method will run its operators sequentially.
There is another important interface called `Eval`, which take some arguments called targets, and generate a minimal graph which takes targets as the end points and creates a new Block,
after `Run`, `Eval` will get the latest value and return the targets.
There is another important interface called `Eval`, which takes some arguments called targets and generates a minimal graph which treats targets as the end points and creates a new Block. After `Run`, `Eval` will get the latest value and return the targets.
The definition of Eval is as follows:
```c++
// clean a block description by targets using the corresponding dependency graph.
// return a new BlockDesc with minimal number of operators.
// NOTE not return a Block but the block's description so that this can be distributed
// NOTE: The return type is not a Block but the block's description so that this can be distributed
<p>Both deep learning systems and programming languages help users describe computation procedures. These systems use various representations of computation:</p>
<ulclass="simple">
<li>Caffe, Torch, and Paddle: sequences of layers.</li>
<li>TensorFlow, Caffe2, Mxnet: graphs of operators.</li>
<li>TensorFlow, Caffe2, Mxnet: graph of operators.</li>
<li>PaddlePaddle: nested blocks, like C++ and Java programs.</li>
<spanid="block-in-programming-languages-and-deep-learning"></span><h2>Block in Programming Languages and Deep Learning<aclass="headerlink"href="#block-in-programming-languages-and-deep-learning"title="永久链接至标题">¶</a></h2>
<p>In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions, or operators.</p>
<p>In programming languages, a block is a pair of curly braces that includes local variables definitions and a sequence of instructions or operators.</p>
<p>Blocks work with control flow structures like <codeclass="docutils literal"><spanclass="pre">if</span></code>, <codeclass="docutils literal"><spanclass="pre">else</span></code>, and <codeclass="docutils literal"><spanclass="pre">for</span></code>, which have equivalents in deep learning:</p>
<spanid="stack-frames-and-the-scope-hierarchy"></span><h2>Stack Frames and the Scope Hierarchy<aclass="headerlink"href="#stack-frames-and-the-scope-hierarchy"title="永久链接至标题">¶</a></h2>
<p>The existence of the backward makes the execution of a block of traditional programs and PaddlePaddle different to each other:</p>
| push at entering block| push at entering block |
| pop at leaving block | destroy when minibatch completes|</p>
<olclass="simple">
<li>In traditional programs:<ul>
<li>When the execution enters the left curly brace of a block, the runtime pushes a frame into the stack, where it realizes local variables.</li>
...
...
@@ -231,9 +231,9 @@
</li>
<li>In PaddlePaddle<ul>
<li>When the execution enters a block, PaddlePaddle adds a new scope, where it realizes variables.</li>
<li>PaddlePaddle doesn’t pop a scope after the execution of the block because variables therein are to be used by the backward pass. So it has a stack forest known as a <em>scope hierarchy</em>.</li>
<li>PaddlePaddle doesn’t pop a scope after the execution of the block because variables therein are used by the backward pass. So it has a stack forest known as a <em>scope hierarchy</em>.</li>
<li>The height of the highest tree is the maximum depth of nested blocks.</li>
<li>After the process of a minibatch, PaddlePaddle destroys the scope hierarchy.</li>
<li>After the processing of a minibatch, PaddlePaddle destroys the scope hierarchy.</li>
<p>In both examples, the left branch computes <codeclass="docutils literal"><spanclass="pre">x+y</span></code> and <codeclass="docutils literal"><spanclass="pre">softmax(x+y)</span></code>, the right branch computes <codeclass="docutils literal"><spanclass="pre">x+1</span></code> and <codeclass="docutils literal"><spanclass="pre">fc(x)</span></code>.</p>
<p>A difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances. The <codeclass="docutils literal"><spanclass="pre">ie.input(true,</span><spanclass="pre">0)</span></code> invocation returns instances in the 0-th input, <codeclass="docutils literal"><spanclass="pre">x</span></code>, that corresponds to true values in <codeclass="docutils literal"><spanclass="pre">cond</span></code> as the local variable <codeclass="docutils literal"><spanclass="pre">x</span></code>, where <codeclass="docutils literal"><spanclass="pre">ie.input(false,</span><spanclass="pre">0)</span></code> returns instances corresponding to false values.</p>
<p>In both examples, the left branch computes <codeclass="docutils literal"><spanclass="pre">x+y</span></code> and <codeclass="docutils literal"><spanclass="pre">softmax(x+y)</span></code>, the right branch computes <codeclass="docutils literal"><spanclass="pre">fc(x)</span></code> and <codeclass="docutils literal"><spanclass="pre">x+1</span></code>.</p>
<p>The difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances.</p>
<spanid="blocks-with-for-and-rnnop"></span><h3>Blocks with <codeclass="docutils literal"><spanclass="pre">for</span></code> and <codeclass="docutils literal"><spanclass="pre">RNNOp</span></code><aclass="headerlink"href="#blocks-with-for-and-rnnop"title="永久链接至标题">¶</a></h3>
<p>The following RNN model from the <aclass="reference external"href="design/rnn.md">RNN design doc</a></p>
<p>The following RNN model in PaddlePaddle from the <aclass="reference external"href="design/rnn.md">RNN design doc</a> :</p>
<spanid="compilation-and-execution"></span><h2>Compilation and Execution<aclass="headerlink"href="#compilation-and-execution"title="永久链接至标题">¶</a></h2>
<p>Like TensorFlow programs, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest part executes the message for training or inference.</p>
<p>The generation of this protobuf message is like what a compiler generates a binary executable file. The execution of the message that the OS executes the binary file.</p>
<p>Like TensorFlow, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest executes the message for training or inference.</p>
<p>The generation of this protobuf message is similar to how a compiler generates a binary executable file. The execution of the message is similar to how the OS executes the binary file.</p>
<p>Also, the RNN operator in above example is serialized into a protobuf message of type <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> and would look like:</p>
<spanid="the-compilation-of-blocks"></span><h2>The Compilation of Blocks<aclass="headerlink"href="#the-compilation-of-blocks"title="永久链接至标题">¶</a></h2>
<p>During the generation of the Protobuf message, the Block should store VarDesc (the Protobuf message which describes Variable) and OpDesc (the Protobuf message which describes Operator).</p>
<p>VarDesc in a block should have its name scope to avoid local variables affect parent block’s name scope.
Child block’s name scopes should inherit the parent’s so that OpDesc in child block can reference a VarDesc that stored in parent block. For example</p>
Child block’s name scopes should inherit the parent’s so that OpDesc in child block can reference a VarDesc that stored in parent block. For example:</p>
<p>the method <codeclass="docutils literal"><spanclass="pre">pd.get_variable</span></code> can help retrieve a Variable by a name, a Variable may store in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.</p>
<p>The method <codeclass="docutils literal"><spanclass="pre">pd.get_variable</span></code> can help retrieve a Variable by the name. The Variable may be stored in a parent block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.</p>
<p>In compiler design, the symbol table is a data structure created and maintained by compilers to store information about the occurrence of various entities such as variable names, function names, classes, etc.</p>
<p>To store the definition of variables and operators, we define a C++ class <codeclass="docutils literal"><spanclass="pre">SymbolTable</span></code>, like the one used in compilers.</p>
<p><codeclass="docutils literal"><spanclass="pre">SymbolTable</span></code> can do the following stuff:</p>
<p><codeclass="docutils literal"><spanclass="pre">SymbolTable</span></code> can do the following:</p>
<ulclass="simple">
<li>store the definitions (some names and attributes) of variables and operators,</li>
<li>to verify if a variable was declared,</li>
<li>to make it possible to implement type checking (offer Protobuf message pointers to <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> handlers).</li>
<li>verify if a variable was declared,</li>
<li>make it possible to implement type checking (offer Protobuf message pointers to <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> handlers).</li>
</ul>
<divclass="highlight-c++"><divclass="highlight"><pre><span></span><spanclass="c1">// Information in SymbolTable is enough to trace the dependency graph. So maybe</span>
<spanclass="c1">// the Eval() interface takes a SymbolTable is enough.</span>
...
...
@@ -408,19 +408,18 @@ Child block’s name scopes should inherit the parent’s so that OpDesc
@@ -437,7 +436,7 @@ Child block’s name scopes should inherit the parent’s so that OpDesc
</div>
<p>After all the description of variables and operators is added into SymbolTable,
the block has enough information to run.</p>
<p>The <codeclass="docutils literal"><spanclass="pre">Block</span></code> class takes a <codeclass="docutils literal"><spanclass="pre">BlockDesc</span></code> as input, and provide <codeclass="docutils literal"><spanclass="pre">Run</span></code> and <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> functions.</p>
<p>The <codeclass="docutils literal"><spanclass="pre">Block</span></code> class takes a <codeclass="docutils literal"><spanclass="pre">BlockDesc</span></code> as input, and provides<codeclass="docutils literal"><spanclass="pre">Run</span></code> and <codeclass="docutils literal"><spanclass="pre">InferShape</span></code> functions.</p>
@@ -480,12 +479,11 @@ the block has enough information to run.</p>
<spanid="the-execution-of-blocks"></span><h2>The Execution of Blocks<aclass="headerlink"href="#the-execution-of-blocks"title="永久链接至标题">¶</a></h2>
<p>Block inherits from OperatorBase, which has a Run method.
Block’s Run method will run its operators sequentially.</p>
<p>There is another important interface called <codeclass="docutils literal"><spanclass="pre">Eval</span></code>, which take some arguments called targets, and generate a minimal graph which takes targets as the end points and creates a new Block,
after <codeclass="docutils literal"><spanclass="pre">Run</span></code>, <codeclass="docutils literal"><spanclass="pre">Eval</span></code> will get the latest value and return the targets.</p>
<p>There is another important interface called <codeclass="docutils literal"><spanclass="pre">Eval</span></code>, which takes some arguments called targets and generates a minimal graph which treats targets as the end points and creates a new Block. After <codeclass="docutils literal"><spanclass="pre">Run</span></code>, <codeclass="docutils literal"><spanclass="pre">Eval</span></code> will get the latest value and return the targets.</p>
<p>The definition of Eval is as follows:</p>
<divclass="highlight-c++"><divclass="highlight"><pre><span></span><spanclass="c1">// clean a block description by targets using the corresponding dependency graph.</span>
<spanclass="c1">// return a new BlockDesc with minimal number of operators.</span>
<spanclass="c1">// NOTE not return a Block but the block's description so that this can be distributed</span>
<spanclass="c1">// NOTE: The return type is not a Block but the block's description so that this can be distributed</span>
