`optimize_op_attrs` is not in the `VarDesc` message, but kept in the Python instance, as it will be used in the Python space when creating the optimize operator's `OpDesc`, and will be in the `OpDesc` message.
## Layer Functions
## Layer Function
A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.
A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.
### Data Layer
Layer functions take `Variable` and configuration parameters as its input and return the output variable(s).
For example, `FullyConnected` take one or more variable as its input. The input could be input data or another layer's output. There are many configuration options for a `FullyConnected` layer, such as layer size, activation, parameter names, initialization strategies of parameters, and so on. The `FullyConnected` layer will return an output variable.
### Necessity for reusing code between layer functions
There are a lot of code that can be reused. Such as
* Give the default value of configuration. e.g., default initialize strategy for parameters is uniform random with `min = -1.0`, `max = 1.0`. and default initialize strategy for bias is to fill zero.
* Append the activation operator.
* Create a temporary variable.
* Create parameter.
* Generate a unique name.
* Add a bias.
* ...
A mechanism to reuse code between layer functions is necessary. It will be around [150 lines of code](https://github.com/PaddlePaddle/Paddle/pull/4724/files#diff-823b27e07e93914ada859232ae23f846R12) if we write a `FullyConnected` layer without any helper functions.
### Comparision between global functions and helper class
The `FullyConnected` layer will be as follow when we provide global functions:
The input to the feed operator is a special variable in the global scope, which is the output of [Python readers](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md).
We can provide many helpers functions for layer developers. However, there are several disadvantages for global helper functions:
1. We need a namespace for these methods, then layer developers can quickly figure out what method they can use.
2. Global functions will force layer developers to pass its parameter time by time.
So we provide a helper class, `LayerHelper`, to share code between layer functions. The `FullyConnected` Layer will be as follow.
helper = LayerHelper(locals()) # pass all parameter to LayerHelper
mul_results = []
for ipt, param in helper.iter_multiple_input_and_param():
w = helper.create_parameter(shape=ipt.shape[1:] + [size], dtype = ipt.dtype)
tmp = helper.create_tmp_variable()
helper.append_op('mul', {ipt, w}, {tmp})
mul_results.append(tmp)
pre_bias = helper.add_sum(mul_results)
pre_activation = helper.add_bias(pre_bias)
return helper.add_activation(pre_activation)
```
We not only use the fewer lines of code to write `fc_layer` but also make the code clearer to understand. At the same time, layer developers can figure out what function they can invoke by typing `helper.` in a python editor.
### Implementation of layer helper
### FC Layer
We just keep all parameters of a layer function as a dictionary in layer helper as a private data member. Every method of layer helper will look up the dictionary after it is invoked. In that way, we can implement a layer helper for all layer functions even some layer does not contain some operator. For example, The `activation` is used by the FullyConnected layer or convolution layers, but a cross-entropy layer does not use it. The example code of `add_activation` are:
```python
def fc_layer(input, size, ...):
block = program.current_block()
w = block.create_parameter(...)
b = block.create_parameter(...)
out = block.create_var()
op = block.append_operator("FC", X=input, W=w, b=b, out=out)
out.writer = op
return out
class LayerHelper(object):
def __init__(self, **kwargs): # kwargs is short for `keyword arguments`
self.kwargs = kwargs
def add_activation(self, input_var):
act = self.kwargs.get("act", None) # default value is None
<p><codeclass="docutils literal"><spanclass="pre">optimize_op_attrs</span></code> is not in the <codeclass="docutils literal"><spanclass="pre">VarDesc</span></code> message, but kept in the Python instance, as it will be used in the Python space when creating the optimize operator’s <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code>, and will be in the <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> message.</p>
</div>
</div>
<divclass="section"id="layer-functions">
<spanid="layer-functions"></span><h2>Layer Functions<aclass="headerlink"href="#layer-functions"title="Permalink to this headline">¶</a></h2>
<p>A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.</p>
<divclass="section"id="data-layer">
<spanid="data-layer"></span><h3>Data Layer<aclass="headerlink"href="#data-layer"title="Permalink to this headline">¶</a></h3>
<spanid="layer-function"></span><h2>Layer Function<aclass="headerlink"href="#layer-function"title="Permalink to this headline">¶</a></h2>
<p>A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.</p>
<p>Layer functions take <codeclass="docutils literal"><spanclass="pre">Variable</span></code> and configuration parameters as its input and return the output variable(s).</p>
<p>For example, <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> take one or more variable as its input. The input could be input data or another layer’s output. There are many configuration options for a <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer, such as layer size, activation, parameter names, initialization strategies of parameters, and so on. The <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer will return an output variable.</p>
<spanid="necessity-for-reusing-code-between-layer-functions"></span><h3>Necessity for reusing code between layer functions<aclass="headerlink"href="#necessity-for-reusing-code-between-layer-functions"title="Permalink to this headline">¶</a></h3>
<p>There are a lot of code that can be reused. Such as</p>
<ulclass="simple">
<li>Give the default value of configuration. e.g., default initialize strategy for parameters is uniform random with <codeclass="docutils literal"><spanclass="pre">min</span><spanclass="pre">=</span><spanclass="pre">-1.0</span></code>, <codeclass="docutils literal"><spanclass="pre">max</span><spanclass="pre">=</span><spanclass="pre">1.0</span></code>. and default initialize strategy for bias is to fill zero.</li>
<li>Append the activation operator.</li>
<li>Create a temporary variable.</li>
<li>Create parameter.</li>
<li>Generate a unique name.</li>
<li>Add a bias.</li>
<li>...</li>
</ul>
<p>A mechanism to reuse code between layer functions is necessary. It will be around <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/pull/4724/files#diff-823b27e07e93914ada859232ae23f846R12">150 lines of code</a> if we write a <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer without any helper functions.</p>
<spanid="comparision-between-global-functions-and-helper-class"></span><h3>Comparision between global functions and helper class<aclass="headerlink"href="#comparision-between-global-functions-and-helper-class"title="Permalink to this headline">¶</a></h3>
<p>The <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer will be as follow when we provide global functions:</p>
<p>We can provide many helpers functions for layer developers. However, there are several disadvantages for global helper functions:</p>
<olclass="simple">
<li>We need a namespace for these methods, then layer developers can quickly figure out what method they can use.</li>
<li>Global functions will force layer developers to pass its parameter time by time.</li>
</ol>
<p>So we provide a helper class, <codeclass="docutils literal"><spanclass="pre">LayerHelper</span></code>, to share code between layer functions. The <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> Layer will be as follow.</p>
<spanclass="n">helper</span><spanclass="o">=</span><spanclass="n">LayerHelper</span><spanclass="p">(</span><spanclass="nb">locals</span><spanclass="p">())</span><spanclass="c1"># pass all parameter to LayerHelper</span>
<p>The input to the feed operator is a special variable in the global scope, which is the output of <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md">Python readers</a>.</p>
<p>We not only use the fewer lines of code to write <codeclass="docutils literal"><spanclass="pre">fc_layer</span></code> but also make the code clearer to understand. At the same time, layer developers can figure out what function they can invoke by typing <codeclass="docutils literal"><spanclass="pre">helper.</span></code> in a python editor.</p>
</div>
<divclass="section"id="fc-layer">
<spanid="fc-layer"></span><h3>FC Layer<aclass="headerlink"href="#fc-layer"title="Permalink to this headline">¶</a></h3>
<spanid="implementation-of-layer-helper"></span><h3>Implementation of layer helper<aclass="headerlink"href="#implementation-of-layer-helper"title="Permalink to this headline">¶</a></h3>
<p>We just keep all parameters of a layer function as a dictionary in layer helper as a private data member. Every method of layer helper will look up the dictionary after it is invoked. In that way, we can implement a layer helper for all layer functions even some layer does not contain some operator. For example, The <codeclass="docutils literal"><spanclass="pre">activation</span></code> is used by the FullyConnected layer or convolution layers, but a cross-entropy layer does not use it. The example code of <codeclass="docutils literal"><spanclass="pre">add_activation</span></code> are:</p>
<spanclass="k">def</span><spanclass="fm">__init__</span><spanclass="p">(</span><spanclass="bp">self</span><spanclass="p">,</span><spanclass="o">**</span><spanclass="n">kwargs</span><spanclass="p">):</span><spanclass="c1"># kwargs is short for `keyword arguments`</span>
<spanclass="n">act</span><spanclass="o">=</span><spanclass="bp">self</span><spanclass="o">.</span><spanclass="n">kwargs</span><spanclass="o">.</span><spanclass="n">get</span><spanclass="p">(</span><spanclass="s2">"act"</span><spanclass="p">,</span><spanclass="bp">None</span><spanclass="p">)</span><spanclass="c1"># default value is None</span>
<spanclass="k">if</span><spanclass="n">act</span><spanclass="ow">is</span><spanclass="bp">None</span><spanclass="p">:</span><spanclass="c1"># do nothing if no act</span>
`optimize_op_attrs` is not in the `VarDesc` message, but kept in the Python instance, as it will be used in the Python space when creating the optimize operator's `OpDesc`, and will be in the `OpDesc` message.
## Layer Functions
## Layer Function
A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.
A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.
### Data Layer
Layer functions take `Variable` and configuration parameters as its input and return the output variable(s).
For example, `FullyConnected` take one or more variable as its input. The input could be input data or another layer's output. There are many configuration options for a `FullyConnected` layer, such as layer size, activation, parameter names, initialization strategies of parameters, and so on. The `FullyConnected` layer will return an output variable.
### Necessity for reusing code between layer functions
There are a lot of code that can be reused. Such as
* Give the default value of configuration. e.g., default initialize strategy for parameters is uniform random with `min = -1.0`, `max = 1.0`. and default initialize strategy for bias is to fill zero.
* Append the activation operator.
* Create a temporary variable.
* Create parameter.
* Generate a unique name.
* Add a bias.
* ...
A mechanism to reuse code between layer functions is necessary. It will be around [150 lines of code](https://github.com/PaddlePaddle/Paddle/pull/4724/files#diff-823b27e07e93914ada859232ae23f846R12) if we write a `FullyConnected` layer without any helper functions.
### Comparision between global functions and helper class
The `FullyConnected` layer will be as follow when we provide global functions:
The input to the feed operator is a special variable in the global scope, which is the output of [Python readers](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md).
We can provide many helpers functions for layer developers. However, there are several disadvantages for global helper functions:
1. We need a namespace for these methods, then layer developers can quickly figure out what method they can use.
2. Global functions will force layer developers to pass its parameter time by time.
So we provide a helper class, `LayerHelper`, to share code between layer functions. The `FullyConnected` Layer will be as follow.
helper = LayerHelper(locals()) # pass all parameter to LayerHelper
mul_results = []
for ipt, param in helper.iter_multiple_input_and_param():
w = helper.create_parameter(shape=ipt.shape[1:] + [size], dtype = ipt.dtype)
tmp = helper.create_tmp_variable()
helper.append_op('mul', {ipt, w}, {tmp})
mul_results.append(tmp)
pre_bias = helper.add_sum(mul_results)
pre_activation = helper.add_bias(pre_bias)
return helper.add_activation(pre_activation)
```
We not only use the fewer lines of code to write `fc_layer` but also make the code clearer to understand. At the same time, layer developers can figure out what function they can invoke by typing `helper.` in a python editor.
### Implementation of layer helper
### FC Layer
We just keep all parameters of a layer function as a dictionary in layer helper as a private data member. Every method of layer helper will look up the dictionary after it is invoked. In that way, we can implement a layer helper for all layer functions even some layer does not contain some operator. For example, The `activation` is used by the FullyConnected layer or convolution layers, but a cross-entropy layer does not use it. The example code of `add_activation` are:
```python
def fc_layer(input, size, ...):
block = program.current_block()
w = block.create_parameter(...)
b = block.create_parameter(...)
out = block.create_var()
op = block.append_operator("FC", X=input, W=w, b=b, out=out)
out.writer = op
return out
class LayerHelper(object):
def __init__(self, **kwargs): # kwargs is short for `keyword arguments`
self.kwargs = kwargs
def add_activation(self, input_var):
act = self.kwargs.get("act", None) # default value is None
<p><codeclass="docutils literal"><spanclass="pre">optimize_op_attrs</span></code> is not in the <codeclass="docutils literal"><spanclass="pre">VarDesc</span></code> message, but kept in the Python instance, as it will be used in the Python space when creating the optimize operator’s <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code>, and will be in the <codeclass="docutils literal"><spanclass="pre">OpDesc</span></code> message.</p>
<p>A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers.</p>
<p>Layer functions take <codeclass="docutils literal"><spanclass="pre">Variable</span></code> and configuration parameters as its input and return the output variable(s).</p>
<p>For example, <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> take one or more variable as its input. The input could be input data or another layer’s output. There are many configuration options for a <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer, such as layer size, activation, parameter names, initialization strategies of parameters, and so on. The <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer will return an output variable.</p>
<spanid="necessity-for-reusing-code-between-layer-functions"></span><h3>Necessity for reusing code between layer functions<aclass="headerlink"href="#necessity-for-reusing-code-between-layer-functions"title="永久链接至标题">¶</a></h3>
<p>There are a lot of code that can be reused. Such as</p>
<ulclass="simple">
<li>Give the default value of configuration. e.g., default initialize strategy for parameters is uniform random with <codeclass="docutils literal"><spanclass="pre">min</span><spanclass="pre">=</span><spanclass="pre">-1.0</span></code>, <codeclass="docutils literal"><spanclass="pre">max</span><spanclass="pre">=</span><spanclass="pre">1.0</span></code>. and default initialize strategy for bias is to fill zero.</li>
<li>Append the activation operator.</li>
<li>Create a temporary variable.</li>
<li>Create parameter.</li>
<li>Generate a unique name.</li>
<li>Add a bias.</li>
<li>...</li>
</ul>
<p>A mechanism to reuse code between layer functions is necessary. It will be around <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/pull/4724/files#diff-823b27e07e93914ada859232ae23f846R12">150 lines of code</a> if we write a <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer without any helper functions.</p>
<spanid="comparision-between-global-functions-and-helper-class"></span><h3>Comparision between global functions and helper class<aclass="headerlink"href="#comparision-between-global-functions-and-helper-class"title="永久链接至标题">¶</a></h3>
<p>The <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> layer will be as follow when we provide global functions:</p>
<p>We can provide many helpers functions for layer developers. However, there are several disadvantages for global helper functions:</p>
<olclass="simple">
<li>We need a namespace for these methods, then layer developers can quickly figure out what method they can use.</li>
<li>Global functions will force layer developers to pass its parameter time by time.</li>
</ol>
<p>So we provide a helper class, <codeclass="docutils literal"><spanclass="pre">LayerHelper</span></code>, to share code between layer functions. The <codeclass="docutils literal"><spanclass="pre">FullyConnected</span></code> Layer will be as follow.</p>
<spanclass="n">helper</span><spanclass="o">=</span><spanclass="n">LayerHelper</span><spanclass="p">(</span><spanclass="nb">locals</span><spanclass="p">())</span><spanclass="c1"># pass all parameter to LayerHelper</span>
<p>The input to the feed operator is a special variable in the global scope, which is the output of <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md">Python readers</a>.</p>
<p>We not only use the fewer lines of code to write <codeclass="docutils literal"><spanclass="pre">fc_layer</span></code> but also make the code clearer to understand. At the same time, layer developers can figure out what function they can invoke by typing <codeclass="docutils literal"><spanclass="pre">helper.</span></code> in a python editor.</p>
<spanid="implementation-of-layer-helper"></span><h3>Implementation of layer helper<aclass="headerlink"href="#implementation-of-layer-helper"title="永久链接至标题">¶</a></h3>
<p>We just keep all parameters of a layer function as a dictionary in layer helper as a private data member. Every method of layer helper will look up the dictionary after it is invoked. In that way, we can implement a layer helper for all layer functions even some layer does not contain some operator. For example, The <codeclass="docutils literal"><spanclass="pre">activation</span></code> is used by the FullyConnected layer or convolution layers, but a cross-entropy layer does not use it. The example code of <codeclass="docutils literal"><spanclass="pre">add_activation</span></code> are:</p>
<spanclass="k">def</span><spanclass="fm">__init__</span><spanclass="p">(</span><spanclass="bp">self</span><spanclass="p">,</span><spanclass="o">**</span><spanclass="n">kwargs</span><spanclass="p">):</span><spanclass="c1"># kwargs is short for `keyword arguments`</span>
<spanclass="n">act</span><spanclass="o">=</span><spanclass="bp">self</span><spanclass="o">.</span><spanclass="n">kwargs</span><spanclass="o">.</span><spanclass="n">get</span><spanclass="p">(</span><spanclass="s2">"act"</span><spanclass="p">,</span><spanclass="bp">None</span><spanclass="p">)</span><spanclass="c1"># default value is None</span>
<spanclass="k">if</span><spanclass="n">act</span><spanclass="ow">is</span><spanclass="bp">None</span><spanclass="p">:</span><spanclass="c1"># do nothing if no act</span>
