A PaddlePaddle program, or a block, is a sequence of operators operating variables. A training program needs to do three kinds of works:
1. the forward pass, which computes intermediate results and the cost(s),
1. the backward pass, which derives gradients from intermediate results and costs, and
1. the optimization pass, which update model parameters to optimize the cost(s).
These works rely on three kinds of operators:
1. forward operators,
1. gradient operators, and
1. optimization operators.
It's true that users should be able to create all these operators manually by calling some low-level API, but it would be much more convenient if they could only describe the forward pass and let PaddlePaddle create the backward and optimization operators automatically.
In this design, we propose a high-level API that automatically derives the optimisation pass and operators from the forward pass.
### High-level Python API to describe the training process
1. User write code to describe the network:
```python
images = layer.data("images")
labels = layer.data("labels")
w1 = pd.var("w1")
b1 = pd.var("b1")
hidden = layer.fc(images, w=w1, b=b1)
cost = layer.mse(hidden, labels)
```
The above code snippet will create forward operators in [Block](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md).
2. Users create a certain kind of Optimizer with some argument.
```python
optimizer = AdagradOptimizer(learing_rate=0.001)
```
3. Users use the optimizer to `minimize` a certain `cost` through updating parameters in parameter_list.
The above code snippet will create gradient and optimization operators in Block. The return value of `minimize()` is list of optimization operators that will be run by session.
4. Users use Session/Executor to run this opt_op_list as target to do training.
<liclass="toctree-l2"><aclass="reference internal"href="../getstarted/build_and_install/index_en.html">Install and Build</a><ul>
<liclass="toctree-l3"><aclass="reference internal"href="../getstarted/build_and_install/docker_install_en.html">PaddlePaddle in Docker Containers</a></li>
<liclass="toctree-l3"><aclass="reference internal"href="../getstarted/build_and_install/build_from_source_en.html">Installing from Sources</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/usage/k8s/k8s_en.html">Paddle On Kubernetes</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/usage/k8s/k8s_aws_en.html">Distributed PaddlePaddle Training on AWS with Kubernetes</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/dev/build_en.html">Build PaddlePaddle from Source Code and Run Unit Test</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../howto/dev/new_layer_en.html">Write New Layers</a></li>
<spanid="optimizer-design"></span><h1>Optimizer Design<aclass="headerlink"href="#optimizer-design"title="Permalink to this headline">¶</a></h1>
<divclass="section"id="the-problem">
<spanid="the-problem"></span><h2>The Problem<aclass="headerlink"href="#the-problem"title="Permalink to this headline">¶</a></h2>
<p>A PaddlePaddle program, or a block, is a sequence of operators operating variables. A training program needs to do three kinds of works:</p>
<olclass="simple">
<li>the forward pass, which computes intermediate results and the cost(s),</li>
<li>the backward pass, which derives gradients from intermediate results and costs, and</li>
<li>the optimization pass, which update model parameters to optimize the cost(s).</li>
</ol>
<p>These works rely on three kinds of operators:</p>
<olclass="simple">
<li>forward operators,</li>
<li>gradient operators, and</li>
<li>optimization operators.</li>
</ol>
<p>It’s true that users should be able to create all these operators manually by calling some low-level API, but it would be much more convenient if they could only describe the forward pass and let PaddlePaddle create the backward and optimization operators automatically.</p>
<p>In this design, we propose a high-level API that automatically derives the optimisation pass and operators from the forward pass.</p>
<spanid="high-level-python-api-to-describe-the-training-process"></span><h2>High-level Python API to describe the training process<aclass="headerlink"href="#high-level-python-api-to-describe-the-training-process"title="Permalink to this headline">¶</a></h2>
<ol>
<li><pclass="first">User write code to describe the network:</p>
<li><pclass="first">Users use the optimizer to <codeclass="docutils literal"><spanclass="pre">minimize</span></code> a certain <codeclass="docutils literal"><spanclass="pre">cost</span></code> through updating parameters in parameter_list.</p>
<p>The above code snippet will create gradient and optimization operators in Block. The return value of <codeclass="docutils literal"><spanclass="pre">minimize()</span></code> is list of optimization operators that will be run by session.</p>
</li>
<li><pclass="first">Users use Session/Executor to run this opt_op_list as target to do training.</p>
<spanid="optimizer-python-interface"></span><h3>Optimizer Python interface:<aclass="headerlink"href="#optimizer-python-interface"title="Permalink to this headline">¶</a></h3>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
A PaddlePaddle program, or a block, is a sequence of operators operating variables. A training program needs to do three kinds of works:
1. the forward pass, which computes intermediate results and the cost(s),
1. the backward pass, which derives gradients from intermediate results and costs, and
1. the optimization pass, which update model parameters to optimize the cost(s).
These works rely on three kinds of operators:
1. forward operators,
1. gradient operators, and
1. optimization operators.
It's true that users should be able to create all these operators manually by calling some low-level API, but it would be much more convenient if they could only describe the forward pass and let PaddlePaddle create the backward and optimization operators automatically.
In this design, we propose a high-level API that automatically derives the optimisation pass and operators from the forward pass.
### High-level Python API to describe the training process
1. User write code to describe the network:
```python
images = layer.data("images")
labels = layer.data("labels")
w1 = pd.var("w1")
b1 = pd.var("b1")
hidden = layer.fc(images, w=w1, b=b1)
cost = layer.mse(hidden, labels)
```
The above code snippet will create forward operators in [Block](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md).
2. Users create a certain kind of Optimizer with some argument.
```python
optimizer = AdagradOptimizer(learing_rate=0.001)
```
3. Users use the optimizer to `minimize` a certain `cost` through updating parameters in parameter_list.
The above code snippet will create gradient and optimization operators in Block. The return value of `minimize()` is list of optimization operators that will be run by session.
4. Users use Session/Executor to run this opt_op_list as target to do training.
<p>A PaddlePaddle program, or a block, is a sequence of operators operating variables. A training program needs to do three kinds of works:</p>
<olclass="simple">
<li>the forward pass, which computes intermediate results and the cost(s),</li>
<li>the backward pass, which derives gradients from intermediate results and costs, and</li>
<li>the optimization pass, which update model parameters to optimize the cost(s).</li>
</ol>
<p>These works rely on three kinds of operators:</p>
<olclass="simple">
<li>forward operators,</li>
<li>gradient operators, and</li>
<li>optimization operators.</li>
</ol>
<p>It’s true that users should be able to create all these operators manually by calling some low-level API, but it would be much more convenient if they could only describe the forward pass and let PaddlePaddle create the backward and optimization operators automatically.</p>
<p>In this design, we propose a high-level API that automatically derives the optimisation pass and operators from the forward pass.</p>
<spanid="high-level-python-api-to-describe-the-training-process"></span><h2>High-level Python API to describe the training process<aclass="headerlink"href="#high-level-python-api-to-describe-the-training-process"title="永久链接至标题">¶</a></h2>
<ol>
<li><pclass="first">User write code to describe the network:</p>
<li><pclass="first">Users use the optimizer to <codeclass="docutils literal"><spanclass="pre">minimize</span></code> a certain <codeclass="docutils literal"><spanclass="pre">cost</span></code> through updating parameters in parameter_list.</p>
<p>The above code snippet will create gradient and optimization operators in Block. The return value of <codeclass="docutils literal"><spanclass="pre">minimize()</span></code> is list of optimization operators that will be run by session.</p>
</li>
<li><pclass="first">Users use Session/Executor to run this opt_op_list as target to do training.</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
