Skip to content

P
Paddle

PaddlePaddle / Paddle
大约 1 年 前同步成功

通知 2299
Star 20931
Fork 5422
P
Paddle
提交 4c2433d2 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: 32b09b51

上级 a5478808
显示空白变更内容
内联 并排
Showing with 470 addition and 1518 deletion
develop/doc/_sources/design/refactor/distributed_architecture.md.txt → develop/doc/_sources/design/dist_refactor/distributed_architecture.md.txt
浏览文件 @ 4c2433d2
......@@ -52,8 +52,9 @@ The IR for PaddlePaddle after refactoring is called a `Block`, it specifies the
The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly.
This could be fixed by making the parameter server run the same computation definition as the trainer (the user's Python module). For a detailed explanation, refer to this document -
[Design Doc: Operation Graph Based Parameter Server](./parameter_server.md)
This could be fixed by making the parameter server also run an IR, which can be different to the trainer side
For a detailed explanation, refer to this document -
[Design Doc: Parameter Server](./parameter_server.md)
## Distributed Training Architecture
......@@ -61,68 +62,111 @@ The revamped distributed training architecture can address the above discussed l
<img src="src/distributed_architecture.png"/>
The major components in the architecture are: *PaddlePaddle Python*, *PaddlePaddle converter* and *PaddlePaddle runtime*.
The major components are: *Python API*, *Distribute Transpiler* and *Remote Executor*.
### PaddlePaddle Python
### Python API
PaddlePaddle Python is the Python library that user's Python code invokes, to read the data. build the neural network topology, start training, etc.
Python API is the Python library that user's Python code invokes, to read the data, build the neural network topology, and start training, etc.
```Python
paddle.init()
input = paddle.op.recordIO("/home/data/mnist.recordio") # file stored on the cluster
img, label = input[0], input[1]
hidden = paddle.layer.fc(input=img, size=200, act=paddle.activation.Tanh())
prediction = paddle.layer.fc(input=img, size=10, act=paddle.activation.Softmax())
cost = paddle.layer.classification_cost(input=prediction, label=label)
optimizer = paddle.optimizer.SGD(cost, learning_rate=0.01)
session = paddle.session.NewRemote(num_trainer=3, num_ps=2, GPU_per_trainer=1)
for i in range(1000):
_, cost_val = session.eval(targets=[cost, optimizer])
print cost_val
images = fluid.layers.data(name='pixel', shape=[1, 28, 28], dtype='float32')
label = fluid.layers.data(name='label', shape=[1], dtype='int64')
...
predict = fluid.layers.fc(input=conv_pool_2, size=10, act="softmax")
cost = fluid.layers.cross_entropy(input=predict, label=label)
avg_cost = fluid.layers.mean(x=cost)
optimizer = fluid.optimizer.Adam(learning_rate=0.01)
optimizer.minimize(avg_cost)
train_reader = paddle.batch(
paddle.reader.shuffle(
paddle.dataset.mnist.train(), buf_size=500),
batch_size=BATCH_SIZE)
place = fluid.CPUPlace()
exe = fluid.Executor(place)
for pass_id in range(10):
for data in train_reader():
loss, acc = exe.run(trainer_prog,
feed=feeder.feed(data),
fetch_list=[avg_cost])
```
The above code is what a typical Python trainer code is, the neural network topology is built using the helper functions such as `paddle.layer.fc`. Training is done by calling `session.eval` iteratively.
#### session.eval
As shown in the graph, `session.eval` sends the IR and the evaluation inputs or targets to the PaddlePaddle cluster for evaluation.
The targets can be any variable in the computation graph. When the target is say, the `optimizer` variable, the neural network will be optimized once. When the target is the `cost` variable, `session.eval` returns the cost value. Based on what the target is, an appropriate action is taken.
The Python `session` is a wrapper of the C++ `Session` class. For more information about `Session`, refer to this document - [Design Doc: Session](./session.md).
### PaddlePaddle Converter
The PaddlePaddle converter automatically converts the IR in the request (IR and evaluation inputs/targets) from PaddlePaddle Python to partitioned IRs and dispatches the new IRs and evaluation inputs/targets to different PaddlePaddle runtimes. Below are the steps that are followed :
1. Add a `feed` OP that feeds the eval inputs, and a `fetch` OP that fetches the eval targets to the IR.
2. Extract a new computation (sub)graph with the `feed` and `fetch` OPs as the boundary. The runtime does not need to run the OP that is not dependent on the `fetch` OP.
3. Optimize the computation graph.
4. Place the OPs in the graph onto different devices on different PaddlePaddle runtime according to a placement algorithm and the device constraints specified by the user.
5. Partition the graph according to runtime boundaries and add `send` / `recv` OP pair on the runtime boundaries.
The code above is a typical local training program, the "Training Program" is built using helper functions such as
`fluid.layer.fc`. The training is done by calling `Executor.run`
iteratively.
For more details, the implementation of IR is [Program](../program.md), and `ProgramDesc` is the protobuf type.
[Executor](../executor.md) simply runs the `ProgramDesc`. For local training you generally use
`Executor` to run the program locally. For any kind of distributed training, you can use
`RemoteExecutor` to specify desired distributed training method with some optional arguments.
### Distributed Transpiler
The Distributed Transpiler automatically converts the IR (in protobuf format) to partitioned IRs. Then
the Remote Executor dispatches the new IRs to Remote Executors across the cluster.
Below are the steps that are followed :
1. User only need to change `Executor` to `RemoteExecutor` to change local program to distributed program.
1. `RemoteExecutor` calls `Distributed Transpiler` to "transpile" user's program to several IRs representing a
distributed training program:
1. Parse configurations from `RemoteExecutor`.
1. Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming.
1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. Take
DataParallelism type for example, it removes the optimization operators and add a `send` OP to the
"trainer" role, then add the optimization operators to the parameter server role within the `recv` OP.
1. Dispatch the partitioned graph to different `RemoteExecutor` in the cluster.
1. `RemoteExecutor` on each node run the received `ProgramDesc` utill the end.
### RemoteExecutor
As shown in the graph, `RemoteExecutor.run` sends the IR to the cluster for Execution.
You can also use parameter `fetch_list` to interactively fetch variable back to local for
log printing.
The Python `RemoteExecutor` is derived from `Executor` class.
```python
exe = RemoteExecutor(
feed=feeder.feed(data),
fetch_list=[avg_cost],
job_desc=JobDesc(
jobname,
num_trainer,
num_pserver,
cpu_per_trainer,
gpu_per_trainer,
mem_per_trainer,
cpu_per_pserver,
mem_per_pserver
))
for data in train_reader():
loss, acc = exe.run(trainer_prog,
feed=feeder.feed(data),
fetch_list=[avg_cost])
```
6. Dispatch the partitioned graph to different PaddlePaddle runtimes.
`JobDesc` object describe the distributed job resource specification to run on
Cluster environment.
7. PaddlePaddle runtimes with the `fetch` OP reports evaluation results back to the converter, the converter reports the evaluation results back to the PaddlePaddle Python.
<img src="src/remote_executor.png"/>
The output IRs will be cached to optimize the conversion latency.
`RemoteExecutor.run` sends the `ProgramDesc` and
[TrainingJob](https://github.com/PaddlePaddle/cloud/blob/develop/doc/autoscale/README.md#training-job-resource)
to a server in the cluster which executes `RemoteExecutor.listen`. This server is responsible
to start the final Kubernetes Jobs to run the different role of `ProgramDesc`.
#### Placement Algorithm
### Placement Algorithm
Our first implementation will only support "trainer-parameter server" placement: the parameters, initializers, and optimizers are all placed on the PaddlePaddle runtimes with the parameter server role. Everything else will be placed on the PaddlePaddle runtimes with the trainer role. This has the same functionality as the "trainer-parameter server" architecture of PaddlePaddle v0.10.0, but is more generic and flexible.
In the future, a more general placement algorithm should be implemented, which makes placements according to the input IR, and a model of device computation time and device communication time. Model parallelism requires the generic placement algorithm.
### PaddlePaddle Runtime
The PaddlePaddle runtime owns multiple devices (e.g., CPUs, GPUs) and runs the IR. The runtime does not need to do OP placement since it is already done by the converter.
### Local Training Architecture
The local training architecture will be the same as the distributed training architecture, the difference is that everything runs locally, and there is just one PaddlePaddle runtime:
......@@ -132,9 +176,18 @@ The local training architecture will be the same as the distributed training arc
### Training Data
In PaddlePaddle v0.10.0, training data is typically read with a [data reader](../reader/README.md) from Python. This approach is no longer efficient when training in a distributed fashion since the Python process no longer runs on the same node with the trainer processes. The Python reader will need to read from the distributed filesystem (assuming it has the required access) and send to the trainers, doubling the network traffic.
When doing distributed training, the user can still use Python data reader: the training data are sent with `session.eval`. However this should be used for debugging purpose only. The users are encouraged to use the read data OPs.
In PaddlePaddle v0.10.0, training data is typically read
with [data reader](../reader/README.md) from Python. This approach is
no longer efficient when training distributedly since the Python
process no longer runs on the same node with the trainer processes,
the Python reader will need to read from the distributed filesystem
(assuming it has the access) and send to the trainers, doubling the
network traffic.
When doing distributed training, the user can still use Python data
reader: the training data are sent with `Executor.run`. However, should
be used for debugging purpose only. The users are encouraged to use
the read data OPs.
## References:
......
develop/doc/_sources/design/refactor/parameter_server.md.txt → develop/doc/_sources/design/dist_refactor/parameter_server.md.txt
浏览文件 @ 4c2433d2
# Design Doc: Operation Graph Based Parameter Server
# Design Doc: Parameter Server
## Abstract
......@@ -10,7 +10,7 @@ different purposes.
## Background
The previous implementations of the parameter server does not run a
subgraph. parameter initialization, optimizer computation, network
fluid sub-program. Parameter initialization, optimizer computation, network
communication and checkpointing are implemented twice on both the
trainer and the parameter server.
......@@ -23,10 +23,10 @@ server becomes a natural extension.
## Design
### Graph Converter
### Distributed Transpiler
The *graph converter* converts the user-defined operation (OP) graph
into subgraphs to be scheduled on different nodes with the following
The *Distributed Transpiler* converts the user-defined fluid program
into sub-programs to be scheduled on different nodes with the following
steps:
1. OP placement: the OPs will be placed on different nodes according
......@@ -34,7 +34,6 @@ steps:
time. Currently we will use a simple heuristic that puts parameter
varable on parameter server workers and everything else on trainer
workers.
1. Add communication OPs to enable the communication between nodes.
We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*.
......@@ -48,8 +47,8 @@ After converting:
<img src="src/dist-graph.png" width="700"/>
1. The parameter variable W and it's optimizer subgraph are placed on the parameter server.
1. Operators are added to the subgraphs.
1. The parameter variable W and it's optimizer program are placed on the parameter server.
1. Operators are added to the program.
- *Send* sends data to the connected *Recv* operator. The
scheduler on the receive node will only schedule *Recv* operator
to run when the *Send* operator has ran (the *Send* OP will mark
......@@ -64,39 +63,30 @@ After converting:
### Benefits
- Model parallelism become easier to implement: it's an extension to
the trainer - parameter server approach. we already have the
communication OPs, but need to extend the graph converter's
placement functionality.
the trainer - parameter server approach. We can have several "Transpilers"
to achieve different goals.
- User-defined optimizer is easier to add - user can now express it as
a subgraph.
a sub-program.
- No more duplication logic inside the trainer and the parameter
server mentioned in the background section.
### Challenges
- It might be hard for the graph converter to cut a general graph
(without any hint for which subgraph is the optimizer). We may need
to label which subgraph inside the OP graph is the optimizer.
- It's important to balance the parameter shards of on multiple
parameter server. If a single parameter is very big (some
word-embedding, fully connected, softmax layer), we need to
automatically partition the single parameter onto different
parameter servers when possible (only element-wise optimizer depends
on the parameter variable).
- In the "Aync SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently. See
[here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more
details about concurrent program in fluid.
### Discussion
- In the "Aync SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently, what is our locking strategy?
E.g., each variable have a lock cpp method to be invoked by every
OP, or, have a lock OP.
- Can the Enqueue OP be implemented under our current tensor design
(puts the input tensor into the queue tensor)?
- *Dequeue* OP will have variable numbers of output (depends on the
`min_count` attribute), does our current design support it? (similar
question for the *Add* OP)
......
develop/doc/_sources/design/refactor/session.md.txt 已删除 100644 → 0
浏览文件 @ a5478808
# Design Doc: Session
## Abstract
The *session* object encapsulates the environment in which the
computation graph is executed.
We will have the *local* session and *remote* session, they offer the
same [interface](#interface). The local session encapsulates the local
runtime environment and the remote session encapsulates the cluster
runtime environment.
The local runtime environment contains:
1. computation devices (i.e., CPU, GPU) handles, and
1. the [scope](../scope.md) which holds all variables.
The remote runtime environment contains:
1. computation devices (i.e., CPU and GPU on node 0, 1) in a cluster,
and
1. the distributed [scope](../scope.md) in a cluster which holds all
variables.
The user can create a remote session on Paddle Cloud and evaluate the
computation graph with it. In this way, the user can control the
remote computation resource in a cluster from his local computer.
## Background
The current design has an implicit global session in which
`paddle.eval()` is executed. The pain point is:
Since the user is not able to explicitly switch between runtime
environments, the user cannot run a topology in two independent
environments.
For example, in reinforcement learning, the user may want to have a
stale model for inference and a fresh model for training, and only
replace the stale model with the fresh model periodically.
Furthermore, we have no concept that encapsulates a remote environment
that executes a computation graph.
We need the session object to address above issues.
## Session
A session is an object that owns the runtime environment. All
computations are executed through `session.eval()`.
### Interface
```python
eval(
targets,
feed_dict=None,
)
```
Evaluates the target Operations or Variables in `targets`.
- *targets*: the evaluation targets. Can be a single Operation or
Variable, or a list with the Operations or Variables as
elements. The value returned by `eval()` has the same shape as the
`target` argument.
The PaddlePaddle program is represented by
the [ProgramDesc](../design/program.md), `eval()` will infer the
ProgramDesc from the given targets and run the PaddlePaddle
program. Please
see
[this graph](./distributed_architecture.md#local-training-architecture) for
the detailed illustration for the local session
and
[this graph](./distributed_architecture.md#distributed-training-architecture) for
the detailed illustration for the remote session.
- *feed_dict*: a dictionary that contains the tensors which override
the edges of the computation graph.
feed_dict not only can provide the input data, it can override any
OP's input as well:
```python
a = pd.constant(2.0, name="a")
b = pd.variable(name="b")
c = pd.mul(a,b)
sess.eval(targets=c, feed_dict={"b":3.0}) # returns 6.0
```
```python
close()
```
Closes the session and releases the scope that the session owns.
### Create a Local Session
```python
session(
devices=None
)
```
Creates a new session. One session owns one global scope, so creating
multiple sessions will create different scopes.
- *devices*: a single `string` or a list of `string` of device names,
the corresponding devices will be the computation devices for
`eval()`. If not specified, all available devices (e.g., all GPUs)
will be used. The user doesn't need to specify the CPU device since
it will be always used. Multiple sessions can use the same device.
#### Example
```Python
a = paddle.constant(1.0)
b = paddle.constant(2.0)
c = a + b
sess = paddle.session(devices=["gpu:0", "gpu:1", "fpga:0"])
sess.eval(c)
sess.close()
```
### Create a Remote Session
```python
create_cloud_job(
name,
num_trainer,
mem_per_trainer,
gpu_per_trainer,
cpu_per_trainer,
num_ps,
mem_per_ps,
cpu_per_ps,
)
```
Creates a Paddle Cloud job. Fails if the job name exists.
```python
get_cloud_job(
name
)
```
Gets a Paddle Cloud job.
```python
remote_session(
job
)
```
- *job*: the Paddle Cloud job.
#### Example
```Python
reader = paddle.reader.recordio("/pfs/home/peter/mnist-train-*") # data stored on Paddle Cloud
image = reader.column(0)
label = reader.column(1)
fc1 = paddle.op.fc(image, size=256, act="sigmoid")
fc2 = paddle.op.fc(fc1, size=10, act="softmax")
cost = paddle.op.cross_entropy(fc2, label)
opt = paddle.optimizer.sgd(cost)
job = paddle.create_cloud_job("test", 3, "1G", 1, 1, 2, "1G", 1)
sess = paddle.remote_ession(job)
for i in range(1000):
sess.eval(opt)
sess.close()
```
develop/doc/design/refactor/distributed_architecture.html → develop/doc/design/dist_refactor/distributed_architecture.html
浏览文件 @ 4c2433d2
......@@ -251,62 +251,109 @@ computation is only specified in Python code which sits outside of PaddlePaddle,
<div class="section" id="limitation-3">
<span id="limitation-3"></span><h3>Limitation 3<a class="headerlink" href="#limitation-3" title="Permalink to this headline"></a></h3>
<p>The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly.</p>
<p>This could be fixed by making the parameter server run the same computation definition as the trainer (the user&#8217;s Python module). For a detailed explanation, refer to this document -
<a class="reference internal" href="parameter_server.html"><span class="doc">Design Doc: Operation Graph Based Parameter Server</span></a></p>
<p>This could be fixed by making the parameter server also run an IR, which can be different to the trainer side
For a detailed explanation, refer to this document -
<a class="reference internal" href="parameter_server.html"><span class="doc">Design Doc: Parameter Server</span></a></p>
</div>
</div>
<div class="section" id="distributed-training-architecture">
<span id="distributed-training-architecture"></span><h2>Distributed Training Architecture<a class="headerlink" href="#distributed-training-architecture" title="Permalink to this headline"></a></h2>
<p>The revamped distributed training architecture can address the above discussed limitations. Below is the illustration of how it does so:</p>
<p><img src="src/distributed_architecture.png"/></p>
<p>The major components in the architecture are: <em>PaddlePaddle Python</em>, <em>PaddlePaddle converter</em> and <em>PaddlePaddle runtime</em>.</p>
<div class="section" id="paddlepaddle-python">
<span id="paddlepaddle-python"></span><h3>PaddlePaddle Python<a class="headerlink" href="#paddlepaddle-python" title="Permalink to this headline"></a></h3>
<p>PaddlePaddle Python is the Python library that user&#8217;s Python code invokes, to read the data. build the neural network topology, start training, etc.</p>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">paddle</span><span class="o">.</span><span class="n">init</span><span class="p">()</span>
<span class="nb">input</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">recordIO</span><span class="p">(</span><span class="s2">&quot;/home/data/mnist.recordio&quot;</span><span class="p">)</span> <span class="c1"># file stored on the cluster</span>
<span class="n">img</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="nb">input</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">input</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">hidden</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">img</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="n">paddle</span><span class="o">.</span><span class="n">activation</span><span class="o">.</span><span class="n">Tanh</span><span class="p">())</span>
<span class="n">prediction</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">img</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="n">paddle</span><span class="o">.</span><span class="n">activation</span><span class="o">.</span><span class="n">Softmax</span><span class="p">())</span>
<span class="n">cost</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">classification_cost</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">prediction</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">cost</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">NewRemote</span><span class="p">(</span><span class="n">num_trainer</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">num_ps</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">GPU_per_trainer</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">):</span>
<span class="n">_</span><span class="p">,</span> <span class="n">cost_val</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">targets</span><span class="o">=</span><span class="p">[</span><span class="n">cost</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">])</span>
<span class="k">print</span> <span class="n">cost_val</span>
<p>The major components are: <em>Python API</em>, <em>Distribute Transpiler</em> and <em>Remote Executor</em>.</p>
<div class="section" id="python-api">
<span id="python-api"></span><h3>Python API<a class="headerlink" href="#python-api" title="Permalink to this headline"></a></h3>
<p>Python API is the Python library that user&#8217;s Python code invokes, to read the data, build the neural network topology, and start training, etc.</p>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;pixel&#39;</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;float32&#39;</span><span class="p">)</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;label&#39;</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;int64&#39;</span><span class="p">)</span>
<span class="o">...</span>
<span class="n">predict</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">conv_pool_2</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="s2">&quot;softmax&quot;</span><span class="p">)</span>
<span class="n">cost</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">predict</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">avg_cost</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="n">cost</span><span class="p">)</span>
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span>
<span class="n">optimizer</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">avg_cost</span><span class="p">)</span>
<span class="n">train_reader</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">batch</span><span class="p">(</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">dataset</span><span class="o">.</span><span class="n">mnist</span><span class="o">.</span><span class="n">train</span><span class="p">(),</span> <span class="n">buf_size</span><span class="o">=</span><span class="mi">500</span><span class="p">),</span>
<span class="n">batch_size</span><span class="o">=</span><span class="n">BATCH_SIZE</span><span class="p">)</span>
<span class="n">place</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">CPUPlace</span><span class="p">()</span>
<span class="n">exe</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">Executor</span><span class="p">(</span><span class="n">place</span><span class="p">)</span>
<span class="k">for</span> <span class="n">pass_id</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
<span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="n">train_reader</span><span class="p">():</span>
<span class="n">loss</span><span class="p">,</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">exe</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">trainer_prog</span><span class="p">,</span>
<span class="n">feed</span><span class="o">=</span><span class="n">feeder</span><span class="o">.</span><span class="n">feed</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">fetch_list</span><span class="o">=</span><span class="p">[</span><span class="n">avg_cost</span><span class="p">])</span>
</pre></div>
</div>
<p>The above code is what a typical Python trainer code is, the neural network topology is built using the helper functions such as <code class="docutils literal"><span class="pre">paddle.layer.fc</span></code>. Training is done by calling <code class="docutils literal"><span class="pre">session.eval</span></code> iteratively.</p>
<div class="section" id="session-eval">
<span id="session-eval"></span><h4>session.eval<a class="headerlink" href="#session-eval" title="Permalink to this headline"></a></h4>
<p>As shown in the graph, <code class="docutils literal"><span class="pre">session.eval</span></code> sends the IR and the evaluation inputs or targets to the PaddlePaddle cluster for evaluation.
The targets can be any variable in the computation graph. When the target is say, the <code class="docutils literal"><span class="pre">optimizer</span></code> variable, the neural network will be optimized once. When the target is the <code class="docutils literal"><span class="pre">cost</span></code> variable, <code class="docutils literal"><span class="pre">session.eval</span></code> returns the cost value. Based on what the target is, an appropriate action is taken.</p>
<p>The Python <code class="docutils literal"><span class="pre">session</span></code> is a wrapper of the C++ <code class="docutils literal"><span class="pre">Session</span></code> class. For more information about <code class="docutils literal"><span class="pre">Session</span></code>, refer to this document - <a class="reference internal" href="session.html"><span class="doc">Design Doc: Session</span></a>.</p>
<p>The code above is a typical local training program, the &#8220;Training Program&#8221; is built using helper functions such as
<code class="docutils literal"><span class="pre">fluid.layer.fc</span></code>. The training is done by calling <code class="docutils literal"><span class="pre">Executor.run</span></code>
iteratively.</p>
<p>For more details, the implementation of IR is <a class="reference internal" href="../program.html"><span class="doc">Program</span></a>, and <code class="docutils literal"><span class="pre">ProgramDesc</span></code> is the protobuf type.</p>
<p><a class="reference internal" href="../executor.html"><span class="doc">Executor</span></a> simply runs the <code class="docutils literal"><span class="pre">ProgramDesc</span></code>. For local training you generally use
<code class="docutils literal"><span class="pre">Executor</span></code> to run the program locally. For any kind of distributed training, you can use
<code class="docutils literal"><span class="pre">RemoteExecutor</span></code> to specify desired distributed training method with some optional arguments.</p>
</div>
</div>
<div class="section" id="paddlepaddle-converter">
<span id="paddlepaddle-converter"></span><h3>PaddlePaddle Converter<a class="headerlink" href="#paddlepaddle-converter" title="Permalink to this headline"></a></h3>
<p>The PaddlePaddle converter automatically converts the IR in the request (IR and evaluation inputs/targets) from PaddlePaddle Python to partitioned IRs and dispatches the new IRs and evaluation inputs/targets to different PaddlePaddle runtimes. Below are the steps that are followed :</p>
<div class="section" id="distributed-transpiler">
<span id="distributed-transpiler"></span><h3>Distributed Transpiler<a class="headerlink" href="#distributed-transpiler" title="Permalink to this headline"></a></h3>
<p>The Distributed Transpiler automatically converts the IR (in protobuf format) to partitioned IRs. Then
the Remote Executor dispatches the new IRs to Remote Executors across the cluster.
Below are the steps that are followed :</p>
<ol class="simple">
<li>Add a <code class="docutils literal"><span class="pre">feed</span></code> OP that feeds the eval inputs, and a <code class="docutils literal"><span class="pre">fetch</span></code> OP that fetches the eval targets to the IR.</li>
<li>Extract a new computation (sub)graph with the <code class="docutils literal"><span class="pre">feed</span></code> and <code class="docutils literal"><span class="pre">fetch</span></code> OPs as the boundary. The runtime does not need to run the OP that is not dependent on the <code class="docutils literal"><span class="pre">fetch</span></code> OP.</li>
<li>Optimize the computation graph.</li>
<li>Place the OPs in the graph onto different devices on different PaddlePaddle runtime according to a placement algorithm and the device constraints specified by the user.</li>
<li>Partition the graph according to runtime boundaries and add <code class="docutils literal"><span class="pre">send</span></code> / <code class="docutils literal"><span class="pre">recv</span></code> OP pair on the runtime boundaries.</li>
<li>Dispatch the partitioned graph to different PaddlePaddle runtimes.</li>
<li>PaddlePaddle runtimes with the <code class="docutils literal"><span class="pre">fetch</span></code> OP reports evaluation results back to the converter, the converter reports the evaluation results back to the PaddlePaddle Python.</li>
<li>User only need to change <code class="docutils literal"><span class="pre">Executor</span></code> to <code class="docutils literal"><span class="pre">RemoteExecutor</span></code> to change local program to distributed program.</li>
<li><code class="docutils literal"><span class="pre">RemoteExecutor</span></code> calls <code class="docutils literal"><span class="pre">Distributed</span> <span class="pre">Transpiler</span></code> to &#8220;transpile&#8221; user&#8217;s program to several IRs representing a
distributed training program:<ol>
<li>Parse configurations from <code class="docutils literal"><span class="pre">RemoteExecutor</span></code>.</li>
<li>Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming.</li>
<li>Partition the <code class="docutils literal"><span class="pre">ProgramDesc</span></code> according to type and add <code class="docutils literal"><span class="pre">send</span></code> / <code class="docutils literal"><span class="pre">recv</span></code> OP pair on the boundaries. Take
DataParallelism type for example, it removes the optimization operators and add a <code class="docutils literal"><span class="pre">send</span></code> OP to the
&#8220;trainer&#8221; role, then add the optimization operators to the parameter server role within the <code class="docutils literal"><span class="pre">recv</span></code> OP.</li>
</ol>
</li>
<li>Dispatch the partitioned graph to different <code class="docutils literal"><span class="pre">RemoteExecutor</span></code> in the cluster.</li>
<li><code class="docutils literal"><span class="pre">RemoteExecutor</span></code> on each node run the received <code class="docutils literal"><span class="pre">ProgramDesc</span></code> utill the end.</li>
</ol>
<p>The output IRs will be cached to optimize the conversion latency.</p>
</div>
<div class="section" id="remoteexecutor">
<span id="remoteexecutor"></span><h3>RemoteExecutor<a class="headerlink" href="#remoteexecutor" title="Permalink to this headline"></a></h3>
<p>As shown in the graph, <code class="docutils literal"><span class="pre">RemoteExecutor.run</span></code> sends the IR to the cluster for Execution.
You can also use parameter <code class="docutils literal"><span class="pre">fetch_list</span></code> to interactively fetch variable back to local for
log printing.</p>
<p>The Python <code class="docutils literal"><span class="pre">RemoteExecutor</span></code> is derived from <code class="docutils literal"><span class="pre">Executor</span></code> class.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">exe</span> <span class="o">=</span> <span class="n">RemoteExecutor</span><span class="p">(</span>
<span class="n">feed</span><span class="o">=</span><span class="n">feeder</span><span class="o">.</span><span class="n">feed</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">fetch_list</span><span class="o">=</span><span class="p">[</span><span class="n">avg_cost</span><span class="p">],</span>
<span class="n">job_desc</span><span class="o">=</span><span class="n">JobDesc</span><span class="p">(</span>
<span class="n">jobname</span><span class="p">,</span>
<span class="n">num_trainer</span><span class="p">,</span>
<span class="n">num_pserver</span><span class="p">,</span>
<span class="n">cpu_per_trainer</span><span class="p">,</span>
<span class="n">gpu_per_trainer</span><span class="p">,</span>
<span class="n">mem_per_trainer</span><span class="p">,</span>
<span class="n">cpu_per_pserver</span><span class="p">,</span>
<span class="n">mem_per_pserver</span>
<span class="p">))</span>
<span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="n">train_reader</span><span class="p">():</span>
<span class="n">loss</span><span class="p">,</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">exe</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">trainer_prog</span><span class="p">,</span>
<span class="n">feed</span><span class="o">=</span><span class="n">feeder</span><span class="o">.</span><span class="n">feed</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">fetch_list</span><span class="o">=</span><span class="p">[</span><span class="n">avg_cost</span><span class="p">])</span>
</pre></div>
</div>
<p><code class="docutils literal"><span class="pre">JobDesc</span></code> object describe the distributed job resource specification to run on
Cluster environment.</p>
<p><img src="src/remote_executor.png"/></p>
<p><code class="docutils literal"><span class="pre">RemoteExecutor.run</span></code> sends the <code class="docutils literal"><span class="pre">ProgramDesc</span></code> and
<a class="reference external" href="https://github.com/PaddlePaddle/cloud/blob/develop/doc/autoscale/README.md#training-job-resource">TrainingJob</a>
to a server in the cluster which executes <code class="docutils literal"><span class="pre">RemoteExecutor.listen</span></code>. This server is responsible
to start the final Kubernetes Jobs to run the different role of <code class="docutils literal"><span class="pre">ProgramDesc</span></code>.</p>
</div>
<div class="section" id="placement-algorithm">
<span id="placement-algorithm"></span><h4>Placement Algorithm<a class="headerlink" href="#placement-algorithm" title="Permalink to this headline"></a></h4>
<span id="placement-algorithm"></span><h3>Placement Algorithm<a class="headerlink" href="#placement-algorithm" title="Permalink to this headline"></a></h3>
<p>Our first implementation will only support &#8220;trainer-parameter server&#8221; placement: the parameters, initializers, and optimizers are all placed on the PaddlePaddle runtimes with the parameter server role. Everything else will be placed on the PaddlePaddle runtimes with the trainer role. This has the same functionality as the &#8220;trainer-parameter server&#8221; architecture of PaddlePaddle v0.10.0, but is more generic and flexible.</p>
<p>In the future, a more general placement algorithm should be implemented, which makes placements according to the input IR, and a model of device computation time and device communication time. Model parallelism requires the generic placement algorithm.</p>
</div>
</div>
<div class="section" id="paddlepaddle-runtime">
<span id="paddlepaddle-runtime"></span><h3>PaddlePaddle Runtime<a class="headerlink" href="#paddlepaddle-runtime" title="Permalink to this headline"></a></h3>
<p>The PaddlePaddle runtime owns multiple devices (e.g., CPUs, GPUs) and runs the IR. The runtime does not need to do OP placement since it is already done by the converter.</p>
</div>
<div class="section" id="local-training-architecture">
<span id="local-training-architecture"></span><h3>Local Training Architecture<a class="headerlink" href="#local-training-architecture" title="Permalink to this headline"></a></h3>
<p>The local training architecture will be the same as the distributed training architecture, the difference is that everything runs locally, and there is just one PaddlePaddle runtime:</p>
......@@ -314,8 +361,17 @@ The targets can be any variable in the computation graph. When the target is say
</div>
<div class="section" id="training-data">
<span id="training-data"></span><h3>Training Data<a class="headerlink" href="#training-data" title="Permalink to this headline"></a></h3>
<p>In PaddlePaddle v0.10.0, training data is typically read with a <a class="reference internal" href="../reader/README.html"><span class="doc">data reader</span></a> from Python. This approach is no longer efficient when training in a distributed fashion since the Python process no longer runs on the same node with the trainer processes. The Python reader will need to read from the distributed filesystem (assuming it has the required access) and send to the trainers, doubling the network traffic.</p>
<p>When doing distributed training, the user can still use Python data reader: the training data are sent with <code class="docutils literal"><span class="pre">session.eval</span></code>. However this should be used for debugging purpose only. The users are encouraged to use the read data OPs.</p>
<p>In PaddlePaddle v0.10.0, training data is typically read
with <a class="reference internal" href="../reader/README.html"><span class="doc">data reader</span></a> from Python. This approach is
no longer efficient when training distributedly since the Python
process no longer runs on the same node with the trainer processes,
the Python reader will need to read from the distributed filesystem
(assuming it has the access) and send to the trainers, doubling the
network traffic.</p>
<p>When doing distributed training, the user can still use Python data
reader: the training data are sent with <code class="docutils literal"><span class="pre">Executor.run</span></code>. However, should
be used for debugging purpose only. The users are encouraged to use
the read data OPs.</p>
</div>
</div>
<div class="section" id="references">
......
develop/doc/design/refactor/parameter_server.html → develop/doc/design/dist_refactor/parameter_server.html
浏览文件 @ 4c2433d2
......@@ -8,7 +8,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Design Doc: Operation Graph Based Parameter Server &mdash; PaddlePaddle documentation</title>
<title>Design Doc: Parameter Server &mdash; PaddlePaddle documentation</title>
......@@ -199,7 +199,7 @@
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li>Design Doc: Operation Graph Based Parameter Server</li>
<li>Design Doc: Parameter Server</li>
</ul>
</div>
......@@ -208,8 +208,8 @@
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="design-doc-operation-graph-based-parameter-server">
<span id="design-doc-operation-graph-based-parameter-server"></span><h1>Design Doc: Operation Graph Based Parameter Server<a class="headerlink" href="#design-doc-operation-graph-based-parameter-server" title="Permalink to this headline"></a></h1>
<div class="section" id="design-doc-parameter-server">
<span id="design-doc-parameter-server"></span><h1>Design Doc: Parameter Server<a class="headerlink" href="#design-doc-parameter-server" title="Permalink to this headline"></a></h1>
<div class="section" id="abstract">
<span id="abstract"></span><h2>Abstract<a class="headerlink" href="#abstract" title="Permalink to this headline"></a></h2>
<p>We propose an approach to implement the parameter server. In this
......@@ -220,7 +220,7 @@ different purposes.</p>
<div class="section" id="background">
<span id="background"></span><h2>Background<a class="headerlink" href="#background" title="Permalink to this headline"></a></h2>
<p>The previous implementations of the parameter server does not run a
subgraph. parameter initialization, optimizer computation, network
fluid sub-program. Parameter initialization, optimizer computation, network
communication and checkpointing are implemented twice on both the
trainer and the parameter server.</p>
<p>It would be great if we can write code once and use them on both the
......@@ -232,10 +232,10 @@ server becomes a natural extension.</p>
</div>
<div class="section" id="design">
<span id="design"></span><h2>Design<a class="headerlink" href="#design" title="Permalink to this headline"></a></h2>
<div class="section" id="graph-converter">
<span id="graph-converter"></span><h3>Graph Converter<a class="headerlink" href="#graph-converter" title="Permalink to this headline"></a></h3>
<p>The <em>graph converter</em> converts the user-defined operation (OP) graph
into subgraphs to be scheduled on different nodes with the following
<div class="section" id="distributed-transpiler">
<span id="distributed-transpiler"></span><h3>Distributed Transpiler<a class="headerlink" href="#distributed-transpiler" title="Permalink to this headline"></a></h3>
<p>The <em>Distributed Transpiler</em> converts the user-defined fluid program
into sub-programs to be scheduled on different nodes with the following
steps:</p>
<ol class="simple">
<li>OP placement: the OPs will be placed on different nodes according
......@@ -252,8 +252,8 @@ subgraphs for the trainer and the parameter server:</p>
<p>After converting:</p>
<p><img src="src/dist-graph.png" width="700"/></p>
<ol class="simple">
<li>The parameter variable W and it&#8217;s optimizer subgraph are placed on the parameter server.</li>
<li>Operators are added to the subgraphs.<ul>
<li>The parameter variable W and it&#8217;s optimizer program are placed on the parameter server.</li>
<li>Operators are added to the program.<ul>
<li><em>Send</em> sends data to the connected <em>Recv</em> operator. The
scheduler on the receive node will only schedule <em>Recv</em> operator
to run when the <em>Send</em> operator has ran (the <em>Send</em> OP will mark
......@@ -271,11 +271,10 @@ tensors.</li>
<span id="benefits"></span><h3>Benefits<a class="headerlink" href="#benefits" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li>Model parallelism become easier to implement: it&#8217;s an extension to
the trainer - parameter server approach. we already have the
communication OPs, but need to extend the graph converter&#8217;s
placement functionality.</li>
the trainer - parameter server approach. We can have several &#8220;Transpilers&#8221;
to achieve different goals.</li>
<li>User-defined optimizer is easier to add - user can now express it as
a subgraph.</li>
a sub-program.</li>
<li>No more duplication logic inside the trainer and the parameter
server mentioned in the background section.</li>
</ul>
......@@ -283,24 +282,21 @@ server mentioned in the background section.</li>
<div class="section" id="challenges">
<span id="challenges"></span><h3>Challenges<a class="headerlink" href="#challenges" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li>It might be hard for the graph converter to cut a general graph
(without any hint for which subgraph is the optimizer). We may need
to label which subgraph inside the OP graph is the optimizer.</li>
<li>It&#8217;s important to balance the parameter shards of on multiple
parameter server. If a single parameter is very big (some
word-embedding, fully connected, softmax layer), we need to
automatically partition the single parameter onto different
parameter servers when possible (only element-wise optimizer depends
on the parameter variable).</li>
<li>In the &#8220;Aync SGD&#8221; figure, the &#8220;W&#8221; variable on the parameter server
could be read and wrote concurrently. See
<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/pull/6394">here</a> for more
details about concurrent program in fluid.</li>
</ul>
</div>
<div class="section" id="discussion">
<span id="discussion"></span><h3>Discussion<a class="headerlink" href="#discussion" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li>In the &#8220;Aync SGD&#8221; figure, the &#8220;W&#8221; variable on the parameter server
could be read and wrote concurrently, what is our locking strategy?
E.g., each variable have a lock cpp method to be invoked by every
OP, or, have a lock OP.</li>
<li>Can the Enqueue OP be implemented under our current tensor design
(puts the input tensor into the queue tensor)?</li>
<li><em>Dequeue</em> OP will have variable numbers of output (depends on the
......
develop/doc/design/refactor/session.html 已删除 100644 → 0
浏览文件 @ a5478808
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Design Doc: Session &mdash; PaddlePaddle documentation</title>
<link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
<link rel="index" title="Index"
href="../../genindex.html"/>
<link rel="search" title="Search" href="../../search.html"/>
<link rel="top" title="PaddlePaddle documentation" href="../../index.html"/>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/css/perfect-scrollbar.min.css" type="text/css" />
<link rel="stylesheet" href="../../_static/css/override.css" type="text/css" />
<script>
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
hm.src = "//hm.baidu.com/hm.js?b9a314ab40d04d805655aab1deee08ba";
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
</script>
<script src="../../_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav" role="document">
<header class="site-header">
<div class="site-logo">
<a href="/"><img src="../../_static/images/PP_w.png"></a>
</div>
<div class="site-nav-links">
<div class="site-menu">
<a class="fork-on-github" href="https://github.com/PaddlePaddle/Paddle" target="_blank"><i class="fa fa-github"></i>Fork me on Github</a>
<div class="language-switcher dropdown">
<a type="button" data-toggle="dropdown">
<span>English</span>
<i class="fa fa-angle-up"></i>
<i class="fa fa-angle-down"></i>
</a>
<ul class="dropdown-menu">
<li><a href="/doc_cn">中文</a></li>
<li><a href="/doc">English</a></li>
</ul>
</div>
<ul class="site-page-links">
<li><a href="/">Home</a></li>
</ul>
</div>
<div class="doc-module">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../getstarted/index_en.html">GET STARTED</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../howto/index_en.html">HOW TO</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/index_en.html">API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mobile/index_en.html">MOBILE</a></li>
</ul>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
</div>
</header>
<div class="main-content-wrap">
<nav class="doc-menu-vertical" role="navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../getstarted/index_en.html">GET STARTED</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../getstarted/build_and_install/index_en.html">Install and Build</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../getstarted/build_and_install/pip_install_en.html">Install Using pip</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../getstarted/build_and_install/docker_install_en.html">Run in Docker Containers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/dev/build_en.html">Build using Docker</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../getstarted/build_and_install/build_from_source_en.html">Build from Sources</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../howto/index_en.html">HOW TO</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../howto/usage/cmd_parameter/index_en.html">Set Command-line Parameters</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cmd_parameter/use_case_en.html">Use Case</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cmd_parameter/arguments_en.html">Argument Outline</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cmd_parameter/detail_introduction_en.html">Detail Description</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/usage/cluster/cluster_train_en.html">Distributed Training</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/fabric_en.html">fabric</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/openmpi_en.html">openmpi</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/k8s_en.html">kubernetes</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/k8s_aws_en.html">kubernetes on AWS</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/dev/new_layer_en.html">Write New Layers</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/dev/contribute_to_paddle_en.html">Contribute Code</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/dev/write_docs_en.html">Contribute Documentation</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/deep_model/rnn/index_en.html">RNN Models</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../howto/deep_model/rnn/rnn_config_en.html">RNN Configuration</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/optimization/gpu_profiling_en.html">Tune GPU Performance</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../api/index_en.html">API</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/model_configs.html">Model Configuration</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/activation.html">Activation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/layer.html">Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/evaluators.html">Evaluators</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/optimizer.html">Optimizer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/pooling.html">Pooling</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/networks.html">Networks</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/attr.html">Parameter Attribute</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/data.html">Data Reader Interface and DataSets</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/data/data_reader.html">Data Reader Interface</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/data/image.html">Image Interface</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/data/dataset.html">Dataset</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/run_logic.html">Training and Inference</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/fluid.html">Fluid</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/layers.html">Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/data_feeder.html">DataFeeder</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/executor.html">Executor</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/initializer.html">Initializer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/evaluator.html">Evaluator</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/nets.html">Nets</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/optimizer.html">Optimizer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/param_attr.html">ParamAttr</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/profiler.html">Profiler</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/regularizer.html">Regularizer</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../mobile/index_en.html">MOBILE</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../mobile/cross_compiling_for_android_en.html">Build PaddlePaddle for Android</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mobile/cross_compiling_for_ios_en.html">Build PaddlePaddle for iOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mobile/cross_compiling_for_raspberry_en.html">Build PaddlePaddle for Raspberry Pi</a></li>
</ul>
</li>
</ul>
</nav>
<section class="doc-content-wrap">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li>Design Doc: Session</li>
</ul>
</div>
<div class="wy-nav-content" id="doc-content">
<div class="rst-content">
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="design-doc-session">
<span id="design-doc-session"></span><h1>Design Doc: Session<a class="headerlink" href="#design-doc-session" title="Permalink to this headline"></a></h1>
<div class="section" id="abstract">
<span id="abstract"></span><h2>Abstract<a class="headerlink" href="#abstract" title="Permalink to this headline"></a></h2>
<p>The <em>session</em> object encapsulates the environment in which the
computation graph is executed.</p>
<p>We will have the <em>local</em> session and <em>remote</em> session, they offer the
same <a class="reference external" href="#interface">interface</a>. The local session encapsulates the local
runtime environment and the remote session encapsulates the cluster
runtime environment.</p>
<p>The local runtime environment contains:</p>
<ol class="simple">
<li>computation devices (i.e., CPU, GPU) handles, and</li>
<li>the <a class="reference internal" href="../scope.html"><span class="doc">scope</span></a> which holds all variables.</li>
</ol>
<p>The remote runtime environment contains:</p>
<ol class="simple">
<li>computation devices (i.e., CPU and GPU on node 0, 1) in a cluster,
and</li>
<li>the distributed <a class="reference internal" href="../scope.html"><span class="doc">scope</span></a> in a cluster which holds all
variables.</li>
</ol>
<p>The user can create a remote session on Paddle Cloud and evaluate the
computation graph with it. In this way, the user can control the
remote computation resource in a cluster from his local computer.</p>
</div>
<div class="section" id="background">
<span id="background"></span><h2>Background<a class="headerlink" href="#background" title="Permalink to this headline"></a></h2>
<p>The current design has an implicit global session in which
<code class="docutils literal"><span class="pre">paddle.eval()</span></code> is executed. The pain point is:</p>
<p>Since the user is not able to explicitly switch between runtime
environments, the user cannot run a topology in two independent
environments.</p>
<p>For example, in reinforcement learning, the user may want to have a
stale model for inference and a fresh model for training, and only
replace the stale model with the fresh model periodically.</p>
<p>Furthermore, we have no concept that encapsulates a remote environment
that executes a computation graph.</p>
<p>We need the session object to address above issues.</p>
</div>
<div class="section" id="session">
<span id="session"></span><h2>Session<a class="headerlink" href="#session" title="Permalink to this headline"></a></h2>
<p>A session is an object that owns the runtime environment. All
computations are executed through <code class="docutils literal"><span class="pre">session.eval()</span></code>.</p>
<div class="section" id="interface">
<span id="interface"></span><h3>Interface<a class="headerlink" href="#interface" title="Permalink to this headline"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="nb">eval</span><span class="p">(</span>
<span class="n">targets</span><span class="p">,</span>
<span class="n">feed_dict</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Evaluates the target Operations or Variables in <code class="docutils literal"><span class="pre">targets</span></code>.</p>
<ul>
<li><p class="first"><em>targets</em>: the evaluation targets. Can be a single Operation or
Variable, or a list with the Operations or Variables as
elements. The value returned by <code class="docutils literal"><span class="pre">eval()</span></code> has the same shape as the
<code class="docutils literal"><span class="pre">target</span></code> argument.</p>
<p>The PaddlePaddle program is represented by
the <a class="reference external" href="design/design/program.md">ProgramDesc</a>, <code class="docutils literal"><span class="pre">eval()</span></code> will infer the
ProgramDesc from the given targets and run the PaddlePaddle
program. Please
see
<a class="reference external" href="/design/refactor/distributed_architecture.html#local-training-architecture">this graph</a> for
the detailed illustration for the local session
and
<a class="reference external" href="/design/refactor/distributed_architecture.html#distributed-training-architecture">this graph</a> for
the detailed illustration for the remote session.</p>
</li>
<li><p class="first"><em>feed_dict</em>: a dictionary that contains the tensors which override
the edges of the computation graph.</p>
<p>feed_dict not only can provide the input data, it can override any
OP&#8217;s input as well:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="mf">2.0</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;a&quot;</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">variable</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;b&quot;</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">mul</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">)</span>
<span class="n">sess</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">targets</span><span class="o">=</span><span class="n">c</span><span class="p">,</span> <span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;b&quot;</span><span class="p">:</span><span class="mf">3.0</span><span class="p">})</span> <span class="c1"># returns 6.0</span>
</pre></div>
</div>
</li>
</ul>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>Closes the session and releases the scope that the session owns.</p>
</div>
<div class="section" id="create-a-local-session">
<span id="create-a-local-session"></span><h3>Create a Local Session<a class="headerlink" href="#create-a-local-session" title="Permalink to this headline"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">session</span><span class="p">(</span>
<span class="n">devices</span><span class="o">=</span><span class="bp">None</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Creates a new session. One session owns one global scope, so creating
multiple sessions will create different scopes.</p>
<ul class="simple">
<li><em>devices</em>: a single <code class="docutils literal"><span class="pre">string</span></code> or a list of <code class="docutils literal"><span class="pre">string</span></code> of device names,
the corresponding devices will be the computation devices for
<code class="docutils literal"><span class="pre">eval()</span></code>. If not specified, all available devices (e.g., all GPUs)
will be used. The user doesn&#8217;t need to specify the CPU device since
it will be always used. Multiple sessions can use the same device.</li>
</ul>
<div class="section" id="example">
<span id="example"></span><h4>Example<a class="headerlink" href="#example" title="Permalink to this headline"></a></h4>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="mf">2.0</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="n">sess</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">session</span><span class="p">(</span><span class="n">devices</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;gpu:0&quot;</span><span class="p">,</span> <span class="s2">&quot;gpu:1&quot;</span><span class="p">,</span> <span class="s2">&quot;fpga:0&quot;</span><span class="p">])</span>
<span class="n">sess</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="n">sess</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="create-a-remote-session">
<span id="create-a-remote-session"></span><h3>Create a Remote Session<a class="headerlink" href="#create-a-remote-session" title="Permalink to this headline"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">create_cloud_job</span><span class="p">(</span>
<span class="n">name</span><span class="p">,</span>
<span class="n">num_trainer</span><span class="p">,</span>
<span class="n">mem_per_trainer</span><span class="p">,</span>
<span class="n">gpu_per_trainer</span><span class="p">,</span>
<span class="n">cpu_per_trainer</span><span class="p">,</span>
<span class="n">num_ps</span><span class="p">,</span>
<span class="n">mem_per_ps</span><span class="p">,</span>
<span class="n">cpu_per_ps</span><span class="p">,</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Creates a Paddle Cloud job. Fails if the job name exists.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">get_cloud_job</span><span class="p">(</span>
<span class="n">name</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Gets a Paddle Cloud job.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">remote_session</span><span class="p">(</span>
<span class="n">job</span>
<span class="p">)</span>
</pre></div>
</div>
<ul class="simple">
<li><em>job</em>: the Paddle Cloud job.</li>
</ul>
<div class="section" id="example">
<span id="id1"></span><h4>Example<a class="headerlink" href="#example" title="Permalink to this headline"></a></h4>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">reader</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">recordio</span><span class="p">(</span><span class="s2">&quot;/pfs/home/peter/mnist-train-*&quot;</span><span class="p">)</span> <span class="c1"># data stored on Paddle Cloud</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">fc1</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="s2">&quot;sigmoid&quot;</span><span class="p">)</span>
<span class="n">fc2</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">fc1</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="s2">&quot;softmax&quot;</span><span class="p">)</span>
<span class="n">cost</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="n">fc2</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span>
<span class="n">opt</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">sgd</span><span class="p">(</span><span class="n">cost</span><span class="p">)</span>
<span class="n">job</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">create_cloud_job</span><span class="p">(</span><span class="s2">&quot;test&quot;</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="s2">&quot;1G&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="s2">&quot;1G&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">sess</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">remote_ession</span><span class="p">(</span><span class="n">job</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">):</span>
<span class="n">sess</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">opt</span><span class="p">)</span>
<span class="n">sess</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>
&copy; Copyright 2016, PaddlePaddle developers.
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'../../',
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: ".txt",
};
</script>
<script type="text/javascript" src="../../_static/jquery.js"></script>
<script type="text/javascript" src="../../_static/underscore.js"></script>
<script type="text/javascript" src="../../_static/doctools.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/javascript" src="../../_static/js/theme.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/js/perfect-scrollbar.jquery.min.js"></script>
<script src="../../_static/js/paddle_doc_init.js"></script>
</body>
</html>
\ No newline at end of file
develop/doc/searchindex.js
浏览文件 @ 4c2433d2
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc_cn/_sources/design/refactor/distributed_architecture.md.txt → develop/doc_cn/_sources/design/dist_refactor/distributed_architecture.md.txt
浏览文件 @ 4c2433d2
......@@ -52,8 +52,9 @@ The IR for PaddlePaddle after refactoring is called a `Block`, it specifies the
The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly.
This could be fixed by making the parameter server run the same computation definition as the trainer (the user's Python module). For a detailed explanation, refer to this document -
[Design Doc: Operation Graph Based Parameter Server](./parameter_server.md)
This could be fixed by making the parameter server also run an IR, which can be different to the trainer side
For a detailed explanation, refer to this document -
[Design Doc: Parameter Server](./parameter_server.md)
## Distributed Training Architecture
......@@ -61,68 +62,111 @@ The revamped distributed training architecture can address the above discussed l
<img src="src/distributed_architecture.png"/>
The major components in the architecture are: *PaddlePaddle Python*, *PaddlePaddle converter* and *PaddlePaddle runtime*.
The major components are: *Python API*, *Distribute Transpiler* and *Remote Executor*.
### PaddlePaddle Python
### Python API
PaddlePaddle Python is the Python library that user's Python code invokes, to read the data. build the neural network topology, start training, etc.
Python API is the Python library that user's Python code invokes, to read the data, build the neural network topology, and start training, etc.
```Python
paddle.init()
input = paddle.op.recordIO("/home/data/mnist.recordio") # file stored on the cluster
img, label = input[0], input[1]
hidden = paddle.layer.fc(input=img, size=200, act=paddle.activation.Tanh())
prediction = paddle.layer.fc(input=img, size=10, act=paddle.activation.Softmax())
cost = paddle.layer.classification_cost(input=prediction, label=label)
optimizer = paddle.optimizer.SGD(cost, learning_rate=0.01)
session = paddle.session.NewRemote(num_trainer=3, num_ps=2, GPU_per_trainer=1)
for i in range(1000):
_, cost_val = session.eval(targets=[cost, optimizer])
print cost_val
images = fluid.layers.data(name='pixel', shape=[1, 28, 28], dtype='float32')
label = fluid.layers.data(name='label', shape=[1], dtype='int64')
...
predict = fluid.layers.fc(input=conv_pool_2, size=10, act="softmax")
cost = fluid.layers.cross_entropy(input=predict, label=label)
avg_cost = fluid.layers.mean(x=cost)
optimizer = fluid.optimizer.Adam(learning_rate=0.01)
optimizer.minimize(avg_cost)
train_reader = paddle.batch(
paddle.reader.shuffle(
paddle.dataset.mnist.train(), buf_size=500),
batch_size=BATCH_SIZE)
place = fluid.CPUPlace()
exe = fluid.Executor(place)
for pass_id in range(10):
for data in train_reader():
loss, acc = exe.run(trainer_prog,
feed=feeder.feed(data),
fetch_list=[avg_cost])
```
The above code is what a typical Python trainer code is, the neural network topology is built using the helper functions such as `paddle.layer.fc`. Training is done by calling `session.eval` iteratively.
#### session.eval
As shown in the graph, `session.eval` sends the IR and the evaluation inputs or targets to the PaddlePaddle cluster for evaluation.
The targets can be any variable in the computation graph. When the target is say, the `optimizer` variable, the neural network will be optimized once. When the target is the `cost` variable, `session.eval` returns the cost value. Based on what the target is, an appropriate action is taken.
The Python `session` is a wrapper of the C++ `Session` class. For more information about `Session`, refer to this document - [Design Doc: Session](./session.md).
### PaddlePaddle Converter
The PaddlePaddle converter automatically converts the IR in the request (IR and evaluation inputs/targets) from PaddlePaddle Python to partitioned IRs and dispatches the new IRs and evaluation inputs/targets to different PaddlePaddle runtimes. Below are the steps that are followed :
1. Add a `feed` OP that feeds the eval inputs, and a `fetch` OP that fetches the eval targets to the IR.
2. Extract a new computation (sub)graph with the `feed` and `fetch` OPs as the boundary. The runtime does not need to run the OP that is not dependent on the `fetch` OP.
3. Optimize the computation graph.
4. Place the OPs in the graph onto different devices on different PaddlePaddle runtime according to a placement algorithm and the device constraints specified by the user.
5. Partition the graph according to runtime boundaries and add `send` / `recv` OP pair on the runtime boundaries.
The code above is a typical local training program, the "Training Program" is built using helper functions such as
`fluid.layer.fc`. The training is done by calling `Executor.run`
iteratively.
For more details, the implementation of IR is [Program](../program.md), and `ProgramDesc` is the protobuf type.
[Executor](../executor.md) simply runs the `ProgramDesc`. For local training you generally use
`Executor` to run the program locally. For any kind of distributed training, you can use
`RemoteExecutor` to specify desired distributed training method with some optional arguments.
### Distributed Transpiler
The Distributed Transpiler automatically converts the IR (in protobuf format) to partitioned IRs. Then
the Remote Executor dispatches the new IRs to Remote Executors across the cluster.
Below are the steps that are followed :
1. User only need to change `Executor` to `RemoteExecutor` to change local program to distributed program.
1. `RemoteExecutor` calls `Distributed Transpiler` to "transpile" user's program to several IRs representing a
distributed training program:
1. Parse configurations from `RemoteExecutor`.
1. Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming.
1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. Take
DataParallelism type for example, it removes the optimization operators and add a `send` OP to the
"trainer" role, then add the optimization operators to the parameter server role within the `recv` OP.
1. Dispatch the partitioned graph to different `RemoteExecutor` in the cluster.
1. `RemoteExecutor` on each node run the received `ProgramDesc` utill the end.
### RemoteExecutor
As shown in the graph, `RemoteExecutor.run` sends the IR to the cluster for Execution.
You can also use parameter `fetch_list` to interactively fetch variable back to local for
log printing.
The Python `RemoteExecutor` is derived from `Executor` class.
```python
exe = RemoteExecutor(
feed=feeder.feed(data),
fetch_list=[avg_cost],
job_desc=JobDesc(
jobname,
num_trainer,
num_pserver,
cpu_per_trainer,
gpu_per_trainer,
mem_per_trainer,
cpu_per_pserver,
mem_per_pserver
))
for data in train_reader():
loss, acc = exe.run(trainer_prog,
feed=feeder.feed(data),
fetch_list=[avg_cost])
```
6. Dispatch the partitioned graph to different PaddlePaddle runtimes.
`JobDesc` object describe the distributed job resource specification to run on
Cluster environment.
7. PaddlePaddle runtimes with the `fetch` OP reports evaluation results back to the converter, the converter reports the evaluation results back to the PaddlePaddle Python.
<img src="src/remote_executor.png"/>
The output IRs will be cached to optimize the conversion latency.
`RemoteExecutor.run` sends the `ProgramDesc` and
[TrainingJob](https://github.com/PaddlePaddle/cloud/blob/develop/doc/autoscale/README.md#training-job-resource)
to a server in the cluster which executes `RemoteExecutor.listen`. This server is responsible
to start the final Kubernetes Jobs to run the different role of `ProgramDesc`.
#### Placement Algorithm
### Placement Algorithm
Our first implementation will only support "trainer-parameter server" placement: the parameters, initializers, and optimizers are all placed on the PaddlePaddle runtimes with the parameter server role. Everything else will be placed on the PaddlePaddle runtimes with the trainer role. This has the same functionality as the "trainer-parameter server" architecture of PaddlePaddle v0.10.0, but is more generic and flexible.
In the future, a more general placement algorithm should be implemented, which makes placements according to the input IR, and a model of device computation time and device communication time. Model parallelism requires the generic placement algorithm.
### PaddlePaddle Runtime
The PaddlePaddle runtime owns multiple devices (e.g., CPUs, GPUs) and runs the IR. The runtime does not need to do OP placement since it is already done by the converter.
### Local Training Architecture
The local training architecture will be the same as the distributed training architecture, the difference is that everything runs locally, and there is just one PaddlePaddle runtime:
......@@ -132,9 +176,18 @@ The local training architecture will be the same as the distributed training arc
### Training Data
In PaddlePaddle v0.10.0, training data is typically read with a [data reader](../reader/README.md) from Python. This approach is no longer efficient when training in a distributed fashion since the Python process no longer runs on the same node with the trainer processes. The Python reader will need to read from the distributed filesystem (assuming it has the required access) and send to the trainers, doubling the network traffic.
When doing distributed training, the user can still use Python data reader: the training data are sent with `session.eval`. However this should be used for debugging purpose only. The users are encouraged to use the read data OPs.
In PaddlePaddle v0.10.0, training data is typically read
with [data reader](../reader/README.md) from Python. This approach is
no longer efficient when training distributedly since the Python
process no longer runs on the same node with the trainer processes,
the Python reader will need to read from the distributed filesystem
(assuming it has the access) and send to the trainers, doubling the
network traffic.
When doing distributed training, the user can still use Python data
reader: the training data are sent with `Executor.run`. However, should
be used for debugging purpose only. The users are encouraged to use
the read data OPs.
## References:
......
develop/doc_cn/_sources/design/refactor/parameter_server.md.txt → develop/doc_cn/_sources/design/dist_refactor/parameter_server.md.txt
浏览文件 @ 4c2433d2
# Design Doc: Operation Graph Based Parameter Server
# Design Doc: Parameter Server
## Abstract
......@@ -10,7 +10,7 @@ different purposes.
## Background
The previous implementations of the parameter server does not run a
subgraph. parameter initialization, optimizer computation, network
fluid sub-program. Parameter initialization, optimizer computation, network
communication and checkpointing are implemented twice on both the
trainer and the parameter server.
......@@ -23,10 +23,10 @@ server becomes a natural extension.
## Design
### Graph Converter
### Distributed Transpiler
The *graph converter* converts the user-defined operation (OP) graph
into subgraphs to be scheduled on different nodes with the following
The *Distributed Transpiler* converts the user-defined fluid program
into sub-programs to be scheduled on different nodes with the following
steps:
1. OP placement: the OPs will be placed on different nodes according
......@@ -34,7 +34,6 @@ steps:
time. Currently we will use a simple heuristic that puts parameter
varable on parameter server workers and everything else on trainer
workers.
1. Add communication OPs to enable the communication between nodes.
We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*.
......@@ -48,8 +47,8 @@ After converting:
<img src="src/dist-graph.png" width="700"/>
1. The parameter variable W and it's optimizer subgraph are placed on the parameter server.
1. Operators are added to the subgraphs.
1. The parameter variable W and it's optimizer program are placed on the parameter server.
1. Operators are added to the program.
- *Send* sends data to the connected *Recv* operator. The
scheduler on the receive node will only schedule *Recv* operator
to run when the *Send* operator has ran (the *Send* OP will mark
......@@ -64,39 +63,30 @@ After converting:
### Benefits
- Model parallelism become easier to implement: it's an extension to
the trainer - parameter server approach. we already have the
communication OPs, but need to extend the graph converter's
placement functionality.
the trainer - parameter server approach. We can have several "Transpilers"
to achieve different goals.
- User-defined optimizer is easier to add - user can now express it as
a subgraph.
a sub-program.
- No more duplication logic inside the trainer and the parameter
server mentioned in the background section.
### Challenges
- It might be hard for the graph converter to cut a general graph
(without any hint for which subgraph is the optimizer). We may need
to label which subgraph inside the OP graph is the optimizer.
- It's important to balance the parameter shards of on multiple
parameter server. If a single parameter is very big (some
word-embedding, fully connected, softmax layer), we need to
automatically partition the single parameter onto different
parameter servers when possible (only element-wise optimizer depends
on the parameter variable).
- In the "Aync SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently. See
[here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more
details about concurrent program in fluid.
### Discussion
- In the "Aync SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently, what is our locking strategy?
E.g., each variable have a lock cpp method to be invoked by every
OP, or, have a lock OP.
- Can the Enqueue OP be implemented under our current tensor design
(puts the input tensor into the queue tensor)?
- *Dequeue* OP will have variable numbers of output (depends on the
`min_count` attribute), does our current design support it? (similar
question for the *Add* OP)
......
develop/doc_cn/_sources/design/refactor/session.md.txt 已删除 100644 → 0
浏览文件 @ a5478808
# Design Doc: Session
## Abstract
The *session* object encapsulates the environment in which the
computation graph is executed.
We will have the *local* session and *remote* session, they offer the
same [interface](#interface). The local session encapsulates the local
runtime environment and the remote session encapsulates the cluster
runtime environment.
The local runtime environment contains:
1. computation devices (i.e., CPU, GPU) handles, and
1. the [scope](../scope.md) which holds all variables.
The remote runtime environment contains:
1. computation devices (i.e., CPU and GPU on node 0, 1) in a cluster,
and
1. the distributed [scope](../scope.md) in a cluster which holds all
variables.
The user can create a remote session on Paddle Cloud and evaluate the
computation graph with it. In this way, the user can control the
remote computation resource in a cluster from his local computer.
## Background
The current design has an implicit global session in which
`paddle.eval()` is executed. The pain point is:
Since the user is not able to explicitly switch between runtime
environments, the user cannot run a topology in two independent
environments.
For example, in reinforcement learning, the user may want to have a
stale model for inference and a fresh model for training, and only
replace the stale model with the fresh model periodically.
Furthermore, we have no concept that encapsulates a remote environment
that executes a computation graph.
We need the session object to address above issues.
## Session
A session is an object that owns the runtime environment. All
computations are executed through `session.eval()`.
### Interface
```python
eval(
targets,
feed_dict=None,
)
```
Evaluates the target Operations or Variables in `targets`.
- *targets*: the evaluation targets. Can be a single Operation or
Variable, or a list with the Operations or Variables as
elements. The value returned by `eval()` has the same shape as the
`target` argument.
The PaddlePaddle program is represented by
the [ProgramDesc](../design/program.md), `eval()` will infer the
ProgramDesc from the given targets and run the PaddlePaddle
program. Please
see
[this graph](./distributed_architecture.md#local-training-architecture) for
the detailed illustration for the local session
and
[this graph](./distributed_architecture.md#distributed-training-architecture) for
the detailed illustration for the remote session.
- *feed_dict*: a dictionary that contains the tensors which override
the edges of the computation graph.
feed_dict not only can provide the input data, it can override any
OP's input as well:
```python
a = pd.constant(2.0, name="a")
b = pd.variable(name="b")
c = pd.mul(a,b)
sess.eval(targets=c, feed_dict={"b":3.0}) # returns 6.0
```
```python
close()
```
Closes the session and releases the scope that the session owns.
### Create a Local Session
```python
session(
devices=None
)
```
Creates a new session. One session owns one global scope, so creating
multiple sessions will create different scopes.
- *devices*: a single `string` or a list of `string` of device names,
the corresponding devices will be the computation devices for
`eval()`. If not specified, all available devices (e.g., all GPUs)
will be used. The user doesn't need to specify the CPU device since
it will be always used. Multiple sessions can use the same device.
#### Example
```Python
a = paddle.constant(1.0)
b = paddle.constant(2.0)
c = a + b
sess = paddle.session(devices=["gpu:0", "gpu:1", "fpga:0"])
sess.eval(c)
sess.close()
```
### Create a Remote Session
```python
create_cloud_job(
name,
num_trainer,
mem_per_trainer,
gpu_per_trainer,
cpu_per_trainer,
num_ps,
mem_per_ps,
cpu_per_ps,
)
```
Creates a Paddle Cloud job. Fails if the job name exists.
```python
get_cloud_job(
name
)
```
Gets a Paddle Cloud job.
```python
remote_session(
job
)
```
- *job*: the Paddle Cloud job.
#### Example
```Python
reader = paddle.reader.recordio("/pfs/home/peter/mnist-train-*") # data stored on Paddle Cloud
image = reader.column(0)
label = reader.column(1)
fc1 = paddle.op.fc(image, size=256, act="sigmoid")
fc2 = paddle.op.fc(fc1, size=10, act="softmax")
cost = paddle.op.cross_entropy(fc2, label)
opt = paddle.optimizer.sgd(cost)
job = paddle.create_cloud_job("test", 3, "1G", 1, 1, 2, "1G", 1)
sess = paddle.remote_ession(job)
for i in range(1000):
sess.eval(opt)
sess.close()
```
develop/doc_cn/design/refactor/distributed_architecture.html → develop/doc_cn/design/dist_refactor/distributed_architecture.html
浏览文件 @ 4c2433d2
......@@ -264,62 +264,109 @@ computation is only specified in Python code which sits outside of PaddlePaddle,
<div class="section" id="limitation-3">
<span id="limitation-3"></span><h3>Limitation 3<a class="headerlink" href="#limitation-3" title="永久链接至标题"></a></h3>
<p>The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly.</p>
<p>This could be fixed by making the parameter server run the same computation definition as the trainer (the user&#8217;s Python module). For a detailed explanation, refer to this document -
<a class="reference internal" href="parameter_server.html"><span class="doc">Design Doc: Operation Graph Based Parameter Server</span></a></p>
<p>This could be fixed by making the parameter server also run an IR, which can be different to the trainer side
For a detailed explanation, refer to this document -
<a class="reference internal" href="parameter_server.html"><span class="doc">Design Doc: Parameter Server</span></a></p>
</div>
</div>
<div class="section" id="distributed-training-architecture">
<span id="distributed-training-architecture"></span><h2>Distributed Training Architecture<a class="headerlink" href="#distributed-training-architecture" title="永久链接至标题"></a></h2>
<p>The revamped distributed training architecture can address the above discussed limitations. Below is the illustration of how it does so:</p>
<p><img src="src/distributed_architecture.png"/></p>
<p>The major components in the architecture are: <em>PaddlePaddle Python</em>, <em>PaddlePaddle converter</em> and <em>PaddlePaddle runtime</em>.</p>
<div class="section" id="paddlepaddle-python">
<span id="paddlepaddle-python"></span><h3>PaddlePaddle Python<a class="headerlink" href="#paddlepaddle-python" title="永久链接至标题"></a></h3>
<p>PaddlePaddle Python is the Python library that user&#8217;s Python code invokes, to read the data. build the neural network topology, start training, etc.</p>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">paddle</span><span class="o">.</span><span class="n">init</span><span class="p">()</span>
<span class="nb">input</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">recordIO</span><span class="p">(</span><span class="s2">&quot;/home/data/mnist.recordio&quot;</span><span class="p">)</span> <span class="c1"># file stored on the cluster</span>
<span class="n">img</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="nb">input</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">input</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">hidden</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">img</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="n">paddle</span><span class="o">.</span><span class="n">activation</span><span class="o">.</span><span class="n">Tanh</span><span class="p">())</span>
<span class="n">prediction</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">img</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="n">paddle</span><span class="o">.</span><span class="n">activation</span><span class="o">.</span><span class="n">Softmax</span><span class="p">())</span>
<span class="n">cost</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">classification_cost</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">prediction</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">cost</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">NewRemote</span><span class="p">(</span><span class="n">num_trainer</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">num_ps</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">GPU_per_trainer</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">):</span>
<span class="n">_</span><span class="p">,</span> <span class="n">cost_val</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">targets</span><span class="o">=</span><span class="p">[</span><span class="n">cost</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">])</span>
<span class="k">print</span> <span class="n">cost_val</span>
<p>The major components are: <em>Python API</em>, <em>Distribute Transpiler</em> and <em>Remote Executor</em>.</p>
<div class="section" id="python-api">
<span id="python-api"></span><h3>Python API<a class="headerlink" href="#python-api" title="永久链接至标题"></a></h3>
<p>Python API is the Python library that user&#8217;s Python code invokes, to read the data, build the neural network topology, and start training, etc.</p>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;pixel&#39;</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;float32&#39;</span><span class="p">)</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;label&#39;</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;int64&#39;</span><span class="p">)</span>
<span class="o">...</span>
<span class="n">predict</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">conv_pool_2</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="s2">&quot;softmax&quot;</span><span class="p">)</span>
<span class="n">cost</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">predict</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">avg_cost</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">layers</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="n">cost</span><span class="p">)</span>
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span>
<span class="n">optimizer</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">avg_cost</span><span class="p">)</span>
<span class="n">train_reader</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">batch</span><span class="p">(</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span>
<span class="n">paddle</span><span class="o">.</span><span class="n">dataset</span><span class="o">.</span><span class="n">mnist</span><span class="o">.</span><span class="n">train</span><span class="p">(),</span> <span class="n">buf_size</span><span class="o">=</span><span class="mi">500</span><span class="p">),</span>
<span class="n">batch_size</span><span class="o">=</span><span class="n">BATCH_SIZE</span><span class="p">)</span>
<span class="n">place</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">CPUPlace</span><span class="p">()</span>
<span class="n">exe</span> <span class="o">=</span> <span class="n">fluid</span><span class="o">.</span><span class="n">Executor</span><span class="p">(</span><span class="n">place</span><span class="p">)</span>
<span class="k">for</span> <span class="n">pass_id</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
<span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="n">train_reader</span><span class="p">():</span>
<span class="n">loss</span><span class="p">,</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">exe</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">trainer_prog</span><span class="p">,</span>
<span class="n">feed</span><span class="o">=</span><span class="n">feeder</span><span class="o">.</span><span class="n">feed</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">fetch_list</span><span class="o">=</span><span class="p">[</span><span class="n">avg_cost</span><span class="p">])</span>
</pre></div>
</div>
<p>The above code is what a typical Python trainer code is, the neural network topology is built using the helper functions such as <code class="docutils literal"><span class="pre">paddle.layer.fc</span></code>. Training is done by calling <code class="docutils literal"><span class="pre">session.eval</span></code> iteratively.</p>
<div class="section" id="session-eval">
<span id="session-eval"></span><h4>session.eval<a class="headerlink" href="#session-eval" title="永久链接至标题"></a></h4>
<p>As shown in the graph, <code class="docutils literal"><span class="pre">session.eval</span></code> sends the IR and the evaluation inputs or targets to the PaddlePaddle cluster for evaluation.
The targets can be any variable in the computation graph. When the target is say, the <code class="docutils literal"><span class="pre">optimizer</span></code> variable, the neural network will be optimized once. When the target is the <code class="docutils literal"><span class="pre">cost</span></code> variable, <code class="docutils literal"><span class="pre">session.eval</span></code> returns the cost value. Based on what the target is, an appropriate action is taken.</p>
<p>The Python <code class="docutils literal"><span class="pre">session</span></code> is a wrapper of the C++ <code class="docutils literal"><span class="pre">Session</span></code> class. For more information about <code class="docutils literal"><span class="pre">Session</span></code>, refer to this document - <a class="reference internal" href="session.html"><span class="doc">Design Doc: Session</span></a>.</p>
<p>The code above is a typical local training program, the &#8220;Training Program&#8221; is built using helper functions such as
<code class="docutils literal"><span class="pre">fluid.layer.fc</span></code>. The training is done by calling <code class="docutils literal"><span class="pre">Executor.run</span></code>
iteratively.</p>
<p>For more details, the implementation of IR is <a class="reference internal" href="../program.html"><span class="doc">Program</span></a>, and <code class="docutils literal"><span class="pre">ProgramDesc</span></code> is the protobuf type.</p>
<p><a class="reference internal" href="../executor.html"><span class="doc">Executor</span></a> simply runs the <code class="docutils literal"><span class="pre">ProgramDesc</span></code>. For local training you generally use
<code class="docutils literal"><span class="pre">Executor</span></code> to run the program locally. For any kind of distributed training, you can use
<code class="docutils literal"><span class="pre">RemoteExecutor</span></code> to specify desired distributed training method with some optional arguments.</p>
</div>
</div>
<div class="section" id="paddlepaddle-converter">
<span id="paddlepaddle-converter"></span><h3>PaddlePaddle Converter<a class="headerlink" href="#paddlepaddle-converter" title="永久链接至标题"></a></h3>
<p>The PaddlePaddle converter automatically converts the IR in the request (IR and evaluation inputs/targets) from PaddlePaddle Python to partitioned IRs and dispatches the new IRs and evaluation inputs/targets to different PaddlePaddle runtimes. Below are the steps that are followed :</p>
<div class="section" id="distributed-transpiler">
<span id="distributed-transpiler"></span><h3>Distributed Transpiler<a class="headerlink" href="#distributed-transpiler" title="永久链接至标题"></a></h3>
<p>The Distributed Transpiler automatically converts the IR (in protobuf format) to partitioned IRs. Then
the Remote Executor dispatches the new IRs to Remote Executors across the cluster.
Below are the steps that are followed :</p>
<ol class="simple">
<li>Add a <code class="docutils literal"><span class="pre">feed</span></code> OP that feeds the eval inputs, and a <code class="docutils literal"><span class="pre">fetch</span></code> OP that fetches the eval targets to the IR.</li>
<li>Extract a new computation (sub)graph with the <code class="docutils literal"><span class="pre">feed</span></code> and <code class="docutils literal"><span class="pre">fetch</span></code> OPs as the boundary. The runtime does not need to run the OP that is not dependent on the <code class="docutils literal"><span class="pre">fetch</span></code> OP.</li>
<li>Optimize the computation graph.</li>
<li>Place the OPs in the graph onto different devices on different PaddlePaddle runtime according to a placement algorithm and the device constraints specified by the user.</li>
<li>Partition the graph according to runtime boundaries and add <code class="docutils literal"><span class="pre">send</span></code> / <code class="docutils literal"><span class="pre">recv</span></code> OP pair on the runtime boundaries.</li>
<li>Dispatch the partitioned graph to different PaddlePaddle runtimes.</li>
<li>PaddlePaddle runtimes with the <code class="docutils literal"><span class="pre">fetch</span></code> OP reports evaluation results back to the converter, the converter reports the evaluation results back to the PaddlePaddle Python.</li>
<li>User only need to change <code class="docutils literal"><span class="pre">Executor</span></code> to <code class="docutils literal"><span class="pre">RemoteExecutor</span></code> to change local program to distributed program.</li>
<li><code class="docutils literal"><span class="pre">RemoteExecutor</span></code> calls <code class="docutils literal"><span class="pre">Distributed</span> <span class="pre">Transpiler</span></code> to &#8220;transpile&#8221; user&#8217;s program to several IRs representing a
distributed training program:<ol>
<li>Parse configurations from <code class="docutils literal"><span class="pre">RemoteExecutor</span></code>.</li>
<li>Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming.</li>
<li>Partition the <code class="docutils literal"><span class="pre">ProgramDesc</span></code> according to type and add <code class="docutils literal"><span class="pre">send</span></code> / <code class="docutils literal"><span class="pre">recv</span></code> OP pair on the boundaries. Take
DataParallelism type for example, it removes the optimization operators and add a <code class="docutils literal"><span class="pre">send</span></code> OP to the
&#8220;trainer&#8221; role, then add the optimization operators to the parameter server role within the <code class="docutils literal"><span class="pre">recv</span></code> OP.</li>
</ol>
</li>
<li>Dispatch the partitioned graph to different <code class="docutils literal"><span class="pre">RemoteExecutor</span></code> in the cluster.</li>
<li><code class="docutils literal"><span class="pre">RemoteExecutor</span></code> on each node run the received <code class="docutils literal"><span class="pre">ProgramDesc</span></code> utill the end.</li>
</ol>
<p>The output IRs will be cached to optimize the conversion latency.</p>
</div>
<div class="section" id="remoteexecutor">
<span id="remoteexecutor"></span><h3>RemoteExecutor<a class="headerlink" href="#remoteexecutor" title="永久链接至标题"></a></h3>
<p>As shown in the graph, <code class="docutils literal"><span class="pre">RemoteExecutor.run</span></code> sends the IR to the cluster for Execution.
You can also use parameter <code class="docutils literal"><span class="pre">fetch_list</span></code> to interactively fetch variable back to local for
log printing.</p>
<p>The Python <code class="docutils literal"><span class="pre">RemoteExecutor</span></code> is derived from <code class="docutils literal"><span class="pre">Executor</span></code> class.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">exe</span> <span class="o">=</span> <span class="n">RemoteExecutor</span><span class="p">(</span>
<span class="n">feed</span><span class="o">=</span><span class="n">feeder</span><span class="o">.</span><span class="n">feed</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">fetch_list</span><span class="o">=</span><span class="p">[</span><span class="n">avg_cost</span><span class="p">],</span>
<span class="n">job_desc</span><span class="o">=</span><span class="n">JobDesc</span><span class="p">(</span>
<span class="n">jobname</span><span class="p">,</span>
<span class="n">num_trainer</span><span class="p">,</span>
<span class="n">num_pserver</span><span class="p">,</span>
<span class="n">cpu_per_trainer</span><span class="p">,</span>
<span class="n">gpu_per_trainer</span><span class="p">,</span>
<span class="n">mem_per_trainer</span><span class="p">,</span>
<span class="n">cpu_per_pserver</span><span class="p">,</span>
<span class="n">mem_per_pserver</span>
<span class="p">))</span>
<span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="n">train_reader</span><span class="p">():</span>
<span class="n">loss</span><span class="p">,</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">exe</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">trainer_prog</span><span class="p">,</span>
<span class="n">feed</span><span class="o">=</span><span class="n">feeder</span><span class="o">.</span><span class="n">feed</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">fetch_list</span><span class="o">=</span><span class="p">[</span><span class="n">avg_cost</span><span class="p">])</span>
</pre></div>
</div>
<p><code class="docutils literal"><span class="pre">JobDesc</span></code> object describe the distributed job resource specification to run on
Cluster environment.</p>
<p><img src="src/remote_executor.png"/></p>
<p><code class="docutils literal"><span class="pre">RemoteExecutor.run</span></code> sends the <code class="docutils literal"><span class="pre">ProgramDesc</span></code> and
<a class="reference external" href="https://github.com/PaddlePaddle/cloud/blob/develop/doc/autoscale/README.md#training-job-resource">TrainingJob</a>
to a server in the cluster which executes <code class="docutils literal"><span class="pre">RemoteExecutor.listen</span></code>. This server is responsible
to start the final Kubernetes Jobs to run the different role of <code class="docutils literal"><span class="pre">ProgramDesc</span></code>.</p>
</div>
<div class="section" id="placement-algorithm">
<span id="placement-algorithm"></span><h4>Placement Algorithm<a class="headerlink" href="#placement-algorithm" title="永久链接至标题"></a></h4>
<span id="placement-algorithm"></span><h3>Placement Algorithm<a class="headerlink" href="#placement-algorithm" title="永久链接至标题"></a></h3>
<p>Our first implementation will only support &#8220;trainer-parameter server&#8221; placement: the parameters, initializers, and optimizers are all placed on the PaddlePaddle runtimes with the parameter server role. Everything else will be placed on the PaddlePaddle runtimes with the trainer role. This has the same functionality as the &#8220;trainer-parameter server&#8221; architecture of PaddlePaddle v0.10.0, but is more generic and flexible.</p>
<p>In the future, a more general placement algorithm should be implemented, which makes placements according to the input IR, and a model of device computation time and device communication time. Model parallelism requires the generic placement algorithm.</p>
</div>
</div>
<div class="section" id="paddlepaddle-runtime">
<span id="paddlepaddle-runtime"></span><h3>PaddlePaddle Runtime<a class="headerlink" href="#paddlepaddle-runtime" title="永久链接至标题"></a></h3>
<p>The PaddlePaddle runtime owns multiple devices (e.g., CPUs, GPUs) and runs the IR. The runtime does not need to do OP placement since it is already done by the converter.</p>
</div>
<div class="section" id="local-training-architecture">
<span id="local-training-architecture"></span><h3>Local Training Architecture<a class="headerlink" href="#local-training-architecture" title="永久链接至标题"></a></h3>
<p>The local training architecture will be the same as the distributed training architecture, the difference is that everything runs locally, and there is just one PaddlePaddle runtime:</p>
......@@ -327,8 +374,17 @@ The targets can be any variable in the computation graph. When the target is say
</div>
<div class="section" id="training-data">
<span id="training-data"></span><h3>Training Data<a class="headerlink" href="#training-data" title="永久链接至标题"></a></h3>
<p>In PaddlePaddle v0.10.0, training data is typically read with a <a class="reference internal" href="../reader/README.html"><span class="doc">data reader</span></a> from Python. This approach is no longer efficient when training in a distributed fashion since the Python process no longer runs on the same node with the trainer processes. The Python reader will need to read from the distributed filesystem (assuming it has the required access) and send to the trainers, doubling the network traffic.</p>
<p>When doing distributed training, the user can still use Python data reader: the training data are sent with <code class="docutils literal"><span class="pre">session.eval</span></code>. However this should be used for debugging purpose only. The users are encouraged to use the read data OPs.</p>
<p>In PaddlePaddle v0.10.0, training data is typically read
with <a class="reference internal" href="../reader/README.html"><span class="doc">data reader</span></a> from Python. This approach is
no longer efficient when training distributedly since the Python
process no longer runs on the same node with the trainer processes,
the Python reader will need to read from the distributed filesystem
(assuming it has the access) and send to the trainers, doubling the
network traffic.</p>
<p>When doing distributed training, the user can still use Python data
reader: the training data are sent with <code class="docutils literal"><span class="pre">Executor.run</span></code>. However, should
be used for debugging purpose only. The users are encouraged to use
the read data OPs.</p>
</div>
</div>
<div class="section" id="references">
......
develop/doc_cn/design/refactor/parameter_server.html → develop/doc_cn/design/dist_refactor/parameter_server.html
浏览文件 @ 4c2433d2
......@@ -8,7 +8,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Design Doc: Operation Graph Based Parameter Server &mdash; PaddlePaddle 文档</title>
<title>Design Doc: Parameter Server &mdash; PaddlePaddle 文档</title>
......@@ -212,7 +212,7 @@
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li>Design Doc: Operation Graph Based Parameter Server</li>
<li>Design Doc: Parameter Server</li>
</ul>
</div>
......@@ -221,8 +221,8 @@
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="design-doc-operation-graph-based-parameter-server">
<span id="design-doc-operation-graph-based-parameter-server"></span><h1>Design Doc: Operation Graph Based Parameter Server<a class="headerlink" href="#design-doc-operation-graph-based-parameter-server" title="永久链接至标题"></a></h1>
<div class="section" id="design-doc-parameter-server">
<span id="design-doc-parameter-server"></span><h1>Design Doc: Parameter Server<a class="headerlink" href="#design-doc-parameter-server" title="永久链接至标题"></a></h1>
<div class="section" id="abstract">
<span id="abstract"></span><h2>Abstract<a class="headerlink" href="#abstract" title="永久链接至标题"></a></h2>
<p>We propose an approach to implement the parameter server. In this
......@@ -233,7 +233,7 @@ different purposes.</p>
<div class="section" id="background">
<span id="background"></span><h2>Background<a class="headerlink" href="#background" title="永久链接至标题"></a></h2>
<p>The previous implementations of the parameter server does not run a
subgraph. parameter initialization, optimizer computation, network
fluid sub-program. Parameter initialization, optimizer computation, network
communication and checkpointing are implemented twice on both the
trainer and the parameter server.</p>
<p>It would be great if we can write code once and use them on both the
......@@ -245,10 +245,10 @@ server becomes a natural extension.</p>
</div>
<div class="section" id="design">
<span id="design"></span><h2>Design<a class="headerlink" href="#design" title="永久链接至标题"></a></h2>
<div class="section" id="graph-converter">
<span id="graph-converter"></span><h3>Graph Converter<a class="headerlink" href="#graph-converter" title="永久链接至标题"></a></h3>
<p>The <em>graph converter</em> converts the user-defined operation (OP) graph
into subgraphs to be scheduled on different nodes with the following
<div class="section" id="distributed-transpiler">
<span id="distributed-transpiler"></span><h3>Distributed Transpiler<a class="headerlink" href="#distributed-transpiler" title="永久链接至标题"></a></h3>
<p>The <em>Distributed Transpiler</em> converts the user-defined fluid program
into sub-programs to be scheduled on different nodes with the following
steps:</p>
<ol class="simple">
<li>OP placement: the OPs will be placed on different nodes according
......@@ -265,8 +265,8 @@ subgraphs for the trainer and the parameter server:</p>
<p>After converting:</p>
<p><img src="src/dist-graph.png" width="700"/></p>
<ol class="simple">
<li>The parameter variable W and it&#8217;s optimizer subgraph are placed on the parameter server.</li>
<li>Operators are added to the subgraphs.<ul>
<li>The parameter variable W and it&#8217;s optimizer program are placed on the parameter server.</li>
<li>Operators are added to the program.<ul>
<li><em>Send</em> sends data to the connected <em>Recv</em> operator. The
scheduler on the receive node will only schedule <em>Recv</em> operator
to run when the <em>Send</em> operator has ran (the <em>Send</em> OP will mark
......@@ -284,11 +284,10 @@ tensors.</li>
<span id="benefits"></span><h3>Benefits<a class="headerlink" href="#benefits" title="永久链接至标题"></a></h3>
<ul class="simple">
<li>Model parallelism become easier to implement: it&#8217;s an extension to
the trainer - parameter server approach. we already have the
communication OPs, but need to extend the graph converter&#8217;s
placement functionality.</li>
the trainer - parameter server approach. We can have several &#8220;Transpilers&#8221;
to achieve different goals.</li>
<li>User-defined optimizer is easier to add - user can now express it as
a subgraph.</li>
a sub-program.</li>
<li>No more duplication logic inside the trainer and the parameter
server mentioned in the background section.</li>
</ul>
......@@ -296,24 +295,21 @@ server mentioned in the background section.</li>
<div class="section" id="challenges">
<span id="challenges"></span><h3>Challenges<a class="headerlink" href="#challenges" title="永久链接至标题"></a></h3>
<ul class="simple">
<li>It might be hard for the graph converter to cut a general graph
(without any hint for which subgraph is the optimizer). We may need
to label which subgraph inside the OP graph is the optimizer.</li>
<li>It&#8217;s important to balance the parameter shards of on multiple
parameter server. If a single parameter is very big (some
word-embedding, fully connected, softmax layer), we need to
automatically partition the single parameter onto different
parameter servers when possible (only element-wise optimizer depends
on the parameter variable).</li>
<li>In the &#8220;Aync SGD&#8221; figure, the &#8220;W&#8221; variable on the parameter server
could be read and wrote concurrently. See
<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/pull/6394">here</a> for more
details about concurrent program in fluid.</li>
</ul>
</div>
<div class="section" id="discussion">
<span id="discussion"></span><h3>Discussion<a class="headerlink" href="#discussion" title="永久链接至标题"></a></h3>
<ul class="simple">
<li>In the &#8220;Aync SGD&#8221; figure, the &#8220;W&#8221; variable on the parameter server
could be read and wrote concurrently, what is our locking strategy?
E.g., each variable have a lock cpp method to be invoked by every
OP, or, have a lock OP.</li>
<li>Can the Enqueue OP be implemented under our current tensor design
(puts the input tensor into the queue tensor)?</li>
<li><em>Dequeue</em> OP will have variable numbers of output (depends on the
......
develop/doc_cn/design/refactor/session.html 已删除 100644 → 0
浏览文件 @ a5478808
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Design Doc: Session &mdash; PaddlePaddle 文档</title>
<link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
<link rel="index" title="索引"
href="../../genindex.html"/>
<link rel="search" title="搜索" href="../../search.html"/>
<link rel="top" title="PaddlePaddle 文档" href="../../index.html"/>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/css/perfect-scrollbar.min.css" type="text/css" />
<link rel="stylesheet" href="../../_static/css/override.css" type="text/css" />
<script>
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
hm.src = "//hm.baidu.com/hm.js?b9a314ab40d04d805655aab1deee08ba";
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
</script>
<script src="../../_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav" role="document">
<header class="site-header">
<div class="site-logo">
<a href="/"><img src="../../_static/images/PP_w.png"></a>
</div>
<div class="site-nav-links">
<div class="site-menu">
<a class="fork-on-github" href="https://github.com/PaddlePaddle/Paddle" target="_blank"><i class="fa fa-github"></i>Fork me on Github</a>
<div class="language-switcher dropdown">
<a type="button" data-toggle="dropdown">
<span>English</span>
<i class="fa fa-angle-up"></i>
<i class="fa fa-angle-down"></i>
</a>
<ul class="dropdown-menu">
<li><a href="/doc_cn">中文</a></li>
<li><a href="/doc">English</a></li>
</ul>
</div>
<ul class="site-page-links">
<li><a href="/">Home</a></li>
</ul>
</div>
<div class="doc-module">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../getstarted/index_cn.html">新手入门</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../howto/index_cn.html">进阶指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/index_cn.html">API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../faq/index_cn.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mobile/index_cn.html">MOBILE</a></li>
</ul>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
</div>
</header>
<div class="main-content-wrap">
<nav class="doc-menu-vertical" role="navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../getstarted/index_cn.html">新手入门</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../getstarted/build_and_install/index_cn.html">安装与编译</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../getstarted/build_and_install/pip_install_cn.html">使用pip安装</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../getstarted/build_and_install/docker_install_cn.html">使用Docker安装运行</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/dev/build_cn.html">用Docker编译和测试PaddlePaddle</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../getstarted/build_and_install/build_from_source_cn.html">从源码编译</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../getstarted/concepts/use_concepts_cn.html">基本使用概念</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../howto/index_cn.html">进阶指南</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../howto/usage/cmd_parameter/index_cn.html">设置命令行参数</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cmd_parameter/use_case_cn.html">使用案例</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cmd_parameter/arguments_cn.html">参数概述</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cmd_parameter/detail_introduction_cn.html">细节描述</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/usage/cluster/cluster_train_cn.html">分布式训练</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/fabric_cn.html">fabric集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/openmpi_cn.html">openmpi集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/k8s_cn.html">kubernetes单机</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/k8s_distributed_cn.html">kubernetes distributed分布式</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/usage/cluster/k8s_aws_cn.html">AWS上运行kubernetes集群训练</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/dev/contribute_to_paddle_cn.html">如何贡献代码</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/dev/write_docs_cn.html">如何贡献/修改文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/deep_model/rnn/index_cn.html">RNN相关模型</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../howto/deep_model/rnn/rnn_config_cn.html">RNN配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/deep_model/rnn/recurrent_group_cn.html">Recurrent Group教程</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/deep_model/rnn/hierarchical_layer_cn.html">支持双层序列作为输入的Layer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../howto/deep_model/rnn/hrnn_rnn_api_compare_cn.html">单双层RNN API对比介绍</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../howto/optimization/gpu_profiling_cn.html">GPU性能分析与调优</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../api/index_cn.html">API</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/model_configs.html">模型配置</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/activation.html">Activation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/layer.html">Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/evaluators.html">Evaluators</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/optimizer.html">Optimizer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/pooling.html">Pooling</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/networks.html">Networks</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/config/attr.html">Parameter Attribute</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/data.html">数据访问</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/data/data_reader.html">Data Reader Interface</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/data/image.html">Image Interface</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/data/dataset.html">Dataset</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/run_logic.html">训练与应用</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../api/v2/fluid.html">Fluid</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/layers.html">Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/data_feeder.html">DataFeeder</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/executor.html">Executor</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/initializer.html">Initializer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/evaluator.html">Evaluator</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/nets.html">Nets</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/optimizer.html">Optimizer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/param_attr.html">ParamAttr</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/profiler.html">Profiler</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../api/v2/fluid/regularizer.html">Regularizer</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../faq/index_cn.html">FAQ</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../faq/build_and_install/index_cn.html">编译安装与单元测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../faq/model/index_cn.html">模型配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../faq/parameter/index_cn.html">参数设置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../faq/local/index_cn.html">本地训练与预测</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../faq/cluster/index_cn.html">集群训练与预测</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../mobile/index_cn.html">MOBILE</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../mobile/cross_compiling_for_android_cn.html">Android平台编译指南</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mobile/cross_compiling_for_ios_cn.html">iOS平台编译指南</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mobile/cross_compiling_for_raspberry_cn.html">Raspberry Pi平台编译指南</a></li>
</ul>
</li>
</ul>
</nav>
<section class="doc-content-wrap">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li>Design Doc: Session</li>
</ul>
</div>
<div class="wy-nav-content" id="doc-content">
<div class="rst-content">
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="design-doc-session">
<span id="design-doc-session"></span><h1>Design Doc: Session<a class="headerlink" href="#design-doc-session" title="永久链接至标题"></a></h1>
<div class="section" id="abstract">
<span id="abstract"></span><h2>Abstract<a class="headerlink" href="#abstract" title="永久链接至标题"></a></h2>
<p>The <em>session</em> object encapsulates the environment in which the
computation graph is executed.</p>
<p>We will have the <em>local</em> session and <em>remote</em> session, they offer the
same <a class="reference external" href="#interface">interface</a>. The local session encapsulates the local
runtime environment and the remote session encapsulates the cluster
runtime environment.</p>
<p>The local runtime environment contains:</p>
<ol class="simple">
<li>computation devices (i.e., CPU, GPU) handles, and</li>
<li>the <a class="reference internal" href="../scope.html"><span class="doc">scope</span></a> which holds all variables.</li>
</ol>
<p>The remote runtime environment contains:</p>
<ol class="simple">
<li>computation devices (i.e., CPU and GPU on node 0, 1) in a cluster,
and</li>
<li>the distributed <a class="reference internal" href="../scope.html"><span class="doc">scope</span></a> in a cluster which holds all
variables.</li>
</ol>
<p>The user can create a remote session on Paddle Cloud and evaluate the
computation graph with it. In this way, the user can control the
remote computation resource in a cluster from his local computer.</p>
</div>
<div class="section" id="background">
<span id="background"></span><h2>Background<a class="headerlink" href="#background" title="永久链接至标题"></a></h2>
<p>The current design has an implicit global session in which
<code class="docutils literal"><span class="pre">paddle.eval()</span></code> is executed. The pain point is:</p>
<p>Since the user is not able to explicitly switch between runtime
environments, the user cannot run a topology in two independent
environments.</p>
<p>For example, in reinforcement learning, the user may want to have a
stale model for inference and a fresh model for training, and only
replace the stale model with the fresh model periodically.</p>
<p>Furthermore, we have no concept that encapsulates a remote environment
that executes a computation graph.</p>
<p>We need the session object to address above issues.</p>
</div>
<div class="section" id="session">
<span id="session"></span><h2>Session<a class="headerlink" href="#session" title="永久链接至标题"></a></h2>
<p>A session is an object that owns the runtime environment. All
computations are executed through <code class="docutils literal"><span class="pre">session.eval()</span></code>.</p>
<div class="section" id="interface">
<span id="interface"></span><h3>Interface<a class="headerlink" href="#interface" title="永久链接至标题"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="nb">eval</span><span class="p">(</span>
<span class="n">targets</span><span class="p">,</span>
<span class="n">feed_dict</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Evaluates the target Operations or Variables in <code class="docutils literal"><span class="pre">targets</span></code>.</p>
<ul>
<li><p class="first"><em>targets</em>: the evaluation targets. Can be a single Operation or
Variable, or a list with the Operations or Variables as
elements. The value returned by <code class="docutils literal"><span class="pre">eval()</span></code> has the same shape as the
<code class="docutils literal"><span class="pre">target</span></code> argument.</p>
<p>The PaddlePaddle program is represented by
the <a class="reference external" href="design/design/program.md">ProgramDesc</a>, <code class="docutils literal"><span class="pre">eval()</span></code> will infer the
ProgramDesc from the given targets and run the PaddlePaddle
program. Please
see
<a class="reference external" href="/design/refactor/distributed_architecture.html#local-training-architecture">this graph</a> for
the detailed illustration for the local session
and
<a class="reference external" href="/design/refactor/distributed_architecture.html#distributed-training-architecture">this graph</a> for
the detailed illustration for the remote session.</p>
</li>
<li><p class="first"><em>feed_dict</em>: a dictionary that contains the tensors which override
the edges of the computation graph.</p>
<p>feed_dict not only can provide the input data, it can override any
OP&#8217;s input as well:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="mf">2.0</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;a&quot;</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">variable</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;b&quot;</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">mul</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">)</span>
<span class="n">sess</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">targets</span><span class="o">=</span><span class="n">c</span><span class="p">,</span> <span class="n">feed_dict</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;b&quot;</span><span class="p">:</span><span class="mf">3.0</span><span class="p">})</span> <span class="c1"># returns 6.0</span>
</pre></div>
</div>
</li>
</ul>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>Closes the session and releases the scope that the session owns.</p>
</div>
<div class="section" id="create-a-local-session">
<span id="create-a-local-session"></span><h3>Create a Local Session<a class="headerlink" href="#create-a-local-session" title="永久链接至标题"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">session</span><span class="p">(</span>
<span class="n">devices</span><span class="o">=</span><span class="bp">None</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Creates a new session. One session owns one global scope, so creating
multiple sessions will create different scopes.</p>
<ul class="simple">
<li><em>devices</em>: a single <code class="docutils literal"><span class="pre">string</span></code> or a list of <code class="docutils literal"><span class="pre">string</span></code> of device names,
the corresponding devices will be the computation devices for
<code class="docutils literal"><span class="pre">eval()</span></code>. If not specified, all available devices (e.g., all GPUs)
will be used. The user doesn&#8217;t need to specify the CPU device since
it will be always used. Multiple sessions can use the same device.</li>
</ul>
<div class="section" id="example">
<span id="example"></span><h4>Example<a class="headerlink" href="#example" title="永久链接至标题"></a></h4>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">constant</span><span class="p">(</span><span class="mf">2.0</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="n">sess</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">session</span><span class="p">(</span><span class="n">devices</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;gpu:0&quot;</span><span class="p">,</span> <span class="s2">&quot;gpu:1&quot;</span><span class="p">,</span> <span class="s2">&quot;fpga:0&quot;</span><span class="p">])</span>
<span class="n">sess</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="n">sess</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="create-a-remote-session">
<span id="create-a-remote-session"></span><h3>Create a Remote Session<a class="headerlink" href="#create-a-remote-session" title="永久链接至标题"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">create_cloud_job</span><span class="p">(</span>
<span class="n">name</span><span class="p">,</span>
<span class="n">num_trainer</span><span class="p">,</span>
<span class="n">mem_per_trainer</span><span class="p">,</span>
<span class="n">gpu_per_trainer</span><span class="p">,</span>
<span class="n">cpu_per_trainer</span><span class="p">,</span>
<span class="n">num_ps</span><span class="p">,</span>
<span class="n">mem_per_ps</span><span class="p">,</span>
<span class="n">cpu_per_ps</span><span class="p">,</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Creates a Paddle Cloud job. Fails if the job name exists.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">get_cloud_job</span><span class="p">(</span>
<span class="n">name</span>
<span class="p">)</span>
</pre></div>
</div>
<p>Gets a Paddle Cloud job.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">remote_session</span><span class="p">(</span>
<span class="n">job</span>
<span class="p">)</span>
</pre></div>
</div>
<ul class="simple">
<li><em>job</em>: the Paddle Cloud job.</li>
</ul>
<div class="section" id="example">
<span id="id1"></span><h4>Example<a class="headerlink" href="#example" title="永久链接至标题"></a></h4>
<div class="highlight-Python"><div class="highlight"><pre><span></span><span class="n">reader</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">recordio</span><span class="p">(</span><span class="s2">&quot;/pfs/home/peter/mnist-train-*&quot;</span><span class="p">)</span> <span class="c1"># data stored on Paddle Cloud</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">fc1</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="s2">&quot;sigmoid&quot;</span><span class="p">)</span>
<span class="n">fc2</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">fc1</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="s2">&quot;softmax&quot;</span><span class="p">)</span>
<span class="n">cost</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="n">fc2</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span>
<span class="n">opt</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">sgd</span><span class="p">(</span><span class="n">cost</span><span class="p">)</span>
<span class="n">job</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">create_cloud_job</span><span class="p">(</span><span class="s2">&quot;test&quot;</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="s2">&quot;1G&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="s2">&quot;1G&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">sess</span> <span class="o">=</span> <span class="n">paddle</span><span class="o">.</span><span class="n">remote_ession</span><span class="p">(</span><span class="n">job</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">):</span>
<span class="n">sess</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">opt</span><span class="p">)</span>
<span class="n">sess</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>
&copy; Copyright 2016, PaddlePaddle developers.
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'../../',
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: ".txt",
};
</script>
<script type="text/javascript" src="../../_static/jquery.js"></script>
<script type="text/javascript" src="../../_static/underscore.js"></script>
<script type="text/javascript" src="../../_static/doctools.js"></script>
<script type="text/javascript" src="../../_static/translations.js"></script>
<script type="text/javascript" src="https://cdn.bootcss.com/mathjax/2.7.0/MathJax.js"></script>
<script type="text/javascript" src="../../_static/js/theme.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/js/perfect-scrollbar.jquery.min.js"></script>
<script src="../../_static/js/paddle_doc_init.js"></script>
</body>
</html>
\ No newline at end of file
develop/doc_cn/searchindex.js
浏览文件 @ 4c2433d2
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或