提交 e1cea8ca 编写于 作者: T typhoonzero

follow comments

上级 430a9111
7 合并请求!11636[IMPORTANT] MKLDNN layout: Support for sum operator,!11358Merge release 0.12.0 and 0.13.0 to master,!11107merge release/0.12.0 release/0.13.0,!11094Master release/0.12.0 to master,!11092Release/0.12.0,!7531Add baseline code for PaddlePaddle AI competition,!5776Update design of dist train refactor
...@@ -52,8 +52,9 @@ The IR for PaddlePaddle after refactoring is called a `Block`, it specifies the ...@@ -52,8 +52,9 @@ The IR for PaddlePaddle after refactoring is called a `Block`, it specifies the
The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly. The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly.
This could be fixed by making the parameter server run the same computation definition as the trainer (the user's Python module). For a detailed explanation, refer to this document - This could be fixed by making the parameter server also run an IR, which can be different to the trainer side
[Design Doc: Operation Graph Based Parameter Server](./parameter_server.md) For a detailed explanation, refer to this document -
[Design Doc: Parameter Server](./parameter_server.md)
## Distributed Training Architecture ## Distributed Training Architecture
...@@ -113,7 +114,7 @@ Below are the steps that are followed : ...@@ -113,7 +114,7 @@ Below are the steps that are followed :
distributed training program: distributed training program:
1. Parse configurations from `RemoteExecutor`. 1. Parse configurations from `RemoteExecutor`.
1. Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming. 1. Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming.
1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. For 1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. Take
DataParallelism type for example, it removes the optimization operators and add a `send` OP to the DataParallelism type for example, it removes the optimization operators and add a `send` OP to the
"trainer" role, then add the optimization operators to the parameter server role within the `recv` OP. "trainer" role, then add the optimization operators to the parameter server role within the `recv` OP.
1. Dispatch the partitioned graph to different `RemoteExecutor` in the cluster. 1. Dispatch the partitioned graph to different `RemoteExecutor` in the cluster.
...@@ -129,12 +130,9 @@ log printing. ...@@ -129,12 +130,9 @@ log printing.
The Python `RemoteExecutor` is derived from `Executor` class. The Python `RemoteExecutor` is derived from `Executor` class.
```python ```python
run(self, exe = RemoteExecutor(
program=None, feed=feeder.feed(data),
feed=None, fetch_list=[avg_cost],
fetch_list=None,
feed_var_name='feed',
fetch_var_name='fetch',
job_desc=JobDesc( job_desc=JobDesc(
jobname, jobname,
num_trainer, num_trainer,
...@@ -145,6 +143,10 @@ run(self, ...@@ -145,6 +143,10 @@ run(self,
cpu_per_pserver, cpu_per_pserver,
mem_per_pserver mem_per_pserver
)) ))
for data in train_reader():
loss, acc = exe.run(trainer_prog,
feed=feeder.feed(data),
fetch_list=[avg_cost])
``` ```
`JobDesc` object describe the distributed job resource specification to run on `JobDesc` object describe the distributed job resource specification to run on
......
# Design Doc: Operation Graph Based Parameter Server # Design Doc: Parameter Server
## Abstract ## Abstract
...@@ -10,7 +10,7 @@ different purposes. ...@@ -10,7 +10,7 @@ different purposes.
## Background ## Background
The previous implementations of the parameter server does not run a The previous implementations of the parameter server does not run a
subgraph. parameter initialization, optimizer computation, network fluid sub-program. Parameter initialization, optimizer computation, network
communication and checkpointing are implemented twice on both the communication and checkpointing are implemented twice on both the
trainer and the parameter server. trainer and the parameter server.
...@@ -23,10 +23,10 @@ server becomes a natural extension. ...@@ -23,10 +23,10 @@ server becomes a natural extension.
## Design ## Design
### Graph Converter ### Distributed Transpiler
The *graph converter* converts the user-defined operation (OP) graph The *Distributed Transpiler* converts the user-defined fluid program
into subgraphs to be scheduled on different nodes with the following into sub-programs to be scheduled on different nodes with the following
steps: steps:
1. OP placement: the OPs will be placed on different nodes according 1. OP placement: the OPs will be placed on different nodes according
...@@ -34,7 +34,6 @@ steps: ...@@ -34,7 +34,6 @@ steps:
time. Currently we will use a simple heuristic that puts parameter time. Currently we will use a simple heuristic that puts parameter
varable on parameter server workers and everything else on trainer varable on parameter server workers and everything else on trainer
workers. workers.
1. Add communication OPs to enable the communication between nodes. 1. Add communication OPs to enable the communication between nodes.
We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*. We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*.
...@@ -48,8 +47,8 @@ After converting: ...@@ -48,8 +47,8 @@ After converting:
<img src="src/dist-graph.png" width="700"/> <img src="src/dist-graph.png" width="700"/>
1. The parameter variable W and it's optimizer subgraph are placed on the parameter server. 1. The parameter variable W and it's optimizer program are placed on the parameter server.
1. Operators are added to the subgraphs. 1. Operators are added to the program.
- *Send* sends data to the connected *Recv* operator. The - *Send* sends data to the connected *Recv* operator. The
scheduler on the receive node will only schedule *Recv* operator scheduler on the receive node will only schedule *Recv* operator
to run when the *Send* operator has ran (the *Send* OP will mark to run when the *Send* operator has ran (the *Send* OP will mark
...@@ -64,39 +63,30 @@ After converting: ...@@ -64,39 +63,30 @@ After converting:
### Benefits ### Benefits
- Model parallelism become easier to implement: it's an extension to - Model parallelism become easier to implement: it's an extension to
the trainer - parameter server approach. we already have the the trainer - parameter server approach. We can have several "Transpilers"
communication OPs, but need to extend the graph converter's to achieve different goals.
placement functionality.
- User-defined optimizer is easier to add - user can now express it as - User-defined optimizer is easier to add - user can now express it as
a subgraph. a sub-program.
- No more duplication logic inside the trainer and the parameter - No more duplication logic inside the trainer and the parameter
server mentioned in the background section. server mentioned in the background section.
### Challenges ### Challenges
- It might be hard for the graph converter to cut a general graph
(without any hint for which subgraph is the optimizer). We may need
to label which subgraph inside the OP graph is the optimizer.
- It's important to balance the parameter shards of on multiple - It's important to balance the parameter shards of on multiple
parameter server. If a single parameter is very big (some parameter server. If a single parameter is very big (some
word-embedding, fully connected, softmax layer), we need to word-embedding, fully connected, softmax layer), we need to
automatically partition the single parameter onto different automatically partition the single parameter onto different
parameter servers when possible (only element-wise optimizer depends parameter servers when possible (only element-wise optimizer depends
on the parameter variable). on the parameter variable).
- In the "Aync SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently. See
[here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more
details about concurrent program in fluid.
### Discussion ### Discussion
- In the "Aync SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently, what is our locking strategy?
E.g., each variable have a lock cpp method to be invoked by every
OP, or, have a lock OP.
- Can the Enqueue OP be implemented under our current tensor design - Can the Enqueue OP be implemented under our current tensor design
(puts the input tensor into the queue tensor)? (puts the input tensor into the queue tensor)?
- *Dequeue* OP will have variable numbers of output (depends on the - *Dequeue* OP will have variable numbers of output (depends on the
`min_count` attribute), does our current design support it? (similar `min_count` attribute), does our current design support it? (similar
question for the *Add* OP) question for the *Add* OP)
......
文件已添加
doc/design/refactor/src/distributed_architecture.png

185.7 KB | W: 0px | H: 0px

doc/design/dist_refactor/src/distributed_architecture.png

189.2 KB | W: 0px | H: 0px

doc/design/refactor/src/distributed_architecture.png
doc/design/dist_refactor/src/distributed_architecture.png
doc/design/refactor/src/distributed_architecture.png
doc/design/dist_refactor/src/distributed_architecture.png
  • 2-up
  • Swipe
  • Onion skin
文件已删除
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册
反馈
建议
客服 返回
顶部