未验证 提交 7081f214 编写于 作者: K kavyasrinet 提交者: GitHub

Update the parameter_server doc (#7805)

上级 7ed48bd0
...@@ -9,16 +9,16 @@ different purposes. ...@@ -9,16 +9,16 @@ different purposes.
## Background ## Background
The previous implementations of the parameter server does not run a The previous implementations of the parameter server do not run a
fluid sub-program. Parameter initialization, optimizer computation, network fluid sub-program. Parameter initialization, optimizer computation, network
communication and checkpointing are implemented twice on both the communication and checkpointing are implemented twice on both the
trainer and the parameter server. trainer as well as the parameter server.
It would be great if we can write code once and use them on both the It would be great if we can write code once and use them on both: the
trainer and the parameter server: reduces code duplication and trainer and the parameter server, since this reduces code duplication and
improves extensibility. Given that after the current refactor, we are improves extensibility. Given that after the current refactoring, we are
representing everything as a computing graph on the representing everything as a computation graph on the
trainer. Representing everything as a computing graph on the parameter trainer. Representing everything as a computation graph on the parameter
server becomes a natural extension. server becomes a natural extension.
## Design ## Design
...@@ -30,9 +30,9 @@ into sub-programs to be scheduled on different nodes with the following ...@@ -30,9 +30,9 @@ into sub-programs to be scheduled on different nodes with the following
steps: steps:
1. OP placement: the OPs will be placed on different nodes according 1. OP placement: the OPs will be placed on different nodes according
to heuristic that minimizes estimated total computation to a heuristic that minimizes the estimated total computation
time. Currently we will use a simple heuristic that puts parameter time. Currently we will use a simple heuristic that puts parameter
varable on parameter server workers and everything else on trainer variable on parameter server workers and everything else on trainer
workers. workers.
1. Add communication OPs to enable the communication between nodes. 1. Add communication OPs to enable the communication between nodes.
...@@ -47,22 +47,22 @@ After converting: ...@@ -47,22 +47,22 @@ After converting:
<img src="src/dist-graph.png" width="700"/> <img src="src/dist-graph.png" width="700"/>
1. The parameter variable W and it's optimizer program are placed on the parameter server. 1. The parameter variable W and its optimizer program are placed on the parameter server.
1. Operators are added to the program. 1. Operators are added to the program.
- *Send* sends data to the connected *Recv* operator. The - *Send* sends data to the connected *Recv* operator. The
scheduler on the receive node will only schedule *Recv* operator scheduler on the receive node will only schedule *Recv* operator
to run when the *Send* operator has ran (the *Send* OP will mark to run when the *Send* operator has ran (the *Send* OP will mark
the *Recv* OP runnable automatically). the *Recv* OP runnable automatically).
- *Enueue* enqueues the input variable, it can block until space - *Enqueue* enqueues the input variable, it can block until space
become available in the queue. become available in the queue.
- *Dequeue* outputs configurable numbers of tensors from the - *Dequeue* outputs configurable numbers of tensors from the
queue. It will block until the queue have the required number of queue. It will block until the queue has the required number of
tensors. tensors.
### Benefits ### Benefits
- Model parallelism become easier to implement: it's an extension to - Model parallelism becomes easier to implement: it is an extension to
the trainer - parameter server approach. We can have several "Transpilers" the trainer - parameter server approach. We can have several "Transpilers"
to achieve different goals. to achieve different goals.
- User-defined optimizer is easier to add - user can now express it as - User-defined optimizer is easier to add - user can now express it as
...@@ -72,22 +72,22 @@ After converting: ...@@ -72,22 +72,22 @@ After converting:
### Challenges ### Challenges
- It's important to balance the parameter shards of on multiple - It is important to balance the parameter shards on multiple
parameter server. If a single parameter is very big (some parameter servers. If a single parameter is very big (for example: some
word-embedding, fully connected, softmax layer), we need to word-embedding, fully connected, softmax layer), we need to
automatically partition the single parameter onto different automatically partition the single parameter onto different
parameter servers when possible (only element-wise optimizer depends parameter servers when possible (only element-wise optimizer depends
on the parameter variable). on the parameter variable).
- In the "Aync SGD" figure, the "W" variable on the parameter server - In the "Async SGD" figure, the "W" variable on the parameter server
could be read and wrote concurrently. See could be read and written concurrently. See
[here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more [here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more
details about concurrent program in fluid. details about concurrent program in Fluid.
### Discussion ### Discussion
- Can the Enqueue OP be implemented under our current tensor design - Can the Enqueue OP be implemented under our current tensor design
(puts the input tensor into the queue tensor)? (put the input tensor into the queue tensor)?
- *Dequeue* OP will have variable numbers of output (depends on the - *Dequeue* OP will have variable numbers of output (depending on the
`min_count` attribute), does our current design support it? (similar `min_count` attribute), does our current design support it? (similar
question for the *Add* OP) question for the *Add* OP)
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册