diff --git a/doc/design/refactor/distributed_architecture.md b/doc/design/dist_refactor/distributed_architecture.md similarity index 95% rename from doc/design/refactor/distributed_architecture.md rename to doc/design/dist_refactor/distributed_architecture.md index f228d481e7e949e1badd7086737d58197c2ddf42..3a741f95866fb6c301ca9097af7916281f2278cf 100644 --- a/doc/design/refactor/distributed_architecture.md +++ b/doc/design/dist_refactor/distributed_architecture.md @@ -52,8 +52,9 @@ The IR for PaddlePaddle after refactoring is called a `Block`, it specifies the The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly. -This could be fixed by making the parameter server run the same computation definition as the trainer (the user's Python module). For a detailed explanation, refer to this document - -[Design Doc: Operation Graph Based Parameter Server](./parameter_server.md) +This could be fixed by making the parameter server also run an IR, which can be different to the trainer side +For a detailed explanation, refer to this document - +[Design Doc: Parameter Server](./parameter_server.md) ## Distributed Training Architecture @@ -113,7 +114,7 @@ Below are the steps that are followed : distributed training program: 1. Parse configurations from `RemoteExecutor`. 1. Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming. - 1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. For + 1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. Take DataParallelism type for example, it removes the optimization operators and add a `send` OP to the "trainer" role, then add the optimization operators to the parameter server role within the `recv` OP. 1. Dispatch the partitioned graph to different `RemoteExecutor` in the cluster. @@ -129,12 +130,9 @@ log printing. The Python `RemoteExecutor` is derived from `Executor` class. ```python -run(self, - program=None, - feed=None, - fetch_list=None, - feed_var_name='feed', - fetch_var_name='fetch', +exe = RemoteExecutor( + feed=feeder.feed(data), + fetch_list=[avg_cost], job_desc=JobDesc( jobname, num_trainer, @@ -145,6 +143,10 @@ run(self, cpu_per_pserver, mem_per_pserver )) +for data in train_reader(): + loss, acc = exe.run(trainer_prog, + feed=feeder.feed(data), + fetch_list=[avg_cost]) ``` `JobDesc` object describe the distributed job resource specification to run on diff --git a/doc/design/refactor/multi_cpu.md b/doc/design/dist_refactor/multi_cpu.md similarity index 100% rename from doc/design/refactor/multi_cpu.md rename to doc/design/dist_refactor/multi_cpu.md diff --git a/doc/design/refactor/parameter_server.md b/doc/design/dist_refactor/parameter_server.md similarity index 76% rename from doc/design/refactor/parameter_server.md rename to doc/design/dist_refactor/parameter_server.md index fa3c5d7990213cf2b0d236e66e592dd2699da876..1094f06d461275a9ad4034d5e48b39856d967b71 100644 --- a/doc/design/refactor/parameter_server.md +++ b/doc/design/dist_refactor/parameter_server.md @@ -1,4 +1,4 @@ -# Design Doc: Operation Graph Based Parameter Server +# Design Doc: Parameter Server ## Abstract @@ -10,7 +10,7 @@ different purposes. ## Background The previous implementations of the parameter server does not run a -subgraph. parameter initialization, optimizer computation, network +fluid sub-program. Parameter initialization, optimizer computation, network communication and checkpointing are implemented twice on both the trainer and the parameter server. @@ -23,10 +23,10 @@ server becomes a natural extension. ## Design -### Graph Converter +### Distributed Transpiler -The *graph converter* converts the user-defined operation (OP) graph -into subgraphs to be scheduled on different nodes with the following +The *Distributed Transpiler* converts the user-defined fluid program +into sub-programs to be scheduled on different nodes with the following steps: 1. OP placement: the OPs will be placed on different nodes according @@ -34,7 +34,6 @@ steps: time. Currently we will use a simple heuristic that puts parameter varable on parameter server workers and everything else on trainer workers. - 1. Add communication OPs to enable the communication between nodes. We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*. @@ -48,8 +47,8 @@ After converting: -1. The parameter variable W and it's optimizer subgraph are placed on the parameter server. -1. Operators are added to the subgraphs. +1. The parameter variable W and it's optimizer program are placed on the parameter server. +1. Operators are added to the program. - *Send* sends data to the connected *Recv* operator. The scheduler on the receive node will only schedule *Recv* operator to run when the *Send* operator has ran (the *Send* OP will mark @@ -64,39 +63,30 @@ After converting: ### Benefits - Model parallelism become easier to implement: it's an extension to - the trainer - parameter server approach. we already have the - communication OPs, but need to extend the graph converter's - placement functionality. - + the trainer - parameter server approach. We can have several "Transpilers" + to achieve different goals. - User-defined optimizer is easier to add - user can now express it as - a subgraph. - + a sub-program. - No more duplication logic inside the trainer and the parameter server mentioned in the background section. ### Challenges -- It might be hard for the graph converter to cut a general graph - (without any hint for which subgraph is the optimizer). We may need - to label which subgraph inside the OP graph is the optimizer. - - It's important to balance the parameter shards of on multiple parameter server. If a single parameter is very big (some word-embedding, fully connected, softmax layer), we need to automatically partition the single parameter onto different parameter servers when possible (only element-wise optimizer depends on the parameter variable). +- In the "Aync SGD" figure, the "W" variable on the parameter server + could be read and wrote concurrently. See + [here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more + details about concurrent program in fluid. ### Discussion -- In the "Aync SGD" figure, the "W" variable on the parameter server - could be read and wrote concurrently, what is our locking strategy? - E.g., each variable have a lock cpp method to be invoked by every - OP, or, have a lock OP. - - Can the Enqueue OP be implemented under our current tensor design (puts the input tensor into the queue tensor)? - - *Dequeue* OP will have variable numbers of output (depends on the `min_count` attribute), does our current design support it? (similar question for the *Add* OP) diff --git a/doc/design/refactor/src/compiler.graffle b/doc/design/dist_refactor/src/compiler.graffle similarity index 100% rename from doc/design/refactor/src/compiler.graffle rename to doc/design/dist_refactor/src/compiler.graffle diff --git a/doc/design/refactor/src/compiler.png b/doc/design/dist_refactor/src/compiler.png similarity index 100% rename from doc/design/refactor/src/compiler.png rename to doc/design/dist_refactor/src/compiler.png diff --git a/doc/design/refactor/src/dist-graph.graffle b/doc/design/dist_refactor/src/dist-graph.graffle similarity index 100% rename from doc/design/refactor/src/dist-graph.graffle rename to doc/design/dist_refactor/src/dist-graph.graffle diff --git a/doc/design/refactor/src/dist-graph.png b/doc/design/dist_refactor/src/dist-graph.png similarity index 100% rename from doc/design/refactor/src/dist-graph.png rename to doc/design/dist_refactor/src/dist-graph.png diff --git a/doc/design/dist_refactor/src/distributed_architecture.graffle b/doc/design/dist_refactor/src/distributed_architecture.graffle new file mode 100644 index 0000000000000000000000000000000000000000..d1b60141342232e06227c2d430ebc60ec349a907 Binary files /dev/null and b/doc/design/dist_refactor/src/distributed_architecture.graffle differ diff --git a/doc/design/refactor/src/distributed_architecture.png b/doc/design/dist_refactor/src/distributed_architecture.png similarity index 58% rename from doc/design/refactor/src/distributed_architecture.png rename to doc/design/dist_refactor/src/distributed_architecture.png index 0da49f44123be7c0b84bf420ad81c33f0ac0a105..29c7b0c0783f97c6d33b1db1ed484d6a2b9dd356 100644 Binary files a/doc/design/refactor/src/distributed_architecture.png and b/doc/design/dist_refactor/src/distributed_architecture.png differ diff --git a/doc/design/refactor/src/local-graph.graffle b/doc/design/dist_refactor/src/local-graph.graffle similarity index 100% rename from doc/design/refactor/src/local-graph.graffle rename to doc/design/dist_refactor/src/local-graph.graffle diff --git a/doc/design/refactor/src/local-graph.png b/doc/design/dist_refactor/src/local-graph.png similarity index 100% rename from doc/design/refactor/src/local-graph.png rename to doc/design/dist_refactor/src/local-graph.png diff --git a/doc/design/refactor/src/local_architecture.graffle b/doc/design/dist_refactor/src/local_architecture.graffle similarity index 100% rename from doc/design/refactor/src/local_architecture.graffle rename to doc/design/dist_refactor/src/local_architecture.graffle diff --git a/doc/design/refactor/src/local_architecture.png b/doc/design/dist_refactor/src/local_architecture.png similarity index 100% rename from doc/design/refactor/src/local_architecture.png rename to doc/design/dist_refactor/src/local_architecture.png diff --git a/doc/design/refactor/src/multi-threads.graffle b/doc/design/dist_refactor/src/multi-threads.graffle similarity index 100% rename from doc/design/refactor/src/multi-threads.graffle rename to doc/design/dist_refactor/src/multi-threads.graffle diff --git a/doc/design/refactor/src/multi-threads/multi-threads@3x.png b/doc/design/dist_refactor/src/multi-threads/multi-threads@3x.png similarity index 100% rename from doc/design/refactor/src/multi-threads/multi-threads@3x.png rename to doc/design/dist_refactor/src/multi-threads/multi-threads@3x.png diff --git a/doc/design/refactor/src/multi-threads/single-thread@3x.png b/doc/design/dist_refactor/src/multi-threads/single-thread@3x.png similarity index 100% rename from doc/design/refactor/src/multi-threads/single-thread@3x.png rename to doc/design/dist_refactor/src/multi-threads/single-thread@3x.png diff --git a/doc/design/refactor/src/paddle-compile.graffle b/doc/design/dist_refactor/src/paddle-compile.graffle similarity index 100% rename from doc/design/refactor/src/paddle-compile.graffle rename to doc/design/dist_refactor/src/paddle-compile.graffle diff --git a/doc/design/refactor/src/paddle-compile.png b/doc/design/dist_refactor/src/paddle-compile.png similarity index 100% rename from doc/design/refactor/src/paddle-compile.png rename to doc/design/dist_refactor/src/paddle-compile.png diff --git a/doc/design/refactor/src/remote_executor.graffle b/doc/design/dist_refactor/src/remote_executor.graffle similarity index 100% rename from doc/design/refactor/src/remote_executor.graffle rename to doc/design/dist_refactor/src/remote_executor.graffle diff --git a/doc/design/refactor/src/remote_executor.png b/doc/design/dist_refactor/src/remote_executor.png similarity index 100% rename from doc/design/refactor/src/remote_executor.png rename to doc/design/dist_refactor/src/remote_executor.png diff --git a/doc/design/refactor/src/distributed_architecture.graffle b/doc/design/refactor/src/distributed_architecture.graffle deleted file mode 100644 index 1ebbe70db04ebe0321f252a1de68aaefdaa32bd6..0000000000000000000000000000000000000000 Binary files a/doc/design/refactor/src/distributed_architecture.graffle and /dev/null differ