-
-
-
| C++ | -CUDA C++ | -Go | -
|---|---|---|
| cc_library | -nv_library | -go_library | -
| cc_binary | -nv_binary | -go_binary | -
| cc_test | -nv_test | -go_test | -
| programming languages | -PaddlePaddle | -
|---|---|
| for, while loop | -RNN, WhileOp | -
| if, if-else, switch | -IfElseOp, SwitchOp | -
| sequential execution | -a sequence of layers | -
| programming languages | -PaddlePaddle | -
|---|---|
| stack | -scope hierarchy | -
| stack frame | -scope | -
| push at entering block | -push at entering block | -
| pop at leaving block | -destroy when minibatch completes | -
| C++ functions/functors | -mul | -add | -- | - |
|---|---|---|---|---|
| C++ operator class | -mulOp | -addOp | -FCOp | -- |
| Python binding | -operator.mul | -operator.add | -operator.fc | -- |
| Python function | -- | - | - | layer.fc | -
| - | TensorFlow | -PaddlePaddle | -
|---|---|---|
| RNN | -Support | -Support | -
| recursive RNN | -Support | -Support | -
| padding zeros | -Must | -No need | -
| blob data type | -Tensor | -LoDTensor | -
| - | compile time | -runtime | -
|---|---|---|
| Data | -VarDesc(proto) | -Variable(cpp) | -
| Operation | -OpDesc(proto) | -Operator(cpp) | -
-
-
-
-
| - | Go | -Fluid | -
|---|---|---|
| user-defined functions | --layers | -- |
| control-flow and built-in functions | --intrinsics/operators | -- |
| goroutines, channels | --class ThreadPool | -- |
| runtime | --class Executor | -- |
| concurrent programming model | -implementation | -
|---|---|
| mutex | -types and functions in standard libraries | -
| semaphore | -types and functions in standard libraries | -
| communicating sequential processes (CSP) | -Go programming language | -
| actor model | -Erlang programming language | -
| message passing | -MPI | -
| bulk synchronous parallel (BSP) | -Pregel distributed programming framework | -
-
-
-
-As the figure above, we describe a global view of the asynchronous update process and use
-the parameter `w1` as an example to introduce the steps:
-1. For each gradient variables, they may distribute on different GPU card and aggregate
-them while they are all calculated.
-1. Split the gradient variable into multiple blocks according to the number of PS
-instances and then send them.
-1. PS would run an `Optimize Block` using a specified optimize algorithm to update
-the specified parameter.
-1. The trainer will fetch the latest parameter from PS before running forward Op which depends
-on the specified parameter.
-1. Broadcast the received variable into multiple GPU cards and continue to run the next
-mini-batch.
-
-### Trainer
-
-- For the multiple devices distributed training, we need to aggregate the gradient
-variables which placed on different devices firstly and then schedule a `SendVars` Operator to
-send the gradient variables to the multiple PS instances.
-- Schedule `FetchVars` operator to fetch the latest parameter from PS before running
-the forward ops.
-- There could be a large number of gradient variables to be sent, so we need to use another
-thread pool(IO Threadpool) whose a number of the schedulable threads is larger than the
-computing thread pool to avoid competitive the thread resources with computing.
-
-### Parameter Server
-
-
-
-- There should be multiple trainer instances want to optimize the same parameter at
-the same time, to avoid the racing, we need one `BlockingQueue` for each gradient
-variable to process them one by one.
-- We need a `Map` structure to map a gradient variable name to the `OptimizeBlock` which
-can optimize the respective parameter.
diff --git a/doc/fluid/design/dist_train/dist_train_nccl2.md b/doc/fluid/design/dist_train/dist_train_nccl2.md
deleted file mode 100644
index b8b8427811cddcddf872db5badfd37c96a76c3e3..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/dist_train_nccl2.md
+++ /dev/null
@@ -1,35 +0,0 @@
-# Distributed Training with NCCL2
-
-We design a pattern that can enable training with `ParallelExecutor` and
-use [NCCL2](https://developer.nvidia.com/nccl) as it's collective
-communication library.
-
-In `ParallelExecutor` we can use `AllReduce` or `Reduce` and `Broadcast`
-to do multi GPU training. And if we initialize NCCL2 communicators as
-ranks in a distributed environment, we can simply run the `ParallelExecutor`
-as a distributed program! The only thing that may be different than in
-the single node version is that we need to broadcast the NCCL unique ID
-to all the nodes and initialize communicators using that ID, so NCCL2
-can know each other as ranks.
-
-To achieve this feature, we introduce a new operator: `gen_nccl_id` op,
-so we are ***not*** "bind to" running NCCL2 with MPI, we can run it in
-whatever platform you like.
-
-It has two running modes:
-
-1. Generate and broadcast mode, which should be used on trainer 0;
-1. Listen and fetch mode, which should be used on trainers other than 0.
-
-In both two modes, this op can save the NCCL ID into current scope as a
-persistable variable, Then we can insert this op at the end of
-"startup program" of fluid, so that all workers can get the same ID to
-initialize NCCL communicator objects.
-
-
-
-The above figure indicates the general process when training with NCCL2
-distributed. Each trainer has the number of communicators equal to the
-number of GPUs, but the ranks should match the global ranks number: here
-we have total 8 GPUs, so `nranks==8`, for each trainer, the ranks should
-be from 0 ~ 3 on trainer 0 and 4 ~ 7 on trainer 1.
diff --git a/doc/fluid/design/dist_train/distributed_architecture.md b/doc/fluid/design/dist_train/distributed_architecture.md
deleted file mode 100644
index 371bbeebf7559eccc77ba0eea4f6f87a1bc5b54a..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/distributed_architecture.md
+++ /dev/null
@@ -1,197 +0,0 @@
-# Design Doc: Fluid Distributed Training Architecture
-
-## Abstract
-
-PaddlePaddle version 0.10.0 uses the "trainer-parameter server" architecture. We run multiple instances of trainers (where each trainer runs the same model) and parameter servers for distributed training. This architecture serves well, but has few limitations:
-
-1. There is a need to write special code that handles tasks which should only be run on a single trainer. E.g., initializing the model, saving the model etc.
-
-2. Model parallelism is hard: It would need all the if-else branches conditioned on the trainer ID to partition the model onto the trainers, and eventually manually writing out the inter-model-shard communication code to communicate between different trainers.
-
-3. The user can not directly specify the parameter update rule: This would need to modify the parameter server code and compile a new binary. This makes things more complicated for researchers: A lot of extra effort is required to make this work. Besides, the training job submission program may not allow running arbitrary binaries.
-
-This design doc discusses PaddlePaddle's new distributed training architecture that addresses the above mentioned limitations.
-
-## Analysis
-
-The assumption is that the user writes the trainer program in either Python or C++.
-
-### Limitation 1
-
-There are two basic functionalities in the trainer program:
-
-1. The training logic such as loading / saving the model and printing out the logs.
-2. The neural network definition such as the definition of the data layer, the fully connected layer, the cost function and the
- optimizer.
-
-When we train using PaddlePaddle v0.10.0 in a distributed fashion, multiple instances of the same Python code are run on different nodes, hence both: the
-training logic as well as the neural network computation logic, is replicated.
-
-The tasks that only need to be run once belong to the training logic. Hence if we only replicate the neural network computation part, and do **not**
-replicate the training logic, the limitation mentioned above can be avoided.
-
-### Limitation 2
-
-Model parallelism means that a single model is partitioned into different components and each node runs one of the component separately. This comes at the extra cost of managing the
-inter-model-shard communication between nodes.
-
-PaddlePaddle should ideally be able to modify the neural network computation and figure out the support for model parallelism automatically. However, the
-computation is only specified in Python code which sits outside of PaddlePaddle, hence PaddlePaddle can not support the feature in this setup.
-
-Similar to how a compiler uses an intermediate representation (IR) so that the programmer does not need to manually optimize their code for most of the cases, we can have an intermediate representation in PaddlePaddle as well. The compiler optimizes the IR as follows:
-
-
-
-PaddlePaddle can support model parallelism by converting the IR so that the user no longer needs to manually perform the computation and operations in the Python component:
-
-
-
-The IR for PaddlePaddle after refactoring is called a `Block`, it specifies the computation dependency graph and the variables used in the computation.
-
-### Limitation 3
-
-The user can not directly specify the parameter update rule for the parameter server in the Python module, since the parameter server does not use the same computation definition as the trainer. Instead, the update rule is baked inside the parameter server. The user can not specify the update rule explicitly.
-
-This could be fixed by making the parameter server also run an IR, which can be different to the trainer side
-For a detailed explanation, refer to this document -
-[Design Doc: Parameter Server](./parameter_server.md)
-
-## Distributed Training Architecture
-
-The revamped distributed training architecture can address the above discussed limitations. Below is the illustration of how it does so:
-
-
-
-The major components are: *Python API*, *Distribute Transpiler* and *Remote Executor*.
-
-### Python API
-
-Python API is the Python library that user's Python code invokes, to read the data, build the neural network topology, and start training, etc.
-
-```Python
-images = fluid.layers.data(name='pixel', shape=[1, 28, 28], dtype='float32')
-label = fluid.layers.data(name='label', shape=[1], dtype='int64')
-...
-predict = fluid.layers.fc(input=conv_pool_2, size=10, act="softmax")
-cost = fluid.layers.cross_entropy(input=predict, label=label)
-avg_cost = fluid.layers.mean(x=cost)
-optimizer = fluid.optimizer.Adam(learning_rate=0.01)
-optimizer.minimize(avg_cost)
-
-train_reader = paddle.batch(
- paddle.reader.shuffle(
- paddle.dataset.mnist.train(), buf_size=500),
- batch_size=BATCH_SIZE)
-
-place = fluid.CPUPlace()
-exe = fluid.Executor(place)
-
-for pass_id in range(10):
- for data in train_reader():
- loss, acc = exe.run(trainer_prog,
- feed=feeder.feed(data),
- fetch_list=[avg_cost])
-```
-
-The code above is a typical local training program, the "Training Program" is built using helper functions such as
-`fluid.layer.fc`. The training is done by calling `Executor.run`
-iteratively.
-
-For more details, the implementation of IR is [Program](../program.md), and `ProgramDesc` is the protobuf type.
-
-[Executor](../executor.md) simply runs the `ProgramDesc`. For local training you generally use
-`Executor` to run the program locally. For any kind of distributed training, you can use
-`RemoteExecutor` to specify desired distributed training method with some optional arguments.
-
-### Distributed Transpiler
-
-The Distributed Transpiler automatically converts the IR (in protobuf format) to partitioned IRs. Then
-the Remote Executor dispatches the new IRs to Remote Executors across the cluster.
-Below are the steps that are followed :
-
-1. User only need to change `Executor` to `RemoteExecutor` to change local program to distributed program.
-1. `RemoteExecutor` calls `Distributed Transpiler` to "transpile" user's program to several IRs representing a
- distributed training program:
- 1. Parse configurations from `RemoteExecutor`.
- 1. Determine the type of distributed program, can be DataParallelism, ModelParallelism or Streaming.
- 1. Partition the `ProgramDesc` according to type and add `send` / `recv` OP pair on the boundaries. Take
- DataParallelism type for example, it removes the optimization operators and add a `send` OP to the
- "trainer" role, then add the optimization operators to the parameter server role within the `recv` OP.
-1. Dispatch the partitioned graph to different `RemoteExecutor` in the cluster.
-1. `RemoteExecutor` on each node run the received `ProgramDesc` utill the end.
-
-
-### RemoteExecutor
-
-As shown in the graph, `RemoteExecutor.run` sends the IR to the cluster for Execution.
-You can also use parameter `fetch_list` to interactively fetch variable back to local for
-log printing.
-
-The Python `RemoteExecutor` is derived from `Executor` class.
-
-```python
-exe = RemoteExecutor(
- feed=feeder.feed(data),
- fetch_list=[avg_cost],
- job_desc=JobDesc(
- jobname,
- num_trainer,
- num_pserver,
- cpu_per_trainer,
- gpu_per_trainer,
- mem_per_trainer,
- cpu_per_pserver,
- mem_per_pserver
- ))
-for data in train_reader():
- loss, acc = exe.run(trainer_prog,
- feed=feeder.feed(data),
- fetch_list=[avg_cost])
-```
-
-`JobDesc` object describe the distributed job resource specification to run on
-Cluster environment.
-
-
-
-`RemoteExecutor.run` sends the `ProgramDesc` and
-[TrainingJob](https://github.com/PaddlePaddle/cloud/blob/unreleased-tpr/doc/autoscale/README.md#training-job-resource)
-to a server in the cluster which executes `RemoteExecutor.listen`. This server is responsible
-to start the final Kubernetes Jobs to run the different role of `ProgramDesc` from `ConfigMap`.
-
-
-### Placement Algorithm
-
-Our first implementation will only support "trainer-parameter server" placement: the parameters, initializers, and optimizers are all placed on the PaddlePaddle runtimes with the parameter server role. Everything else will be placed on the PaddlePaddle runtimes with the trainer role. This has the same functionality as the "trainer-parameter server" architecture of PaddlePaddle v0.10.0, but is more generic and flexible.
-
-In the future, a more general placement algorithm should be implemented, which makes placements according to the input IR, and a model of device computation time and device communication time. Model parallelism requires the generic placement algorithm.
-
-
-### Local Training Architecture
-
-The local training architecture will be the same as the distributed training architecture, the difference is that everything runs locally, and there is just one PaddlePaddle runtime:
-
-
-
-
-### Training Data
-
-In PaddlePaddle v0.10.0, training data is typically read
-with [data reader](./README.md) from Python. This approach is
-no longer efficient when training distributedly since the Python
-process no longer runs on the same node with the trainer processes,
-the Python reader will need to read from the distributed filesystem
-(assuming it has the access) and send to the trainers, doubling the
-network traffic.
-
-When doing distributed training, the user can still use Python data
-reader: the training data are sent with `Executor.run`. However, should
-be used for debugging purpose only. The users are encouraged to use
-the read data OPs.
-
-
-## References:
-
-[1] [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf)
-
-[2] [TensorFlow: A System for Large-Scale Machine Learning](https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf)
diff --git a/doc/fluid/design/dist_train/distributed_lookup_table_design.md b/doc/fluid/design/dist_train/distributed_lookup_table_design.md
deleted file mode 100644
index e284e1ec5cdd18d0049ce3c1a8349bbe1248cb48..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/distributed_lookup_table_design.md
+++ /dev/null
@@ -1,89 +0,0 @@
-# Design Doc: Distributed Lookup Table Operator
-
-A distribute lookup table operator in PaddlePaddle where the table could be out
-of the memory of a computer.
-
-## Background
-
-A lookup table operator is well-used in deep learning for learning the
-representation, or the
-[*embedding*](http://www.cs.toronto.edu/~fritz/absps/ieee-lre.pdf), of
-symbols.
-
-### The Forward Algorithm
-
-The forward algorithm of the lookup table is a multiplication of the
-input vector x and the lookup table matrix W:
-
-$$y = x * W$$
-
-When x is a sparse vector of symbols, the above multiplication
-simplifies into looking up rows in W that correspond to symbols in x,
-denoted by W(x). Please be aware that W could be huge and out of the
-memory, so we'd need a distributed storage service, which supports the
-lookup of rows.
-
-The following figure illustrates the multiplication of x with two
-non-zero elements, or say two symbols, and a lookup table W:
-
-
-
-### The Backward Algorithm
-
-The backward algorithm computes W'(x) using W(x). W'(x) has the same
-the scale of size as W(x) and is much smaller than W.
-
-To optimize W given W', we can do simple SGD update:
-
-$$W = f(W') = \lambda * W'$$
-
-or some more sophisticated algorithms that rely on both W' and W:
-
-$$W = f(W, W')$$
-
-The following figure illustrates the backward pass of the lookup
-operator: 
-
-## Distributed Lookup Table
-### Problem 1: The lookup table may be very large.
-
- In the condition like the search engine and recommendation system, the number of feature Id may be very large, say 100,000,000,000, then for a float value lookup table of size 8, the total size of the table is:
-
- ```
- 100,000,000,000 * 8 * 4(Bytes) = 2980.23 GB
- ```
-
-### Solution: Distributed storage
-
-1. Paddle use [SelectedRows](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/selected_rows.md) as the storage format for the lookup table, the lookup table parameter will be split to multi-machine according to the hash of the feature ID, and data will also be split and send to the same machine to prefetch the parameter.
-
-1. For common parameters, the trainer will get the whole parameter for training, but for the big lookup table, the trainer can not store the whole parameter. Because the input data feature is very sparse, every time we only need a few parameters for training, so we use `prefetch_op` to only prefetch the parameter needed to trainer.
-
-### Problem 2. The Id in the lookup table is not sure before training.
-
- The feature Id is calculated by the hash function because the feature data source is so large, we can not get all the Id before training. So we can not initialize the table before training.
-
-### Solution: Id auto growth
-
-At the beginning of training, paddle only malloc the memory for the lookup table at parameter server side, the Id and it's value will not be initialized. During training, when a parameter server received an Id, if it is already in the lookup table, it will return the existing parameter, if the Id does not exist, paddle will add it into the lookup table and initialize the value for it.
-
-### Problem 3: parameter load and save
-
-For common parameters, paddle use trainer to save and load them. But for distributed lookup table, trainer cannot do this because it's large size.
-
-### Solution: Parameter server side save and load
-
-Paddle support parameter server side save and load for distribute lookup table. Each machine of parameter servers will only save and load part of the whole table.
-
-## Architecture
-The whole architecture of the distribute lookup table is as below:
-
-### Training steps:
-1. Read a batch of data, the data is feature ids.
-1. The input ids will be split by `split_ids_op` with the same hash function of the lookup table.
-1. The `prefetch_op` use the split result to prefetch parameters back from the lookup table.
-1. Run forward-backward to get the gradient of the lookup table.
-1. `split_ids_op` split the gradient and then use `send_op` to the parameter server.
-1. parameter server update the table with the received gradient.
-
-
diff --git a/doc/fluid/design/dist_train/distributed_traing_review.md b/doc/fluid/design/dist_train/distributed_traing_review.md
deleted file mode 100644
index c09b7c99159ace9b3df989f803ede20bc3585d92..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/distributed_traing_review.md
+++ /dev/null
@@ -1,44 +0,0 @@
-# Parallelism, Asynchronous, Synchronous, Codistillation
-
-
-For valuable models, it’s worth using more hardware resources to reduce the training time and improve the final model quality. This doc discuss various solutions, their empirical results and some latest researches.
-
-# Model Parallelism
-In some situations, larger and more complex models can improve the model quality. Sometimes, such models cannot fit in one device. Sometimes, parts of the model can be executed in parallel to improve speed. Model Parallelism address the issues by partitioning a single model and place the shards on several devices for execution.
-
-A common way of model parallelism is partition the logic of “gradient application” to parameter servers, while leaving the forward and backward computation at training servers.
-
-More flexible model parallelism is challenging. For example, multi-level-single-direction LSTM can be partitioned by layers, while such solution is not helpful for bi-directional LSTM. Different models can have quite different ways of partitioning and the benefits also depend on the underlying hardware. Framework needs to provide flexible APIs for user to define the customized partition scheme. For example, in TensorFlow, user can use tf.device() to specify the device placement. In MxNet, mx.AttrScope(ctx_group='dev1') does similar things. Recent research proposes to automatically find the optimal partition scheme with Reinforcement Learning, which is essentially solution space search algorithm that could cost a lot of extra hardware sources.
-
-# Data Parallelism
-Data Parallelism runs the same model on multiple devices, each taking in a partition of the input batch. It’s more commonly used for a few reasons. It generally applies to common SGD mini-batch training. Compared with model parallelism, which requires users to carefully partition their model and tune for good performance, data parallelism usually involves no more than calling an extra API and speed up is more predictable.
-
-# Asynchronous Training
-In asynchronous training, it usually involves a set of trainers and a set of parameter servers. The parameter servers collectively hold a single copy of shared parameters. While the trainers each holds a unique copy of model and trains the model independently. Each trainer pulls parameters from parameter servers and sends gradients to the parameter servers independently. Similarly the parameter servers applies the gradients to parameters as soon as the gradients are received and sends parameters whenever they are requested.
-
-In theory, asynchronous training is not safe and unstable. Each trainer is very likely using stale copy of parameters and parameters are also likely to apply stale gradients. However, in practice, especially for large-scale nonconvex optimization, it is effective [1]. Compared with synchronous solution, which will be discussed later, asynchronous distributed training is easier to implement and scales to a few dozen workers without losing much performance due to network communication or other overhead. Besides, asynchronous training can make progress even in case of random trainer failure in the cluster.
-
-Many production models, such as [3], are trained with distributed asynchronous solutions due to its scalability and effectiveness in practice. However, asynchronous training has its limitations. Usually, it’s not as stable as synchronous training. A warm-up phase is sometimes needed. Learning rate is usually smaller compared with synchronous training and decay is also often needed. Normally, asynchronous training doesn’t scale beyond 100 trainers. In other words, when putting more trainers beyond that, the model cannot converge faster.
-
-# Synchronous Training
-Unlike asynchronous training, synchronous training requires step barriers. Parameter servers needs to wait for gradients from all trainers before they are applied to parameters and trainers will always pull the latest parameters.
-
-An obvious advantage of synchronous training is that the behavior is more clearly defined. Usually, it's more stable than asynchronous training. Learning rate can be set larger and for some vision tasks, the final accuracy can be slightly higher. (In my practical experience, for some models, it can actually be worse).
-
-Synchronous training usually faces scalability and performance issues, if not carefully implemented or deployed. In [2], native synchronous training can be 20%~40% slower than asynchronous training. A common trick to avoid slowness, discussed in [1] and [2], is to have backups. N+M replicas are scheduled while only the first N is needed for the training step the proceed.
-
-Similar to asynchronous training, the benefit of synchronous training diminishes quickly. Depending on the models, increasing the number of trainers (effectively batch size) beyond a point won’t delivers faster converge time or better final model quality.
-
-# Codistillation
-Codistillation is a technique that tries to scale the training further. A few training instance (each training instance can be distributed) are performed during the same period. Each training instance has extra losses that comes from the prediction of other training instances. (likey teacher and student) The training process converges faster and usually converge to a better model quality. [4]
-
-
-# Reference
-
-[1] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks.
-
-[2] Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting distributed synchronous SGD.
-
-[3] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation.
-
-[4] LARGE SCALE DISTRIBUTED NEURAL NETWORK TRAINING THROUGH ONLINE DISTILLATION
diff --git a/doc/fluid/design/dist_train/index_cn.rst b/doc/fluid/design/dist_train/index_cn.rst
deleted file mode 100644
index ed6f3dda271d2de58d92aa7ec804fa9e68dfc48a..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/index_cn.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-分布式训练
-------------
-
-.. toctree::
- :maxdepth: 1
-
- distributed_architecture.md
- distributed_lookup_table_design.md
- parameter_server.md
diff --git a/doc/fluid/design/dist_train/index_en.rst b/doc/fluid/design/dist_train/index_en.rst
deleted file mode 100644
index f84688f168021113bd933802709bcd787b474bca..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/index_en.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-Distributed Training
----------------------
-
-.. toctree::
- :maxdepth: 1
-
- distributed_architecture.md
- distributed_lookup_table_design.md
- parameter_server.md
diff --git a/doc/fluid/design/dist_train/mpi_enabled_design.md b/doc/fluid/design/dist_train/mpi_enabled_design.md
deleted file mode 100644
index 4ad3afc7b7522c60460c6f1f387f9415d3738778..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/mpi_enabled_design.md
+++ /dev/null
@@ -1,46 +0,0 @@
-# MPI-enabled PaddlePaddle Design doc
-
-# Background
-When we do distribute multi GPU training, the communication overhead between servers become the major bottleneck, because of the following reasons:
-1. Must copy at least once from GPU to CPU memory so that the data can be ready to transfer. And for the pserver side, copy data from CPU to GPU introduce more overhead.
-2. GPU->CPU data transfer is 10 times slower than data transfer between GPUs or between PCIe devices.
-3. TCP connections can not make full use of RDMA 100Gb devices.
-
-We will use OpenMPI API to PaddlePaddle, which can bring two benefits to PaddlePaddle:
-1. Enable RDMA with PaddlePaddle, which bring high-performance low latency networks.
-2. Enable GPUDriect with PaddlePaddle, which bring the highest throughput and lowest latency GPU read and write.
-
-# Change list
-* Compile args: Need add compile args to enable MPI support.
-* Execute args: Need add execute args to assign when and how to use MPI operations.
-* New ops: Need new op ```mpi_send_op``` and ```mpi_listenandserve_op``` to support MPI send and receive.
-* Transpiler optimized: Which can add ```mpi_send_op``` and ```mpi_listenandserve_op``` to the running graph.
-* MPI utils package: Need MPI utils package as the low-level API supported.
-
-## Compile args
-Because MPI or CUDA need hardware supported, so we will add compile args to enable MPI support and control compiling.Add ```WITH_MPI``` compile args to control MPI to use or not. If the ```WITH_MPI``` is ```ON```, compile system will find openMPI codes in configuration. We should prepare openMPI environment before compiling.
-
-## Execute args
-Launch the script using the ```mpirun``` launcher, For example: ```mpirun -np 3 -hosts node1,node2,node3 python train.py```. By doing this, We can number the actors (trainer/pserver/master) with o .. (n-1). The node's number is the Rank of the calling process in a group of comm (integer), The MPI processes identify each other using a Rank ID. We have to create a mapping between PaddlePaddle's nodes and their Rank ID so that we can communicate with the correct destinations when using MPI operations.
-
-## New ops
-We won't replace all the gRPC requests to MPI requests, the standard gRPC library is used for all administrative operations and the MPI API will be used to transfer tensor or selectRows to Pservers. The base of this idea, we create two new operators to handle requests and receives, the two operators are ```mpi_send_op``` and ```mpi_listenandserve_op```. They are a little similar to [send_op](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/send_op.cc) and [listen_and_serv_op](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/listen_and_serv_op.cc), also, We will build a new module to package MPI send and receive process.
-
-### mpi_send_op
-Very similar with ```send_op```, we will replace gRPC code which used to send gradient with ```mpi_module```, at the same time, we will wrap it with ```framework::Async```.
-
-### mpi_listenandserve_op
-Very similar with ```listen_and_serv_op```, we will replace gRPC code which used to receive gradient with ```mpi_module```, at the same time, we will wrap it with ```framework::Async```.
-
-## Transpiler optimized
-**We can get env ```OMPI_COMM_WORLD_SIZE``` and ```OMPI_COMM_WORLD_RANK``` to distinguish use MPI or not, If we use openMPI, the variable in env must exist.**
- if confirm to use MPI, we will modify ```send_op``` to ```mpi_send_op``` in distribute_transpiler, and modify ```listenandserve_op``` to ```mpi_listenandserve_op``` also.
-
-## MPI utils package
-In this package, We will write openMPI low-level API to use MPI.
-The API included in this package are:
-* MPI send and receive module, We will build a new module to package MPI send and receive process. MPI send and receive are different to gRPC, the MPI [recvice](https://www.open-mpi.org/doc/v1.8/man3/MPI_Irecv.3.php) must know receive buffer size and receive buffer element. For this reason, We have to make communications twice, the first one is to send metadata about gradient through gRPC, the second one is the real communication through MPI which send gradient data to mpi_listenandserve_op.
-The detailed flow is below:
-
-* MPI global configurations, which store the Rank ID and the mapping in global variables, for example:
-gRPC client : MPI nodes :``` 127.0.0.1:32004 : 3 ```
diff --git a/doc/fluid/design/dist_train/multi_cpu.md b/doc/fluid/design/dist_train/multi_cpu.md
deleted file mode 100644
index 38222d083084ebfca3099ce96b47868c42d55101..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/multi_cpu.md
+++ /dev/null
@@ -1,43 +0,0 @@
-# Design Doc: Execute the Program with Multi CPU
-
-## Abstract
-
-This Design Doc propose an approach to make the user-defined Op graph
-running with multi-CPU, we will use an auto transpiler to convert the user-defined
-Op graph to a multi-CPU Op graph, and run `ParallelDo` Op to run the graph.
-
-## Transpiler
-
-
-
-After converted:
-
-
-
-## Implement
-
-- `Multi-CPU Transpiler` will convert the graph to a multi-CPU graph
- which would be executed with multi-threads.
-- `BlockingCounter` will `Init/Decrement` an atomic counter, and Blocking `Wait`
- for the atomic counter become `0`:
- ```cpp
- BlockingCounter bc(thread_count);
- for (int i = 0; i < thread_count; ++i) {
- thread_pool->Start([&bc] {bc.DecrementCount(); })
- }
- bc.Wait();
- ```
-- `ParallelDo` Operator
- - Initialize a thread pool which is a Singleton.
- - Use a block id as the input, and create run the specify Block on independent scope
- with multi-threads.
- - Initialize a `BlockingCounter` instance and wait until all threads are done.
-- `Split` Operator will split the Input Tensor into a TensorArray.
-- `Merge` merge all the gradients which calculated in different threads
- with `mean/sum/max/min...` method, and then run the Optimizer Op to optimize `W`.
-
-## TODO
-
-- Improve the optimizer stage with multi-threads, since we could
- assign the parameters to the different threads and execute
- optimizer with multi-threads.
diff --git a/doc/fluid/design/dist_train/parameter_server.md b/doc/fluid/design/dist_train/parameter_server.md
deleted file mode 100644
index 563b70bc0e852bec953eb40dda3c46b3d45d7e68..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dist_train/parameter_server.md
+++ /dev/null
@@ -1,106 +0,0 @@
-# Design Doc: Parameter Server
-
-## Abstract
-
-We propose an approach to implement the parameter server. In this
-approach, there is no fundamental difference between the trainer and
-the parameter server: they both run subgraphs, but subgraphs of
-different purposes.
-
-## Background
-
-The previous implementations of the parameter server do not run a
-fluid sub-program. Parameter initialization, optimizer computation, network
-communication and checkpointing are implemented twice on both the
-trainer as well as the parameter server.
-
-It would be great if we can write code once and use them on both: the
-trainer and the parameter server, since this reduces code duplication and
-improves extensibility. Given that after the current refactoring, we are
-representing everything as a computation graph on the
-trainer. Representing everything as a computation graph on the parameter
-server becomes a natural extension.
-
-## Design
-
-### Distributed Transpiler
-
-The *Distributed Transpiler* converts the user-defined fluid program
-into sub-programs to be scheduled on different nodes with the following
-steps:
-
-1. OP placement: the OPs will be placed on different nodes according
- to a heuristic that minimizes the estimated total computation
- time. Currently we will use a simple heuristic that puts parameter
- variable on parameter server workers and everything else on trainer
- workers.
-1. Add communication OPs to enable the communication between nodes.
-
-We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*.
-
-Below is an example of converting the user defined graph to the
-subgraphs for the trainer and the parameter server:
-
-
-
-After converting:
-
-
-
-1. The parameter variable W and its optimizer program are placed on the parameter server.
-1. Operators are added to the program.
- - *Send* sends data to the connected *Recv* operator. The
- scheduler on the receive node will only schedule *Recv* operator
- to run when the *Send* operator has ran (the *Send* OP will mark
- the *Recv* OP runnable automatically).
- - *Enqueue* enqueues the input variable, it can block until space
- become available in the queue.
- - *Dequeue* outputs configurable numbers of tensors from the
- queue. It will block until the queue has the required number of
- tensors.
-
-### Sparse Update
-
-For embedding layers, the gradient may have many rows containing only 0 when training,
-if the gradient uses a dense tensor to do parameter optimization,
-it could spend unnecessary memory, slow down the calculations and waste
-the bandwidth while doing distributed training.
-In Fluid, we introduce [SelectedRows](../modules/selected_rows.md) to represent a list of rows containing
-non-zero gradient data. So when we do parameter optimization both locally and remotely,
-we only need to send those non-zero rows to the optimizer operators:
-
-
-### Benefits
-
-- Model parallelism becomes easier to implement: it is an extension to
- the trainer - parameter server approach. We can have several "Transpilers"
- to achieve different goals.
-- User-defined optimizer is easier to add - user can now express it as
- a sub-program.
-- No more duplication logic inside the trainer and the parameter
- server mentioned in the background section.
-
-### Challenges
-
-- It is important to balance the parameter shards on multiple
- parameter servers. If a single parameter is very big (for example: some
- word-embedding, fully connected, softmax layer), we need to
- automatically partition the single parameter onto different
- parameter servers when possible (only element-wise optimizer depends
- on the parameter variable).
-- In the "Async SGD" figure, the "W" variable on the parameter server
- could be read and written concurrently. See
- [here](https://github.com/PaddlePaddle/Paddle/pull/6394) for more
- details about concurrent program in Fluid.
-
-### Discussion
-
-- Can the Enqueue OP be implemented under our current tensor design
- (put the input tensor into the queue tensor)?
-- *Dequeue* OP will have variable numbers of output (depending on the
- `min_count` attribute), does our current design support it? (similar
- question for the *Add* OP)
-
-### References
-
-[1] [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf)
diff --git a/doc/fluid/design/dist_train/src/async_distributed_training.png b/doc/fluid/design/dist_train/src/async_distributed_training.png
deleted file mode 100644
index 3b53ab59c0cd7b44b2956f16f1adc47fe85909d3..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/async_distributed_training.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/async_pserver.graffle b/doc/fluid/design/dist_train/src/async_pserver.graffle
deleted file mode 100644
index d2301611774fcb3866473e3e6470568d1e1312cf..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/async_pserver.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/async_pserver.png b/doc/fluid/design/dist_train/src/async_pserver.png
deleted file mode 100644
index 7d900b0c0eb291c67537b9cf93227c671bafdc73..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/async_pserver.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/async_update.graffle b/doc/fluid/design/dist_train/src/async_update.graffle
deleted file mode 100644
index 3a631888688a0d564a873fcb16d943958c91223e..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/async_update.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/async_update.png b/doc/fluid/design/dist_train/src/async_update.png
deleted file mode 100644
index 3e8db973f45d6d9ac8dcce1dc7878067e79e6dcc..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/async_update.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/compiler.graffle b/doc/fluid/design/dist_train/src/compiler.graffle
deleted file mode 100644
index 8cc678fea3c820103e7ce81f7a5d625d6c1d92de..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/compiler.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/compiler.png b/doc/fluid/design/dist_train/src/compiler.png
deleted file mode 100644
index 65d34f841afce9756def07dd8ecb9ca44e658bfe..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/compiler.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/dist-graph.graffle b/doc/fluid/design/dist_train/src/dist-graph.graffle
deleted file mode 100644
index 941399c6ced8d5f65b6c595522b770c88259df4b..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/dist-graph.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/dist-graph.png b/doc/fluid/design/dist_train/src/dist-graph.png
deleted file mode 100644
index 3546b09f1c2ee3e4f60f519d5e47f823f08051a7..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/dist-graph.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/distributed_architecture.graffle b/doc/fluid/design/dist_train/src/distributed_architecture.graffle
deleted file mode 100644
index d1b60141342232e06227c2d430ebc60ec349a907..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/distributed_architecture.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/distributed_architecture.png b/doc/fluid/design/dist_train/src/distributed_architecture.png
deleted file mode 100644
index 29c7b0c0783f97c6d33b1db1ed484d6a2b9dd356..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/distributed_architecture.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/distributed_lookup_table.graffle b/doc/fluid/design/dist_train/src/distributed_lookup_table.graffle
deleted file mode 100644
index 65dfdbbacd219739db6ddfdf243cc16c3c4e8d1e..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/distributed_lookup_table.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/distributed_lookup_table.jpeg b/doc/fluid/design/dist_train/src/distributed_lookup_table.jpeg
deleted file mode 100644
index 5353a16fd329f62ff893d32706b9c3c0bcc46a07..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/distributed_lookup_table.jpeg and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/distributed_training.graffle b/doc/fluid/design/dist_train/src/distributed_training.graffle
deleted file mode 100644
index 1168801bc1fadfce310a74cb3110695bd1629f6b..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/distributed_training.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/fluid_lookup_remote_table.graffle b/doc/fluid/design/dist_train/src/fluid_lookup_remote_table.graffle
deleted file mode 100644
index 96ca6d48f43bd9f49c6861dab006e2037873db87..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/fluid_lookup_remote_table.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/fluid_lookup_remote_table.png b/doc/fluid/design/dist_train/src/fluid_lookup_remote_table.png
deleted file mode 100644
index afa25ab3b4e427bc595a855b12ab966478e01ed0..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/fluid_lookup_remote_table.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/local-graph.graffle b/doc/fluid/design/dist_train/src/local-graph.graffle
deleted file mode 100644
index 19e509bd9af3c1e9a3f5e0f16ddd281457a339c5..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/local-graph.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/local-graph.png b/doc/fluid/design/dist_train/src/local-graph.png
deleted file mode 100644
index ada51200f793a9bb18911e7d63cfdb3244b967d7..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/local-graph.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/local_architecture.graffle b/doc/fluid/design/dist_train/src/local_architecture.graffle
deleted file mode 100644
index 49fcc663ebe3824aa234e3a67aadf285cb417877..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/local_architecture.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/local_architecture.png b/doc/fluid/design/dist_train/src/local_architecture.png
deleted file mode 100644
index 14adc9fd72b855bb9f74fbf2c84ac9ec0cf2b122..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/local_architecture.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/lookup_table.png b/doc/fluid/design/dist_train/src/lookup_table.png
deleted file mode 100644
index 72dfe3547f731d0d090338afb206b0549dff472e..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/lookup_table.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/lookup_table_training.png b/doc/fluid/design/dist_train/src/lookup_table_training.png
deleted file mode 100644
index cc7cc4aeb3b885850fe2f70f19fb84d5873bed1e..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/lookup_table_training.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/mpi_module.png b/doc/fluid/design/dist_train/src/mpi_module.png
deleted file mode 100644
index e6b6a3e5d6f68baeeb67d7f71154bd8d85f32b6f..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/mpi_module.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/multi-threads.graffle b/doc/fluid/design/dist_train/src/multi-threads.graffle
deleted file mode 100644
index e71173715fff92a0a933d0c7d83599ba948552c6..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/multi-threads.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/multi-threads/multi-threads@3x.png b/doc/fluid/design/dist_train/src/multi-threads/multi-threads@3x.png
deleted file mode 100644
index e40a869987dbbf5019d4cb03c1dab55b74d6c9f9..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/multi-threads/multi-threads@3x.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/multi-threads/single-thread@3x.png b/doc/fluid/design/dist_train/src/multi-threads/single-thread@3x.png
deleted file mode 100644
index 4083aebfdd45af5fbac25fa2c4176bc08c3cb44a..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/multi-threads/single-thread@3x.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/ncc2_design.graffle b/doc/fluid/design/dist_train/src/ncc2_design.graffle
deleted file mode 100644
index 7d2753bbb03bc28c7a0054bb0aa424deb072ffbf..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/ncc2_design.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/ncc2_design.png b/doc/fluid/design/dist_train/src/ncc2_design.png
deleted file mode 100644
index da0d5ee81f5dfeb4ca1356601b0bb5870456e3d6..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/ncc2_design.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/paddle-compile.graffle b/doc/fluid/design/dist_train/src/paddle-compile.graffle
deleted file mode 100644
index a6348cc3dbcaca923c6e794681b2edb85cb9f8f6..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/paddle-compile.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/paddle-compile.png b/doc/fluid/design/dist_train/src/paddle-compile.png
deleted file mode 100644
index e0f13d551ac41afaec627a57dea79356464bf0bf..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/paddle-compile.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/remote_executor.graffle b/doc/fluid/design/dist_train/src/remote_executor.graffle
deleted file mode 100644
index 41b2067311694b56d211a4f32d1b76884eeffd2d..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/remote_executor.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/remote_executor.png b/doc/fluid/design/dist_train/src/remote_executor.png
deleted file mode 100644
index 744e2fb2e0f1bbe058e991ba7b2a09000965ee79..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/remote_executor.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/sparse_update.graffle b/doc/fluid/design/dist_train/src/sparse_update.graffle
deleted file mode 100644
index 08d689a58f83698d8c1158ee3990ed8abf3a7a9a..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/sparse_update.graffle and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/sparse_update.png b/doc/fluid/design/dist_train/src/sparse_update.png
deleted file mode 100644
index 8c872e6ac479f7d1b818a4a207956c43155d0ad7..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/sparse_update.png and /dev/null differ
diff --git a/doc/fluid/design/dist_train/src/sync_distributed_training.png b/doc/fluid/design/dist_train/src/sync_distributed_training.png
deleted file mode 100644
index e4f9a221fea4b7238e8a1d84e609c0371f6ef7a2..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dist_train/src/sync_distributed_training.png and /dev/null differ
diff --git a/doc/fluid/design/dynamic_rnn/2_level_rnn.dot b/doc/fluid/design/dynamic_rnn/2_level_rnn.dot
deleted file mode 100644
index 5d77865061ca7bbbfcf254dd938f09aef5553505..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dynamic_rnn/2_level_rnn.dot
+++ /dev/null
@@ -1,56 +0,0 @@
-digraph G {
-
- rnn [label="1st level RNN" shape=box]
-
- subgraph cluster0 {
- label = "time step 0"
-
- sent0 [label="sentence"]
- sent1 [label="sentence"]
-
- rnn1 [label="2nd level RNN" shape=box]
-
- sent0 -> rnn1
- sent1 -> rnn1
- }
-
- subgraph cluster1 {
- label = "time step 1"
-
- sent2 [label="sentence"]
- sent3 [label="sentence"]
-
- rnn2 [label="2nd level RNN" shape=box]
-
- sent2 -> rnn2
- sent3 -> rnn2
- }
-
- subgraph cluster2 {
- label = "time step 2"
-
- sent4 [label="sentence"]
- sent5 [label="sentence"]
-
- rnn3 [label="2nd level RNN" shape=box]
-
- sent4 -> rnn3
- sent5 -> rnn3
- }
-
-
- para0 [label="paragraph info 0"]
- para1 [label="paragraph info 1"]
- para2 [label="paragraph info 2"]
-
- rnn1 -> para0
- rnn2 -> para1
- rnn3 -> para2
-
- para0 -> rnn
- para1 -> rnn
- para2 -> rnn
-
- chapter [label="chapter info"]
- rnn -> chapter
-}
diff --git a/doc/fluid/design/dynamic_rnn/2_level_rnn.png b/doc/fluid/design/dynamic_rnn/2_level_rnn.png
deleted file mode 100644
index 0537a75beb175c0c284717421f7aa908da2a5038..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dynamic_rnn/2_level_rnn.png and /dev/null differ
diff --git a/doc/fluid/design/dynamic_rnn/index_cn.rst b/doc/fluid/design/dynamic_rnn/index_cn.rst
deleted file mode 100644
index 1d224d22cf7103616f44115db01f0ae55f1cb88a..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dynamic_rnn/index_cn.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-动态RNN
-------------
-
-.. toctree::
- :maxdepth: 1
-
- rnn.md
- rnn_design.md
diff --git a/doc/fluid/design/dynamic_rnn/index_en.rst b/doc/fluid/design/dynamic_rnn/index_en.rst
deleted file mode 100644
index 568f496e4ffe21a5e730488aef905f7e2d98839e..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dynamic_rnn/index_en.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-Dynamic RNN
-------------
-
-.. toctree::
- :maxdepth: 1
-
- rnn.md
- rnn_design.md
diff --git a/doc/fluid/design/dynamic_rnn/rnn.dot b/doc/fluid/design/dynamic_rnn/rnn.dot
deleted file mode 100644
index c1141cd9c981bb3cbf50d8bf7a6ed210280d79a5..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dynamic_rnn/rnn.dot
+++ /dev/null
@@ -1,87 +0,0 @@
-digraph G {
- label = "simple RNN implementation"
-
- ranksep=2;
-
- //graph [nodesep=1, ranksep=1];
-
- node[nodesep=1]
-
- subgraph cluster0 {
- label = "global scope"
- rankdir = TB
- W
- boot_memory
- input
- output
- }
-
- subgraph cluster1 {
- label = "step-scope 0"
- rankdir = TB
- memory0[label="memory"]
- prememory0[label="pre-memory"]
- step_input0[label="step input"]
- step_output0[label="step output"]
- }
-
- subgraph cluster2 {
- label = "step-scope 1"
- rankdir = TB
- memory1[label="memory"]
- prememory1[label="pre-memory"]
- step_input1[label="step input"]
- step_output1[label="step output"]
- }
-
- subgraph cluster3 {
- label = "step-scope 2"
- rankdir = TB
- memory2[label="memory"]
- prememory2[label="pre-memory"]
- step_input2[label="step input"]
- step_output2[label="step output"]
- }
-
- stepnet [shape=box]
- stepnet0 [shape=box, style=dashed]
- stepnet1 [shape=box, style=dashed]
- stepnet2 [shape=box, style=dashed]
-
-
- edge[color=blue]
- boot_memory -> prememory0 [label="init" color="blue"]
- memory0 -> prememory1 [label="copy/reference" color="blue"]
- memory1 -> prememory2 [label="copy/reference" color="blue"]
-
- edge[color=black]
- W -> stepnet0[constraint=false, style=dashed]
- W -> stepnet1[constraint=false, style=dashed]
- W -> stepnet2[constraint=false, style=dashed]
-
- memory0 -> stepnet0[style=dashed]
- prememory0 -> stepnet0 -> step_output0[style=dashed]
-
- memory1 -> stepnet1[style=dashed]
- prememory1 -> stepnet1 -> step_output1[style=dashed]
-
- memory2 -> stepnet2[style=dashed]
- prememory2 -> stepnet2 -> step_output2[style=dashed]
-
- input -> step_input0
- input -> step_input1
- input -> step_input2
-
- step_input0 -> stepnet0 [style=dashed]
- step_input1 -> stepnet1[style=dashed]
- step_input2 -> stepnet2[style=dashed]
-
- step_output0 -> output
- step_output1 -> output
- step_output2 -> output
-
- stepnet0 -> stepnet[style=dashed]
- stepnet1 -> stepnet[style=dashed]
- stepnet2 -> stepnet[style=dashed]
-
-}
diff --git a/doc/fluid/design/dynamic_rnn/rnn.jpg b/doc/fluid/design/dynamic_rnn/rnn.jpg
deleted file mode 100644
index 9867e404cf959df0dce6ded5222b466c788fb840..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/dynamic_rnn/rnn.jpg and /dev/null differ
diff --git a/doc/fluid/design/dynamic_rnn/rnn.md b/doc/fluid/design/dynamic_rnn/rnn.md
deleted file mode 100644
index b39ae0675c45e56852293d97f45e91861cf31667..0000000000000000000000000000000000000000
--- a/doc/fluid/design/dynamic_rnn/rnn.md
+++ /dev/null
@@ -1,153 +0,0 @@
-# RNNOp design
-
-This document describes the RNN (Recurrent Neural Network) operator and how it is implemented in PaddlePaddle. The RNN op requires that all instances in a mini-batch have the same length. We will have a more flexible dynamic RNN operator in the future.
-
-## RNN Algorithm Implementation
-
-
-
-
-
-Figure 2 illustrates the RNN's data flow
-
-
-
-
-
-
-cudnn provides APIs to finish the whole series of computation, we can use them in our GPU kernel.
-
-### Python
-
-`batch_norm_op` is warpped as a layer in Python:
-
-```python
-def batch_norm_layer(net,
- input,
- output,
- scale,
- bias,
- use_global_est = False,
- epsilon = 1e-6,
- momentum = 0.99):
- mean_cache = scope.new_var(name = 'estimated_mean', trainable = False)
- var_cache = scop.new_var(name = 'estimated_var', trainable = False)
- batch_mean = scope.new_var(name = 'batch_mean')
- batch_var = scope.new_var(name = 'batch_var')
- batch_norm_op = Operator('batch_norm_op',
- x = input,
- estimated_mean = mean_cache,
- estimated_mean = var_cache,
- scale = scale,
- bias = bias,
- y = output,
- batch_mean = batch_mean,
- batch_var = batch_var,
- saved_mean = mean_cache,
- saved_var = var_cache,
- is_infer = False,
- use_global_est = use_global_est,
- epsilon = epsilon,
- momentum = momentum)
- net.append_op(batch_norm_op)
- return output
-```
-
-Because Python API has not been finally decided, the code above can be regarded as pseudo code. There are a few key points we shall note:
-
-1. `estimated_mean` and `estimated_var` are assigned the same variables with `saved_mean` and `saved_var` respectively. So they share same the memories. The output mean and variance values(`saved_mean` and `saved_var`) of a certain batch will be the inputs(`estimated_mean` and `estimated_var`) of the next batch.
-
-2. `is_infer` decided whether `batch_norm_op` will run in training mode or inferencing mode. However, a network may contains both training and inferencing parts. And user may switch `batch_norm_op`'s running mode in Python `for` loop like this:
-
-```python
-for pass_id in range(PASS_NUM):
- # ...
- net.train() # run training model
- if pass_id % 100 == 0:
- net.infer(test_image) # run inferencing model
- # ...
-```
-
-`is_infer` is an attribute. Once an operator is created, its attributes can not be changed. It suggests us that we shall maintain two `batch_norm_op` in the model, one's `is_infer` is `True`(we call it `infer_batch_norm_op`) and the other one's is `False`(we call it `train_batch_norm_op`). They share all parameters and variables, but be placed in two different branches. That is to say, if a network contains a `batch_norm_op`, it will fork into two branches, one go through `train_batch_norm_op` and the other one go through `infer_batch_norm_op`:
-
-
-| Python classes | -Protobuf messages | -
|---|---|
| Program | -ProgramDesc | -
| Block | -BlockDesc | -
| Operator | -OpDesc | -
| Variable | -VarDesc | -
-
-```python
-def G(in):
- # over-simplified example as G has only one layers:
- return paddle.layer.fc(in, parameter_name="G")
-
-def D(in);
- # again, over-simplified:
- return paddle.layer.fc(in, parameter_name="D")
-
-# Construct the first topology, which contains both D and G.
-# By learning this topology, we update parameters of G.
-d0 = paddle.layer.should_be_false(D(G(paddle.layer.data())))
-
-# Construct a second topology d1, which contains only D. By
-# training this topology, we update parameters of D. Note
-# that d1 share parameters with d0.
-d1 = paddle.layer.should_be_true(D(paddle.layer.data()))
-
-# Create parameters from a list of multiple topologies (models) for
-# the chance to share parameters between these topologies.
-parameters = paddle.parameters.create([d0, d1])
-
-# Iterative training of GAN.
-for ...:
- train(d0, parameters, reader=read_from_rng, immutable_parameters={"D"})
- train(d1, parameters, reader=read_from_realistic_images)
-
-# Use d1 for inference:
-print "D thinks a batch of images are realistic ", infer(d1, parameters, read_mnist_images)
-```
-
-
-### Summarization
-
-
-Above two programs reveal some important design concerns:
-
-1. Users describe a topology as an expression of layers. Every layer
- has a *parameter name*. If the users don't specify it explicitly, it's automatically generated as a unique name. By
- specifying the parameter name, users can specify the sharing of
- parameters between layers and even between topologies.
-
-1. `paddle.parameters.create` figures out parameters required by one
- or more topologies from parameter names of layers. It creates these
- parameters and returns a `ParameterSet` object, which is in essence
- a map from *parameter names* to *parameters*.
-
-1. At training and inference time, `paddle.train` and `paddle.infer`
- requires both a topology and the parameter set that holds the parameters of that topology. There are some reasons:
-
- 1. This prevents users from forgetting to call
- `paddle.parameters.create`.
- 1. `paddle.train` needs to know which parameter set to update.
- 1. Users could load another (pre-trained) parameter set and use it
- with a topology in `train.infer`.
-
-1. By specifying the `immutable_parameters` parameter of
- `paddle.train`, we can forbid the update of these parameters.
-
-
-## Reader
-
-Not all programming frameworks allow users to define I/O functions.
-An example is Google MapReduce, which can only read from text,
-SSTable, and RecordIO files. Hadoop MapReduce allows users to define
-readers and writers by deriving from base classes `Reader` and
-`Writer`. The former is less flexible but also less error-prone. We
-decide to provide the flexibility to users to define their readers.
-
-
-There are some open questions here:
-
-1. **Should a reader return a Python dictionary?**
-
-1. **How to map multiple outputs from a reader to multiple data layers?**
-
-1. **How to easily compose some existing readers to read more data and
- feed a topology with more data layers?**
-
-
-## Training
-
-The recommended way to training a model is to call `paddle.train`,
-which simply calls `paddle.trainer.Default`, a global variable of
-type `paddle.trainer.SGD`. Equivalently, we can do
-
-```python
-opt = paddle.trainer.SGD(..., paddle.updater.Adam(...))
-opt.train(topology, parameters, reader=read, ...)
-```
-
-### Updater
-
-Please be aware that a trainer can accept an updater as its data
-member, where an updater is a class derived from
-`paddle.trainer.Updater`. This is to make it easier to customize
-trainers, as discussed
-[here](https://github.com/PaddlePaddle/Paddle/issues/1319).
-
-### Event Handler
-
-`paddle.train` and `paddle.trainer.XXX.train` take an optional
-parameter `event_handler`, which should be either `None` or a function
-that handle some events:
-
-1. BeginTraining
-1. EndTraining
-1. BeginIteration
-1. EndIteration
-1. BeginPass
-1. EndPass
-
-where EndPass is sent if and only if the reader yields
-`end_pass=True`.
-
-An example as follows:
-
-```python
-def event_handler(event):
- if ininstance(event, paddle.event.EndIteration):
- print paddle.test(...)
-
-paddle.train(topology, parameters, reader, event_handler)
-```
-
-If we are writing a PaddlePaddle program in and for iPython/Jypyter,
-we can use metaplotlib in the event handler to plot a curve of
-cost/error versus iterations, as shown
-[here](https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/).
-
-### Distributed Training
-
-If users want to do distributed training on a cluster, s/he should
-call `paddle.dist_train` and provides access tokens to the cluster as
-a parameter.
-
-For example, if the user has a TLS certificate that allows him to
-access a Kubernetes cluster, s/he should be able to call
-
-```python
-paddle.dist_train(model,
- trainer=paddle.trainer.SGD(...,
- paddle.updater.Adam(...)),
- reader=read,
- k8s_user="yi",
- k8s_token="kube_cluster_tls.pem",
- k8s_job="hello",
- num_parameter_servers=15)
-```
-
-The pseudo code of `paddle.dist_train` is as follows:
-
-```python
-def dist_train(topology, parameters, trainer, reader, ...):
- if os.getenv("KUBERNETES_SERVICE_HOST") == None:
- image_name = k8s_user + '/' + k8s_job
- docker_build(image_name)
- docker_push()
- kube_ctrl_start_job(image_name, k8s_user, k8s_token)
- else:
- rank = kube_list_containers_in_job_and_return_current_containers_rank()
- if rank == 0:
- master()
- elif rank < 15:
- parameter_server()
- else:
- trainer.train(model, reader=read)
-```
-
-Please be aware that if a process is running on the Kubernetes
-cluster, it will have some environment variables pre-defined.
-
-If `dist_train` doesn't see these environment variables, it knows
-that it's running on users' personal computer, and it should work as a
-*launcher*. Otherwise, it knows that it's running on the cluster and
-need to figure out its role as either the master, or a trainer, or a
-parameter server.
diff --git a/doc/fluid/design/motivation/fluid-compiler.graffle b/doc/fluid/design/motivation/fluid-compiler.graffle
deleted file mode 100644
index c933df2cb855462c52b2d25f7f9a99b95652961d..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/motivation/fluid-compiler.graffle and /dev/null differ
diff --git a/doc/fluid/design/motivation/fluid-compiler.png b/doc/fluid/design/motivation/fluid-compiler.png
deleted file mode 100644
index 1b0ffed2039c91a3a00bbb719da08c91c3acf7bb..0000000000000000000000000000000000000000
Binary files a/doc/fluid/design/motivation/fluid-compiler.png and /dev/null differ
diff --git a/doc/fluid/design/motivation/fluid.md b/doc/fluid/design/motivation/fluid.md
deleted file mode 100644
index 4b7696cc1bbf57ace72c4d31ffc2bfe6c1071939..0000000000000000000000000000000000000000
--- a/doc/fluid/design/motivation/fluid.md
+++ /dev/null
@@ -1,140 +0,0 @@
-# Design Doc: PaddlePaddle Fluid
-
-## Why Fluid
-
-When Baidu developed PaddlePaddle in 2013, the only well-known open source deep learning system at the time was Caffe. However, when PaddlePaddle was open-sourced in 2016, many other choices were available. There was a challenge -- what is the need for open sourcing yet another deep learning framework?
-
-Fluid is the answer. Fluid is similar to PyTorch and TensorFlow Eager Execution, which describes the "process" of training or inference using the concept of a model. In fact in PyTorch, TensorFlow Eager Execution and Fluid, there is no concept of a model at all. The details are covered in the sections below. Fluid is currently more extreme in the above mentioned idea than PyTorch and Eager Execution, and we are trying to push Fluid towards the directions of a compiler and a new programming language for deep learning.
-
-## The Evolution of Deep Learning Systems
-
-Deep learning infrastructure is one of the fastest evolving technologies. Within four years, there have already been three generations of technologies invented.
-
-| Existed since | -model as sequence of layers | -model as graph of operators | -No model | -
|---|---|---|---|
| 2013 | -Caffe, Theano, Torch, PaddlePaddle | -- | - |
| 2015 | -- | TensorFlow, MxNet, Caffe2, ONNX, n-graph | -- |
| 2016 | -- | - | PyTorch, TensorFlow Eager Execution, PaddlePaddle Fluid | -
| - | Representation (protobuf messages) | -Realization (C++ class objects) | -
|---|---|---|
| Data | --VarDesc | --Variable | -
| Operation | --OpDesc | --Operator | -
| Block | -BlockDesc | -Block | - -
| Roadmap | -Description | -Parallelizable Tasks | -
|---|---|---|
| Phase I | -Simplified model & components | -Task 1 ~ Task 8 | -
| Phase II | -Standard model & benchmarking & profiling | -Task 9 ~ Task 12 | -
| Phase III | -Documentations | -Task13 ~ Task14 | -
| Required Components | -PaddlePaddle Support | -Need to Develop | -
|---|---|---|
| Data Layer I (Spectrogram) | -Not supported yet. | -TBD (Task 3) | -
| Data Layer II (Transcription) | -paddle.data_type.integer_value_sequence | -- | -
| 2D Convolution Layer | -paddle.layer.image_conv_layer | -- | -
| DataType Converter (vec2seq) | -paddle.layer.block_expand | -- | -
| Bi-/Uni-directional RNNs | -paddle.layer.recurrent_group | -- | -
| Row Convolution Layer | -Not supported yet. | -TBD (Task 4) | -
| CTC-loss Layer | -paddle.layer.warp_ctc | -- | -
| Batch Normalization Layer | -paddle.layer.batch_norm | -- | -
| CTC-Beam search | -Not supported yet. | -TBD (Task 6) | -
- 
-
-
-
-
-Figure 1. The overall running logic of GAN. The black solid arrows indicate the forward pass; the green dashed arrows indicate the backward pass of generator training; the red dashed arrows indicate the backward pass of the discriminator training. The BP pass of the green (red) arrow should only update the parameters in the green (red) boxes. The diamonds indicate the data providers. d\_loss and g\_loss marked in red and green are the two targets we would like to run.
-
-
-Figure 2. Photo borrowed from the original DC-GAN paper.
-
-
-Figure 1. Forward in training with simulated quantization.
-
-
-Figure 2. Equivalent forward in training with simulated quantization.
-
-
-Figure 3. Backward and weight updating in training with simulated quantization.
-
| 内容 | -定义位置 | -
|---|---|
| OpProtoMake定义 | -.cc 文件，Backward Op不需要定义OpProtoMake | -
| Op定义 | -.cc 文件 | -
| Kernel实现 | -CPU、CUDA共享Kernel实现在.h 文件中，否则，CPU 实现在.cc 文件中，CUDA 实现在.cu 文件中。 | -
| 注册Op | -Op注册实现在.cc 文件；Kernel注册CPU实现在.cc 文件中，CUDA实现在.cu 文件中 | -
| Information | -Where is it defined | -
|---|---|
| OpProtoMake definition | -`.cc`files, Backward Op does not need an OpProtoMake interface. | -
| Op definition | -`.cc` files | -
| Kernel implementation | -The kernel methods shared between CPU and CUDA are defined in `.h` files. CPU-specific kernels live in `.cc` files, while CUDA-specific kernels are implemented in `.cu`files. | -
| Registering the Op | -Ops are registered in `.cc` files; For Kernel registration, `.cc` files contain the CPU implementation, while `.cu` files contain the CUDA implementation. | -
-1. 等待编译完成后可以在此页面的"Artifacts"下拉框中找到生成的3个二进制文件，分别对应CAPI，`cp27m`和`cp27mu`的版本。
-1. 由于pypi.python.org目前遵循[严格的命名规范PEP 513](https://www.python.org/dev/peps/pep-0513)，在使用twine上传之前，需要重命名wheel包中platform相关的后缀，比如将`linux_x86_64`修改成`manylinux1_x86_64`。
-1. 上传：
-```
-cd build/python
-pip install twine
-twine upload dist/[package to upload]
-```
-
-* 注：CI环境使用 https://github.com/PaddlePaddle/buildtools 这里的DockerImage作为编译环境以支持更多的Linux
- 发型版，如果需要手动编译，也可以使用这些镜像。这些镜像也可以从 https://hub.docker.com/r/paddlepaddle/paddle_manylinux_devel/tags/ 下载得到。
-* pypi不支持覆盖上传，所以一个版本号的wheel包发布之后，不可以更改。下一个wheel包需要更新版本号才可以上传。
-
-## 发布Docker镜像
-
-上述PaddlePaddle CI编译wheel完成后会自动将Docker镜像push到DockerHub，所以，发布Docker镜像只需要对自动push的镜像打上
-版本号对应的tag即可：
-
-```
-docker pull [镜像]:latest
-docker tag [镜像]:latest [镜像]:[version]
-docker push [镜像]:[version]
-```
-
-需要更新的镜像tag包括：
-
-* `[version]`: CPU版本
-* `[version]-openblas`: openblas版本
-* `[version]-gpu`: GPU版本（CUDA 8.0 cudnn 5）
-* `[version]-gpu-[cudaver]-[cudnnver]`: 不同cuda, cudnn版本的镜像
-
-之后可进入 https://hub.docker.com/r/paddlepaddle/paddle/tags/ 查看是否发布成功。
-
-## PaddlePaddle 分支规范
-
-PaddlePaddle开发过程使用[Trunk Based Development](https://trunkbaseddevelopment.com/) 开发规范。
-
-* `develop`分支为开发(develop branch)版本分支。每一个`develop`分支的版本都经过单元测试。并且会经过模型回归测试。
-* `release/版本号`分支为每一次Release时建立的临时分支。release分支主要用于测试，bug修复和最终发版。
-* `master`分支因为历史原因，已经废弃。
-
-* 其他开发者fork的feature branch。
- * 建议，开发者的feature branch需要同步主版本库的`develop`分支。
- * 建议，开发者的feature branch需要基于主版本库中的`develop`分支。
- * 当feature branch开发完毕后，向PaddlePaddle的主版本库提交`Pull Reuqest`，进而进行代码评审。
- * 在评审过程中，开发者修改自己的代码，可以继续在自己的feature branch提交代码。
-
-## PaddlePaddle回归测试列表
-
-TODO
-
-### PaddlePaddle Book中所有章节
-
-PaddlePaddle每次发版本首先要保证PaddlePaddle Book中所有章节功能的正确性。功能的正确性包括验证PaddlePaddle目前的`paddle_trainer`训练和纯使用`Python`训练（V2和Fluid）模型正确性。
-
-| - | 新手入门章节 | -识别数字 | -图像分类 | -词向量 | -情感分析 | -语意角色标注 | -机器翻译 | -个性化推荐 | -
|---|---|---|---|---|---|---|---|---|
| API.V2 + Docker + GPU | -- | - | - | - | - | - | - | - |
| API.V2 + Docker + CPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Docker + GPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Docker + CPU | -- | - | - | - | - | - | - | - |
| API.V2 + Ubuntu + GPU | -- | - | - | - | - | - | - | - |
| API.V2 + Ubuntu + CPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Ubuntu + GPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Ubuntu + CPU | -- | - | - | - | - | - | - | - |
-1. After the build succeeds, download the outputs under "Artifacts" including capi, `cp27m` and `cp27mu`.
-1. Since pypi.python.org follows [PEP 513](https://www.python.org/dev/peps/pep-0513), before we
- upload the package using `twine`, we need to rename the package from `linux_x86_64` to
- `manylinux1_x86_64`.
-1. Start the upload:
- ```
- cd build/python
- pip install twine
- twine upload dist/[package to upload]
- ```
-
-* NOTE: We use a special Docker image to build our releases to support more Linux distributions, you can
- download it from https://hub.docker.com/r/paddlepaddle/paddle_manylinux_devel/tags/, or build it using
- scripts under `tools/manylinux1`.
-* pypi does not allow overwrite the already uploaded version of wheel package, even if you delete the
- old version. you must change the version number before upload a new one.
-
-### Publish wheel Packages for MacOS
-
-You need to build the binary wheel package for MacOS before publishing, to
-make sure that the package can be used by many versions of MacOS
-(10.11, 10.12, 10.13) and different python installs (python.org, homebrew, etc.),
-you must build the package ***exactly*** following below steps:
-
-Build steps:
-
-1. install python from python.org downloads, and make sure it's currently in use
- in your system.
-1. `export MACOSX_DEPLOYMENT_TARGET=10.11`, use `10.11` is enough for recent versions.
-1. `git clone https://github.com/PaddlePaddle/Paddle.git && cd Paddle && mkdir build && cd build`
-1. `cmake -DWITH_GPU=OFF -DWITH_MKL=OFF -DWITH_SYSTEM_BLAS=OFF ..`, make sure the output of `cmake` command is using the correct python interpreter installed from python.org
-1. `make -j`
-1. `pip install delocate`
-1. `mkdir fixed_wheel && delocate-wheel -w fixed_wheel python/dist/*.whl`
-
-Then the whl under `fixed_wheel` is ready to upload.
-
-Install steps:
-
-1. run `pip install paddlepaddle...whl`
-1. find the `libpython.dylib` that are currently in use:
- - for python.org package installs, do nothing.
- - for other python installs, find the path of `libpython*.dylib` and `export LD_LIBRARY_PATH=you path && DYLD_LIBRARY_PATH=your path`
-
-## Publish Docker Images
-
-Our CI tool will push latest images to DockerHub, so we only need to push a version tag like:
-
-```
-docker pull [image]:latest
-docker tag [image]:latest [image]:[version]
-docker push [image]:[version]
-```
-
-Tags that need to be updated are:
-* `[version]`: CPU only version image
-* `[version]-openblas`: openblas version image
-* `[version]-gpu`: GPU version（using CUDA 8.0 cudnn 5）
-* `[version]-gpu-[cudaver]-[cudnnver]`: tag for different cuda, cudnn versions
-
-You can then checkout the latest pushed tags at https://hub.docker.com/r/paddlepaddle/paddle/tags/.
-
-## Branching Model
-
-PaddlePaddle uses [Trunk Based Development](https://trunkbaseddevelopment.com/) as our branching model.
-
-* `develop` branch is used for development. Each comment to `develop` branc goes through unit tests and model regression tests.
-* `release/[version]` branch is used for each release. Release branch is used for tests, bug fix and evetual release.
-* `master` branch as been deprecated for historical reasons
-
-* Developer's feature branch。
- * Developer's feature branch should sync with upstream `develop` branch.
- * Developer's feature branch should be forked from upstream `develop` branch.
- * After feature branch is ready, create a `Pull Request` against the Paddle repo and go through code review.
- * In the review process, develop modify codes and push to their own feature branch.
-
-## PaddlePaddle Regression Test List
-
-TODO
-
-### All Chapters of PaddlePaddle Book
-
-We need to guarantee that all the chapters of PaddlePaddle Book can run correctly. Including
-V1 (`paddle_trainer` training) and V2 training and Fluid training.
-
-| - | Linear Regression | -Recognize Digits | -Image Classification | -Word2Vec | -Personalized Recommendation | -Sentiment Analysis | -Semantic Role Labeling | -Machine Translation | -
|---|---|---|---|---|---|---|---|---|
| API.V2 + Docker + GPU | -- | - | - | - | - | - | - | - |
| API.V2 + Docker + CPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Docker + GPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Docker + CPU | -- | - | - | - | - | - | - | - |
| API.V2 + Ubuntu + GPU | -- | - | - | - | - | - | - | - |
| API.V2 + Ubuntu + CPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Ubuntu + GPU | -- | - | - | - | - | - | - | - |
| `paddle_trainer` + Ubuntu + CPU | -- | - | - | - | - | - | - | - |