diff --git a/doc/fluid/design/algorithm/parameter_average.md b/doc/fluid/design/algorithm/parameter_average.md index 940d37fb31dcd0c50ea6c4c42b052d7cb23a9c47..340bc302d57429a9bf10a9d23ed9b0cdc7a2a568 100644 --- a/doc/fluid/design/algorithm/parameter_average.md +++ b/doc/fluid/design/algorithm/parameter_average.md @@ -49,9 +49,9 @@ In the new design, we propose to create a new operation for averaging parameter - the optimizer - the window_size to keep the updates -The ParameterAverageOptimizer op can be like any other operator with its own CPU/GPU implementation either using Eigen or separate CPU and GPU kernels. As the initial implementation, we can implement the kernel using Eigen following the abstraction pattern implemented for [Operators](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/rmsprop_op.h). We also want to support the case when the Trainer/Optimizer runs on the GPU while ParameterAverageOptimizer runs on a CPU. +The ParameterAverageOptimizer op can be like any other operator with its own CPU/GPU implementation either using Eigen or separate CPU and GPU kernels. As the initial implementation, we can implement the kernel using Eigen following the abstraction pattern implemented for [Operators](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/rmsprop_op.h). We also want to support the case when the Trainer/Optimizer runs on the GPU while ParameterAverageOptimizer runs on a CPU. -The idea of building an op for averaging is in sync with the refactored PaddlePaddle philosophy of using operators to represent any computation unit. The way the op will be added to the computation graph will be decided by the [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) in Python API. +The idea of building an op for averaging is in sync with the refactored PaddlePaddle philosophy of using operators to represent any computation unit. The way the op will be added to the computation graph will be decided by the [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/python_api.md#layer-function) in Python API. ### Python API implementation for ParameterAverageOptimizer @@ -59,8 +59,8 @@ Based on Polyak and Juditsky (1992), we can generalize the averaging of updates - Any optimizer (RMSProp , AdaGrad etc.) - A window size. The op keeps accumulating updated parameter values over a window of N batches and takes an average. Move the averaged value to a buffer when window is full to avoid loss of precision. -Using the ParameterAverageOptimizer op, any user can add the operation to their computation graphs. However, this will require a lot of lines of code and we should design Python APIs that support averaging. As per the PaddlePaddle [Python API design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md), the layer functions are responsible for creating operators, operator parameters and variables. Since ParameterAverageOptimizer will be an operator, it makes sense to create it in the layer functions. -We will have a wrapper written in Python that will support the functionality and implement the actual core computation in C++ core as we have done for other [Optimizers](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/rmsprop_op.cc) +Using the ParameterAverageOptimizer op, any user can add the operation to their computation graphs. However, this will require a lot of lines of code and we should design Python APIs that support averaging. As per the PaddlePaddle [Python API design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/python_api.md), the layer functions are responsible for creating operators, operator parameters and variables. Since ParameterAverageOptimizer will be an operator, it makes sense to create it in the layer functions. +We will have a wrapper written in Python that will support the functionality and implement the actual core computation in C++ core as we have done for other [Optimizers](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/rmsprop_op.cc) #### Creation of the ParameterAverageOptimizer operator There are two ways for creating the ParameterAverageOptimizer op: @@ -71,4 +71,4 @@ The proposal is to add the op immediately while building the computation graph. #### High-level API -In PaddlePaddle Python API, users will primarily rely on [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) to create neural network layers. Hence, we also need to provide parameter average functionality in layer functions. +In PaddlePaddle Python API, users will primarily rely on [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/python_api.md#layer-function) to create neural network layers. Hence, we also need to provide parameter average functionality in layer functions. diff --git a/doc/fluid/design/concepts/block.md b/doc/fluid/design/concepts/block.md index 3b626bd89cd83a9428997abccfeeebbbbdbb3d38..3757cd055c818be1e63ee8c0f000f4dd299b59f4 100644 --- a/doc/fluid/design/concepts/block.md +++ b/doc/fluid/design/concepts/block.md @@ -113,7 +113,7 @@ if (cond) { ``` -An equivalent PaddlePaddle program from the design doc of the [IfElseOp operator](./if_else_op.md) is as follows: +An equivalent PaddlePaddle program from the design doc of the [IfElseOp operator](../execution/if_else_op.md) is as follows: ```python import paddle as pd @@ -140,7 +140,7 @@ The difference is that variables in the C++ program contain scalar values, where ### Blocks with `for` and `RNNOp` -The following RNN model in PaddlePaddle from the [RNN design doc](./rnn.md) : +The following RNN model in PaddlePaddle from the [RNN design doc](../dynamic_rnn/rnn.md) : ```python x = sequence([10, 20, 30]) # shape=[None, 1] diff --git a/doc/fluid/design/concepts/executor.md b/doc/fluid/design/concepts/executor.md index 2d4b371cc56db82ce5747da6db07f05aa7f7e6c1..3fcddf4dd90f826ee1a16713f4371fb010f8eac5 100644 --- a/doc/fluid/design/concepts/executor.md +++ b/doc/fluid/design/concepts/executor.md @@ -1,7 +1,7 @@ # Executor Design Doc ## Motivation -In [fluid](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/fluid.md), we encourage the user to use deep learning programming paradigms to describe the training process. When the user-written Python program is executed, it will first create a protobuf message +In [fluid](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/motivation/fluid.md), we encourage the user to use deep learning programming paradigms to describe the training process. When the user-written Python program is executed, it will first create a protobuf message [`ProgramDesc`](https://github.com/PaddlePaddle/Paddle/blob/a91efdde6910ce92a78e3aa7157412c4c88d9ee8/paddle/framework/framework.proto#L145) that describes the process and is conceptually like an [abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree). The executor runs the `ProgramDesc` like an interpreter. `ProgramDesc` contains the intrinsics (operators in this case) and variables which will be used, executor explicitly executes the stored precompiled code. diff --git a/doc/fluid/design/concepts/program.md b/doc/fluid/design/concepts/program.md index bd2456787c4e336d357a65255a8274a7c9e465cc..cfcd21ecdb9d2844bf93ed98a56db09651077c40 100644 --- a/doc/fluid/design/concepts/program.md +++ b/doc/fluid/design/concepts/program.md @@ -4,7 +4,7 @@ A PaddlePaddle program consists of two parts -- the first generates a `ProgramDesc` protobuf message that describes the program, and the second runs this message using a C++ class `Executor`. -A simple example PaddlePaddle program can be found in [graph.md](./graph.md): +A simple example PaddlePaddle program can be found in [graph.md](../others/graph.md): ```python x = layer.data("images") diff --git a/doc/fluid/design/concurrent/concurrent_programming.md b/doc/fluid/design/concurrent/concurrent_programming.md index 1859f983e9133674e69ecd506d7683ea926b2b8f..0428e74f9e00a87f6b0972057f48479b8ae56ad6 100644 --- a/doc/fluid/design/concurrent/concurrent_programming.md +++ b/doc/fluid/design/concurrent/concurrent_programming.md @@ -1,6 +1,6 @@ # Design Doc: Concurrent Programming with Fluid -With PaddlePaddle Fluid, users describe a program other than a model. The program is a [`ProgramDesc`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto) protobuf message. TensorFlow/MxNet/Caffe2 applications generate protobuf messages too, but their protobuf messages represent the model, a graph of operators, but not the program that trains/uses the model. +With PaddlePaddle Fluid, users describe a program other than a model. The program is a [`ProgramDesc`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/framework.proto) protobuf message. TensorFlow/MxNet/Caffe2 applications generate protobuf messages too, but their protobuf messages represent the model, a graph of operators, but not the program that trains/uses the model. Many know that when we program TensorFlow, we can specify the device on which each operator runs. This allows us to create a concurrent/parallel AI application. An interesting questions is **how does a `ProgramDesc` represents a concurrent program?** @@ -28,19 +28,19 @@ The following table compares concepts in Fluid and Go control-flow and built-in functions -intrinsics/operators +intrinsics/operators goroutines, channels -class ThreadPool +class ThreadPool runtime -class Executor +class Executor @@ -78,7 +78,7 @@ message ProgramDesc { } ``` -Then, the default `main` function calls `fluid.run()`, which creates an instance of the [`class Executor`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/executor.h) and calls `Executor.Run(block[0])`, where `block[0]` is the first and only block defined in above `ProgramDesc` message. +Then, the default `main` function calls `fluid.run()`, which creates an instance of the [`class Executor`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/executor.h) and calls `Executor.Run(block[0])`, where `block[0]` is the first and only block defined in above `ProgramDesc` message. The default `main` function is defined as follows: @@ -146,7 +146,7 @@ An explanation of the above program: - `fluid.k8s` is a package that provides access to Kubernetes API. - `fluid.k8s.get_worker_addrs` returns the list of IP and ports of all pods of the current job except for the current one (the master pod). -- `fluid.tensor_array` creates a [tensor array](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor_array.h). `fluid.parallel_for` creates a `ParallelFor` intrinsic, which, when executed, +- `fluid.tensor_array` creates a [tensor array](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/lod_tensor_array.h). `fluid.parallel_for` creates a `ParallelFor` intrinsic, which, when executed, 1. creates `len(L)` scopes, each for the concurrent running of the sub-block (block 1 in this case), and initializes a variable named "index" in the scope to an integer value in the range `[0, len(L)-1]`, and 2. creates `len(L)` threads by calling into the `ThreadPool` singleton, each thread @@ -175,7 +175,7 @@ where 1. listens on the current pod's IP address, as returned by `fliud.k8s.self_addr()`, 2. once a connection is established, 1. creates a scope of two parameters, "input" and "output", - 2. reads a [Fluid variable](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h) and saves it into "input", + 2. reads a [Fluid variable](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/variable.h) and saves it into "input", 3. creates an Executor instance and calls `Executor.Run(block)`, where the block is generated by running the lambda specified as the second parameter of `fluid.listen_and_do`. ## Summarization diff --git a/doc/fluid/design/dist_train/distributed_architecture.md b/doc/fluid/design/dist_train/distributed_architecture.md index 229cb47c17d633be6848bb35e58d33ec9b47ec3b..371bbeebf7559eccc77ba0eea4f6f87a1bc5b54a 100644 --- a/doc/fluid/design/dist_train/distributed_architecture.md +++ b/doc/fluid/design/dist_train/distributed_architecture.md @@ -177,7 +177,7 @@ The local training architecture will be the same as the distributed training arc ### Training Data In PaddlePaddle v0.10.0, training data is typically read -with [data reader](../reader/README.md) from Python. This approach is +with [data reader](./README.md) from Python. This approach is no longer efficient when training distributedly since the Python process no longer runs on the same node with the trainer processes, the Python reader will need to read from the distributed filesystem diff --git a/doc/fluid/design/dist_train/parameter_server.md b/doc/fluid/design/dist_train/parameter_server.md index 73c85da5e89eee0ac7857a0b808bc64ae673fdad..563b70bc0e852bec953eb40dda3c46b3d45d7e68 100644 --- a/doc/fluid/design/dist_train/parameter_server.md +++ b/doc/fluid/design/dist_train/parameter_server.md @@ -65,7 +65,7 @@ For embedding layers, the gradient may have many rows containing only 0 when tra if the gradient uses a dense tensor to do parameter optimization, it could spend unnecessary memory, slow down the calculations and waste the bandwidth while doing distributed training. -In Fluid, we introduce [SelectedRows](../selected_rows.md) to represent a list of rows containing +In Fluid, we introduce [SelectedRows](../modules/selected_rows.md) to represent a list of rows containing non-zero gradient data. So when we do parameter optimization both locally and remotely, we only need to send those non-zero rows to the optimizer operators: diff --git a/doc/fluid/design/dynamic_rnn/rnn.md b/doc/fluid/design/dynamic_rnn/rnn.md index 7b61b050f640814d6949cf6847b431da53d59581..b39ae0675c45e56852293d97f45e91861cf31667 100644 --- a/doc/fluid/design/dynamic_rnn/rnn.md +++ b/doc/fluid/design/dynamic_rnn/rnn.md @@ -22,7 +22,7 @@ There are several important concepts here: There could be local variables defined in each step-net. PaddlePaddle runtime realizes these variables in *step-scopes* which are created for each step.

-
+
Figure 2 illustrates the RNN's data flow

@@ -93,7 +93,7 @@ For example, we could have a 2-level RNN, where the top level corresponds to par The following figure illustrates feeding in text into the lower level, one sentence at a step, and the feeding in step outputs to the top level. The final top level output is about the whole text.

- +

```python @@ -149,5 +149,5 @@ If the `output_all_steps` is set to False, it will only output the final time st

- +

diff --git a/doc/fluid/design/index_cn.rst b/doc/fluid/design/index_cn.rst index e9f55214f411abb11bef180d7af4716ad85a0b09..31b62a5eb3cd9b5b68d51abcd001fd5b8c39a914 100644 --- a/doc/fluid/design/index_cn.rst +++ b/doc/fluid/design/index_cn.rst @@ -9,7 +9,7 @@ concepts/index_cn.rst data_type/index_cn.rst memory/index_cn.rst - muti_devices/index_cn.rst + multi_devices/index_cn.rst dynamic_rnn/index_cn.rst concurrent/index_cn.rst algorithm/index_cn.rst diff --git a/doc/fluid/design/index_en.rst b/doc/fluid/design/index_en.rst index 2802dc3a31d540c5a19bf9042053496aad152f98..2bfee02ad4626633b08ddff747e2886faf9ba99f 100644 --- a/doc/fluid/design/index_en.rst +++ b/doc/fluid/design/index_en.rst @@ -9,7 +9,7 @@ Design concepts/index_en.rst data_type/index_en.rst memory/index_en.rst - muti_devices/index_en.rst + multi_devices/index_en.rst dynamic_rnn/index_en.rst concurrent/index_en.rst algorithm/index_en.rst diff --git a/doc/fluid/design/modules/python_api.md b/doc/fluid/design/modules/python_api.md index f83ad3b6a4e8b4d82d8fe8d4154a2739a9b9628b..265732a348ea77d21005e335390d99abcdfbd045 100644 --- a/doc/fluid/design/modules/python_api.md +++ b/doc/fluid/design/modules/python_api.md @@ -36,7 +36,7 @@ Please be aware that these Python classes need to maintain some construction-tim ### Program -A `ProgramDesc` describes a [DL program](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/program.md), which is composed of an array of `BlockDesc`s. The `BlockDesc`s in a `ProgramDesc` can have a tree-like hierarchical structure. However, the `ProgramDesc` onlys stores a flattened array of `BlockDesc`s. A `BlockDesc` refers to its parent block by its index in the array. For example, operators in the step block of an RNN operator need to be able to access variables in its ancestor blocks. +A `ProgramDesc` describes a [DL program](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/concepts/program.md), which is composed of an array of `BlockDesc`s. The `BlockDesc`s in a `ProgramDesc` can have a tree-like hierarchical structure. However, the `ProgramDesc` onlys stores a flattened array of `BlockDesc`s. A `BlockDesc` refers to its parent block by its index in the array. For example, operators in the step block of an RNN operator need to be able to access variables in its ancestor blocks. Whenever we create a block, we need to set its parent block to the current block, hence the Python class `Program` needs to maintain a data member `current_block`. @@ -70,7 +70,7 @@ class Program(objects): ### Block -A [Block](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md) includes +A [Block](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/concepts/block.md) includes 1. a map from variable names to an instance of the Python `Variable` class, and 1. a list of `Operator` instances. diff --git a/doc/fluid/design/modules/regularization.md b/doc/fluid/design/modules/regularization.md index 8cd5ff71d193f03e1ac923724b52f28c6057d25d..519a9143033386678351ff78a465e5ba6e220c52 100644 --- a/doc/fluid/design/modules/regularization.md +++ b/doc/fluid/design/modules/regularization.md @@ -32,9 +32,9 @@ In the new design, we propose to create new operations for regularization. For n - L2_regularization_op - L1_regularization_op -These ops can be like any other ops with their own CPU/GPU implementations either using Eigen or separate CPU and GPU kernels. As the initial implementation, we can implement their kernels using Eigen following the abstraction pattern implemented for [Activation Ops](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/accuracy_op.h). This abstraction pattern can make it very easy to implement new regularization schemes other than L1 and L2 norm penalties. +These ops can be like any other ops with their own CPU/GPU implementations either using Eigen or separate CPU and GPU kernels. As the initial implementation, we can implement their kernels using Eigen following the abstraction pattern implemented for [Activation Ops](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/accuracy_op.h). This abstraction pattern can make it very easy to implement new regularization schemes other than L1 and L2 norm penalties. -The idea of building ops for regularization is in sync with the refactored Paddle philosophy of using operators to represent any computation unit. The way these ops will be added to the computation graph, will be decided by the [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) in Python API. +The idea of building ops for regularization is in sync with the refactored Paddle philosophy of using operators to represent any computation unit. The way these ops will be added to the computation graph, will be decided by the [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/python_api.md#layer-function) in Python API. ### Computation Graph @@ -48,7 +48,7 @@ The Python API will modify this computation graph to add regularization operator     ### Python API implementation for Regularization -Using the low level ops, `L2_regularization_op` and `L1_regularization_op`, any user can add regularization to their computation graphs. However, this will require a lot of lines of code and we should design Python APIs that support regularization. An example of such an API can be seen in [Keras](https://keras.io/regularizers/). As per the PaddlePaddle [Python API design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md), the layer functions are responsible for creating operators, operator parameters and variables. Since regularization is a property of parameters, it makes sense to create these in the layer functions. +Using the low level ops, `L2_regularization_op` and `L1_regularization_op`, any user can add regularization to their computation graphs. However, this will require a lot of lines of code and we should design Python APIs that support regularization. An example of such an API can be seen in [Keras](https://keras.io/regularizers/). As per the PaddlePaddle [Python API design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/python_api.md), the layer functions are responsible for creating operators, operator parameters and variables. Since regularization is a property of parameters, it makes sense to create these in the layer functions. #### Creation of Regularization ops There are two possibilities for creating the regularization ops: @@ -63,4 +63,4 @@ Since we want to create the regularization ops in a lazy manner, the regularizat #### High-level API -In PaddlePaddle Python API, users will primarily rely on [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) to create neural network layers. Hence, we also need to provide regularization functionality in layer functions. The design of these APIs can be postponed for later right now. A good reference for these APIs can be found in [Keras](https://keras.io/regularizers/) and also by looking at Tensorflow in [`tf.contrib.layers`](https://www.tensorflow.org/api_guides/python/contrib.layers). +In PaddlePaddle Python API, users will primarily rely on [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/modules/python_api.md#layer-function) to create neural network layers. Hence, we also need to provide regularization functionality in layer functions. The design of these APIs can be postponed for later right now. A good reference for these APIs can be found in [Keras](https://keras.io/regularizers/) and also by looking at Tensorflow in [`tf.contrib.layers`](https://www.tensorflow.org/api_guides/python/contrib.layers). diff --git a/doc/fluid/design/motivation/fluid_compiler.md b/doc/fluid/design/motivation/fluid_compiler.md index 2a6beafc52e815fa067b273bb5887ddcf6ab15ae..6dd3840a0734e8593890dcf8044746197350c6f5 100644 --- a/doc/fluid/design/motivation/fluid_compiler.md +++ b/doc/fluid/design/motivation/fluid_compiler.md @@ -23,7 +23,7 @@ func paddlepaddle() { } ``` -This program consists of a [block](block.md) of three operators -- +This program consists of a [block](../concepts/block.md) of three operators -- `read`, `assign`, and `mult`. Its `ProgramDesc` message looks like the following @@ -39,7 +39,7 @@ message ProgramDesc { } } ``` - + ## Transpilers We can write a transpiler program that takes a `ProgramDesc`, e.g., @@ -93,7 +93,7 @@ specific hardware platform, for example, the `mult` operator, the generated code should call its CUDA kernel: ```c++ -paddle::Tensor fluid_cuda_mult(const paddle::Tensor& a, +paddle::Tensor fluid_cuda_mult(const paddle::Tensor& a, const paddle::Tensor& b) { paddle::Tensor t; paddle::operator::Mult m(a, b, ...); @@ -107,4 +107,4 @@ where `cuda_context` could be a global variable of type ## Multi-Block Code Generation Most Fluid application programs may have more than one blocks. To -execute them, we need to trace [scopes](scope.md). +execute them, we need to trace [scopes](../concepts/scope.md). diff --git a/doc/fluid/design/motivation/refactorization.md b/doc/fluid/design/motivation/refactorization.md index f199cc892f5e84f0a12abe3b8e5cace9849e7fa8..4e1d660cef6369f04db8e1e83360f6af25259f96 100644 --- a/doc/fluid/design/motivation/refactorization.md +++ b/doc/fluid/design/motivation/refactorization.md @@ -11,7 +11,7 @@ The goals of refactoring include: 1. PaddlePaddle represents the computation, training and inference of Deep Learning models, by computation graphs. - 1. Please refer to [computation graphs](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md) for a concrete example. + 1. Please refer to [computation graphs](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/others/graph.md) for a concrete example. 1. Users write Python programs to describe the graphs and run them (locally or remotely). @@ -28,7 +28,7 @@ The goals of refactoring include: 1. the C++ library `libpaddle.so` for local execution, 1. the master process of a distributed training job for training, or 1. the server process of a Kubernetes serving job for distributed serving. - 1. *Execution* executes the graph by constructing instances of class [`Variable`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24) and [`OperatorBase`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70), according to the protobuf message. + 1. *Execution* executes the graph by constructing instances of class [`Variable`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/variable.h#L24) and [`OperatorBase`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/operator.h#L70), according to the protobuf message. ## Description and Realization of Computation Graph @@ -48,16 +48,16 @@ At runtime, the C++ program realizes the graph and runs it. Data -VarDesc +VarDesc -Variable +Variable Operation -OpDesc +OpDesc -Operator +Operator Block @@ -85,7 +85,7 @@ The word *graph* is interchangeable with *block* in this document. A graph cons 1. The invocation of `train` or [`infer`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108) methods in the Python program does the following: - 1. Create a new Scope instance in the [scope hierarchy](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/scope.md) for each run of a block, + 1. Create a new Scope instance in the [scope hierarchy](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/concepts/scope.md) for each run of a block, 1. realize local variables defined in the BlockDesc message in the new scope, 1. a scope is similar to the stack frame in programming languages, @@ -195,7 +195,7 @@ Maintaining a map, whose key is the type name and the value is the corresponding ## Related Concepts ### Op_Maker -It's constructor takes `proto` and `checker`. They are completed during Op_Maker's construction. ([ScaleOpMaker](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/scale_op.cc#L37)) +It's constructor takes `proto` and `checker`. They are completed during Op_Maker's construction. ([ScaleOpMaker](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/scale_op.cc#L37)) ### Register Macros ```cpp @@ -236,7 +236,7 @@ REGISTER_OP_WITHOUT_GRADIENT(op_type, op_class, op_maker_class) * `Tensor` is an n-dimension array with type. * Only dims and data pointers are stored in `Tensor`. * All operations on `Tensor` are written in `Operator` or global functions. - * Variable length Tensor design [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md) + * Variable length Tensor design [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/concepts/lod_tensor.md) * `Variable` instances are the inputs and the outputs of an operator, not just `Tensor`. * `step_scopes` in RNN is a variable and not a tensor. * `Scope` is where variables are stored. diff --git a/doc/fluid/design/muti_devices/index_cn.rst b/doc/fluid/design/multi_devices/index_cn.rst similarity index 100% rename from doc/fluid/design/muti_devices/index_cn.rst rename to doc/fluid/design/multi_devices/index_cn.rst diff --git a/doc/fluid/design/muti_devices/index_en.rst b/doc/fluid/design/multi_devices/index_en.rst similarity index 100% rename from doc/fluid/design/muti_devices/index_en.rst rename to doc/fluid/design/multi_devices/index_en.rst diff --git a/doc/fluid/design/muti_devices/kernel_hint_design.md b/doc/fluid/design/multi_devices/kernel_hint_design.md similarity index 80% rename from doc/fluid/design/muti_devices/kernel_hint_design.md rename to doc/fluid/design/multi_devices/kernel_hint_design.md index 58e44b64169d8c942174de86986403570b271641..6edc14ca73b1abf824981b59511a9aca4e0f3b47 100644 --- a/doc/fluid/design/muti_devices/kernel_hint_design.md +++ b/doc/fluid/design/multi_devices/kernel_hint_design.md @@ -1,7 +1,7 @@ # Kernel Hint Design ## Problem -In PaddlePaddle's [Design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/switch_kernel.md), one Operator may have multiple kernels. Users may have some personal preference to choose a certain type of kernel for an operator, such as `force_cpu` to choose a CPU kernel, `use_cudnn` to choose a CUDNN kernel, we need to provide a way for users to do this. +In PaddlePaddle's [Design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/execution/switch.md), one Operator may have multiple kernels. Users may have some personal preference to choose a certain type of kernel for an operator, such as `force_cpu` to choose a CPU kernel, `use_cudnn` to choose a CUDNN kernel, we need to provide a way for users to do this. In the current design, we use KernelType to describe one kernel. @@ -14,7 +14,7 @@ struct KernelType { ``` `place_` `data_type_` and `layout_` can be got from the input tensors of the operator, `GetActualKernelType(inputs)` use inputs to infer the proper kernel key that fit the incoming data, but users can not directly configure it. -The [design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/switch_kernel.md) also provides a virtual method `GetExpectedKernelType` that user can overload and use to choose the KernelType they want to use. +The [design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/execution/switch.md) also provides a virtual method `GetExpectedKernelType` that user can overload and use to choose the KernelType they want to use. So we should send the information user defined in proto to `GetExpectedKernelType` for choosing a kernel. diff --git a/doc/fluid/design/muti_devices/kernel_selection.md b/doc/fluid/design/multi_devices/kernel_selection.md similarity index 100% rename from doc/fluid/design/muti_devices/kernel_selection.md rename to doc/fluid/design/multi_devices/kernel_selection.md diff --git a/doc/fluid/design/muti_devices/operator_kernel_type.md b/doc/fluid/design/multi_devices/operator_kernel_type.md similarity index 97% rename from doc/fluid/design/muti_devices/operator_kernel_type.md rename to doc/fluid/design/multi_devices/operator_kernel_type.md index f86e6b7a564ed23f2bddbec25da1c110014f941d..8c1bc8f76a337006497e5ab5e5a710f9f49261b8 100644 --- a/doc/fluid/design/muti_devices/operator_kernel_type.md +++ b/doc/fluid/design/multi_devices/operator_kernel_type.md @@ -8,7 +8,7 @@ struct OpKernelType { proto::DataType data_type_; }; ``` -For more details, please refer to [codes](https://github.com/PaddlePaddle/Paddle/blob/2d5ec16bc8a09fb8e0f62c89b116b0cd1d333907/paddle/framework/operator.h#L348-L374) in github. +For more details, please refer to [codes](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/operator.h#L348-L374) in github. It contains two keys, `Place` and `DataType`. And these two keys will be hashed to a unique key to represent a certain type of kernel. However, these two keys do not provide enough information. We need a more complete representation of `OpKernelType`. diff --git a/doc/fluid/design/network/sequence_decoder.md b/doc/fluid/design/network/sequence_decoder.md index f13d30ca9fe09c9525c711436f605bb280e11000..b95773c50ca0dcbd1b93529332e035d4de90faa8 100644 --- a/doc/fluid/design/network/sequence_decoder.md +++ b/doc/fluid/design/network/sequence_decoder.md @@ -11,7 +11,7 @@ In the old version of PaddlePaddle, the C++ class `RecurrentGradientMachine` imp There are a lot of heuristic tricks in the sequence generation tasks, so the flexibility of sequence decoder is very important to users. -During the refactoring of PaddlePaddle, some new concepts are proposed such as: [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md) and [TensorArray](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md) that can better support the sequence usage, and they can also help make the implementation of beam search based sequence decoder **more transparent and modular** . +During the refactoring of PaddlePaddle, some new concepts are proposed such as: [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/concepts/lod_tensor.md) and [TensorArray](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/concepts/tensor_array.md) that can better support the sequence usage, and they can also help make the implementation of beam search based sequence decoder **more transparent and modular** . For example, the RNN states, candidates IDs and probabilities of beam search can be represented all as `LoDTensors`; the selected candidate's IDs in each time step can be stored in a `TensorArray`, and `Packed` to the sentences translated. diff --git a/doc/fluid/dev/name_convention.md b/doc/fluid/dev/name_convention.md index 75830ef28c67dc4694d899efe503084b7b5852e1..6b4244d0f506c8cd6c08739141eabad27c581ca7 100644 --- a/doc/fluid/dev/name_convention.md +++ b/doc/fluid/dev/name_convention.md @@ -4,7 +4,7 @@ To make the operator document itself more clear, we recommend operator names obe ## OpProtoMaker names -When defining an operator in Paddle, a corresponding [OpProtoMaker](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L170) (TODO: OpProtoMaker Doc)need to be defined. All the Input/Output and Attributes will write into the [OpProto](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto#L61) , and will be used in client language to create operator. +When defining an operator in Paddle, a corresponding [OpProtoMaker](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/operator.h#L170) (TODO: OpProtoMaker Doc)need to be defined. All the Input/Output and Attributes will write into the [OpProto](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/framework.proto#L61) , and will be used in client language to create operator. - Input/Output. - Input/Output names follow the **CamelCase**. e.g. `X`, `Y`, `Matrix`, `LastAxisInMatrix`. Input/Output much more like Variables, we prefer to meaningful English words. diff --git a/doc/fluid/dev/new_op_cn.md b/doc/fluid/dev/new_op_cn.md index 0fa3252d220a49c88a5ea01a15073e74d6489c99..587d819f79fcf82549826359fbf04ad3af404446 100644 --- a/doc/fluid/dev/new_op_cn.md +++ b/doc/fluid/dev/new_op_cn.md @@ -147,7 +147,7 @@ class MulOp : public framework::OperatorWithKernel { }; ``` -[`MulOp`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/mul_op.cc#L22)继承自`OperatorWithKernel`。`public`成员： +[`MulOp`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/mul_op.cc#L22)继承自`OperatorWithKernel`。`public`成员： ```cpp using framework::OperatorWithKernel::OperatorWithKernel; @@ -173,7 +173,7 @@ MulOp(const std::string &type, const framework::VariableNameMap &inputs, `MulKernel`继承自`framework::OpKernel`，带有下面两个模板参数: -- `typename DeviceContext`: 表示设备类型，不同设备(CPU、CUDA)共享同一个Kernel时，需加该模板参数，不共享则不加，一个不共享的例子是[`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/cross_entropy_op.h#L43)。 +- `typename DeviceContext`: 表示设备类型，不同设备(CPU、CUDA)共享同一个Kernel时，需加该模板参数，不共享则不加，一个不共享的例子是[`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/cross_entropy_op.h#L43)。 - `typename T` : 表示数据类型，如`float`, `double`等。 @@ -201,9 +201,9 @@ MulOp(const std::string &type, const framework::VariableNameMap &inputs, 需要注意：**不同设备(CPU、CUDA)共享一个Op定义，是否则共享同一个`OpKernel`，取决于`Compute`调用的函数是否支持不同设备。** -`MulOp`的CPU、CUDA实现共享同一个`Kernel`。`OpKernel`不共享的例子可以参考：[`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/cross_entropy_op.h#L43)。 +`MulOp`的CPU、CUDA实现共享同一个`Kernel`。`OpKernel`不共享的例子可以参考：[`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/cross_entropy_op.h#L43)。 -为了使`OpKernel`的计算过程书写更加简单，并且CPU、CUDA的代码可以复用，我们通常借助 Eigen unsupported Tensor模块来实现`Compute`接口。关于在PaddlePaddle中如何使用Eigen库，请参考[使用文档](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/dev/use_eigen_cn.md)。 +为了使`OpKernel`的计算过程书写更加简单，并且CPU、CUDA的代码可以复用，我们通常借助 Eigen unsupported Tensor模块来实现`Compute`接口。关于在PaddlePaddle中如何使用Eigen库，请参考[使用文档](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/dev/use_eigen_cn.md)。 到此，前向Op实现完成。接下来，需要在`.cc`文件中注册该op和kernel。 反向Op类的定义，反向OpKernel的定义与前向Op类似，这里不再赘述。**但需注意反向Op没有`ProtoMaker`**。 diff --git a/doc/fluid/dev/new_op_en.md b/doc/fluid/dev/new_op_en.md index c8fb7e3b3bfe200aeed43f2c580b8409a2e9008a..f8de271ed4e5e0fb4018478bffd4b525d4319738 100644 --- a/doc/fluid/dev/new_op_en.md +++ b/doc/fluid/dev/new_op_en.md @@ -26,13 +26,6 @@ Here are the base types needed. For details, please refer to the design docs. Operators can be categorized into two groups: operator with kernel(s) and operator without kernel(s). An operator with kernel(s) inherits from `OperatorWithKernel` while the one without kernel(s) inherits from `OperatorBase`. This tutorial focuses on implementing operators with kernels. In short, an operator includes the following information: - Information | Where is it defined --------------- | :---------------------- -OpProtoMake definition | `.cc`files, Backward Op does not need an OpProtoMake interface. -Op definition | `.cc` files -Kernel implementation | The kernel methods shared between CPU and CUDA are defined in `.h` files. CPU-specific kernels live in `.cc` files, while CUDA-specific kernels are implemented in `.cu`files. -Registering the Op | Ops are registered in `.cc` files; For Kernel registration, `.cc` files contain the CPU implementation, while `.cu` files contain the CUDA implementation. - @@ -176,7 +169,7 @@ Usually `OpProtoMaker` and `Op`'s type definitions are written in `.cc` files, w `MulKernel` inherits `framework::OpKernel`, which includes the following templates: -- `typename DeviceContext` denotes device context type. When different devices, namely the CPUDeviceContext and the CUDADeviceContext, share the same kernel, this template needs to be added. If they don't share kernels, this must not be added. An example of a non-sharing kernel is [`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/cross_entropy_op.h#L43). +- `typename DeviceContext` denotes device context type. When different devices, namely the CPUDeviceContext and the CUDADeviceContext, share the same kernel, this template needs to be added. If they don't share kernels, this must not be added. An example of a non-sharing kernel is [`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/cross_entropy_op.h#L43). - `typename T` denotes data type, such as `float` or `double`. @@ -207,7 +200,7 @@ Note that **different devices (CPU, CUDA)share one Op definition; whether or not `MulOp`'s CPU and CUDA share the same `Kernel`. A non-sharing `OpKernel` example can be seen in [`OnehotCrossEntropyOpKernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/cross_entropy_op.cc). -To ease the writing of `OpKernel` compute, and for reusing code cross-device, [`Eigen-unsupported Tensor`](https://bitbucket.org/eigen/eigen/src/default/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default) module is used to implement `Compute` interface. To learn about how the Eigen library is used in PaddlePaddle, please see [usage document](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/dev/use_eigen_cn.md). +To ease the writing of `OpKernel` compute, and for reusing code cross-device, [`Eigen-unsupported Tensor`](https://bitbucket.org/eigen/eigen/src/default/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default) module is used to implement `Compute` interface. To learn about how the Eigen library is used in PaddlePaddle, please see [usage document](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/dev/use_eigen_en.md). This concludes the forward implementation of an operator. Next its operation and kernel need to be registered in a `.cc` file. diff --git a/doc/fluid/dev/new_op_kernel.md b/doc/fluid/dev/new_op_kernel.md index 55dea8d0a39232ede59d4663d6e1a47fbfc60853..87e617d44041bde9c9051151878ffb4304689b3c 100644 --- a/doc/fluid/dev/new_op_kernel.md +++ b/doc/fluid/dev/new_op_kernel.md @@ -4,13 +4,13 @@ PaddlePaddle Fluid have hundreds of operators. Each operator could have one or more kernels. A kernel is an implementation of the operator for a certain device, which could be a hardware device, e.g., the CUDA GPU, or a library that utilizes a device, e.g., Intel MKL that makes full use of the Xeon CPU. -[This document](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/dev/new_op_en.md) explains how to add an operator, and its kernels. The kernels of an operator are indexed by a C++ type [`OpKernelType`](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/operator_kernel_type.md). An operator chooses the right kernel at runtime. This choosing mechanism is described [here](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/switch_kernel.md). +[This document](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/dev/new_op_en.md) explains how to add an operator, and its kernels. The kernels of an operator are indexed by a C++ type [`OpKernelType`](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/multi_devices/operator_kernel_type.md). An operator chooses the right kernel at runtime. This choosing mechanism is described [here](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/execution/switch.md). ## Write Kernels for A New Device ### Add A New Device - For some historical reaons, we misuse the word *library* for *device*. For example, we call the deivce type by *library type*. An example is the header file [`library_type.h`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/library_type.h#L24). We will correct this ASAP. + For some historical reaons, we misuse the word *library* for *device*. For example, we call the deivce type by *library type*. An example is the header file [`library_type.h`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/library_type.h#L24). We will correct this ASAP. To register a new device, we need to add an enum value to `LibraryType`: @@ -23,9 +23,9 @@ enum class LibraryType { ``` -### Add A New [Place](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/platform/place.h#L53) +### Add A New [Place](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/platform/place.h#L53) -If you have a new kind of Device, firstly you need to add a new kind of [`Place`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/platform/place.h#L53). For example `CUDAPlace`: +If you have a new kind of Device, firstly you need to add a new kind of [`Place`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/platform/place.h#L53). For example `CUDAPlace`: ```cpp struct CUDAPlace { @@ -45,8 +45,8 @@ struct CUDAPlace { typedef boost::variant Place; ``` -### Add [device context]((https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/platform/device_context.h#L37)) -After a new kind of Device is added, you should add a corresponding [DeviceContext](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/platform/device_context.h#L37) for it. +### Add [device context]((https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/platform/device_context.h#L37)) +After a new kind of Device is added, you should add a corresponding [DeviceContext](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/platform/device_context.h#L37) for it. ```cpp class DeviceContext { @@ -58,9 +58,9 @@ class DeviceContext { }; ``` -### Implement new [OpKernel](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L351) for your Device. +### Implement new [OpKernel](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/operator.h#L351) for your Device. -A detailed documentation can be found in [`new_op_and_kernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/dev/new_op_en.md) +A detailed documentation can be found in [`new_op_and_kernel`](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/dev/new_op_en.md) ```cpp class OpKernelBase { @@ -101,7 +101,7 @@ REGISTER_OP_KERNEL( kernel0, kernel1 are kernels that have the same `op_type`, `library_type`, `place_type` but different `data_types`. -take [`conv2d`]((https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/conv_cudnn_op.cu.cc#L318)) as an example: +take [`conv2d`]((https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/conv_cudnn_op.cu.cc#L318)) as an example: ```cpp REGISTER_OP_KERNEL(conv2d, CPU, paddle::platform::CPUPlace, diff --git a/doc/fluid/dev/support_new_device.md b/doc/fluid/dev/support_new_device.md index 8983df900460127fc130043c52373dab505363ba..051a463cfcf97df2e2d5b6a880923ca70fefbd6e 100644 --- a/doc/fluid/dev/support_new_device.md +++ b/doc/fluid/dev/support_new_device.md @@ -13,7 +13,7 @@ So, how to support a new Device/Library in Fluid becomes a challenge. ## Basic: Integrate A New Device/Library -For a general overview of fluid, please refer to the [overview doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/read_source.md). +For a general overview of fluid, please refer to the [overview doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/read_source.md). There are mainly three parts that we have to consider while integrating a new device/library: @@ -28,7 +28,7 @@ There are mainly three parts that we have to consider while integrating a new de Please note that device and computing library are not one-to-one corresponding. A device can have a lot of computing libraries and a computing library can also support several devices. #### Place -Fluid uses class [Place](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/platform/place.h#L55) to represent the device memory where data is located. If we add another device, we have to add the corresponding `DevicePlace`. +Fluid uses class [Place](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/platform/place.h#L55) to represent the device memory where data is located. If we add another device, we have to add the corresponding `DevicePlace`. ``` | CPUPlace @@ -44,7 +44,7 @@ typedef boost::variant Place; #### DeviceContext -Fluid uses class [DeviceContext](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/platform/device_context.h#L30) to manage the resources in different libraries, such as CUDA stream in `CDUADeviceContext`. There are also inheritance relationships between different kinds of `DeviceContext`. +Fluid uses class [DeviceContext](https://github.com/PaddlePaddle/Paddle/blob/develop/fluid/paddle/platform/device_context.h#L30) to manage the resources in different libraries, such as CUDA stream in `CDUADeviceContext`. There are also inheritance relationships between different kinds of `DeviceContext`. ``` @@ -73,7 +73,7 @@ class CUDADeviceContext : public DeviceContext { Place GetPlace() const override { return place_; } private: CUDAPlace place_; - cudaStream_t stream_; + cudaStream_t stream_; cublasHandle_t cublas_handle_; std::unique_ptr eigen_device_; // binds with stream_ }; @@ -84,7 +84,7 @@ private: #### memory module -Fluid provides the following [memory interfaces](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/memory/memory.h#L36): +Fluid provides the following [memory interfaces](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/memory/memory.h#L36): ``` template @@ -102,7 +102,7 @@ To implement these interfaces, we have to implement MemoryAllocator for differen #### Tensor -[Tensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/tensor.h#L36) holds data with some shape in a specific Place. +[Tensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/framework/tensor.h#L36) holds data with some shape in a specific Place. ```cpp class Tensor { @@ -161,7 +161,7 @@ t.mutable_data(place); Fluid implements computing units based on different DeviceContexts. Some computing units are shared between operators. This common part will be put in operators/math directory as basic Functors. -Let's take [MaxOutFunctor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/math/maxouting.h#L27) as an example: +Let's take [MaxOutFunctor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/math/maxouting.h#L27) as an example: The interface is defined in the header file. @@ -210,7 +210,7 @@ The implementation of `OpKernel` is similar to math functors, the extra thing we Fluid provides different register interfaces in op_registry.h -Let's take [Crop](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/crop_op.cc#L134) operator as an example: +Let's take [Crop](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/crop_op.cc#L134) operator as an example: In .cc file: @@ -236,5 +236,5 @@ Generally, we will implement OpKernel for all Device/Library of an Operator. We For more details, please refer to following docs: -- operator kernel type [doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/operator_kernel_type.md) -- switch kernel [doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/switch_kernel.md) +- operator kernel type [doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/multi_devices/operator_kernel_type.md) +- switch kernel [doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/fluid/design/execution/switch.md) diff --git a/paddle/fluid/framework/tensor_util_test.cc b/paddle/fluid/framework/tensor_util_test.cc index 9687a86ca25be7886e67028a38e54b3065c8e4b5..6e10885890cd2d4a0d77834944b37e291197b637 100644 --- a/paddle/fluid/framework/tensor_util_test.cc +++ b/paddle/fluid/framework/tensor_util_test.cc @@ -105,16 +105,14 @@ TEST(TensorCopy, Tensor) { } TEST(TensorFromVector, Tensor) { - using namespace paddle::framework; - using namespace paddle::platform; { std::vector src_vec = {1, 2, 3, 4, 5, 6, 7, 8, 9}; - Tensor cpu_tensor; + paddle::framework::Tensor cpu_tensor; // Copy to CPU Tensor - cpu_tensor.Resize(make_ddim({3, 3})); + cpu_tensor.Resize(paddle::framework::make_ddim({3, 3})); auto cpu_place = new paddle::platform::CPUPlace(); - TensorFromVector(src_vec, &cpu_tensor); + paddle::framework::TensorFromVector(src_vec, &cpu_tensor); // Compare Tensors const int* cpu_ptr = cpu_tensor.data(); @@ -125,8 +123,8 @@ TEST(TensorFromVector, Tensor) { } src_vec.erase(src_vec.begin(), src_vec.begin() + 5); - cpu_tensor.Resize(make_ddim({2, 2})); - TensorFromVector(src_vec, &cpu_tensor); + cpu_tensor.Resize(paddle::framework::make_ddim({2, 2})); + paddle::framework::TensorFromVector(src_vec, &cpu_tensor); cpu_ptr = cpu_tensor.data(); src_ptr = src_vec.data(); ASSERT_NE(src_ptr, cpu_ptr); @@ -140,23 +138,23 @@ TEST(TensorFromVector, Tensor) { #ifdef PADDLE_WITH_CUDA { std::vector src_vec = {1, 2, 3, 4, 5, 6, 7, 8, 9}; - Tensor cpu_tensor; - Tensor gpu_tensor; - Tensor dst_tensor; + paddle::framework::Tensor cpu_tensor; + paddle::framework::Tensor gpu_tensor; + paddle::framework::Tensor dst_tensor; // Copy to CPU Tensor cpu_tensor.Resize(make_ddim({3, 3})); auto cpu_place = new paddle::platform::CPUPlace(); - CPUDeviceContext cpu_ctx(*cpu_place); - TensorFromVector(src_vec, cpu_ctx, &cpu_tensor); + paddle::platform::CPUDeviceContext cpu_ctx(*cpu_place); + paddle::framework::TensorFromVector(src_vec, cpu_ctx, &cpu_tensor); // Copy to GPUTensor - gpu_tensor.Resize(make_ddim({3, 3})); + gpu_tensor.Resize(paddle::framework::make_ddim({3, 3})); auto gpu_place = new paddle::platform::CUDAPlace(); - CUDADeviceContext gpu_ctx(*gpu_place); - TensorFromVector(src_vec, gpu_ctx, &gpu_tensor); + paddle::platform::CUDADeviceContext gpu_ctx(*gpu_place); + paddle::framework::TensorFromVector(src_vec, gpu_ctx, &gpu_tensor); // Copy from GPU to CPU tensor for comparison - TensorCopy(gpu_tensor, *cpu_place, gpu_ctx, &dst_tensor); + paddle::framework::TensorCopy(gpu_tensor, *cpu_place, gpu_ctx, &dst_tensor); // Sync before Compare Tensors gpu_ctx.Wait(); @@ -172,11 +170,11 @@ TEST(TensorFromVector, Tensor) { src_vec.erase(src_vec.begin(), src_vec.begin() + 5); - cpu_tensor.Resize(make_ddim({2, 2})); - TensorFromVector(src_vec, cpu_ctx, &cpu_tensor); - gpu_tensor.Resize(make_ddim({2, 2})); - TensorFromVector(src_vec, gpu_ctx, &gpu_tensor); - TensorCopy(gpu_tensor, *cpu_place, gpu_ctx, &dst_tensor); + cpu_tensor.Resize(paddle::framework::make_ddim({2, 2})); + paddle::framework::TensorFromVector(src_vec, cpu_ctx, &cpu_tensor); + gpu_tensor.Resize(paddle::framework::make_ddim({2, 2})); + paddle::framework::TensorFromVector(src_vec, gpu_ctx, &gpu_tensor); + paddle::framework::TensorCopy(gpu_tensor, *cpu_place, gpu_ctx, &dst_tensor); // Sync before Compare Tensors gpu_ctx.Wait(); @@ -197,18 +195,16 @@ TEST(TensorFromVector, Tensor) { } TEST(TensorToVector, Tensor) { - using namespace paddle::framework; - using namespace paddle::platform; { - Tensor src; - int* src_ptr = src.mutable_data({3, 3}, CPUPlace()); + paddle::framework::Tensor src; + int* src_ptr = src.mutable_data({3, 3}, paddle::platform::CPUPlace()); for (int i = 0; i < 3 * 3; ++i) { src_ptr[i] = i; } - CPUPlace place; + paddle::platform::CPUPlace place; std::vector dst; - TensorToVector(src, &dst); + paddle::framework::TensorToVector(src, &dst); for (int i = 0; i < 3 * 3; ++i) { EXPECT_EQ(src_ptr[i], dst[i]); @@ -217,13 +213,13 @@ TEST(TensorToVector, Tensor) { #ifdef PADDLE_WITH_CUDA { std::vector src_vec = {1, 2, 3, 4, 5, 6, 7, 8, 9}; - Tensor gpu_tensor; - CUDAPlace place; - CUDADeviceContext gpu_ctx(place); - TensorFromVector(src_vec, gpu_ctx, &gpu_tensor); + paddle::framework::Tensor gpu_tensor; + paddle::platform::CUDAPlace place; + paddle::platform::CUDADeviceContext gpu_ctx(place); + paddle::framework::TensorFromVector(src_vec, gpu_ctx, &gpu_tensor); std::vector dst; - TensorToVector(gpu_tensor, gpu_ctx, &dst); + paddle::framework::TensorToVector(gpu_tensor, gpu_ctx, &dst); for (int i = 0; i < 3 * 3; ++i) { EXPECT_EQ(src_vec[i], dst[i]); @@ -233,54 +229,54 @@ TEST(TensorToVector, Tensor) { } TEST(TensorContainsNAN, CPU) { - using namespace paddle::framework; - using namespace paddle::platform; { - Tensor src; - float* buf = src.mutable_data({3}, CPUPlace()); + paddle::framework::Tensor src; + float* buf = src.mutable_data({3}, paddle::platform::CPUPlace()); buf[0] = 0.0; buf[1] = NAN; buf[2] = 0.0; - ASSERT_TRUE(TensorContainsNAN(src)); + ASSERT_TRUE(paddle::framework::TensorContainsNAN(src)); buf[1] = 0.0; - ASSERT_FALSE(TensorContainsNAN(src)); + ASSERT_FALSE(paddle::framework::TensorContainsNAN(src)); } { - Tensor src; - float16* buf = src.mutable_data({3}, CPUPlace()); + paddle::framework::Tensor src; + paddle::platform::float16* buf = + src.mutable_data( + {3}, paddle::platform::CPUPlace()); buf[0] = 0.0; buf[1].x = 0x7fff; buf[2] = 0.0; - ASSERT_TRUE(TensorContainsNAN(src)); + ASSERT_TRUE(paddle::framework::TensorContainsNAN(src)); buf[1] = 0.0; - ASSERT_FALSE(TensorContainsNAN(src)); + ASSERT_FALSE(paddle::framework::TensorContainsNAN(src)); } } TEST(TensorContainsInf, CPU) { - using namespace paddle::framework; - using namespace paddle::platform; { - Tensor src; - double* buf = src.mutable_data({3}, CPUPlace()); + paddle::framework::Tensor src; + double* buf = src.mutable_data({3}, paddle::platform::CPUPlace()); buf[0] = 1.0; buf[1] = INFINITY; buf[2] = 0.0; - ASSERT_TRUE(TensorContainsInf(src)); + ASSERT_TRUE(paddle::framework::TensorContainsInf(src)); buf[1] = 1.0; - ASSERT_FALSE(TensorContainsInf(src)); + ASSERT_FALSE(paddle::framework::TensorContainsInf(src)); } { - Tensor src; - float16* buf = src.mutable_data({3}, CPUPlace()); + paddle::framework::Tensor src; + paddle::platform::float16* buf = + src.mutable_data( + {3}, paddle::platform::CPUPlace()); buf[0] = 1.0; buf[1].x = 0x7c00; buf[2] = 0.0; - ASSERT_TRUE(TensorContainsInf(src)); + ASSERT_TRUE(paddle::framework::TensorContainsInf(src)); buf[1] = 1.0; - ASSERT_FALSE(TensorContainsInf(src)); + ASSERT_FALSE(paddle::framework::TensorContainsInf(src)); } } diff --git a/paddle/fluid/framework/tensor_util_test.cu b/paddle/fluid/framework/tensor_util_test.cu index 4766ec28aa3cff6be3259f258f1c9543ae471f5d..b4cff1e6c2293fa44f0fd0bb398a538c08dd4fb1 100644 --- a/paddle/fluid/framework/tensor_util_test.cu +++ b/paddle/fluid/framework/tensor_util_test.cu @@ -45,9 +45,8 @@ static __global__ void FillInf(platform::float16* buf) { } TEST(TensorContainsNAN, GPU) { - using namespace paddle::platform; - CUDAPlace gpu(0); - auto& pool = DeviceContextPool::Instance(); + paddle::platform::CUDAPlace gpu(0); + auto& pool = paddle::platform::DeviceContextPool::Instance(); auto* cuda_ctx = pool.GetByPlace(gpu); { Tensor tensor; @@ -58,7 +57,8 @@ TEST(TensorContainsNAN, GPU) { } { Tensor tensor; - float16* buf = tensor.mutable_data({3}, gpu); + paddle::platform::float16* buf = + tensor.mutable_data({3}, gpu); FillNAN<<<1, 1, 0, cuda_ctx->stream()>>>(buf); cuda_ctx->Wait(); ASSERT_TRUE(TensorContainsNAN(tensor)); @@ -66,9 +66,8 @@ TEST(TensorContainsNAN, GPU) { } TEST(TensorContainsInf, GPU) { - using namespace paddle::platform; - CUDAPlace gpu(0); - auto& pool = DeviceContextPool::Instance(); + paddle::platform::CUDAPlace gpu(0); + auto& pool = paddle::platform::DeviceContextPool::Instance(); auto* cuda_ctx = pool.GetByPlace(gpu); { Tensor tensor; @@ -79,7 +78,8 @@ TEST(TensorContainsInf, GPU) { } { Tensor tensor; - float16* buf = tensor.mutable_data({3}, gpu); + paddle::platform::float16* buf = + tensor.mutable_data({3}, gpu); FillInf<<<1, 1, 0, cuda_ctx->stream()>>>(buf); cuda_ctx->Wait(); ASSERT_TRUE(TensorContainsInf(tensor)); diff --git a/paddle/fluid/operators/activation_op.cc b/paddle/fluid/operators/activation_op.cc index 549629ffd664825c40b8cd89811f31c6ab390fd3..87ef55c50b0be46492a695928625d140345d415d 100644 --- a/paddle/fluid/operators/activation_op.cc +++ b/paddle/fluid/operators/activation_op.cc @@ -74,105 +74,105 @@ class ActivationOpGrad : public framework::OperatorWithKernel { } }; -constexpr char SigmoidDoc[] = R"DOC( +__attribute__((unused)) constexpr char SigmoidDoc[] = R"DOC( Sigmoid Activation Operator $$out = \frac{1}{1 + e^{-x}}$$ )DOC"; -constexpr char LogSigmoidDoc[] = R"DOC( +__attribute__((unused)) constexpr char LogSigmoidDoc[] = R"DOC( Logsigmoid Activation Operator $$out = \log \frac{1}{1 + e^{-x}}$$ )DOC"; -constexpr char ExpDoc[] = R"DOC( +__attribute__((unused)) constexpr char ExpDoc[] = R"DOC( Exp Activation Operator. $out = e^x$ )DOC"; -constexpr char ReluDoc[] = R"DOC( +__attribute__((unused)) constexpr char ReluDoc[] = R"DOC( Relu Activation Operator. $out = \max(x, 0)$ )DOC"; -constexpr char TanhDoc[] = R"DOC( +__attribute__((unused)) constexpr char TanhDoc[] = R"DOC( Tanh Activation Operator. $$out = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$ )DOC"; -constexpr char TanhShrinkDoc[] = R"DOC( +__attribute__((unused)) constexpr char TanhShrinkDoc[] = R"DOC( TanhShrink Activation Operator. $$out = x - \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$ )DOC"; -constexpr char SqrtDoc[] = R"DOC( +__attribute__((unused)) constexpr char SqrtDoc[] = R"DOC( Sqrt Activation Operator. $out = \sqrt{x}$ )DOC"; -constexpr char AbsDoc[] = R"DOC( +__attribute__((unused)) constexpr char AbsDoc[] = R"DOC( Abs Activation Operator. $out = |x|$ )DOC"; -constexpr char CeilDoc[] = R"DOC( +__attribute__((unused)) constexpr char CeilDoc[] = R"DOC( Ceil Activation Operator. $out = ceil(x)$ )DOC"; -constexpr char FloorDoc[] = R"DOC( +__attribute__((unused)) constexpr char FloorDoc[] = R"DOC( Floor Activation Operator. $out = floor(x)$ )DOC"; -constexpr char CosDoc[] = R"DOC( +__attribute__((unused)) constexpr char CosDoc[] = R"DOC( Cosine Activation Operator. $out = cos(x)$ )DOC"; -constexpr char SinDoc[] = R"DOC( +__attribute__((unused)) constexpr char SinDoc[] = R"DOC( Sine Activation Operator. $out = sin(x)$ )DOC"; -constexpr char RoundDoc[] = R"DOC( +__attribute__((unused)) constexpr char RoundDoc[] = R"DOC( Round Activation Operator. $out = [x]$ )DOC"; -constexpr char ReciprocalDoc[] = R"DOC( +__attribute__((unused)) constexpr char ReciprocalDoc[] = R"DOC( Reciprocal Activation Operator. $$out = \frac{1}{x}$$ )DOC"; -constexpr char LogDoc[] = R"DOC( +__attribute__((unused)) constexpr char LogDoc[] = R"DOC( Log Activation Operator. $out = \ln(x)$ @@ -181,21 +181,21 @@ Natural logarithm of x. )DOC"; -constexpr char SquareDoc[] = R"DOC( +__attribute__((unused)) constexpr char SquareDoc[] = R"DOC( Square Activation Operator. $out = x^2$ )DOC"; -constexpr char SoftplusDoc[] = R"DOC( +__attribute__((unused)) constexpr char SoftplusDoc[] = R"DOC( Softplus Activation Operator. $out = \ln(1 + e^{x})$ )DOC"; -constexpr char SoftsignDoc[] = R"DOC( +__attribute__((unused)) constexpr char SoftsignDoc[] = R"DOC( Softsign Activation Operator. $$out = \frac{x}{1 + |x|}$$