For further information, please refer to: [multi_devices](https://github.com/PaddlePaddle/FluidDoc/tree/develop/doc/fluid/design/multi_devices) , [scope](https://github.com/PaddlePaddle/FluidDoc/Blob/develop/doc/fluid/design/concepts/scope.md) , [Developer's_Guide_to_Paddle_Fluid](https://github.com/PaddlePaddle/FluidDoc/blob/release/1.2/doc/fluid/getstarted/Developer's_Guide_to_Paddle_Fluid.md)
For further information, please refer to: [multi_devices](https://github.com/PaddlePaddle/FluidDoc/tree/develop/doc/fluid/design/multi_devices) , [scope](https://github.com/PaddlePaddle/FluidDoc/blob/develop/doc/fluid/design/concepts/scope.md) , [Developer's_Guide_to_Paddle_Fluid](https://github.com/PaddlePaddle/FluidDoc/blob/release/1.2/doc/fluid/getstarted/Developer's_Guide_to_Paddle_Fluid.md)
### 2.Op's registration logic
### 2.Op's registration logic
The registration entries for each Operator include:
The registration entries for each Operator include:
从iterable生成的元素应该是单个数据条目，而不是mini batch。数据输入可以是单个项目，也可以是项目的元组，但应为 `支持的类型 <http://www.paddlepaddle.org/doc/ui/data_provider/pydataprovider2.html?highlight=dense_vector#input-types>`_ （如, numpy 1d array of float32, int, list of int）
从iterable生成的元素应该是单个数据条目，而不是mini batch。数据输入可以是单个项目，也可以是项目的元组，但应为 `支持的类型 <../../user_guides/howto/prepare_data/feeding_data.html#fluid>`_ （如, numpy 1d array of float32, int, list of int）
Currently PaddlePaddle Fluid has 2 branches of API interfaces:
- Low-level API:
- It is highly flexible and relatively mature. The model trained by it can directly support C++ inference deployment and release.
- There are a large number of models as examples, including all chapters in [book](https://github.com/PaddlePaddle/book), and [models](https://github.com/PaddlePaddle/models).
- Recommended for users who have a certain understanding of deep learning and need to customize a network for training/inference/online deployment.
- High-level API:
- Simple to use
- Still under development. the interface is temporarily in [paddle.fluid.contrib](https://github.com/PaddlePaddle/Paddle/tree/develop/python/paddle/fluid/contrib).
This section introduces the Fluid API structure and usage, to help you quickly get the full picture of the PaddlePaddle Fluid API. This section is divided into the following modules:
This section introduces the Fluid API structure and usage, to help you quickly get the full picture of the PaddlePaddle Fluid API. This section is divided into the following modules:
Fluid supports parallelism asynchronous distributed training. :code:`DistributedTranspiler` converts a single node network configuration into a :code:`pserver` side program and the :code:`trainer` side program that can be executed on multiple machines. The user executes the same piece of code on different nodes. Depending on the environment variables or startup parameters, the corresponding :code:`pserver` or :code:`trainer` role can be executed.
Fluid supports parallelism asynchronous distributed training. :code:`DistributeTranspiler` converts a single node network configuration into a :code:`pserver` side program and the :code:`trainer` side program that can be executed on multiple machines. The user executes the same piece of code on different nodes. Depending on the environment variables or startup parameters, the corresponding :code:`pserver` or :code:`trainer` role can be executed.
**Asynchronous distributed training in Fluid only supports the pserver mode** . The main difference between asynchronous training and `synchronous training <../distributed/sync_training_en.html>`_ is that the gradients of each trainer are asynchronously applied on the parameters, but in synchronous training, the gradients of all trainers must be combined first and then they are used to update the parameters. Therefore, the hyperparameters of synchronous training and asynchronous training need to be adjusted separately.
**Asynchronous distributed training in Fluid only supports the pserver mode** . The main difference between asynchronous training and `synchronous training <../distributed/sync_training_en.html>`_ is that the gradients of each trainer are asynchronously applied on the parameters, but in synchronous training, the gradients of all trainers must be combined first and then they are used to update the parameters. Therefore, the hyperparameters of synchronous training and asynchronous training need to be adjusted separately.
Fluid supports parallelism distributed synchronous training, the API uses the :code:`DistributedTranspiler` to convert a single node network configuration into a :code:`pserver` side and :code:`trainer` side program that can be executed on multiple machines. The user executes the same piece of code on different nodes. Depending on the environment variables or startup parameters, you can execute the corresponding :code:`pserver` or :code:`trainer` role. Fluid distributed synchronous training supports both pserver mode and NCCL2 mode. There are differences in the use of the API, to which you need to pay attention.
Fluid supports parallelism distributed synchronous training, the API uses the :code:`DistributeTranspiler` to convert a single node network configuration into a :code:`pserver` side and :code:`trainer` side program that can be executed on multiple machines. The user executes the same piece of code on different nodes. Depending on the environment variables or startup parameters, you can execute the corresponding :code:`pserver` or :code:`trainer` role. Fluid distributed synchronous training supports both pserver mode and NCCL2 mode. There are differences in the use of the API, to which you need to pay attention.
Distributed training in pserver mode
Distributed training in pserver mode
======================================
======================================
For API Reference, please refer to :ref:`DistributeTranspiler`. A simple example :
For API Reference, please refer to :ref:`DistributeTranspiler`. A simple example :
.. code-block:: python
.. code-block:: python
config = fluid.DistributedTranspilerConfig()
config = fluid.DistributeTranspilerConfig()
#Configuring policy config
#Configuring policy config
config.slice_var_up = False
config.slice_var_up = False
t = fluid.DistributedTranspiler(config=config)
t = fluid.DistributeTranspiler(config=config)
t.transpile(trainer_id,
t.transpile(trainer_id,
program=main_program,
program=main_program,
pservers="192.168.0.1:6174,192.168.0.2:6174",
pservers="192.168.0.1:6174,192.168.0.2:6174",
...
@@ -51,7 +51,7 @@ Configuration for general environment variables:
...
@@ -51,7 +51,7 @@ Configuration for general environment variables:
- :code:`FLAGS_rpc_deadline` : int, the longest waiting time for RPC communication, in milliseconds, default 180000
- :code:`FLAGS_rpc_deadline` : int, the longest waiting time for RPC communication, in milliseconds, default 180000
Distributed training in NCCL2 mode
Distributed training in NCCL2 mode
====================================
====================================
The multi-node synchronous training mode based on NCCL2 (Collective Communication) is only supported in the GPU cluster.
The multi-node synchronous training mode based on NCCL2 (Collective Communication) is only supported in the GPU cluster.
...
@@ -65,7 +65,7 @@ Use the following code to convert the current :code:`Program` to a Fluid :code:`
...
@@ -65,7 +65,7 @@ Use the following code to convert the current :code:`Program` to a Fluid :code:`
The logic of :code:`Executor` is very simple. It is suggested to thoroughly run the model with :code:`Executor` in debugging phase on one computer and then switch to mode of multiple devices or multiple computers to compute.
The logic of :code:`Executor` is very simple. It is suggested to thoroughly run the model with :code:`Executor` in debugging phase on one computer and then switch to mode of multiple devices or multiple computers to compute.
:code:`Executor` receives a :code:`Place` at construction, which can either be :ref:`api_fluid_CPUPlace` or :ref:`api_fluid_CUDAPlace`.
:code:`Executor` receives a :code:`Place` at construction, which can either be :ref:`api_fluid_CPUPlace` or :ref:`api_fluid_CUDAPlace`.
.. code-block:: python
.. code-block:: python
...
@@ -18,14 +18,14 @@ The logic of :code:`Executor` is very simple. It is suggested to thoroughly run
...
@@ -18,14 +18,14 @@ The logic of :code:`Executor` is very simple. It is suggested to thoroughly run
# Run the startup program once and only once.
# Run the startup program once and only once.
exe.run(fluid.default_startup_program())
exe.run(fluid.default_startup_program())
# Run the main program directly.
# Run the main program directly.
loss, = exe.run(fluid.default_main_program(),
loss, = exe.run(fluid.default_main_program(),
feed=feed_dict,
feed=feed_dict,
fetch_list=[loss.name])
fetch_list=[loss.name])
For simple example please refer to `quick_start_fit_a_line <http://paddlepaddle.org/documentation/docs/zh/1.1/beginners_guide/quick_start/fit_a_line/README.html>`_
For simple example please refer to `basics_fit_a_line <../../beginners_guide/basics/fit_a_line/README.html>`_
@@ -37,9 +37,9 @@ API reference :ref:`api_fluid_layers_embedding` . Here is a simple example:
...
@@ -37,9 +37,9 @@ API reference :ref:`api_fluid_layers_embedding` . Here is a simple example:
The parameters:
The parameters:
- :code:`is_sparse` : Whether the gradient is a sparse tensor in the backward calculation. If not set, the gradient is a `LodTensor <https://github.com/PaddlePaddle/FluidDoc/blob/develop/doc/fluid/user_guides/howto/prepare_data/lod_tensor.md>`_ . The default is False.
- :code:`is_sparse` : Whether the gradient is a sparse tensor in the backward calculation. If not set, the gradient is a `LodTensor <https://github.com/PaddlePaddle/FluidDoc/blob/develop/doc/fluid/user_guides/howto/basic_concept/lod_tensor_en.html>`_ . The default is False.
- :code:`is_distributed` : Whether the current training is in a distributed scenario. Generally, this parameter can only be set in large-scale sparse updates (the 0th dimension of embedding is very large, such as several million or more). For details, please refer to the large-scale sparse API guide :ref:`api_guide_async_training`. The default is False.
- :code:`is_distributed` : Whether the current training is in a distributed scenario. Generally, this parameter can only be set in large-scale sparse updates (the 0th dimension of embedding is very large, such as several million or more). For details, please refer to the large-scale sparse API guide :ref:`api_guide_async_training`. The default is False.
@@ -32,8 +32,8 @@ For API Reference, please refer to :ref:`api_fluid_nets_img_conv_group`
...
@@ -32,8 +32,8 @@ For API Reference, please refer to :ref:`api_fluid_nets_img_conv_group`
--------------------
--------------------
:code:`sequence_conv_pool` is got by concatenating :ref:`api_fluid_layers_sequence_conv` with :ref:`api_fluid_layers_sequence_pool`.
:code:`sequence_conv_pool` is got by concatenating :ref:`api_fluid_layers_sequence_conv` with :ref:`api_fluid_layers_sequence_pool`.
The module is widely used in the field of `natural language processing <https://en.wikipedia.org/wiki/Natural_language_processing>`_ and `speech recognition <https://en.wikipedia.org/wiki/Speech_recognition>`_ . Models such as the `text classification model <https://github.com/PaddlePaddle/models/blob/develop/fluid/PaddleNLP/text_classification/nets.py>`_ ,
The module is widely used in the field of `natural language processing <https://en.wikipedia.org/wiki/Natural_language_processing>`_ and `speech recognition <https://en.wikipedia.org/wiki/Speech_recognition>`_ . Models such as the `text classification model <https://github.com/PaddlePaddle/models/blob/develop/PaddleNLP/text_classification/nets.py>`_ ,
`TagSpace <https://github.com/PaddlePaddle/models/blob/develop/fluid/PaddleRec/tagspace/train.py>`_ and `Multi view Simnet <https://github.com/PaddlePaddle/models/blob/develop/fluid/PaddleRec/multiview_simnet/nets.py>`_.
`TagSpace <https://github.com/PaddlePaddle/models/blob/develop/PaddleRec/tagspace/train.py>`_ and `Multi view Simnet <https://github.com/PaddlePaddle/models/blob/develop/PaddleRec/multiview_simnet/nets.py>`_.
For API Reference, please refer to :ref:`api_fluid_nets_sequence_conv_pool`
For API Reference, please refer to :ref:`api_fluid_nets_sequence_conv_pool`
...
@@ -54,6 +54,6 @@ For the input data :code:`Queries` , :code:`Key` and :code:`Values`, calculate t
...
@@ -54,6 +54,6 @@ For the input data :code:`Queries` , :code:`Key` and :code:`Values`, calculate t
.. math::
.. math::
Attention(Q, K, V)= softmax(QK^\mathrm{T})V
Attention(Q, K, V)= softmax(QK^\mathrm{T})V
This module is widely used in the model of `machine translation <https://en.wikipedia.org/wiki/Machine_translation>`_, such as `Transformer <https://github.com/PaddlePaddle/models/tree/develop/Fluid/PaddleNLP/neural_machine_translation/transformer>`_ .
This module is widely used in the model of `machine translation <https://en.wikipedia.org/wiki/Machine_translation>`_, such as `Transformer <https://github.com/PaddlePaddle/models/tree/develop/PaddleNLP/neural_machine_translation/transformer>`_ .
For API Reference, please refer to :ref:`api_fluid_nets_scaled_dot_product_attention`
For API Reference, please refer to :ref:`api_fluid_nets_scaled_dot_product_attention`
As mentioned above, :code:`Block` in Fluid describes a set of Operators that include sequential execution, conditional selection or loop execution, and the operating object of Operator: Tensor.
As mentioned above, :code:`Block` in Fluid describes a set of Operators that include sequential execution, conditional selection or loop execution, and the operating object of Operator: Tensor.
...
@@ -53,7 +53,7 @@ This is because some common operations on Tensor may consist of more basic opera
...
@@ -53,7 +53,7 @@ This is because some common operations on Tensor may consist of more basic opera
More information can be read for reference. `Fluid Design Idea <../../advanced_usage/design_idea/fluid_design_idea.html>`_
More information can be read for reference. `Fluid Design Idea <../../advanced_usage/design_idea/fluid_design_idea.html>`_
=========
=========
...
@@ -75,4 +75,4 @@ Related API
...
@@ -75,4 +75,4 @@ Related API
* Users can also use :ref:`api_fluid_program_guard` with :code:`with` to modify the configured :ref:`api_fluid_default_startup_program` and :ref:`api_fluid_default_main_program` .
* Users can also use :ref:`api_fluid_program_guard` with :code:`with` to modify the configured :ref:`api_fluid_default_startup_program` and :ref:`api_fluid_default_main_program` .
* In Fluid，the execution order in a Block is determined by control flow，such as :ref:`api_fluid_layers_IfElse` , :ref:`api_fluid_layers_While` and :ref:`api_fluid_layers_Switch` . For more information, please refer to： :ref:`api_guide_control_flow_en`
* In Fluid，the execution order in a Block is determined by control flow，such as :ref:`api_fluid_layers_IfElse` , :ref:`api_fluid_layers_While` and :ref:`api_fluid_layers_Switch` . For more information, please refer to： :ref:`api_guide_control_flow_en`
This section collects six documents arranging from the simplest to the most challenging, which will guide you through the basic deep learning tasks in PaddlePaddle.
This section collects 8 documents arranging from the simplest to the most challenging, which will guide you through the basic deep learning tasks in PaddlePaddle.
The documentation in this chapter covers a lot of deep learning basics and how to implement them with PaddlePaddle. See the instructions below for how to use:
The documentation in this chapter covers a lot of deep learning basics and how to implement them with PaddlePaddle. See the instructions below for how to use:
...
@@ -15,6 +15,8 @@ The book you are reading is an "interactive" e-book - each chapter can be run in
...
@@ -15,6 +15,8 @@ The book you are reading is an "interactive" e-book - each chapter can be run in
.. toctree::
.. toctree::
:titlesonly:
:titlesonly:
fit_a_line/README.md
recognize_digits/README.md
image_classification/index_en.md
image_classification/index_en.md
word2vec/index_en.md
word2vec/index_en.md
recommender_system/index_en.md
recommender_system/index_en.md
...
@@ -45,7 +47,7 @@ Just run these in shell:
...
@@ -45,7 +47,7 @@ Just run these in shell:
docker run -d -p 8888:8888 paddlepaddle/book
docker run -d -p 8888:8888 paddlepaddle/book
It downloads the Docker image for running books from DockerHub.com.
It downloads the Docker image for running books from DockerHub.com.
To read and edit this book on-line, please visit http://localhost:8888 in your browser.
To read and edit this book on-line, please visit http://localhost:8888 in your browser.
If the Internet connection to DockerHub.com is compromised, try our spare docker image named docker.paddlepaddlehub.com:
If the Internet connection to DockerHub.com is compromised, try our spare docker image named docker.paddlepaddlehub.com:
@@ -15,8 +15,6 @@ If you have been armed with certain level of deep learning knowledge, and it hap
...
@@ -15,8 +15,6 @@ If you have been armed with certain level of deep learning knowledge, and it hap
- `Programming with Fluid <../beginners_guide/programming_guide/programming_guide_en.html>`_ ： Core concepts and basic usage of Fluid
- `Programming with Fluid <../beginners_guide/programming_guide/programming_guide_en.html>`_ ： Core concepts and basic usage of Fluid
- `Quick Start <../beginners_guide/quick_start/index_en.html>`_： Two easy-to-go models, linear regression model and digit recognition model, are in place to speed up your study of training neural networks
- `Deep Learning Basics <../beginners_guide/basics/index_en.html>`_： This section encompasses various fields of fundamental deep learning knowledge, such as image classification, customized recommendation, machine translation, and examples implemented by Fluid are provided.
- `Deep Learning Basics <../beginners_guide/basics/index_en.html>`_： This section encompasses various fields of fundamental deep learning knowledge, such as image classification, customized recommendation, machine translation, and examples implemented by Fluid are provided.
...
@@ -24,6 +22,5 @@ If you have been armed with certain level of deep learning knowledge, and it hap
...
@@ -24,6 +22,5 @@ If you have been armed with certain level of deep learning knowledge, and it hap
@@ -414,7 +414,7 @@ Firstly, define input data format, model structure,loss function and optimized a
...
@@ -414,7 +414,7 @@ Firstly, define input data format, model structure,loss function and optimized a
```
```
Now we discover that predicted value is nearly close to real value and the loss value descends from original value 9.05 to 0.01 after iteration for 100 times.
Now we discover that predicted value is nearly close to real value and the loss value descends from original value 9.05 to 0.01 after iteration for 100 times.
Congratulations! You have succeed to create a simple network. If you want to try advanced linear regression —— predict model of housing price, please read [linear regression](../../beginners_guide/quick_start/fit_a_line/README.en.html). More examples of model can be found in [models](../../user_guides/models/index_en.html).
Congratulations! You have succeed to create a simple network. If you want to try advanced linear regression —— predict model of housing price, please read [linear regression](../../beginners_guide/basics/fit_a_line/README.en.html). More examples of model can be found in [models](../../user_guides/models/index_en.html).
<aname="what_next"></a>
<aname="what_next"></a>
## What's next
## What's next
...
@@ -427,4 +427,4 @@ After the construction of network, you can start training your network in single
...
@@ -427,4 +427,4 @@ After the construction of network, you can start training your network in single
In addition, there are three learning levels in documentation according to developer's background and experience: [Beginner's Guide](../../beginners_guide/index_en.html) , [User Guides](../../user_guides/index_en.html) and [Advanced User Guides](../../advanced_usage/index_en.html).
In addition, there are three learning levels in documentation according to developer's background and experience: [Beginner's Guide](../../beginners_guide/index_en.html) , [User Guides](../../user_guides/index_en.html) and [Advanced User Guides](../../advanced_usage/index_en.html).
If you want to read examples in more application scenarios, you can go to [quick start](../../beginners_guide/quick_start/index_en.html) and [basic knowledge of deep learning](../../beginners_guide/basics/index_en.html) .If you have learned basic knowledge of deep learning, you can read from [user guide](../../user_guides/index_en.html).
If you want to read examples in more application scenarios, you can go to [basic knowledge of deep learning](../../beginners_guide/basics/index_en.html) .If you have learned basic knowledge of deep learning, you can read from [user guide](../../user_guides/index_en.html).
