未验证 提交 f39df8d7 编写于 作者: Z zhongpu 提交者: GitHub

update feeding_data.rst doc (#1813)

* update feeding_data.rst, test=develop

* poish description, test=develop

* fix data shape, test=develop
上级 230b6915
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
同步数据读取 同步数据读取
############## ##############
PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层; PaddlePaddle Fluid支持使用 :code:`fluid.data()` 配置数据层;
再使用 Numpy Array 或者直接使用Python创建C++的 再使用 Numpy Array 或者直接使用Python创建C++的
:code:`fluid.LoDTensor` , 通过 :code:`Executor.run(feed=...)` 传给 :code:`fluid.LoDTensor` , 通过 :code:`Executor.run(feed=...)` 传给
:code:`fluid.Executor` 或 :code:`fluid.ParallelExecutor` 。 :code:`fluid.Executor` 或 :code:`fluid.ParallelExecutor` 。
...@@ -12,29 +12,25 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层; ...@@ -12,29 +12,25 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层;
数据层配置 数据层配置
########## ##########
通过 :code:`fluid.layers.data()` 可以配置神经网络中需要的数据层。具体方法为: 通过 :code:`fluid.data()` 可以配置神经网络中需要的数据层。具体方法为:
.. code-block:: python .. code-block:: python
import paddle.fluid as fluid import paddle.fluid as fluid
image = fluid.layers.data(name="image", shape=[3, 224, 224]) image = fluid.data(name="image", shape=[None, 3, 224, 224])
label = fluid.layers.data(name="label", shape=[1], dtype="int64") label = fluid.data(name="label", shape=[None, 1], dtype="int64")
# use image/label as layer input # use image/label as layer input
prediction = fluid.layers.fc(input=image, size=1000, act="softmax") prediction = fluid.layers.fc(input=image, size=1000, act="softmax")
loss = fluid.layers.cross_entropy(input=prediction, label=label) loss = fluid.layers.cross_entropy(input=prediction, label=label)
... ...
上段代码中,:code:`image` 和 :code:`label` 是通过 :code:`fluid.layers.data` 上段代码中,:code:`image` 和 :code:`label` 是通过 :code:`fluid.data`
创建的两个输入数据层。其中 :code:`image` 是 :code:`[3, 224, 224]` 维度的浮点数据; 创建的两个输入数据层。其中 :code:`image` 是 :code:`[None, 3, 224, 224]` 维度的浮点数据;
:code:`label` 是 :code:`[1]` 维度的整数数据。这里需要注意的是: :code:`label` 是 :code:`[None, 1]` 维度的整数数据。这里需要注意的是:
1. Fluid中默认使用 :code:`-1` 表示 batch size 维度,默认情况下会在 :code:`shape` 1. Executor在执行的时候,会检查定义的数据层数据和feed的数据的 :code:`shape` 和 :code:`dtype` 是否一致,如果不一致,程序会报错退出。对于一些任务,在不同的轮数,数据的某些维度会变化,可以将维度的值设置为None,例如第0维会变化,可以将 :code:`shape` 设置为 :code:`[None, 3, 224, 224]` 。
的第一个维度添加 :code:`-1` 。 所以 上段代码中, 我们可以接受将一个
:code:`[32, 3, 224, 224]` 的numpy array传给 :code:`image` 。 如果想自定义batch size
维度的位置的话,请设置 :code:`fluid.layers.data(append_batch_size=False)` 。
请参考进阶使用中的 :ref:`user_guide_customize_batch_size_rank` 。
2. Fluid中用来做类别标签的数据类型是 :code:`int64`,并且标签从0开始。可用数据类型请参考 :ref:`user_guide_paddle_support_data_types`。 2. Fluid中用来做类别标签的数据类型是 :code:`int64`,并且标签从0开始。可用数据类型请参考 :ref:`user_guide_paddle_support_data_types`。
...@@ -69,17 +65,17 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层; ...@@ -69,17 +65,17 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层;
序列数据是PaddlePaddle Fluid支持的特殊数据类型,可以使用 :code:`LoDTensor` 作为 序列数据是PaddlePaddle Fluid支持的特殊数据类型,可以使用 :code:`LoDTensor` 作为
输入数据类型。它需要用户: 1. 传入一个mini-batch需要被训练的所有数据; 输入数据类型。它需要用户: 1. 传入一个mini-batch需要被训练的所有数据;
2.每个序列的长度信息。 2.每个序列的长度信息。
用户可以使用 :code:`fluid.create_lod_tensor` 来创建 :code:`LoDTensor`。 用户可以使用 :code:`fluid.create_lod_tensor` 来创建 :code:`LoDTensor`
传入序列信息的时候,需要设置序列嵌套深度,:code:`lod_level`。 传入序列信息的时候,需要设置序列嵌套深度,:code:`lod_level`
例如训练数据是词汇组成的句子,:code:`lod_level=1`;训练数据是 词汇先组成了句子, 例如训练数据是词汇组成的句子,:code:`lod_level=1` ;训练数据是 词汇先组成了句子,
句子再组成了段落,那么 :code:`lod_level=2`。 句子再组成了段落,那么 :code:`lod_level=2`
例如: 例如:
.. code-block:: python .. code-block:: python
sentence = fluid.layers.data(name="sentence", dtype="int64", shape=[1], lod_level=1) sentence = fluid.data(name="sentence", dtype="int64", shape=[None, 1], lod_level=1)
... ...
...@@ -91,8 +87,8 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层; ...@@ -91,8 +87,8 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层;
) )
}) })
训练数据 :code:`sentence` 包含三个样本,他们的长度分别是 :code:`4, 1, 2`。 训练数据 :code:`sentence` 包含三个样本,他们的长度分别是 :code:`4, 1, 2`
他们分别是 :code:`data[0:4]`, :code:`data[4:5]` 和 :code:`data[5:7]`。 他们分别是 :code:`data[0:4]`, :code:`data[4:5]` 和 :code:`data[5:7]`
如何分别设置ParallelExecutor中每个设备的训练数据 如何分别设置ParallelExecutor中每个设备的训练数据
------------------------------------------------ ------------------------------------------------
...@@ -123,36 +119,6 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层; ...@@ -123,36 +119,6 @@ PaddlePaddle Fluid支持使用 :code:`fluid.layers.data()` 配置数据层;
上述代码中,GPU0会训练 32 个样本,而 GPU1训练 16 个样本。 上述代码中,GPU0会训练 32 个样本,而 GPU1训练 16 个样本。
.. _user_guide_customize_batch_size_rank:
自定义BatchSize维度
-------------------
PaddlePaddle Fluid默认batch size是数据的第一维度,以 :code:`-1` 表示。但是在高级
使用中,batch_size 可以固定,也可以是其他维度或者多个维度来表示。这都需要设置
:code:`fluid.layers.data(append_batch_size=False)` 来完成。
1. 固定batch size维度
.. code-block:: python
image = fluid.layers.data(name="image", shape=[32, 784], append_batch_size=False)
这里,:code:`image` 永远是一个 :code:`[32, 784]` 大小的矩阵。
2. 使用其他维度表示batch size
.. code-block:: python
sentence = fluid.layers.data(name="sentence",
shape=[80, -1, 1],
append_batch_size=False,
dtype="int64")
这里 :code:`sentence` 的中间维度是batch size。这种数据排布会用在定长的循环神经
网络中。
.. _user_guide_paddle_support_data_types: .. _user_guide_paddle_support_data_types:
Fluid目前支持的数据类型 Fluid目前支持的数据类型
......
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
Take Numpy Array as Training Data Take Numpy Array as Training Data
################################# #################################
PaddlePaddle Fluid supports configuring data layer with :code:`fluid.layers.data()` . PaddlePaddle Fluid supports configuring data layer with :code:`fluid.data()` .
Then you can use Numpy Array or directly use Python to create C++ Then you can use Numpy Array or directly use Python to create C++
:code:`fluid.LoDTensor` , and then feed it to :code:`fluid.Executor` or :code:`fluid.ParallelExecutor` :code:`fluid.LoDTensor` , and then feed it to :code:`fluid.Executor` or :code:`fluid.ParallelExecutor`
through :code:`Executor.run(feed=...)` . through :code:`Executor.run(feed=...)` .
...@@ -12,23 +12,23 @@ through :code:`Executor.run(feed=...)` . ...@@ -12,23 +12,23 @@ through :code:`Executor.run(feed=...)` .
Configure Data Layer Configure Data Layer
############################ ############################
With :code:`fluid.layers.data()` , you can configure data layer in neural network. Details are as follows: With :code:`fluid.data()` , you can configure data layer in neural network. Details are as follows:
.. code-block:: python .. code-block:: python
import paddle.fluid as fluid import paddle.fluid as fluid
image = fluid.layers.data(name="image", shape=[3, 224, 224]) image = fluid.data(name="image", shape=[None, 3, 224, 224])
label = fluid.layers.data(name="label", shape=[1], dtype="int64") label = fluid.data(name="label", shape=[None, 1], dtype="int64")
# use image/label as layer input # use image/label as layer input
prediction = fluid.layers.fc(input=image, size=1000, act="softmax") prediction = fluid.layers.fc(input=image, size=1000, act="softmax")
loss = fluid.layers.cross_entropy(input=prediction, label=label) loss = fluid.layers.cross_entropy(input=prediction, label=label)
... ...
In the code above, :code:`image` and :code:`label` are two input data layers created by :code:`fluid.layers.data` . :code:`image` is float data of shape :code:`[3, 224, 224]` ; :code:`label` is the int data of shape :code:`[1]` . Note that: In the code above, :code:`image` and :code:`label` are two input data layers created by :code:`fluid.data` . :code:`image` is float data of shape :code:`[None, 3, 224, 224]` ; :code:`label` is the int data of shape :code:`[None, 1]` . Note that:
1. :code:`-1` is represented for the dimension of batch size by default in Fluid. And :code:`-1` is added to the first dimension of :code:`shape` by default. Therefore in the code above, it would be alright to transfer numpy array of :code:`[32, 3, 224, 224]` to :code:`image` . If you want to customize the position of the batch size dimension, please set :code:`fluid.layers.data(append_batch_size=False)` .Please refer to the tutorial in the advanced user guide: :ref:`user_guide_customize_batch_size_rank_en` . 1. When the program is executing, executor will check whether the :code:`shape` and :code:`dtype` defined and feeded are consistent. If they are not consistent, the program will exit with an error. In some tasks, the dimension will change in different training steps. For this case, the value of the dimension can be set to None. For example, the :code:`shape` can be set to :code:`[None, 3, 224, 224]` when the 0th dimension will change.
2. Data type of category labels in Fluid is :code:`int64` and the label starts from 0. About the supported data types,please refer to :ref:`user_guide_paddle_support_data_types_en` . 2. Data type of category labels in Fluid is :code:`int64` and the label starts from 0. About the supported data types,please refer to :ref:`user_guide_paddle_support_data_types_en` .
...@@ -76,7 +76,7 @@ For example: ...@@ -76,7 +76,7 @@ For example:
.. code-block:: python .. code-block:: python
sentence = fluid.layers.data(name="sentence", dtype="int64", shape=[1], lod_level=1) sentence = fluid.data(name="sentence", dtype="int64", shape=[None, 1], lod_level=1)
... ...
...@@ -122,32 +122,6 @@ For example: ...@@ -122,32 +122,6 @@ For example:
In the code above, GPU0 will train 32 samples and GPU1 will train 16 samples. In the code above, GPU0 will train 32 samples and GPU1 will train 16 samples.
.. _user_guide_customize_batch_size_rank_en:
Customize the BatchSize dimension
------------------------------------
Batch size is the first dimension of data by default in PaddlePaddle Fluid, indicated by :code:`-1` .But in advanced usage, batch_size could be fixed or respresented by other dimension or multiple dimensions, which could be implemented by setting :code:`fluid.layers.data(append_batch_size=False)` .
1. fixed BatchSize dimension
.. code-block:: python
image = fluid.layers.data(name="image", shape=[32, 784], append_batch_size=False)
Here :code:`image` is always a matrix with size of :code:`[32, 784]` .
2. batch size expressed by other dimension
.. code-block:: python
sentence = fluid.layers.data(name="sentence",
shape=[80, -1, 1],
append_batch_size=False,
dtype="int64")
Here the middle dimension of :code:`sentence` is batch size. This type of data layout is applied in fixed-length recurrent neural networks.
.. _user_guide_paddle_support_data_types_en: .. _user_guide_paddle_support_data_types_en:
Data types supported by Fluid Data types supported by Fluid
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册