Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
黄金旺铺
Paddle
提交
74d1bf4a
P
Paddle
项目概览
黄金旺铺
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
74d1bf4a
编写于
6月 20, 2018
作者:
C
chengduoZH
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add doc of data reader
上级
49f23e63
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
98 addition
and
0 deletion
+98
-0
python/paddle/fluid/data_feeder.py
python/paddle/fluid/data_feeder.py
+98
-0
未找到文件。
python/paddle/fluid/data_feeder.py
浏览文件 @
74d1bf4a
...
...
@@ -70,6 +70,62 @@ class DataToLoDTensorConverter(object):
class
DataFeeder
(
object
):
"""
DataFeeder converts the data that returned by paddle.reader into a
data structure of Arguments which is defined in the API. The paddle.reader
usually returns a list of mini-batch data entries. Each data entry in
the list is one sample. Each sample is a list or a tuple with one feature
or multiple features. DataFeeder converts this mini-batch data entries
into Arguments in order to feed it to C++ interface.
The simple usage shows below:
.. code-block:: python
place = fluid.CPUPlace()
data = fluid.layers.data(
name='data', shape=[1], dtype='int64', lod_level=2)
label = fluid.layers.data(name='label', shape=[1], dtype='int64')
feeder = fluid.DataFeeder([data, label], place)
result = feeder.feed(
[([[1, 2, 3], [4, 5]], [1]), ([[6, 7, 8, 9]], [1])])
If you want to feed data into GPU side separately in advance when you
use multi-GPU to train a model, you can use `decorate_reader` function.
.. code-block:: python
place=fluid.CUDAPlace(0)
feeder = fluid.DataFeeder(place=place, feed_list=[data, label])
reader = feeder.decorate_reader(
paddle.batch(flowers.train(), batch_size=16))
Args:
feed_list(list): The Variables or Variables'name that will
feed into model.
place(Place): fluid.CPUPlace() or fluid.CUDAPlace(i).
program(Program): The Program that will feed data into, if program
is None, it will use default_main_program(). Default None.
Raises:
ValueError: If the some Variable is not in the Program.
Examples:
.. code-block:: python
# ...
place = fluid.CPUPlace()
feed_list = [
main_program.global_block().var(var_name) for var_name in feed_vars_name
]
feeder = fluid.DataFeeder(feed_list, place)
for data in reader():
outs = exe.run(program=main_program,
feed=feeder.feed(data))
"""
def
__init__
(
self
,
feed_list
,
place
,
program
=
None
):
self
.
feed_dtypes
=
[]
self
.
feed_names
=
[]
...
...
@@ -99,6 +155,16 @@ class DataFeeder(object):
self
.
place
=
place
def
feed
(
self
,
iterable
):
"""
According to feed_list and iterable converter the input data
into a dictionary that can feed into Executor or ParallelExecutor.
Args:
iterable(list|tuple): the input data.
Returns:
dict: the result of conversion.
"""
converter
=
[]
for
lod_level
,
shape
,
dtype
in
six
.
zip
(
self
.
feed_lod_level
,
self
.
feed_shapes
,
self
.
feed_dtypes
):
...
...
@@ -121,6 +187,20 @@ class DataFeeder(object):
return
ret_dict
def
feed_parallel
(
self
,
iterable
,
num_places
=
None
):
"""
Takes multiple mini-batches. Each mini-batch will be feed on each
device.
Args:
iterable(list|tuple): the input data.
num_places(int): the number of places. Default None.
Returns:
dict: the result of conversion.
Notes:
The number of devices and number of mini-batches must be same.
"""
if
isinstance
(
self
.
place
,
core
.
CUDAPlace
):
places
=
[
core
.
CUDAPlace
(
i
)
...
...
@@ -159,6 +239,24 @@ class DataFeeder(object):
multi_devices
,
num_places
=
None
,
drop_last
=
True
):
"""
Converter the input data into a data that returned by reader into
multiple mini-batches. Each mini-batch will be feed on each device.
Args:
reader(fun): the input data.
multi_devices(bool): the number of places. Default None.
num_places(int): the number of places. Default None.
drop_last(bool): the number of places. Default None.
Returns:
dict: the result of conversion.
Raises:
ValueError: If drop_last is False and the data batch which cannot
fit for devices.
"""
def
__reader_creator__
():
if
not
multi_devices
:
for
item
in
reader
():
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录