Paddle reads data from *data reader* during training. *data reader creator* (or *reader creator*) creates a *data reader* when invoked. *reader creator* will be passed into `paddle.train` as a parameter.
Paddle reads data from *data reader* during training. *data reader* will be passed into `paddle.train` as a parameter.
## Data Reader Creator Interface
## Data Reader Interface
Data reader creator is a function with no parameter that creates a iterable (anything can be used in `for x in iterable`):
Data reader is a function with no parameter that creates a iterable (anything can be used in `for x in iterable`):
```
```
iterable = data_reader_creator()
iterable = data_reader()
```
```
Element produced for the iterable should be a **single** entry of data, **not** a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of [supported type](http://www.paddlepaddle.org/doc/ui/data_provider/pydataprovider2.html?highlight=dense_vector#input-types)(e.g., numpy 1d array of float32, int, list of int)
Element produced for the iterable should be a **single** entry of data, **not** a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of [supported type](http://www.paddlepaddle.org/doc/ui/data_provider/pydataprovider2.html?highlight=dense_vector#input-types)(e.g., numpy 1d array of float32, int, list of int)
An example implementation for single item data reader creator:
An example implementation for single item data reader:
```python
```python
defdata_reader_creator_fake_image():
defdata_reader_fake_image():
whileTrue:
whileTrue:
yieldnumpy.random.uniform(-1,1,size=20*20)
yieldnumpy.random.uniform(-1,1,size=20*20)
```
```
An example implementation for multiple item data reader creator:
An example implementation for multiple item data reader:
```python
```python
defdata_reader_creator_fake_image_and_label():
defdata_reader_fake_image_and_label():
whileTrue:
whileTrue:
yieldnumpy.random.uniform(-1,1,size=20*20),False
yieldnumpy.random.uniform(-1,1,size=20*20),False
```
```
## Usage
## Usage
data reader creator, mapping from item(s) read to data layer, batch size and number of total pass will be passed into `paddle.train`:
data reader, mapping from item(s) read to data layer, batch size and number of total pass will be passed into `paddle.train`:
*Data reader creator decorator* (or *reader creator decorator*) takes a single or multiple data reader creator, returns a new data reader creator. It is similar to a [python decorator](https://wiki.python.org/moin/PythonDecorators), but it does not use `@` syntax.
*Data reader decorator* takes a single or multiple data reader, returns a new data reader. It is similar to a [python decorator](https://wiki.python.org/moin/PythonDecorators), but it does not use `@` syntax.
Since we have a strict interface for data readers creators (no parameter, return a single data item). Data reader creators can be used flexiable via data reader creator decorators. Following are a few examples:
Since we have a strict interface for data readers (no parameter, return a single data item). Data reader can be used flexiable via data reader decorators. Following are a few examples:
### Prefetch Data
### Prefetch Data
...
@@ -54,63 +54,63 @@ Since reading data may take time and training can not proceed without data. It i
...
@@ -54,63 +54,63 @@ Since reading data may take time and training can not proceed without data. It i
`buffered_reader_creator` will try to buffer (prefetch) `100` data entries.
`buffered_reader` will try to buffer (prefetch) `100` data entries.
### Compose Multiple Data Reader Creators
### Compose Multiple Data Readers
For example, we want to use a source of real images (reusing mnist dataset), and a source of fake images as input for [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661).
For example, we want to use a source of real images (reusing mnist dataset), and a source of fake images as input for [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661).
Given shuffle buffer size `n`, `paddle.reader.shuffle` will return a data reader creator that buffers `n` data entries and shuffle them before a data entry is read.
Given shuffle buffer size `n`, `paddle.reader.shuffle` will return a data reader that buffers `n` data entries and shuffle them before a data entry is read.
### Why return only a single entry, but not a mini batch?
### Why return only a single entry, but not a mini batch?
If a mini batch is returned, data reader creator need to take care of batch size. But batch size is a concept for training, it makes more sense for user to specify batch size as a parameter for `train`.
If a mini batch is returned, data reader need to take care of batch size. But batch size is a concept for training, it makes more sense for user to specify batch size as a parameter for `train`.
Practically, always return a single entry make reusing existing data reader creators much easier (e.g., if existing reader creator return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2).
Practically, always return a single entry make reusing existing data readers much easier (e.g., if existing reader return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2).
### Why use a dictionary but not a list to provide mapping?
### Why use a dictionary but not a list to provide mapping?
We decided to use dictionary (`{"image":0, "label":1}`) instead of list (`["image", "label"]`) is because that user can easily resue item (e.g., using `{"image_a":0, "image_b":0, "label":1}`) or skip item (e.g., using `{"image_a":0, "label":2}`).
We decided to use dictionary (`{"image":0, "label":1}`) instead of list (`["image", "label"]`) is because that user can easily resue item (e.g., using `{"image_a":0, "image_b":0, "label":1}`) or skip item (e.g., using `{"image_a":0, "label":2}`).