diff --git a/doc/design/reader/README.md b/doc/design/reader/README.md index fa37ad7b2920032c42ac5b8c5d9ad59cdbae240a..7fa62350c40d531ec065592a50e8ed373c719ab4 100644 --- a/doc/design/reader/README.md +++ b/doc/design/reader/README.md @@ -1,28 +1,86 @@ -# Python Data Provider Design Doc +# Python Data Reader Design Doc -Data provider provides data for training. It will be passed into `paddle.train` as a parameter. +Paddle reads data from data reader during training. It will be passed into `paddle.train` as a parameter. -## Data Provider Interface +## Data Reader Interface -Data provider is a function with no parameter that creates a iterable (anything can be used in `for x in iterable`): +Data reader is a function with no parameter that creates a iterable (anything can be used in `for x in iterable`): ``` -iterable = data_provider() +iterable = data_reader() ``` -Element produced for the iterable should be a **single** entry of data, in format `[column_0_item, column_1_item, ...]`. Each element of the list needs to be supported data type (e.g., numpy 1d array of float32, list of int). +Element produced for the iterable should be a **single** entry of data, **not** a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of [supported type](http://www.paddlepaddle.org/doc/ui/data_provider/pydataprovider2.html?highlight=dense_vector#input-types) (e.g., numpy 1d array of float32, int, list of int) -For example, `column_0_item` could be image pixels of format numpy 1d array of float32, and `column_1_item` could be image label of format single int value: +An example implementation for single item data reader: +```python +def data_reader_fake_image(): + while True: + yield numpy.random.uniform(-1, 1, size=20*20) +``` + +An example implementation for multiple item data reader: +```python +def data_reader_fake_image_and_label(): + while True: + yield numpy.random.uniform(-1, 1, size=20*20), False ``` -for single_entry in iterable: - pixel = entry[0] - label = entry[1] + +## Data Reader Decorators + +Data Reader Decorators takes a single or multiple data reader, returns a new data reader. It is similar to a [python decorator](https://wiki.python.org/moin/PythonDecorators), but it does not use `@` syntax. + +Since we have a strict interface for data readers (no parameter, return a single data item). Data reader can be used flexiable via data reader decorators. Following a few examples: + +### Prefetch Data + +Since reading data may take time and training can not proceed without data. It is generally a good idea to prefetch data. + +Use `paddle.reader.buffered` to prefetch data: + +```python +buffered_reader = paddle.reader.buffered(paddle.dataset.mnist, 100) +``` + +`buffered_reader` will try to buffer (prefetch) `100` data entries. + +### Compose Multiple Data Readers + +For example, we want to use a source of real images (reusing mnist dataset), and a source of fake images as input for [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661). + +We can do: + +```python +def data_reader_fake_image(): + while True: + yield numpy.random.uniform(-1, 1, size=20*20) + +def data_reader_bool(t): + while True: + yield t + +true_reader = lambda : data_reader_bool(True) +false_reader = lambda : data_reader_bool(False) + +reader = paddle.reader.combine(paddle.dataset.mnist, data_reader_fake_image, true_reader, false_reader) +# skipped 1 because paddle.dataset.mnist produces two items per data entry. +# We don't care second item at this time. +paddle.train(reader, {"true_image":0, "fake_image": 2, "true_label": 3, "false_label": 4}, ...) +``` + +### Shuffle + +Given shuffle buffer size `n`, `paddle.reader.shuffle` will return a data reader decorater that buffers `n` data entries and shuffle them before a data entry is read. + +Example: +```python +reader = paddle.reader.shuffle(paddle.dataset.mnist, 512) ``` ## Usage -data provider, mapping from data provider column to data layer, batch size and number of total pass will be passed into `paddle.train`: +data reader, mapping from item(s) read to data layer, batch size and number of total pass will be passed into `paddle.train`: ```python # two data layer is created: @@ -31,32 +89,38 @@ label_layer = paddle.layer.data("label", ...) # ... -paddle.train(paddle.data.mnist, ["image", "label"], 128, 10, ...) +paddle.train(paddle.dataset.mnist, {"image":0, "label":1}, 128, 10, ...) ``` + ## Q & A ### Why return only a single entry, but not a mini batch? -If return a mini batch, data provider need to take care of batch size. But batch size is a concept for training, it makes more sense for user to specify batch size as a parameter for `train`. +If return a mini batch, data reader need to take care of batch size. But batch size is a concept for training, it makes more sense for user to specify batch size as a parameter for `train`. -Concretely, always return a single entry make reusing existing data providers much easier (e.g., if existing data provider return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2). +Practically, always return a single entry make reusing existing data reader much easier (e.g., if existing data reader return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2). -### How to create custom data provider +### Why use a dictionary but not a list to provide mapping? + +We decided to use dictionary (`{"image":0, "label":1}`) instead of list (`["image", "label"]`) is because that user can easily resue item (e.g., using `{"image_a":0, "image_b":0, "label":1}`) or skip item (e.g., using `{"image_a":0, "label":2}`). + +### How to create custom data reader ```python -def image_provider(path): - f = open(path) +def image_reader(image_path, label_path): + f = open(image_path) + l = open(label_path) images = numpy.fromfile( f, 'ubyte', count=n * 28 * 28).reshape((n, 28 * 28)).astype('float32') images = images / 255.0 * 2.0 - 1.0 labels = numpy.fromfile(l, 'ubyte', count=n).astype("int") for i in xrange(n): - yield [images[i, :], labels[i]] # a single entry of data is created each time + yield images[i, :], labels[i] # a single entry of data is created each time f.close() -# use python lambda to change image_provier into a function with no parameters. -provider = lambda : image_provier("/path/to/data/") -paddle.train(provider, ["image", "label"], ...) +# use python lambda to change image_reader into a function with no parameters. +reader = lambda : image_reader("/path/to/image_file", "/path/to/label_file") +paddle.train(reader, {"image":0, "label":1}, ...) ``` ### How is `paddle.train` implemented @@ -64,16 +128,14 @@ paddle.train(provider, ["image", "label"], ...) An example implementation of paddle.train could be: ```python -def make_minibatch_generator(reader, minibatch_size): +def minibatch_decorater(reader, minibatch_size): buf = [reader.next() for x in xrange(minibatch_size)] while len(buf) > 0: yield buf buf = [reader.next() for x in xrange(minibatch_size)] -def train(provider, mapping, batch_size, total_pass): +def train(reader, mapping, batch_size, total_pass): for pass_idx in range(total_pass): - for mini_batch in make_minibatch_generator(provider(pass_idx)): # this loop will never end in online learning. + for mini_batch in minibatch_decorater(reader()): # this loop will never end in online learning. do_forward_backward(mini_batch, mapping) ``` - -This is just an example implementation, more complicated logic like data prefetch could be implemented.