Python Data Reader Design Doc¶

At training and testing time, PaddlePaddle programs need to read data. To ease the users’ work to write data reading code, we define that

A reader is a function that reads data (from file, network, random number generator, etc) and yields data items.
A reader creator is a function that returns a reader function.
A reader decorator is a function, which accepts one or more readers, and returns a reader.
A batch reader is a function that reads data (from reader, file, network, random number generator, etc) and yields a batch of data items.

and provide function which converts reader to batch reader, frequently used reader creators and reader decorators.

Data Reader Interface¶

Indeed, data reader doesn’t have to be a function that reads and yields data items. It can be any function with no parameter that creates a iterable (anything can be used in for x in iterable):

iterable = data_reader()

Element produced from the iterable should be a single entry of data, not a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of supported type (e.g., numpy 1d array of float32, int, list of int)

An example implementation for single item data reader creator:

def reader_creator_random_image(width, height):
    def reader():
        while True:
            yield numpy.random.uniform(-1, 1, size=width*height)
    return reader

An example implementation for multiple item data reader creator:

def reader_creator_random_image_and_label(width, height, label):
    def reader():
        while True:
            yield numpy.random.uniform(-1, 1, size=width*height), label
    return reader

Batch Reader Interface¶

batch reader can be any function with no parameter that creates a iterable (anything can be used in for x in iterable). The output of the iterable should be a batch (list) of data items. Each item inside the list must be a tuple.

Here are valid outputs:

# a mini batch of three data items. Each data item consist three columns of data, each of which is 1.
[(1, 1, 1),
(2, 2, 2),
(3, 3, 3)]

# a mini batch of three data items, each data item is a list (single column).
[([1,1,1],),
([2,2,2],),
([3,3,3],),

Please note that each item inside the list must be a tuple, below is an invalid output:

 # wrong, [1,1,1] needs to be inside a tuple: ([1,1,1],).
 # Otherwise it's ambiguous whether [1,1,1] means a single column of data [1, 1, 1],
 # or three column of datas, each of which is 1.
[[1,1,1],
[2,2,2],
[3,3,3]]

It’s easy to convert from reader to batch reader:

mnist_train = paddle.dataset.mnist.train()
mnist_train_batch_reader = paddle.batch(mnist_train, 128)

Also easy to create custom batch reader:

def custom_batch_reader():
    while True:
        batch = []
        for i in xrange(128):
            batch.append((numpy.random.uniform(-1, 1, 28*28),)) # note that it's a tuple being appended.
        yield batch

mnist_random_image_batch_reader = custom_batch_reader

Usage¶

batch reader, mapping from item(s) read to data layer, batch size and number of total pass will be passed into paddle.train:

# two data layer is created:
image_layer = paddle.layer.data("image", ...)
label_layer = paddle.layer.data("label", ...)

# ...
batch_reader = paddle.batch(paddle.dataset.mnist.train(), 128)
paddle.train(batch_reader, {"image":0, "label":1}, 128, 10, ...)

Data Reader Decorator¶

Data reader decorator takes a single or multiple data reader, returns a new data reader. It is similar to a python decorator, but it does not use @ syntax.

Since we have a strict interface for data readers (no parameter, return a single data item). Data reader can be used flexiable via data reader decorators. Following are a few examples:

Prefetch Data¶

Since reading data may take time and training can not proceed without data. It is generally a good idea to prefetch data.

Use paddle.reader.buffered to prefetch data:

buffered_reader = paddle.reader.buffered(paddle.dataset.mnist.train(), 100)

buffered_reader will try to buffer (prefetch) 100 data entries.

Compose Multiple Data Readers¶

For example, we want to use a source of real images (reusing mnist dataset), and a source of random images as input for Generative Adversarial Networks.

We can do:

def reader_creator_random_image(width, height):
    def reader():
        while True:
            yield numpy.random.uniform(-1, 1, size=width*height)
    return reader

def reader_creator_bool(t):
    def reader:
        while True:
            yield t
    return reader

true_reader = reader_creator_bool(True)
false_reader = reader_creator_bool(False)

reader = paddle.reader.compose(paddle.dataset.mnist.train(), data_reader_creator_random_image(20, 20), true_reader, false_reader)
# Skipped 1 because paddle.dataset.mnist.train() produces two items per data entry.
# And we don't care second item at this time.
paddle.train(paddle.batch(reader, 128), {"true_image":0, "fake_image": 2, "true_label": 3, "false_label": 4}, ...)

Shuffle¶

Given shuffle buffer size n, paddle.reader.shuffle will return a data reader that buffers n data entries and shuffle them before a data entry is read.

Example:

reader = paddle.reader.shuffle(paddle.dataset.mnist.train(), 512)

Q & A¶

Why reader return only a single entry, but not a mini batch?¶

Always returning a single entry make reusing existing data readers much easier (e.g., if existing reader return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2).

We provide function paddle.batch to turn (single entry) reader into batch reader.

Why do we need batch reader, isn’t train take reader and batch_size as arguments sufficient?¶

In most of the case, train taking reader and batch_size as arguments would be sufficent. However sometimes user want to customize order of data entries inside a mini batch. Or even change batch size dynamically.

Why use a dictionary but not a list to provide mapping?¶

We decided to use dictionary ({"image":0, "label":1}) instead of list (["image", "label"]) is because that user can easily resue item (e.g., using {"image_a":0, "image_b":0, "label":1}) or skip item (e.g., using {"image_a":0, "label":2}).

How to create custom data reader creator¶

def image_reader_creator(image_path, label_path, n):
    def reader():
        f = open(image_path)
        l = open(label_path)
        images = numpy.fromfile(
            f, 'ubyte', count=n * 28 * 28).reshape((n, 28 * 28)).astype('float32')
        images = images / 255.0 * 2.0 - 1.0
        labels = numpy.fromfile(l, 'ubyte', count=n).astype("int")
        for i in xrange(n):
            yield images[i, :], labels[i] # a single entry of data is created each time
        f.close()
        l.close()
    return reader

# images_reader_creator creates a reader
reader = image_reader_creator("/path/to/image_file", "/path/to/label_file", 1024)
paddle.train(paddle.batch(reader, 128), {"image":0, "label":1}, ...)

How is `paddle.train` implemented¶

An example implementation of paddle.train could be:

def train(batch_reader, mapping, batch_size, total_pass):
    for pass_idx in range(total_pass):
        for mini_batch in batch_reader(): # this loop will never end in online learning.
            do_forward_backward(mini_batch, mapping)