README.md 5.2 KB
Newer Older
1
# Python Data Reader Design Doc
2

3
Paddle reads data from *data reader* during training. *data reader* will be passed into `paddle.train` as a parameter.
4

5
## Data Reader Interface
6

7
Data reader is a function with no parameter that creates a iterable (anything can be used in `for x in iterable`):
8 9

```
10
iterable = data_reader()
11 12
```

H
Helin Wang 已提交
13
Element produced for the iterable should be a **single** entry of data, **not** a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of [supported type](http://www.paddlepaddle.org/doc/ui/data_provider/pydataprovider2.html?highlight=dense_vector#input-types) (e.g., numpy 1d array of float32, int, list of int)
14

15
An example implementation for single item data reader:
16

H
Helin Wang 已提交
17
```python
18
def data_reader_fake_image():
H
Helin Wang 已提交
19 20 21 22
	while True:
		yield numpy.random.uniform(-1, 1, size=20*20)
```

23
An example implementation for multiple item data reader:
H
Helin Wang 已提交
24
```python
25
def data_reader_fake_image_and_label():
H
Helin Wang 已提交
26 27
	while True:
		yield numpy.random.uniform(-1, 1, size=20*20), False
28
```
H
Helin Wang 已提交
29

30 31
## Usage

32
data reader, mapping from item(s) read to data layer, batch size and number of total pass will be passed into `paddle.train`:
33 34 35 36 37 38 39 40 41 42 43

```python
# two data layer is created:
image_layer = paddle.layer.data("image", ...)
label_layer = paddle.layer.data("label", ...)

# ...

paddle.train(paddle.dataset.mnist, {"image":0, "label":1}, 128, 10, ...)
```

44
## Data Reader Decorator
H
Helin Wang 已提交
45

46
*Data reader decorator* takes a single or multiple data reader, returns a new data reader. It is similar to a [python decorator](https://wiki.python.org/moin/PythonDecorators), but it does not use `@` syntax.
H
Helin Wang 已提交
47

48
Since we have a strict interface for data readers (no parameter, return a single data item). Data reader can be used flexiable via data reader decorators. Following are a few examples:
H
Helin Wang 已提交
49 50 51 52 53 54 55 56

### Prefetch Data

Since reading data may take time and training can not proceed without data. It is generally a good idea to prefetch data.

Use `paddle.reader.buffered` to prefetch data:

```python
57
buffered_reader = paddle.reader.buffered(paddle.dataset.mnist, 100)
H
Helin Wang 已提交
58 59
```

60
`buffered_reader` will try to buffer (prefetch) `100` data entries.
H
Helin Wang 已提交
61

62
### Compose Multiple Data Readers
H
Helin Wang 已提交
63 64 65 66 67 68

For example, we want to use a source of real images (reusing mnist dataset), and a source of fake images as input for [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661).

We can do:

```python
69
def data_reader_fake_image():
H
Helin Wang 已提交
70 71 72
	while True:
		yield numpy.random.uniform(-1, 1, size=20*20)

73
def data_reader_creator_bool(t):
74
	def reader:
75 76
		while True:
			yield t
77
	return reader
H
Helin Wang 已提交
78

79 80
true_reader = data_reader_creator_bool(True)
false_reader = data_reader_creator_bool(False)
H
Helin Wang 已提交
81

82
reader = paddle.reader.compose(paddle.dataset.mnist, data_reader_fake_image, true_reader, false_reader)
83 84
# Skipped 1 because paddle.dataset.mnist produces two items per data entry.
# And we don't care second item at this time.
85
paddle.train(reader, {"true_image":0, "fake_image": 2, "true_label": 3, "false_label": 4}, ...)
H
Helin Wang 已提交
86 87 88 89
```

### Shuffle

90
Given shuffle buffer size `n`, `paddle.reader.shuffle` will return a data reader that buffers `n` data entries and shuffle them before a data entry is read.
H
Helin Wang 已提交
91 92 93

Example:
```python
94
reader = paddle.reader.shuffle(paddle.dataset.mnist, 512)
95 96 97 98 99 100
```

## Q & A

### Why return only a single entry, but not a mini batch?

101
If a mini batch is returned, data reader need to take care of batch size. But batch size is a concept for training, it makes more sense for user to specify batch size as a parameter for `train`.
102

103
Practically, always return a single entry make reusing existing data readers much easier (e.g., if existing reader return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2).
104

H
Helin Wang 已提交
105 106 107 108
### Why use a dictionary but not a list to provide mapping?

We decided to use dictionary (`{"image":0, "label":1}`) instead of list (`["image", "label"]`) is because that user can easily resue item (e.g., using `{"image_a":0, "image_b":0, "label":1}`) or skip item (e.g., using `{"image_a":0, "label":2}`).

109
### How to create custom data reader
110 111

```python
112
def image_reader_creator(image_path, label_path, n):
113
	def reader():
114 115 116 117 118 119 120 121 122 123
		f = open(image_path)
		l = open(label_path)
		images = numpy.fromfile(
			f, 'ubyte', count=n * 28 * 28).reshape((n, 28 * 28)).astype('float32')
		images = images / 255.0 * 2.0 - 1.0
		labels = numpy.fromfile(l, 'ubyte', count=n).astype("int")
		for i in xrange(n):
			yield images[i, :], labels[i] # a single entry of data is created each time
		f.close()
		l.close()
124
	return reader
125

126 127 128
# images_reader_creator creates a reader
reader = image_reader_creator("/path/to/image_file", "/path/to/label_file", 1024)
paddle.train(reader, {"image":0, "label":1}, ...)
129 130 131 132 133 134 135
```

### How is `paddle.train` implemented

An example implementation of paddle.train could be:

```python
136
def minibatch_decorater(reader, minibatch_size):
137
	def ret():
138
		r = reader()
139 140 141 142 143
		buf = [r.next() for x in xrange(minibatch_size)]
		while len(buf) > 0:
			yield buf
			buf = [r.next() for x in xrange(minibatch_size)]
	return ret
144

145
def train(reader, mapping, batch_size, total_pass):
146
	for pass_idx in range(total_pass):
147
		for mini_batch in minibatch_decorater(reader): # this loop will never end in online learning.
148 149
			do_forward_backward(mini_batch, mapping)
```