# Data Reader Interface

## DataTypes

`paddle.v2.data_type.``dense_array`(dim, seq_type=0)

  Dense Array. It means the input feature is dense array with float type. For example, if the input is an image with 28*28 pixels, the input of Paddle neural network could be a dense vector with dimension 784 or a numpy array with shape (28, 28).

  For the 2-D convolution operation, each sample in one mini-batch must have the similarly size in PaddlePaddle now. But, it supports variable-dimension feature across mini-batch. For the variable-dimension, the param dim is not used. While the data reader must yield numpy array and the data feeder will set the data shape correctly.

  Parameters:

  - dim (int) – dimension of this vector.
  - seq_type (int) – sequence type of input.

  Returns: | An input type object.
  Return type: | InputType

`paddle.v2.data_type.``dense_vector`(dim, seq_type=0)

  Dense Array. It means the input feature is dense array with float type. For example, if the input is an image with 28*28 pixels, the input of Paddle neural network could be a dense vector with dimension 784 or a numpy array with shape (28, 28).

  For the 2-D convolution operation, each sample in one mini-batch must have the similarly size in PaddlePaddle now. But, it supports variable-dimension feature across mini-batch. For the variable-dimension, the param dim is not used. While the data reader must yield numpy array and the data feeder will set the data shape correctly.

  Parameters:

  - dim (int) – dimension of this vector.
  - seq_type (int) – sequence type of input.

  Returns: | An input type object.
  Return type: | InputType

`paddle.v2.data_type.``dense_vector_sequence`(dim)

  Data type of a sequence of dense vector.

  Parameters: | dim (int) – dimension of dense vector.
  Returns: | An input type object
  Return type: | InputType

`paddle.v2.data_type.``integer_value`(value_range, seq_type=0)

  Data type of integer.

  Parameters:

  - seq_type (int) – sequence type of this input.
  - value_range (int) – range of this integer.

  Returns: | An input type object
  Return type: | InputType

`paddle.v2.data_type.``integer_value_sequence`(value_range)

  Data type of a sequence of integer.

  Parameters: | value_range (int) – range of each element.

`paddle.v2.data_type.``sparse_binary_vector`(dim, seq_type=0)

  Sparse binary vector. It means the input feature is a sparse vector and the every element in this vector is either zero or one.

  Parameters:

  - dim (int) – dimension of this vector.
  - seq_type (int) – sequence type of this input.

  Returns: | An input type object.
  Return type: | InputType

`paddle.v2.data_type.``sparse_binary_vector_sequence`(dim)
  Data type of a sequence of sparse vector, which every element is either zero
    or one.

  Parameters: | dim (int) – dimension of sparse vector.
  Returns: | An input type object
  Return type: | InputType

`paddle.v2.data_type.``sparse_float_vector`(dim, seq_type=0)

  Sparse vector. It means the input feature is a sparse vector. Most of the elements in this vector are zero, others could be any float value.

  Parameters:

  - dim (int) – dimension of this vector.
  - seq_type (int) – sequence type of this input.

  Returns: | An input type object.
  Return type: | InputType

`paddle.v2.data_type.``sparse_float_vector_sequence`(dim)

  Data type of a sequence of sparse vector, which most elements are zero, others could be any float value.

  Parameters: | dim (int) – dimension of sparse vector.
  Returns: | An input type object
  Return type: | InputType

`paddle.v2.data_type.``sparse_non_value_slot`(dim, seq_type=0)

  Sparse binary vector. It means the input feature is a sparse vector and the every element in this vector is either zero or one.

  Parameters:

  - dim (int) – dimension of this vector.
  - seq_type (int) – sequence type of this input.

  Returns: | An input type object.
  Return type: | InputType

`paddle.v2.data_type.``sparse_value_slot`(dim, seq_type=0)

  Sparse vector. It means the input feature is a sparse vector. Most of the elements in this vector are zero, others could be any float value.

  Parameters:

  - dim (int) – dimension of this vector.
  - seq_type (int) – sequence type of this input.

  Returns: | An input type object.
  Return type: | InputType

class `paddle.v2.data_type.``InputType`(dim, seq_type, tp)

  InputType is the base class for paddle input types.

  Note

  this is a base class, and should never be used by user.

  Parameters:

  - dim (int) – dimension of input. If the input is an integer, it means the value range. Otherwise, it means the size of layer.
  - seq_type (int) – sequence type of input. 0 means it is not a sequence. 1 means it is a variable length sequence. 2 means it is a nested sequence.
  - type (int) – data type of input.

## DataFeeder

## Reader

At training and testing time, PaddlePaddle programs need to read data. To ease the users’ work to write data reading code, we define that

- A reader is a function that reads data (from file, network, random number generator, etc) and yields data items.
- A reader creator is a function that returns a reader function.
- A reader decorator is a function, which accepts one or more readers, and returns a reader.
- A batch reader is a function that reads data (from reader, file, network, random number generator, etc) and yields a batch of data items.

### Data Reader Interface

Indeed, data reader doesn’t have to be a function that reads and yields data items. It can be any function with no parameter that creates a iterable (anything can be used in `for x in iterable`):

    iterable = data_reader()

Element produced from the iterable should be a single entry of data, not a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of supported type (e.g., numpy 1d array of float32, int, list of int)

An example implementation for single item data reader creator:

    def reader_creator_random_image(width, height):
        def reader():
            while True:
                yield numpy.random.uniform(-1, 1, size=width*height)
    return reader

An example implementation for multiple item data reader creator:

    def reader_creator_random_image_and_label(width, height, label):
        def reader():
            while True:
                yield numpy.random.uniform(-1, 1, size=width*height), label
    return reader

TODO(yuyang18): Should we add whole design doc here?

`paddle.v2.reader.``map_readers`(func, *readers)

  Creates a data reader that outputs return value of function using output of each data readers as arguments.

  Parameters:

  - func – function to use. The type of func should be (Sample) => Sample
  - readers – readers whose outputs will be used as arguments of func.

  Type: | callable
  Returns: | the created data reader.
  Return type: | callable

`paddle.v2.reader.``buffered`(reader, size)

  Creates a buffered data reader.

  The buffered data reader will read and save data entries into a buffer. Reading from the buffered data reader will proceed as long as the buffer is not empty.

  Parameters:

  - reader (callable) – the data reader to read from.
  - size (int) – max buffer size.

  Returns: | the buffered data reader.

`paddle.v2.reader.``compose`(*readers, **kwargs)

  Creates a data reader whose output is the combination of input readers.

  If input readers output following data entries: (1, 2) 3 (4, 5) The composed reader will output: (1, 2, 3, 4, 5)

  Parameters:

  - readers – readers that will be composed together.
  - check_alignment (bool) – if True, will check if input readers are aligned correctly. If False, will not check alignment and trailing outputs will be discarded. Defaults to True.

  Returns: | the new data reader.
  Raises: | ComposeNotAligned – outputs of readers are not aligned. Will not raise when check_alignment is set to False.

`paddle.v2.reader.``chain`(*readers)

  Creates a data reader whose output is the outputs of input data readers chained together.

  If input readers output following data entries: [0, 0, 0] [1, 1, 1] [2, 2, 2] The chained reader will output: [0, 0, 0, 1, 1, 1, 2, 2, 2]

  Parameters: | readers – input readers.
  Returns: | the new data reader.
  Return type: | callable

`paddle.v2.reader.``shuffle`(reader, buf_size)

  Creates a data reader whose data output is shuffled.

  Output from the iterator that created by original reader will be buffered into shuffle buffer, and then shuffled. The size of shuffle buffer is determined by argument buf_size.

  Parameters:

  - reader (callable) – the original reader whose output will be shuffled.
  - buf_size (int) – shuffle buffer size.

  Returns: | the new reader whose output is shuffled.
  Return type: | callable

`paddle.v2.reader.``firstn`(reader, n)

  Limit the max number of samples that reader could return.

  Parameters:

  - reader (callable) – the data reader to read from.
  - n (int) – the max number of samples that return.

  Returns: | the decorated reader.
  Return type: | callable

`paddle.v2.reader.``xmap_readers`(mapper, reader, process_num, buffer_size, order=False)

  Use multiprocess to map samples from reader by a mapper defined by user. And this function contains a buffered decorator. :param mapper: a function to map sample. :type mapper: callable :param reader: the data reader to read from :type reader: callable :param process_num: process number to handle original sample :type process_num: int :param buffer_size: max buffer size :type buffer_size: int :param order: keep the order of reader :type order: bool :return: the decarated reader :rtype: callable

`paddle.v2.reader.``pipe_reader`(left_cmd, parser, bufsize=8192, file_type='plain', cut_lines=True, line_break='\n')

  > pipe_reader read data by stream from a command, take it’s stdout into a pipe buffer and redirect it to the parser to parse, then yield data as your desired format.

  > You can using standard linux command or call another program to read data, from HDFS, Ceph, URL, AWS S3 etc:

  > cmd = “hadoop fs -cat /path/to/some/file” cmd = “cat sample_file.tar.gz” cmd = “curl http://someurl” cmd = “python print_s3_bucket.py”

  > A sample parser:

  > def sample_parser(lines):

  >   # parse each line as one sample data, # return a list of samples as batches. ret = [] for l in lines:

  >   > ret.append(l.split(” ”)[1:5])

  >   return ret

  > param left_cmd: | command to excute to get stdout from.
  > type left_cmd: | string
  > param parser: | parser function to parse lines of data. if cut_lines is True, parser will receive list of lines. if cut_lines is False, parser will receive a raw buffer each time. parser should return a list of parsed values.
  > type parser: | callable
  > param bufsize: | the buffer size used for the stdout pipe.
  > type bufsize: | int
  > param file_type:
  > | can be plain/gzip, stream buffer data type.
  > type file_type: | string
  > param cut_lines:
  > | whether to pass lines instead of raw buffer to the parser
  > type cut_lines: | bool
  > param line_break:
  > | line break of the file, like

  or

    type line_break:
    | string
    return: | the reader generator.
    rtype: | callable

Creator package contains some simple reader creator, which could be used in user program.

`paddle.v2.reader.creator.``np_array`(x)

  Creates a reader that yields elements of x, if it is a numpy vector. Or rows of x, if it is a numpy matrix. Or any sub-hyperplane indexed by the highest dimension.

  Parameters: | x – the numpy array to create reader from.
  Returns: | data reader created from x.

`paddle.v2.reader.creator.``text_file`(path)

  Creates a data reader that outputs text line by line from given text file. Trailing new line (‘\n’) of each line will be removed.

  Path: | path of the text file.
  Returns: | data reader of text file

`paddle.v2.reader.creator.``cloud_reader`(paths, etcd_endpoints, timeout_sec=5, buf_size=64)
  Create a data reader that yield a record one by one from
    the paths:

  Paths: | path of recordio files, can be a string or a string list.
  Etcd_endpoints: | the endpoints for etcd cluster
  Returns: | data reader of recordio files.

## minibatch

`paddle.v2.minibatch.``batch`(reader, batch_size)

  Create a batched reader.

  Parameters:

  - reader (callable) – the data reader to read from.
  - batch_size (int) – size of each mini-batch

  Returns: | the batched reader.
  Return type: | callable