Data Providers

Base DataProvider

class paddle::DataProvider

Base class for DataProvider, which supplies data for training.

Note

It can supplies multiple streams of data. For typical supervised training, there are two streams: one is for input, one is for label.

Subclassed by paddle::DataProviderGroup< T >, paddle::DummyDataProvider, paddle::MultiDataProvider, paddle::ProtoDataProvider, paddle::PyDataProvider, paddle::PyDataProvider2, paddle::SimpleDataProviderBase

Public Functions

DataProvider(const DataConfig &config, bool useGpu)

virtual ~DataProvider()

const DataConfig &getConfig() const

void setSkipShuffle()

int64_t getNextBatch(int64_t size, DataBatch *batch)

Get next batch of training samples.

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples

virtual void shuffle() = 0

Shuffle the data set.

virtual void reset()

reset all the value of index

Note

reset() must be called before any calls to getNextBatch() IMPORTANT: subclass reset() should always call the base class reset() at the end of the function

virtual int64_t getSize() = 0

Get the size of training samples.

Return

the number of training samples in the data set.

Note

return -1 to indicate unlimited number of samples.

virtual int64_t getNextBatchInternal(int64_t size, DataBatch *batch) = 0

Get next batch training samples internally.

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples

Public Static Functions

DataProvider *create(const DataConfig &config, const ModelConfig &modelConfig, bool useGpu = FLAGS_use_gpu)

static DataProvider *create(const DataConfig &config, bool useGpu = FLAGS_use_gpu)

create only used for unittest.

Public Static Attributes

ClassRegistrar<DataProvider, DataConfig, ModelConfig, bool> registrar_

Protected Functions

int64_t getNextBatchFromBuffer(int64_t size, DataBatch *batch)

@brief Get next batch training samples from buffer

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples

void initAsyncLoader()

Protected Attributes

DataConfig config_

bool skipShuffle_

float usageRatio_

bool useGpu_

std::unique_ptr<DoubleBuffer> doubleBuffer_

ThreadLocal<std::vector<MatrixPtr>> constantSlots_

DataProviderGroup

template <class T>

class paddle::DataProviderGroup

Inherits from paddle::DataProvider

Public Functions

DataProviderGroup(const DataConfig &config, bool useGpu)

~DataProviderGroup()

void reset()

reset all the value of index

Note

reset() must be called before any calls to getNextBatch() IMPORTANT: subclass reset() should always call the base class reset() at the end of the function

virtual void shuffle()

Shuffle the data set.

virtual int64_t getSize()

Get the size of training samples.

Return

the number of training samples in the data set.

Note

return -1 to indicate unlimited number of samples.

int64_t getNextBatchInternal(int64_t size, DataBatch *batch)

Get next batch training samples internally.

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples

Protected Types

typedef T ProviderType

typedef std::shared_ptr<ProviderType> ProviderPtrType

Protected Attributes

ProviderPtrType provider_

std::vector<std::string> fileList_

std::mutex lock_

std::unique_ptr<MultiThreadWorker<ProviderType>> loader_

MultiDataProvider

class paddle::MultiDataProvider

Inherits from paddle::DataProvider

Public Functions

MultiDataProvider(const DataConfig &config, const ModelConfig &modelConfig, bool useGpu)

~MultiDataProvider()

void reset()

reset all the value of index

Note

reset() must be called before any calls to getNextBatch() IMPORTANT: subclass reset() should always call the base class reset() at the end of the function

void shuffle()

Shuffle the data set.

virtual int64_t getSize()

Get the size of training samples.

Return

the number of training samples in the data set.

Note

return -1 to indicate unlimited number of samples.

int64_t getNextBatchInternal(int64_t size, DataBatch *batch)

Get next batch training samples internally.

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples

bool isTestMode() const

Protected Attributes

std::vector<std::unique_ptr<DataProvider>> subDataProviders_

PyDataProvider

IFieldScanner

class paddle::IFieldScanner

FieldScanner Interface.

It will read python object, and fill to argument’s each slot. There are two steps, prepare and fill. Scanner will alloc memory during prepare step, fill data into argument during fill step.

Subclassed by paddle::DenseScanner, paddle::IndexScanner, paddle::SequenceScanner, paddle::SparseNonValueScanner

Public Functions

DISABLE_COPY(IFieldScanner)

IFieldScanner(SlotHeader *headerPtr)

Ctor.

Parameters

• headerPtr: slot header that scanner belong to.

virtual ~IFieldScanner()

virtual void startPrepare(Argument &argument)

Start prepare step.

virtual void prepare(Argument &argument, PyObject *obj)

Prepare step.

Note

the obj could be a timestep of sample or whole sample. It depends what scanner it is.

virtual void finishPrepare(Argument &argument)

Finish Prepare step.

virtual void startFill(Argument &argument)

Start fill step.

virtual void fill(Argument &argument, PyObject *obj)

Fill step.

Note

the obj could be a timestep of sample or whole sample. It depends what scanner it is.

virtual void finishFill(Argument &argument)

Finish fill step.

Public Static Functions

IFieldScanner *create(SlotHeader *header)

Factory method. Create a scanner by header. The final scanner may be combine many scanners.

Note

Fatal if header is not support.

Protected Attributes

SlotHeader *headerPtr_

DenseScanner

class paddle::DenseScanner

Scanner for dense slot.

Inherits from paddle::IFieldScanner

Public Functions

DenseScanner(SlotHeader *ptr)

virtual void prepare(Argument &argument, PyObject *obj)

Prepare.

Parameters

• argument: target argument
• obj: each timestep of a sample.

virtual void finishPrepare(Argument &argument)

Finish Prepare step.

virtual void fill(Argument &argument, PyObject *obj)

Fill argument from obj.

Parameters

• argument:
• obj:

IndexScanner

class paddle::IndexScanner

Scanner for index slot

Inherits from paddle::IFieldScanner

Public Functions

IndexScanner(SlotHeader *ptr)

virtual void prepare(Argument &argument, PyObject *obj)

Prepare memory space.

Note

obj is a single timestep of sample

virtual void finishPrepare(Argument &argument)

Finish Prepare step.

virtual void fill(Argument &argument, PyObject *obj)

Fill one index to argument.

SparseNonValueScanner

class paddle::SparseNonValueScanner

Inherits from paddle::IFieldScanner

Subclassed by paddle::SparseValueScanner

Public Functions

SparseNonValueScanner(SlotHeader *ptr)

virtual void prepare(Argument &argument, PyObject *obj)

Prepare memory space

Note

obj is a timestep of one sample.

virtual void finishPrepare(Argument &argument)

Finish Prepare step.

virtual void startFill(Argument &argument)

Start fill step.

virtual void fill(Argument &argument, PyObject *obj)

Fill one sparse vector to argument.

Note

obj is a timestep of one sample.

Protected Functions

virtual void setData(int *col, real *dat, PyObject *obj)

Set a single sparse index and value.

Parameters

• col: sparse index
• dat: sparse value
• obj: Python Object. For sparse_non_value is a PyInt or PyLong. For sparse_value is a Tuple (int, float).

Protected Attributes

size_t nnz_

size_t height_

SparseValueScanner

class paddle::SparseValueScanner

Inherits from paddle::SparseNonValueScanner

Public Functions

SparseValueScanner(SlotHeader *ptr)

virtual void finishPrepare(Argument &argument)

Finish Prepare step.

Protected Functions

virtual void setData(int *col, real *dat, PyObject *obj)

Set a single sparse index and value.

Parameters

• col: sparse index
• dat: sparse value
• obj: Python Object. For sparse_non_value is a PyInt or PyLong. For sparse_value is a Tuple (int, float).

SequenceScanner

class paddle::SparseValueScanner

Inherits from paddle::SparseNonValueScanner

Public Functions

SparseValueScanner(SlotHeader *ptr)

virtual void finishPrepare(Argument &argument)

Finish Prepare step.

Protected Functions

virtual void setData(int *col, real *dat, PyObject *obj)

Set a single sparse index and value.

Parameters

• col: sparse index
• dat: sparse value
• obj: Python Object. For sparse_non_value is a PyInt or PyLong. For sparse_value is a Tuple (int, float).

IPyDataProviderCache

class paddle::IPyDataProviderCache

Py Data Provider Cache Interface.

Subclassed by paddle::CacheOnePassInMemory, paddle::NoCacheStrategy

Public Functions

virtual ~IPyDataProviderCache()

virtual bool reset() = 0

invoke when DataProvider::reset()

Return

true if read data from python.

virtual void drop(std::deque<PyObjectPtr> *data) = 0

invoke when these data are used by DataProvider, and need to clear.

Note

The implemented class must clear these data array. Or if you want to delete the PyObjectPtr later, you should make sure the paddle process only have one active thread calling python code (use PyGuard otherwise).

Parameters

• data: used data.

virtual std::deque<PyObjectPtr> *load() = 0

Return whole data in cache.

Public Static Functions

IPyDataProviderCache *create(CacheType ct)

Factory method. Convert CacheType to IPyDataProviderCache*

NoCacheStrategy

class paddle::NoCacheStrategy

No Cache Strategy. Will destruct old data immediately and load data from python every pass.

Inherits from paddle::IPyDataProviderCache

Public Functions

virtual bool reset()

invoke when DataProvider::reset()

Return

true if read data from python.

virtual void drop(std::deque<PyObjectPtr> *data)

invoke when these data are used by DataProvider, and need to clear.

Note

The implemented class must clear these data array. Or if you want to delete the PyObjectPtr later, you should make sure the paddle process only have one active thread calling python code (use PyGuard otherwise).

Parameters

• data: used data.

virtual std::deque<PyObjectPtr> *load()

Return whole data in cache.

CacheOnePassInMemory

class paddle::CacheOnePassInMemory

Cache One Pass In Memory strategy.

In first pass, will load data from python and store them in memory. The rest passes, will load data from memory.

Inherits from paddle::IPyDataProviderCache

Public Functions

CacheOnePassInMemory()

virtual bool reset()

invoke when DataProvider::reset()

Return

true if read data from python.

virtual void drop(std::deque<PyObjectPtr> *data)

invoke when these data are used by DataProvider, and need to clear.

Note

The implemented class must clear these data array. Or if you want to delete the PyObjectPtr later, you should make sure the paddle process only have one active thread calling python code (use PyGuard otherwise).

Parameters

• data: used data.

virtual std::deque<PyObjectPtr> *load()

Return whole data in cache.

IPyDataProvider

class paddle::PyDataProvider2

PyDataProvider2.

For usage, please refer python module ‘paddle.trainer.PyDataProvider2’

Here, we start a thread to read data. It is totally asynchronous for reading data. And it support cache strategies.

Inherits from paddle::DataProvider

Public Functions

PyDataProvider2(const DataConfig &config, const ModelConfig &modelConfig, bool useGpu)

Ctor

virtual ~PyDataProvider2()

Dtor

Note

will stop loading thread when destructing

virtual void reset()

Resetting the PyDataProvider. May start reading thread here.

void shuffle()

Shuffle. Do nothing because PyDataProvider do shuffle implicitly by random select data from datapool.

int64_t getSize()

Not limited size.

int64_t getNextBatchInternal(int64_t size_, DataBatch *batch)

Loading a batch of data.

Proto Data Provider

ProtoDataProvider

class paddle::ProtoDataProvider

Provider data from protobuf data file with each sample specified by proto message.

DataSample defined in DataFormat.proto.

The file format is

header

sample1

sample2

...

sampleN

Note

: In the data file, each message is prefixed with its length. The read/write of the protbuf are implemented in ProtoReader.h

Inherits from paddle::DataProvider

Subclassed by paddle::ProtoSequenceDataProvider

Public Functions

ProtoDataProvider(const DataConfig &config, bool useGpu, bool loadDataAll = true)

void reset()

reset all the value of index

Note

reset() must be called before any calls to getNextBatch() IMPORTANT: subclass reset() should always call the base class reset() at the end of the function

virtual int64_t getSize()

Note

this size includes the sequences which are skipped because they are longer than the batch size.

void shuffle()

Shuffle the data set.

void loadData(const std::vector<std::string> &fileList)

int64_t getNextBatchInternal(int64_t size, DataBatch *batch)

Get next batch training samples internally.

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples

Protected Functions

void loadData(const std::string &fileName)

load protobuf data from a list of file

Parameters

• fileName: file name of a file which contains a list of file names

void loadDataFile(const std::string &fileName)

load protobuf data from file

Parameters

• fileName: data file name

void checkDataHeader(const DataHeader &header)

check data header of each data sample

Parameters

• header: data header read from protobuf data

void fillSlots(const DataSample &sample)

fill protobuf data into slot_, slot_ is a vector of ProtoSlot in memory.

Parameters

• sample: data sample read from protobuf data

bool iidData() const

return true if each sample is one sequence, i.e., independent of other samples.

void checkSample(const DataSample &sample)

check that sample is consistent with header_

template <class Op>

int64_t sequenceLoop(Op op, int64_t size)

template <class Op>

int64_t sampleLoop(Op op, int64_t size)

template <class Op>

int64_t subSampleLoop(Op op, int64_t size, int slot)

void showDataStats()

Protected Attributes

DataHeader header_

int numVecSlots_

std::vector<ProtoSlot> slots_

size_t sampleNums_

std::vector<size_t> sequenceStartPositions_

The starting position of each sequence in samples. The last element should be num of samples. If empty, each sample is one sequence.

int64_t currentSequenceIndex_

std::vector<size_t> shuffledSequenceIds_

ThreadLocalD<DataBatch> cpuBatch_

ThreadLocalD<DataBatch> gpuBatch_

RWLock lock_

std::vector<StatPtr> nnzStats_

struct ProtoSlot

Public Members

SlotDef::SlotType type

int dim

std::vector<int> indexData

std::vector<real> denseData

std::vector<sparse_non_value_t> sparseNonValueData

std::vector<sparse_float_value_t> sparseFloatValueData

std::vector<int64_t> indices

std::vector<int64_t> subIndices

std::vector<ProtoVarSlot> varDenseData

std::vector<std::vector<int>> varIndices

std::vector<std::string> strData

struct ProtoVarSlot

Public Members

std::vector<real> data

std::vector<int> dims

ProtoSequenceDataProvider

class paddle::ProtoSequenceDataProvider

Special use for Proto data: instances should contain sparse-non-value slots and label.

Note

ProtoSequenceDataProvider treats each SPARSE SLOT as a SEQUENCE

Inherits from paddle::ProtoDataProvider

Public Functions

ProtoSequenceDataProvider(const DataConfig &config, bool useGpu, bool loadDataAll = true)

~ProtoSequenceDataProvider()

int64_t getNextBatchInternal(int64_t size, DataBatch *batch)

Get next batch training samples internally.

Return

actual size of obtained training samples

Parameters

• size: size of training samples to get
• batch: a batch of training samples