Gradient machines

Networks

namespace paddle

class MultiNetwork

Inherits from paddle::NeuralNetwork

Public Functions

MultiNetwork(std::string subModelName = "")

virtual void init(const ModelConfig &config, ParamInitCallback callback, const std::vector<ParameterType> &parameterTypes, bool useGpu)

virtual void prefetch(const std::vector<Argument> &inArgs)

Prefetch row ids of sparse parameter.

virtual void forward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType)

Forward propagation.

Calculate outputs (outArgs) based the inputs (inArgs)

Note

: if passType==PASS_TEST, then backward() should not be called

virtual void backward(const UpdateCallback &callback = nullptr)

Backward propagation.

Calculate the gradient of inArgs and parameter.

This function should only be called after a corresponding forward() call. The caller is responsible for filling the correct grad for the outArgs obtained using forward().

It may also change the grad field for the inArgs supplied at forward()

virtual void forwardBackward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType, const UpdateCallback &callback)

Combine forward() and backward(). For multithread training, this may be faster.

Note

: passType PASS_TEST is not allowed for forwardBackward().

virtual void onPassEnd()

virtual Evaluator *makeEvaluator()

Create an evaluator which can be used for eval()

virtual void eval(Evaluator *evaluator)

evaluate using the given evaluator

const std::vector<std::unique_ptr<NeuralNetwork>> &getSubNetworks() const

virtual void start(const TrainerConfig &config, DataProviderPtr dataProvider)

Used before formal training, start work-threads and set trainer Parameters;.

Note

This function will only been implemented and used in a multithreaded environment.

virtual void finish()

check each work-thread whether is failed/error/finish, if not, return ture, and yes return false.

Note

This function will only been implemented and used in a multithreaded environment.

Protected Attributes

std::vector<std::unique_ptr<NeuralNetwork>> subNetworks_

namespace paddle

Enums

enum TaskType

Values:

TASK_FORWARD = 0

TASK_BACKWARD = 1

TASK_END_LAYER = 2

TASK_THREAD_FINISH = 3

class ParallelNeuralNetwork

#include <ParallelNeuralNetwork.h>

A ParallelNeuralNetwork is capable of calculating a neural network through multiple threads in parallel.

Inherits from paddle::NeuralNetwork

Public Functions

ParallelNeuralNetwork(std::string subModelName = "", NeuralNetwork *rootNetwork = nullptr)

virtual void paddle::ParallelNeuralNetwork::init(const ModelConfig & config, ParamInitCallback callback = nullptr, const std::vector< ParameterType > & parameterTypes = std::vector< ParameterType >{PARAMETER_VALUE, PARAMETER_GRADIENT, PARAMETER_MOMENTUM}, bool useGpu = FLAGS_use_gpu)

virtual void forward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType)

Forward propagation.

Calculate outputs (outArgs) based the inputs (inArgs)

Note

: if passType==PASS_TEST, then backward() should not be called

virtual void backward(const UpdateCallback &callback = nullptr)

Backward propagation.

Calculate the gradient of inArgs and parameter.

This function should only be called after a corresponding forward() call. The caller is responsible for filling the correct grad for the outArgs obtained using forward().

It may also change the grad field for the inArgs supplied at forward()

virtual void forwardBackward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType, const UpdateCallback &callback = NULL)

Combine forward() and backward(). For multithread training, this may be faster.

Note

: passType PASS_TEST is not allowed for forwardBackward().

virtual void start(const TrainerConfig &config, DataProviderPtr dataProvider)

Used before formal training, start work-threads and set trainer Parameters;.

Note

This function will only been implemented and used in a multithreaded environment.

void addComputeThread(int deviceId)

void dispatchByDeviceId(int deviceId, LayerPtr layer, TaskType task)

void waitAllThread()

Protected Attributes

bool useGpu_

int numDevices_

std::vector<std::unique_ptr<ParallelThread>> threads_

class ParallelThread

Public Types

typedef Queue<Job> JobQueue

Public Functions

ParallelThread(int threadId, int deviceId, bool useGpu)

~ParallelThread()

void jobEnqueue(LayerPtr layer, TaskType task)

void start()

void stop()

int getDeviceId() const

void setBackwardCallback(const UpdateCallback &callback)

void setForwardPassType(PassType passType)

Public Members

JobQueue queue_

Protected Functions

void computeThread()

Protected Attributes

int threadId_

int deviceId_

bool useGpu_

std::unique_ptr<std::thread> computeThread_

bool stopping_

UpdateCallback backwardCallback_

PassType passType_

struct Job

Public Members

LayerPtr layer_

TaskType task_

Gradient Machines

namespace paddle

Typedefs

typedef std::vector<LayerStatePtr> MachineState

A gradient machine is capable of calculating some outputs given some inputs and performing gradient calculation based on the derivative from the outputs.

A gradient machine can be either a full neural network or part of a neural network.

Usage for training:

1. Prepare inArgs. Put your input data into inArgs[i].value.
2. Call forward(inArgs, &outArgs)
3. Calculate gradient with respect to outArgs[i]->value and fill them into outArgs[i]->grad. This step can be skipped if your the outputs are from cost layers.
4. Call backward(). After backward, gradient of each parameter is accumulated to getParameters()[i]->getBuf(PARAMETER_GRADIENT)
5. Update parameter value getParameters()[i]->getBuf(PARAMETER_VALUE) using gradients.
6. Clear gradients to zero.

Usage for prediction:

1. Prepare inArgs. Put your input data into inArgs[i].value.
2. Call forward(inArgs, &outArgs)
3. Obtain the prediction result from outArgs[i]

typedef std::shared_ptr<GradientMachine> GradientMachinePtr

class GradientMachine

Subclassed by paddle::MultiGradientMachine, paddle::NeuralNetwork

Public Types

enum CreateMode

Values:

kNormal = 0

kSgdSparseCpuTraining = 3

kTesting = 4

kCustom = 10

Public Functions

virtual ~GradientMachine()

virtual void prefetch(const std::vector<Argument> &inArgs)

Prefetch row ids of sparse parameter.

virtual void forward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType) = 0

Forward propagation.

Calculate outputs (outArgs) based the inputs (inArgs)

Note

: if passType==PASS_TEST, then backward() should not be called

virtual void backward(const UpdateCallback &callback = nullptr) = 0

Backward propagation.

Calculate the gradient of inArgs and parameter.

This function should only be called after a corresponding forward() call. The caller is responsible for filling the correct grad for the outArgs obtained using forward().

It may also change the grad field for the inArgs supplied at forward()

virtual void forwardBackward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType, const UpdateCallback &callback = nullptr)

Combine forward() and backward(). For multithread training, this may be faster.

Note

: passType PASS_TEST is not allowed for forwardBackward().

virtual void resetState()

virtual void setState(const MachineState &machineState)

virtual void getState(MachineState &machineState)

virtual void onPassEnd() = 0

virtual Evaluator *makeEvaluator() = 0

Create an evaluator which can be used for eval()

virtual void eval(Evaluator *evaluator) = 0

evaluate using the given evaluator

std::vector<ParameterPtr> &getParameters()

std::vector<ParameterPtr> &getNonStaticParameters()

bool hasStaticParameters()

virtual void start(const TrainerConfig &config, DataProviderPtr dataProvider)

Used before formal training, start work-threads and set trainer Parameters;.

Note

This function will only been implemented and used in a multithreaded environment.

virtual void finish()

check each work-thread whether is failed/error/finish, if not, return ture, and yes return false.

Note

This function will only been implemented and used in a multithreaded environment.

virtual bool trainIsOn()

set the training status a “finished” value, the sub_work_threads will option the change, and then exit.

Note

This function will only been implemented and used in a multithreaded environment.

virtual void restart()

when all or some of the sub-workThreads are suspended to waiting controller’s instructions, and after some processing done in the controller, it will call this function to wake up all the pending thread.

Note

This function will only been implemented and used in a multithreaded environment.

virtual void setOutputGrad(const std::vector<Argument> &args)

void saveParameters(const std::string &dir) const

void loadParameters(const std::string &dir)

void randParameters()

virtual void getStats(real &cost, int64_t &numProcessed)

Public Static Functions

GradientMachine * GradientMachine::create(const ModelConfig & config, int mode = kNormal, const std::vector< ParameterType > & parameterTypes = std::vector< ParameterType >{PARAMETER_VALUE, PARAMETER_GRADIENT, PARAMETER_MOMENTUM})

Create a gradient machine from ModelConfig Parameter will have parameterTypes

GradientMachine *create(const std::string &modelFile, DataConfig *dataConfig)

Create a gradient machine from the merged model file. The merged model file can be generated using tools/merge_model If dataConfig is not null, it will be filled with the DataConfig from the TrainerConfig

GradientMachine *create(std::istream &is, DataConfig *dataConfig)

Create a gradient machine from a stream which contains the merged model file. The merged model file can be generated using tools/merge_model If dataConfig is not null, it will be filled with the DataConfig from the TrainerConfig

GradientMachine *create(const std::string &modelFile, TrainerConfig *trainerConfig)

Create a gradient machine from the merged model file. The merged model file can be generated using tools/merge_model If trainerConfig is not null, it will be filled with the TrainerConfig

GradientMachine *create(std::istream &is, TrainerConfig *trainerConfig)

Create a gradient machine from a stream which contains the merged model file. The merged model file can be generated using tools/merge_model If trainerConfig is not null, it will be filled with the TrainerConfig

Protected Functions

virtual void onLoadParameter()

Protected Attributes

std::vector<ParameterPtr> parameters_

std::vector<ParameterPtr> nonStaticParameters_

namespace paddle

Typedefs

typedef Queue<int> PidQueue

typedef std::unique_ptr<TrainerThread> TrainerThreadPtr

struct GradBuffer

Public Members

int paramId

Semaphore sem

std::vector<VectorPtr> bufs

class MultiGradientMachine

#include <MultiGradientMachine.h>

A MultiGradientMachine is a synchronous GradientMachine which devides one data batch into several smaller batches and assign each one small batch to one computint thread for computation. After each thread finishes computation, it merges result (including output Argument and gradient during backward()). It basically is the same as single thread gradient machine, except that it uses multi-thread to do the computation.

It handles GPU and Cpu parameters differently. In GPU, one computing thread generally corresponds to one GPU device. Thus, each thread keeps a separate copy of the parameter in its own device’s memory. In CPU, we only need to keep one copy of the parameters in the main memory. After, each computing thread computes its own parameter gradient, the update process needs to accumulate the parameter gradients from all the computing threads, and update the accumulated parameter gradient to the corresponding parameter value.

Each GPU parameter is assigned to a thread called its main thread. For each parameter, the accumulation of its gradients and the update of its value happens in its main thread. The main thread first gather the parameter gradients from all the computing thread. Then, it performs parameter update. After a gradient is updated by the main thread, it is scattered to all the computing thread so that the parameters in all the computing threads are synchronized. The scatter and gather process are implemented by ring-style communication. Assume we have N computing threads, its thread ids will be 0, 1, ..., N-1. For each parameter, the id of the main thread is specified in paraMainThread_[pid], where pid is the id of the parameter. Each thread i only sends data to its partner thread (i - 1) % N. For example, for a parameter gradient that is computed in thread 4, and its main thread is 2. Its traveling process would be 4, 5,..., N-1, 0, 1, 2. In each step, the gradient buffer is added to the local gradient, and the local gradient is then copied to the gradient buffer of the next thread. At last, its main thread 2 will get the accumulated parameter gradient. For the same parameter, after its value is updated, the value’s traveling process would be 2, 1, 0, N-1, ... 3. At the end, all the computing threads would have the updated parameter value.

A computing thread (TrainerThread) uses 4 threads to do different jobs:

1. computeThread(): performing forward(), backward(), prefetch().
2. valueDispatchThread(): copying parameter values to partner thread.
3. copyGradToBufferThread(): copying parameter gradient to partner thread.
4. gradCollectThread(): merging the gradient from step 3 with local gradient and call the callback supplied by the user to update parameter value.

CPU parameter value has only one copy. And their gradients are merged at the end of backward().

• Handling of sparse update Currently, sparse update is only supported for CPU parameters.

Sparse updates refers to gradient caculation where the gradient is sparse. For example, if the input argument to a ‘fc’ layer is sparse, the gradient of the weight matrix of this layer will be sparse. It is usually more efficient to treat the gradient explicitly as sparse vector during the parameter update.

There are two types of sparse updates called local sparse update and remote sparse update.

For both types of sparse updates, there is one copy of parameter value and gradient called main parameter value and gradient, and there is a copy of parameter value and gradient for each computing thread called slave parameter value and gradient. The slave parameter values are always shared with the corresponding main parameter value. The slave parameter grad is a sparse row matrix. The sparse pattern for slave parameter grads are different, because the small batches for each computing thread might have different sparsity pattern.

1. Local sparse update

   Main parameter value type is MAT_NORMAL. It is a dense matrix.

   Main parameter grad type is MAT_SPARSE_ROW_IDS (SparseRowIdsCpuMatrix) It is also a dense matrix, but the updated values are specified by IDS.

   Slave parameter value shares with main parameter value.

   Slave parameter grad type is MAT_SPARSE_ROW_AUTO_GROW (SparseAutoGrowRowCpuMatrix). It is a sparse row matrix.

   During backward() of each TrainerThread, SparseAutoGrowRowCpuMatrix will gather all the non-zero gradient. And After backward(), they will be merged into main parameter grad (SparseRowIdsCpuMatrix), with indices indicating which rows have nonzero gradient.

2. Remote sparse update

   Main parameter value type is MAT_SPARSE_ROW_PREFETCH(_FULL_SIZE) (SparsePrefetchRowCpuMatrix). MAT_SPARSE_ROW_PREFETCH is a sparse matrix. MAT_SPARSE_ROW_PREFETCH_FULL_SIZE is a dense matrix. However, only the parameter values that are prefetched is up-to-date.

   Main parameter grad type is MAT_SPARSE_ROW (SparseRowCpuMatrix). And it shares sparse pattern with value by sharing indexDictHandle_, which is an internal data structure used by SparseRowCpuMatrixto specify the sparsity pattern of Slave parameter value shares with main parameter value.

   Slave parameter grad type is MAT_SPARSE_ROW_AUTO_GROW (SparsePrefetchRowCpuMatrix). It is a sparse row matrix

   During prefetch(), all the layers will indicates which rows of each parameter are needed. Then the framework will retrieve those rows from parameter server.

   During backward() of each TrainerThread, SparseAutoGrowRowCpuMatrix will gather all the non-zero gradient. And After backward(), they will be merged into main parameter grad (SparseRowCpuMatrix). And the framework will send the merged gradient to parameter server.

Inherits from paddle::GradientMachine

Public Types

enum TaskType

Values:

TASK_FORWARD_BACKWARD = 0

TASK_FORWARD = 1

TASK_BACKWARD = 2

TASK_COPY_IN_ARGS = 3

Public Functions

MultiGradientMachine(const ModelConfig &config, bool useGpu)

virtual void prefetch(const std::vector<Argument> &inArgs)

Prefetch row ids of sparse parameter.

virtual void forward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType)

Forward propagation.

Calculate outputs (outArgs) based the inputs (inArgs)

Note

: if passType==PASS_TEST, then backward() should not be called

virtual void backward(const UpdateCallback &callback = nullptr)

Backward propagation.

Calculate the gradient of inArgs and parameter.

This function should only be called after a corresponding forward() call. The caller is responsible for filling the correct grad for the outArgs obtained using forward().

It may also change the grad field for the inArgs supplied at forward()

virtual void forwardBackward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType, const UpdateCallback &callback)

Combine forward() and backward(). For multithread training, this may be faster.

Note

: passType PASS_TEST is not allowed for forwardBackward().

virtual void onPassEnd()

virtual void finish()

check each work-thread whether is failed/error/finish, if not, return ture, and yes return false.

Note

This function will only been implemented and used in a multithreaded environment.

virtual Evaluator *makeEvaluator()

Create an evaluator which can be used for eval()

virtual void eval(Evaluator *evaluator)

evaluate using the given evaluator

bool useGpu() const

bool isPassGrad()

void setPassGrad(bool isPass)

virtual void setOutputGrad(const std::vector<Argument> &args)

Protected Functions

std::vector<TrainerThreadPtr> &getAllThreads()

int logicalDeviceId2RealDeviceId(int logicalId, int threadId = 0) const

int realDeviceId2LogicalDeviceId(int realId, int threadId = 0) const

std::vector<const std::vector<ParameterPtr> *> getSlaveParameters()

bool hasNonstaticCpuParamters() const

void waitBeforeMerge()

void waitAfterMerge()

void waitForCopyInArgs()

TrainerThreadPtr &getThread(int threadId)

std::vector<GradBuffer> &getGradBuf(int threadId)

PassType getPassType() const

void notifyGradientTransfer(int paramId)

const std::vector<Argument> &getInArgs()

TaskType getTaskType() const

const UpdateCallback &getBackwardCallback() const

int getNumDevices() const

int getNumLogicalDevices() const

int getNumThreads() const

int paraMainThread(int pid) const

virtual void forwardImp(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType, TaskType taskType)

virtual void backwardImp(const UpdateCallback &callback = NULL)

void updateThreadParameters()

void startTask(TaskType taskType)

void getOutArgs(std::vector<Argument> *outArgs, PassType passType)

void allocGradBufs()

Protected Attributes

bool useGpu_

bool hasNonstaticCpuParamters_

std::unique_ptr<GradientMachine> gradientMachine_

std::vector<TrainerThreadPtr> threads_

std::vector<int> paraMainThread_

std::vector<std::vector<GradBuffer>> gradBufs_

std::vector<size_t> bufferSizes_

PassType passType_

TaskType taskType_

PidQueue gradQueue_

std::vector<Argument> inArgs_

std::vector<Argument> outArgs_

hl_stream_t outArgStream_

std::vector<ParameterType> mergeTypes_

int numDevices_

int numLogicalDevices_

int numThreads_

UpdateCallback backwardCallback_

ThreadBarrier trainerBarrier_

ThreadBarrier allBarrier_

bool inArgsCopied_

bool isPassGrad_

Friends

friend paddle::TrainerThread

class TrainerThread

Public Functions

TrainerThread(const ModelConfig &config, int threadId, MultiGradientMachine *multiMachine)

~TrainerThread()

void start()

void onPassEnd()

void waitOutArgsReady()

void notifyTaskReady()

int getDeviceId() const

GradientMachine *getGradientMachine()

const std::vector<ParameterPtr> &getParameters()

void stop()

void notifyValueReady(int paramId)

const VectorPtr &getValueBuf(int paramId)

const std::vector<Argument> &getOutArgs()

void incUpdateCounter(int n = 1)

void notifyGradientCollect(int paramId)

void notifyCopyGradToBuffer(int paramId)

void notifyValueDispatch(int paramId)

void prefetch()

void copyOutputGrad()

Protected Functions

void mergeCpuGradients()

void mergeGradSparse(Parameter *para, std::vector<const std::vector<ParameterPtr> *> &slaveParameters)

void mergeGradSparseRemote(Parameter *para, std::vector<const std::vector<ParameterPtr> *> &slaveParameters)

void mergeGradDense(Parameter *para, std::vector<const std::vector<ParameterPtr> *> &slaveParameters)

void computeThread()

void valueDispatchThread()

void copyGradToBufferThread()

void gradCollectThread()

void copyInArgs()

void forward()

void backward()

void backwardCallback(Parameter *para)

void doCallback(int pid)

Protected Attributes

MultiGradientMachine *multiMachine_

ModelConfig config_

bool stopping_

int partnerId_

int threadId_

int deviceId_

std::unique_ptr<GradientMachine> gradientMachine_

std::vector<ParameterPtr> parameters_

std::vector<ParameterType> mergeTypes_

std::unique_ptr<std::thread> computeThread_

std::vector<Argument> inArgs_

std::vector<Argument> outArgs_

Semaphore taskReadySem_

Semaphore outArgsReadySem_

std::unique_ptr<std::thread> copyThread_

PidQueue gradBufQueue_

hl_stream_t gradStream_

std::unique_ptr<std::thread> gradCollectThread_

PidQueue gradQueue_

UpdateCallback backwardCallback_

std::unique_ptr<std::thread> valueDispatchThread_

PidQueue valueReadyQueue_

LockedCondition valueReadyCond_

hl_stream_t valueStream_

std::atomic<int> updateCounter_

bool parameterUpdated_

bool inArgsCopied_

Recurrent Gradient Machines

namespace paddle

class RecurrentGradientMachine

Inherits from paddle::NeuralNetwork

Public Types

typedef std::function<void(const std::vector<std::vector<int> *>&, NeuralNetwork *, const int)> BeamSearchCandidatesAdjustCallback

BeamSearchCandidatesAdjustCallback.

Adjust searching candidates to restrict beam search searching within a limited subset of all possibile paths.

The first parameter is the prefixes of all formed paths in current beam search step, whose type is basically int[][].

The second parameter is a pointer to the network used to generate sequence, user can use this pointer to tranverse each layer in the network to modify behaivors of a particular layer.

The third parameter is an integer to indicate the iteration number of beam search, so that user can customize different operations in different beam search iterations.

typedef std::function<bool(int seqId, const std::vector<int>&, const std::vector<real>&)> DropCallback

DropCallback.

Drop a whole prefix or one candidate in beam search or not.

The first parameter is sequence index in a batch

The second parameter is one path in beam search, which is made up of node indices.

The third parameter is probabilites for each node in this path.

Return true if this prefix or candidate is expected to be dropped.

typedef std::function<void(int seqId, const std::vector<int>&, std::vector<real>&, real*)> NormOrDropNodeCallback

NormOrDropNodeCallback.

Normalize a path’s probabilities or just drop it by modifying path.logProb

The first parameter is sequence index in a batch

The second parameter is path.ids

The third parameter is probabilites for each node in this path.

The fourth parameter is the probability of the whole path.

typedef std::function<void(int)> EachStepCallback

EachStepCallback.

Invoke with beam search step.

Public Functions

RecurrentGradientMachine(const std::string &subModelName, NeuralNetwork *rootNetwork)

RecurrentGradientMachine(const RecurrentGradientMachine &other)

RecurrentGradientMachine &operator=(const RecurrentGradientMachine &other)

virtual ~RecurrentGradientMachine()

virtual void init(const ModelConfig &config, ParamInitCallback callback, const std::vector<ParameterType> &parameterTypes, bool useGpu)

virtual void prefetch(const std::vector<Argument> &inArgs)

Prefetch row ids of sparse parameter.

virtual void forward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType)

Forward propagation.

Calculate outputs (outArgs) based the inputs (inArgs)

Note

: if passType==PASS_TEST, then backward() should not be called

virtual void backward(const UpdateCallback &callback = nullptr)

Backward propagation.

Calculate the gradient of inArgs and parameter.

This function should only be called after a corresponding forward() call. The caller is responsible for filling the correct grad for the outArgs obtained using forward().

It may also change the grad field for the inArgs supplied at forward()

virtual void forwardBackward(const std::vector<Argument> &inArgs, std::vector<Argument> *outArgs, PassType passType, const UpdateCallback &callback)

Combine forward() and backward(). For multithread training, this may be faster.

Note

: passType PASS_TEST is not allowed for forwardBackward().

virtual void resetState()

virtual void eval(Evaluator *evaluator)

evaluate using the given evaluator

const std::vector<int> &getParameterIds()

void registerBeamSearchControlCallbacks(const BeamSearchCandidatesAdjustCallback &adjustBeamSearch, const NormOrDropNodeCallback &normOrDropNode, const DropCallback &stopBeamSearch)

Register beam search control callbacks. Used for prediction.

Parameters

• queryBeamSearch -

  Give the sequences already formed, return the nodes expected to be expanded. Input: A pointer to an array holding pathes which have been expanded Return: A pointer to an array holding nodes wanted to be expanded.

• dropOneNode -

  Early drop a node in one beam search step. Given the path formed and probability history, decide whether a node should be dropped or not.

• stopBeamSearch -

  Early stop a path in one beam search step. Given the path and probability history, decide whether a path should be dropped or not.

void removeBeamSearchControlCallbacks()

Remove user costumized beam search callbacks,.

make sequence generation acts like normal beam search.

void registerBeamSearchStatisticsCallbacks(const EachStepCallback &onEachStepStarted, const EachStepCallback &onEachStepStoped)

register statistics methods for performance profile of beam search.

Parameters

• onEachStepStarted -

  invoke once a beam search step starts. Its input is index of the beam search step.

• onEachStepStoped -

  invoke once a beam search step ends. Its input is index of the beam search step.

void removeBeamSearchStatisticsCallbacks()

Remove beam search callbacks.

void stopBeamSearch()

Stop beam search for current source.

Will restart beam search in the next forward

const std::vector<std::vector<Path>> &getFinalPaths() const

access beam search results.

Return

beam search results.

Protected Functions

void resizeOrCreateFrames(int numFrames)

void resizeBootFrame(int numSequences)

void generateSequence()

void oneWaySearch(size_t batchSize)

void beamSearch(size_t batchSize)

void createInFrameInfo(const Argument &input, PassType passType)

void createMemoryFrameInfo(MemoryFrameLine *memoryFrameLine, PassType passType)

void copyScattedId(std::vector<int> &srcIds, IVectorPtr *dstIds, int size)

void selectRowsOneTime(LayerPtr layer, const IVectorPtr &allIds, Argument *arg, PassType passType)

void createSeqPos(const std::vector<int> &sequenceStartPosition, ICpuGpuVectorPtr *sequenceStartPositions)

Protected Attributes

std::vector<InFrameLine> inFrameLines_

std::vector<OutFrameLine> outFrameLines_

std::vector<MemoryFrameLine> memoryFrameLines_

Info info_

std::vector<std::tuple<int, int, int, int>> seqLengthAndStart_

std::unique_ptr<EosFrameLine> eosFrameLine_

Generator generator_

std::vector<std::unique_ptr<NeuralNetwork>> frames_

NeuralNetwork *rootNetwork_

bool reversed_

int maxSequenceLength_

bool useGpu_

bool stopBeamSearch_

std::vector<int> parameterIds_

std::unique_ptr<Evaluator> evaluator_

std::vector<Argument> dataArgs_

std::vector<std::vector<Argument>> dataArgsFrame_

size_t dataArgsSize_

IVectorPtr cpuId_

MatrixPtr cpuProb_

IVectorPtr cpuEos_

Private Functions

size_t getBeamSize()

size_t getGenBatchSize()

void copyDataOutlinkFrame(size_t machineCur)

void createDataOutlink(std::vector<int> &machineIdVec)

void connectPrevFrame(int stepId, std::vector<Path> &paths)

void forwardFrame(int machineCur)

size_t beamShrink(std::vector<Path> &newPaths, size_t seqId, size_t totalExpandCount)

void singlePathExpand(Path &curPath, size_t curPathId, std::vector<Path> &newPaths, size_t expandWidth)

void beamExpand(std::vector<Path> &paths, std::vector<Path> &newPaths)

void fillGenOutputs()

Private Members

std::vector<int> machineIds_

std::vector<int> topIds_

std::vector<int> seqIds_

std::vector<int> batchMachineIdVec_

std::vector<std::vector<Path>> finalPaths_

std::vector<real> minFinalPathLogProb_

BeamSearchControlCallbacks *beamSearchCtrlCallbacks_

BeamSearchStatisticsCallbacks *beamSearchStatistics_

struct EosFrameLine

Public Members

std::vector<LayerPtr> layers

struct Generator

Public Members

GeneratorConfig config

std::vector<int> ids

Argument outArg

struct Info

Public Members

IVectorPtr allIds

std::vector<int> idIndex

ICpuGpuVectorPtr sequenceStartPositions

std::vector<int> seqStartPosIndex

struct InFrameLine

Public Members

std::string linkName

LayerPtr inLayer

std::vector<LayerPtr> agents

bool hasSubseq

Argument outArg

struct MemoryFrameLine

Public Members

std::string layerName

std::string linkName

LayerPtr bootLayer

LayerPtr biasLayer

LayerPtr rootLayer

LayerPtr rootAgent

std::vector<LayerPtr> frames

std::vector<LayerPtr> agents

std::vector<LayerPtr> scatterAgents

Argument outArg

bool is_sequence

IVectorPtr allIds

ICpuGpuVectorPtr sequenceStartPositions

struct OutFrameLine

Public Members

std::string layerName

LayerPtr agentLayer

std::vector<LayerPtr> frames

struct Path

Public Functions

Path()

Path default ctor, first logProb is 0.

Path(size_t seqId)

Path(Path &old, int newId, real logProb, int machineId, int topIndex)

Create a new path based on an old path and a new node with probability.

Parameters

• old -

  old path

• newId -

  index of the new node

• logProb -

  probability of the new node.

• machineId -

  sample index of a frame in RNN

• topIndex -

  index of MaxIdLayer output in one sample

bool operator<(const Path &other) const

operator <

Path a < Path b means log probability of a is smaller than that of b

void recordHistory()

Start recording history in this path.

void adjustProb(int calc_id, bool atEos = false)

Adjust probability for DIY beam search interface. In normal situation, it will do nothing.

Parameters

• calc_id -

  the object id for DIY beam search interface.

• atEos -

  at end of sequence or not.

bool isDropable() const

isDropable indacating whether the current node will be dropped or not in beam search.

Note

: if logProb is -inf, current node will be dropped.

Return

true to drop the current node.

Public Members

std::vector<int> ids

ids, path of beam search.

real logProb

logProb, current probability of path.

int machineId

int topIndex

int seqId

std::vector<int> machineIdVec

std::vector<real> probHistory

A record of each node’s probality in a formed path in beam search.

Note

It could be empty when history is not recorded. If the history is wanted to be recorded, recordHistory() MUST be invoked first.

Public Static Functions

static bool greaterPath(const Path &a, const Path &b)

Functions

P_DEFINE_string(diy_beam_search_prob_so, "", "the diy beam search cost so")

Variables

const char *DIY_CALC_PROB_SYMBOL_NAME

const char *DIY_START_CALC_PROB_SYMBOL_NAME

const char *DIY_FINISH_CALC_PROB_SYMBOL_NAME

namespace paddle

Typedefs

typedef int(* paddle::DiyStartCalcProbCallback) (size_t nNodes, int *nodes)

Start Custom Calculate Probability callback type.

Return

: A custom handler id that will passed to another callback.

Parameters

• nNodenodes -

  the path will be explored. nNodes is array size. nodes is array elements.

typedef real(* paddle::DiyCalcProbCallback) (int handler, size_t nNodes, int *nodes, real curProb, bool atEos)

Doing Custom Calculation of Probability callback type.

Return

: Log probability which user calculated, it will be updated to this path. : Return -INFINITY will DROP this path IMMEDIATELY!!

Parameters

• handler -

  User custom handler. The return value from start calc prob.

• nNodenodes -

  Array. The current path.

• curProb -

  The current log probability that neural network returns.

typedef void(* paddle::DiyStopCalcProbCallback) (int handler)

Finish Custom Calculation of Probability callback type.

Parameters

• handler -

  User custom handler. The return value from start calc prob.

Functions

static void exit_diy_prob()

template <typename SymbolType>

static SymbolType loadDiySymbol(const char *symbolName)

static InitFunction paddle::__init__diy_prob_method({std::string soName=FLAGS_diy_beam_search_prob_so;if(!soName.empty()){gDiyProbHandle=dlopen(soName.c_str(), RTLD_LAZY);CHECK(gDiyProbHandle)<< "Cannot Open DIY Prob So "<< soName;atexit(exit_diy_prob);gDiyProbMethod=loadDiySymbol< decltype(gDiyProbMethod)>(DIY_CALC_PROB_SYMBOL_NAME);gDiyProbStart=loadDiySymbol< decltype(gDiyProbStart)>(DIY_START_CALC_PROB_SYMBOL_NAME);gDiyProbStop=loadDiySymbol< decltype(gDiyProbStop)>(DIY_FINISH_CALC_PROB_SYMBOL_NAME);}}, std::numeric_limits< int >:: max())

Variables

DiyCalcProbCallback gDiyProbMethod

DiyStartCalcProbCallback gDiyProbStart

DiyStopCalcProbCallback gDiyProbStop

void *gDiyProbHandle

class BeamSearchControlCallbacks

Public Functions

BeamSearchControlCallbacks(const RecurrentGradientMachine::BeamSearchCandidatesAdjustCallback &candidateAdjust, const RecurrentGradientMachine::NormOrDropNodeCallback &norm, const RecurrentGradientMachine::DropCallback &stop)

for gcc46 aggregate initialization is not very well, so we need to explicit

Public Members

RecurrentGradientMachine::BeamSearchCandidatesAdjustCallback beamSearchCandidateAdjust

RecurrentGradientMachine::NormOrDropNodeCallback normOrDropNode

RecurrentGradientMachine::DropCallback stopDetermineCandidates

class BeamSearchStatisticsCallbacks

Public Functions

BeamSearchStatisticsCallbacks(const RecurrentGradientMachine::EachStepCallback &start, const RecurrentGradientMachine::EachStepCallback &stop)

Public Members

RecurrentGradientMachine::EachStepCallback onEachStepStarted

RecurrentGradientMachine::EachStepCallback onEachStepStoped

class BootBiasLayer

bias layer, as input of memory frame 0 will give vector of zeros if bias parameter is not set.

boot bias layer create directly in recurrent gradient machine, because:

1. It is only one frame, so it should not be placed in layer group, which is one instance for every one frame.
2. It is no input layer, so it need resetHeight() before forward(), and resetHeight() must be called in recurrent gradient machine, so it’s should not be placed in root network.

Inherits from paddle::Layer

Public Functions

BootBiasLayer(const LayerConfig &config)

virtual bool init(const LayerMap &layerMap, const ParameterMap &parameterMap)

Intialization. For example, adding input layers from layerMap and parameterMap.

void resetHeight(int height)

virtual void forward(PassType passType)

Forward propagation. All inherited implementation should call Layer::foward() function.

virtual void backward(const UpdateCallback &callback)

Backward propagation. Should only be called after Layer::forward() function.

Protected Attributes

std::unique_ptr<Weight> biases_

IVectorPtr cpuIds_