Optimizer

namespace paddle

class SgdOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

SgdOptimizer(const OptimizationConfig &optConfig)

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

virtual void finishBatch()

called by Trainer after backward() of a batch

class SparseMomentumParameterOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

SparseMomentumParameterOptimizer(const OptimizationConfig &optConfig)

void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

ParameterOptimizer::TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

void finishBatch()

called by Trainer after backward() of a batch

Protected Attributes

int64_t timer_

std::vector<int64_t> t0Vec_

bool isParameterSparse_

Private Members

real alpha_

real beta_

real tau_

real gamma_

real threshold_

real momentum_

real decayRate_

class AdagradParameterOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

AdagradParameterOptimizer(const OptimizationConfig &optConfig)

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

ParameterOptimizer::TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

Protected Attributes

int64_t numUpdates_

Protected Static Attributes

const int64_t kMaxNumAccumulates

class AdaDeltaParameterOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

AdaDeltaParameterOptimizer(const OptimizationConfig &optConfig)

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

Protected Attributes

real rou_

real epsilon_

class RMSPropParameterOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

RMSPropParameterOptimizer(const OptimizationConfig &optConfig)

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual void finishBatch()

called by Trainer after backward() of a batch

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

Protected Attributes

real rou_

real epsilon_

int64_t timer_

counting batches, donot need catch up with t(timer_) is current time, t0(t0Vec_) are last occur time of i rows. if one block is update by multi threads, caller should hash sparse ids to avoid write conflict in t0Vec_.

std::vector<int64_t> t0Vec_

class DecayedAdagradParameterOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

DecayedAdagradParameterOptimizer(const OptimizationConfig &optConfig)

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual void finishBatch()

called by Trainer after backward() of a batch

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

Protected Attributes

real rou_

real epsilon_

int64_t timer_

counting batches, donot need catch up with t(timer_) is current time, t0(t0Vec_) are last occur time of i rows. if one block is update by multi threads, caller should hash sparse ids to avoid write conflict in t0Vec_.

std::vector<int64_t> t0Vec_

class AdamParameterOptimizer

#include <FirstOrderOptimizer.h>

Adam Optimizer. Reference Paper: http://arxiv.org/abs/1412.6980 Algorithm 1

Inherits from paddle::ParameterOptimizer

Public Functions

AdamParameterOptimizer(const OptimizationConfig &optConfig)

virtual void finishBatch()

called by Trainer after backward() of a batch

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

Protected Attributes

real beta1_

real beta2_

real epsilon_

int64_t step_

real learningRate_

class AdamaxParameterOptimizer

#include <FirstOrderOptimizer.h>

AdaMax Optimizer. Reference Paper: http://arxiv.org/abs/1412.6980 Algorithm 2

Inherits from paddle::ParameterOptimizer

Public Functions

AdamaxParameterOptimizer(const OptimizationConfig &optConfig)

virtual void finishBatch()

called by Trainer after backward() of a batch

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

Protected Attributes

real beta1_

real beta2_

int64_t step_

real learningRate_

class AddOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

AddOptimizer(const OptimizationConfig &optConfig)

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

class DummyOptimizer

Inherits from paddle::ParameterOptimizer

Public Functions

DummyOptimizer(const OptimizationConfig &optConfig)

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

class OptimizerWithGradientClipping

Inherits from paddle::ParameterOptimizer

Public Functions

OptimizerWithGradientClipping(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer)

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void startPass()

virtual void finishPass()

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual void finishBatch()

called by Trainer after backward() of a batch

virtual TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

virtual void setNoDecay()

Protected Attributes

std::unique_ptr<ParameterOptimizer> optimizer_

namespace paddle

class AverageOptimizer

Inherits from paddle::ParameterOptimizer

Subclassed by paddle::AverageSparseOptimizer

Public Functions

AverageOptimizer(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer, bool useParameterApply)

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void startPass()

virtual void finishPass()

void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

void finishBatch()

called by Trainer after backward() of a batch

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

ParameterOptimizer::TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

virtual TraverseCallback startCatchUpWith() const

following hooks catch up with current time for sparse update, In the beginning, call startCatchUpWith() and check return. In the end, call finishCatchUpWith() to finish state. callback do the actual works, can call many times for sparse data. e.g. Trainer call like this:

auto callback = startCatchUpWith();
if (callback) {
  // do catch up with, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : rows_in_block) {callback();}
  }
  // finish catch up with, main thread
  finishCatchUpWith();
}

Return

callback if need catch up with, else return nullptr. It should be no state change.

virtual void finishCatchUpWith()

ParameterOptimizer::TraverseCallback apply()

following two hooks used by averager, apply to final parameter value (PARAMETER_VALUE or PARAMETER_APPLY).

restore() will restore orginal value if it apply to PARAMETER_VALUE. Caller must ensure it’s catched up with current time before apply.

Use returned callback same way as callback returned by ParameterOptimizer::needSpecialTraversal()

ParameterOptimizer::TraverseCallback restore()

virtual void setNoDecay()

Public Static Functions

ParameterOptimizer *create(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer, bool isParameterSparse = false, bool useParameterApply = false)

Protected Functions

void updateAverageWindowLimit()

bool isAverageWindowTooLong() const

Protected Attributes

std::unique_ptr<ParameterOptimizer> optimizer_

bool useApply_

int64_t numUpdates_

int64_t prevNumUpdates_

int64_t numAccumulates_

int64_t oldNumAccumulates_

int64_t minAverageWindow_

int64_t maxAverageWindow_

Protected Static Attributes

const int64_t kMaxNumAccumulates

class AverageSparseOptimizer

Inherits from paddle::AverageOptimizer

Public Functions

AverageSparseOptimizer(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer, bool useParameterApply)

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void finishBatch()

called by Trainer after backward() of a batch

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

void catchUpWith(const VectorPtr vecs[], const ParameterConfig &paraConfig, size_t sparseId) const

ParameterOptimizer::TraverseCallback startCatchUpWith() const

following hooks catch up with current time for sparse update, In the beginning, call startCatchUpWith() and check return. In the end, call finishCatchUpWith() to finish state. callback do the actual works, can call many times for sparse data. e.g. Trainer call like this:

auto callback = startCatchUpWith();
if (callback) {
  // do catch up with, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : rows_in_block) {callback();}
  }
  // finish catch up with, main thread
  finishCatchUpWith();
}

Return

callback if need catch up with, else return nullptr. It should be no state change.

virtual void finishCatchUpWith()

Protected Attributes

int timer_

counting batches, clear after catch up with t(timer_) is current time, t0(t0Vec_) are last occur time of i rows. if one block is update by multi threads, caller should hash sparse ids to avoid write conflict in t0Vec_.

std::vector<int32_t> t0Vec_

namespace paddle

class ParameterOptimizer

#include <ParameterOptimizer.h>

Some member functions are set to const for two reasons:

1. For sparse update thread safe: update(), traverse callback(const this) may be called many times, each time one row, and these function can be called parallelly by multi worker, to speed up large block.
2. For predicate functions, needSpecialTraversal(), startCatchUpWith() may be called many times, should be no state change between calls.

Subclassed by paddle::AdaDeltaParameterOptimizer, paddle::AdagradParameterOptimizer, paddle::AdamaxParameterOptimizer, paddle::AdamParameterOptimizer, paddle::AddOptimizer, paddle::AverageOptimizer, paddle::DecayedAdagradParameterOptimizer, paddle::DummyOptimizer, paddle::OptimizerWithGradientClipping, paddle::OptimizerWithRegularizer, paddle::RMSPropParameterOptimizer, paddle::SgdOptimizer, paddle::SparseMomentumParameterOptimizer

Public Types

typedef std::function<void(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId)> TraverseCallback

Public Functions

ParameterOptimizer(const OptimizationConfig &optConfig)

real calcLearningRate(int64_t numSamplesProcessed, int64_t pass)

virtual ~ParameterOptimizer()

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void startPass()

virtual void finishPass()

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

virtual void finishBatch()

called by Trainer after backward() of a batch

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId = -1LU) const = 0

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

virtual TraverseCallback startCatchUpWith() const

following hooks catch up with current time for sparse update, In the beginning, call startCatchUpWith() and check return. In the end, call finishCatchUpWith() to finish state. callback do the actual works, can call many times for sparse data. e.g. Trainer call like this:

auto callback = startCatchUpWith();
if (callback) {
  // do catch up with, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : rows_in_block) {callback();}
  }
  // finish catch up with, main thread
  finishCatchUpWith();
}

Return

callback if need catch up with, else return nullptr. It should be no state change.

virtual void finishCatchUpWith()

virtual TraverseCallback apply()

following two hooks used by averager, apply to final parameter value (PARAMETER_VALUE or PARAMETER_APPLY).

restore() will restore orginal value if it apply to PARAMETER_VALUE. Caller must ensure it’s catched up with current time before apply.

Use returned callback same way as callback returned by ParameterOptimizer::needSpecialTraversal()

virtual TraverseCallback restore()

const std::vector<ParameterType> &getParameterTypes() const

return the parameter types used by this updater

void addParameterType(ParameterType type)

real getLearningRate() const

virtual void setNoDecay()

Public Static Functions

ParameterOptimizer *create(const OptimizationConfig &optConfig, bool inPserver = false)

Protected Types

typedef std::vector<ParameterOptimizer::TraverseCallback> TraverseCallbackVec

Protected Attributes

bool applyDecay_

const OptimizationConfig &optConfig_

std::vector<ParameterType> parameterTypes_

real learningRate_

global learning rate, init value is opt_config.learning_rate, sparse regularizer get this value per batch, after StartBatch() called so, if lr change in StartBatch, please assign to learningRate_

std::unique_ptr<LearningRateScheduler> learningRateScheduler_

int64_t pass_

bool firstTime_

Protected Static Functions

static TraverseCallback composeCallbacks(const TraverseCallbackVec &callbacks)

namespace paddle

class OptimizerWithRegularizer

Inherits from paddle::ParameterOptimizer

Subclassed by paddle::OptimizerWithRegularizerEveryNumBatches, paddle::OptimizerWithRegularizerSparse

Public Functions

OptimizerWithRegularizer(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer, Regularizer *regularizer)

virtual void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

virtual void startPass()

virtual void finishPass()

virtual void startBatch(int64_t numSamplesProcessed)

called by Trainer before forward() of a batch.

virtual void finishBatch()

called by Trainer after backward() of a batch

virtual TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

Public Static Functions

ParameterOptimizer *create(const OptimizationConfig &optConfig, const ParameterConfig &paraConfig, bool isParameterSparse, bool inPserver)

Protected Attributes

std::unique_ptr<ParameterOptimizer> optimizer_

Regularizer *regularizer_

int timer_

counting batches, clear after catch up with t(timer_) is current time, t0(t0Vec_) are last occur time of i rows. if one block is update by multi threads, caller should hash sparse ids to avoid write conflict in t0Vec_.

class OptimizerWithRegularizerEveryNumBatches

Inherits from paddle::OptimizerWithRegularizer

Public Functions

OptimizerWithRegularizerEveryNumBatches(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer, Regularizer *regularizer)

virtual void startPass()

virtual void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

ParameterOptimizer::TraverseCallback needSpecialTraversal(const ParameterConfig &config) const

following hooks useful for sparse update, because the traversal in block costs. called by Trainer after update and before finishBatch e.g. Trainer call like this:

startBatch();
if (dense) {
  update(blockVec);
} else {//sparse
  for (row : rows_in_block) {update(rowVec)}
}
auto callback = needSpecialTraversal();
if (callback) {
  // do traverse, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : all_rows_in_block) {callback();}
  }
}
finishBatch();

Return

callback if need traverse, else return nullptr. It should be no state change.

void doTraversal(const VectorPtr vecs[], const ParameterConfig &config) const

void catchUpWith(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

ParameterOptimizer::TraverseCallback startCatchUpWith() const

following hooks catch up with current time for sparse update, In the beginning, call startCatchUpWith() and check return. In the end, call finishCatchUpWith() to finish state. callback do the actual works, can call many times for sparse data. e.g. Trainer call like this:

auto callback = startCatchUpWith();
if (callback) {
  // do catch up with, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : rows_in_block) {callback();}
  }
  // finish catch up with, main thread
  finishCatchUpWith();
}

Return

callback if need catch up with, else return nullptr. It should be no state change.

virtual void finishCatchUpWith()

Protected Functions

bool isRegularizationBatch(const ParameterConfig &config) const

Protected Attributes

int baseTimer_

recored the timer_ value while catchUpWith called.

class OptimizerWithRegularizerSparse

Inherits from paddle::OptimizerWithRegularizer

Public Functions

OptimizerWithRegularizerSparse(const OptimizationConfig &optConfig, ParameterOptimizer *optimizer, Regularizer *regularizer)

void init(size_t numRows, const ParameterConfig *config)

For sparse update, optimizer can maintain numRows of timer(t0). Some sparse optimizer depends on parameter config in functions such as startBatch(). Optimizer can get it here. But notice that, not all callers can pass config here, so the optimizer should check config passed in is not null ptr.

void update(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

between startBatch() and finishBatch(), update() will be called by the trainer multiple times, each time for updating one Parameter with its gradient in PARAMETER_GRADIENT. sparseId is row id, when sparseId set, update is sparse, each time one row.

void catchUpWith(const VectorPtr vecs[], const ParameterConfig &config, size_t sparseId) const

ParameterOptimizer::TraverseCallback startCatchUpWith() const

following hooks catch up with current time for sparse update, In the beginning, call startCatchUpWith() and check return. In the end, call finishCatchUpWith() to finish state. callback do the actual works, can call many times for sparse data. e.g. Trainer call like this:

auto callback = startCatchUpWith();
if (callback) {
  // do catch up with, maybe multi-thread
  if (dense) {
    callback();
  } else {//sparse
    for (row : rows_in_block) {callback();}
  }
  // finish catch up with, main thread
  finishCatchUpWith();
}

Return

callback if need catch up with, else return nullptr. It should be no state change.

virtual void finishCatchUpWith()

Protected Attributes

std::vector<int32_t> t0Vec_

t0Vec_ are last occur time of i rows if one block is update by multi threads, caller should hash sparse ids to avoid write conflict in t0Vec_.