Base Evaluator

Evaluator

class paddle::Evaluator

Subclassed by paddle::AucEvaluator, paddle::ChunkEvaluator, paddle::ClassificationErrorEvaluator, paddle::ColumnSumEvaluator, paddle::CombinedEvaluator, paddle::CTCErrorEvaluator, paddle::DummyEvaluator, paddle::GradientPrinter, paddle::MaxFramePrinter, paddle::MaxIdPrinter, paddle::MultiCombinedEvaluator, paddle::PnpairEvaluator, paddle::PrecisionRecallEvaluator, paddle::RankAucEvaluator, paddle::SequenceTextPrinter, paddle::SumEvaluator, paddle::ValuePrinter

Public Functions

Evaluator()
virtual ~Evaluator()
virtual void init(const EvaluatorConfig &config)
virtual void start()

start to evaluate some data

virtual void eval(const NeuralNetwork &nn)

Process a batch of data.

virtual real evalImp(std::vector<Argument> &arguments) = 0

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

void mergeResultsOfAllClients(ParameterClient2 *client)
virtual void finish()

finish the evaluation.

virtual void printStats(std::ostream &os)

finish() should be called before printStats

Public Static Functions

Evaluator *create(const EvaluatorConfig &config)

Public Static Attributes

ClassRegistrar<Evaluator> registrar_

Protected Attributes

EvaluatorConfig config_
double numSamples_
double totalScore_

Friends

std::ostream &operator<<(std::ostream &os, Evaluator &evaluator)
std::ostream &&operator<<(std::ostream &&os, Evaluator &evaluator)

Utils

SumEvaluator

class paddle::SumEvaluator

Inherits from paddle::Evaluator

Public Functions

SumEvaluator()
virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

ColumnSumEvaluator

class paddle::ColumnSumEvaluator

Inherits from paddle::Evaluator

Public Functions

ColumnSumEvaluator(int32_t colIdx)
virtual void start()

start to evaluate some data

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void printStats(std::ostream &os)

finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

Classification

ClassificationErrorEvaluator

class paddle::ClassificationErrorEvaluator

Inherits from paddle::Evaluator

Subclassed by paddle::ClassificationErrorPrinter, paddle::SequenceClassificationErrorEvaluator

Public Functions

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

MatrixPtr calcError(std::vector<Argument> &arguments)
virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

SequenceClassificationErrorEvaluator

class paddle::SequenceClassificationErrorEvaluator

Inherits from paddle::ClassificationErrorEvaluator

Public Functions

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

AucEvaluator

class paddle::AucEvaluator

Inherits from paddle::Evaluator

Public Functions

AucEvaluator(int32_t colIdx)

evaluate AUC using colIdx-th column as prediction.

  • colIdx = 0: the 0-th column.
  • colIdx > 0: the colIdx-th column.
  • colIdx < 0: the last colIdx-th column.

virtual void start()

start to evaluate some data

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void printStats(std::ostream &os)

finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

PrecisionRecallEvaluator

class paddle::PrecisionRecallEvaluator

Inherits from paddle::Evaluator

Public Functions

PrecisionRecallEvaluator()
virtual void start()

start to evaluate some data

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void printStats(std::ostream &os)

finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

struct StatsInfo

Public Functions

StatsInfo()

Public Members

double TP

numbers of true positives

double TN

numbers of true negatives

double FP

numbers of false positives

double FN

numbers of false negatives

ChunkEvaluator

class paddle::ChunkEvaluator

Chunk evaluator is used to evaluate segment labelling accuracy for a sequence. It calculates the chunk detection F1 score.

A chunk is correctly detected if its beginning, end and type are correct. Other chunk type is ignored. For each label in the label sequence, we have

tagType = label % numTagType
chunkType = label / numTagType
otherChunkType = numChunkTypes

The total number of different labels is numTagType*numChunkTypes+1 We support 4 labelling scheme The tag type for each of the scheme is shown as follows:

Scheme Begin Inside End   Single
 plain  0     -      -     -
 IOB    0     1      -     -
 IOE    -     0      1     -
 IOBES  0     1      2     3

‘plain’ means the whole chunk must contain exactly the same chunk label.

Inherits from paddle::Evaluator

Public Functions

virtual void init(const EvaluatorConfig &config)
virtual void start()

start to evaluate some data

virtual void printStats(std::ostream &os)

finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

void eval1(int *output, int *label, int length)
void getSegments(int *label, int length, std::vector<Segment> &segments)
bool isChunkEnd(int prevTag, int prevType, int tag, int type)
bool isChunkBegin(int prevTag, int prevType, int tag, int type)

CTCEvaluator

class paddle::CTCErrorEvaluator

calculate sequence-to-sequence edit distance

Inherits from paddle::Evaluator

Public Functions

CTCErrorEvaluator()
virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void eval(const NeuralNetwork &nn)

Process a batch of data.

virtual void start()

start to evaluate some data

virtual void printStats(std::ostream &os)

finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

Rank

PnpairEvaluator

class paddle::PnpairEvaluator

Positive-negative pair rate Evaluator

Inherits from paddle::Evaluator

Public Functions

PnpairEvaluator()
virtual void start()

start to evaluate some data

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

void printPredictResults()
void stat(size_t start, size_t end, PredictionResult *answers, double &pos, double &neg, double &spe)
void calc(std::vector<PredictionResult> &predictArray)
virtual void finish()

finish the evaluation.

virtual void printStats(std::ostream &os)

finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

Public Members

std::vector<PredictionResult> predictArray_
struct PredictionResult

Public Functions

PredictionResult(real __out, int __label, int __queryid, real __weight)

Public Members

real out
int label
int queryid
real weight

AucEvaluator

class paddle::RankAucEvaluator

RankAucEvaluator calculates the AUC of each list (i.e., titles under the same query), and averages them.

Each list should be organized as a sequence. The inputs of this evaluator is [output, click, pv]. If pv is not provided, it will be set to 1. The types of click and pv are dense value.

Inherits from paddle::Evaluator

Public Functions

virtual void start()

start to evaluate some data

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)

finish() should be called before distributeEval

Printer

ValuePrinter

class paddle::ValuePrinter

Inherits from paddle::Evaluator

Public Functions

ValuePrinter()
virtual void eval(const NeuralNetwork &nn)

Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

GradientPrinter

class paddle::GradientPrinter

Inherits from paddle::Evaluator

Public Functions

GradientPrinter()
virtual void eval(const NeuralNetwork &nn)

Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

MaxIdPrinter

class paddle::MaxIdPrinter

Inherits from paddle::Evaluator

Public Functions

MaxIdPrinter()
virtual void eval(const NeuralNetwork &nn)

Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

MaxFramePrinter

class paddle::MaxFramePrinter

Inherits from paddle::Evaluator

Public Functions

MaxFramePrinter()
virtual void eval(const NeuralNetwork &nn)

Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

SequenceTextPrinter

class paddle::SequenceTextPrinter

Sequence text printer will print text according to index matrix and a dictionary. There can be multiple input to this layer:

1) If there is only one input, the input must be a matrix containing the sequence of indices;

2) If there are more than one input, the first input should be ids, and are interpreted as sample ids.

The output format will be:

1) sequence without sub-sequence, and there is probability.

id \t prob space_seperated_tokens_from_dictionary_according_to_seq

2) sequence without sub-sequence, and there is not probability.

id \t space_seperated_tokens_from_dictionary_according_to_seq

3) sequence with sub-sequence, and there is not probability.

id \t space_seperated_tokens_from_dictionary_according_to_sub_seq
\t \t space_seperated_tokens_from_dictionary_according_to_sub_seq
...

Typically SequenceTextPrinter layer takes output of maxid or RecurrentGroup with maxid (when generating) as an input.

Inherits from paddle::Evaluator

Public Functions

SequenceTextPrinter()
virtual void init(const EvaluatorConfig &config)
virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

ClassificationErrorPrinter

class paddle::ClassificationErrorPrinter

Inherits from paddle::ClassificationErrorEvaluator

Public Functions

virtual void updateSamplesNum(const std::vector<Argument> &arguments)

Update the number of processed samples

virtual real evalImp(std::vector<Argument> &arguments)

Process a batch of data. return the score for the batch if it make sense to sum the score across batches. Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.