Base Evaluator¶

Evaluator¶

class paddle::Evaluator¶

Base class for Evaluator Evaluating the performance of a model is very important. It indicates how successful the scores(predictions) of a datasets has been by a trained model.

Subclassed by paddle::AucEvaluator, paddle::ChunkEvaluator, paddle::ClassificationErrorEvaluator, paddle::ColumnSumEvaluator, paddle::CombinedEvaluator, paddle::CTCErrorEvaluator, paddle::DummyEvaluator, paddle::GradientPrinter, paddle::MaxFramePrinter, paddle::MaxIdPrinter, paddle::MultiCombinedEvaluator, paddle::PnpairEvaluator, paddle::PrecisionRecallEvaluator, paddle::RankAucEvaluator, paddle::SequenceTextPrinter, paddle::SumEvaluator, paddle::ValuePrinter

Public Functions

Evaluator()¶

virtual ~Evaluator()¶

virtual void init(const EvaluatorConfig &config)¶

virtual void start()¶: start to evaluate some data

virtual void eval(const NeuralNetwork &nn)¶: Process a batch of data.

virtual real evalImp(std::vector<Argument> &arguments) = 0¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

void mergeResultsOfAllClients(ParameterClient2 *client)¶

virtual void finish()¶: finish the evaluation.

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

Public Static Functions

Evaluator *create(const EvaluatorConfig &config)¶

Public Static Attributes

ClassRegistrar<Evaluator> registrar_¶

Protected Attributes

EvaluatorConfig config_¶

double numSamples_¶

double totalScore_¶

Friends

std::ostream &operator<<(std::ostream &os, Evaluator &evaluator)¶

std::ostream &&operator<<(std::ostream &&os, Evaluator &evaluator)¶

Utils¶

SumEvaluator¶

class paddle::SumEvaluator¶

sum Evaluator Calculate the sum of output or label

The config file api is sum_evaluator.

Inherits from paddle::Evaluator

Public Functions

SumEvaluator()¶

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

ColumnSumEvaluator¶

class paddle::ColumnSumEvaluator¶

column sum Evaluator

The config file api is column_sum_evaluator.

Note

column sum for the colIdx-th column *

colIdx = 0: the 0-th column.
colIdx > 0: the colIdx-th column.
colIdx < 0: the last colIdx-th column.

Inherits from paddle::Evaluator

Public Functions

ColumnSumEvaluator(int32_t colIdx)¶

virtual void start()¶: start to evaluate some data

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

Classification¶

ClassificationErrorEvaluator¶

class paddle::ClassificationErrorEvaluator¶

classification error Evaluator

The config file api is classification_error_evaluator.

Inherits from paddle::Evaluator

Subclassed by paddle::ClassificationErrorPrinter, paddle::SequenceClassificationErrorEvaluator

Public Functions

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

MatrixPtr calcError(std::vector<Argument> &arguments)¶

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

SequenceClassificationErrorEvaluator¶

class paddle::SequenceClassificationErrorEvaluator¶

sequence classification error Evaluator

Note: sequence level classification error stats, if any frame in one sequence has error, the sequence is error

Inherits from paddle::ClassificationErrorEvaluator

Public Functions

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

AucEvaluator¶

class paddle::AucEvaluator¶

evaluate AUC using colIdx-th column as prediction. The AUC(Area Under the Curve) is a common evaluation metric for binary classification problems. It computes the area under the receiver operating characteristic(ROC) curve.

colIdx = 0: the 0-th column.
colIdx > 0: the colIdx-th column.
colIdx < 0: the last colIdx-th column.

Note: colIdx-th column

The config file api is auc_evaluator.

Inherits from paddle::Evaluator

Public Functions

AucEvaluator(int32_t colIdx)¶

virtual void start()¶: start to evaluate some data

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

PrecisionRecallEvaluator¶

class paddle::PrecisionRecallEvaluator¶

precision, recall and f1 score Evaluator

\[\begin{split} precision = \frac{tp}{tp+tn} \\ recall=\frac{tp}{tp+fn} \\ f1=2*\frac{precsion*recall}{precision+recall} \end{split}\]

The config file api is precision_recall_evaluator.

Inherits from paddle::Evaluator

Public Functions

PrecisionRecallEvaluator()¶

virtual void start()¶: start to evaluate some data

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

struct StatsInfo¶

Public Functions

StatsInfo()¶

Public Members

double TP¶: numbers of true positives

double TN¶: numbers of true negatives

double FP¶: numbers of false positives

double FN¶: numbers of false negatives

ChunkEvaluator¶

class paddle::ChunkEvaluator¶

Chunk evaluator is used to evaluate segment labelling accuracy for a sequence. It calculates the chunk detection F1 score.

A chunk is correctly detected if its beginning, end and type are correct. Other chunk type is ignored. For each label in the label sequence, we have

tagType = label % numTagType
chunkType = label / numTagType
otherChunkType = numChunkTypes

The total number of different labels is numTagType*numChunkTypes+1 We support 4 labelling scheme The tag type for each of the scheme is shown as follows:

Scheme Begin Inside End   Single
 plain  0     -      -     -
 IOB    0     1      -     -
 IOE    -     0      1     -
 IOBES  0     1      2     3

‘plain’ means the whole chunk must contain exactly the same chunk label.

Inherits from paddle::Evaluator

Public Functions

virtual void init(const EvaluatorConfig &config)¶

virtual void start()¶: start to evaluate some data

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

void eval1(int *output, int *label, int length)¶

void getSegments(int *label, int length, std::vector<Segment> &segments)¶

bool isChunkEnd(int prevTag, int prevType, int tag, int type)¶

bool isChunkBegin(int prevTag, int prevType, int tag, int type)¶

CTCEvaluator¶

class paddle::CTCErrorEvaluator¶

calculate sequence-to-sequence edit distance

Inherits from paddle::Evaluator

Public Functions

CTCErrorEvaluator()¶

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void eval(const NeuralNetwork &nn)¶: Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual void start()¶: start to evaluate some data

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

Rank¶

PnpairEvaluator¶

class paddle::PnpairEvaluator¶

Inherits from paddle::Evaluator

Public Functions

PnpairEvaluator()¶

virtual void start()¶: start to evaluate some data

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

void printPredictResults()¶

void stat(size_t start, size_t end, PredictionResult *answers, double &pos, double &neg, double &spe)¶

void calc(std::vector<PredictionResult> &predictArray)¶

virtual void finish()¶: finish the evaluation.

virtual void printStats(std::ostream &os)¶

print the statistics of evaluate result

Note: finish() should be called before printStats

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

Public Members

std::vector<PredictionResult> predictArray_¶

struct PredictionResult¶

Public Functions

PredictionResult(real __out, int __label, int __queryid, real __weight)¶

Public Members

real out¶

int label¶

int queryid¶

real weight¶

AucEvaluator¶

class paddle::RankAucEvaluator¶

RankAucEvaluator calculates the AUC of each list (i.e., titles under the same query), and averages them. Each list should be organized as a sequence. The inputs of this evaluator is [output, click, pv]. If pv is not provided, it will be set to 1. The types of click and pv are dense value.

Inherits from paddle::Evaluator

Public Functions

virtual void start()¶: start to evaluate some data

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

virtual void distributeEval(ParameterClient2 *client)¶: finish() should be called before distributeEval

Printer¶

ValuePrinter¶

class paddle::ValuePrinter¶

print value of each layer.

The config file api is value_printer_evaluator.

Inherits from paddle::Evaluator

Public Functions

ValuePrinter()¶

virtual void eval(const NeuralNetwork &nn)¶: Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

GradientPrinter¶

class paddle::GradientPrinter¶

print gradient of each layer.

The config file api is gradient_printer_evaluator.

Inherits from paddle::Evaluator

Public Functions

GradientPrinter()¶

virtual void eval(const NeuralNetwork &nn)¶: Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

MaxIdPrinter¶

class paddle::MaxIdPrinter¶

print row max id vctor of each layer

The config file api is maxid_printer_evaluator.

Inherits from paddle::Evaluator

Public Functions

MaxIdPrinter()¶

virtual void eval(const NeuralNetwork &nn)¶: Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

MaxFramePrinter¶

class paddle::MaxFramePrinter¶

print sequence max frames of each layer

The config file api is maxframe_printer_evaluator.

Inherits from paddle::Evaluator

Public Functions

MaxFramePrinter()¶

virtual void eval(const NeuralNetwork &nn)¶: Process a batch of data.

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

SequenceTextPrinter¶

class paddle::SequenceTextPrinter¶

print text according to index matrix and a dictionary.

There can be multiple input to this layer:

If there is only one input, the input must be a matrix containing the sequence of indices;
If there are more than one input, the first input should be ids, and are interpreted as sample ids.

The output format will be:

sequence without sub-sequence, and there is probability.

id \t prob space_seperated_tokens_from_dictionary_according_to_seq

sequence without sub-sequence, and there is not probability.

id \t space_seperated_tokens_from_dictionary_according_to_seq

sequence with sub-sequence, and there is not probability.

id \t space_seperated_tokens_from_dictionary_according_to_sub_seq
\t \t space_seperated_tokens_from_dictionary_according_to_sub_seq
...

Typically SequenceTextPrinter layer takes output of maxid or RecurrentGroup with maxid (when generating) as an input.

The config file api is seqtext_printer_evaluator.

Inherits from paddle::Evaluator

Public Functions

SequenceTextPrinter()¶

virtual void init(const EvaluatorConfig &config)¶

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

ClassificationErrorPrinter¶

class paddle::ClassificationErrorPrinter¶

print classification error.

The config file api is classification_error_printer_evaluator.

Inherits from paddle::ClassificationErrorEvaluator

Public Functions

virtual void updateSamplesNum(const std::vector<Argument> &arguments)¶: Update the number of processed samples.

virtual real evalImp(std::vector<Argument> &arguments)¶

Process a batch of data.

Return: the score for the batch if it make sense to sum the score across batches.
Note: Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

Base Evaluator¶

Evaluator¶

Utils¶

SumEvaluator¶

ColumnSumEvaluator¶

Classification¶

ClassificationErrorEvaluator¶

SequenceClassificationErrorEvaluator¶

AucEvaluator¶

PrecisionRecallEvaluator¶

ChunkEvaluator¶

CTCEvaluator¶

Rank¶

PnpairEvaluator¶

AucEvaluator¶

Printer¶

ValuePrinter¶

GradientPrinter¶

MaxIdPrinter¶

MaxFramePrinter¶

SequenceTextPrinter¶

ClassificationErrorPrinter¶

Table Of Contents

Previous topic

Next topic

This Page