# Base Evaluator

## Evaluator

class `paddle::``Evaluator`

  Base class for Evaluator Evaluating the performance of a model is very important. It indicates how successful the scores(predictions) of a datasets has been by a trained model.

  Subclassed by paddle::AucEvaluator, paddle::ChunkEvaluator, paddle::ClassificationErrorEvaluator, paddle::ColumnSumEvaluator, paddle::CombinedEvaluator, paddle::CTCErrorEvaluator, paddle::DummyEvaluator, paddle::GradientPrinter, paddle::MaxFramePrinter, paddle::MaxIdPrinter, paddle::MultiCombinedEvaluator, paddle::PnpairEvaluator, paddle::PrecisionRecallEvaluator, paddle::RankAucEvaluator, paddle::SequenceTextPrinter, paddle::SumEvaluator, paddle::ValuePrinter

  Public Functions

  `Evaluator`()
  virtual `~Evaluator`()
  virtual void `init`(const EvaluatorConfig &config)
  virtual void `start`()

    start to evaluate some data

  void `eval`(const NeuralNetwork &nn)

    Process a batch of data.

  virtual real `evalImp`(std::vector<Argument> &arguments) = 0

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

  void `mergeResultsOfAllClients`(ParameterClient2 *client)
  virtual void `finish`()

    finish the evaluation.

  virtual void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats

  Public Static Functions

  Evaluator *`create`(const EvaluatorConfig &config)

  Public Static Attributes

  ClassRegistrar<Evaluator> `registrar_`

  Protected Attributes

  EvaluatorConfig `config_`
  double `numSamples_`
  double `totalScore_`

  Friends

  std::ostream &`operator<<`(std::ostream &os, Evaluator &evaluator)
  std::ostream &&`operator<<`(std::ostream &&os, Evaluator &evaluator)

# Utils

## SumEvaluator

class `paddle::``SumEvaluator`

  sum Evaluator Calculate the sum of output or label

  The config file api is sum_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `SumEvaluator`()
  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

## ColumnSumEvaluator

class `paddle::``ColumnSumEvaluator`

  column sum Evaluator

  The config file api is column_sum_evaluator.

  Note
    column sum for the colIdx-th column *
    - colIdx = 0: the 0-th column.
    - colIdx > 0: the colIdx-th column.
    - colIdx < 0: the last colIdx-th column.

  Inherits from paddle::Evaluator

  Public Functions

  `ColumnSumEvaluator`(int32_t colIdx)
  virtual void `start`()

    start to evaluate some data

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats
  void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

# Classification

## ClassificationErrorEvaluator

class `paddle::``ClassificationErrorEvaluator`

  classification error Evaluator

  The config file api is classification_error_evaluator.

  Inherits from paddle::Evaluator

  Subclassed by paddle::ClassificationErrorPrinter, paddle::SequenceClassificationErrorEvaluator

  Public Functions

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  MatrixPtr `calcError`(std::vector<Argument> &arguments)
  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

## SequenceClassificationErrorEvaluator

class `paddle::``SequenceClassificationErrorEvaluator`

  sequence classification error Evaluator

  Note
    sequence level classification error stats, if any frame in one sequence has error, the sequence is error

  Inherits from paddle::ClassificationErrorEvaluator

  Public Functions

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

## AucEvaluator

class `paddle::``AucEvaluator`

  evaluate AUC using colIdx-th column as prediction. The AUC(Area Under the Curve) is a common evaluation metric for binary classification problems. It computes the area under the receiver operating characteristic(ROC) curve.

  - colIdx = 0: the 0-th column.
  - colIdx > 0: the colIdx-th column.
  - colIdx < 0: the last colIdx-th column.
  Note
    colIdx-th column

  The config file api is auc_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `AucEvaluator`(int32_t colIdx)
  void `start`()

    start to evaluate some data

  real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats
  void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

## PrecisionRecallEvaluator

class `paddle::``PrecisionRecallEvaluator`

  precision, recall and f1 score Evaluator

  \[\begin{split} precision = \frac{tp}{tp+tn} \\ recall=\frac{tp}{tp+fn} \\ f1=2*\frac{precsion*recall}{precision+recall} \end{split}\]

  The config file api is precision_recall_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `PrecisionRecallEvaluator`()
  void `start`()

    start to evaluate some data

  real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats
  void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

  struct `StatsInfo`

    Public Functions

    `StatsInfo`()

    Public Members

    double `TP`

      numbers of true positives

    double `TN`

      numbers of true negatives

    double `FP`

      numbers of false positives

    double `FN`

      numbers of false negatives

## ChunkEvaluator

class `paddle::``ChunkEvaluator`

  Chunk evaluator is used to evaluate segment labelling accuracy for a sequence. It calculates the chunk detection F1 score.

  A chunk is correctly detected if its beginning, end and type are correct. Other chunk type is ignored. For each label in the label sequence, we have

      tagType = label % numTagType
      chunkType = label / numTagType
      otherChunkType = numChunkTypes

  The total number of different labels is numTagType*numChunkTypes+1 We support 4 labelling scheme The tag type for each of the scheme is shown as follows:

      Scheme Begin Inside End   Single
       plain  0     -      -     -
       IOB    0     1      -     -
       IOE    -     0      1     -
       IOBES  0     1      2     3

  ‘plain’ means the whole chunk must contain exactly the same chunk label.

  Inherits from paddle::Evaluator

  Public Functions

  virtual void `init`(const EvaluatorConfig &config)
  virtual void `start`()

    start to evaluate some data

  virtual void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  void `eval1`(int *output, int *label, int length)
  void `getSegments`(int *label, int length, std::vector<Segment> &segments)
  bool `isChunkEnd`(int prevTag, int prevType, int tag, int type)
  bool `isChunkBegin`(int prevTag, int prevType, int tag, int type)

## CTCEvaluator

class `paddle::``CTCErrorEvaluator`

  calculate sequence-to-sequence edit distance

  Inherits from paddle::Evaluator

  Public Functions

  `CTCErrorEvaluator`()
  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `eval`(const NeuralNetwork &nn)

    Process a batch of data.

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual void `start`()

    start to evaluate some data

  virtual void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

# Rank

## PnpairEvaluator

class `paddle::``PnpairEvaluator`

  Inherits from paddle::Evaluator

  Public Functions

  `PnpairEvaluator`()
  void `start`()

    start to evaluate some data

  real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  void `printPredictResults`()
  void `stat`(size_t start, size_t end, PredictionResult *answers, double &pos, double &neg, double &spe)
  void `calc`(std::vector<PredictionResult> &predictArray)
  virtual void `finish`()

    finish the evaluation.

  virtual void `printStats`(std::ostream &os)

    print the statistics of evaluate result

    Note
      finish() should be called before printStats
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

  Public Members

  std::vector<PredictionResult> `predictArray_`
  struct `PredictionResult`

    Public Functions

    `PredictionResult`(real __out, int __label, int __queryid, real __weight)

    Public Members

    real `out`
    int `label`
    int `queryid`
    real `weight`

## AucEvaluator

class `paddle::``RankAucEvaluator`

  RankAucEvaluator calculates the AUC of each list (i.e., titles under the same query), and averages them. Each list should be organized as a sequence. The inputs of this evaluator is [output, click, pv]. If pv is not provided, it will be set to 1. The types of click and pv are dense value.

  Inherits from paddle::Evaluator

  Public Functions

  void `start`()

    start to evaluate some data

  void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.
  virtual void `distributeEval`(ParameterClient2 *client)

    finish() should be called before distributeEval

# Printer

## ValuePrinter

class `paddle::``ValuePrinter`

  print value of each layer.

  The config file api is value_printer_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `ValuePrinter`()
  virtual void `eval`(const NeuralNetwork &nn)

    Process a batch of data.

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

## GradientPrinter

class `paddle::``GradientPrinter`

  print gradient of each layer.

  The config file api is gradient_printer_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `GradientPrinter`()
  virtual void `eval`(const NeuralNetwork &nn)

    Process a batch of data.

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

## MaxIdPrinter

class `paddle::``MaxIdPrinter`

  print row max id vctor of each layer

  The config file api is maxid_printer_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `MaxIdPrinter`()
  virtual void `eval`(const NeuralNetwork &nn)

    Process a batch of data.

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

## MaxFramePrinter

class `paddle::``MaxFramePrinter`

  print sequence max frames of each layer

  The config file api is maxframe_printer_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `MaxFramePrinter`()
  virtual void `eval`(const NeuralNetwork &nn)

    Process a batch of data.

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

## SequenceTextPrinter

class `paddle::``SequenceTextPrinter`

  print text according to index matrix and a dictionary.

  There can be multiple input to this layer:

  - If there is only one input, the input must be a matrix containing the sequence of indices;
  - If there are more than one input, the first input should be ids, and are interpreted as sample ids.

  The output format will be:

  - sequence without sub-sequence, and there is probability.

      id \t prob space_seperated_tokens_from_dictionary_according_to_seq

  - sequence without sub-sequence, and there is not probability.

      id \t space_seperated_tokens_from_dictionary_according_to_seq

  - sequence with sub-sequence, and there is not probability.

      id \t space_seperated_tokens_from_dictionary_according_to_sub_seq
      \t \t space_seperated_tokens_from_dictionary_according_to_sub_seq
      ...

  Typically SequenceTextPrinter layer takes output of maxid or RecurrentGroup with maxid (when generating) as an input.

  The config file api is seqtext_printer_evaluator.

  Inherits from paddle::Evaluator

  Public Functions

  `SequenceTextPrinter`()
  virtual void `init`(const EvaluatorConfig &config)
  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.

## ClassificationErrorPrinter

class `paddle::``ClassificationErrorPrinter`

  print classification error.

  The config file api is classification_error_printer_evaluator.

  Inherits from paddle::ClassificationErrorEvaluator

  Public Functions

  virtual void `updateSamplesNum`(const std::vector<Argument> &arguments)

    Update the number of processed samples.

  virtual real `evalImp`(std::vector<Argument> &arguments)

    Process a batch of data.

    Return
      the score for the batch if it make sense to sum the score across batches.
    Note
      Otherwise evaluator should return 0 and override finish() and printStats() to do the right calculation.