Evaluators¶
Evaluator will evaluate the network status while training/testing.
User can use evaluator by classify/regression job. For example.
classify(prediction, output, evaluator=classification_error_evaluator)
And user could define evaluator separately as follow.
classification_error_evaluator("ErrorRate", prediction, label)
The evaluator often contains a name parameter. It will also be printed when evaluating network. The printed information may look like the following.
Batch=200 samples=20000 AvgCost=0.679655 CurrentCost=0.662179 Eval:
classification_error_evaluator=0.4486
CurrentEval: ErrorRate=0.3964
-
paddle.trainer_config_helpers.evaluators.classification_error_evaluator(*args, **kwargs)¶ Classification Error Evaluator. It will print error rate for classification.
The classification error is:
\[classification\_error = \frac{NumOfWrongPredicts}{NumOfAllSamples}\]The simple usage is:
eval = classification_error_evaluator(input=prob,label=lbl)
Parameters: - name (basestring) – Evaluator name.
- input (LayerOutput) – Input Layer name. The output prediction of network.
- label (basestring) – Label layer name.
- weight (LayerOutput) – Weight Layer name. It should be a matrix with size [sample_num, 1]. And will just multiply to NumOfWrongPredicts and NumOfAllSamples. So, the elements of weight are all one, then means not set weight. The larger weight it is, the more important this sample is.
- threshold (float) – The classification threshold.
Returns: None.
-
paddle.trainer_config_helpers.evaluators.auc_evaluator(*args, **kwargs)¶ Auc Evaluator which adapts to binary classification.
The simple usage:
eval = auc_evaluator(input, label)
Parameters: - name (None|basestring) – Evaluator name.
- input (LayerOutput) – Input Layer name. The output prediction of network.
- label (None|basestring) – Label layer name.
- weight (LayerOutput) – Weight Layer name. It should be a matrix with size [sample_num, 1].
-
paddle.trainer_config_helpers.evaluators.pnpair_evaluator(*args, **kwargs)¶ Positive-negative pair rate Evaluator which adapts to rank task like learning to rank. This evaluator must contain at least three layers.
The simple usage:
eval = pnpair_evaluator(input, info, label)
Parameters: - name (None|basestring) – Evaluator name.
- input (LayerOutput) – Input Layer name. The output prediction of network.
- label (LayerOutput) – Label layer name.
- info (LayerOutput) – Label layer name. (TODO, explaination)
- weight (LayerOutput) – Weight Layer name. It should be a matrix with size [sample_num, 1]. (TODO, explaination)
-
paddle.trainer_config_helpers.evaluators.precision_recall_evaluator(*args, **kwargs)¶ An Evaluator to calculate precision and recall, F1-score. It is adapt to the task with multiple labels.
- If positive_label=-1, it will print the average precision, recall, F1-score of all labels.
- If use specify positive_label, it will print the precision, recall, F1-score of this label.
The simple usage:
eval = precision_recall_evaluator(input, label)
Parameters: - name (None|basestring) – Evaluator name.
- input (LayerOutput) – Input Layer name. The output prediction of network.
- label (LayerOutput) – Label layer name.
- positive_label (LayerOutput.) – The input label layer.
- weight (LayerOutput) – Weight Layer name. It should be a matrix with size [sample_num, 1]. (TODO, explaination)
-
paddle.trainer_config_helpers.evaluators.ctc_error_evaluator(*args, **kwargs)¶ This evaluator is to calculate sequence-to-sequence edit distance.
The simple usage is :
eval = ctc_error_evaluator(input)
Parameters: - name (None|basestring) – Evaluator name.
- input (LayerOutput) – Input Layer.
-
paddle.trainer_config_helpers.evaluators.chunk_evaluator(*args, **kwargs)¶ Chunk evaluator is used to evaluate segment labelling accuracy for a sequence. It calculates the chunk detection F1 score.
A chunk is correctly detected if its beginning, end and type are correct. Other chunk type is ignored.
For each label in the label sequence, we have:
tagType = label % numTagType chunkType = label / numTagType otherChunkType = numChunkTypes
The total number of different labels is numTagType*numChunkTypes+1. We support 4 labelling scheme. The tag type for each of the scheme is shown as follows:
Scheme Begin Inside End Single plain 0 - - - IOB 0 1 - - IOE - 0 1 - IOBES 0 1 2 3
‘plain’ means the whole chunk must contain exactly the same chunk label.
The simple usage is:
eval = chunk_evaluator(input)
Parameters: - input (LayerOutput) – The input layers.
- name (basename|None) – The Evaluator name, it is not necessary.
- chunk_scheme (basestring) – The labelling schemes support 4 types. It is one of “IOB”, “IOE”, “IOBES”, “plain”.This Evaluator must contain this chunk_scheme.
- num_chunk_types – number of chunk types other than “other”
-
paddle.trainer_config_helpers.evaluators.sum_evaluator(*args, **kwargs)¶ An Evaluator to sum the result of input.
The simple usage:
eval = sum_evaluator(input)
Parameters: - name (None|basestring) – Evaluator name.
- input (LayerOutput) – Input Layer name.
- weight (LayerOutput) – Weight Layer name. It should be a matrix with size [sample_num, 1]. (TODO, explaination)
-
paddle.trainer_config_helpers.evaluators.column_sum_evaluator(*args, **kwargs)¶ This Evaluator is used to sum the last column of input.
The simple usage is:
eval = column_sum_evaluator(input, label)
Parameters: - name (None|basestring) – Evaluator name.
- input (LayerOutput) – Input Layer name.
-
paddle.trainer_config_helpers.evaluators.value_printer_evaluator(*args, **kwargs)¶ This Evaluator is used to print the values of input layers. It contains one or more input layers.
The simple usage is:
eval = value_printer_evaluator(input)
Parameters: - input (LayerOutput|list) – One or more input layers.
- name (None|basestring) – Evaluator name.
-
paddle.trainer_config_helpers.evaluators.gradient_printer_evaluator(*args, **kwargs)¶ This Evaluator is used to print the gradient of input layers. It contains one or more input layers.
The simple usage is:
eval = gradient_printer_evaluator(input)
Parameters: - input (LayerOutput|list) – One or more input layers.
- name (None|basestring) – Evaluator name.
-
paddle.trainer_config_helpers.evaluators.maxid_printer_evaluator(*args, **kwargs)¶ This Evaluator is used to print maximum top k values and their indexes of each row of input layers. It contains one or more input layers. k is specified by num_results.
The simple usage is:
eval = maxid_printer_evaluator(input)
Parameters: - input (LayerOutput|list) – Input Layer name.
- num_results (int.) – This number is used to specify the top k numbers. It is 1 by default.
- name (None|basestring) – Evaluator name.
-
paddle.trainer_config_helpers.evaluators.maxframe_printer_evaluator(*args, **kwargs)¶ This Evaluator is used to print the top k frames of each input layers. The input layers should contain sequences info or sequences type. k is specified by num_results. It contains one or more input layers.
Note
The width of each frame is 1.
The simple usage is:
eval = maxframe_printer_evaluator(input)
Parameters: - input (LayerOutput|list) – Input Layer name.
- name (None|basestring) – Evaluator name.
-
paddle.trainer_config_helpers.evaluators.seqtext_printer_evaluator(*args, **kwargs)¶ Sequence text printer will print text according to index matrix and a dictionary. There can be multiple input to this layer:
1. If there is only one input, the input must be a matrix containing the sequence of indices;
2. If there are more than one input, the first input should be ids, and are interpreted as sample ids.
The output format will be:
- sequence without sub-sequence, and there is probability.
id prob space_seperated_tokens_from_dictionary_according_to_seq
- sequence without sub-sequence, and there is not probability.
id space_seperated_tokens_from_dictionary_according_to_seq
- sequence with sub-sequence, and there is not probability.
id space_seperated_tokens_from_dictionary_according_to_sub_seq space_seperated_tokens_from_dictionary_according_to_sub_seq ...
Typically SequenceTextPrinter layer takes output of maxid or RecurrentGroup with maxid (when generating) as an input.
The simple usage is:
eval = seqtext_printer_evaluator(input, dict_file=dict_file, result_file=result_file)
Parameters: - input (LayerOutput|list) – Input Layer name.
- dict_file (basestring) – The input dictionary which contains a list of tokens.
- result_file (basestring) – The file is to save the results.
- delimited (bool) – Whether to use space to separate output tokens. Default is True. No space is added if set to False.
- name (None|basestring) – Evaluator name.
-
paddle.trainer_config_helpers.evaluators.classification_error_printer_evaluator(*args, **kwargs)¶ This Evaluator is used to print the classification error of each sample.
The simple usage is:
eval = classification_error_printer_evaluator(input)
Parameters: - input (LayerOutput) – Input layer.
- label (LayerOutput) – Input label layer.
- name (None|basestring) – Evaluator name.