Created by: Superjomn

Model CI

The users occasionally found a negligible performance or precision difference between different Paddle versions. Though we have unit tests for each class and Travis-CI to ensures the precision of each operator, there is no any logic to ensure the model (a composition of several operators) works as reliable as the operators.

There are several conditions where an existing model will fail either in performance or precision:

1. the incomplete coverage test cases.
2. poor performance update, currently, we have no performance tracker for each operator.
3. API changes, developers are likely to forget to update usages in some other repos such as paddle/models.

The model-CI module is proposed to enhance the weaknesses above and track the overall performance and precision of the model granularity.

Module function

Inputs:

• a compiled paddle whl of some version
• some test models
• historical information of the execution of the test models

Outputs:

• execution information of the test models
• give an alarm if there is an obvious difference

Indicators tracked

1. success or not of a model execution
2. execution time of a model
3. train cost trend
4. infer precision trend