Created by: wojtuss

PR types

New features

PR changes

OPs

Describe

This patch adds support for INT8 quantization of fusion_gru op. It includes commits from PR https://github.com/PaddlePaddle/Paddle/pull/27481 and provides the rest of functionality required for https://github.com/PaddlePaddle/Paddle/issues/27330.

This patch adds also a test with transformation of a quant GRU model into int8 model. The saved int8 model can be used for testing accuracy and performance:

ctest -R save_quant2_model_gru -V

Performance benchmarking will make sense only after bumping up oneDNN version commit with an optimized GRU INT8 primitive, as the current oneDNN version provides unoptimized GRU INT8 kernel only. The oneDNN version will be updated most probably by the end of this week.

With these changes INT8 quantization of the fusion_gru op will be enabled. However, quantization of all the quantizable operators in the GRU model does not work yet because other operators like concat does not support quantization with shift yet. For performance reasons it is desirable to have a sequence of quantized operators without dequantization/quantization in between, so support for quantization of concat op with shift will be implemented as well. A PR with the changes should come by the end of this week as well.

[Update] Now the patch has updated oneDNN commit hash containing optimized version of GRU INT8 kernel. Here are the benchmark results of the saved GRU INT8 model on CLX 6248:   | fp32 | qat (fp32) | int8 | int8-qat diff | fp32/int8 ratio -- | -- | -- | -- | -- | -- Precision | 0.89211 | 0.89198 | 0.89221 | 0.00023 |   Recall | 0.89442 | 0.89449 | 0.89412 | -0.00037 |   F1 score | 0.89326 | 0.89323 | 0.89316 | -0.00007 |   batch latency (ms) | 25.3818 | 27.8914 | 15.9434 |   | 1.59

The command for GRU INT8 model benchmarking:

build/paddle/fluid/inference/tests/api/test_analyzer_lexical_analysis \
          --infer_model=build/third_party/inference_demo/quant/GRU_quant2_int8 \
          --infer_data=build/third_party/inference_demo/gru/GRU_eval_data.bin \
          --batch_size=50 \
          --cpu_num_threads=1 \
          --with_accuracy_layer=true \
          --use_analysis=false \
          --iterations=0

For GRU FP32 use the model from http://paddle-inference-dist.bj.bcebos.com/gru/GRU_eval_model_v2.tar.gz

There are still options to improve INT8 performance, we are working on them.