pvanet model INT8 model accuracy drop 10%
Created by: lidanqing-intel
Customer model pvanet accuracy after QAT INT8 the model accuracy drop 10%
System information -PaddlePaddle version: develop -CPU: i9
To Reproduce
Download test, dataset and model from
shanghai-6271-ip:/data/models/ocr_test_intel_with_data.tgz
Run following command:
sh test_metric.sh models_pvanet/model_folder/ 28
Describe your current behavior
I have set all relu output datatype, and also set concat to INT8 concat kernel. But the accuracy only improved from 75.03%
to 75.28%
, while the FP32 model accuracy is 85%
Describe current result fp32:
hit_count: 3437 gt_count: 3886 det_count: 3904
recall: 88.45 precision: 88.04 f_measure: 88.24
0.995053826011
Qat:
hit_count: 3379 gt_count: 3886 det_count: 3890
recall: 86.95 precision: 86.86 f_measure: 86.91
0.994968925718
INT8:
hit_count: 2937 gt_count: 3886 det_count: 3917
recall: 75.58 precision: 74.98 f_measure: 75.28
0.973782771536