QAT int8 accuracy little improvement (#21074)

test=develop

QAT int8 accuracy little improvement (#21074)
test=develop
37e0e7a9 · joanna.wozna.intel · Tao Luo · 8414575b · 37e0e7a9
隐藏空白更改
内联并排

Showing with 3 addition and 3 deletion

python/paddle/fluid/contrib/slim/quantization/quantization_mkldnn_pass.py ...uid/contrib/slim/quantization/quantization_mkldnn_pass.py +3 -3

未找到文件。
--- a/python/paddle/fluid/contrib/slim/quantization/quantization_mkldnn_pass.py
+++ b/python/paddle/fluid/contrib/slim/quantization/quantization_mkldnn_pass.py
@@ -514,11 +514,11 @@ class FakeQAT2MkldnnINT8PerfPass(object):
                    weights = np.array(
                        self._load_param(self._scope, weight_var_name))
                    scales = 1.0 / np.amax(
-                        np.abs(weights.reshape(weights.shape[0], -1)),
+                        np.abs(weights.reshape(weights.shape[0], -1)).astype(
+                            np.float64),
                        axis=axis)
-                    lod_tensor = self._convert_scale2tensor(
+                    lod_tensor = self._convert_scale2tensor(scales)
-                        scales.astype(np.float64))
                    use_unsigned_int = False
                    self._var_quant_scales[weight_var_name] = (use_unsigned_int,
                                                               lod_tensor)