* Use correct master weights in AdamW.

* Just modify the master weight.

* Update for CI Coverage.

post quantize support insert fake_quantize_dequantize node before the OPs that will be used in VIS's faceid models (#30659)

test=develop

* Fix the error of save_quantized_model

* add fleet amp.init()

* add unittest for fleet_amp_init

* Collect weight threshold of lstm, test=develop

* skip quantizing ops in cpu inference, test=develop

* Implemented AddQuantDequantPass in imperative quantization.

* Supported LeakyReLU Quantization

* For meeting coverage rate.

* Changed the file name of test of AddQuantDequant

* Implemented more Quantized NoWeightLayers.

* Fix the loss cannot align problem between static and dynamic model quantization, add swish as supported quantized layer in imperative quantization.

* remove noweight_list

* support 2.0 API such as Pool2D and ReLu

* add sparse embedding & load vars for 2.0

Change-Id: I36b59ed5f015189dc9d9d2e34a9357722d369f1b

* fix hdfs gloo

Change-Id: Ia84d579053720ad804183e54c9a04b4f031c79c6

* fix gloo hdfs

Change-Id: I5ab982fd483cddc10adcdef0b8aa83aca976cb9e

* move loadvar/sparse embedding from incubute to static

Change-Id: I57081d3545ad2efab78c72420d2162c0eacaf3a0

* add cast ops before and after unsupported fp16 ops.

* Keep partial net in FP32 pattern.

* Support check_finite_and_unscale and update_loss_scaling for FP16 calculation mode.

* Add fp16 support for adam op.

* add multi precision attr for adam.

* Fix the bug of test_multi_precision_fp16_train UT.

* Code format for CI.

* Fix the redefine error about MPTypeTrait on windows.

* fix bugs of the _create_accumulators func in Momentum.

* fix bug when inserting post cast op.

* Add the update_loss_scaling op in allow_set of UnusedVarCheck.

* Update for ci coverage.

* Add some doc for OptimizerWithMixedPrecision.

* Fix the code style.

* Imporve the doc of `amp_init`.

* Change for fp16 testing if users have the infer program defined in separate way.

* Quantization supports 2.0 APIs

* Fix the error of save_quantized_model

* Avoid the scale to be infinity in quant2_int8_mkldnn_pass, test=develop
* support quantized model for paddle2.0 dygraph, test=develop

* Support quantizing program_desc, test=develop

* Fix unittest:test_imperative_skip_out

* rm check_import_scipy, rm chunk_eval and mean_iou in paddle.metric.__init__.py

* Revert "rm check_import_scipy, rm chunk_eval and mean_iou in paddle.metric.__init__.py"

This reverts commit 179ba8c2b22bc31fe8d8a126e31820792cbd0f4e.

* delete paddle.metric.chunk_eval and paddle.metric.mean_iou

* delete paddle.nn.clip and paddle.nn.clip_by_norm

* delete paddle.nn.functional.activation.hard_sigmoid and paddle.nn.functional.activation.hard_swish

* delete paddle.nn.Pool2D, paddle.nn.BilinearTensorProduct, paddle.nn.RowConv, paddle.nn.functional.row_conv

* fix extension import error

* fix unittest for row_conv and Pool2D

* merge amp related function in Momentum from paddle.fluid.contrib.optimizer into paddle.optimizer.

* Add unittest for 2.0  Momentum API.

* fix some bugs in weight_decay.

* add static.amp into setup.pu.in

* add unittest for api

* add alias for fluid.contrib.mixed_precision

* add the weight decay func for the momentum op

* Add the multi_precision function in Momentum Optimizer.

* Make sure that the initial value of master weights are same with the fp16 weights.

* add static loss scaling.

* add the rescale_grad function in the pure fp16 training.

* use the original momentum updating method.

* Polish some codes, such as variable names.

* add docstring for apis.

* update the var creation details of _create_master_weight.

* not modify codes about imperative momentum updating.

* Fix the error of test_dist_sparse_tensor_load_momentum UT.

* add unit test for multi precision fp16 training.

* add more unit tests for CI.

* Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT.

* For CI Coverage Checking.

* add fp16 for layer_norm op

* revert layernorm api

* fix forward

* fix forward

* fix backward for layernorm with fp16

* fix unit test for layernorm with fp16

* fix with_mkldnn compile error for layernorm with fp16

* 1. revert to PADDLE_ENFORCE_NOT_NULL, 2. change static_cast<float> to static_cast<U>

* fix with_mkldnn compile error for layernorm with fp16

* fix with_mkldnn compile error for layernorm with fp16
Co-authored-by: Nzhiqiu <chenqiuliang@baidu.com>

* Optimiz the unittest test_imperative_out_scale

test=develop

* Impelement 2.0 API version Conv2d and Linear layer quantization in imperative mode.

* use cudnn softmax in static Lenet

* Modified ChannelwiseQAT Unittest for 2.0 API.

* For CI python coverage.

* upgrade comment string to raw string

* fix string in

* fix string with ' '

* revert update on comments

* upgrade only necessary

* fix sample code checker

* fix comments with '''

* refactor momentum op to combine weight_decay (scale op and sum op)

* add debuging code

* change seed & add debug message

* support user-defined quant and preprocess

* add matmtl_v2 to amp list

* support dygraph