* exclude lr scheduler's state from accumulators_holder

* fix when there is no learning rate scheduler

* make a copy of the loaded state dict to avoid modifying it

* opt dygraph python API, test=develop

* Fix unbind bug in manipulation.py

This reverts commit b2afc8df.

* fix Undefined variables

* fix Undefined variables

* fuse L2Decay and momentum when param.regularizer is set

* add unittest

* refine

* refine _create_regularization_of_grad of momentum

* improve append_optimizer_op

* support params groups, test=develop

* simplify updating opt attr

* update according to review

add __all__=[] to python files not in API public list; import * only support in API public list files (#32643)

* add input EpsilonTensor for adam

* update python api

* add unit test

* add npu test

* add more ut

* fixed docs.

* Fixed docs. test=document_fix

code bak.

fixed docs. test=document_fix

* Revert to previous version of python/paddle/fluid/backward.py

* fixed bugs.

* test=document_fix. Fixed examples.

* add support for optimizer with varbase input

* refine cond

* fix failed unittest

* add test for coverage

* Use correct master weights in AdamW.

* Just modify the master weight.

* Update for CI Coverage.

* Modify the calculation logic of LambOptimizer

* add cast ops before and after unsupported fp16 ops.

* Keep partial net in FP32 pattern.

* Support check_finite_and_unscale and update_loss_scaling for FP16 calculation mode.

* Add fp16 support for adam op.

* add multi precision attr for adam.

* Fix the bug of test_multi_precision_fp16_train UT.

* Code format for CI.

* Fix the redefine error about MPTypeTrait on windows.

* fix bugs of the _create_accumulators func in Momentum.

* fix bug when inserting post cast op.

* Add the update_loss_scaling op in allow_set of UnusedVarCheck.

* Update for ci coverage.

* Add some doc for OptimizerWithMixedPrecision.

* Fix the code style.

* Imporve the doc of `amp_init`.

* Change for fp16 testing if users have the infer program defined in separate way.

* fix code bugs

* fix code bugs test=document_fix

* fix code bugs test=document_fix

* merge amp related function in Momentum from paddle.fluid.contrib.optimizer into paddle.optimizer.

* Add unittest for 2.0  Momentum API.

* fix some bugs in weight_decay.

* fix 3 doc

* fix 3 doc

* Update adadelta.py

* add lamb optimizer and unittest

* fix momentum resume training

* fix momentum acc

* The leaf tensor concept is exposed and the gradient accumulation of leaf tensor

* The leaf tensor concept is exposed and the gradient accumulation of leaf tensor

* fix coverage

* fix api doc

* fix CI unittest

* fix CI unittest

* fix unitest

* empty tensor does’t need inner_var_

* fix some error message

* fix 2.0 api, test=develop

* fix api, test=develop

* add reducer

* refine envent for memorycopy

* add concat&split for allreduce

* apply concat & split for fuse tensor

* fix nccl dep

* fix the untest, compile problem and ddp initialize problem

* fix untest for mac & add some comments & solve the repeated param in sublayers

* fix untest for windows & fix document

* add lamb optimizer and unittest

* fix lamb

* fix lamb v2 op

* fix sampling id

* fix lamb sample code

* Update lamb.py

* fix doc

* fix doc

* Update lamb.py

* upgrade comment string to raw string

* fix string in

* fix string with ' '

* revert update on comments

* upgrade only necessary

* fix sample code checker

* fix comments with '''

* fix doc of save/load

* fix API optimizer get_lr

* fix API optimizer get_lr

Fix no_grad setting in AdamW.
test=develop

* make Numpy version is below 1.19.3

* fix 2.0 doc

* Fix lr setting of AdamW when lr is an instance of LRScheduler.
test=develop

* Fix static graph test mode in test_adamw_op.py.
test=develop