* move temporal_shift to functional

* add the weight decay func for the momentum op

* Add the multi_precision function in Momentum Optimizer.

* Make sure that the initial value of master weights are same with the fp16 weights.

* add static loss scaling.

* add the rescale_grad function in the pure fp16 training.

* use the original momentum updating method.

* Polish some codes, such as variable names.

* add docstring for apis.

* update the var creation details of _create_master_weight.

* not modify codes about imperative momentum updating.

* Fix the error of test_dist_sparse_tensor_load_momentum UT.

* add unit test for multi precision fp16 training.

* add more unit tests for CI.

* Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT.

* For CI Coverage Checking.

* add fp16 for layer_norm op

* revert layernorm api

* fix forward

* fix forward

* fix backward for layernorm with fp16

* fix unit test for layernorm with fp16

* fix with_mkldnn compile error for layernorm with fp16

* 1. revert to PADDLE_ENFORCE_NOT_NULL, 2. change static_cast<float> to static_cast<U>

* fix with_mkldnn compile error for layernorm with fp16

* fix with_mkldnn compile error for layernorm with fp16
Co-authored-by: Nzhiqiu <chenqiuliang@baidu.com>

* Update IterableDataset API

* Update TensorDataset API

* Update APIs in paddle/text/datasets

* Update dataset.py

* Update get_worker_info API

* Update dataloader_iter.py

* Update dataloader_iter.py

* Update dataloader_iter.py

* Update conv3d API

* Update nn.py

* Update nn.py

* Update nn.py

* Update nn.py

* Update nn.py

* Update nn.py

* Update nn.py

* Update nn.py

This PR fixes several problems in dy2stat for Deoldify model in PaddleGan.

In model, software engineer wrote if x.shape == y.shape, the Tenser shape is a tuple in dygraph so the == returns True/False, but in static graph the == becomes element-wise comparison, which is a different behavior. In this PR we reduce the element-wise comparison result.

If software engineer write computations which uses parameters in hooks, the static graph can loss the parameter variable because we put param_guard at forward of a Layer. In this PR we made param_guard cover pre-hook and post-hook.

In PaddleGan, software engineer calculated some parameter values in __init__ by running some dygraph code. Those code also run during dy2stat. So some variables may be assign as a VarBase (Tensor) first and then Variable, which raised an error. We fixed the bug in this PR by handling the case.

TODO: We just added testcase for the 1. shape comparison. Should add test case for 2. and 3. But since we are chasing 2.0RC, I will do it in the near future PR

* pass stop_gradient for cast op

* improve performance of elementwise_add grad

* use tensor copy async

* dygraph branch

* fix dygraph branch

* add ut

* rebase develop

* update deformable_conv op on xpu

* update deformable_conv op on xpu

* hot fix complle failed in gcc4.8

* fix failed unittest

* add stop_gradient property and remove reduce redundant information

* refine code

* update conv2d & softmax to new xpu api
* test=kunlun

* remove useless comments
* test=kunlun

* remote softmax xpu op
* test=kunlun

* update kunlun softmax
* test=kunlun

* update xpu unitest
* test=kunlun

* fix elementwise_grad bug for kunlun
*test=kunlun

* fix 3 doc

* fix 3 doc

* Update adadelta.py

* add lamb optimizer and unittest

* fix momentum resume training

* fix momentum acc

* add complex64 and complex128 type; add +-*/@ and slice opreator for complex types

* add test cases for complex elementwise, matmul and getitem unittest

* add test cases for complex types

* add test cases for complex matmul unittest

* The leaf tensor concept is exposed and the gradient accumulation of leaf tensor

* The leaf tensor concept is exposed and the gradient accumulation of leaf tensor

* fix coverage

* fix api doc

* fix CI unittest

* fix CI unittest

* fix unitest

* empty tensor does’t need inner_var_

* fix some error message

* fix en doc, test=document_fix

* add blank after code declare, test=document_fix

* refine doc of dropout, test=document_fix

* refine npair_loss and dropout, test=document_fix

* fix paramete prefetch & device guard
Co-authored-by: NMrChengmo <cmchengmo@163.com>
Co-authored-by: Nchengmo <chengmo@baidu.com>

* Add a class TensorInplaceVersion to count the inplace version and put it in framework::Tensor instead of Allocation or Variable.

* Add a new attribute `_inplace_version` for VarBase.

* Raise exception if an inplace operation can result in incorrect gradient computation.

* Add a new interface _bump_inplace_version() for VarBase to bump the version whenever the Tensor is modified through an inplace operation.

* For api assign, call _bump_inplace_version() when it's an inplace operation inn dynamic mode.

* Use original var_wrapper if the inplace_version is not changed.

* Replace SnapshotVarWrapperList with SnapshotVarWrapper to optimize performane.

* update, test=develop

As the title

* fix failed tests in yingchun gived list

* add unittests into static_mode_white_list

* add enable static

* fix dist unittest

* skip test_sigmoid_focal_loss_op & add gym

* revert no need skip unittests

* remove gym

* test=develop, rm pathlib

* Changed a variable name error

* Add comments

* Move member functions of TranslatedLayer out of function

* edit code according to review

* Edit input argument of '_run_static_graph'

* reset due to Segmentation fault

* rename variables when stitching graph

* modify code according CI

* Add comments to '__i_m_p_l__'

* remove blanks befor 'Get...'

* edit code according to review

* Add a comment to '_execution_method_creator'

* Edit a comment to '_execution_method_creator'

* Generate code coverage reports only for incremental files, test=develop

* Generate code coverage reports only for incremental files, test=develop

* Generate code coverage reports only for incremental files, test=develop

* test for diff python file, test=develop

* fix no python diff report, test=develop

* add cc test file, test=develop

* fix bug in generic.cmake, test=develop

* for debug no cc report, test=develp

* modify compire branch form test_pr to test, test=develop

* fix bug, test=develop

* test for h file changed, test=develop

* debug for redefinition of argument optimize error, test=develop

* close -o3 for test, test=develop

* remove -o3 for test, test=develop

* remove coverage option for nvcc, test=develop

* use CMAKE_CXX_FLAGS open coverage option when header file changed, test=develop

* reopen -o3, test=develop

* remove debug code, test=develop

* remove unused code, test=develop

test_mnist failed on CUDA11. We found that it is due to PaddleInference IR Optimization after debugging. We disable it in this PR and we will re-enable it after PaddleInference fixes it.