* dygraph amp support param_group

* remove unused code

* fix doc

* support margin loss (arcface, cosface, sphereface)

* Support NPU OP hard_swish and hard_swish_grad

* Support NPU OP hard_swish and hard_swish_grad

* add the unittest to compare the result between npu ans cpu

* format the prompt of exception

* replace Min and Max op by ClipByValue op

* fix the precision problem for fp16

* Using HardtanhGrad to improve performace

* fix dice_loss bug

* notest;pm-op-benchmark

* notest;pm-op-benchmark

* notest;pm-op-benchmark

* notest;pm-op-benchmark

* notest;pm-op-benchmark

* notest;pm-op-benchmark

* notest;test=op_benchmark

* notest;test=op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* notest;test=op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* notest;op_benchmark

* fix

* fix

* add p_norm_op_npu

* remove p_norm_grad op

* update

* OP dot: refactor CPU kernels and get better loop performance.

* Minor fix on code format.

* Fixed minor errors.

* Add new API: einsum

* Update the Einsum unit test.

One case failed with matmul_v2, where the dtype is int64:

a = np.arange(2 * 3 * 1).reshape(2, 3, 1)
b = np.arange(1)
paddle.einsum("...i, ...i", a, b)

* Test cases in test_einsum test floating point dtypes only.

As of now Paddle only supports float/double dtypes in matmul, which is
one of building blocks of this Einsum implementation. We decide not to
test einsum against other dtypes.

* Polish format.

* More formatting.

* Format...

* Einsum: improve test coverage.

* Einsum: bug fixes and more testcases for testing error messages

* Einsum: fix format..

* Einsum: fixed typo and format.

* Einsum: format again...

* Einsum: applied suggested changes.

* Einsum API: improve API documentation.

* Einsum API: apply suggested changes.

* Einsum API: Add dygraph only note.

* Einsum API: Add dygraph only note.

* Einsum API: fixed unittest.

* Fix a bug : can't load more than one custom op module

* Fix a bug : can't load more than one custom op module

* add test for load multiple modules of custom c++ op

* add config for Coverage CI

* Add EmptyGradOpMaker CI Approval, test=develop

* Fix typo in echo_line

* fix bug of py36 import

* remove unmatched signal error stack

* fix error writing for cond

This reverts commit 0a5c99e8.

This PR adds fused transformer related files defining c interface including class, function etc..

* Fix safety-bug of functional.linear

* Fix safety-bug of functional.linear

* Fix safety-bug of functional.linear

* Fix safety-bug of functional.linear

* add recompute for pp

* add recompute offload

* add recompute partition

* [NPU] Support npu op expand_v2 and expand_v2_grad

* [NPU] Support npu op expand_v2 and expand_v2_grad

* [NPU] Support npu op expand_v2 and expand_v2_grad

* update test_expand_v2_op_npu.py

* update test_expand_v2_op_npu.py

* modify expand_v2_op_npu.cc

* modify expand_v2_op_npu.cc

* add det_mv3_db & LeViT test case in pr-ci-inference

* fix LeViT model dir bugs

* fix grammar error

* - Added softmax without caching

* - Binary is no longer manually cached

* - Activation onednn caching removed

* - Removed manual caching of activation

* - modified UT

* - fix

* - fix

* - fixes to building

* - fix

* - fix

* - fix to UT

* - Faulty UT workaround

* - approval workaround

* - Fixes after review

* - compilation fixes

* - more lint fixes

* - more fixes after review

* - fixes after another round of review

* add state_dict and load_state_dict and unittest for class GradScaler

* refine unittest for coverage of load_state_dict

* refine comments of code-block

* refine some comments

* refine state_dict code and unittest

* add #require gpu, xpu for GradScaler get/set example code

* add #require gpu, xpu for GradScaler get/set example code

* refine example code

* refine unittest for state_dict

* refine unittest for state_dict

* fix bug of DataLoader in TestGradScalerStateDict

* add flag FLAGS_cudnn_deterministic

* add set_value_grad op

* add unittest.

* polish unittest.

* polish code.

* support cuda kernel

* polish code according to CI

* polish code.

* polish code

* remove *.pyc

* polish code.

* add unittest to improve coverage.

* polish code.

* fix_fc_reshape_convert

* fix

* add while read_from_array write_to_array npu op

* optimize unittest