reduce the  occupied size  of memory for the fused pattern of elementwise_add Op and activation Op(relu Op for example) (#29885)

* - Added UT for testing elementwise_mul caching

* lint fixes

* add cast ops before and after unsupported fp16 ops.

* Keep partial net in FP32 pattern.

* Support check_finite_and_unscale and update_loss_scaling for FP16 calculation mode.

* Add fp16 support for adam op.

* add multi precision attr for adam.

* Fix the bug of test_multi_precision_fp16_train UT.

* Code format for CI.

* Fix the redefine error about MPTypeTrait on windows.

* fix bugs of the _create_accumulators func in Momentum.

* fix bug when inserting post cast op.

* Add the update_loss_scaling op in allow_set of UnusedVarCheck.

* Update for ci coverage.

* Add some doc for OptimizerWithMixedPrecision.

* Fix the code style.

* Imporve the doc of `amp_init`.

* Change for fp16 testing if users have the infer program defined in separate way.

* fix dtype of ungenerated grad var

* update ut

* refine code

* set default dtype

* fix could_use_cudnn bug

* remove debug code

* re-implement

* fix bug

* add tensor table

* enhance error message of nll_loss op test=develop

Improve some error messages in parallel_executor.cc, conditional_block_op.cc, recurrent_op.cc

* test=develop, add index_select_cuda kernel

* enhance error info for py_func

* update

* add aarch64 and sunway kunlun lib

* minor

* optimize elementwise_add for kunlun

* update kunlun dependence

* minor

* minor

1. when slice_item is a slice: 
 1) the start of __getitem__ should be std::max(start, 0) if slice
 2) the start of __getitem__ should be std::min(end, dim) 
2. when slice_item is an integer, it should be in [-dim_len, dim_len) 
3. Fix error message to use accurate data

* add trace op_register_version and fix defaulf bug; test=op_version

* add trace op_register_version; test=op_version

* add trace op_register_version; test=op_version

* add trace op_register_version; test=op_version

* fix missing the template bug of vector; test=op_version

* fix the formate of raising error in randperm op

* p_norm fix op version info. test=develop

* for inference checkpoint

Change-Id: I36c979240ffa55bf1ef0c9315402960762af6be4

* for inference checkpoint

Change-Id: I82025365d5b792cbea1ead506df685aecc8ac198

* register instance norm, test=op_version

Co-authored-by: Nzhoushunjie <zhoushunjie@baidu.com>

* Register op version for linspace,test=op_version

* Register op version for linspace,test=op_version

* Register op version for linspace,test=op_version

* Register op version for linspace,test=op_version

* Register op version for linspace,test=op_version

* test=develop, add op_register_version for roll_op

* dot op support complex types

* matmul support complex types

* add test case

* matmul broadcast gradient support complex

* move conjFunctor to complex_functor.h

* fix rank_attention, test=op_version

* Add operator checkpoints for new attributes.

* Fix adding subsequent checkpoint to quantize op.

* Add mkldnn nearest_interp and bilinear_interp op
* don't run mkldnn interpolate in default
* add interpolate_mkldnn_pass