* add use_hierarchical_allreduce

* support hierarchical allreduce for more cases

* remove cudaDeviceContext

* remove more template

* fix rocm compile

* remove alias name CUDADeviceContext

* fix compile

* fix tests

* revert changes

* add some fp16 op for kunlun resnet50 model
*test=kunlun

* tmp
*test=kunlun

*test=kunlun

* fix cpu lars_momentum bug & add xpu grad_add/log_softmax/log_softmax_grad
*test=kunlun

* minor
*test=kunlun

* remov merged_adam_op to phi

* refine code

* rmsprop for xpu. test=kunlun

* minor fix (follow comments). test=kunlun

* fix init()

* delete test_device_context

* replace CPUDeviceContext with CPUContext

* fix test_scalar

* remove dot_op.cc

* fix compile

* add nproc_per_node for DistributedFusedLamb

* fix nproc_per_node communicator bug

* fix ring_id = 1 init bug

* fix ci

* fix test_parallel_executor_mnist.py

* add use_master_acc_grad

* add ut

* refactor the optional tensor

* remove optiona<MetaTensor> in InferMeta

* fix bug

* fix optional<vector<Tensor>>

* fix bug

* fix rmsprop

* fix amp of eager_gen

* polish code

* fix deleted code

* fix merge conflict

* polish code

* remove is_nullopt_

* fix merge conflict

* fix merge conflict

* Add the new XDNN implementation. test=kunlun

* Add the new XDNN implementation. test=kunlun

* Modify the code based on review, test=kunlun

* add gradient merge for DistributedFusedLamb

* use master acc gradient

* fix CI ut

* polish

* remove math_function_impl.h change

* fix test_update_loss_scaling_op.py

* try to fix XPU/NPU CI

* add gm ut

* add sgd yaml

* change python api

* open eager mode in sgd

* fix bug

* add Output(Step) to distributed fused lamb op

* add _set_step

* momentum support l2decay for xpu. test=kunlun

* fix include file. test=kunlun

* fix cmake for device_worker. test=kunlun

* add some infer meta

* fix bug

* fix bugs;

* fix bug and add set data type

* revert infer shape of lookup table

* recover test

* move momentum, rmsprop to phi; test=develop

* update

* update

* update

* update

* udpate; test=develop

* fix xpu npu bugs; test=develop

* fix npu bug; test=develop

* fix windows compile error; test=develop

* fix windows compile error; test=develop

* polish code; test=develop

* fix conflict; test=develop

* add meshgrid;

* update

* polish code

* polish code;

* fix bug

* format; remove useless code

* fix npu bug

* fix bug

* fix lars optitmizer bug

* Update optimizer.py

* [Phi] Migrate Adam and Adamw into Phi

* fix compile error and unittest ok

* fix compile error and unittest ok

* fix undefined reference to fLI::FLAGS

* test depend on operator

* fix cmake

* fix xpu compile

* fix infrt

* fix amp_type_traits

* fix amp_type_traits

* modify according reviewer

* modify according reviewer

* fix dtype float16

* fix typo

* fix Cmake

* fix code style

* [NPU] add merged_momentum

* fix

* fix device

* [Phi]Migrate Adamax into phi

* Add adadelta kernel

* clean distribution_helper, index_impl, aligned_vector code in fluid

* fix conflicts