* add use_hierarchical_allreduce

* support hierarchical allreduce for more cases

* move average_accumulates op to phi kernel

* add nproc_per_node for DistributedFusedLamb

* fix nproc_per_node communicator bug

* fix ring_id = 1 init bug

* fix ci

* fix test_parallel_executor_mnist.py

* add use_master_acc_grad

* add ut

* use yapf to format all python file

* yapf exclude two unittests file for they rely on writing and reading file, and format will break them

* disable diff_py_file because too many diff files cause command following failed

* improve introduction of bfgs args; test=document_fix

* modify according to zhouwei's comment; test=document_fix

* add gradient merge for DistributedFusedLamb

* use master acc gradient

* fix CI ut

* polish

* remove math_function_impl.h change

* fix test_update_loss_scaling_op.py

* try to fix XPU/NPU CI

* add gm ut

* fix bfgs_doc; test=document_fix

* add parameter name; test=document_fix

* modify according to chenlong's comments;test=document_fix

* fix image cv2 import

* fix lamb

* add Output(Step) to distributed fused lamb op

* add _set_step

* cut off relation between xk and initial_position's graph

* fix_bug

* add detach to cut off with original graph

* change vjp to paddle.grad

* use grad and gradients api

* fix preprocess for x

* fix a bug, val_and_grad should return a Tensor

* detach value and grad to avoid assign error
Co-authored-by: Nlevi131 <limaolin01@baidu.com>

* [New API]: miminize_bfgs and miminize_lbfgs

* modify for python module call correctly

* add functional package, add error raise in static_graph, change assign to set_value

* unify static_graph and dygraph, fix bug when x or H0 is float64

* now only accept input is tensor, put check args in utils.py, put exception test together

* temp

* add more detailed algorithm illustration and comment, reduce test case to limit test time in 15s

* change in_dygraph_mode to in_dynamic_mode

* fix bug of sample code; reduce test case to reduce test time

* change dir to incubate

* refactor eager flags

* fix flags error when we switch from eager to dygraph

* fix ci problem

* fix ci

* fix ci

* merge develop and fix code style

* merge develop and fix code style

* fix op test error

* fix op test error

* fix op test error

* fix op test error

* fix op test error

* merge develop

* vectorize lamb kernel

* remove flags, add ut

* remove useless codes

* refine code, add param order

* add multi tensor apply l2 norm

* add multi_tensor_apply code

* make sizeof(TensorMeta) smalller

* move code to distributed_fused_lamb_op.cu

* remove useless FLAGS

* add DistributedFusedLamb op

* polish code

* fix compile error

* compatible with pten changement

* fix rocm compile error

* improve converage

* update upstream/develop

* fix cast_with_ptr.h

* add FLAGS_distributed_lamb_divide_nranks_when_allreduce=1

* fix clip before allreduce

* add use_master_param_norm

* code polish

* fix bug

* fix ROCM ci

* cache core.ops, test=develop

* refine, test=develop

* update 2.0 public api in all left files

* reverse device.py all list;
fix some flake8 errors

* test=develop, fix test_lookahead

* test=develop, add model_average and lookahead