* support xpu multi-card infer

* add ut

* clean code

* clean code

* fix

* fix

* fix

* fix

* 1219

* temporarily change the num_diff_files limit, test=document_fix

* Revert "temporarily change the num_diff_files limit, test=document_fix"

This reverts commit 8e70f00ef468d2dad0e38b3da06295ed62990d20.

* for codestyle

* remove duplicate license

* `static mode` -> `static graph mode`

* Update hybrid_parallel_inference.py

* Update layer_function_generator.py

* Update manipulation.py

* reset
Co-authored-by: NLigoml <39876205+Ligoml@users.noreply.github.com>
Co-authored-by: NSigureMo <sigure.qaq@gmail.com>

* lbfgs right version

* delete log

* code_style

* add test

* delete sparse grad

* add test for converage

* modify example in dygraph

* revert lbfgs

* add test for linesearch

* code_style

* add test

* modify test

* modify test

* modify set

* modify set

* modify test

* modiy code_block; test=docs_preview

* modiy code_block; test=docs_preview

* isort all files

* revert conflicting files

* revert conflicting files

* revert conflicting files

* remove apis in fluid.ops

* fix test_activation_nn_grad

* fix circle import error

* fix ops

* fix cos

* fix divide not inplace

* remove lazy-import part

* fix flake8 CodeStyle E266

* fix comments

* [CodeStyle][py2][U008] remove unnecessary args in `super()`

* remove remained args

* revert changes in test_pylayer_op

* Revert "revert changes in test_pylayer_op"

This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4.

* revert some changes in example code

* Remove reduntant numpy input in Example code

* Remove reduntant numpy input in Example code, test=document_fix

* update config

* re-blacken python code

* temporarily disable date and diff_py_file

* skip a format

[CodeStyle][F401] remove unused imports in python_paddle/inference_device_profiler_text_metric_incubate_quantization_libs_audio_amp_jit. (#46762)

* update for py3.6.

* update for py_3.6

* trim trailing whitespace

* fix `.cmake-format.py`

* revert npu ut changes, avoid npu ci error

* add use_hierarchical_allreduce

* support hierarchical allreduce for more cases

* move average_accumulates op to phi kernel

* add nproc_per_node for DistributedFusedLamb

* fix nproc_per_node communicator bug

* fix ring_id = 1 init bug

* fix ci

* fix test_parallel_executor_mnist.py

* add use_master_acc_grad

* add ut

* use yapf to format all python file

* yapf exclude two unittests file for they rely on writing and reading file, and format will break them

* disable diff_py_file because too many diff files cause command following failed

* improve introduction of bfgs args; test=document_fix

* modify according to zhouwei's comment; test=document_fix

* add gradient merge for DistributedFusedLamb

* use master acc gradient

* fix CI ut

* polish

* remove math_function_impl.h change

* fix test_update_loss_scaling_op.py

* try to fix XPU/NPU CI

* add gm ut

* fix bfgs_doc; test=document_fix

* add parameter name; test=document_fix

* modify according to chenlong's comments;test=document_fix

* fix image cv2 import

* fix lamb

* add Output(Step) to distributed fused lamb op

* add _set_step

* cut off relation between xk and initial_position's graph

* fix_bug

* add detach to cut off with original graph

* change vjp to paddle.grad

* use grad and gradients api

* fix preprocess for x

* fix a bug, val_and_grad should return a Tensor

* detach value and grad to avoid assign error
Co-authored-by: Nlevi131 <limaolin01@baidu.com>

* [New API]: miminize_bfgs and miminize_lbfgs

* modify for python module call correctly

* add functional package, add error raise in static_graph, change assign to set_value

* unify static_graph and dygraph, fix bug when x or H0 is float64

* now only accept input is tensor, put check args in utils.py, put exception test together

* temp

* add more detailed algorithm illustration and comment, reduce test case to limit test time in 15s

* change in_dygraph_mode to in_dynamic_mode

* fix bug of sample code; reduce test case to reduce test time

* change dir to incubate

* refactor eager flags

* fix flags error when we switch from eager to dygraph

* fix ci problem

* fix ci

* fix ci

* merge develop and fix code style

* merge develop and fix code style

* fix op test error

* fix op test error

* fix op test error

* fix op test error

* fix op test error

* merge develop

* vectorize lamb kernel

* remove flags, add ut

* remove useless codes

* refine code, add param order

* add multi tensor apply l2 norm

* add multi_tensor_apply code

* make sizeof(TensorMeta) smalller

* move code to distributed_fused_lamb_op.cu

* remove useless FLAGS

* add DistributedFusedLamb op

* polish code

* fix compile error

* compatible with pten changement

* fix rocm compile error

* improve converage

* update upstream/develop

* fix cast_with_ptr.h

* add FLAGS_distributed_lamb_divide_nranks_when_allreduce=1

* fix clip before allreduce

* add use_master_param_norm

* code polish

* fix bug

* fix ROCM ci

* cache core.ops, test=develop

* refine, test=develop

* update 2.0 public api in all left files

* reverse device.py all list;
fix some flake8 errors