【New IR】 backward gradients accumulate test and pulish append_backward_ops func for op_pattern  (#56265)

* [prim][newir] add basic framework for primitive

* support desctensor in new ir

* add vjp interface

* support vjp in new ir

* support vjp in new ir

* polish vjp interface

* fix stop_gradients set

* fix vjp dispatch

* add comment

* add vjp test for new ir

* add test for tanh vjp

* [prim][newir] add basic framework for primitive

* support desctensor in new ir

* support vjp in new ir

* support vjp in new ir

* polish vjp interface

* fix stop_gradients set

* fix vjp dispatch

* add comment

* add vjp test for new ir

* add test for tanh vjp

* add eager and static backend for warp lower level api

* support call_vjp pybind

* polish code and add test for vjp

* remove useless code

* polish code

* remove useless code

* support mean vjp

* backward origin code

* add test for mean vjp and support has_vjp function

* fix call_vjp

* polish code

* add attrs and dtype interface

* add primitive ops set for backend

* fix compile bugs

* fix some bugs

* fix windows bugs

* add vjp test for tanh_

* fix inference CI

* fix inference ci

* modify fluid cmake

* origin test of tanh and mean passed

* fix conflict

* modify stop_gradient

* remove useless deps

* add cmake

* modify block.ops

* modify test

* fix conflict

* reply review comments

* reply review comments

* pulish code

* fix comment

* fix test

* polish code

* modify backward stop_gradients

* modify static_backend.cc

* refactor grad_op

* support add and add_inplace vjp

* remove useless code

* remove useless code

* remove cout

* modify add_n

* modify add_n with add_vjp test

* modify add_n with add_vjp test

* fix conflict and concat call_vjp

* modify backward test

* Add more gen api

---------
Co-authored-by: Ncxxly <chenxx_id@163.com>
Co-authored-by: NCharles-hit <wanghao107@baidu.com>
Co-authored-by: Nzhangbo9674 <zhangbo54@baidu.com>
Co-authored-by: NYuanRisheng <yuanrisheng@baidu.com>
Co-authored-by: N0x45f <wangzhen45@baidu.com>

* update

* fix codestyle

* update

* update

* test=docs_preview

* test=docs_preview

* Fix and add unittest

* don't introduce assign when already in global block.

* fix more

* fix bugs

* fix bugs

* fix ci

* fix bfgs

* make function local

* support setting backward prim blacklist

* add test case

* polish prim forward flag

* fix test case

* [Custom Dice]add run_check support for custom device

* fix error msg

* fix typo

* update for all custom device

* fix

* add warning msg

* add lu_unpack data check

* add error input api test

* add error type info

* [WIP] Add mp_all_reduce asynchronize overlap.

* Fix some problems.

* Fix dw compute bug, and use a temporary solution to achieve overlap.

* Use fused_linear_param_grad_add to compute dw.

* Reformat ColumnParallel _overlap_linear. Use environment flags to
control following behaviors:
1. export Flags_mp_aysnc_allreduce=True to turn on mp async all_reduce
2. export Flags_skip_mp_c_identity=True to skip two c_identity operators
   in dygraph mode.
3. export Flags_fused_linear_param_grad_add to enable fused_linear_param_grad_add
   in ColumnParallel backward with mp async all_reduce.

* Polish code.

* Remove useless communication API.

* Fix some problems in mp_async_all_reduce and skip_c_identity.

* Add test cases.

* Remove environment variable Flags_fused_linear_param_grad_add in test case.

* Reset error threshold.

* Reset threshold in test case.

* Add useful log. Remove useless test cases.

* refine static op return val

* fix inset get_parameter op bug

* fix bug: insert  only for parameters

* fix bug: wrong idx in vector

---------
Co-authored-by: Nzhangbo9674 <zhangbo54@baidu.com>

* test=docs_preview

* test=docs_preview

* test=docs_preview

* test=docs_preview

* fix indent in math.py

* test=docs_preview

* test=docs_preview

* test=docs_preview

* test=docs_preview

---------
Co-authored-by: NSigureMo <sigure.qaq@gmail.com>

* add range support in indexing

* add getitem ut case

* fix: updated code examples.

* fix: updated blank lines.

* fix: updated code style

* fix: removed extra changes

* fix: refine detail

[xdoctest][task 108] Reformat example code with google style in python/paddle/optimizer/lbfgs.py (#56224)

* [xdoctest][task 108] test=docs_preview

* Apply suggestions from code review

---------
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* make params_grads order same bewteen dynamic and static mode

* revert inplace clip

* use sorted attribute to control

* tiny fix

* fix find loss_grad_op

[xdoctest][task 120] Reformat example code with google style in python/paddle/quantization/observers/abs_max.py (#56231)
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

---------
Co-authored-by: Ntianhaodongbd <tianhaodong@baidu.com>

* support_mmha
* add_python_api
* add_api_doc
* fix_doc_error
* fix_infermeta
* add_infermeta
* add_bf16_cuda_check
* add_bf16_check
* fix_ci_windows
* fix_ci_windows_kernel_register
* fix_test_mmha
* add_cumoffsets
* remove_bias
* delete_mmha_reshape_input_output
* rename_delete_hfile
* remove_fluid

---------
Co-authored-by: Nyangjianfengo1 <yangjianfeng01@baidu.com>

[xdoctest][task 142] Reformat example code with google style in quantization/quant_int8_mkldnn_pass.py (#56239)

* [xdoctest][task 142] test=docs_preview

* Update python/paddle/static/quantization/quant_int8_mkldnn_pass.py

* Update python/paddle/static/quantization/quant_int8_mkldnn_pass.py

---------
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* [dtype] add fp16 support for dist_kernel

* fix typo

* fix CE

* fix CE

* fix CE

* fix CE

* fix CE

* refactor

* fix CE

* fix CE

* fix varname

* add bf16

* add ut for bf16

* fix CE

* Improve GC for pipeline parallel

* Delete print

* fix bug of nop_op and sharding

---------
Co-authored-by: Nchenruibiao <chenruibiao@baidu.com>

* fix sample codes

* fix bug

* Update tess.py

* Update dlpack.py

* Apply suggestions from code review

* Apply suggestions from code review

* add reference of lbfgs

* add reference of lbfgs

* input.py

* Update python/paddle/nn/functional/input.py

* Update input.py

* Update all_gather.py

* Update all_gather.py

[xdoctest][task 123] Reformat example code with google style in python/paddle/quantization/quanters/abs_max.py (#56234)

[xdoctest][task 107]reformat example code with google style in python/paddle/optimizer/lamb.py test=docs_preivew (#56223)

* [AMP] add fp16 op detection for custom device

* resolve conflicts

[xdoctest][task 106] Reformat example code with google style in python/paddle/optimizer/adamw.py (#56222)

* [xdoctest][task 106]reformat example code with google style in python/paddle/optimizer/adamw.py

* fix word typo test=docs_preview

* fix word typo test=docs_preview

* fix whitespace test=docs_preview

[xdoctest][task 110] Reformat example code with google style in python/paddle/optimizer/momentum.py (#56226)

* [xdoctest][task 110] test=docs_preview

* fix whitespace test=docs_preview

[xdoctest][task 111] Reformat example code with google style in python/paddle/optimizer/rmsprop.py (#56227)

* [xdoctest][task 111] test=docs_preview

* fix whitespace test=docs_preview

[xdoctest][task 112] Reformat example code with google style in python/paddle/optimizer/sgd.py (#56228)

* temp commit

* distribute best cfg

* update metric extracting

* fix bugs of prune and reading log

* fix adding cfg bug

* reset status

* remove alarm and set logdir

* deepcopy ctx

* change alarm

* fix restart bug

* best no need alarm

* add gbs search, add gpu memory to history csv, add memory detect

* fix bug

* fix memory read bug; fix etcd connection bug

* fix memory read bug, add oom detection for all ranks

* fix read log and oom detaction, add error code for read log

* add unit test

* Update master.py

---------
Co-authored-by: Ncaozhou <caozhou@radi.ac.cn>

* weight_norm_hook

* Update weight_norm_hook.py

* Update weight_norm_hook.py

* Update python/paddle/nn/utils/weight_norm_hook.py

* Update python/paddle/nn/utils/weight_norm_hook.py

* Update python/paddle/nn/utils/weight_norm_hook.py
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

---------
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* add rmsnorm residual bias add and quant

* refine python interface

* add rmsnorm unittest

* Add layernorm

* fix layernorm unittest

* refine unittest

* fix example code

* fix review comment