* for verify

fluid operator support new comm library

* u

* u

* u

* compatiable new comm library upgrade for c_allgather, c_reduce, c_reduce_scatter and c_scatter.

* Remove useless comments in process_group.py

* Polish code style.

* Fix some problems.

* Remove use fluid api in phi comm_context_manager.

* Add PPADDLE_WITH_CUDA and PADDLE_WITH_NCCL micro judgement.

* Fix bug of HIP architecture.

* Fix some problems.
1. remove useless loggings.
2. Fix conditional compilation for HIP.
3. Fix problems of test_pass_generation_pipeline.py. It calls paddle.distributed.init_parallel_env() at first,
then auto.Engine calls _init_comm(), which will calls process_group.instantiate(). However, init_parallel_env() will call
paddle.distributed.barrier(), it will call CreateNCCLEnvCache and create corresponding NCCLCommContext. But dev_id is not
set, as a result, NCCLCommContext's dev_ctx is not initialized.

* Fix some problems.

* Polish code.

* Polish code.

* Revert compatiable upgrade for communication operators. Their upgrades
will be submitted in another PR.

* Remove StaticTCPStore.

* Remove useless modification.

* Remove useless set_cuda_device_id.

* Polish code.

* Remove fluid header files in phi files.

* Remove useless comments.

* Fix problems of hip arch.

* Fix some problems.

* Polish code.

* Polish code style.

---------
Co-authored-by: hitywt <yuwentao126@126.com>

* weight_norm_hook

* Update weight_norm_hook.py

* Update weight_norm_hook.py

* Update python/paddle/nn/utils/weight_norm_hook.py

* Update python/paddle/nn/utils/weight_norm_hook.py

* Update python/paddle/nn/utils/weight_norm_hook.py
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* xdoc

* Apply suggestions from code review

* Apply suggestions from code review

---------
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

[xdoctest][task 181-183] reformat example code with google style in `sparse/multiary.py`,`distributed/auto_parallel/*` (#56665)

* [Doctest]fix No.181-183, test=docs_preview

* add env skip

[xdoctest][task 213,215-217] reformat example code with google style in `python/paddle/distributed/fleet/base` (#56651)

* [xdoctest][task 213,215-217] reformat example code with google style in python/paddle/distributed/fleet/base

* fix output as comments

* fix eval_loss bug in parallelizer_v2.py

* fix fetch problem in pass_utils.py

* fix pylayer py39 mem leak

* Update input.py

* Update input.py

* Update gather.py

* Update broadcast.py

* Update batch_isend_irecv.py

* Update all_to_all.py

* Update all_reduce.py

* Update all_gather.py

* rollback

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* modify feed_data for dataloader in pipline parallel mode

* add pre-commit

* remove read op, use feed op

* fix validate batch_size

* tiny fix

* support catch EOFException

* fix conflict

* fix conflict

* fix executor if cond

---------
Co-authored-by: Frida-a <2624653516@qq.com>

* shard_tensor support static graph

* add comments

* add dy2static ut

* use property in c++ side

* 203

* 204

* 205

* 206

* 207

* 208

* 209

* 210

* 211

* Update all_to_all.py

* Apply suggestions from code review

* fix: updated code examples.

* fix: added paddle.seed

* fix: updated code style

* Apply suggestions from code review

* refactor: refine detail of code examples

* Update python/paddle/distributed/auto_parallel/static/process_mesh_v2.py

* fix: refine detail

* fix: refine detail

* Update python/paddle/distributed/auto_parallel/static/process_mesh_v2.py
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* refactor: refine detail

* refactor: refine detail

* fix: refine doc

---------
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* optimize the memory

* fix bug in static_build.cc

* fix bug when using logging

* change the static_build

* fix bug in windows

* fix code accordding to review

* remove empty block program

* update implementation

* [Custom Dice]add run_check support for custom device

* fix error msg

* fix typo

* update for all custom device

* fix

* add warning msg

* [WIP] Add mp_all_reduce asynchronize overlap.

* Fix some problems.

* Fix dw compute bug, and use a temporary solution to achieve overlap.

* Use fused_linear_param_grad_add to compute dw.

* Reformat ColumnParallel _overlap_linear. Use environment flags to
control following behaviors:
1. export Flags_mp_aysnc_allreduce=True to turn on mp async all_reduce
2. export Flags_skip_mp_c_identity=True to skip two c_identity operators
   in dygraph mode.
3. export Flags_fused_linear_param_grad_add to enable fused_linear_param_grad_add
   in ColumnParallel backward with mp async all_reduce.

* Polish code.

* Remove useless communication API.

* Fix some problems in mp_async_all_reduce and skip_c_identity.

* Add test cases.

* Remove environment variable Flags_fused_linear_param_grad_add in test case.

* Reset error threshold.

* Reset threshold in test case.

* Add useful log. Remove useless test cases.

* make params_grads order same bewteen dynamic and static mode

* revert inplace clip

* use sorted attribute to control

* tiny fix

* fix find loss_grad_op

* Improve GC for pipeline parallel

* Delete print

* fix bug of nop_op and sharding

---------
Co-authored-by: Nchenruibiao <chenruibiao@baidu.com>

* input.py

* Update python/paddle/nn/functional/input.py

* Update input.py

* Update all_gather.py

* Update all_gather.py

* temp commit

* distribute best cfg

* update metric extracting

* fix bugs of prune and reading log

* fix adding cfg bug

* reset status

* remove alarm and set logdir

* deepcopy ctx

* change alarm

* fix restart bug

* best no need alarm

* add gbs search, add gpu memory to history csv, add memory detect

* fix bug

* fix memory read bug; fix etcd connection bug

* fix memory read bug, add oom detection for all ranks

* fix read log and oom detaction, add error code for read log

* add unit test

* Update master.py

---------
Co-authored-by: Ncaozhou <caozhou@radi.ac.cn>

* remove the optimizer base and learning rate base

* fix bug

* fix bug

repacle fluid.io.load_inference_model, fluid.io.save_inference_model in fluid with 2.0 version  (#55345)

* repacle fluid.io.load_inference_model

* replace fluid.io.save_inference_model

* fix some bug

* fix some bugs of load & save model

* fix some bug

* fix test_inference_model_io bug

* fix word2vec_inference_model bug

* fix some bug

* fix valueError bug

* fix some bug

* fix a warning error

* for debug

* for debug

* fix io error

* fix test_wordvec_book error

* remove debug print

* fix load_var bug

* for debug cinn test

* revert cinn & fix inference_pass_test in windows

* fix some bugs

* revert cinn & fix inference_pass_test in windows

* for debug vars

* for debug

* fix quant_dequant_test

* fix some path errors

* remove fluid save/load

* fix incubate-fleet save

* move some from fluid.io to static.io

* move fluid apis

* fix type error

* remove static exponential_decay

* fix some import error

* remove nn.py

* fix some error

* fix type error

remove the AdamOptimizer、SGDOptimizer、MomentumOptimizer、ModelAverage、LookaheadOptimizer、FtrlOptimizer、DecayedAdagradOptimizer、DpsgdOptimizer in fluid and relocate the ExponentialMovingAverage、PipelineOptimizer、GradientMergeOptimizer and change optimizer base for LarsMomentumOptimizer and RecomputeOptimizer (#55970)

* change the optimizer base for SGDOptimizer

* change the optimizer base for SGDOptimizer

* replace the SGDOptimizer with SGD

* fix bug of sgd

* change the optimizer base for MomentumOptimizer

* fix the remaining tests

* remove the Momentum in fluid/optimizer.py

* fix bug

* fix bug

* fix bug

* fix bug

* Update test_resnet_cinn.py

* Update test_resnet_prim_cinn.py

* fix bug

* fix bug

* fix bug

* remove the ModelAverage in fluid

* remove the LookaheadOptimizer in fluid

* fix bug

* remove AdamOptimizer in fluid

* Update test_image_classification_fp16.py

* fix bug

* relocate the ExponentialMovingAverage in fluid

* restore the static api

* remove the FtrlOptimizer in fluid

* remove the DecayedAdagradOptimizer in fluid

* remove the DpsgdOptimizer in fluid

* fix bug

* fix codestyle

* fix bug

* fix bug

* relocate the PipelineOptimizer

* relocate the GradientMergeOptimizer

* fix bug

* fix bug

* fix bug

* fix doc

* Update __init__.py

* Update test_fleet_qat_meta_optimizer.py

* change optimizer base for LarsMomentumOptimizer

* fix bug

* fix conflict

* fix code-style

* fix sample codes

* fix bug

* fix bug

* fix cinn bug

* fix bug

* fix bug

* Update qat_optimizer.py

* Update __init__.py

* fix bug

* change optimizer base for RecomputeOptimizer

* fix bug

* fix bug

* Update test_imperative_optimizer_v2.py

* Improve GC for pipeline parallel

* Delete print

* open

* update

* make tcp store a global instance

* fix windows compile error