* [WIP] Add mp_all_reduce asynchronize overlap.

* Fix some problems.

* Fix dw compute bug, and use a temporary solution to achieve overlap.

* Use fused_linear_param_grad_add to compute dw.

* Reformat ColumnParallel _overlap_linear. Use environment flags to
control following behaviors:
1. export Flags_mp_aysnc_allreduce=True to turn on mp async all_reduce
2. export Flags_skip_mp_c_identity=True to skip two c_identity operators
   in dygraph mode.
3. export Flags_fused_linear_param_grad_add to enable fused_linear_param_grad_add
   in ColumnParallel backward with mp async all_reduce.

* Polish code.

* Remove useless communication API.

* Fix some problems in mp_async_all_reduce and skip_c_identity.

* Add test cases.

* Remove environment variable Flags_fused_linear_param_grad_add in test case.

* Reset error threshold.

* Reset threshold in test case.

* Add useful log. Remove useless test cases.

* [dygraph]unify _non_static_mode() in_dygraph_mode() and in_dynamic_mode()

* [dygraph]unify _non_static_mode() in_dygraph_mode() and in_dynamic_mode()

* [dygraph]unify _non_static_mode() in_dygraph_mode() and in_dynamic_mode()

* [dygraph]unify _non_static_mode() in_dygraph_mode() and in_dynamic_mode()

* [dygraph]unify _non_static_mode() in_dygraph_mode() and in_dynamic_mode()

* [dygraph]unify _non_static_mode() in_dygraph_mode() and in_dynamic_mode()

* fixed cyclic reference that caused patial import

* fixed bad change

* fix bad import

* fix bad import

* fix bad import

* fix ut failed caused by change in_dynamic_mode

* fix ut failed caused by change in_dynamic_mode

* fixed usage of in_dynamic_mode() or in_dygraph_mode()

* revert python3 to python in .pre-commit-config.yaml

* fix merge conflicts

* rename _varbase_creator as create_tensor

* rename _varbase_creator as create_tensor

* add seed control

* fix bug

* rem is_compiled_with_npu

* rem nup related code

* make lint happy

* rem test

* remove some tests

* Update grad_scaler.py

* fix an error

* autofix
Co-authored-by: NLiyulingyue <83450930+Liyulingyue@users.noreply.github.com>

* revert changes in python/paddle/distributed/fleet/utils/hybrid_parallel_util.py

* empty commit, trigger ci

* fix test_slice

---------
Co-authored-by: NSigureMo <sigure.qaq@gmail.com>

* Support ignore_index for c_softmax_with_cross_entropy_op.

* Polish code. Remove useless comments and add Testcase.

* Polish code for TestCase.

* Polish code.

* Polish code style.

* Polish code.

* Change loss calculation formula and ignore_index dtype.

* Polish TestCase.

* Fix bug of c_softmax_with_cross_entropy_op_xpu_op. Attribute 'ignore_index'
dtype is int64_t.

* Support ignore_index for c_softmax_with_cross_entropy_op.

* Polish code. Remove useless comments and add Testcase.

* Polish code for TestCase.

* Polish code.

* Polish code style.

* Polish code.

* Change loss calculation formula and ignore_index dtype.

* Polish TestCase.

* fix flash_attention

* Update mp_layers.py

* remove no used fluid beam_search_decoder

* move Layer and related helper to paddle.nn.common

* modify Layer references from dygraph.layers.Layer to paddle.nn.common.layers

* stash changge

* remove fluid layer_object_helper, layers.py

* remove fluid layers init

* add setip

* fix unitest

* delete layers in fluid.dygraph

* merge paddle.tensor.stat,py

* fix circle import

* fix curcle import

* remove redundant in_dygraph_mode import

* revoce paddle.nn.common.* in fluid.__init__

* recovery nn.rnn

* paddle.frame use lazy import import paddle.jit to avoid circle import

* remove left dygraph.layers ref

* merge develop

* fix import error

* fix test error

* fxi merge error

* fix test fluid.Layer

* fix test error

* fix test error

* fix import error

* fix import error

* fix comments

* fix circle import

* fix rnn import error

* fix circle import

* fluid clean: remove dygraph_utils._append_bias_in_dygraph

* fix func name and imports

remove fluid.initializer.UniformInitializer, ConstantInitializer, NormalInitializer, TruncatedNormalInitializer, XavierInitializer, BilinearInitializer, MSRAInitializer, NumpyArrayInitializer and calculate_gain.. (#49498)

* move UniformInitializer and ConstantInitializer

* more modify

* circular import resolved

* another circular import resolved?

* more circular import 2

* circular import 3

* change import paddle in metric.py

* BuildStrategy import from fluid

* modify the framework import path in common.py

* change rnn.py import, from static to original framework

* change import static in the nn folder

* default_main_program should import from common_ops_import

* add import paddle in param_attr.py

* use core not paddle module for using VarDesc

* another old uniform

* mistake that use Uniform instead of UniformInitializer

* modify UniformInitializer doc

* move fluid.NormalInitializer to nn.initializer.NormalInitializer

* remove import of Normal in fluid.layers.nn.py

* remove more import of old Normal

* remove more import of old Normal

* sample code modify and tests modify import

* is_listen_failed passing arg should be log file

* problem solved

* a mistake solved

* comments resoleved and remove paddle.fluid.initializer.TruncatedNormalInitializer

* remove paddle.fluid.initializer.XavierInitializer and paddle.fluid.initializer.MSRAInitializer

* remove paddle.fluid.initializer.BilinearInitializer NumpyArrayInitializer and set_global_initializer

* change fluid to static

* change static to fluid to avoid circular import in distributed_strategy.py

* fix example code and test_initializer

* ValueType

* sample code fix

* change set_global_initializer back to fluid

* put paddle.static.BuildStrategy.ReduceStrategy into the fuction to avoid circular import

* remove calculate_gain, delete BilinearInitializer and revert set_global_initializer

* change the time of using UniformInitializer, ConstantInitializer, NormalInitializer, TruncatedNormalInitializer, XavierInitializer, MSRAInitializer, NumpyArrayInitializer as few as possible

* fix argument incampatible

* fix more arg incompatible

* fix test_prelu_op_xpu.py Constant

* fix inaccurate doc

* more doc fix: default value

* rm op_function_generator

* rm op_func_generator.h

* rm op_function

* modify cmake

* rm op_function.h

* rm check for op_function_generator.cc

* reset imperative

* rm python part

* fix imperative

* lint

* lint

* modify legacy_c

* review

* modify

* modify legacy

* rm gen op_functions code

* reset framework

* rm core.ops for test

* core.ops->core.eager.ops.legacy

* not raiseError for xpu

* delete legacy dygraph code in python/paddle/distributed

* refine

* isort all files

* revert conflicting files

* revert conflicting files

* revert conflicting files

* add index sample fp16 support

* remove fluid APIs in distributed_strategy.py and role_maker.py

* Revert "remove fluid APIs in distributed_strategy.py and role_maker.py"

This reverts commit 223bbee990d3bf69e252fc3c0f19e3873550a264.

* remove fluid APIs in distributed_strategy.py and role_maker.py

* remove index sample op changes

* remove fluid APIs under fleet.base

* remove fluid APIs under fleet.layers.mpu

* remove fluid APIs under fleet.meta_optimizers

* fix fluid error

* fix util_factory.py

* reset fluid.io.load_inference_model API

[CodeStyle][F821] fix undefined variables due to missing imports, misspelled variable names (#47899)

* `hann` -> `_hann`

* `false` -> `False`

* a missing passed argument `reduce_all`

* some missing imports

* `device_type` -> `heter_device_type`

* `PKVClient` -> `KVClient`

* fix some typos and missing imports

* [CodeStyle][py2][U008] remove unnecessary args in `super()`

* remove remained args

* revert changes in test_pylayer_op

* Revert "revert changes in test_pylayer_op"

This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4.

* revert some changes in example code

* [CodeStyle][E712] use `if cond`/`if cond is True` for comparison with `True`

* revert changes in fluid

* revert unrelated file

* revert changes in norm

* revert changes in auto_parallel_amp

* fix norm and auto_parallel_amp

* revert a typo fix due to fixed at #47477

* update config

* re-blacken python code

* temporarily disable date and diff_py_file

* skip a format

* [CodeStyle][F401] remove unused import in python/paddle/distributed

* remove pass

* empty commit

* Fix ValueError: list.remove(x): x not in list for meta_optimizer_names.

Fix ValueError: list.remove(x): x not in list for meta_optimizer_names.

* Fix split import.

Fix split import.

* add noqa after meta_optimizers in factory

* restort collective ops

* expand `import *`

* add noqa after required imports

* try to fix APIs without core.ops

* Revert "try to fix APIs without core.ops"

This reverts commit 6172beaf601e84bf61f2490c12c4739f0edaa5eb.

* fix an increment

* empty commit

* add noqa after required imports

* expand `import *`, fix ci error
Co-authored-by: NShuangchi He <34329208+Yulv-git@users.noreply.github.com>

* refactor mp.

* update setup.py.

* update mp_layers.py for compatibility.

* add documents for mp_layers.py

* update init.py

* update collective.py.

* update.

* update mp_ops.py

* update.

* update code style.

* update code style.