* support sharding stage2 + mp hybrid_parallel.

* fix the group of check_nan_inf.

* update hcg.

* [CodeStyle][py2][U008] remove unnecessary args in `super()`

* remove remained args

* revert changes in test_pylayer_op

* Revert "revert changes in test_pylayer_op"

This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4.

* revert some changes in example code

* add unbalanced data

* fix utest

* [CodeStyle][E711] use `is`/`is not` for comparison with `None`

* `self.assertTrue($A is None)` -> `self.assertIsNone($A)`

* `self.assertTrue($A is not None)` -> `self.assertIsNotNone($A)`

* `self.assertFalse($A is None)` -> `self.assertIsNotNone($A)`

* `self.assertEqual($A, None)` -> `self.assertIsNone($A)`

* `self.assertNotEqual($A, None)` -> `self.assertIsNotNone($A)`

* [CodeStyle][E712] use `if cond`/`if cond is True` for comparison with `True`

* revert changes in fluid

* revert unrelated file

* revert changes in norm

* revert changes in auto_parallel_amp

* fix norm and auto_parallel_amp

* revert a typo fix due to fixed at #47477

* [CodeStyle][py2] remove `six` package (part2)

* six.ensure_str

* remove unused `import six`

* remove six from BUILTIN_LIKELY_MODULES

* remove six in example code

* remove some decode

* try to fix example code

* fix MockEtcdClient get/get_prefix returns data type

* fix MockEtcdClient get_prefix returns data

* fix MockEtcdClient get returns data

* remove `six` in pypi and conda requirements

* fix MockEtcdClient add_watch_callback/add_watch_prefix_callback returns data type

* refine MockEtcdClient

* add fused_allreduce_gradients_with_group

* add scale

* fix ci

* Fix F522: remove unused arguments

* add redirect_stderr argument in _run_cmd

* update config

* re-blacken python code

* temporarily disable date and diff_py_file

* skip a format

* Fix W605 in tools folder by adding escape symbols

* Fix W605 in incubate and some other folders

* Fix W605 in /fluid/test folders

* Update tools/analysisPyXml.py
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* Add some changes to manual and auto escape symbols

* revert changes in transformer.py

* Fix new code with W605 error: add escape symbols

* revert changes in transformer.py

* revert changes in transformer.py
Co-authored-by: NNyakku Shigure <sigure.qaq@gmail.com>

* add test for stage2 + dp

* update test for stage2 + dp.

* update.

* update.

* Fix bug of reduce_sum op. When input.numel() > INT32_MAX, its result
is wrong.

* Support allow_partial switch, which can be configure in
pipeline_configs. If sent tensor are not the same from
different hosts, they shouldn't been sent partially and
then concated as a whole tensor.

* Change name allow_partial to enable_partial_send_recv.

* Add global variable _enable_partial_send_recv

* Fix bug of reduce_sum op. When input.numel() > INT32_MAX, its result
is wrong.

* support pure bfloat16

* support bf16 linear

* update PR to pass CI

* tiny fix where_grad_kernel.cu

* Support bfloat16 type for reducer and sharding.

* Fix some bug.

* Polish code.

* Polise code.

* Add bfloat16 datatype in fill_grad kernels.
Co-authored-by: Nsneaxiy <sneaxiy@126.com>

* combine dp and stage2 hybrid parallel.

* update condition.

* [Zero-Dim] support input 0D Tensor for unary api

* fix CI

* [CodeStyle][F401] remove unused import in python/paddle/distributed

* remove pass

* empty commit

* Fix ValueError: list.remove(x): x not in list for meta_optimizer_names.

Fix ValueError: list.remove(x): x not in list for meta_optimizer_names.

* Fix split import.

Fix split import.

* add noqa after meta_optimizers in factory

* restort collective ops

* expand `import *`

* add noqa after required imports

* try to fix APIs without core.ops

* Revert "try to fix APIs without core.ops"

This reverts commit 6172beaf601e84bf61f2490c12c4739f0edaa5eb.

* fix an increment

* empty commit

* add noqa after required imports

* expand `import *`, fix ci error
Co-authored-by: NShuangchi He <34329208+Yulv-git@users.noreply.github.com>

* Update README.md

* Update README.md

* Fix E713: convert 'not ... in' to 'not in'

* Update README.md

* Update README.md

* Fix F541 by converting f-string to normal strings

* [CodeStyle] remove all future import

* revert test_error.py

* restore future import in example code