* refactor reducer

* modify cmakelists

* solve conflicts

* rename group and update process_group

* fix bugs of ProcessGroupNCCL

* modify for CIs

* refactoring reducer

* [MLU] add fleet init api and collective api pytest for mlu

* fix no value for argument 'data_type' in method call

* add reducer

* fix bug of dp in pfp16

* fix topo

* fix data parallel when VOCAB var in program

* Added HCCL backend support in dynamic graph mode

* fix segmentation fault

* add ut

* add no_sync for parameters sync

* add pipeline for moe

* Add solutions to PyLayer which is unsupported in DataParallel

* modify note format for parallel.py

* modify docs of dataparallel

* add docs of dp with pylayer

* modify docs format

* modify example format

* change example of dp with pylayer

* add unittest for dp with pylayer

* modify ut

* merge latest codes

* update

* modify for CI-Coverage

* modify text-indent

* Add no_sync in data parallel for dynamic graph

* modify UT of no_sync

* delete test_parallel_dygraph_dataparallel_no_sync.py

* add test_parallel_dygraph_no_sync.py

* modify run_trainer_with_spawn in UTs

* Add UT of complex control flow in no_sync

* add specific descriptions and notes for no_sync

* check code style

* modify UT's TIMEOUT in CMakeLists.txt

* fix find_unused_parameters default value

* add clip/check

* add amp & clip grad in dygraph

* add logging

* support dp & mp

* fix sublayer error with include_sublayers=False

* add ut

* refactor include_sublayers related api

* fix ut

* fix ut of transformer

* fix ut of transformer

* remove useless code

* change sublayer api

* polish code

* add test for include_self=True

* support control flow

* supoort sync_parameters_buffers

* fix the bug of sparse embedding

* fix fleet for multi-stream

* fix memcpy for ncclid

* use sync to solve move operation

* add tensor_indices in AssignGroupBySize

* add rebuild group in reducer

* fix the bug of reducer in embedding

* add comment

* add reducer

* refine envent for memorycopy

* add concat&split for allreduce

* apply concat & split for fuse tensor

* fix nccl dep

* fix the untest, compile problem and ddp initialize problem

* fix untest for mac & add some comments & solve the repeated param in sublayers

* fix untest for windows & fix document

* polish parallel api impl & doc details

* add unittest for coverage

* remove spawn test in py2.7

* add parallel api into white list

* remove data parallel scale loss & apply collective_grads

* move apply in minimize

* fix failed unittests

* polish Program api doc & example

* polish CompiledProgram api doc & example

* polish ParallelEnv api doc & examples

* polish details, test=document_fix

* polish program doc details, test=document_fix

* polish details, test=document_fix

* fix note format error, test=document_fix

* add lost example, test=document_fix

* fix lost example, test=document_fix

* remove inplace argument

* fix sample code

* fix sample code

* init commit of new save/load

* fix failed unittests

* fix save_load_v2 unittest failed

* fix failed unittest & polish doc

* add tests for coverage

* add more tests & move static apis

* fix example code error

* polish emample code

* fix detail example code problem

* update set_dict method name & add aliases

* fix var name error

* fix alias formats

* use set_state_dict in unittest

* add decorator solve compatible problem

* polish decorator

* replace layer set_state_dict by patched method

* remove import monkey path layer

* fix import function error

* add unittest for coverage

* remove backend argument of init_parallel_env

* remove keep name table in transformer

* add cpu version check

* add skip unittest for init_parallel_env

* polish doc: remove func use & update example

* add dygraph parallel run interface

* polish implement & unified env property name

* add print config arg

* refactor init_parallel_env function

* Compatible with multiprocessing and launch modes

* set default trainer start port

* support run in python 2

* polish python2 support code

* remove python2 support

* refine launch import

* polish dome design details

* refactor api implemention & path

* use new method _set_expected_place

* add spawn unittest framework & mnist test

* add more unittests & doc

* fix unittest failed

* polish english doc

* self review and polish details

* refactor code by reviewer's comments

* fix unittest failed

* fix parallel_env unittest

* fix several typos

* fix error introduced when fixing typos

* add unpublic note for start_processes

* polish details by xiaoguang's comment

* verify correctly when spawn nprocs=-1

* refactor spawn & init_parallel_env design

* polish doc details

* open spawn unittests

* try to fix doc compile error

* try to fix unknown doc format error

* add skip unittest when not gpu

* Upgrade `no_grad` decorator

test=develop

- match torch decorator usage (i.e., with parenthesis)
- handle generator functions
- add `paddle.no_grad` alias

* Switch from `functools` to `decorator`

preserves signature

* Reword decorator usage note

* support selectedrows allreduce in multi-cards dygraph, test=develop

* remove useless import modules in unittests, test=develop

* add nccl cmake to get nccl version, test=develop

* add if-condition to compiled correctly, test=develop

* add detail version parseing for old nccl, test=develop

* polish camke details, test=develop

* fix remove test cmake error, test=develop

* fix cmake condition, test=develop

* change unittest camke list, test=develop

* fix unittest cmake rule, test=develop, test=framep0

* test=develop, test=document_fix

* test=develop, test=document_fix
Co-authored-by: Nswtkiwi <1208425345@qq.com>