* fix fleet util and gloo

* fix worker endpoints

* fix

* fix UT

* fix gloo

* fix gloo

* update gloo

* update gloo

* update gloo

* update gloo

* update gloo

* fix gloo wrapper for hdfs

* add file gloo and UT

* fix UT

* fix UT

* fix UT

* hide public method of RoleMaker

* fix UT

* GPU fleetrun support gloo

* parameterserver fleetrun support gloo

* add UT

* add UT

* fix UT

* fix get server endpoint

* fix get server endpoint

* fix UT

* hide public method of rolemaker

* hide public method of rolemaker

* hide public method of rolemaker

* Update test_fleet_rolemaker_new.py

* hide public method of rolemaker

* hide public method of rolemaker

* fix worker endpoints

* fix gloo wrapper for hdfs

* GPU fleetrun support gloo

* parameterserver fleetrun support gloo

* fix get server endpoint

* test=develop, util documents

* fix fleet dynamic-mode en-doc;test=develop

* add adaptivelsgd

* Todo fix the code to avoid the conflict.

* fix ports conflict when launching multi-nodes in paddlecloud;test=develop

* add DISTRIBUTED_TRAINER_ENDPOINTS env for cloud;test=develop

* fix strategy, test=develop

* fix can_apply

* rm auto from localsgd

* parameter_server_optimizer support auto_strategy

add lars to fleet meta optimizer

* refine launch and distributed repr string for print

* test=develop, add ps auto

* fix the localsgd

* add auto parallel L1 implementation
test=develop

support heter-xpu-ps

* fix doc

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* fix doc, test=develop

* update localsgd doc
test=develop

* update localsgd doc
test=develop

* fix fleet dgc amp doc, test=develop

* fix, test=develop

* fix async configs
Co-authored-by: Nliuyi05 <gavin1332@gmail.com>
Co-authored-by: NWangXi <wangxi16@baidu.com>
Co-authored-by: NseiriosPlus <tangwei12@baidu.com>

paddle.distributed.fleet supports dynamic graph execution.

* print detailed and clear log infos; test=develop

* support running python train.py for fleet-task; test=develop

* fix sample codes, test=develop

* fix fluid.embedding

* remove backend argument of init_parallel_env

* remove keep name table in transformer

* add cpu version check

* add skip unittest for init_parallel_env

* polish doc: remove func use & update example

* add FleetAPI doc
Co-authored-by: Nliuyuhui <liuyuhui@baidu.com>

* Support Heter Parameter Server

* fix api document

* add dygraph parallel run interface

* polish implement & unified env property name

* add print config arg

* refactor init_parallel_env function

* Compatible with multiprocessing and launch modes

* set default trainer start port

* support run in python 2

* polish python2 support code

* remove python2 support

* refine launch import

* polish dome design details

* refactor api implemention & path

* use new method _set_expected_place

* add spawn unittest framework & mnist test

* add more unittests & doc

* fix unittest failed

* polish english doc

* self review and polish details

* refactor code by reviewer's comments

* fix unittest failed

* fix parallel_env unittest

* fix several typos

* fix error introduced when fixing typos

* add unpublic note for start_processes

* polish details by xiaoguang's comment

* verify correctly when spawn nprocs=-1

* refactor spawn & init_parallel_env design

* polish doc details

* open spawn unittests

* try to fix doc compile error

* try to fix unknown doc format error

* add skip unittest when not gpu

* modified timeout value on windows and mac (#26690)

* add Local SGD algorithm referenced paper
test=develop

add collective op for cpu using gloo and paddle.distributed.* apis

* add cudnn related strategies to DistributedStrategy

* solve the initial configuration about fleet and rolemaker
Co-authored-by: NseiriosPlus <tangwei12@baidu.com>

* add documentation for DistributedStrategy

* consider the combination of different strategies to work together