* Fix the dowanload bug in the case of multiple machines (#29551)

* fix the dowanload bug
* add sort for ips

* Fix bug of matmul_v2 for broadcast case (#29599)

* fix bug of matmul_v2 for broadcast

* Rebuild group automatically in dynamic graph distributed (#29255)

* add tensor_indices in AssignGroupBySize

* add rebuild group in reducer

* fix error message of gather nd (#29521)

* Change the api of DataParallel and Fleet (#29224)

* fix fleet api doc

* add lars to fleet meta optimizer

* add lamb to proto

* add lamb to fleet meta optimizer

* fixed syntax bug

* fixed syntax bug

* fixed syntax error in lamb, add config setter of lamb in distributed_strategy

* trigger unitest to rerun

* add new unitest func for lamb

* revise unitest for lars and lamb

* revise dgc meta unitest

* revise lars document in distribute_strategy

* revise lars lamb document in distributed_strategy.py

* revise lars lamb document in distributed_strategy.py

* add weight decay exclude logic to lars

* restore optimzier.py

* restore optimizer.py as develop except lars

* add epsilon and exclude fn to distributed_sttrategy

* add lars epsilon

* revise unitest for fleet lars and lamb

* revise lars lamb unitest for CI coverage

* revise lars argument api

* revise lars argument api

* revise lars argument api

* revise api doc of lars

* fix op role

* add sharding save and add_sync_comm_for_test function

* add comm_analyse to utlis

* revise sharding_utils

* add sharding saving unittest

* revise sharding utils for unittest

* revise sharding en doc

* update sharding utils api

* add doc for sharding

* fixed bug in sharding var size count

* update varsize count in sharding

* fix sharding num_nccl_comm

* Revert "fix sharding num_nccl_comm"

This reverts commit d51587c15e9323acf226ddd36154275f0d1daf76.

* upgrade comment string to raw string

* fix string in

* fix string with ' '

* revert update on comments

* upgrade only necessary

* fix sample code checker

* fix comments with '''

* add sharding

* add get final strategy for user to print final strategy

* add adaptivelsgd

* Todo fix the code to avoid the conflict.

* rm auto from localsgd

* refine launch and distributed repr string for print

* add auto parallel L1 implementation
test=develop

* fix doc

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* test=develop

* fix doc, test=develop

* update localsgd doc
test=develop

* update localsgd doc
test=develop

* fix fleet dgc amp doc, test=develop

* fix, test=develop

* fix async configs
Co-authored-by: Nliuyi05 <gavin1332@gmail.com>
Co-authored-by: NWangXi <wangxi16@baidu.com>
Co-authored-by: NseiriosPlus <tangwei12@baidu.com>

* fix api document

* modified timeout value on windows and mac (#26690)

* add Local SGD algorithm referenced paper
test=develop

* add cudnn related strategies to DistributedStrategy

* add documentation for DistributedStrategy

* add feature to fleet2.0 role_maker, distribute_strategy, test=develop

* move paddle.fleet to paddle.distributed.fleet

add lamb to fleet meta optimizer

* add paddle.fleet.AsyncOptimizer
Co-authored-by: Ndongdaxiang <dongdaxiang@baidu.com>

Add dgc to fleet meta optimizer, rm dgc from optimizer all

* split meta optimizer files
* add graph execution in execution, update two properties in DistributedStrategy, unit tests for these features

* add gradient Merge optimizer to meta, test=develop

* refine strategy compiler and meta optimizers
make async as a_sync

* add more settings for distributed strategy
Basically, DistributedStrategy has several parts of configurations:
- BuildStrategy: the same as paddle.fluid.BuildStrategy, but the distributed arguments are moved out of BuildStrategy
- ExecutionStrategy: the same as paddle.fluid.ExecutionStrategy
- collective communication configs: nccl_comm_num, hierarchical allreduce and so on
- distributed algorithms: async_update(mainly used in PS), lars, lamb and so on

refactor fleet api under paddle.fleet
update DistributedStrategy

test=develop

* add paddle.fleet.DistributedStrategy for 2.0