[cherry-pick][Hybrid Performance] Move the cast op of AMP which cast fp32 param to fp16 param to the optimizer (#34965) (#35296)
Co-authored-by: NWangXi <wangxi16@baidu.com>

* revert commit id 34212

* add trainer desc config to distributed strategy

* code style modified

* add fix op run order pass

* add ut for fix_op_run_order

* fix ci error

* improve coverage

* improve coverge again and fix cpu test case

* follow some comments

* dygraph sharding

* update unitest hybrid_parallel_communicate_group

* fix precision of mp

* fix bug of seed

* fix dp

* print group

* fix find_unused_parameters default value

* add raw program, test=develop

* update

* add model parallel support in dygraph

* pscore support heterps

* fleet cmake

* fleet wrapper

* macro

* solve conflict

* solve conflict

* add unitest

* paddle enforce

* unitest

* unitest

* unitest

* support control flow

* supoort sync_parameters_buffers

* fix the bug of sparse embedding

* add 1f1b scheduler for pp, test=develop

* remove the send/recv of tensor size, but users have to specify the shape of the received var explicitly.

* update, test=develop

* add tensor table

* Sharding add hybrid-dp feature

* update sharding in distributed_strategy

* update sharding unitest

* revise code format for sharding

* add sharding

* refine fleetrun.ps_launch

* update fleet run for multi device support

* ps_graph support ps-gpu

* fix heter save

* add heter save unittest

* fix unittest & simple code

* update fleetrun

* fix fleetrun

* fix launch barrier

* fix role maker

* add paddlecloud rolemaker unittest

* rename heter_worker_device_guard

* fix test_dist_fleet_heter_ctr & peformance update

* add adaptivelsgd

* Todo fix the code to avoid the conflict.

* rm auto from localsgd

add lars to fleet meta optimizer

* add cudnn related strategies to DistributedStrategy

add lamb to fleet meta optimizer

* add paddle.fleet.AsyncOptimizer
Co-authored-by: Ndongdaxiang <dongdaxiang@baidu.com>