* add usecache

* add p2p cache fix

* add cache

* support param group in sharding

* fix utest

* add debug information

* fix log

* fix log

* add detach for pp

* dp and sharding coexist

* dp

* fix bug

* fix bug

* fix bug

* fix bug

* fix bug

* Add virtual pp and dp overlap

* add sharding/dp overlap

* add dp/vpp overlap

* fix code

* fix log

* make sharding reduce mode by default

* Update dygraph_sharding_optimizer.py

* Update hybrid_parallel_optimizer.py

* Update pipeline_parallel.py

* new_frl_shard_redece

* add mp guard

* add test

* fix new launch

* fix ps uit

* refine dygraph_sharding_optimizer.py by sorting parameters

* Update dygraph_sharding_optimizer.py

Make FLAGS_sharding_sort_parameters=1 by default.

* support add(x_float32, bfloa16_) or add(x_float32, y_float16)

* polisg

* make FLAGS_force_align_vpp_grad_sum_order default to false

* polish code

* add p2p calc stream

* rm code

* rm code

* rm assert

* rm code

* support sharding stage1

* fix unittest

* format

* pass sharded sharding params_and_grads to inner_opt apply_pptimize

* change sharding gradient allreduce to reduce

* support save state_dict adptively and support sharding with mp

* fix sharding test

* test set_state_dict

* add more unit test

* fix global norm of mp case

* polish

* hack to calculate global norm in order to remove diff in calculating global norm values in HybridParallelClipGrad compared to dp

* remove print

* polish

* polish

* polish

* polish

* polish

* polish

* opt memory

* rm args

* add rm ptr

* add timer to pp (#53831)

* [Hybrid Performance] Sharding stage 1 PP/VP overlap (#54312)

* add utest

* rm hack code

add utest

fix utest