* new_frl_shard_redece

* add mp guard

* add test

* make top_p_sampling supports threshold

* delete __nv_bfloat16

* fix new launch

* fix ps uit

* fix mea backward seg fault

* fix bias stride error

* fix MemoryEfficientAttentionGradInferMeta

* fix mea backward seg fault

* fix bias stride error

* fix hybrid_parallel_sharding_model.py

* Update hybrid_parallel_sharding_model.py

* refine dygraph_sharding_optimizer.py by sorting parameters

* Update dygraph_sharding_optimizer.py

Make FLAGS_sharding_sort_parameters=1 by default.

* support add(x_float32, bfloa16_) or add(x_float32, y_float16)

* polisg

* make FLAGS_force_align_vpp_grad_sum_order default to false

* polish code

* add p2p calc stream

* rm code

* rm code

* rm assert

* rm code

* support sharding stage1

* fix unittest

* format

* pass sharded sharding params_and_grads to inner_opt apply_pptimize

* change sharding gradient allreduce to reduce

* support save state_dict adptively and support sharding with mp

* fix sharding test

* test set_state_dict

* add more unit test

* fix global norm of mp case

* polish

* hack to calculate global norm in order to remove diff in calculating global norm values in HybridParallelClipGrad compared to dp

* remove print

* polish

* polish

* polish

* polish

* polish

* polish

* modify multiply_grad_node create

* modify build conflict

* add place choose

* ci segment fault

* clear branch

* opt memory

* rm args

* add rm ptr

* add timer to pp (#53831)

* [Hybrid Performance] Sharding stage 1 PP/VP overlap (#54312)

Co-authored-by: Ntianshuo78520a <707759223@qq.com>

* add log for memory stats

* fix string_split in einsum

* Set random seed for test_tensordot (#53004)

---------
Co-authored-by: NRuibiao Chen <chenruibiao@baidu.com>

* add host memory stats

* add ut

* Add large dim test of log_softmax

* fix