* add p2p calc stream

* rm code

* rm code

* rm assert

* rm code

* support sharding stage1

* fix unittest

* format

* pass sharded sharding params_and_grads to inner_opt apply_pptimize

* change sharding gradient allreduce to reduce

* support save state_dict adptively and support sharding with mp

* fix sharding test

* test set_state_dict

* add more unit test

* fix global norm of mp case

* polish

* hack to calculate global norm in order to remove diff in calculating global norm values in HybridParallelClipGrad compared to dp

* remove print

* polish

* polish

* polish

* polish

* polish

* polish

* modify multiply_grad_node create

* modify build conflict

* add place choose

* ci segment fault

* clear branch

* opt memory

* rm args

* add rm ptr

* add timer to pp (#53831)

* [Hybrid Performance] Sharding stage 1 PP/VP overlap (#54312)

Co-authored-by: Ntianshuo78520a <707759223@qq.com>

* add log for memory stats

* fix string_split in einsum

* Set random seed for test_tensordot (#53004)

---------
Co-authored-by: NRuibiao Chen <chenruibiao@baidu.com>

* add host memory stats

* add ut

* Add large dim test of log_softmax

* fix

* delete bf16 of cross entropy in new frl

* delete bf16 of cross entropy grad

* add utest

* rm hack code

* add log overwrite flag

* use strtobool

Co-authored-by: Nsneaxiy <32832641+sneaxiy@users.noreply.github.com>

add utest

fix utest

* print modifed flags

* fix ref, opt print

* fix default getter

* fix ut