* skip CopyOrAdd when tmp grad is None (#55679)

* Optim fused linear grad add (#55927)

* make flash attn v1 available

* add deps error

* refine cmake dependencies

* fix cmake error

* [FlashAttn] add flash randomness control (#52902)

* add flash randomness control

* fix VLOG undefied

* [WIP] Integration flash attention 2 (#55758)

* Work for fa-2 padded fwd. Code to be cleaned.

* Work for fa2 unpadded fwd.

* Work for padded-bwd, dk get small diff on np.random.seed(0)

* Anyway I pass paddle's utest, except return softmax without dropout.

* Clean code.

* Modify interface.

* Clean code and add some check.

* Easy compile for dev.

* Fix ci.

* Fix ci-build.

* Add std c++17 option again.

* Limit max job when compiling fa2.

* Remove const_cast

* Add fwd params, to be cleaned.

* Clean code.

* Add bwd params.

* Clean code.

* Add enforce.

* Use v2.0.4

* Pass RNG state to fa2 capi

* Fix review.

* Add assert

* Skip compile for sm less than 80.

---------
Co-authored-by: NChitsing KUI <kuizhiqing@msn.com>

* Add fused_rope forward op (#54351)

* style

* more

* update ctest

* Update legacy_backward.yaml

* Update legacy_ops.yaml

* Update legacy_ops.yaml

* update

* update

* update for move

* Update the rope op according to the comments (#54985)

* Update multiary.cc

* Update __init__.py

* for int64_t and assert

* more

* remove useless assert first

---------
Co-authored-by: Nsneaxiy <sneaxiy@126.com>

* make top_p_sampling supports threshold

* delete __nv_bfloat16

* fix mea backward seg fault

* fix bias stride error

* fix MemoryEfficientAttentionGradInferMeta

* fix mea backward seg fault

* fix bias stride error

* support add(x_float32, bfloa16_) or add(x_float32, y_float16)

* polisg

* support sharding stage1

* fix unittest

* format

* pass sharded sharding params_and_grads to inner_opt apply_pptimize

* change sharding gradient allreduce to reduce

* support save state_dict adptively and support sharding with mp

* fix sharding test

* test set_state_dict

* add more unit test

* fix global norm of mp case

* polish

* hack to calculate global norm in order to remove diff in calculating global norm values in HybridParallelClipGrad compared to dp

* remove print

* modify multiply_grad_node create

* modify build conflict

* add place choose

* ci segment fault

* clear branch

* opt memory

* rm args

* add rm ptr

* add timer to pp (#53831)

* [Hybrid Performance] Sharding stage 1 PP/VP overlap (#54312)

* add log for memory stats

* fix string_split in einsum

* Set random seed for test_tensordot (#53004)

---------
Co-authored-by: NRuibiao Chen <chenruibiao@baidu.com>

* add host memory stats

* add ut

* delete bf16 of cross entropy in new frl

* delete bf16 of cross entropy grad

Co-authored-by: Nsneaxiy <32832641+sneaxiy@users.noreply.github.com>

* print modifed flags

* fix ref, opt print

* fix default getter

* fix ut

Co-authored-by: Ngongweibao <gongweibao@baidu.com>

* Fix the calculation of layer_norm_bwd

* fix

Co-authored-by: Ntianshuo78520a <707759223@qq.com>