* add usecache

* add p2p cache fix

* add cache

* fix bug in synchronize

* fix bug in synchronize

* support param group in sharding

* fix utest

* add debug information

* fix log

* fix log

* add detach for pp

* dp and sharding coexist

* dp

* skip CopyOrAdd when tmp grad is None (#55679)

* Optim fused linear grad add (#55927)

* make flash attn v1 available

* add deps error

* refine cmake dependencies

* fix cmake error

* [FlashAttn] add flash randomness control (#52902)

* add flash randomness control

* fix VLOG undefied

* [WIP] Integration flash attention 2 (#55758)

* Work for fa-2 padded fwd. Code to be cleaned.

* Work for fa2 unpadded fwd.

* Work for padded-bwd, dk get small diff on np.random.seed(0)

* Anyway I pass paddle's utest, except return softmax without dropout.

* Clean code.

* Modify interface.

* Clean code and add some check.

* Easy compile for dev.

* Fix ci.

* Fix ci-build.

* Add std c++17 option again.

* Limit max job when compiling fa2.

* Remove const_cast

* Add fwd params, to be cleaned.

* Clean code.

* Add bwd params.

* Clean code.

* Add enforce.

* Use v2.0.4

* Pass RNG state to fa2 capi

* Fix review.

* Add assert

* Skip compile for sm less than 80.

---------
Co-authored-by: NChitsing KUI <kuizhiqing@msn.com>

* Add fused_rope forward op (#54351)

* style

* more

* update ctest

* Update legacy_backward.yaml

* Update legacy_ops.yaml

* Update legacy_ops.yaml

* update

* update

* update for move

* Update the rope op according to the comments (#54985)

* Update multiary.cc

* Update __init__.py

* for int64_t and assert

* more

* remove useless assert first

---------
Co-authored-by: Nsneaxiy <sneaxiy@126.com>

* fix bug

* fix bug

* fix bug

* fix bug

* fix bug

* Fix test_resnet and test_resnet_v2 ut

* Remove ut

* Add virtual pp and dp overlap

* add sharding/dp overlap

* add dp/vpp overlap

* fix code

* fix log

* make sharding reduce mode by default

* Update dygraph_sharding_optimizer.py

* Update hybrid_parallel_optimizer.py

* Update pipeline_parallel.py

* add paddle.async_save to reduce time cost by checkpoint saving

* adapt save_for_auto_inference to paddle.async_save

* modify UT

* modify UT

* fix on cpu only version

* revert commit on save_auto_inference

* fix threading

* new_frl_shard_redece

* add mp guard

* add test

* make top_p_sampling supports threshold

* delete __nv_bfloat16

* fix new launch

* fix ps uit

* fix mea backward seg fault

* fix bias stride error

* fix MemoryEfficientAttentionGradInferMeta

* fix mea backward seg fault

* fix bias stride error

* fix hybrid_parallel_sharding_model.py

* Update hybrid_parallel_sharding_model.py

* refine dygraph_sharding_optimizer.py by sorting parameters

* Update dygraph_sharding_optimizer.py

Make FLAGS_sharding_sort_parameters=1 by default.

* support add(x_float32, bfloa16_) or add(x_float32, y_float16)

* polisg

* make FLAGS_force_align_vpp_grad_sum_order default to false

* polish code