* [FlashAttn] add flash randomness control (#52902)

* add flash randomness control

* fix VLOG undefied

* [WIP] Integration flash attention 2 (#55758)

* Work for fa-2 padded fwd. Code to be cleaned.

* Work for fa2 unpadded fwd.

* Work for padded-bwd, dk get small diff on np.random.seed(0)

* Anyway I pass paddle's utest, except return softmax without dropout.

* Clean code.

* Modify interface.

* Clean code and add some check.

* Easy compile for dev.

* Fix ci.

* Fix ci-build.

* Add std c++17 option again.

* Limit max job when compiling fa2.

* Remove const_cast

* Add fwd params, to be cleaned.

* Clean code.

* Add bwd params.

* Clean code.

* Add enforce.

* Use v2.0.4

* Pass RNG state to fa2 capi

* Fix review.

* Add assert

* Skip compile for sm less than 80.

---------
Co-authored-by: NChitsing KUI <kuizhiqing@msn.com>

* add paddle.async_save to reduce time cost by checkpoint saving

* adapt save_for_auto_inference to paddle.async_save

* modify UT

* modify UT

* fix on cpu only version

* revert commit on save_auto_inference

* fix threading

* new_frl_shard_redece

* add mp guard

* add test

* make top_p_sampling supports threshold

* delete __nv_bfloat16

* fix new launch

* fix ps uit

* fix hybrid_parallel_sharding_model.py

* Update hybrid_parallel_sharding_model.py

* support add(x_float32, bfloa16_) or add(x_float32, y_float16)

* polisg

* support sharding stage1

* fix unittest

* format

* pass sharded sharding params_and_grads to inner_opt apply_pptimize

* change sharding gradient allreduce to reduce

* support save state_dict adptively and support sharding with mp

* fix sharding test

* test set_state_dict

* add more unit test

* fix global norm of mp case

* polish

* hack to calculate global norm in order to remove diff in calculating global norm values in HybridParallelClipGrad compared to dp

* remove print

* add timer to pp (#53831)

* [Hybrid Performance] Sharding stage 1 PP/VP overlap (#54312)

* add log for memory stats

* fix string_split in einsum

* Set random seed for test_tensordot (#53004)

---------
Co-authored-by: NRuibiao Chen <chenruibiao@baidu.com>

* add host memory stats

* add ut

* Add large dim test of log_softmax

* fix

* delete bf16 of cross entropy in new frl

* delete bf16 of cross entropy grad

* add utest

* rm hack code

add utest

fix utest

Co-authored-by: Ngongweibao <gongweibao@baidu.com>

* save env log for each worker

* fix ut

* Add digamma, dirichlet tests

* Fix code

* add erf FP16 test

1. modify set_value op, use Scalars to represent attr `values`, instead of a bunch of attributs of various types; (#52408)

2. add program converter and set_value op as an example, which provides the functionality to convert `paddle::framework::ProgramDesc` between old and new formats(the differences are mainly some operators with incompatible updates in the definition);
3. program version and operator version map now are always saved when serializing `paddle::framework::ProgramDesc` to identify the version;
3. provide an option `legacy_format=false` in  serialization of `paddle::framework::ProgramDesc`, it decided whether to convert ProgramDesc back to a legacy format, which is compatible for paddle 2.4.2 or earlier versions to load and execute;
4. deserialization of `paddle::framework::ProgramDesc` is now automatically detecting whether the bytes it receives is in legacy format(contains any of the operators that has been incompatibly updated and have any attribute of type `Scalar`) and convert it to new format. But if you want a faithful deserialization without the automatic conversion, you can use protobuf's deserialization instead. Though it is not recommended, it can be used for the purpose of testing.