* update TestAPI arguments to enable param_attr and bias_attr in test_layer_norm_op

* add bf16 condition in test_layer_norm_op

* add fast_math condition

* Fix LayerNormForward perf issue

* Bugfix, fast_layer_norm OOB

* apply pre-commit

---------
Co-authored-by: NShijie Wang <jaywan@nvidia.com>

* block bf16 tests on ROCM

* block more bf16 tests on ROCM

* some unittest cases doesn't have kernels on ROCm

* some unittest cases doesn't have kernels on ROCm

* fix code style

* support layer_norm prim op bf16 dtype

* polish code

* resolve conflict

* mv all unittests test

* fix error

* fix error

* fix

* fix

* del unittests

* fix paddle_build.sh

* fix

* fix test

* fix add test

* fix

* fix

* fix

* merge develop

* fix

* fix

* fix

* fix

* fix

* merge develop

* fix test_async_read_write

* fix test_async_read_write

* merge develop

* fix

* fix import legacy_test

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix bug

* fix

* fix coverage test bug

* fix

* fix

* fix

* fix

* fix

* fix code sstyle

* fix code

* fix code

* fix

* fix

* fix

* del test_sequence_enumerate_op.py

* fix

* support layer_norm prim and cinn test

* enable cinn test

* fix merge conflict

* polish input for check_output_with_place

* fix merge conflict

* add more test case

* fix merge conflict

* polish test case

* polish op_test

* change ln_g rules

* modify scale is none case

* modify scale is none case

* add public_python_api for check prim

* modify setoutputgrad and fp64bug

* add todo & delete log

* recover

* fix some errors

* recover

* recover

* recover

* recover

* fix merge conflicts

---------
Co-authored-by: Nwangruting <wangruting@baidu.com>

* delete old dygraph op test

* first commit

* fix code bugs in for_loop

* fix bugs in cuLoadAddStridedInputs.

* optimization for LayerNormBackwardComputeGradInput

* add unitest for validating the optimization

* fix windows ci error

* first commit

* main codes has been developed

* fix all bugs

* add vectorize input&output

* a test for optimization_of_layer_norm_fwd

* add some changes

* fix memory coalesced access for more optimization.

* fix addition ctest error

* fix according to ci-approval

* remove change on slice

* replace paddle.fluid.layers.data and remove io.data

* partial commit

* partial commit

* partial commit

* partial commit

* partial commit

* partial commit

* remove data in fluid.layers.io.__all__

* fix errors

* fix unitests

* fix unitest

* fix unitests

* fix unitest

* fix unitest

* fix unitests

* fix unitest

* fix test_layers unitests

* fix typro

* fix unitest

* fix unitest

* fix unitest

* fix typro

* fix unitest test_model_cast_to_bf16

* fix test_reducescatter

* fix collective unitest

* fix collective unitests

* fix collective unitests

* add coverage

* fix add layers.data

* re run ci

* fix some typro

* fix samplecode error

* fix samplecode error

* remove paddle.fluid.layers.layer_norm

* templatedoc import from paddle.fluid.layers.layer_function_generator

* del import of fluid.layers.layer_norm in __init__.py

* add import of ..common.layer_norm in __init__.py

* fix bug in UT

* fix doc

* [remove fluid] drop_out PI

* [remove fluid] drop_out PI

* [remove fluid] drop_out layernorm

* [remove fluid] drop_out layernorm

* [remove fluid] drop_out layernorm

* [remove fluid] drop_out layernorm

* isort all files

* revert conflicting files

* revert conflicting files

* revert conflicting files

* revert conflicting files

* revert conflicting files

* update config

* re-blacken python code

* temporarily disable date and diff_py_file

* skip a format

* [CodeStyle] remove all future import

* revert test_error.py

* restore future import in example code

[CodeStyle][NPU] use np.testing.assert_allclose instead of self.assertTrue(np.allclose(...)) (part 1) (#44988)

* autofix

* try resolve precision issues

* revert some changes

* clean some `err_msg`

* 0.0001 -> 1e-4

* update commented assert code

* try to fix some shape errors

* `numpy` -> `np`

* empty commit, trigger kunlun ci, test=kunlun

* empty commit, retrigger kunlun ci, test=kunlun

* empty commit, trigger kunlun ci, try fix npu memcpy_h2d, test=kunlun

* try fix npu import error, test=kunlun

[CodeStyle] use np.testing.assert_array_equal instead of self.assertTrue(np.array_equal(...)) (#44947)

* automatically fix

* update comments

* numpy -> np

* self.assertEqual(..., True)

* wrong usage (err_msg=True)

这不是修复导致的错误，这些是原来 `self.assertTrue(..., True)`
的错误用法，因此在修复后将其认为位置参数 `err_msg`

* some missing fix

* use yapf to format all python file

* yapf exclude two unittests file for they rely on writing and reading file, and format will break them

* disable diff_py_file because too many diff files cause command following failed

* update

* fix bugs; test=develop

* update; test=develop

* fix test compile error; test=develop

* fix cpu compile error; test=develop

* fix test error; test=develo

* fix layer_norm_op plugin error; test=develop

* fix error; test=develop

* fix test bug; test=develop

* update; test=develop

* polish code; test=develop

* fix bugs; test=develop

* remove unused depency; test=develop

* polish code; test=develop

* add layer norm

* add p norm

* add reduce sum

* refine layer norm register bf16 for cudnn811

* add bf16 cast for hip

* add unittest

* refine rocm

* refine layer_norm unittest

* refine reduce op

* refine unittest

* enhance atol for reduce unittest

* Optimize layer_norm fwd when cols is 1024.

* support multi precision update for LAMB

* hide some api

* fix ci uts

* fix lamb output of dygraph

* remove some changes to some PR

* try to fix Py3 CI compile error

* fix test_imperative_optimizer, add lars ut, add layer_norm ut

* fix ut, fix format

* fix ut

* fix windows ci

The bug is that access to mean and var is incorrect, and the array will be out of bounds: the shape of mean and var is [batch_size], and the range of thread idx is 0~feature_size, so mean[idx] and var[idx] is incorrect.

When batch_size=1, the correct access is mean[0] and var[0], and a unit test with batch_size=1 is added.

* add fp16 for layer_norm op

* revert layernorm api

* fix forward

* fix forward

* fix backward for layernorm with fp16

* fix unit test for layernorm with fp16

* fix with_mkldnn compile error for layernorm with fp16

* 1. revert to PADDLE_ENFORCE_NOT_NULL, 2. change static_cast<float> to static_cast<U>

* fix with_mkldnn compile error for layernorm with fp16

* fix with_mkldnn compile error for layernorm with fp16
Co-authored-by: Nzhiqiu <chenqiuliang@baidu.com>

restrict block num of layer_norm_grad cuda kernel to 128, test=develop

* copy stop_gradient property in clone, test=develop

* fix pruning issue, test=develop

* fix kLookupTablePath issue, test=develop

* sync program, test=develop

* copy subblocks, test=develop

* add unittests, test=develop

* Fix docs of gru_unit and dynamic_gru.

Fix basic_gru in rnn_impl.py.

Add error messages for param_attr setting in layer_norm api.

Add int64 dtype for expand.

test=develop

* Reopen unit-tests of basic_gru/basic_lstm in rnn_impl.py.
test=develop

* Add unit test for layer_norm api.
test=develop

* Remove the deprecated gru doc fix. test=develop

* Fix basic_gru test coverage. test=develop

* Update API.spec. test=develop

* Update API.spec. test=develop

* Fix test_basic_gru coverage test. test=develop

* Update test_basic_gru in test_layers to use fluid.data
test=develop

* Update test_basic_gru for coverage. test=develop

* Add fc_elementwise_layernorm_fuse pass and unittest.

* Add fused_fc_elementwise_layernorm op and its GPU kernel.
test=develop

* Apply fc_elementwise_layernorm_fuse_pass to GPU inference.

* Add the setting of attrs in the definition of binary_op.
test=develop

* Add comment.

* Implement the unittest.
test=develop

* Change the unittest name of layer_norm.
test=develop

* Enable ngraph layer_norm operator

test=develop

* Disable/Enable cuda, new unit-test test=develop

* Fix use_cudnn test=develop

* Fixed test_layer test, new funciton is added test=develop

* set use_cudnn by default test=develop