* multihead matmul add fp16

* fix windows error

* fix rocm error

* fix rocm error

* remove cudaDeviceContext

* remove more template

* fix rocm compile

* remove alias name CUDADeviceContext

* fix compile

* fix tests

* revert changes

* remove cudaDeviceContext

* remove more template

* fix rocm compile

* add some fp16 op for kunlun resnet50 model
*test=kunlun

* tmp
*test=kunlun

* fused_fc_elementwise_layernorm support fp16

* fused_fc_elementwise_layernorm support double

* multi stream support handle lazy init.

* support eigen lazy init

* update

* fix ci problem

* Rename BOOST_GET macros

* Fix conflicts

* add xpu resnet_unit
*test=kunlun

* tmp
*test=kunlun

* Fix nan in fast_ln_fwd_kernel when cols > 1024

* delete blas

* conv_fusion

* fix init()

* delete test_device_context

* replace CPUDeviceContext with CPUContext

* fix test_scalar

* remove dot_op.cc

* fix compile

* 2nd part of transpose update

* add switch_auto_tune option.

* add some changes according to Ci

* refine the structure of auto_tune_base.

* merge develop changes

* reset the switch_set_range and change unittest of transpose auto-tune

* change the kernel auto-tune logits

* fused_gate_attention manual code in eager

* refine

* refine

* refine

* refine

* refine

* refine

* Add new attr of fused_multi_transformer

* fix format

* add note

* add in layer

* fixfixfixfix

* support more dimensions

* fix

* Support optional residual add in fused_attention and fused_feedforward.

* Add checkpoint and add the check of add_residual when pre_layer_norm is false.

* Add TODO and change the python api to add add_residual argument.

* Optimize prod's python implementation for dygraph.

* Change key_dim to head_dim.

* Add comment in unittest.

* Disable TF32 in unittest.

Co-authored-by: NLiu Yiqun <liuyiqun01@baidu.com>

* Polish codes and memory usage for fused_gate_attention.

* Fix wrong reduce_dims in fused_gate_attention when computing gradient of nonbatched_bias.

* Support head_dim = 96 in fused_multi_transformer in PLATO-XL

* add notes

* extend forward fast_ln_kernel to support more column values.

* support weight transpose

* add ut

* add template

* fix transpose error

* fix transpose_comment

* add api tests

* add skipif

* add doc

* replace dropout_is_test with is_test.
* improve atol on a100.