* convert multihead to oss

* fix:bug

* fix:delete const cast

* fix:don't support bias_qk

* add vit pass

* fix:convert bug and add preln_residual_bias

* support length=-1

* add UT for convert

* add no_bias_qk support for gpu_multihead_op

* delete infer_shape depends on bias_qk

* oss just can be used in T4 and A*

* fix:change api for ROCM CI

* convert_fp16 support multi block

* update

* update

* add fused_multi_transformer post_layer_norm

* add test post_layer_norm

* multihead matmul add fp16

* fix windows error

* fix rocm error

* fix rocm error

* remove cudaDeviceContext

* remove more template

* fix rocm compile

* remove alias name CUDADeviceContext

* fix compile

* fix tests

* revert changes

* remove cudaDeviceContext

* remove more template

* fix rocm compile

* add some fp16 op for kunlun resnet50 model
*test=kunlun

* tmp
*test=kunlun

* fused_fc_elementwise_layernorm support fp16

* fused_fc_elementwise_layernorm support double

* multi stream support handle lazy init.

* support eigen lazy init

* update

* fix ci problem

* Rename BOOST_GET macros

* Fix conflicts

* add xpu resnet_unit
*test=kunlun

* tmp
*test=kunlun

* Fix nan in fast_ln_fwd_kernel when cols > 1024

* delete blas

* conv_fusion

* fix init()

* delete test_device_context

* replace CPUDeviceContext with CPUContext

* fix test_scalar

* remove dot_op.cc

* fix compile

* 2nd part of transpose update

* add switch_auto_tune option.

* add some changes according to Ci

* refine the structure of auto_tune_base.

* merge develop changes

* reset the switch_set_range and change unittest of transpose auto-tune

* change the kernel auto-tune logits

* fused_gate_attention manual code in eager

* refine

* refine

* refine

* refine

* refine

* refine

* Add new attr of fused_multi_transformer

* fix format

* add note

* add in layer

* fixfixfixfix

* support more dimensions

* fix

* Support optional residual add in fused_attention and fused_feedforward.

* Add checkpoint and add the check of add_residual when pre_layer_norm is false.

* Add TODO and change the python api to add add_residual argument.

* Optimize prod's python implementation for dygraph.

* Change key_dim to head_dim.

* Add comment in unittest.

* Disable TF32 in unittest.

Co-authored-by: NLiu Yiqun <liuyiqun01@baidu.com>

* Polish codes and memory usage for fused_gate_attention.

* Fix wrong reduce_dims in fused_gate_attention when computing gradient of nonbatched_bias.