[Kunlun]Multi xpu dygraph performance optimization , add distributed.spawn support for multi xpu and some bug-fixes (#31130)

* fix python full coverage decrease issue

* fix

prepare remove grad op and kernel script.
update Paddle_CI_Inference pipeline.

* fix modified_retry_method_only_win

* fix bug

* fix retry bug on windows

* improve performance of depthwise_conv2d

* add unittest

Fix wrong code comment

Fix Read-Only Attribute as while_loop Output:

Usually, our convert_while_loop will be like:
```
    [a, b, c] = paddle.jit.dy2static.convert_while_loop(
            condition_name, body_name, [a, b, c])
```
where a, b, c are in loop_var_names.

However, if loop_var_names contains property such as foo.x, we cannot
assign the attribute as output of convert_while_loop because Python
property is a kind of read-only attribute. To handle the case, we replace
the attributes which are output of convert_while_loop with generated
variables, then if we know the attribute is not read-only at runtime, we
assign the attribute. The created statements are like:
```
    [a, b, __attribute_variable_1] = paddle.jit.dy2static.convert_while_loop(
            condition_name, body_name, [a, b, foo.x])
    if not isinstance(getattr(type(foo), x, None), property): foo.x = __attribute_variable_1
```

* [ROCM] update fluid elementwise op for rocm (part10), test=develop

* update, test=develop

* address review comments, test=develop

* [ROCM] update fluid operators for rocm (part3), test=develop

* fix clang format error, test=develop

* add attrs deformable_groups

* fix sycn training error

Change-Id: Ie2feebcf0b5b2984fd59cfcdde0c817840e203d2

* [ROCM] update fluid operators for rocm (part5), test=develop

* address review comments, test=develop

* fix typo, test=develop

* Decrease threshold for failed ut retry

* retry Method upgrade

* second method upgrade

* fix error

* Remove the comment lines

* test for modified_retry_times

* fix error

* fix some error

* fix error

* fix error

* remove test content

* fix error

* Reduce duplicate code

* fix more than 10 ut failed bug

* fix more than 10 ut failed  bug on mac

* add n-d input support for trt scale converter

* add flatten for ut

* fix dims

* support trt serialize when load model from memory

* delete conv_bn_fuse_pass before tensorrt, with which trt serialize engine id is not stable

* Revert "delete conv_bn_fuse_pass before tensorrt, with which trt serialize engine id is not stable"

performance degradation, fix in the future

This reverts commit fa6cd17e60b15df351efda379ddd00e9e9c1fea9.

* add delete conv_bn

* delete path when delete_cache_files

* lamb_op_xpu;test=kunlun

* modify lamb_op_xpu.cc;test=kunlun

* delete atol lamb_op_xpu; test=kunlun

* update xpu.cmake;test=kunlun

* test_error 1e-5,lamb_op_xpu;test=kunlun

* error1e-5,lamb_op_xpu,test=kunlun

* delete atol lamb_xpu;test=kunlun

* modify atol,lamb_op_xpy;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu, XPUOptest;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu,modify xpu_cmake; test=kunlun

* lamb_op_xpu;test=kunlun

* lamb_op_xpu,modify xpucmake;test=kunlun

* topo and memory performance for heterps; test=develop
* add trainwithprofiler in heter trainier; test=develop