* support hybrid parallel inference helper class

* [ROCM] fix top_k_v2 with large shape

* [ROCM] fix top_k_v2 with large shape

* implemented clip op bf16/fp32

* added skipping if not cpu or bf16

* CI rerun after bf16 package change

* added parentheses to ensure formatting

* refactor gc, test=develop

* refine, test=develop

* refine, test=develop

* refine, test=develop

* gc each tensor, test=develop

* refine, test=develop

* change metaclass of Layer from pybind11_builtins.pybind11_type to type

* fix cast

* add ut

* sharding support dp

* sharding support mp

* sharding support pp

* fix warning error , test=document_fix

* fix warning error , test=document_fix

* fix warning error , test=document_fix

* fix warning error , test=document_fix

* fix warning error , test=document_fix

* fix warning error , test=document_fix

* fix warning error , test=document_fix

* add test_cumprod_op

* Revert "add test_cumprod_op"

This reverts commit c96cf6dff5d09ae7d8cc72c1e8ae4369a153aa19.

* recommit

* add error message

* test input(x) initialize

* test use cpu

* update test code

* add test type

* add test case

* solve ci problem

* add complex case test

* add complex case test

* fix review problem

* fix conflict

* fix some docs

* change test case

* change test case

* fix review problems again

* fix docs

* fix inclusivescan bug

This PR supports gradient clip (ClipGradByGlobalNorm) when training with AMP(auto mixed precision).

* add llvm::SmallVector to paddle

* rename small vector file

* merge paddle small vector to one file

* add small_vector_test

* modify smallvector test argument type

* add string header

* import ska flat_hash_map

* add define NOMINMAX macro to fix windows build failed bug

* add brackets to std::max in flat_hash_map

* move flat_hash_map directions

* modify namespace to paddle

* modify namespace to paddle

* modify namespace to paddle

* modify namespace to paddle

* rm not used map.h and replace with op_info

* add as_extra to dropout op and lrn op

* refine details

* fix dropout op maker

* fix extra op for expand, expand_as, tile, unstack

* Update expand_v2_op.cc

* fix bn/in/squeeze/syncbn extra

* update bn

* update

* update

* add opdef extra

* add reduce mean

* update style

* fix scatter gather bug:

* fix windows ci

* conv3d

* remove const_cast

* modify ut

* disable dynamic shape for trt6.0

* remove trt5

* add asExtra for nce op

* fix unittest error in macos

* remove asExtra for is_test

* mark extra attr for unsqueeze2

* debug for inference