* remove -DSUPPORTS_CUDA_FP16 in cuda.cmake

* comile with cuda9

* add some unittest

* notest;test=coverage

* add unittest for trt plugin swish && split

* update ernie unittest

* fix some error message

* remove repeated judgement of CUDA version in mbEltwiseLayerNormOpConverter

* fix comile errror when CUDA_ARCH_NAME < Pascal"

* fix comile error

* update unittest timeout

* compile with cuda9

* update error msg

* fix code style

* add some comments

* add define IF_CUDA_ARCH_SUPPORT_FP16

* rename IF_CUDA_ARCH_SUPPORT_FP16 to CUDA_ARCH_FP16_SUPPORTED

* add symlink force for unittest

* open unittest

* fix interpolate example, test=develop;test=document_fix

* fix format, test=develop, test=document_fix

* update upsample doc, test=develop, test=document_fix

* 1. grid_sample
1.1 fix has_print
2. conv1d_transpose
2.1 fix code_example error
3. conv1d
4. affine_grid
4.1 has_print
4.2 has_disable_static
5. Conv1DTranspose
5.1 fix code_example error
5.2 has_disable_static
6. Conv1d
6.1 code_example
6.2 has_disable_static

* update, test=develop

Fix ops doc for some ops 

* split train_mode and has_grad

* fix format

* fix ci problems

* fix sample code

* add Inmemorydataset

This reverts commit 63e90ee3.

* add lars to fleet meta optimizer

* add lamb to proto

* add lamb to fleet meta optimizer

* fixed syntax bug

* fixed syntax bug

* fixed syntax error in lamb, add config setter of lamb in distributed_strategy

* trigger unitest to rerun

* add new unitest func for lamb

* revise unitest for lars and lamb

* revise dgc meta unitest

* revise lars document in distribute_strategy

* revise lars lamb document in distributed_strategy.py

* revise lars lamb document in distributed_strategy.py

* add weight decay exclude logic to lars

* restore optimzier.py

* restore optimizer.py as develop except lars

* add epsilon and exclude fn to distributed_sttrategy

* add lars epsilon

* revise unitest for fleet lars and lamb

* revise lars lamb unitest for CI coverage

* revise lars argument api

* revise lars argument api

* revise lars argument api

* revise api doc of lars

* fix op role

* add sharding save and add_sync_comm_for_test function

* add comm_analyse to utlis

* revise sharding_utils

* add sharding saving unittest

* revise sharding utils for unittest

* revise sharding en doc

* update sharding utils api

* add doc for sharding

* fixed bug in sharding var size count

* update varsize count in sharding

* fix sharding num_nccl_comm

* Revert "fix sharding num_nccl_comm"

This reverts commit d51587c15e9323acf226ddd36154275f0d1daf76.

* update, test=develop

* add static_only for static api

* addd static_only for class init

* remove static_only for default_main_program

* remove creater_parameter & startup_program

* remove failed apis

* revert py_func import

* remove global scope

* remove some api

* remove cuda pinned place

* fix examples, test=document_fix

* fix, test=document_fix

add API serialize_program, serialize_persistables, save_to_file, deserialize_program, deserialize_persistables, load_from_file. (#29034)

* remove BatchSampler type check. test=develop

* add hapi api flops

* fix bug

* fix some bug

* add unit test

* fix unit test

* solve ci coverage

* fix doc

* fix doc

* fix static flops

* delete the comment

* fix some grammar problem in doc

* fix some bug

* fix some doc

* fix some doc

* Add bf16 pool2d and unify bf16 unit tests

* Add change default ops test

* Fix cpu_bfloat16_pass

* Add output_format

* Fix incorrect SetOutput

* Change fromating

* fix win ci failure, test=develop

* add ci test, test=develop

* default not show cpp statck & add hint

* fix failed unittest

* fix failed unittests