add dist_multi_trainer for distributed training, add trainer_factory and device_worker_factory so that we can easily extend new training mode, add pull dense worker which is a singleton for parameter fetching

add dist_multi_trainer for distributed training, add trainer_factory and device_worker_factory so that we can easily extend new training mode, add pull dense worker which is a singleton for parameter fetching

* fuse optimizer

* Revert "[MKL-DNN] Fix to crash of Transformer when mkldnn is to be used (#16233)"

This reverts commit 13816dd4.
Apart from enabling transformer for MKL-DNN

* Revert "- MKL-DNN pooling updated to set_prim_desc"

This reverts commit c63f6b20.

Conflicts:
	paddle/fluid/operators/mkldnn/concat_mkldnn_op.cc

* Revert "[MKL-DNN] MKL-DNN specific Tensor modification (#15429)"

test=develop

This reverts commit dec9cf53.

* - concat compilation fix

- lint

test=develop

- Lint fixes

test=develop

- Lint fixes

test=develop

- Fix Transpose MKLDNN op

test=develop

* fix threaded executor bug
test=develop

* change the order of class member
test=develop

* Fix Travis CI
test=develop

* modify the interface of Pass::Allay
test=develop

* Polish code
test=develop

* Fix Travis CI
test=develop

* fix Pass::Apply interface
test=develop

* Fix Travis CI
test=develop

test=develop

add include headers to fix travis-ci
test=develop

* Refine default MKL-DNN Pass order

test=develop

* Add comment to default MKL-DNN Pass list

test=develop

* C-API quantization core

test=develop
Co-authored-by: NSylwester Fraczek <sylwester.fraczek@intel.com>

* Decouple Quantizer from AnalysisPredictor

test=develop

* fixes after review

test=develop

* renamed mkldnn quantize stuff

test=develop

* remove ifdef from header file

test=develop

* test=develop, fix space_to_depth_doc

* test=develop, refine indent

* test=develop, refine code

* test=develop, add api spec

test=develop

test=develop

test=develop

* Disable compare for accuracy issue.

test=develop

* Add todo comments to show more information.

test=develop

* test=develop, add install check

* test=develop, add install check scripts

* test=develop, refine language

* test=develop, add api spec

* test=develop, change cdn to bj to pass ci

* extend the slice function for python
test=develop

* test=develop, add some Layers and tests

* test=develop, add more layers

* test=develop, add more layers

* test=develop, add force cpu option

* Update test_layers.py

remove pdb

* test=develop, refine code

* fix cdn issue, test=develop

* fix memory optimize bugs, test=develop

* fix memory optimize bugs, test=develop

* remove add/sub_2 op, test=develop

* disable memory_optimize by default, test=develop

* disable inplace activation in python, test=develop

* fix unittests, test=develop

* fix unittests, test=develop

* bug-fix, test=develop

test=develop

* update jitkernel doc

test=develop

* follow comments

* follow comments

test=develop

test=develop

* test fix fetch bar place for ce

* fix ps mode dist train in develop test=develop

* fix style check test=develop

* update test=develop

test=develop

test=develop

test=develop

* add layer norm to Layers, add transformer prepare encoding

* little change

* finish encoder part

* add decoder part

* finish model part

* add test case and part of data feed

* add transformer test

* add to_parameter, add remove in set_attr

* test=develop, fix pos encoding bug, create_parameter with stantard name

* test=develop, rm dropout test in imperative

* test=develop, fix cpu error

* test=develop, fix minize bug

* test=develop, fix one hot not stop gradient

* test=develop, fix one hot not stop gradient

* test=develop, refine parameter name

* test=develop, fix transformer test in imperative mode

* test=develop, fix transformer test in imperative mode

* test=develop, fix boost and mkl download error

* test=develop, fix boost and mkl download error

* test=develop, fix ci and refine code

* test=develop, fix ci and refine code

test=develop

test=develop

fix ctest eager deletion disable bug
test=develop

test=develop

Use SpectralNormGradOpDescMaker instead of DefaultGradOpDescMaker
to avoid registering useless variables to improve GPU usage.

test=develop

* Add fsp operator.
1 Add unitest.
2. Add python API.
3. Add layer test.

* Add quantization strategy.
1. Add API.
2. Add unitest.

* Add distillatoin strategy.

* Add unitest config file for quantization

* Fix Copyright
test=develop

* Fix setup.py

* Fix document of layers.py.
test=develop

* Fix unitest in python3.
test=develop

* Fix documents.
test=develop

* 1. refine fsp op by batched gemm
2. remove unused import
test=develop

* Fix test_dist_se_resnext.
1. disable test distillation.
2. reset framework.py
test=develop

* Enable unitest of distillation after fixing Block._clone_variable
test=develop

* Fix cdn issue.
test=develop

* fix cdn issue, test=develop

* fix cdn issue, test=develop

test=develop