* fix dist train with rmsprop

* add rmsprop transpiler test

* update by comment

* Add Preface

* Add demo code

* Save file

* Refine code

* seems can work

* use elementwise strategy

* Use ElementwiseComputeEx

* Add comments

* extract functions from operator

* Refine code

* Follow comment

* code refine

* follow comments

* follow comments

* polish dist se resnext ut

* update

* update

* update

* avoid cpu initializer differ

* change to use executor for now

* update by comment

* remove lr decay use para exe, should fix para exe bug later

* update by comment

* fix the order of sum

* add doc

* check whether need to copy

* follow comments

* fix the result of unit test

* fix resnext

* compare the result of PE and Exe

* compare the result of reduce and allreduce

Revert part of a3ca4c99

enforce test_parallel_executor_transformer and test_image_classification_resnet to run in serial (#12545)

* api clean

* update API.spec

* "cherry picked cpp tests"

* "cherry picked"

* "cherry picked tests"

* "merge develop branch"

* "add gradient register"

* "make some enhance"

* "better format"

* "fix typo"

* "fix reuse"

* "fix get expected kernel"

* "change the mkldnn code"

* "fix mkldnn"

* "fix mkldnn failed test"

* "add comment"