* hide misc APIs

* update

* fix transformer test

* update API.spec

* fix dist train with rmsprop

* add rmsprop transpiler test

* update by comment

Fix pb_protobuf2 FromString problem

* Add Preface

* Add demo code

* Save file

* Refine code

* seems can work

* use elementwise strategy

* Use ElementwiseComputeEx

* Add comments

* extract functions from operator

* Refine code

* Follow comment

* code refine

* follow comments

* follow comments

Fix str and bytes problem
Fix sorted problem
Fix math problem
Fix CI problem

* polish dist se resnext ut

* update

* update

* update

* avoid cpu initializer differ

* change to use executor for now

* update by comment

* remove lr decay use para exe, should fix para exe bug later

* update by comment

* fix the order of sum

* add doc

* check whether need to copy

* follow comments

* fix the result of unit test

* fix resnext

* compare the result of PE and Exe

* compare the result of reduce and allreduce

enforce test_parallel_executor_transformer and test_image_classification_resnet to run in serial (#12545)