* add apply for Layer

* add fleetrun command for distributed running; test=develop

Update the code for the compare ops for the broadcast function

* fix eval() sync problem

* add unittest

* modify according reviewer

* fix unittest failed

* test=develop,test=document_fix, remove the out args

* delete out in inverse

* test=develop,test=document_fix, remove the out args

* Revert "test=develop,test=document_fix, remove the out args"

This reverts commit f430799b.

* test=develop,test=document_fix, remove the out args

* remove all  the out args

* remove all  the out args

* test=develop, remove all  the out args

* test=develop, remove all  the out args

* test=develop, remove out args in matmul testcase

* test=develop, fix the bugs of out args testcase in logsumexp

* test=develop, fix the bugs of out args testcase in elementwise_add

* test=develop, fix the bugs of out args testcase in elementwise_div

* test=develop, fix the bugs of out args testcase in activation

* test=develop, fix the bugs of out args testcase in addcmul

* test=develop, fix the bugs of out args

* test=develop, fix the bugs of out args in API example

* test=develop, fix the bugs of out args in kron testcase

* test=develop, fix the bugs of out args in kron testcase

* test=develop, remove out args

* test=develop, remove out args

* test=develop, fix the coverage

* test=develop, polish codestyle

* test=develop, Update math.py

* test=develop, polish codestyle

* test=develop, polish codestyle

* test=develop, polish codestyle

* test=develop, polish codestyle

* test=develop, polish the test_activation_op.py

* test=develop, Update math.py

* test=develop, fix the failed CI

* test=develop, add core.ops

* test=develop, add core.ops change clamp to clip

* test=develop, add core.ops change clamp to clip

* test=develop, write testcase for clamp OP

* split meta optimizer files
* add graph execution in execution, update two properties in DistributedStrategy, unit tests for these features

* add localsgd meta optimizer

* add pipeline optimizer

* fix inverse bug, test=develop

* fix the untest, test=develop

* add singular checking, test=develop

* fix the utest, test=develop

* use memory::copy, test=develop

* fix bost_get, test=develop

* fix position, test=develop

* hidden the explicit setting of dataset for pipeline training.

* modify pipeline demo, test=develop

Update the code for the compare_ops, update the api and doc 

* fix test_pipeline, test=develop

* add gradient Merge optimizer to meta, test=develop

Integrated Trainer of Parameter Server (API add `fluid.contrib.layers.sparse_embedding` only) (#22957)

* Integrated Trainer of Parameter Server

[Dy2Stat-ErrorMessage]  Add interface:create_origin_info_map and attach_origin_info for AST node (#25627)

* Add interface:create_origin_info_map and attach_origin_info for AST node. test=develop

* Fix code according to comments from reviewers. test=develop

* fix double grad visitid unit; test=develop

* change name hash_pair to HashPair; test=develop

* follow comment; test=develop

* reduce hidden size and loop number; test=develop

* change loop number; remove useless code; test=develop

Fix the argsort and sort op for the api2.0, and update the api 

* Remove the output for moving_average_abs_max_scale op, test=develop

* refine strategy compiler and meta optimizers
make async as a_sync

* remove ProgramTranslator.save_inference_model

* adapt save_quantized_model

* revert buffer check implemention

* remove useless import function

* simplify buffered reader to improve DataLoader performance

* fix 22 failed unittests

* fix cuda pinned context condition

* fix test_reader_reset failed

* fix two failed unittests

* change unittest place

* polish error messaage

* polish cast op GetExpecctedKernelType

* remove debug info in unittest

Enhance TracedLayer Error Message

Note: this PR uses assert to check type somewhere and check_type somewhere, the reason is that the check_type skips checking when it is under dygraph mode.

* add more settings for distributed strategy
Basically, DistributedStrategy has several parts of configurations:
- BuildStrategy: the same as paddle.fluid.BuildStrategy, but the distributed arguments are moved out of BuildStrategy
- ExecutionStrategy: the same as paddle.fluid.ExecutionStrategy
- collective communication configs: nccl_comm_num, hierarchical allreduce and so on
- distributed algorithms: async_update(mainly used in PS), lars, lamb and so on

Based on the comment here https://github.com/PaddlePaddle/Paddle/blob/b5f8784cab94eae785659787fc529870c87b254c/paddle/fluid/framework/details/build_strategy.h#L49

The unit test which compares Reduce and AllReduce must have diff. The PR_CI_Night runs on P40 machine and it has 8GB GPU, which is smaller than the 16GB normal CI machines. So we decrease the batch size in the past to make it runnable: https://github.com/PaddlePaddle/Paddle/pull/24651/files  . Decreasing the batch size makes the difference occurs often. So this PR replace the absolute delta by relative delta.

Before this PR, the unit test failure happens with probability about 1/100. After this PR it doesn't happen.

* fix the double grad bug for the star gan. test=develop

* update the retain_graph parameter doc. test=develop

* add the unit test for the retain_graph parameter. test=develop

* fix jit.save input type change error

* add unittes

* add fleet distributed metrics
* test=develop