* default dist op

* add dist_attr for dist op

* add unitest

* update inputname

* update function name

* add unitest

* update CMakeLists.txt for CI

* fix dis_matmul

* fix compile error

* update matmul to matmul_v2

* unify api

* unify api

* todo

* update distop forward func

* update distop forward func

* auto parallel backward

* update dist op

* autoparallel backward

* add backward for embedding

* temp1

* temp2

* temp3

* temp4

* backward done1

* backward done2

* backward done3

* dist embedding remove mp mode

* dist matmul remove mp mode

* update dist embedding
『

* dist op init1

* dist op init 2

* update unitest

* context remove parallel mode

* partitioner remove parallel mode

* update unitest

* a more general method to support varying mesh in pipeline parallel

* support varying mesh in pipeline parallel

* embedding support varying mesh in pipeline parallel

* matmul support varying mesh in pipeline parallel

* default dist op support varying mesh in pipeline parallel

* dist attribute for startup program

* default dist op support varying mesh in pipeline parallel 2

* partitoner support varying mesh in pipeline parallel

* revise logic for auto compeletion

* revise framework.py

* revise reshard unitest

* revise unitest for parallelize

* chmod

* fixed bug for dist embedding name mapping
Co-authored-by: Nzhaoyingli <zhaoyingli@baidu.com>

* native commit for triple grad of sigmod

* Updated unittests files

* init functional jacobian api

* Updated trible_test func

* Updated gradient_checker & test_script

* finish test with dtype float32

* add float64 test case

* polish code

* use atol=1e-5 with dtype float64

* fix for ci

* set timeout for test_jacobian

* fix dygraph grad to support high differential

* polish API docstring

* Updated gradient checker and some related files

* fix double grad strip error for high differential

* fix double grad strip error for high differential

* Add Sigmoid triple grad tests

* fix dygraph double grad dtype error when calling for high differential senario

* Updated triple grad teses func

* Use np.random to initialize ddx

* Updated triple_grad_check func

* add todo for gradient checker and refine some comments

* remove additional code

* add test for warnging in backward.py

* format python code
Co-authored-by: Nveyron95 <veyron_wu@163.com>
Co-authored-by: Nlevi131 <limaolin01@baidu.com>

* [HIP] fix op not support AMD GPU bug, the flag PADDLE_WITH_ROCM is invalid

* [HIP] fix op not support AMD GPU bug, the flag PADDLE_WITH_ROCM is invalid

* [HIP] fix op not support AMD GPU bug

* [hybrid] seed and dropout op support force-cpu

* [hybrid] seed and dropout op support force-cpu

* [hybrid] seed and dropout op support force-cpu

* [hybrid] seed and dropout op support force-cpu

* [hybrid] seed and dropout op support force-cpu

* [hybrid] fix seed ci failed issue

* add AsExtra for force_cpu of seed op

* add gradients_with_optimizer api

* modify gradients_with_optimizer

* add gradients_with_optimizer api into paddle.auto.backward_mode

* add gradients_with_optimizer test case

* add doc for gradients_with_optimizer

* add doc for gradients_with_optimizer

* modify API nn.Bilinear's doc, test=develop

* Modified params of some APIs to support tuple and list.

* fixed bug.

* oneps (3/4)
Co-authored-by: NMrChengmo <cmchengmo@163.com>
Co-authored-by: Nmalin10 <malin10@baidu.com>
Co-authored-by: Nchengmo <chengmo@baidu.com>

* add static_only for static api

* addd static_only for class init

* remove static_only for default_main_program

* remove creater_parameter & startup_program

* remove failed apis

* revert py_func import

* remove global scope

* remove some api

* remove cuda pinned place

* modify sample code

* variable -> tensor

* migrate program_guard sample code

* refine error message

* migrate program_guard

* refine comment style

* fix indent

* support use add instead of sum to do gradient accumulation

* add inplace addto pass

* add grad_add op and inplace addto pass

* remove debug code

* code refine

* fix bug when sereral sum ops inserts at same op_idx

* fix Flags type

* add addto attribute for conv3d

* fix ut

* code clean

* fix type

* fix calcu_gradients

* fix code place

* fix embedding interface usage

[while grad]Support pruning op in find_op_path about while sub-block when appending backward (#25330)

Prune OPs which are not related with loss in while sub-block when constructing backward OP path.

* test=develop, test=document_fix

* test=develop, test=document_fix
Co-authored-by: Nswtkiwi <1208425345@qq.com>

* Rename internal gradient variables in multiple backward
* so that they have different names with previous backward
* For example:
*  y = x * x, grad = fluid.gradients(fluid.gradients(y, x) + y * y, x)
* In second-time backward, gradient variable names of partial
* forward network (y * y) may be have same names with first-time
* fluid.gradients(y, x).

test=develop

* allow amp and recompute working together

* API/OP (append_backward) error message enhancement test=develop

* polish check_type test=develop

* fix unittest failed test=develop

* merge develop test=develop

* API(fluid.gridents) error message enhancement test=develop

* fix unitest failed test=develop

* add double grad implementation for dygraph, test=develop

* polish code, add uts, test=develop

* fix place bug, test=develop

* polish codes, add more uts for coverages, test=develop

* add no_grad_set, test=develop

* add star gan ut, test=develop

* follow comments, test=develop

* add fluid.device_guard to specify the device type for Op

* polish backward api doc test=develop, test=document_preview,
       test=document_fix

* polish backward api doc test=develop, test=document_preview, test=document_fix

* no_grad supports set of Variable test=develop, test=document_preview

* polish sample code of append_backward test=develop, test=document_preview

* modify assert into Raise TypeError test=develop,test=document_preview

* fix unittest failed test=develop

* rm useless file test=develop

* polish en doc test=develop

* polish code of no_grad_set test=develop

* polish code of no_grad_set test=develop

* polish backward prune, test=develop

* fix control flow op bug, test=develop

* add some unittests, test=develop

* fix unittest args, test=develop

* follow huihuang's comments, test=develop

* append optimize op in the grad block of current block if current block is in control flow. test=develop

* add conditional grad op when optimizer used in control flow. test=develop

* add comment and modify typo. test=develop

* fix append_backward to support control flow. test=develop

* add test. test=develop

* fix copy_var_to_parent_block and conditional_block_grad. test=develop

* fix bug: revert to append conditional_block_grad vars to sub grad block. test=develop

* fix bug: revert to assign var to parent block even if var already is in parent block

* fix bug: consider outputs is empty. test=develop

* move _rename_grad_ out. test=develop

* modify code according to reviews from Huihuang. test=develop

* modify code according to reviews from Jinle. test=develop

* update doc, test=develop

* fix related unittests, test=develop

* fix str incompatible error, test=develop

The fixed bugs:

1. The condition sub-graph is not pruned
2. When backward graph is extremely simple, the whole backward ops are pruned.

* add seed op

Add tests to use dy/dx to make sure the gradient values calculated by the control flow backward is correct. Also fixed bugs detected by those tests.

Fix bugs:

1. Unlike sum_op, optimizer ops don't allow uninitialized input tensor. But in conditional_block_grad_op, since the conditional_block may not run, the output gradient tensor may be uninitialized, which will cause the optimizer op error. To fix it, we should let optimizer ops support uninitialized input like sum_op or assign the uninitialized gradient to 0 when the conditional_block_grad_op doesn't run. I found there are about 10+ optimizer ops. **To be simpler, I just assign output gradient of the conditional_block_grad_op to 0 in this PR**. But it can be further explored whether we can make optimizer ops like sum_op to support uninitialized input tensor because theoretically we can speed up without the assigning in conditional_block_grad_op.

2. Infer parameter shapes during append_backward. I didn't know that all our parameters are in global block. When op_desc is inferring shapes at the sub-block, it may not know the shape of gradients of parameters whose shape information is at global block. I fixed it by inferring shapes of gradients from forward var.

This PR also did some code clean up:
1. Print the var name when sgd_op catches shape error so that it is easier to debug
2. Fix a typo: dicta -> dict