Enhance vjp/jvp/Jacobian/Hessian API for supporting dynamic, static graph and batched, unbatched mode (#40692)

* modify vjp/jvp for both dynamic and static graph

* enforce jacobian class for supporting first/last batch

* add unittest for jvp, jacobian withlast batch, jacobian with first batch

* fix the incorrect shape when multi-index Jacobian

* enforce Hessian class for supporting dynamic graph

* add Hessian class unittest

* bugfix, jvp double_backward_trick zeros_like return stop_gradient=True in static graph

* add API beta warnnings

* add white_list for cuda11.x ci windows.

* optimize some code snippets and documments

* set unittest timeout to 100 seconds

* move vjp,jvp,Jacobian,Hessian to incubate

* fix vjp,vjp import path of sample code

* fix code style error of augtograd/__init__ file

* add nll_loss yaml

* fix nll loss

* fix nll loss bug

* fix bug

* fix bug

* fix infrt problem
Co-authored-by: Nxiongkun <xiongkun03@baidu.com>

* [ControlFlow] Fix contrib API bug in while_loop

* format code

* fix inplace fill_ and zero_ API

* add eager unittest

* change inplace_version checking log

* fix

* support C_ops assign

* open unittest

* fix clone

* add interploate cpu kernel

* fix nullptr bug

* add interpolate gpu kernel

* fix unit test error

* remove raw kernels

* add cuda kernel impl

* add infermeta

* recover accidentally deleted kernels in interpolate op

* fix grad x_grad name error

* remove interpolate_v2_op.h

* rm unused codes

* fix xpu build error

* fix build error

* fix namespace error

* add register header for nup

* fix infermeta error

* modify by review

* add the missing args in test_trt_convert_nearest_interp_v2

* add notes for quantile UT

* Supoort quantile in static-mode and add UT

* support multi_layer of bilstm,*test=kunlun

* support multi_layer of bilstm, *test=kunlun

* support multi_layer of bilstm, *test=kunlun

* support multi_layer of bilstm, *test=kunlun

* add yaml for ele_max ele_min

* fig

* push

* xxx

* add several yaml

* polish strided slice kernel & add yaml

* reorder yaml

* add several yaml

* revert yaml config change

* resolve conflict

* Update test_strided_slice_op.py

* fix error; test=develop

* update

* close some yaml

* fix backward attrite error; test=develop

* add div test

* polish code; test=develop

* update

* update

* fix bug

* update bitwise code; test=develop

* update

* update

* fix some bug

* update

* revert cmakelist

* fix optional bug;

* fix bug

* fix bug;

* add backward test

* open bn

* update

* update

* revert eager_gen

* polish code

* fix topk error

* update

* update

* fix bug;

* move label smooth, nll loss

* revert topk

* fix topk label smooth bug;

* remove batch_norm

* remove topk

* change flip infer meta

* fix flip bug

* update yaml

* close abs

* fix histogram bug

* fix histogram bug

* add abs

* fix histogram kernel

* remove expand

* [Yaml] Migrate sqrt/square/reciprocal yaml

* clean file

* fix unittest error

* Enhance test_eigh_op

* Use eigen decomposition to validate eigen values and vectors
* Fix that TestEighBatchAPI didn't run the batched input

* Enhance test_eigvalsh_op

* Align cusolver tolerance to validate eigenvalues
* Fix that BatchAPI didn't run the batched input

* Add abs for |d_ref|

* Remove comment

* add multiclass_nms3 converter

* add flatten2,reshape2,squueze2_trt_fuse_pass  test cast

* add flatten2,reshape2,squueze2_trt_fuse_pass  test cast

* add flatten2,reshape2,squueze2_trt_fuse_pass  test cast

* [New API]: miminize_bfgs and miminize_lbfgs

* modify for python module call correctly

* add functional package, add error raise in static_graph, change assign to set_value

* unify static_graph and dygraph, fix bug when x or H0 is float64

* now only accept input is tensor, put check args in utils.py, put exception test together

* temp

* add more detailed algorithm illustration and comment, reduce test case to limit test time in 15s

* change in_dygraph_mode to in_dynamic_mode

* fix bug of sample code; reduce test case to reduce test time

* change dir to incubate

* fix load bug and add distributed strategy from pslib

* add unittest

* use cvm config

* trainer and worker config

* add unittest

* add unittest

* add test

* code style

* fix dependency of fused optimizer

* add ut

* no

* fix operator summary table

* update unit test

* add random varaiable transformations API for paddle's distribution package

* add TransformedDistribution API for paddle's probability distribution package

* add random variable transformation unitests for static graph

* replace math.prod which not support python3.7 with functools.reduce

* add Independent and TransformedDistribution distribution

* add unittests for constraint

* fix typo and AffineTransform sample code error

* add mean,variance,rsample abstract method for Distribution

* no

* fix bugs

* fix doc according to review

* fix api doc format

* fix api doc according to review

* fix bug and add unit test

* fix record event bug

* optimize chrome tracing display

* fix bug

* add comment

* add unit test

* fix a bug

* fix

* fix

* fix format

* support view strategy in eager_fluid state

* little change

* little change

* optimize unittest

* fix

* Supported Complex2Real Conversion for Eager Dygraph

* Supported Complex2Real Conversion for Eager Dygraph

* Enabled complex type promotion test for matmul_v2

* pylayer, test=develop

* Fix CI issues

* Support initializing specific grad tensors to zero for selected operators

* finish forward, test=develop

* create grad node finish, test=develop

* Merged adj_edges_ with GradSlotMeta

* Fixed monir issue

* backward finish, start dbg, test=develop

* Adjusted num runs

* Recovered Eager performance tests configurations

* Recovered Eager performance tests configurations

* finish, test=develop

* polish, test=develop

* polish, test=develop

* refine, test=develop

* eager, test=develop

* Adjusted performance tests configurations

* Fixed Minor Issues with performance tests

* [Phi] Fix macro name typo

* support set_materialize_grads, test=develop

* suppotr mark_non_differentiable, test=develop

* support once_differentiable, test=develop

* refine, test=develop

* refine, test=develop

* Moved out Edge from GradSlotMeta

* Fixed issues from merge

* Fixed typo

* Addressed review comments

* Fixed merge issues

* Fixed minor issues

* Fixed minor issue

* refine, test=develop

* refine, test=develop

* refine, test=develop

* Fixed major issues and enabled auto_prune test cases

* Fixed issues from merge

* refine, test=develop

* refine, test=develop

* refine, test=develop

* refine, test=develop

* refine, test=develop

* support inplace for pylayer
Co-authored-by: Njim19930609 <jim19930609@gmail.com>
Co-authored-by: NWang Huan <wanghuan29@baidu.com>
Co-authored-by: NAurelius84 <zhangliujie@baidu.com>

* fix converter when sliced_shape is 1

* update unittest

* fix uncontrolled randomness of op

* fix bugs

Add new APIs for GPU memory monitoring (max_memory_allocated, max_memory_reserved, memory_allocated, memory_reserved) (#38657)

* Add new API memory_reserved

* Add memory_allocated, max_memory_reserved and max_memory_allocater

* Fix CI error

* Fix CI error

* Enhance UT

* Add FLAGS_memory_stats_opt

* Add STATS macro functions

* Add StatAllocator

* Fix CI errors

* Add UT

* Fix CI errors

This reverts commit 05f3d48e.