* fix dtype of ungenerated grad var

* update ut

* refine code

* set default dtype

* fix could_use_cudnn bug

* remove debug code

* re-implement

* fix bug

* change to tensor copy sync

* change to tensor copy sync

* make copy_to safe when use TensorCopy

* refine code

* add ut

* add cudapinned garbagecollector

* add testcase: cpu place -> cuda pinned place

* simplify prepared op impl to improve performance

* fix kunlun compile error

* continue fix kunlun compile error

* only transform diff place when dtype diff

* fix failed unittests

* remove useless file

* polish impl by review comment

* support dygraph in xpu place; test=develop

* fix cpu/gpu compile error; test=develop

* fix compile error; test=develop

* fix xpu compile error; testd=develop

* try to add fwd op input dtypes

* refactor base impl

* return tmp_ins after dygraph prepare data

* fix typo found in debug

* polish comment & add complex net test

* revert detail change

* fix unittest failed

* add complex kernel condition control

* fix xpu test failed & polish comment

* polish details by review comments

* add support for complex grad accumulated

* add unittest for coverage

* update test dtype

* remove useless blank line

* fix fleet for multi-stream

* fix memcpy for ncclid

* use sync to solve move operation

* support deepcopy for Layer/Tensor/Paramerbase

* fix some code

* add tensor_indices in AssignGroupBySize

* add rebuild group in reducer

* use has_grad instead of train_mode

* add vlog for debug

* fix ut

* fix ut

* The leaf tensor concept is exposed and the gradient accumulation of leaf tensor

* The leaf tensor concept is exposed and the gradient accumulation of leaf tensor

* fix coverage

* fix api doc

* fix CI unittest

* fix CI unittest

* fix unitest

* empty tensor does’t need inner_var_

* fix some error message

* Add a class TensorInplaceVersion to count the inplace version and put it in framework::Tensor instead of Allocation or Variable.

* Add a new attribute `_inplace_version` for VarBase.

* Raise exception if an inplace operation can result in incorrect gradient computation.

* Add a new interface _bump_inplace_version() for VarBase to bump the version whenever the Tensor is modified through an inplace operation.

* For api assign, call _bump_inplace_version() when it's an inplace operation inn dynamic mode.

* Use original var_wrapper if the inplace_version is not changed.

* Replace SnapshotVarWrapperList with SnapshotVarWrapper to optimize performane.

* add reducer

* refine envent for memorycopy

* add concat&split for allreduce

* apply concat & split for fuse tensor

* fix nccl dep

* fix the untest, compile problem and ddp initialize problem

* fix untest for mac & add some comments & solve the repeated param in sublayers

* fix untest for windows & fix document

* split train_mode and has_grad

* fix format

* fix ci problems

* fix sample code

* add base hook classes and reduce hook impl

* fix constructor typo

* polish comment format

* refactor baisc hook class design

* polish design details

* remove slelected rows all reduce over height check

* polish unittest

* add retry for dygraph parallel socket bind

* change to loop always

* fix writing error

* nccl init support hostname and ip; test=develop

* add sumary feature

* refine printting tensor

* add sci_mode

* add sample code

* fix indent error

* fix _format_item

* polish code

* support item indent

* add ut

* set place for ut

* fix py2 issue

* fix ut

* Add support for mkldnn ops types selection with FLAGS in dygraph

* use regex to match DNNL verbose

* python3 encoding fix

* refine err msg

* follow comments

* use iwyu clean include, test=develop, test=win

* compilation error, test=develop

* fix compilation error2, test=develop

* fix compilation error3, test=develop

* fix compilation error4, test=develop

* fix compilation error5, test=develop

* fix compilation error6, test=develop

* fix compilation error7, test=develop

* fix compilation error8, test=develop

* fix compilation error8, test=develop

* fix compilation error10, test=develop

* fix compilation error11, test=develop

* get use of global 'use_mkldnn' in layer_helper

* update for CI

* update for CI, relu test

* update for CI, relu test added, make FLAGS_use_mkldnn a public flag

* added more strict tests, fixes after review

* fixes after review

* fixes after review, CI stuff

Use `Tensor` instead of `Variable`  in the doc of paddle.grad.

* update the demo code and the doc of varbase.backward.

* update the doc of the fake interface `paddle.fluid.Variable`.

* remove BackwardStrategy.

* support Baidu AI Accelerator
  * test=kunlun

* minor
 * test=kunlun

* support xpu op in separate file
 * test=kunlun

* update XPU error message and remove duplicated code

 * test=kunlun

* minor
 * test=kunlun

* minor
 * test=kunlun

* enable mkldnn layout conversion

* review fix: remove tmp_place

* fix test mkldnn swish

* add UT for PrepareData CPU->MKLDNN

* add #ifdef PADDLE_WITH_MKLDNN

* Force-push commit
Co-authored-by: Ngrygielski <adam.grygielski@gmail.com>

* add unchaged infershape function

* add broadcast infershape function

* fix bug

* rename infershape functions

* add UnaryOpUnchangedInferShapeCheckAxis

* add error message

* add test for common infer shape functions

* dont update existed ops

* dont update op_desc.h

* add more test

* add error check, refine error message

* add auto_cast, test=develop

* add loss scaler, test=develop

* add comments, test=develop

* refine code, test=develop

* refine code, test=develop

* do not set flags automatically, test=develop

* fix custom op bug, test=develop

* add more test, test=develop

* refine enable logic, test=develop

* enable amp test with GPU, test=develop

* add unittest

* add test for found_inf

* follow comments

* follow comments

* remove global variable, use singleton

* add some notes

* update comments

* update comments

* update comments

* add use_dynamic_loss_scaling argument

* refine found_inf

* refine found_inf

* Add some error messages for the op without double grads.

* fix the test_imperative_double_grad UT.

* fix double grad visitid unit; test=develop

* change name hash_pair to HashPair; test=develop

* follow comment; test=develop

* Added DNNL cache management for DyGraph

* move FLAGS_use_mkldnn to more general CMakeLists, getu use of the flag in ClearGradients

* missing file

* Fixes after review

* Bringing back original idea of place for 'use_mkldnn' flag to be accessible from platform nad imperative.

* Removed duplicate and added docs

* Fixes for CI