* refactor(LrScheduler): rename CosineAnnealing to CosineDecay

reference: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/CosineDecay

* feat(CosineAnnealingLR): add CosineAnnealingLR

* refine parameter name

* auto format by CI

* test(Graph): fix graph test bug when using CosineDecayLR
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add runtime error

* add runtime error

* fix  OF_UNIMPLEMENTED

* Add TensorImpl info

* Add python single test

* rename

* fi UNIMPLEMENTED_THEN_RETURN

* fix UNIMPLEMENTED

* format

* fix UNIMPLEMENT

* auto format by CI

* fix format

* Modify OF_RUNTIME ERROR to OF_RUNTIME_ERROR

* Modify OF_COMPLIE_OPTION_EEEOR to OF_COMPLIE_OPTION_ERROR

* Modify OF_RUNTIME ERROR to OF_RUNTIME_ERROR

* Fix omissions

* auto format by CI
Co-authored-by: NYinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

This reverts commit 0ba9198e.
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* xrt: fix all warnings

* xrt: comment in fix_ostream_nullptr.h

* xrt: clang format
Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support_eager_boxing_Sm_2_Sn

* Update collective_boxing_interpreter.cpp

* Update eager_boxing_interpreter_mgr.cpp

* refine

* auto format by CI

* refine

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Move framework/local_dep_object.*  to the eager directory

* Move framework/local_dep_object.*  to the eager directory

Co-authored-by: Ncheng cheng <472491134@qq.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix in_grads size

* correct comment
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Functional support multiple signatures.

* Add TypeError and refine error message.

* Finish py function.

* Refactor add, sub, mul and div functional apis.

* Fix merge error

* Support scalar add/sub/mul/div tensor.

* auto format by CI

* Fix optional argument

* Set inplace keyword only

* Fix throw and treat long as float

* Fixup merge error

* Check tensor type

* Optimize recursive

* Fix add static zero

* Fix merge

* Fix warning as error

* Fix warning as error

* Fix merge

* Fix merge
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* PlacedNdSbp

* 1) fix compiler complaints; 2) add an unittest

* refactor DECORATE

* PlacedNdSbp

* 1) fix compiler complaints; 2) add an unittest

* tensor.cpu/tensor.cuda

* register tensor dtype modules

* export api docs

* fix comments

* refine

* register module methods

* refine

* format

* add testcases

* upper rmsprop tol

* refine

* add more test case

* auto format by CI

* skip test if only cpu

* auto format by CI

* refine

* refine

* fix test

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor to

* test

* call cast when device_type is same

* add fused_self_attention functor

* add fused_self_attention gradients

* add test case

* fix backward immpl bug

* fix bug

* fix bug

* fix backward bug

* fix comments

* code format

* fix comment

* fix ci error

* fix ci error

* add init FusedSelfAttentionInterpState struct
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* dos2unix
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* remove automerge on fail
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* test ci
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* Revert "test ci"

This reverts commit 0f1051f3.

Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add sys conf

* format

* refine warmup api

* refine config

* format

* add WarmupLR

* add warmup lr

* refactor graph folder structure

* export graph

* add amp conf

* deal with lr init step and test cpu

* add WarmUpLR

* add flow config

* GradScaler and config

* rm useless code

* rm useless code

* add api_enable_cudnn_fused_normalization_add_relu

* move config

* move amp

* refactor WarmUpLrScheduler

* add comment

* add arg
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* reemove redundant codes in ModuleDict

* auto format by CI

* add moduledict test

* refine

* refactor test case

* auto format by CI

* auto format by CI

* refine

* refine

* refine  docs format
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add flow.relu

* add test_relu.py

* fix for comments

* add docstring for oneflow.relu

* fix bugs

* add docstring for flow.Tensor.relu

* fix for comments

* fix for comments v2

* auto format by CI

* fix license bug

* auto format by CI

* fix tets_quantization.py bug
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix var impl bug

* add more test_case

* refine

* add tensor method test

* fix comment

* auto format by CI

* fix comments

* fix ci error

* fix run ci error
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add reduce_prod op

* add test_prod

* add prod backward

* modify  backward

* mod backward

* mod prod_test

* add note

* mod prod test

* conflict fixed

* conflict fixed

* auto format by CI
Co-authored-by: NYongtaoShi <73167956+YongtaoShi@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use clang-tidy-diff.py return code and add -warnings-as-errors
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix python2 shebang
Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* prevent_running_oneflow_in_forked_subprocess

* add line change

* IsFork => IsForkedSubProcess

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix bug int test_tensor_str

* format

* fix comment

* fix bug to(cuda) is unavailable in cpu env
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* cuda base cpu mpi boxing

* cpu_mpi

* fix conflicts

* add cpu mpi unittests

* more checks and unittests

* abstract_consistent_to_consistent_op_expr

* fix compiler complaint

* refactor consistent-to-consistent eager consisitent op interpreter

* fix compiler complaint

* refactor ConsistentToConsistentOpExpr

* lazy interpreter (#5903)

* fix bugs about consistent_id

* more test_consistent_cast unittests

* refactor functional::ToConsistent

* refactor GetNdSbp

* fix compiler complaints

* refactor GetDevice4CurrentProcessCtx

* fix error
Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* check: fix clang-tidy-diff commands

* CI: fix step names
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache

* skip test targets of re2
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* abstract_consistent_to_consistent_op_expr

* fix compiler complaint

* refactor consistent-to-consistent eager consisitent op interpreter

* fix compiler complaint

* refactor ConsistentToConsistentOpExpr

* lazy interpreter (#5903)

* fix bugs about consistent_id

* refactor functional::ToConsistent

* refactor GetNdSbp

* Update eager_consistent_op_interpreter.cpp

* Update eager_mirrored_op_interpreter.cpp

* fix error

* fix error

* auto format by CI

* Update nd_sbp.h

* refine identity boxing

* fix sync checkmeta error

* avoid consistent id check in lazy
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: NLi Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* feat(no_grad): support no_grad decorator

* feat(AutogradMode): export flow.autograd_mode

* feat(GradMode): export some grad_mode class

* docs(GradMode): export documents

* refine

* docs(GradMode): export document for is_grad_enabled

* auto format by CI

* fix(GradMode): fix single client bug

* fix bug
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* CI: enable more checkers for clang-tidy

* .clang-tidy: remove cppcoreguidelines-pro-type-vararg

* CI: remove duplicate checkers

* CI: remove clang-analyzer-alpha.deadcode.*

* .clang-tidy: add performance-*

* oneflow/core/eager: remove unnecessary malloc & free

* .clang-tidy: add clang-analyzer-cplusplus.* to werror

* user_kernel: remove useless move

* quantization_aware_training: fix move return

* .clang-tidy: add google-*

* CI: fix clang tidy command

* CI: fix test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix wrong names

* auto format by CI

* refine

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add some op-test

* fix dims_error in my branch

* Fix the bad backward kernel function by using 'cuda::atomic::Add' (#5614)

* Test `nn.AdaptiveAvgPoolXd` (#5615)

* Fix the bad backward kernel function by using 'cuda::atomic::Add'

* Support the 'NoneType' annotation

* Support objects of 'collections.abc.Iterable' as 'output_size'

* Test with all cases of 'output_size'

* Update adaptive_pool_gpu_kernel.cu

* Skip testing `nn.AdaptiveAvgPool3d` for the current PyTorch

* remove some useless test

* Format TODO

* Add the assertion messages for 'output_size'

* Reformat codes

* Remove raw tests for `flow.negative`

* Remove unnecessary codes and add the assertion messages

* Merge updates for 'generators.py' from master

* Remove unnecessary 'random()'

* Delete the separate test for `AvgPool2d`

* Fix import paths

* Fix import problems

* Remove the PyTorch import

* Denote the annotations for `tile` and `repeat` ops

* Add the test for `nn.AvgPool1d`

* Choose better generators for `nn.MaxPoolXd`

* Randomly choose `dilation` and default values

* auto format by CI

* Test more kwargs for `nn.AvgPoolXd`

* Add tests for `return_indices`

* auto format by CI
Co-authored-by: NTianyu Zhao <guikarist@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>