* Rewrite pow and logical functional apis.

* bugfix

* Fix logical_or and logical_and unittest

* Fix logical_xor unittest

* Dev remove python functions (#6042)

* Fix logical_or and logical_and unittest

* Rewrite clip

* Fix clamp

* reformat

* Rewrite scatter and scatter_add

* Rewrite fmod and where

* Rename clip functor by clamp
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tensor.item tensor.tolist

* add test case

* refine

* auto format by CI

* fix ci fail

* fix doc test
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Graph image gpu decoder

* fix ofrecord path in CI

* fix to

* ImageDecoderRandomCropResizeOpExpr; CUDA < 10.2 warning; eager unimplement

* fix path
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* functional_api: fix build error in mac os

* functional_api: fix interpreter_test

* for all: replace device_type.pb.h with device_type.h

* value_types: impl hash

* revert tools change

* CI: mac only

* cmake: fix of_functional_obj

* cmake: fix of_functional_obj

* CI: revert
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* dev optim_optim_lr_scheduler_multisteplr

* change class name

* ensure values in list milestone is increasing

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* bugfix: data_transport_token_per_placement

* refactor TransportToken::NewDataTransportToken

* NewDataTransportToken(parallel_desc)

* fix bugs

* rename UniqueConsistentIdStorage to ConsistentIdStorage

* refactor TransportToken

* add TransportToken::src_rank and TransportToken::dst_rank

* harded coded thread_consistent_id

* StreamType::SupportingTransportInstructions()

* not thread_consistent_id in single-client mode

* explicit initialize all fields of TransportToken

* remove unused field TransportToken::rank_group_level_

* more bits for TransportToken::seq_id_

* fix oneflow_testexe error
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: clackhan <han_binbin@163.com>

* export pooling api

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add some test_case

* add autotest for topk

* add autotest for topk

* add rsqrt autotest

* add more tests

* add fmod autotest

* add tensor.fmod autotest

* fix autotest codegen bug

* add flip autotest

* add chunk autotest

* restruct constant op autotest

* auto format by CI

* add zeros and ones autotest

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix

* fix

* refine

* refine

* refein

* refine

* refein

* refei

* refein

* refein

* refein

* Revert "refein"

This reverts commit d6440b26.
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* NopKernel

* fix

* move kernel_utils
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* disable cuda_h2d stream

* gpu.LocalCallOpKernel is shared between device  and device
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Turn on NVCC's warnings

* refine

* fix

* use CMAKE_CUDA_COMPILER_VERSION

* 611

* CUDA 10

* clean

* fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* add tril op

* mod tril gradfunc

* add tril test, resolve review bugs

* mod tril test

* add fields init
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: NLuyang <flowingsun007@163.com>

* regist unfold ops

* unfinished

* unfold init

* move padding before and after

* add unfold module

* add functional

* add unfold kernel

* add unfold and fold op

* add init

* remove outdated file

* add fold op

* fix compile

* fix unfold kernel

* add unfold module

* add fold kernels

* add unfold and fold op

* Add functional impl

* add fold module

* add unfold and fold gradient func

* fix bug

* fix a bug

* just for debug

* just for debug

* just for debug

* need to solve segment fault

* Add fold test

* fix unfold and fold bug

* add check

* add dilation equal 2 test case

* fix comment

* fix comment

* add api docs

* fix comment

* remove annotation

* fix clang static anlysis
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix

* fix

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use copy_all_files_in_dir to replace copy_files, rm .so in build_wheel.py
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* restore build_wheel.py
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* trigger ci
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* Remove python function wrapper.

* Increase ref

* Fix doc test

* Update python/oneflow/framework/docstr/utils.py
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: NYao Chi <later@usopp.net>

* add naive logging

* refein

* refein

* check in naive iterface

* refein

* impl more function

* refein

* add naive warning

* refein

* refein

* add module

* add py callback

* refein

* add more

* fmt

* BlockingContext => BlockingInfoContext

* refein

* refein

* refein

* refein

* refine

* remove unnecessary check

* fix for clang tidy

* skp

* auto format by CI

* revert
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* skip sync when abnormal exiting
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* cuda base cpu mpi boxing

* cpu_mpi

* fix conflicts

* add cpu mpi unittests

* more checks and unittests

* abstract_consistent_to_consistent_op_expr

* fix compiler complaint

* refactor consistent-to-consistent eager consisitent op interpreter

* fix compiler complaint

* refactor ConsistentToConsistentOpExpr

* lazy interpreter (#5903)

* fix bugs about consistent_id

* more test_consistent_cast unittests

* refactor functional::ToConsistent

* refactor GetNdSbp

* fix compiler complaints

* Update eager_consistent_op_interpreter.cpp

* Update eager_mirrored_op_interpreter.cpp

* eager_boxing_1_to_n

* add missing files

* del useless file

* minor fix

* refactor GetDevice4CurrentProcessCtx

* refine

* minor fix

* Update naive_1ton_boxing_interpreter.cpp

* refine

* Update eager_boxing_interpreter_mgr.cpp

* fix error

* fix error

* auto format by CI

* fix error

* refine

* make of_format

* Update nd_sbp.h

* fix consistent id check error

* refine

* back up

* refine

* minor fix

* refine

* refine

* refine

* minor fix

* minor fix

* refine

* refine

* Update nccl_boxing_function.cpp

* refine

* fix consistent meta check bug

* zoom kLimitParallelConfString

* refine

* Update symmetric_x_to_b.cpp

* Update boxing_dividor_util.h

* Update boxing_dividor_util.cpp

* Update eager_boxing_interpreter_mgr.cpp

* auto format by CI

* Update asymmetric_x_to_b.cpp

* Update asymmetric_broadcast.cpp

* auto format by CI
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: NLi Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tensor.xxx codegen

* fix tensor shape bug

* add flow.xxx and tensor.xxx codegen, with Tensor args and kwargs

* add flow.xxx and tensor.xxx codegen, with Tensor args and kwargs

* add module callable codegen

* code format
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add broadcast_pow

* fix gpu mode & refine test

* refine

* AddOp to const&
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PlacedNdSbp

* 1) fix compiler complaints; 2) add an unittest

* refactor DECORATE

* backup code

* MainBoxingExpr

* 1) reformat; 2) fix compiler complaints; 3) add more unittests

* merge master
Co-authored-by: Nliufengwei <2472937968@qq.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add fused bias add gelu

* add a simple unittest

* fix backward in fused bias add gelu

* add fused bias dropout

* fix format

* remove print

* add more params

* remove dropout op expr

* fix comment

* fix format

* remove redudant code

* remove redundant code

* add initialize
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* new_X_to_B

* refine

* refine

* Update naive_xtob_boxing_interpreter.cpp

* Update naive_xtob_boxing_interpreter.cpp

* cpu x to b

* refine

* minor fix

* add test case

* Update eager_boxing_interpreter_mgr.cpp

* refine

* refine

* refine

* minor fix

* minor fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* optimize of_include_copy
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* move of_third_party_include_copy into prepare_third_party, change add_custom_target to add_custom_command so it will not always run
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* restore debug change
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* update test.yml
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix cmake
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix clang-12 bug
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix cmake
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* add python/oneflow/include
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix dependency
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix inplace arg
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* restore .gitignore
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* update build_wheel.py to support copy include files when building third party
Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor(LrScheduler): rename CosineAnnealing to CosineDecay

reference: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/CosineDecay

* feat(CosineAnnealingLR): add CosineAnnealingLR

* refine parameter name

* auto format by CI

* test(Graph): fix graph test bug when using CosineDecayLR
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add runtime error

* add runtime error

* fix  OF_UNIMPLEMENTED

* Add TensorImpl info

* Add python single test

* rename

* fi UNIMPLEMENTED_THEN_RETURN

* fix UNIMPLEMENTED

* format

* fix UNIMPLEMENT

* auto format by CI

* fix format

* Modify OF_RUNTIME ERROR to OF_RUNTIME_ERROR

* Modify OF_COMPLIE_OPTION_EEEOR to OF_COMPLIE_OPTION_ERROR

* Modify OF_RUNTIME ERROR to OF_RUNTIME_ERROR

* Fix omissions

* auto format by CI
Co-authored-by: NYinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

This reverts commit 0ba9198e.
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* xrt: fix all warnings

* xrt: comment in fix_ostream_nullptr.h

* xrt: clang format
Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support_eager_boxing_Sm_2_Sn

* Update collective_boxing_interpreter.cpp

* Update eager_boxing_interpreter_mgr.cpp

* refine

* auto format by CI

* refine

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>