* add int attr for functional adaptive average pool

* add unittest

* Treat scalar as scalar list if size is not 0

* reformat

* add torch style signature

* remove redundant function
Co-authored-by: Nhjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add kernel observer

* refine

* refine

* refine

* fix

* fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

This reverts commit 6d062579.

* delete single client nn  modules

* remove single client F
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* delete inplace warning

* add _trunc_normal_

* code format

* code format
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* broadcast support inplace
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* add memory usage test
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(AutoMatedTest): AutoMatedTest support test module.parameter.grad

* fix(randperm): fix randperm 0shape bug and fix test bug

* fix(BatchNorm): fix batch_norm cpu bug

* test(randperm): fix randperm test cpu_only bug

* auto format by CI

* test(Resnet50): update test_resnet50_with_bn weights

* update resnet50 weight path

* fix(BatchNorm): do not use inplace add for num_batches_tracked

* remove num_batches_tracked from batchnorm module

* add comment for batchnorm functor
Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* move get_local_rank

* move get_rank

* move get_world_size

* move  is_multi_client

* split_sbp

* delete  comment

* fix error

* fix error

* of_format

* of_format

* fix error

* fix error

* fix get_rank()()

* fix merge master error

* fix  merge error
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* dev full op

* align document with pytorch

* change according to review

* remove full in tensor.rst

* modify document
Co-authored-by: NLuyang <flowingsun007@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add module __call__ kwags

* add force_mirrored_forward  and consistent_forward

* fix comment

* fix comment
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add var to control only print once

* refein

* refein

* fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix 0dim inplace add

* add testcase

* fix test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* Rewrite pow and logical functional apis.

* bugfix

* Fix logical_or and logical_and unittest

* Fix logical_xor unittest

* Dev remove python functions (#6042)

* Fix logical_or and logical_and unittest

* Rewrite clip

* Fix clamp

* reformat

* Rewrite scatter and scatter_add

* Rewrite fmod and where

* Rename clip functor by clamp
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tensor.item tensor.tolist

* add test case

* refine

* auto format by CI

* fix ci fail

* fix doc test
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Graph image gpu decoder

* fix ofrecord path in CI

* fix to

* ImageDecoderRandomCropResizeOpExpr; CUDA < 10.2 warning; eager unimplement

* fix path
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* functional_api: fix build error in mac os

* functional_api: fix interpreter_test

* for all: replace device_type.pb.h with device_type.h

* value_types: impl hash

* revert tools change

* CI: mac only

* cmake: fix of_functional_obj

* cmake: fix of_functional_obj

* CI: revert
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* dev optim_optim_lr_scheduler_multisteplr

* change class name

* ensure values in list milestone is increasing

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* bugfix: data_transport_token_per_placement

* refactor TransportToken::NewDataTransportToken

* NewDataTransportToken(parallel_desc)

* fix bugs

* rename UniqueConsistentIdStorage to ConsistentIdStorage

* refactor TransportToken

* add TransportToken::src_rank and TransportToken::dst_rank

* harded coded thread_consistent_id

* StreamType::SupportingTransportInstructions()

* not thread_consistent_id in single-client mode

* explicit initialize all fields of TransportToken

* remove unused field TransportToken::rank_group_level_

* more bits for TransportToken::seq_id_

* fix oneflow_testexe error
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: clackhan <han_binbin@163.com>

* export pooling api

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add some test_case

* add autotest for topk

* add autotest for topk

* add rsqrt autotest

* add more tests

* add fmod autotest

* add tensor.fmod autotest

* fix autotest codegen bug

* add flip autotest

* add chunk autotest

* restruct constant op autotest

* auto format by CI

* add zeros and ones autotest

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix

* fix

* refine

* refine

* refein

* refine

* refein

* refei

* refein

* refein

* refein

* Revert "refein"

This reverts commit d6440b26.
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* NopKernel

* fix

* move kernel_utils
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* disable cuda_h2d stream

* gpu.LocalCallOpKernel is shared between device  and device
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Turn on NVCC's warnings

* refine

* fix

* use CMAKE_CUDA_COMPILER_VERSION

* 611

* CUDA 10

* clean

* fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* add tril op

* mod tril gradfunc

* add tril test, resolve review bugs

* mod tril test

* add fields init
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: NLuyang <flowingsun007@163.com>

* regist unfold ops

* unfinished

* unfold init

* move padding before and after

* add unfold module

* add functional

* add unfold kernel

* add unfold and fold op

* add init

* remove outdated file

* add fold op

* fix compile

* fix unfold kernel

* add unfold module

* add fold kernels

* add unfold and fold op

* Add functional impl

* add fold module

* add unfold and fold gradient func

* fix bug

* fix a bug

* just for debug

* just for debug

* just for debug

* need to solve segment fault

* Add fold test

* fix unfold and fold bug

* add check

* add dilation equal 2 test case

* fix comment

* fix comment

* add api docs

* fix comment

* remove annotation

* fix clang static anlysis
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix

* fix

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use copy_all_files_in_dir to replace copy_files, rm .so in build_wheel.py
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* restore build_wheel.py
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* trigger ci
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* Remove python function wrapper.

* Increase ref

* Fix doc test

* Update python/oneflow/framework/docstr/utils.py
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: NYao Chi <later@usopp.net>

* add naive logging

* refein

* refein

* check in naive iterface

* refein

* impl more function

* refein

* add naive warning

* refein

* refein

* add module

* add py callback

* refein

* add more

* fmt

* BlockingContext => BlockingInfoContext

* refein

* refein

* refein

* refein

* refine

* remove unnecessary check

* fix for clang tidy

* skp

* auto format by CI

* revert
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* skip sync when abnormal exiting
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* cuda base cpu mpi boxing

* cpu_mpi

* fix conflicts

* add cpu mpi unittests

* more checks and unittests

* abstract_consistent_to_consistent_op_expr

* fix compiler complaint

* refactor consistent-to-consistent eager consisitent op interpreter

* fix compiler complaint

* refactor ConsistentToConsistentOpExpr

* lazy interpreter (#5903)

* fix bugs about consistent_id

* more test_consistent_cast unittests

* refactor functional::ToConsistent

* refactor GetNdSbp

* fix compiler complaints

* Update eager_consistent_op_interpreter.cpp

* Update eager_mirrored_op_interpreter.cpp

* eager_boxing_1_to_n

* add missing files

* del useless file

* minor fix

* refactor GetDevice4CurrentProcessCtx

* refine

* minor fix

* Update naive_1ton_boxing_interpreter.cpp

* refine

* Update eager_boxing_interpreter_mgr.cpp

* fix error

* fix error

* auto format by CI

* fix error

* refine

* make of_format

* Update nd_sbp.h

* fix consistent id check error

* refine

* back up

* refine

* minor fix

* refine

* refine

* refine

* minor fix

* minor fix

* refine

* refine

* Update nccl_boxing_function.cpp

* refine

* fix consistent meta check bug

* zoom kLimitParallelConfString

* refine

* Update symmetric_x_to_b.cpp

* Update boxing_dividor_util.h

* Update boxing_dividor_util.cpp

* Update eager_boxing_interpreter_mgr.cpp

* auto format by CI

* Update asymmetric_x_to_b.cpp

* Update asymmetric_broadcast.cpp

* auto format by CI
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>
Co-authored-by: NLi Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tensor.xxx codegen

* fix tensor shape bug

* add flow.xxx and tensor.xxx codegen, with Tensor args and kwargs

* add flow.xxx and tensor.xxx codegen, with Tensor args and kwargs

* add module callable codegen

* code format
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add broadcast_pow

* fix gpu mode & refine test

* refine

* AddOp to const&
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PlacedNdSbp

* 1) fix compiler complaints; 2) add an unittest

* refactor DECORATE

* backup code

* MainBoxingExpr

* 1) reformat; 2) fix compiler complaints; 3) add more unittests

* merge master
Co-authored-by: Nliufengwei <2472937968@qq.com>