* add clip_grad to optimizer

* add clip grad unit test to sgd optimizer

* add sgd graph clip grad test

* refine

* format

* refine

* fix

* merge master

* refine

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* where broadcast forward

* delete py broadcast code

* delete op_expr

* add where backward broadcast gradient

* scalar x op

* add where scalar_x, scalar_y, scalar_xy ops & kernels

* add where scalar_x, scalar_y, scalar_xy functors

* fix bug

* add where backward

* add pybind api

* add testcase

* fix

* fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove GlobalWorkStreamId/GlobalThrdId

* refine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add randperm with test and docs

* format code

* format

* fix docs

* format the code and add more tests

* format code

* Update test_randperm.py

* Update randperm.py

* add head

* format codes

* docs

* Update test_randperm.py

* Update test_randperm.py

* add more tests

* format

* Update randperm.py

* Update randperm_kernel.cu

* Update randperm_kernel.cu

* Update randperm_kernel.cpp

* reconstruct the code

* format the code

* 2

* s

* 1

* 1

* refine

* add more test

* refine code

* fix according to comment

* add more test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* lrs

* prototype lrs

* add cosine lr scheduler conf

* add opt group

* test opt group

* add comment

* add comment

* add return for lr conf

* check same optimizer in the opt group

* check same optimizer in the opt group

* check cosine lr compute

* cosine lr_sch pass test

* add_optimizer

* rm outdated comment

* make lr_sheduler optional and add test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* ref count del of session and env

* auto format by CI

* add test

* pass graph in global test

* auto format by CI

* rm cuda

* rm debug lof

* rm debug info

* move graph del sync from c to python

* rm log

* auto format by CI

* refine test to pass ci

* fix env sync call

* auto format by CI

* deal with graph destruction when graph not compiled

* deal with not compiled graph destruction

* print log

* deal with single client sync

* auto format by CI

* rm py object ref

* rm python obj ref count

* add log of env

* address comment

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix(*): fix inplace op circle reference bug

* refine code

* format
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* process api dtype

* process api tensor

* small fix

* fix to name func

* still error

* still error

* revert master

* fix symbol dtype part1

* import error

* still error

* still error

* fix symbol

* add Dtype4Datatype function

* work in process

* undefined symbol of tensorbuffer

* macro bug

* construction func return symbol dtype

* process deprecated api

* fix name

* fix bug

* change to use reference

* fix format

* fix name

* small fix

* fix symbol dtype

* modify functor datatype to symbol dtype

* fix to symbol dtype

* add object as for symbol dtype

* modify return symbol dtype

* modify functor

* modify autograd section

* modify autograd

* modify core

* modify back for functional

* modify functional impl

* modify value types

* modify value types

* fix args

* modify tensor

* modify back

* fix comment

* small fix

* auto format by CI

* fix tensor buffer bug

* modify new interface

* modify new interface

* remove annotation

* auto format by CI

* need refine

* remove dtype4datatype

* modify use Get

* auto format by CI

* small fix

* auto format by CI

* remove annotation

* houjiang fix to symbol dtype pointer

* fix dtype

* fix format

* remove annotation

* small fix

* add null ptr support

* auto format by CI

* use symbol dtype

* fix

* auto format by CI

* small fix

* fix in lazy op interpreter

* auto format by CI

* fix to use symbol dtype

* small fix to use equal instead of is in single client

* fix conflict

* Format

* Fix to use symbol dtype
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* restruct sort op

* add sort autotest

* restruct argsort and add autotest

* add tensor autotest

* auto format by CI

* add l1loss autotest

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix bug of SyncOnMasterFn atexit

* move sync to func
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Sync access and assign indexing tensor.

* Remove unused comments.
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* try to fix prelu insightface bug

* fix comment

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* restruct logsoftmax and abs test

* add hardtanh test

* refine batchnorm autotest

* add meshgrid autotest

* add pow autotest

* add stack autotest

* delete prelu useless code

* change Stack Module Test Api

* fix comments

* fix softmax bug

* fix sign bug

* fix sign

* auto format by CI
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add fmod

* add grad

* fotmat and delete comment

* delete  comment

* delete comment

* delete line

* rename and format

* merge  master

* format

* fix error , add data type

* add int8 ,delete  f16

* modified mod example

* auto format by CI
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>

* Support combined_margin_loss op in flow.nn.modules

* Follow review comment to modify

* Follow review comment to modify

* auto format by CI
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix conv1d conv2d conv3d useless args

* fix showing bugs

* add args for unsample

* fix torch.float
Co-authored-by: NYao Chi <later@usopp.net>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* functional.onehot added

* functional api yaml updated

* api yaml updated

* add one_hot functional API

* add Module

* amend one_hot testcase

* amend docstring

* delete Module

* amend review question

* amend review question

* amend flow.nn.functional.one_hot

* amend functional init.py

* delete one_hot import code

* amend valueerror

* auto format by CI

* amend numclasses is -1

* add testcase

* update test_one_hot

* auto format by CI

* amend one_hot param

* amend docsting

* amend onehot.py docstring

* amend on_value and off_value

* auto format by CI

* amend docsting error

* remove onehot

* auto format by CI
Co-authored-by: Ntangnana925 <85614052+tangnana925@users.noreply.github.com>
Co-authored-by: Ntangnana <tnn_personal@163.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: MARD1NO <359521840@qq.com>

* abstract InferSourceOpParallelDistribution

* support specified device, add consistent arange.

* format

* Fix arange python api and unittest

* Broadcast parallel default

* Fix wrong merge

* Fix error merge
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add cmake option USE_SYSTEM_NCCL

nccl target

* rm mark_as_advanced

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix gather kernel 0 shape

* recover module test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add logical scalar kernel

* add logical scalar op register

* add functional api yaml

* modify math functor

* fix

* reuse functor

* fix

* modify equal

* modify greater

* modify greater equal

* modify less equal

* modify less than

* add not equal

* modify not equal

* fix format

* remove partial sum

* add newline

* reuse base class

* fix bin_op to binary_op

* modify to Scalar

* first restruct and anotate cuda

* modify to no grad user op

* restruct code and add dtype

* export to pybind

* remove redundant logic in python

* bind python as false

* remove annotation

* fix dtype

* support scalar in input or output

* fix

* Add magic method

* add docs

* auto format by CI

* fix randn test

* modify back

* small fix

* fix 0d tensor

* auto format by CI

* fix 0d test

* auto format by CI

* fix ddp bug
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix to use is

* remove [0] in ddp.py
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix unittest

* fix format

* fix wrong unittest

* skip free eager test

* fix to use is not none
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* Fix BUG of LazyInterpret FreeEagerTensor memory shared with regst

* remove note

* remove debug

Co-authored-by: NZZK <42901638+MARD1NO@users.noreply.github.com>

* Rename `ParallelDistribution` to `NdSbp`

* Rename `ParallelDistribution` to `NdSbp`

* Rename `ParallelDistribution` to `NdSbp`

* auto format by CI

* Rename `ParallelDistribution` to `NdSbp`
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add flow.rand

* update docstr

* update docstr

* add consistent_rand, add more tests

* update random op

* refine

* refine, add range and int type to uniform_kernel

* refine

* refine

* update doc

* update doc

* Refactor UniformDistribution

* fix
Co-authored-by: Nhjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* SyncAccessBlobByCallback

* refactor capture-by-reference to capture-by-value

* refactor InstructionsBuilder::SyncAccessBlobByCallback
Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* cmake first class cuda support

* refine

* refien

* refine

* refein

* refein

* refeine

* refine

* refein

* refine

* refien

* refgine

* refien

* refein

* refein

* rm useless

* refien

* refein

* also link cuda libs if build static

* refein

* refien

* add

* Revert "add"

This reverts commit d9e67ad1.

* fix

* refeine

* retine
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Tensor): support Tensor.__bool__()

* test(Tensor): add tensor to bool test

* docs(Tensor): refine is_nonzero document

* format

* fix(Tensor): fix Tensor.__bool___ bug

* auto format by CI

* fix(instancenorm): fix merge bug

* fix(*): fix merge bugs
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* refine code

* refine code

* optimize code

* refine code

* refine

* back up

* add tensor.to func

* make of_format

* remove to in pyTensor

* sync gpu data

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* backup

* refine

* rebase

* check in gen py

* merge master and fix bugs

* address pr comments

* eager boxing

* address pr comments

* fix b2p error

* auto format by CI

* remove boxing

* export sbp

* add tensor to_consistent

* /minor fix

* minor fix

* refine

* remove useless head file

* Fix optional

* remove to in tensor.cpp

* update

* Support symbol placement type in functional.

* add sbp and sbp list arg

* refine

* use functional

* refactor CastConsistentOpExpr

* to_consistent(flow.B) backward

* Cache op expr

* add EagerNcclOpKernelState

* refine

* refine

* refine

* refine

* refine

* refine

* minor fix

* capture OpInterpContext

* unimplemented apply

* add GetNdSbp

* add mutex

* refine

* merge EagerConsistentTensorImpl::NewWithPhyTensor and EagerConsistentTensorImpl::NewWithoutPhyTensor into EagerConsistentTensorImpl::New

* rename functiona SyncData to SyncMetaAndData

* fix function yml

* refine

* refine

* refine collective boxing

* make of_format

* of_format

* add to_local to pybind

* refactor EagerBoxingInterpreter

* minor fix

* optimize CastParallelDistribution

* add placement_sbp_util

* minor fix

* eager boxing backward

* minor fix

* sync shape and data when tensor_to_local

* fix rpc_token bugs

* fix p2s backward bug

* refactor AsyncRpcCtx

* set logical_shape correctly

* simplify implementation of consistent_tensor.to_local

* refine

* initialize rpc_token with zero

* refactor grad functions of to_consistent/to_local

* refine

* reformat and address pr comment

* reformat

* add check_meta_consistency in consistent2sonsistent

* refactor eager_nccl_reduce lernel

* refine

* refine to_consistent api

* ban_non_pod_data_in_eager_boxing

* refine

* refine

* refine

* backup code

* THREAD_LOCAL_CACHED

* Delete thread_local_cache.h

* bugfix: DeviceId4ParallelId -> MachineId4ParallelId

* optimize

* support tensor str

* Init code and can print consistent

* refine format

* remove useless to_consistent and format

* refine code and print according data

* attempt to support multi rank when fetch data

* Revert "attempt to support multi rank when fetch data"

This reverts commit ae56afad.

* skip if tensor is consistent

* delete useless

* add comment

* delete useless

* traversal data to determine if int_mode

* if consistent, return [...]

* refine

* add test and fix bug

* add more assertTrue and delete useless

* getitem using integer return scalar when tensor shape is [1]

* add test cast

* refine

* fix spelling mistake

* add op test and enhance in parse device

* fix bug

* fix docstr test bug and support to print meta

* refine

* auto format by CI

* fix docstr in clip_grad.py

* fix docstr

* fix docstr and bug

* the input shape parameter of reshape changed

* add with flow.no_grad when operate tensor

* fix docstr
Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: Ntsai <jackalcooper@gmail.com>
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: NLi Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Nhjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Nwyg1997 <wyg19970408@gmail.com>
Co-authored-by: Ncheng cheng <472491134@qq.com>

* refine code

* optimize code

* refine code

* refine

* back up

* add tensor.to func

* make of_format

* remove to in pyTensor

* sync gpu data

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* backup

* refine

* rebase

* check in gen py

* merge master and fix bugs

* address pr comments

* eager boxing

* address pr comments

* fix b2p error

* auto format by CI

* remove boxing

* export sbp

* add tensor to_consistent

* /minor fix

* minor fix

* refine

* remove useless head file

* Fix optional

* remove to in tensor.cpp

* update

* Support symbol placement type in functional.

* add sbp and sbp list arg

* refine

* use functional

* refactor CastConsistentOpExpr

* to_consistent(flow.B) backward

* Cache op expr

* add EagerNcclOpKernelState

* refine

* refine

* refine

* refine

* refine

* refine

* minor fix

* capture OpInterpContext

* unimplemented apply

* add GetNdSbp

* add mutex

* refine

* merge EagerConsistentTensorImpl::NewWithPhyTensor and EagerConsistentTensorImpl::NewWithoutPhyTensor into EagerConsistentTensorImpl::New

* rename functiona SyncData to SyncMetaAndData

* fix function yml

* refine

* refine

* refine collective boxing

* make of_format

* of_format

* add to_local to pybind

* refactor EagerBoxingInterpreter

* minor fix

* optimize CastParallelDistribution

* add placement_sbp_util

* minor fix

* eager boxing backward

* minor fix

* sync shape and data when tensor_to_local

* fix rpc_token bugs

* fix p2s backward bug

* refactor AsyncRpcCtx

* set logical_shape correctly

* simplify implementation of consistent_tensor.to_local

* refine

* initialize rpc_token with zero

* refactor grad functions of to_consistent/to_local

* refine

* reformat and address pr comment

* reformat

* add check_meta_consistency in consistent2sonsistent

* refactor eager_nccl_reduce lernel

* refine

* refine to_consistent api

* ban_non_pod_data_in_eager_boxing

* refine

* refine

* refine

* backup code

* THREAD_LOCAL_CACHED

* Delete thread_local_cache.h

* bugfix: DeviceId4ParallelId -> MachineId4ParallelId

* optimize

* minor fix

* LazyInterpreterApplyImplForParallelCastOpExpr

* rm eager constraint

* c2c interp ctx with parallel info

* multi client collective boxing

* test_to_consistent

* support to_consistent grad_sbp

* AsConsistentTensor

* pass bwd test

* add multi graph test

* add ConsistentToConsistentOpExpr

* LazyConsistentToConsistent

* interpret ConsistentToConsistentOpExpr

* update test

* rm useless code

* auto format by CI

* fix conflict

* mod comment

* add message for local_tensor.to_consistent() check and consistent_tensor.to_local() check in lazy

* address review

* fix conflict

* rm check which limit placement changing

* auto format by CI

* fix nd_sbp

* auto format by CI

* refactor to.py

* ConsistentToConsistentOpExpr catch free tensor

* fix copy op's sbp inferring

* refactor empty infer sbp

* refactor constant infer sbp

* mod coco reader sbp inferring

* fix GetSbpFn

* fix consistent_to

* fix (#5857)
Co-authored-by: Nleaves-zwx <kunta0932@gmail.com>

* modify comments

* add test_to_placement case

* clear code

* unready test

* refactor with InferNdSbp4SrcOp

* rm out-dated comment

* tidy code

* SBP str -> cfg::SbpParallel
Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: Ntsai <jackalcooper@gmail.com>
Co-authored-by: NXinqi Li <lixinqi0703106@163.com>
Co-authored-by: NLi Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Noneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Nhjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: NLiang Depeng <liangdepeng@gmail.com>

* GetBroadcastGroup

* fix comment typo.

* broadcast shape and dtype

* 1) rm THREAD_LOCAL_CACHED; 2) fix bugs in ThreadLocal

* fix wrong use of LocalRank

* revert several code from master

* fix compiler complain

* merge master
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>