* mig parallel_conf_util

* mig BuildInitialScope BuildScopeWithNewParallelDesc BuildScopeWithNewParallelConf

* add test of GetDeviceTagAndMachineDeviceIds

* mig GetOpConfSymbol

* fix BuildScopeWithNewParallelDesc input type error

* use TRY

* use symbol::Storage<OperatorConfSymbol>

* _NewOpKernelObject

* mig OpKernelObject

* mig object_storage

* make of_format

* del comment

* std::function<void(Object*)

* mig NewOpKernelObject and _StatefulCallOpKernel

* mig _StatefulCallOpKernel and GetSharedOpKernelObject4ParallelConfSymbol

* del object_storage.cpp

* use name GLOBAL_PARA_SYM2SHARED_OPKENEL_OBJ_MUTEX

* mig CheckRefInBlobObjectParallelDesc and  OperandBlobObjects rel api

* mig _StatelessCall

* mig _StatelessCall

* del comment

* use cfg_op_conf and Object*

* use Object*

* del _

* fix func name error

* use MapAt and shared_ptr

* use shared_ptr or const ref

* minor fix

* add todo

* minor fix
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* reduce unittest cost

* format

* refactor unittest for reduce cost

* refine code for review

* moments test double

* rollback top k shape dim

* fix axis err in test_prelu
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* initial tensor

* combine blobdesc in tensor, initial MirrorTensor

* fix typo

* add consistent blob, tensorimpl and outer consistent tensor

* add public method for tensor

* remove unrelated file change

* remove unrelated file change

* fix: reviewer suggestions

* fix: reviewer suggestions

* Delete device.cpp

* Add storage relative methods

* Remove unchange file

* Remove unchange file

* add storage relative tensor interfaces

* add some basic setter function

* remove expose blob concept

* remove useless header

* remove useless header

* add detail for new tensor implementation without blob desc

* reoriganized

* reoriganized

* add get/set for blob object

* refactor device remove is_lazy()

* replace method in tensor

* refactor device

* refactor base tensor

* remove header

* refactor tensor impl

* refactor tensor and device

* refactor tensor impl

* getters should return reference

* add final

* refactor tensor impl

* add final for tensor impl

* fix typo add #endif

* unsolved comments

* remove useless header

* add protected method for tensor impl

* compile and modified getters to return immutable parameters

* code format

* carefully deal with const property

* remove constness of blob object, rename parallel_conf to parallel_desc
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* disable_group_boxing and change nccl logical order to dst

* remove note

* both support insert nccl logical ops as close as possible to Src/Dst node
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove batch_axis

* rm partial tick

* temp fix bug of flow.math.add

* fix bug of input_blob_def.split_axis init

* [KEY] refine infer sbp order value consider logical shape enable split

* ignore op get sbp sign ERROR.

* filter and check valid sbp sign by logical shape; rm magic num

* fix bug of return

* merge rm sigmoid cross entropy op

* rm sigmoid batch axis fn

* more debug log for check valid in get sbp sign

* rm useless check
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Ntsai <caishenghang@oneflow.org>

Co-authored-by: Ncheng cheng <472491134@qq.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add CtrlConf Proto

* add HostListBootStrapClient

* add HostListBootStrapServer

* del OfOnceCall in host_list_boot_strap_client

* add BootStrapServer/Client

* Update control.proto

del rank2ctrl_addr

* add InitConfFromEnvDesc

* add log

* optimize code

* add CHECK

* InitCtrlConfFromEnvDesc

* del useless args def

* CtrlBootstrap

* minor fix

* refactor CtrlServer/CtrlClient with ProcessCtx

* RankInfoBootstrap

* fix bug and optimize

* minor optimize

* minor fix

* use WorkerProcessInfo

* minor optimize
Co-authored-by: Nlixinqi <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* quantize aware training support cambricon quantize formula

* support custom scope and cambricon backend
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* align with latest fake quant ops
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* align with latest observer ops
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* fix wrong moving_min_max_stop_update_after_iters type
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* quantize aware training support cambricon quantize formula
Co-authored-by: NLdpe2G <liangdepeng@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* only write auto gen header if different
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* remove unused code
Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add CtrlConf Proto

* add HostListBootStrapClient

* add HostListBootStrapServer

* del OfOnceCall in host_list_boot_strap_client

* add BootStrapServer/Client

* Update control.proto

del rank2ctrl_addr

* add InitConfFromEnvDesc

* add log

* optimize code

* add CHECK

* InitCtrlConfFromEnvDesc

* del useless args def

* CtrlBootstrap

* minor fix

* refactor CtrlServer/CtrlClient with ProcessCtx
Co-authored-by: Nlixinqi <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Optimize RngTruncatedNormal

* np.zeros -> np.empty, remove np.where in index
Signed-off-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Migrate sigmoid_cross_entropy_with_logits to user op

* add unittest

* fix license

* sigmoid cross entropy elemwise functor

* add LabelType

* remove KernelUtil::Mul

* replace logf and expf with LogFunctor and ExpFunctor
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove deprecated checkpoint init code in ops unittest

* retain deprecated api for session init in test_interface_op_read_and_write

* higher TF_CPP_MIN_LOG_LEVEL

* rm outdated tests

* larger tol
Co-authored-by: Ntsai <caishenghang@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix reduce_sum_like sbp

* refine

* rename
Co-authored-by: Nguoran <guoran@oneflow.org>

* add CtrlConf Proto

* add HostListBootStrapClient

* add HostListBootStrapServer

* del OfOnceCall in host_list_boot_strap_client

* add BootStrapServer/Client

* Update control.proto

del rank2ctrl_addr

* add InitConfFromEnvDesc

* add log

* optimize code

* add CHECK

* InitCtrlConfFromEnvDesc

* del useless args def

* CtrlBootstrap

* minor fix
Co-authored-by: Nlixinqi <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* startup of device_registry, device_registry_mgr

* refine(move GetDeviceString from devRegMgr to oneflow namespace)

* Add some new Functions related to ToString. (#4185)

* Add some new Functions related to ToString.

* fix code according to review advices.

* fp16 registration pseudocode  (#4188)

* pseudocode for demonstrating
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* GetZeroVal pseudocode
Signed-off-by: Ndaquexian <daquexian566@gmail.com>

* float16 resolved, GetZeroValue unresovled

* refine GetZeroValue func

* refactor float16 type uitls

* remove  useless core/framework/tensor.cpp
Co-authored-by: Ndoombeaker <later@usopp.net>

* move calling cudaRuntimeGetVersion to cpp

* add device_register_gpu.cpp

* refine device_register_gpu.cpp

* try to fix cpu build error

* fix cpu version bug

* auto generated dev regs headers

* remove exception in DeviceTag4Type/DeviceType4Tag temporarily

* fix including bugs for python kernel test

* refine according to review comments

* reformat
Co-authored-by: NWang Tuo <58287331+WangTuoxyty@users.noreply.github.com>
Co-authored-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* test_style_transfer and refine inference session

* fix bug

* rm debug info

* update old test

* format

* revert include header

* refine ofrecord dataset

* add FaceEmoreRecordDataset

* test insightface save and load

* format

* change default model dir

* rm useless code

* add args for convenience of test

* modify by review comment

* support print stack for serving

* new event loop when old one was closed

* fix
Co-authored-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: NTsai <caishenghang@oneflow.org>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: Nvycezhong <izhongyuchen@gmail.com>
Co-authored-by: Ndaquexian <daquexian566@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove PartialTick

* remove op_conf.has_partial_tick_conf()
Co-authored-by: NLi Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add NCCL_VERSION_CODE > 2700 macro in nccl logical kernels

* refine macro position
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* rm LocalWorkStreamId

* rm AllocateLocalWorkStreamId in TaskNode

* rm local work stream id in task node and commnet task node

* rm local_work_stream_id param in NewTaskId

* fix test
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor CtrlClient

* RpcServer

* use Rpc

* Update rpc_client.cpp

add comment
Co-authored-by: Nlixinqi <lixinqi0703106@163.com>
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove keep_header_only and BlobDesc::is_body_disabled

* Remove InputBlobModifier::use_header_only and UserOps set_use_header_only

* mig parallel_conf_util

* mig BuildInitialScope BuildScopeWithNewParallelDesc BuildScopeWithNewParallelConf

* add test of GetDeviceTagAndMachineDeviceIds

* mig GetOpConfSymbol

* fix BuildScopeWithNewParallelDesc input type error

* use TRY

* use symbol::Storage<OperatorConfSymbol>

* _NewOpKernelObject

* mig OpKernelObject

* mig object_storage

* make of_format

* del comment

* std::function<void(Object*)

* mig NewOpKernelObject and _StatefulCallOpKernel

* mig _StatefulCallOpKernel and GetSharedOpKernelObject4ParallelConfSymbol

* del object_storage.cpp

* use name GLOBAL_PARA_SYM2SHARED_OPKENEL_OBJ_MUTEX

* del comment

* use cfg_op_conf and Object*

* use Object*

* del _

* fix func name error

* Update object_storage.cpp

rename mutex
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Enable insert nccl logical op pass

* FindMaxConnectedSubgraphForGpuExecOrder~

* through order and interface

* implement of insert nccl logical op in pass

* add nccl logical op using UserOp Implement and EagerNcclCommMgr

* add NCCL ReduceScatter op/kernel; refine pass impl of topo order

* add NCCL logical op/kernel AllGather

* fix bug of reduce scatter/ all gather infer shape

* refine log and note

* fix complier err build with CPU ONLY

* support NCCL ALL2ALL and test pass of alexnet model parallel

* rollback of diff in checkpointing_pass.cpp

* rename to nccl_use_compute_stream; ResourceDesc::nccl_use_compute_stream; refine name for review; create nccl_comm_ in KernelCompute;

* refine code for review

* add unittest for nccl use compute stream

* format test scripts

* refine align

* Refactor InferBatchAxis

* refine

* source subset tick

* remove useless header files

* insert DstSubsetTickOp

* remove incorrect CHECK

* add tick op for each machine

* TryBindBnWithOneofRegst

* add sink tick op in main_job

* refactor LinkMainJob

* fix typo in task_graph

* refactor AddGlobalCriticalSection

* rename and refactor DstSubsetTick::InferBlobDescs and SrcSubsetTick::InferBlobDescs

* add src_subset_tick for input-output critical section

* refactor AutoSourceTick and AutoSinkTick

* vectorizedly link main job

* resize vectorh identity_tick_op_names then access elements

* SrcSubsetTickCompTaskNode: bind bns and in_regst if bns is valid in current device

* refactor optional input to repeated inputs for SrcSubsetTickOpConf

* fix a bug in CaseCompTaskNode; fix a bug when create identity tick in main_job

* 1) Insert tick between sourc tick and src_subset_tick; 2) Insert tick between dst_subset_tick and sink tick

* stash code

* refactor MakeMainJob by using Range::ForEachSubRange

* refactor MakeMainJob by using Range::ForEachSubRange

* rename ReentrantLockLinkPoint to ReentrantLockBackEdge

* set piece id for regst sent by wait_and_send_ids actor

* callback_notifier_sink_tick
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* source subset tick

* remove useless header files

* insert DstSubsetTickOp

* remove incorrect CHECK

* add tick op for each machine

* TryBindBnWithOneofRegst

* add sink tick op in main_job

* refactor LinkMainJob

* fix typo in task_graph

* refactor AddGlobalCriticalSection

* rename and refactor DstSubsetTick::InferBlobDescs and SrcSubsetTick::InferBlobDescs

* add src_subset_tick for input-output critical section

* refactor AutoSourceTick and AutoSinkTick

* vectorizedly link main job

* resize vectorh identity_tick_op_names then access elements

* SrcSubsetTickCompTaskNode: bind bns and in_regst if bns is valid in current device

* refactor optional input to repeated inputs for SrcSubsetTickOpConf

* fix a bug in CaseCompTaskNode; fix a bug when create identity tick in main_job

* 1) Insert tick between sourc tick and src_subset_tick; 2) Insert tick between dst_subset_tick and sink tick
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* source subset tick

* remove useless header files

* insert DstSubsetTickOp

* remove incorrect CHECK

* add tick op for each machine

* TryBindBnWithOneofRegst

* add sink tick op in main_job

* refactor LinkMainJob

* fix typo in task_graph

* refactor AddGlobalCriticalSection

* rename and refactor DstSubsetTick::InferBlobDescs and SrcSubsetTick::InferBlobDescs

* add src_subset_tick for input-output critical section

* refactor AutoSourceTick and AutoSinkTick

* SrcSubsetTickCompTaskNode: bind bns and in_regst if bns is valid in current device

* refactor optional input to repeated inputs for SrcSubsetTickOpConf
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* source subset tick

* remove useless header files

* insert DstSubsetTickOp

* remove incorrect CHECK

* add tick op for each machine

* TryBindBnWithOneofRegst

* fix typo in task_graph

* rename and refactor DstSubsetTick::InferBlobDescs and SrcSubsetTick::InferBlobDescs

* SrcSubsetTickCompTaskNode: bind bns and in_regst if bns is valid in current device

* refactor optional input to repeated inputs for SrcSubsetTickOpConf
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* mig parallel_conf_util

* mig BuildInitialScope BuildScopeWithNewParallelDesc BuildScopeWithNewParallelConf

* add test of GetDeviceTagAndMachineDeviceIds

* mig GetOpConfSymbol

* fix BuildScopeWithNewParallelDesc input type error

* use TRY

* use symbol::Storage<OperatorConfSymbol>

* _NewOpKernelObject

* mig OpKernelObject

* mig object_storage

* make of_format

* del comment

* del comment

* use cfg_op_conf and Object*

* use Object*

Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* mig parallel_conf_util

* mig BuildInitialScope BuildScopeWithNewParallelDesc BuildScopeWithNewParallelConf

* add test of GetDeviceTagAndMachineDeviceIds

* mig GetOpConfSymbol

* fix BuildScopeWithNewParallelDesc input type error

* use TRY

* use symbol::Storage<OperatorConfSymbol>

* _NewOpKernelObject

* del comment
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add CTC Loss

* Add backward kernel

* Remove tf in test

* Update api document

* Add zero_infinity option

* refine

* Add 1n2d test case

* Switch to consistent_view

* Fix Eager mode

* Remove duplicate license

* Add grad check

* Fix bw test

* Fix bugs

* Add op name

* Refine

* of_format

* Expand annotation

* Performance optimizing for cuda

* Check input_length & target_lengths

* Update __syncthreads
Co-authored-by: Noneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>