* initial commit, add code for async construct tensor from numpy array

* inital commit to change Maybe to Optional

* delete redundant code

* replace Maybe with Optional

* fix compile errors

* format code

* changes based on review

* format code, fix based on review

* format code

* fix multiclient type

* changes based on review

* changes based on review

* unify calling to IsMultiClirnt

* refector multi_client related code

* restore InMultiClient interface

* double check for unnecessary changes

* remove unnecessary changes

* format code

* Update oneflow/api/python/symbol/job_conf_symbol.cpp

* Update oneflow/api/python/symbol/op_conf_symbol.cpp

* Update oneflow/api/python/symbol/op_node_signature_symbol.cpp

* Update oneflow/core/common/optional.h

* Update oneflow/api/python/symbol/string_symbol.cpp

* Update oneflow/api/python/symbol/scope_symbol.cpp

* Update oneflow/api/python/symbol/placement_symbol.cpp

* Update oneflow/api/python/symbol/op_conf_symbol.cpp
Co-authored-by: NHoujiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: NTwice <i@twice.moe>

* mig EagerPhysicalBlobHeader

* slove dtype

* mig EagerPhysicalBlob partical

* mig EagerBlobTrait

* fix EagerBlobTrait shape property

* add CHECK

* slove numpy

* enroll blob_trait

* replace EagerPhysicalBlob with oneflow_api.EagerPhysicalBlob

* replace LazyBlob with oneflow_api.LazyBlob

* fix a SyntaxError

* mig eager_blob

* replace EagerBlob with oneflow_api.EagerBlob

* move parallel_size to c++

* Adjust parameter order

* rename fun

* adjust condition

* del useless fun

* refactor_string_symbol

* run make of_format

* rename blob_type to blob_class

* add eager

* conv's filter_diff is never dynamic

* fix test_coco_reader bug at eager mode

* add blank line at the end of file

* skip model_io & interface_op_read_and_write test case under eager mode

* sikp ops/test_sparse_softmax_cross_entropy_ms under eager mode

* skip test_softmax case under eager mode

* fix test_TestMultiOutputOrder bug

* fix test_pool bug under eager mode

* fix VirtualCloseJob() bug

* fix bug test_pool_pad

* Fix eager mode bug (#3681)

* fix(eager): close eager job but not release Glabol<JobDesc>

* refactor(job): refactor job_build_mgr when closing

* style(*): use clang format

* chore(gdb): remove all gdb function

remove gdb.h(cpp) and calling in kernel.cpp

https://github.com/Oneflow-Inc/oneflow/pull/3681#discussion_r505137787

* fix test_sparse_cross_entropy_ms bug

* fix code style

* add TODO in test_pool & test_pool_pad

* fix test_image_resize bug

* remove magic num

* fix code format
Co-authored-by: NShenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: Nlixinqi <lixinqi0703106@163.com>
Co-authored-by: Nwyg1997 <wyg19970408@gmail.com>

* add license at root dir

* check in empty files

* rm space

* check in script

* update script

* fix bug

* add print

* fix

* add exit

* add to of_format

* add CI task

* fix license

* Revert "fix license"

This reverts commit 818b6d7691d3a8b4a25dd41a47ff2c5922b8ec57.

* only add once

* quick fix

* fix script

* dont fmt empty file

* fix

* quick fix

* fix py

* add license

* fix exit

* add license for hpp

* add license

* license new vm files
Co-authored-by: Ntsai <caishenghang@oneflow.org>

* cmake find python note when version less 3.14 (#2286)

* fix bug: reduce split kernel inplace (#2297)

* Dev bias add (#2299)

* use bias add

* fix

* bias_add

* bias add half

* fix

* reinterpret_cast

* fix half

* HALF

* fix

* ADD_DEFAULT_KERNEL_CREATOR

* fix

* format

* Fix dev python test (#2294)

* add decode random

* fix decode random actor

* fix dev_python test scripts

* fix batch_size test scripts

* fix

* Memory Version 2.0 Step 2:  MemSharedAndReused between jobs (#2267)

* MemBlockProto and ChunkProto

* create mem block and chunk after improver

* interface merge mem block and chunk between sub plans

* merge chunk between jobs for memory reuse

* using memory zone unique id replace memory case hash

* merge interface op mem block between jobs for mem shared

* gen GlobalCriticalSection by mem block id and chunk id

* check mem block and chunk valid before runtime

* Refactor: RegstMgr ;  allocate memory by mem block and chunk instead of regst

* fix bug; and pass test

* fig bug: init chunk_id_count in id_manager

* reuse copyHd out mem between jobs

* PushPlan and PullPlan for memblock and chunk

* refine merge mem block / chunk in oneflow.cpp

* at(i);

* GetOpName2JobId2TaskProtos functional

* using output ptr; pass test AlexNet and Resnet

* Dev cuda 9 arch 70 (#2318)

* kCudaAlignSize = 256

* always compute_70

* __CUDA_API_VERSION >= 10000

* __CUDA_API_VERSION >= 10000

* disable_all_reduce_sequence

* Fix cuda9 cudnn turing issue (#2329)

* fix cuda 9 issus on turing device

* CUDA_VERSION

* no cuda check

* bias add kernel gpu half (#2330)

* mem_block=>header_mem_block (#2338)

* speedup oneflow compilation

* identity_sbp_conf

* DropOut Version2 (#2355)

* random mask like op conf; refine dropout op in python

* remove useless dropout kernel conf

* implement of random mask like op

* refine dropout op

* refine dropout grad op

* refine generate dropout backward

* random mask like kernel

* refine dropout (grad) kernel

* fix link problem for template separated compile

* fix bug and pass test

* dropout kernel for half

* add check for dropout mask input data type

* bugfixs

* Remove IsOpFloat32() in auto_mixed_precision.cpp (#2358)

* fuse op/kernl to 1 cpp

* refine for review

* fix bug

* Refactor Kernel Registry for more flexible registration (#2363)

* feat: update KernelRegistration and add KernelRegValProto

* Refactor Kernel Registry for more flexible registration

* Remove unused kernel_reg_value.proto

* Memory Version 2.0 Step 3: MemReused in job (#2319)

* use_memory_allocation_algorithm_v2 for switch improver mem block id

* reuse plan task graph and ctrl edge for inferred mem block

* refine interface; InJobMemSharingUtil

* navie merge memory big chain; gen regst apply/release queue; handle for inplace hint regst

* generate regst 2 mutual exclusion regsts

* bugfix: apply should before release

* interface for multi-thread run algorithm get mem block offset result

* selet best algorithm to set mem block id and mem block offset

* set mem block for inplace consumer regst

* 3 algorithm interface

* half implement of algo 1

* implement of algorithm0_OfColorImproved

* runnable in 1 machine 1 device

* Memory Chain

* merge MemoryChain and pass Correctness test of alexnet and resnet50

* bugfixs: continues inplace consume relationship in bert-base fp16

* erase useless info in MemoryChain

* implement of BfcAllocator and Tf_Bfc algorithm

* use bfc algo and fix bug

* only use default algo

* renme in_job_* => intra_job_*

* rename: InJob* => IntraJob*

* rename: 1) apply_regsts_queue => alloc_regsts_queue; 2) release_regsts_queue => free_regsts_queue

* rename function name in job/intra_job_mem_sharing_util.cpp

* rename variable names in job/intra_job_mem_sharing_util.cpp: 1) *apply* => *alloc*; 2) *release* => *free*

* refactor FindFreeOffset => FindFreeOffsetAndNewBufferSize

* rename method: DeallocateRaw => FreeRaw

* rename varable for review

* use enum for mem reused algorithm and add python interface

* fix sbp infer (#2373)

* mv addr calculation out of decoder (#2374)

* use tmp blob for temp storage (#2375)

* INDEX_DATA_TYPE_SEQ (#2381)

* refine include (#2382)

* refine include

* format


format

* element_wise_mul (#2383)

* gather refine (#2384)

* Dev fix sbp (#2388)

* fix sbp

* fix sbp

* remove VirtualGenKernelConf

* rename Read to ReadFully (#2389)

* Dev parallel cast (#2391)

* parallel cast

* op_conf

* refine

* Dev auto zero padding (#2393)

* auto_zero_padding

* auto_zero_padding

* fix

* fix input_mask and token_type_id (#2398)

* fix job launch (#2401)

* fix sbp bug (#2402)

* fix sbp

* fix

* add missing header files (#2410)

* refactor cnn model tests (#2411)

* refactor cnn model tests

* reformat README.md

* reformat README.md

* refactor ndarray_reduce (#2412)

* fix inplace reachability bug (#2413)

* refactor gpu relu (#2414)

* refactor gpu relu

* CHECK_KERNEL_SAFE_INT32

* there may be a subtle cuda bug in ((float) x < 0)

* refactor ndarray_reduce (#2405)

* refactor ndarray_reduce

* refactor relu/bias_add

* refactor relu

* refactor relu

* refactor bias_add

* refactor relu/bias_add

* fix inplace_lbi bug

* refactor addition

* IsKernelSafeInt32

* CUDA_1D_KERNEL_LOOP_T

* CUDA_1D_KERNEL_LOOP_T

* If add (#2415)

* refactor ndarray_reduce

* refactor relu/bias_add

* refactor relu

* refactor relu

* refactor bias_add

* refactor relu/bias_add

* fix inplace_lbi bug

* refactor addition

* IsKernelSafeInt32

* CUDA_1D_KERNEL_LOOP_T

* CUDA_1D_KERNEL_LOOP_T

* add unless oprand is nonzero

* Clear session (#2416)

* oneflow.clear_default_session

* fix bugs in oneflow.config.machine

* refactor function return type (#2417)

* fix for py2 (#2418)

* blob parallel conf

* Pr watch scope (#2419)

* pr oneflow.watch*

* merge more code to pass watch_scope.py

* TODO: input_blob_def.parallel_conf

* fix reexport of identity op

* merge dev_quick_dirty_object_detection

* oneflow.cluster (#2423)

* oneflow.cluster

* no alias for oneflow.cluster.*

* mv cpp_logging_conf from config_proto to cluster_proto

* rename: cluster => env

* rename: Environment => Session

* Free port (#2427)

* oneflow.cluster

* no alias for oneflow.cluster.*

* mv cpp_logging_conf from config_proto to cluster_proto

* rename: cluster => env

* rename: Environment => Session

* auto find a free port for single node environment

* localhost only

* Dev single processor test (#2430)

* oneflow.cluster

* no alias for oneflow.cluster.*

* mv cpp_logging_conf from config_proto to cluster_proto

* rename: cluster => env

* rename: Environment => Session

* auto find a free port for single node environment

* localhost only

* single process test

* Cluster::WorkerLoop

* delete unnecessary OF_BARRIER_ALL

* no longer fork children processes to run tests

* format

* fix align byte size bug (#2436)

* fix align bugs (#2440)

* fix: GetNumOfLoDLevels lack return

* minor script fix and update

* update script

* remove redundant function

* oneflow.cluster

* no alias for oneflow.cluster.*

* mv cpp_logging_conf from config_proto to cluster_proto

* rename: cluster => env

* rename: Environment => Session

* Fix python internal (#2133)

* Return error meassage in oneflow_internal

* Refine environment_objects_scope

* add OF_ERROR_STR_CHECK and OFStrCat()

* format

* fix based on review

* fix(oneflow_internal.h): add undef

* fix: expr -> (expr)

* feat: update oneflow_internal_helper to use func