Former-commit-id: d171189a960722606c087020fbb3263f12aa0777

Merge branch 'dev_all_reduce_strong_order' of https://github.com/Oneflow-Inc/oneflow into test-multi-socket


Former-commit-id: c47a69c2ead7a9f7007dccbbb1cee9cdc1181e9e

Former-commit-id: a461fe8eb9215a132475bd97ecc3647ce7913f7e

Former-commit-id: 71e94f021ce940edf3e676a41bd03de14cb23b96

Former-commit-id: 4281e1080da619cc19a7c5cb1f625de67827cefc

* fix a bug in OpGraph::InferNoParallelBlobDesc

* fix a bug in OpGraph::InferNoParallelBlobDesc


Former-commit-id: c0b1071fc6fbe72f1207d02fdb794dc1076eb59a

Former-commit-id: 82527249de96a849a3f85b567ff5173980783ba4

* Global<OpGraph> is only available duraing compilation

* small record_piece_size for InferNoParallelBlobDesc


Former-commit-id: 5eb1012703f8f9389ac8e2f16131bfd36411b0db

* mem_shared_hint_id

* sharable memory block

* rm useless code

* remove useless code

* bugfix: no redundant edges

* rename: MemBlockGroup => MemBlock

* put constrcutor of SharableMemBlockNode into header file

* bugfix

* rename field: MemBlock.block_id => MemBlock.mem_block_id

* replace piece_size with logical_blob_dim0

* BlobParallelConf

* BlobParallelDesc

* infer out blob model_split_axis

* int64_t => int32_t

* InferOutBlobParallelDesc

* gather out blob model split (#1624)

* InferBlobParallelDesc

* let variable op support kModelParallel

* rename lbi2blob_desc_ => lbi2no_parallel_blob_desc_

* Global<OpGraph>

* SplitLogicalInputBlobDesc

* ConcatOutputBlobDescs

* rename: BlobDataParallel => DataBlobParallel; BlobModelParallel => ModelBlobParallel; BlobGridParallel => GridBlobParallel

* OpGraph::CheckBlobDescs(...)

* exact division is unnecessary

* fix bugs

* rename InferOutBlob* => InferOutputBlob

* exact division in variable_op is unnecessary

* bug fix

* fix bugs

* fix bugs

* IsInputBlobAllowedModelSplit

* use Global<OpGraph> to InferModelSize

* add OpGraph::GetDataBalancedSplitter and OpGraph::GetModelBalancedSplitter

* fix IdentityOp::IsInputBlobAllowedModelSplit

* no implementation for pure virtual function Operator::IsInputBlobAllowedModelSplit

* refine BlobParallelDesc: replace CopyParallelConf with operator=

* refine ParallelDesc: remove unused functions

* more checks on ParallelDesc

* remove unused function Operator::MaxModelSplitNum

* bugfix: SoleOp() => op_vec().at(0)


Former-commit-id: be1f820b2927f7f79f55b7891f6575cdeb4b2053

Former-commit-id: 28615613e22379ae2135af09fb2aab1f23b1ec6c

* mem_shared_hint_id

* sharable memory block

* rm useless code

* remove useless code

* bugfix: no redundant edges

* rename: MemBlockGroup => MemBlock

* put constrcutor of SharableMemBlockNode into header file

* bugfix

* rename field: MemBlock.block_id => MemBlock.mem_block_id

* replace piece_size with logical_blob_dim0

* BlobParallelConf

* BlobParallelDesc

* infer out blob model_split_axis

* int64_t => int32_t

* InferOutBlobParallelDesc

* gather out blob model split (#1624)

* InferBlobParallelDesc

* let variable op support kModelParallel

* rename lbi2blob_desc_ => lbi2no_parallel_blob_desc_

* Global<OpGraph>

* SplitLogicalInputBlobDesc

* ConcatOutputBlobDescs

* rename: BlobDataParallel => DataBlobParallel; BlobModelParallel => ModelBlobParallel; BlobGridParallel => GridBlobParallel

* OpGraph::CheckBlobDescs(...)

* exact division is unnecessary

* fix bugs

* rename InferOutBlob* => InferOutputBlob

* exact division in variable_op is unnecessary

* bug fix

* fix bugs

* fix bugs

* IsInputBlobAllowedModelSplit

* use Global<OpGraph> to InferModelSize

* add OpGraph::GetDataBalancedSplitter and OpGraph::GetModelBalancedSplitter

* fix IdentityOp::IsInputBlobAllowedModelSplit

* no implementation for pure virtual function Operator::IsInputBlobAllowedModelSplit

* refine BlobParallelDesc: replace CopyParallelConf with operator=

* refine ParallelDesc: remove unused functions

* more checks on ParallelDesc


Former-commit-id: 2b78c6e1f37e514e39f1dc807ccce455190b00a7

Former-commit-id: c1ad6792caf7850e445ed91038c8ba3ecfced9e9

* accuracy

* accuracy_task_node add fw_buf

* fw_buf=>data_tmp


Former-commit-id: ab900061ce8f1050ec6d2ee39057addcfcd44f57

Former-commit-id: f3344a44fc17076fdbf1e1795d50289bfcd22623

Former-commit-id: e33f02f82811568987d02f1da6d5095bc96715f5

Former-commit-id: c7042f5397f00bec8d30795a356e581732064e89

* multithreads register coloring

* refine code


Former-commit-id: 3be07992331e7f62fe9d56b2f8d1357a72ef1a09

Former-commit-id: bfb3d322c107c14c1cae04b0cc96a56d78419c02

Former-commit-id: 2550030088fb6b15f1784a2bd1cfb78eeabe3b0d

Former-commit-id: 9fbbdc7ad2a68bd049add995e51c4c58b4286791

Former-commit-id: 3655dbad6d2d61ae84e0e484a3a5a2fed2fc66b2

Former-commit-id: 39f21e9c94f9195763f2d9d1ba1b3584b2d1b53c

Former-commit-id: af8ca7ba701ef2eb4c9e003a79d09bc92b3f8243

Former-commit-id: 37fcf27c3f69244c81eba4a41a61b86b6ffe26f1

Former-commit-id: 97b0367c3d63c40a2c2ba6a4b272796ed706088a

Former-commit-id: 8689609b90abed7538a728395ea9880cd5af9178

Former-commit-id: 92b9ff1ae7efc90391581eafe8253347a5712b9c

Former-commit-id: 76c25553437543749ed58495709824bcee8e0b55

* refine CHECK in AllReduce

* move ReduceConcatOpCtx definition to .cpp file


Former-commit-id: 5a50f692cb92c5a6a7074be2063cbc1ec325c1ca

* mem_shared_hint_id

* sharable memory block

* rm useless code

* remove useless code

* bugfix: no redundant edges

* rename: MemBlockGroup => MemBlock

* put constrcutor of SharableMemBlockNode into header file

* bugfix

* rename field: MemBlock.block_id => MemBlock.mem_block_id


Former-commit-id: 6a8fc14c2ba6bbe148a84458fa6119af16cbe672

Former-commit-id: 21079b5b27c7ac6fa0b8f137e2d960e2f631df9c

* share model_diff and out_diff in variable op

* bugfix: model_diff is a produced register

* register_num of model_diff is 1

* add VariableKernelConf

* no mutable

* bugfix


Former-commit-id: 2409c41f525875597e989f718434c30d91e9d874

* group by has_diff

* rm unnecessary identity


Former-commit-id: 63f2cfd61337c821e0fb6215b592231ccee584d4

Former-commit-id: 58b9b56676f69c50ea6cdc80f474a3bde8d4becb

* random shuffle

* fix

* refine

* refine

* single thread

* refine


Former-commit-id: 0dbb1f3d7265f9c55a11b07695efd092cd81a83c

* refine logic to infer reduce_concat_op's elem_cnt of out blob, still have bugs...

* add RoundUp in reduce_concat

* CHECK_LE -> CHECK_EQ

* add CHECK


Former-commit-id: 962817e2a322ba6452c9966bae87fb5da9d4a86a

Former-commit-id: dc508e494e7b9bfcd500d8f125390ab65874b792

Former-commit-id: 8ed2fd1609f3c3ee3b6cbe0a5d1f380afea8bd25

* pseudo chains of OpGraph

* ConvertPseudoChainToChain

* refine pseudo_chain

* refine register coloring algorithm

* rename op_graph log file name

* remove unused code

* EnableMemSharingInVariableOp

* no mem_sharing for out_diff & model_diff in variable_op


Former-commit-id: 224bb63a0576e7da8929ab829dfcf7c709398a97

Former-commit-id: dca95c0e8bd46e73e52fe09d0be388c881b47c82