Former-commit-id: e9ca9288a04d3ce65ef1c711b34c916b29151dbb

* fix a bug in OpGraph::InferNoParallelBlobDesc

* fix a bug in OpGraph::InferNoParallelBlobDesc


Former-commit-id: c0b1071fc6fbe72f1207d02fdb794dc1076eb59a

* Global<OpGraph> is only available duraing compilation

* small record_piece_size for InferNoParallelBlobDesc


Former-commit-id: 5eb1012703f8f9389ac8e2f16131bfd36411b0db

* mem_shared_hint_id

* sharable memory block

* rm useless code

* remove useless code

* bugfix: no redundant edges

* rename: MemBlockGroup => MemBlock

* put constrcutor of SharableMemBlockNode into header file

* bugfix

* rename field: MemBlock.block_id => MemBlock.mem_block_id

* replace piece_size with logical_blob_dim0

* BlobParallelConf

* BlobParallelDesc

* infer out blob model_split_axis

* int64_t => int32_t

* InferOutBlobParallelDesc

* gather out blob model split (#1624)

* InferBlobParallelDesc

* let variable op support kModelParallel

* rename lbi2blob_desc_ => lbi2no_parallel_blob_desc_

* Global<OpGraph>

* SplitLogicalInputBlobDesc

* ConcatOutputBlobDescs

* rename: BlobDataParallel => DataBlobParallel; BlobModelParallel => ModelBlobParallel; BlobGridParallel => GridBlobParallel

* OpGraph::CheckBlobDescs(...)

* exact division is unnecessary

* fix bugs

* rename InferOutBlob* => InferOutputBlob

* exact division in variable_op is unnecessary

* bug fix

* fix bugs

* fix bugs

* IsInputBlobAllowedModelSplit

* use Global<OpGraph> to InferModelSize

* add OpGraph::GetDataBalancedSplitter and OpGraph::GetModelBalancedSplitter

* fix IdentityOp::IsInputBlobAllowedModelSplit

* no implementation for pure virtual function Operator::IsInputBlobAllowedModelSplit

* refine BlobParallelDesc: replace CopyParallelConf with operator=

* refine ParallelDesc: remove unused functions

* more checks on ParallelDesc

* remove unused function Operator::MaxModelSplitNum

* bugfix: SoleOp() => op_vec().at(0)


Former-commit-id: be1f820b2927f7f79f55b7891f6575cdeb4b2053

* mem_shared_hint_id

* sharable memory block

* rm useless code

* remove useless code

* bugfix: no redundant edges

* rename: MemBlockGroup => MemBlock

* put constrcutor of SharableMemBlockNode into header file

* bugfix

* rename field: MemBlock.block_id => MemBlock.mem_block_id

* replace piece_size with logical_blob_dim0

* BlobParallelConf

* BlobParallelDesc

* infer out blob model_split_axis

* int64_t => int32_t

* InferOutBlobParallelDesc

* gather out blob model split (#1624)

* InferBlobParallelDesc

* let variable op support kModelParallel

* rename lbi2blob_desc_ => lbi2no_parallel_blob_desc_

* Global<OpGraph>

* SplitLogicalInputBlobDesc

* ConcatOutputBlobDescs

* rename: BlobDataParallel => DataBlobParallel; BlobModelParallel => ModelBlobParallel; BlobGridParallel => GridBlobParallel

* OpGraph::CheckBlobDescs(...)

* exact division is unnecessary

* fix bugs

* rename InferOutBlob* => InferOutputBlob

* exact division in variable_op is unnecessary

* bug fix

* fix bugs

* fix bugs

* IsInputBlobAllowedModelSplit

* use Global<OpGraph> to InferModelSize

* add OpGraph::GetDataBalancedSplitter and OpGraph::GetModelBalancedSplitter

* fix IdentityOp::IsInputBlobAllowedModelSplit

* no implementation for pure virtual function Operator::IsInputBlobAllowedModelSplit

* refine BlobParallelDesc: replace CopyParallelConf with operator=

* refine ParallelDesc: remove unused functions

* more checks on ParallelDesc


Former-commit-id: 2b78c6e1f37e514e39f1dc807ccce455190b00a7

* accuracy

* accuracy_task_node add fw_buf

* fw_buf=>data_tmp


Former-commit-id: ab900061ce8f1050ec6d2ee39057addcfcd44f57

* multithreads register coloring

* refine code


Former-commit-id: 3be07992331e7f62fe9d56b2f8d1357a72ef1a09

Former-commit-id: 2550030088fb6b15f1784a2bd1cfb78eeabe3b0d

Former-commit-id: 92b9ff1ae7efc90391581eafe8253347a5712b9c

* refine CHECK in AllReduce

* move ReduceConcatOpCtx definition to .cpp file


Former-commit-id: 5a50f692cb92c5a6a7074be2063cbc1ec325c1ca

* mem_shared_hint_id

* sharable memory block

* rm useless code

* remove useless code

* bugfix: no redundant edges

* rename: MemBlockGroup => MemBlock

* put constrcutor of SharableMemBlockNode into header file

* bugfix

* rename field: MemBlock.block_id => MemBlock.mem_block_id


Former-commit-id: 6a8fc14c2ba6bbe148a84458fa6119af16cbe672

Former-commit-id: 21079b5b27c7ac6fa0b8f137e2d960e2f631df9c

* share model_diff and out_diff in variable op

* bugfix: model_diff is a produced register

* register_num of model_diff is 1

* add VariableKernelConf

* no mutable

* bugfix


Former-commit-id: 2409c41f525875597e989f718434c30d91e9d874

* group by has_diff

* rm unnecessary identity


Former-commit-id: 63f2cfd61337c821e0fb6215b592231ccee584d4

Former-commit-id: 58b9b56676f69c50ea6cdc80f474a3bde8d4becb

* random shuffle

* fix

* refine

* refine

* single thread

* refine


Former-commit-id: 0dbb1f3d7265f9c55a11b07695efd092cd81a83c

* refine logic to infer reduce_concat_op's elem_cnt of out blob, still have bugs...

* add RoundUp in reduce_concat

* CHECK_LE -> CHECK_EQ

* add CHECK


Former-commit-id: 962817e2a322ba6452c9966bae87fb5da9d4a86a

Former-commit-id: dc508e494e7b9bfcd500d8f125390ab65874b792

Former-commit-id: 8ed2fd1609f3c3ee3b6cbe0a5d1f380afea8bd25

* pseudo chains of OpGraph

* ConvertPseudoChainToChain

* refine pseudo_chain

* refine register coloring algorithm

* rename op_graph log file name

* remove unused code

* EnableMemSharingInVariableOp

* no mem_sharing for out_diff & model_diff in variable_op


Former-commit-id: 224bb63a0576e7da8929ab829dfcf7c709398a97

Former-commit-id: dca95c0e8bd46e73e52fe09d0be388c881b47c82

Former-commit-id: 307abc0a39a3d6aa5ce4bf183d9dba73c882334d

Former-commit-id: 26cf1f9399a6907bbc8cb4469284b82c7d35ca0b

Former-commit-id: 2a539125119f471319820255be7d5da0e92116f5

* pseudo chains of OpGraph

* ConvertPseudoChainToChain

* refine pseudo_chain

* refine register coloring algorithm

* rename op_graph log file name

* remove unused code


Former-commit-id: 2610dff0f6cf9d8b96c06296b41e6c5a39555dbb

Former-commit-id: b8cca65712bc3bba5f4c3becb08e048a5a9d5c27

Former-commit-id: 37be744b0d2eddf91a8858d33fe961e5328a275c

Former-commit-id: 74efe34ebb508d784f2bac3440f3cf9eefcbe9f5

Former-commit-id: 440f92978c31065fab0b6de34d8a3695547e2857

Former-commit-id: 50c62ae42b54c95654f569923587f7a478d99990

Former-commit-id: 318e892f5e04f74a043d07f3a5c63a0f5a414e2c

Former-commit-id: bc7ad0ede51d5b41c5a9a4c30d2ff82d975b72cd

Former-commit-id: 87205121979d94661bb4ee72d9f1ba0f79c098cb

* profiling

* all_reduce_* option for performance optimization

* faster adam kernel

* refine dropout and transpose


Former-commit-id: a1dd7c9b36f2114ef18e0c5f6303026d91e6fe6b

* profiling

* all_reduce_* option for performance optimization

* faster adam kernel


Former-commit-id: 5885d1ff7eb09cbd97ca13c22dabe3835af528a6

* fix a mem sharing bug

* refine by review

* remove previous if condition

* refine


Former-commit-id: 028244941572194047bfa033aa2fbe7a920c598d

Former-commit-id: 61ba5b711fc218b45f84f327d33d2ee11841b8bb

* profiling

* all_reduce_* option for performance optimization


Former-commit-id: 606e964e640275e354b1280517945b0c95d09747

Former-commit-id: c50a06af6f962751cdccebf7851b0a978be6ac7d

* cudaSetDevice in actor poller threads

* ReduceConcatCompActor ; NaiveActor


Former-commit-id: cf98dd810e1b27f6a19270fc9619f52aa4cfa554