Former-commit-id: ccc3b389

Former-commit-id: 28a6fc98

Former-commit-id: 40c299bc

* sketch of merge reduce project

* add reduce_concat, reduce_split in logical graph (#1160)

* add reduce_concat, reduce_split in logical graph

* init ReduceTaskNodes in CollectReduceTaskNodes

* add CompTaskNode for ReduceConcat & ReduceSplit

* set ReduceConcat/Split color index

* copy blob desc from ReduceConcat in to ReduceSplit out

* refine CollectReduceTaskNodes

* SetMemSharing for ReduceConcat, ReduceSplit regst

* complete ReduceConcat & ReduceSplit op

* fill ReduceConcat & ReduceSplit kernel

* simplify ReduceConcatCompActor

* make ReduceScatter & ReduceSplit as input-wise actor

* reduce_scatter & reduce_split use is_inplace

* use ByteSizeOfBlobBody for reduce related packed blob

* Fix dev merge reduce (#1168)

* check concat and split occur simultaneously

* fix ReduceScatter & ReduceSplit as Inputwise actor

* ReduceConcat & ReduceSplit works

* fix single gpu issue

* Refactor reduce (#1170)

* backup, not complete yet

* remove reduce_id

* rm useless comment

* add reduce_graph (#1169)

* add reduce_graph

* fix iter

* add IsLogicalNodeMergeable and fix bug

* remove needless constructor calls

* node VisualStr may conflict, using node_id_str instead

* reduce group works (#1171)

* refine

* sort nodes in topo (#1172)

* add reduce_group_size in job_conf, fix 121 config of ReduceSplit and MdUpdt

* resolve code review issues (variable names)

* refine variable names

* Dev merge reduce rename reduce group (#1174)

* ReduceGraph=>ChainLogicalGraph

* rename Group=>Chain

* reformat

* use pointer instead of reference for mutable argument

* format change

* worker node only pull sub_plan (#1176)

* log compile time

* use c++11 member initialization syntax

* FixPackedBlobDescOfProducedRegst for ReduceSplit

* Dev merge reduce refine chain logical graph (#1177)

* remove IsMerageable

* split TryMergeOneChain and rename to TryMergeTwoChains

* reformat

* resolve review issues


Former-commit-id: 3aa79c70

Former-commit-id: dc6fbefc

Former-commit-id: a8b7dedb

* refactor EraseEmptyRegst (no dependence on weak_ptr)

* weak_ptr -> shared_ptr

* refine


Former-commit-id: e585bba0

* gdb copy blob

* make BnInOp2Blob called by gdb easily


Former-commit-id: ee70abf7

Former-commit-id: 55b46427

* use resize()

* use .size to calc bitset_num


Former-commit-id: 400e277e

Former-commit-id: 3612c581

* if UseRelayPlacement

* judge if there is only one gpu parallel_conf

* refine

* fix naive error


Former-commit-id: 3ea8ae21

* use Special judgment in InitNodeProducedRegstAct

* abandon kMdUpdtArea ActEvents


Former-commit-id: e853cef6

* clear act_event_logger act_event_bin_filename

* cluster_thrd_ids_key

* simplify ofrecord_decoder multi-thread

* let decoder use AllocateCpuThrdIdEvenly

* let ofrecord_decoder use local thread pool


Former-commit-id: a4860e5b

Former-commit-id: 9f01aa33

Former-commit-id: cae14ff3

* caching the cudnn conv algorithm to eliminate duplicate calculation

* refine cudnn conv algo ctx cache


Former-commit-id: ccb7f43b

Former-commit-id: b611c93d

* rm collect kMdUpdtArea Ancestor

* refine AddOrderingCtrlEdgeInSameChain

* mv ChainGraph to SetChainIdAndOrderInGraphForEachNode

* rm task_node ancestors

* rm emplace()


Former-commit-id: 691704b7

* fix typo

* rm useless IsThisMachineMaster

* refine the var name of naive_plan, mem_shared_plan, improved_plan

* refactor PushPlan and PullPlan

* let master node broadcast subplans instead the whole plan

* remove useless code

* rm useless code

* use total_mbn_name_key


Former-commit-id: b21c190b

* backpropogate model_diff only if is trainable

* bugfix: consume bw task node only if trainable

* bugfix: connect md_updt and bw_node when bw_node is not null

* bugfix: md_updt enter HandlerNormal only if there is model to train

* set all op trainable = false when predicting


Former-commit-id: be213666

* rm MdUpdt chain merge

* use area_id == kMdUpdtArea

* rm judgement

* refine IsSubset


Former-commit-id: f9fe1ee0

Former-commit-id: 3654d164

* refine act_id order condition

* strict act id check (excluding model regst)

* add TODO: figure out the ActNumForEachOutput of model regsts to MdSave area


Former-commit-id: 5be84c50

* remove blob_inited check

* fix inplace feature of reduce add actor and kernel

* rm useless code

* add EnableInplace, support CPU allreduce


Former-commit-id: 40a9b9a5

Former-commit-id: 2e8f33fc

Former-commit-id: 980b0dc3

* refine task_graph construtor

* use const qualifier

* add ordered_chain_nodes_


Former-commit-id: 945139d9

Former-commit-id: e5b4d790

* enable in_diff blobs of bw_add share mem

* use default operator= for BlobDesc

* blob's mem_shared_id -> blob_mem_id

* minor refine

* move mem_case util to memory_allocator.h

* refine blob mem sharing

* refine compute packed blob

* add operator= for blob_desc


Former-commit-id: 839d1b7d

* BldSubGrpBy 121/boxing use same buf task

* remove 121 boxing regst from loss/decode compute task node

* remove B121 regst in fw/bw/compute task node, remove lbi_121/boxing in logical node

* remove b121

* fix bug in bw task node


Former-commit-id: ce5aa7fb

* rm TryMerge

* rm extra loop in CollectAncestorsForEachNode


Former-commit-id: 8616e0d4

Former-commit-id: 10e08d66

* add warmup

* add constant warmup

* rename cur_batch_num

* make warmup and lr decay not work together

* WarmupOnset -> TriggerWarmup


Former-commit-id: 02629a55

Former-commit-id: e74bb463

* refine and move CalcBaseII to ChainActGraph

* use MainByteSize4OneRegst()

* use ReverseDfs

* use shared and unshared

* use regst_act_group

* use SameFakeProducerOutsRegstActGroup

* avoid use HashMap<std::list<T*>>

* use std::set instead of HashSet

* change variant name

* minor refine of variable name


Former-commit-id: 953e1aec

Former-commit-id: ce707908

Former-commit-id: 4d573e18

Former-commit-id: f8561470

* feat(reduce_scatter_kernel): reduce scatter kernel is as before for cpu, and do nothing for gpu

* feat(reduce_local_add_kernel): update reduce_local_add kernel for in-place on GPU

* feat: update reduce_local_add actor to support inplace kernel

* feat: add support for inplace in reduce_global_add/gather

* fix: use ibn.substr(3) to specify inplace in_blob other than in_bn_id


Former-commit-id: e15fea94