* DFS ChainActGraph

* provide interface to improver

* Add RegstAct

* rename to chain_act_graph

* add new file

* Add ForEachRegstActDuration

* use RegstActCtx

* use constructor

* use pair instead of node_str

* Construct RegstAct togather

* change ParseActEvents to std::list<std::unique_ptr<ActEvent>>

* fake_producer_outs

* use Duration4ActEvent

* add CHECK

* delete act_graph.cpp

* use node->ForEach

* add CHECK

* remove false CHECK

* optimzie chain merge with bitset

* fix task_uid_cnt_ initialization

* parallel chain merging

* refine

* extract MergeTaskNodes

* shrink the size of bitset with local task_uid

* add stream for mdUpdt task

* rm kCudaWorkTypeSize

* half impl

* remove eigen

* remove unsupported eigen

* remove blob implement

* remove BlobIf

* fix ptr bug of blob copy from

* set_regst_desc only used by regst manager

* check oom for experimental phrase

* refine code

* initial attempt of removing backward add

* workable

* refine

* add backward_activation for KernelConf

* add 3 functions to support move backward activation.

* simplify 3 functions to 1

* add backward_activation in operator

* add GetBackwardActivationType, refine GetActivationType.

* add AfterBackwardActivation in KernelIf

* SetEnumValue support in protobuf.h
set activation func in operator

* bug fix: set cur_node activation after pre_node

* check cur_node op_vec size

* set bw_node_ nullptr for add_fw_node

* pre_node is in backward area or has loss op

* AfterBackwardActivation -> PostBackwardActivation

* remove activation blob in kernel

* keep removing activation blob ...

* post backward activation for loss kernel

* Retrieve BuildAccuracyPrintStruct

* modify post backward activation

* resolve code review issues

* refine

* refine AddOneBackwardClone

* refine RemoveOneBackwardAdd

* add forward_activation for kernel proto

* add ibn blob for bw clone task node

* rm NeedOutWhenBackward

* remove

* add tuple switch;

* simplify code of add/clone kernel

* add static_assert to check error at compile time

* remove extral parameter

* rename function object

support using synthetic data for training (working around IO bottleneck for testing purpose) (#1036)

fix(actor.cpp): fix the bug that return wrong consumed ctrl Regst in AsyncSendCtrlRegstMsg() (#1031)

* add ctrl edge between copyHd and MdUpdte

* fix bug of add ctrl regst

* hack ctrl regst max regst num

* test undo

* after experiment

* use get task type instead of dynamic cast

* fix for review

* remove hack regst

* init ctrl regst min num

* fix ctrl regst num between copy and mdupdt after improver

* mdupdt actor return ctrl regst k(k = num of piece in batch) one act

* add returned regst num of ctrl regst

* CHECK invariant

* optimize the add op

* remove useless code

* feat: avoid net contention by adding ctrl edge in ReduceStruct

* refine(task_graph.h/cpp): refine AddCtrlEdgeInReduceStruct()

* fix(graph/task_graph.cpp): fix the bug of machine order

* fix(graph/task_graph.cpp): do not add ctrl edge with reduce scatter

* feat: add ReduceGlobalAddCompActor

* fix: fix the bug of reduce_global_actor/kernel

* chore: remove used vim .swp file

* fix(graph/task_graph.cpp): fix the bug of sorting copycomment when build reduce ctrl edge

* fix(graph/task_graph.h/cpp): add CtrlEdge for ReduceGather

* feat: revert add ctrl edge in reduce struct from this PR

* refactor: rename ReduceGlobalAddCompActor to InputWiseCompActor for scalability

* fix(kernel/reduce_global_add_kernel.cpp): use Memcpy other than Memset for first blob to be added

* refactor(actor/input_wise_compute_actor.*): use HashMap and counter instead of HashSet for processed regst_desc

* refactor: let ReduceGlobalAddCompActor inherit InputWiseCompActor

* feature: add ReduceGatherCompActor that inherits InputWiseCompActor

* fix(reduce_gather_kernel.cpp): add missing break

* refactor: replace regst_desc_id2bn_in_op_ with regst_desc_id2in_bn_id_ in InputWiseCompActor

* fix(reduce_global_add_kernel): remove useless class member parallel_id_

* refactor: make ReduceLocalAdd kernel support inputwise, rename ReduceGlobalAddActor to ReduceAddActor for scalibility

* naive version of accuracy module

* add top_k_ prefix to accuracy print

* remove magic number

* speedup clone kernel

* refine

* simplify

* handle with arbitrary number of out_diff

* let NormalMdUpdtTask use independent stream

* let kMix task share the same stream

* remove UseIndependentStream

* refine

* feat: avoid net contention by adding ctrl edge in ReduceStruct

* refine(task_graph.h/cpp): refine AddCtrlEdgeInReduceStruct()

* fix(graph/task_graph.cpp): fix the bug of machine order

* fix(graph/task_graph.cpp): do not add ctrl edge with reduce scatter

* feat: add ReduceGlobalAddCompActor

* fix: fix the bug of reduce_global_actor/kernel

* chore: remove used vim .swp file

* fix(graph/task_graph.cpp): fix the bug of sorting copycomment when build reduce ctrl edge

* fix(graph/task_graph.h/cpp): add CtrlEdge for ReduceGather

* revert: remove the ReduceGlobalAddCompActor from this PR

* feat: add use_ordered_allreduce_in_mdupdt in OtherConf

* refactor comm_network

* refine

* AddWorkToStream

* refine

* add CHECK

* refine

* alexnet, resnet and inception v3 generator

* move net gen apps to tools

* bugfix: fix node2component_id bug according to new depth definition

* only take account of ActNode with output

* carefully handle depth of act_node

* revert dfs topo

* carefully handle depth of act_node

* bfs topo instead of dfs topo

* fix bug

* reduce one copy

* perform rpc call in background thread pool

* comment: stream poller thread is not allowed to perform blocking action