Created by: XinyuZhou-1014

When there's a lot of addto_layers in a network, it will takes exponential parsing time. E.G. 1:

def whole_network(src_embedding):
    enc = src_embedding
    for i in range(depth):
        enc = addto_layer([enc, enc])

    pred = fc_layer(input=fc_layer(
                        input=enc,
                        size=dim_embedding,
                        act=ReluActivation()
                        ),
                    size=label_dict_len,
                    act=SoftmaxActivation())
    return pred

E.G. 2:

def whole_network(src_embedding):
    enc = src_embedding
    for i in range(depth):
        enc_res = fc_layer(input=enc, size=dim_embedding)
        enc_res = fc_layer(input=enc_res, size=dim_embedding)
        enc = addto_layer([enc, enc_res])

    pred = fc_layer(input=fc_layer(
                        input=enc,
                        size=dim_embedding,
                        act=ReluActivation()
                        ),
                    size=label_dict_len,
                    act=SoftmaxActivation())
    return pred

Both will costs a huge amount of time to parse (test by the nest_diagram tool). My parsing time:

depth: 4,   parsing time: 0.16.
depth: 8,   parsing time: 0.16.
depth: 12, parsing time: 0.33.
depth: 16, parsing time: 2.02.
depth: 20, parsing time: 32.08.
depth: 21, parsing time: 67.05.
depth: 22, parsing time: 131.48.
depth: 23, parsing time: 268.82.