Created by: kuke

Now it suffers the bad convergence when using parallel executor

Training on one GPU:

export CUDA_VISIBLE_DEVICES=4
python -u  ../../train.py --train_feature_lst data/train_feature0.lst \
                   --train_label_lst data/train_label0.lst \
                   --val_feature_lst data/train_feature0.lst \
                   --val_label_lst data/train_label0.lst \
                   --mean_var data/global_mean_var \
                   --checkpoints checkpoints \
                   --frame_dim 80  \
                   --class_num 3040  \
                   --print_per_batches 5 \
                   --infer_models '' \
                   --batch_size 16 \
                   --learning_rate 6.4e-5 \

-----------  Configuration Arguments -----------
batch_size: 16
checkpoints: checkpoints
class_num: 3040
device: GPU
frame_dim: 80
hidden_dim: 1024
infer_models:
init_model_path: None
learning_rate: 6.4e-05
mean_var: data/global_mean_var
minimum_batch_size: 1
parallel: False
pass_num: 100
print_per_batches: 5
proj_dim: 512
stacked_num: 5
train_feature_lst: data/train_feature0.lst
train_label_lst: data/train_label0.lst
val_feature_lst: data/train_feature0.lst
val_label_lst: data/train_label0.lst
------------------------------------------------
.....
Batch 5, train cost: 6.923083, train acc: 0.145724
....
Batch 10, train cost: 6.637660, train acc: 0.159152
....
Batch 15, train cost: 6.479028, train acc: 0.163057
....
Batch 20, train cost: 6.434110, train acc: 0.156311
....
Batch 25, train cost: 6.252566, train acc: 0.173141
....
Batch 30, train cost: 6.305423, train acc: 0.163910
....
Batch 35, train cost: 6.233900, train acc: 0.174586
....
Batch 40, train cost: 6.188899, train acc: 0.161546
....
Batch 45, train cost: 6.093045, train acc: 0.174171
....
Batch 50, train cost: 6.119380, train acc: 0.176405
....
Batch 55, train cost: 5.921489, train acc: 0.192880
....
Batch 60, train cost: 5.985956, train acc: 0.161175
....
Batch 65, train cost: 5.804343, train acc: 0.193425
....
Batch 70, train cost: 5.828652, train acc: 0.173742
....
Batch 75, train cost: 5.856163, train acc: 0.155712
....
Batch 80, train cost: 5.582592, train acc: 0.183691
....
Batch 85, train cost: 5.588781, train acc: 0.214424
....
Batch 90, train cost: 5.346203, train acc: 0.246694
....
Batch 95, train cost: 5.503672, train acc: 0.204993
....
Batch 100, train cost: 5.554629, train acc: 0.191487
....
Batch 105, train cost: 5.469702, train acc: 0.192436
....
Batch 110, train cost: 5.289505, train acc: 0.220166
....
Batch 115, train cost: 5.165346, train acc: 0.237995
....
Batch 120, train cost: 5.264880, train acc: 0.223674
....
Batch 125, train cost: 5.127150, train acc: 0.221855

Pass 0, time consumed: 345.763684 s, val cost: 5.111879, val acc: 0.230703