Add distributed training robust cases into fluid benchmark test
Created by: velconia
the PaddlePaddle robust feature combination includes:
| Parallel Executor | slice_var_up | sync_mode | Place | Parameters Update Method | Trainer Count | model | LearningRate Decay | Regularization | Gradient Clipping | Error Clipping |
|---|---|---|---|---|---|---|---|---|---|---|
| True | True | True | GPU | pserver | 2 | Mnist | True | L1 | True | True |
| False | False | False | CPU | nccl2 | False | L2 | False | False | ||
| None |
I construct the test-case-pairs with AllPair as following:
import os
import metacomm.combinatorics.all_pairs2
all_pairs = metacomm.combinatorics.all_pairs2.all_pairs2
parameters = [ [ "--gpus 2", "--gpus 1" ] # If use ParallelExecutor or NOT
, [ "", "--no_split_var"] # If split variables into blocks or NOT
, [ "", "--async_mode" ] # if ASGD
, [ "--device GPU", "--device CPU" ] # if train with GPU
, [ "--model mnist" ] # model
, [ "--update_method pserver", "--update_method nccl2" ] # parameters update method
, [ "--learning_rate_decay_method exponential", "--learning_rate_decay_method natural_exp" ] # learning rate decay
, [ "--weight_decay_regularizer_method L1", "--weight_decay_regularizer_method L2" ]
, [ "--gradient_clip_method Norm", "--gradient_clip_method GlobalNorm" ]
, [ "--error_clip_method Value", "" ]
]
pairwise = all_pairs(parameters)
print "PAIRWISE:"
for i, v in enumerate(pairwise):
cmd = "python fluid_benchmark.py"
for arg in v:
cmd += ' ' + arg
print "%i:\t%s" % (i, cmd)