Created by: panyx0718

se-resnext on 4 device titan-x goes from ~1.25 to ~0.98

Some changes in the PR are due to pre-commit cpplint

The main issue is that nccl_all_reduce stream cannot overlap with computation streams. some slow nccl_all_reduce block other gpus' streams