Created by: gangliao

The biggest finding is that most of the frameworks have abandoned Parameter Server and switched to allreduce.

1. Tensorflow: https://github.com/uber/horovod 

2. ChainerMN:

3. Caffe2: Yangqing gives a talk about return to MPI at AISys, SOSP'17. Sorry, he did not share this slides.

https://github.com/facebookincubator/gloo

Their training ImageNet in one hour is actually used allreduce. https://arxiv.org/abs/1706.02677

Based on above facts, I think this algorithm is faster, at least, it should be considered inside Paddle.

NCCL2: https://developer.nvidia.com/nccl