nccl2_rdma_training.md

# Distributed Training with NCCL2 and RDMA

When doing distributed multi-GPU training, network bandwidth often becomes the
bottleneck. We introduce a way to use NCCL2 to do such training job to
achieve best performance.

## Prepare Hardware with RDMA and Multiple GPUs

I'm using two Linux servers each of them installed with 8 GPUs and
one 100Gb RDMA card.
Base environment is:

* OS: CentOS 7.4
* RDMA device: "Mellanox Technologies MT27700 Family [ConnectX-4]"
* Kernel version: `4.4.88-1.el7.elrepo.x86_64`
* Docker version: `1.12.6`
* Docker storage driver: `overlay2`
* IP addresses: 192.168.16.30,192.168.16.34

In general, the steps including:

1. Install GPU drivers
1. Install RDMA drivers
1. Install "InfiniBand Support"
1. Use docker to run tests and make sure GPUs and RDMA can work inside
   the container.

I'll omit the section "Install GPU drivers" because we can find it easily
somewhere else.

### Install RDMA drivers

For my case, I've got two machines with device
"Mellanox Technologies MT27700 Family [ConnectX-4]" installed. The OS was
"CentOS 7.4" and I updated the kernel to version 4.4 so that docker can
work with the latest overlay2 filesystem.

***NOTE: before you start, make sure you have a way to get a console
of the server other than ssh because we may need to re-configure the
network device.***

1. Go to http://www.mellanox.com/page/products_dyn?product_family=26,
   download `MLNX_OFED` software in the bottom of the page, and upload it
   onto the server.
1. Run `./mlnxofedinstall --add-kernel-support` in the software package.
1. Run `/etc/init.d/openibd restart` to make everything work, note that
   this operation may cause the network goes down if you are using this
   RDMA device as default network device and use ssh to log in the server.
1. Re-configure the network interface, for example:
   `ifconfig eth2 192.168.16.30/20 up`, then add routes if needed:
   `ip route add default via 192.168.16.1 dev eth2`.
1. Do the same thing on the other node.
1. Use `ping` to test if the two nodes have typical ICMP connection.
1. Use either `udaddy` or `ib_write_bw` to test the network connection is
   ready and have the desired bandwidth.

### Prepare Docker Image to Run RDMA Programs

1. Build a docker image using cuda base image like: `nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04` and install paddlepaddle whl
   package in it.
1. Start a docker container and mount GPU driver libs into it (you can
   skip this step if you are using nvidia-docker).
1. Mount RDMA drivers and libs into the docker image (see below section),
   also `udaddy` and `ib_write_bw` if needed.
1. Mount GPU devices and RDMA devices into the container using `--device`
   or just use privileged mode `--privileged`.
1. Start the container using host network mode: `--net=host`

### RDMA Library Files Needed

Usually, `MLNX_OFED` install latest supported libs under
`/usr/lib64/mlnx_ofed/valgrind`. Other libs also needed to run RDMA programs
is listed below. These libs must be mounted into the docker container.

* Libs under `/usr/lib64/mlnx_ofed/valgrind`
  * libibcm.so
  * libibverbs.so
  * libmlx4.so
  * libmlx5.so
  * libmlx5-rdmav2.so
  * librdmacm.so
* Other libs:
  * libnl-3.so.200
  * libnl-route-3.so.200
  * libnuma.so.1

## Start to Run the Training Job

Setting NCCL environment variables to turn NCCL switches on and off:


| Env Name | Description |
| --- | --- |
| NCCL_SOCKET_IFNAME | The RDMA device, e.g. eth2 |
| NCCL_P2P_DISABLE | Set to 1 to disable P2P transfer between GPUs |
| NCCL_IB_DISABLE | Set to 1 to disable using RDMA |
| NCCL_IB_CUDA_SUPPORT | Set to 1 to enable GPU Direct if supported |
| NCCL_DEBUG | Set debug level: VERSION, WARN, INFO |

My two servers are: `192.168.16.30,192.168.16.34`, On node 1, Run :

```bash
PADDLE_TRAINER_ID=0 PADDLE_PORT=48372 PADDLE_WORKERS=192.168.16.30,192.168.16.34 POD_IP=192.168.16.30 stdbuf -oL python vgg16.py
```

On node 2, Run:

```bash
PADDLE_TRAINER_ID=1 PADDLE_PORT=48372 PADDLE_WORKERS=192.168.16.30,192.168.16.34 POD_IP=192.168.16.34 stdbuf -oL python vgg16.py
```