@@ -257,32 +204,12 @@ First run on a single GPU with batch size 64.
train(1, 256, 0.3)
```
```{.json .output n=8}
[
{
"name": "stdout",
"output_type": "stream",
"text": "Running on [gpu(0)]\nEpoch 0, training time = 2.2 sec\n validation accuracy = 0.1001\nEpoch 1, training time = 1.8 sec\n validation accuracy = 0.6264\nEpoch 2, training time = 1.8 sec\n validation accuracy = 0.7881\nEpoch 3, training time = 1.8 sec\n validation accuracy = 0.7849\nEpoch 4, training time = 1.8 sec\n validation accuracy = 0.8259\n"
}
]
```
Running on multiple GPUs, we often want to increase the batch size so that each GPU still gets a large enough batch size for good computation performance. A larger batch size sometimes slows down the convergence, we often want to increases the learning rate as well.
```{.python .input n=9}
train(2, 512, 0.6)
```
```{.json .output n=9}
[
{
"name": "stdout",
"output_type": "stream",
"text": "Running on [gpu(0), gpu(1)]\nEpoch 0, training time = 1.3 sec\n validation accuracy = 0.0995\nEpoch 1, training time = 1.1 sec\n validation accuracy = 0.1009\nEpoch 2, training time = 1.1 sec\n validation accuracy = 0.6300\nEpoch 3, training time = 1.1 sec\n validation accuracy = 0.7381\nEpoch 4, training time = 1.1 sec\n validation accuracy = 0.7972\n"
}
]
```
## Conclusion
We have shown how to implement data parallelism on a deep neural network from scratch. Thanks to the auto-parallelism, we only need to write serial codes while the engine is able to parallelize them on multiple GPUs.