Created by: Yancey1989
model parallelism for FC layer.
-
the principle of the distributed mode of FC layer:
-
the scalability of the distributed FC layer.
| GPUs | fc_w.dims | GPU memory(peak) | ratio |
|---|---|---|---|
| 1 | [2^5, 2^26] | 21103MiB | 1 |
| 2 | [2^5, 2^26] | 12945MiB | 61.34% |
| 4 | [2^5, 2^26] | 9091MiB | 43.07% |
| 8 | [2^5, 2^26] | 6919MiB | 32.78% |
| 8 | [2^5, 2^26 * 3] | 18705MiB | - |