README.md 2.5 KB
Newer Older
T
typhoonzero 已提交
1
# Performance for Distributed vgg16
T
typhoonzero 已提交
2 3 4

## Test Result

T
typhoonzero 已提交
5 6 7 8 9 10
### Hardware Infomation

- CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
- cpu MHz		: 2101.000
- cache size	: 20480 KB

11 12 13 14
### Blas settings

Setting environment variable: `MKL_NUM_THREADS=1`.

T
typhoonzero 已提交
15 16 17
### Single Node Single Thread

- Metrics: samples / sec
T
typhoonzero 已提交
18 19 20

| Batch Size | 32 | 64 | 128 | 256 |
| -- | -- | -- | -- | -- |
T
typhoonzero 已提交
21 22
| PaddlePaddle Fluid | 15.44 | 16.32 | 16.74 | 16.79 |
| PaddlePaddle v2 | 15.97 | 17.04 | 17.60 | 17.83 |
T
typhoonzero 已提交
23 24
| TensorFlow | - | - | - | - |

T
typhoonzero 已提交
25
### Different Batch Size
T
typhoonzero 已提交
26 27 28 29 30 31 32

- PServer Count: 10
- Trainer Count: 20
- Metrics: samples / sec

| Batch Size | 32 | 64 | 128 | 256 |
| -- | -- | -- | -- | -- |
T
typhoonzero 已提交
33 34
| PaddlePaddle Fluid | 190.20 | 222.15 | 247.40 | 258.18 |
| PaddlePaddle v2 | 170.96 | 233.71 | 256.14 | 329.23 |
T
typhoonzero 已提交
35 36 37
| TensorFlow | - | - | - | - |


T
typhoonzero 已提交
38
### Accelerate Rate
T
typhoonzero 已提交
39

T
typhoonzero 已提交
40 41 42 43
- Pserver Count: 20
- Batch Size: 128
- Metrics: samples / sec

T
typhoonzero 已提交
44
| Trainer Count | 20 | 40 | 80 | 100 |
T
typhoonzero 已提交
45
| -- | -- | -- | -- | -- |
T
typhoonzero 已提交
46 47
| PaddlePaddle Fluid | 263.29 (78.64%) | 518.80 (77.47%) | 836.26 (62.44%) | 1019.29 (60.89%) |
| PaddlePaddle v2 (need more tests) | 326.85 (92.85%) | 534.58 (75.93%) | 853.30 (60.60%) | 1041.99 (59.20%) |
T
typhoonzero 已提交
48 49
| TensorFlow | - | - | - | - |

T
typhoonzero 已提交
50
### Different Pserver Count
T
typhoonzero 已提交
51

G
gongweibao 已提交
52
- Trainer Count: 60
T
typhoonzero 已提交
53
- Batch Size: 128
T
typhoonzero 已提交
54
- Metrics: samples/ sec
T
typhoonzero 已提交
55

G
gongweibao 已提交
56
| PServer Count | 3 | 6 |10 | 20 |
T
typhoonzero 已提交
57
| -- | -- | -- | -- | -- |
G
gongweibao 已提交
58
| PaddlePaddle Fluid(should fix in next PR) | 589.1 | 592.6 | 656.4 | 655.8 |
G
gongweibao 已提交
59
| PaddlePaddle v2 | 593.4 | 791.3 | 729.7 | 821.7 |
T
typhoonzero 已提交
60 61
| TensorFlow | - | - | - | - |

G
gongweibao 已提交
62 63
*The performance gap between Fuild and v2 comes from the network interference.*

T
typhoonzero 已提交
64

T
typhoonzero 已提交
65
## Steps to Run the Performance Test
T
typhoonzero 已提交
66 67 68 69 70 71 72 73 74

1. You must re-compile PaddlePaddle and enable `-DWITH_DISTRIBUTE` to build PaddlePaddle with distributed support.
1. When the build finishes, copy the output `whl` package located under `build/python/dist` to current directory.
1. Run `docker build -t [image:tag] .` to build the docker image and run `docker push [image:tag]` to push the image to reponsitory so kubernetes can find it.
1. Run `kubectl create -f pserver.yaml && kubectl create -f trainer.yaml` to start the job on your kubernetes cluster (you must configure the `kubectl` client before this step).
1. Run `kubectl get po` to get running pods, and run `kubectl logs [podID]` to fetch the pod log of pservers and trainers.

Check the logs for the distributed training progress and analyze the performance.

T
typhoonzero 已提交
75
## Enable Verbos Logs
T
typhoonzero 已提交
76

T
typhoonzero 已提交
77
Edit `pserver.yaml` and `trainer.yaml` and add an environment variable `GLOG_v=3` and `GLOG_logtostderr=1` to see what happend in detail.