README.md

### 使用方法

假设数据文件为test.data，配置文件为inference.conf

单进程client
```
cat test.data | python test_client.py inference.conf > result
```
多进程client，若进程数为4
```
python test_client_multithread.py inference.conf test.data 4 > result
```
batch clienit，若batch size为4
```
cat test.data | python test_client_batch.py inference.conf 4 > result
```

### Benchmark

设备 ：Intel(R) Xeon(R)  Gold 6271 CPU @ 2.60GHz * 48

模型 ：IMDB-CNN

测试中，client共发送2500条测试样本，图中数据为单个线程的耗时，时间单位为秒

server thread num ：4

| client  thread num | prepro | client infer | op0    | op1   | op2    | postpro | total |
| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- |
| 1                  | 0.99   | 27.39        | 0.085  | 19.92 | 0.046  | 0.032   | 29.84 |
| 4                  | 0.22   | 7.66         | 0.021  | 4.93  | 0.011  | 0.0082  | 8.28  |
| 8                  | 0.1    | 6.66         | 0.01   | 2.42  | 0.0038 | 0.0046  | 6.95  |
| 12                 | 0.074  | 6.87         | 0.0069 | 1.61  | 0.0059 | 0.0032  | 7.07  |
| 16                 | 0.056  | 7.01         | 0.0053 | 1.23  | 0.0029 | 0.0026  | 7.17  |
| 20                 | 0.045  | 7.02         | 0.0042 | 0.97  | 0.0023 | 0.002   | 7.15  |
| 24                 | 0.039  | 7.012        | 0.0034 | 0.8   | 0.0019 | 0.0016  | 7.12  |

server thread num ： 8

| client  thread num | prepro | client infer | op0    | op1   | op2    | postpro | total |
| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- |
| 1                  | 1.02   | 28.9         | 0.096  | 20.64 | 0.047  | 0.036   | 31.51 |
| 4                  | 0.22   | 7.83         | 0.021  | 5.08  | 0.012  | 0.01    | 8.45  |
| 8                  | 0.11   | 4.44         | 0.01   | 2.5   | 0.0059 | 0.0051  | 4.73  |
| 12                 | 0.074  | 4.11         | 0.0069 | 1.65  | 0.0039 | 0.0029  | 4.31  |
| 16                 | 0.057  | 4.2          | 0.0052 | 1.24  | 0.0029 | 0.0024  | 4.35  |
| 20                 | 0.046  | 4.05         | 0.0043 | 1.01  | 0.0024 | 0.0021  | 4.18  |
| 24                 | 0.038  | 4.02         | 0.0034 | 0.81  | 0.0019 | 0.0015  | 4.13  |

server thread num ： 12

| client  thread num | prepro | client infer | op0    | op1   | op2    | postpro | total |
| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- |
| 1                  | 1.02   | 29.47        | 0.098  | 20.95 | 0.048  | 0.038   | 31.96 |
| 4                  | 0.21   | 7.36         | 0.022  | 5.01  | 0.011  | 0.0081  | 7.95  |
| 8                  | 0.11   | 4.52         | 0.011  | 2.58  | 0.0061 | 0.0051  | 4.83  |
| 12                 | 0.072  | 3.25         | 0.0076 | 1.72  | 0.0042 | 0.0038  | 3.45  |
| 16                 | 0.059  | 3.93         | 0.0055 | 1.26  | 0.0029 | 0.0023  | 4.1   |
| 20                 | 0.047  | 3.79         | 0.0044 | 1.01  | 0.0024 | 0.0021  | 3.92  |
| 24                 | 0.041  | 3.76         | 0.0036 | 0.83  | 0.0019 | 0.0017  | 3.87  |

server thread num ： 16

| client  thread num | prepro | client infer | op0    | op1   | op2    | postpro | total |
| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- |
| 1                  | 1.09   | 28.79        | 0.094  | 20.59 | 0.047  | 0.034   | 31.41 |
| 4                  | 0.22   | 7.41         | 0.023  | 5.01  | 0.011  | 0.0098  | 8.01  |
| 8                  | 0.11   | 4.7          | 0.012  | 2.61  | 0.0062 | 0.0049  | 5.01  |
| 12                 | 0.081  | 4.69         | 0.0078 | 1.72  | 0.0042 | 0.0035  | 4.91  |
| 16                 | 0.058  | 3.46         | 0.0061 | 1.32  | 0.0033 | 0.003   | 3.63  |
| 20                 | 0.049  | 3.77         | 0.0047 | 1.03  | 0.0025 | 0.0022  | 3.91  |
| 24                 | 0.041  | 3.86         | 0.0039 | 0.85  | 0.002  | 0.0017  | 3.98  |

server thread num ： 20

| client  thread num | prepro | client infer | op0    | op1   | op2    | postpro | total |
| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- |
| 1                  | 1.03   | 28.42        | 0.085  | 20.47 | 0.046  | 0.037   | 30.98 |
| 4                  | 0.22   | 7.94         | 0.022  | 5.33  | 0.012  | 0.011   | 8.53  |
| 8                  | 0.11   | 4.54         | 0.01   | 2.58  | 0.006  | 0.0046  | 4.84  |
| 12                 | 0.079  | 4.54         | 0.0076 | 1.78  | 0.0042 | 0.0039  | 4.76  |
| 16                 | 0.059  | 3.41         | 0.0057 | 1.33  | 0.0032 | 0.0027  | 3.58  |
| 20                 | 0.051  | 4.33         | 0.0047 | 1.06  | 0.0025 | 0.0023  | 4.48  |
| 24                 | 0.043  | 4.51         | 0.004  | 0.88  | 0.0021 | 0.0018  | 4.63  |

server thread num ：24

| client  thread num | prepro | client infer | op0    | op1  | op2    | postpro | total |
| ------------------ | ------ | ------------ | ------ | ---- | ------ | ------- | ----- |
| 1                  | 0.93   | 29.28        | 0.099  | 20.5 | 0.048  | 0.028   | 31.61 |
| 4                  | 0.22   | 7.72         | 0.023  | 4.98 | 0.011  | 0.0095  | 8.33  |
| 8                  | 0.11   | 4.77         | 0.012  | 2.65 | 0.0062 | 0.0049  | 5.09  |
| 12                 | 0.081  | 4.22         | 0.0078 | 1.77 | 0.0042 | 0.0033  | 4.44  |
| 16                 | 0.062  | 4.21         | 0.0061 | 1.34 | 0.0032 | 0.0026  | 4.39  |
| 20                 | 0.5    | 3.58         | 0.005  | 1.07 | 0.0026 | 0.0023  | 3.72  |
| 24                 | 0.043  | 4.27         | 0.0042 | 0.89 | 0.0022 | 0.0018  | 4.4   |