README.md 13.1 KB
Newer Older
D
Dong Daxiang 已提交
1 2
<p align="center">
    <br>
D
Dong Daxiang 已提交
3
<img src='https://paddle-serving.bj.bcebos.com/imdb-demo%2FLogoMakr-3Bd2NM-300dpi.png' width = "600" height = "130">
D
Dong Daxiang 已提交
4 5
    <br>
<p>
6

D
Dong Daxiang 已提交
7 8
<p align="center">
    <br>
B
barrierye 已提交
9 10 11
    <a href="https://travis-ci.com/PaddlePaddle/Serving">
        <img alt="Build Status" src="https://img.shields.io/travis/com/PaddlePaddle/Serving/develop">
    </a>
D
Dong Daxiang 已提交
12 13 14 15
    <img alt="Release" src="https://img.shields.io/badge/Release-0.0.3-yellowgreen">
    <img alt="Issues" src="https://img.shields.io/github/issues/PaddlePaddle/Serving">
    <img alt="License" src="https://img.shields.io/github/license/PaddlePaddle/Serving">
    <img alt="Slack" src="https://img.shields.io/badge/Join-Slack-green">
D
Dong Daxiang 已提交
16 17
    <br>
<p>
D
Dong Daxiang 已提交
18

D
Dong Daxiang 已提交
19
<h2 align="center">Motivation</h2>
D
Dong Daxiang 已提交
20

J
Jiawei Wang 已提交
21
We consider deploying deep learning inference service online to be a user-facing application in the future. **The goal of this project**: When you have trained a deep neural net with [Paddle](https://github.com/PaddlePaddle/Paddle), you are also capable to deploy the model online easily. A demo of Paddle Serving is as follows:
D
Dong Daxiang 已提交
22
<p align="center">
D
Dong Daxiang 已提交
23
    <img src="doc/demo.gif" width="700">
D
Dong Daxiang 已提交
24
</p>
D
Dong Daxiang 已提交
25

D
Dong Daxiang 已提交
26
<h2 align="center">Some Key Features</h2>
D
Dong Daxiang 已提交
27

J
Jiawei Wang 已提交
28
- Integrate with Paddle training pipeline seamlessly, most paddle models can be deployed **with one line command**.
D
Dong Daxiang 已提交
29
- **Industrial serving features** supported, such as models management, online loading, online A/B testing etc.
J
Jiawei Wang 已提交
30 31 32 33
- **Distributed Key-Value indexing** supported which is especially useful for large scale sparse features as model inputs.
- **Highly concurrent and efficient communication** between clients and servers supported.
- **Multiple programming languages** supported on client side, such as Golang, C++ and python.
- **Extensible framework design** which can support model serving beyond Paddle.
D
Dong Daxiang 已提交
34

D
Dong Daxiang 已提交
35
<h2 align="center">Installation</h2>
D
Dong Daxiang 已提交
36

D
Dong Daxiang 已提交
37
We highly recommend you to run Paddle Serving in Docker, please visit [Run in Docker](https://github.com/PaddlePaddle/Serving/blob/develop/doc/RUN_IN_DOCKER.md)
M
MRXLT 已提交
38 39 40 41 42 43 44 45 46 47 48 49
```
# Run CPU Docker
docker pull hub.baidubce.com/paddlepaddle/serving:0.2.0
docker run -p 9292:9292 --name test -dit hub.baidubce.com/paddlepaddle/serving:0.2.0
docker exec -it test bash
```
```
# Run GPU Docker
nvidia-docker pull hub.baidubce.com/paddlepaddle/serving:0.2.0-gpu
nvidia-docker run -p 9292:9292 --name test -dit hub.baidubce.com/paddlepaddle/serving:0.2.0-gpu
nvidia-docker exec -it test bash
```
D
Dong Daxiang 已提交
50

D
Dong Daxiang 已提交
51
```shell
M
MRXLT 已提交
52 53 54
pip install paddle-serving-client 
pip install paddle-serving-server # CPU
pip install paddle-serving-server-gpu # GPU
D
Dong Daxiang 已提交
55 56
```

M
MRXLT 已提交
57
You may need to use a domestic mirror source (in China, you can use the Tsinghua mirror source, add `-i https://pypi.tuna.tsinghua.edu.cn/simple` to pip command) to speed up the download.
B
barrierye 已提交
58

M
MRXLT 已提交
59
Client package support Centos 7 and Ubuntu 18, or you can use HTTP service without install client.
60

D
Dong Daxiang 已提交
61
<h2 align="center">Quick Start Example</h2>
D
Dong Daxiang 已提交
62

D
Dong Daxiang 已提交
63
### Boston House Price Prediction model
D
Dong Daxiang 已提交
64
``` shell
D
Dong Daxiang 已提交
65
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
D
Dong Daxiang 已提交
66
tar -xzf uci_housing.tar.gz
D
Dong Daxiang 已提交
67
```
D
Dong Daxiang 已提交
68

D
Dong Daxiang 已提交
69 70
Paddle Serving provides HTTP and RPC based service for users to access

D
Dong Daxiang 已提交
71
### HTTP service
D
Dong Daxiang 已提交
72

J
Jiawei Wang 已提交
73
Paddle Serving provides a built-in python module called `paddle_serving_server.serve` that can start a RPC service or a http service with one-line command. If we specify the argument `--name uci`, it means that we will have a HTTP service with a url of `$IP:$PORT/uci/prediction`
D
Dong Daxiang 已提交
74
``` shell
D
Dong Daxiang 已提交
75
python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292 --name uci
D
Dong Daxiang 已提交
76
```
D
Dong Daxiang 已提交
77 78
<center>

D
Dong Daxiang 已提交
79 80
| Argument | Type | Default | Description |
|--------------|------|-----------|--------------------------------|
D
Dong Daxiang 已提交
81
| `thread` | int | `4` | Concurrency of current service |
D
Dong Daxiang 已提交
82
| `port` | int | `9292` | Exposed port of current service to users|
D
Dong Daxiang 已提交
83 84
| `name` | str | `""` | Service name, can be used to generate HTTP request url |
| `model` | str | `""` | Path of paddle model directory to be served |
M
MRXLT 已提交
85
| `mem_optim` | bool | `False` | Enable memory optimization |
D
Dong Daxiang 已提交
86

D
Dong Daxiang 已提交
87
Here, we use `curl` to send a HTTP POST request to the service we just started. Users can use any python library to send HTTP POST as well, e.g, [requests](https://requests.readthedocs.io/en/master/).
D
Dong Daxiang 已提交
88 89
</center>

D
Dong Daxiang 已提交
90 91 92
``` shell
curl -H "Content-Type:application/json" -X POST -d '{"x": [0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332], "fetch":["price"]}' http://127.0.0.1:9292/uci/prediction
```
D
Dong Daxiang 已提交
93

D
Dong Daxiang 已提交
94
### RPC service
D
Dong Daxiang 已提交
95

J
Jiawei Wang 已提交
96
A user can also start a RPC service with `paddle_serving_server.serve`. RPC service is usually faster than HTTP service, although a user needs to do some coding based on Paddle Serving's python client API. Note that we do not specify `--name` here. 
D
Dong Daxiang 已提交
97 98 99
``` shell
python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292
```
D
Dong Daxiang 已提交
100

D
Dong Daxiang 已提交
101
``` python
D
Dong Daxiang 已提交
102
# A user can visit rpc service through paddle_serving_client API
D
Dong Daxiang 已提交
103 104 105
from paddle_serving_client import Client

client = Client()
D
Dong Daxiang 已提交
106
client.load_client_config("uci_housing_client/serving_client_conf.prototxt")
D
Dong Daxiang 已提交
107
client.connect(["127.0.0.1:9292"])
D
Dong Daxiang 已提交
108
data = [0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727,
D
Dong Daxiang 已提交
109
        -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332]
D
Dong Daxiang 已提交
110
fetch_map = client.predict(feed={"x": data}, fetch=["price"])
D
Dong Daxiang 已提交
111
print(fetch_map)
D
Dong Daxiang 已提交
112 113

```
D
Dong Daxiang 已提交
114
Here, `client.predict` function has two arguments. `feed` is a `python dict` with model input variable alias name and values. `fetch` assigns the prediction variables to be returned from servers. In the example, the name of `"x"` and `"price"` are assigned when the servable model is saved during training.
D
Dong Daxiang 已提交
115

D
Dong Daxiang 已提交
116 117
<h2 align="center"> Pre-built services with Paddle Serving</h2>

D
Dong Daxiang 已提交
118 119
<h3 align="center">Chinese Word Segmentation</h4>

D
Dong Daxiang 已提交
120
- **Description**: 
D
Dong Daxiang 已提交
121
``` shell
D
Dong Daxiang 已提交
122
Chinese word segmentation HTTP service that can be deployed with one line command.
D
Dong Daxiang 已提交
123
```
D
Dong Daxiang 已提交
124

D
Dong Daxiang 已提交
125
- **Download Servable Package**: 
D
Dong Daxiang 已提交
126 127 128
``` shell
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/lac/lac_model_jieba_web.tar.gz
```
D
Dong Daxiang 已提交
129
- **Host web service**: 
D
Dong Daxiang 已提交
130 131 132 133
``` shell
tar -xzf lac_model_jieba_web.tar.gz
python lac_web_service.py jieba_server_model/ lac_workdir 9292
```
D
Dong Daxiang 已提交
134
- **Request sample**: 
D
Dong Daxiang 已提交
135
``` shell
D
Dong Daxiang 已提交
136
curl -H "Content-Type:application/json" -X POST -d '{"words": "我爱北京天安门", "fetch":["word_seg"]}' http://127.0.0.1:9292/lac/prediction
D
Dong Daxiang 已提交
137
```
D
Dong Daxiang 已提交
138
- **Request result**: 
D
Dong Daxiang 已提交
139 140 141 142
``` shell
{"word_seg":"我|爱|北京|天安门"}
```

D
Dong Daxiang 已提交
143
<h3 align="center">Image Classification</h4>
D
Dong Daxiang 已提交
144

D
Dong Daxiang 已提交
145
- **Description**: 
D
Dong Daxiang 已提交
146
``` shell
D
Dong Daxiang 已提交
147
Image classification trained with Imagenet dataset. A label and corresponding probability will be returned.
M
MRXLT 已提交
148
Note: This demo needs paddle-serving-server-gpu. 
D
Dong Daxiang 已提交
149
```
D
Dong Daxiang 已提交
150

D
Dong Daxiang 已提交
151
- **Download Servable Package**: 
D
Dong Daxiang 已提交
152 153 154 155 156 157 158 159 160
``` shell
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/imagenet_demo.tar.gz
```
- **Host web service**: 
``` shell
tar -xzf imagenet_demo.tar.gz
python image_classification_service_demo.py resnet50_serving_model
```
- **Request sample**: 
D
Dong Daxiang 已提交
161 162 163 164 165 166

<p align="center">
    <br>
<img src='https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg' width = "200" height = "200">
    <br>
<p>
167

D
Dong Daxiang 已提交
168 169 170 171 172 173 174 175
``` shell
curl -H "Content-Type:application/json" -X POST -d '{"url": "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg", "fetch": ["score"]}' http://127.0.0.1:9292/image/prediction
```
- **Request result**: 
``` shell
{"label":"daisy","prob":0.9341403245925903}
```

J
Jiawei Wang 已提交
176
<h3 align="center">More Demos</h3>
D
Dong Daxiang 已提交
177

M
MRXLT 已提交
178 179 180 181 182 183
| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | Bert-Base-Baike                                              |
| URL                | [https://paddle-serving.bj.bcebos.com/bert_example/bert_seq128.tar.gz](https://paddle-serving.bj.bcebos.com/bert_example%2Fbert_seq128.tar.gz) |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/bert |
| Description        | Get semantic representation from a Chinese Sentence          |
D
Dong Daxiang 已提交
184

D
Dong Daxiang 已提交
185 186


M
MRXLT 已提交
187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243
| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | Resnet50-Imagenet                                            |
| URL                | [https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet50_vd.tar.gz](https://paddle-serving.bj.bcebos.com/imagenet-example%2FResNet50_vd.tar.gz) |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imagenet |
| Description        | Get image semantic representation from an image              |



| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | Resnet101-Imagenet                                           |
| URL                | https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet101_vd.tar.gz |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imagenet |
| Description        | Get image semantic representation from an image              |



| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | CNN-IMDB                                                     |
| URL                | https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb |
| Description        | Get category probability from an English Sentence            |



| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | LSTM-IMDB                                                    |
| URL                | https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb |
| Description        | Get category probability from an English Sentence            |



| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | BOW-IMDB                                                     |
| URL                | https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb |
| Description        | Get category probability from an English Sentence            |



| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | Jieba-LAC                                                    |
| URL                | https://paddle-serving.bj.bcebos.com/lac/lac_model.tar.gz    |
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/lac |
| Description        | Get word segmentation from a Chinese Sentence                |



| Key                | Value                                                        |
| :----------------- | :----------------------------------------------------------- |
| Model Name         | DNN-CTR                                                      |
M
MRXLT 已提交
244
| URL                | https://paddle-serving.bj.bcebos.com/criteo_ctr_example/criteo_ctr_demo_model.tar.gz                            |
M
MRXLT 已提交
245 246 247 248
| Client/Server Code | https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/criteo_ctr |
| Description        | Get click probability from a feature vector of item          |


D
Dong Daxiang 已提交
249
<h2 align="center">Document</h2>
D
Dong Daxiang 已提交
250

D
Dong Daxiang 已提交
251
### New to Paddle Serving
D
Dong Daxiang 已提交
252
- [How to save a servable model?](doc/SAVE.md)
J
Jiawei Wang 已提交
253
- [An End-to-end tutorial from training to inference service deployment](doc/TRAIN_TO_SERVICE.md)
J
Jiawei Wang 已提交
254
- [Write Bert-as-Service in 10 minutes](doc/BERT_10_MINS.md)
D
Dong Daxiang 已提交
255

D
Dong Daxiang 已提交
256
### Developers
D
Dong Daxiang 已提交
257
- [How to config Serving native operators on server side?](doc/SERVER_DAG.md)
J
Jiawei Wang 已提交
258
- [How to develop a new Serving operator?](doc/NEW_OPERATOR.md)
B
barrierye 已提交
259
- [How to develop a new Web Service?](doc/NEW_WEB_SERVICE.md)
D
Dong Daxiang 已提交
260
- [Golang client](doc/IMDB_GO_CLIENT.md)
J
Jiawei Wang 已提交
261
- [Compile from source code](doc/COMPILE.md)
D
Dong Daxiang 已提交
262

D
Dong Daxiang 已提交
263
### About Efficiency
M
MRXLT 已提交
264
- [How to profile Paddle Serving latency?](python/examples/util)
M
MRXLT 已提交
265 266
- [How to optimize performance?(Chinese)](doc/MULTI_SERVICE_ON_ONE_GPU_CN.md)
- [Deploy multi-services on one GPU(Chinese)](doc/PERFORMANCE_OPTIM_CN.md)
J
Jiawei Wang 已提交
267 268
- [CPU Benchmarks(Chinese)](doc/BENCHMARKING.md)
- [GPU Benchmarks(Chinese)](doc/GPU_BENCHMARKING.md)
D
Dong Daxiang 已提交
269

D
Dong Daxiang 已提交
270
### FAQ
W
wangjiawei04 已提交
271
- [FAQ(Chinese)](doc/deprecated/FAQ.md)
D
Dong Daxiang 已提交
272

D
Dong Daxiang 已提交
273

D
Dong Daxiang 已提交
274
### Design
J
Jiawei Wang 已提交
275
- [Design Doc](doc/DESIGN_DOC.md)
D
Dong Daxiang 已提交
276

D
Dong Daxiang 已提交
277 278 279
<h2 align="center">Community</h2>

### Slack
D
Dong Daxiang 已提交
280

D
Dong Daxiang 已提交
281 282
To connect with other users and contributors, welcome to join our [Slack channel](https://paddleserving.slack.com/archives/CUBPKHKMJ)

D
Dong Daxiang 已提交
283
### Contribution
D
Dong Daxiang 已提交
284

D
Dong Daxiang 已提交
285
If you want to contribute code to Paddle Serving, please reference [Contribution Guidelines](doc/CONTRIBUTE.md)
D
Dong Daxiang 已提交
286 287

### Feedback
D
Dong Daxiang 已提交
288

D
Dong Daxiang 已提交
289 290
For any feedback or to report a bug, please propose a [GitHub Issue](https://github.com/PaddlePaddle/Serving/issues).

D
Dong Daxiang 已提交
291 292
### License

D
Dong Daxiang 已提交
293
[Apache 2.0 License](https://github.com/PaddlePaddle/Serving/blob/develop/LICENSE)