README.md 7.6 KB
Newer Older
D
Dong Daxiang 已提交
1 2
<p align="center">
    <br>
D
Dong Daxiang 已提交
3
<img src='https://paddle-serving.bj.bcebos.com/imdb-demo%2FLogoMakr-3Bd2NM-300dpi.png' width = "600" height = "130">
D
Dong Daxiang 已提交
4 5 6 7 8
    <br>
<p>
    
<p align="center">
    <br>
B
barrierye 已提交
9 10 11
    <a href="https://travis-ci.com/PaddlePaddle/Serving">
        <img alt="Build Status" src="https://img.shields.io/travis/com/PaddlePaddle/Serving/develop">
    </a>
D
Dong Daxiang 已提交
12 13 14 15
    <img alt="Release" src="https://img.shields.io/badge/Release-0.0.3-yellowgreen">
    <img alt="Issues" src="https://img.shields.io/github/issues/PaddlePaddle/Serving">
    <img alt="License" src="https://img.shields.io/github/license/PaddlePaddle/Serving">
    <img alt="Slack" src="https://img.shields.io/badge/Join-Slack-green">
D
Dong Daxiang 已提交
16 17
    <br>
<p>
D
Dong Daxiang 已提交
18

D
Dong Daxiang 已提交
19
<h2 align="center">Motivation</h2>
D
Dong Daxiang 已提交
20

D
Dong Daxiang 已提交
21
We consider deploying deep learning inference service online to be a user-facing application in the future. **The goal of this project**: When you have trained a deep neural net with [Paddle](https://github.com/PaddlePaddle/Paddle), you can put the model online without much effort. A demo of serving is as follows:
D
Dong Daxiang 已提交
22
<p align="center">
D
Dong Daxiang 已提交
23
    <img src="doc/demo.gif" width="700">
D
Dong Daxiang 已提交
24
</p>
D
Dong Daxiang 已提交
25

D
Dong Daxiang 已提交
26
<h2 align="center">Some Key Features</h2>
D
Dong Daxiang 已提交
27

D
Dong Daxiang 已提交
28
- Integrate with Paddle training pipeline seemlessly, most paddle models can be deployed **with one line command**.
D
Dong Daxiang 已提交
29 30 31
- **Industrial serving features** supported, such as models management, online loading, online A/B testing etc.
- **Distributed Key-Value indexing** supported that is especially useful for large scale sparse features as model inputs.
- **Highly concurrent and efficient communication** between clients and servers.
D
Dong Daxiang 已提交
32
- **Multiple programming languages** supported on client side, such as Golang, C++ and python
D
Dong Daxiang 已提交
33
- **Extensible framework design** that can support model serving beyond Paddle.
D
Dong Daxiang 已提交
34

D
Dong Daxiang 已提交
35
<h2 align="center">Installation</h2>
D
Dong Daxiang 已提交
36

D
Dong Daxiang 已提交
37 38
We highly recommend you to run Paddle Serving in Docker, please visit [Run in Docker](https://github.com/PaddlePaddle/Serving/blob/develop/doc/RUN_IN_DOCKER.md)

D
Dong Daxiang 已提交
39 40 41 42 43
```shell
pip install paddle-serving-client
pip install paddle-serving-server
```

D
Dong Daxiang 已提交
44
<h2 align="center">Quick Start Example</h2>
D
Dong Daxiang 已提交
45

D
Dong Daxiang 已提交
46
### Boston House Price Prediction model
D
Dong Daxiang 已提交
47
``` shell
D
Dong Daxiang 已提交
48
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
D
Dong Daxiang 已提交
49
tar -xzf uci_housing.tar.gz
D
Dong Daxiang 已提交
50
```
D
Dong Daxiang 已提交
51

D
Dong Daxiang 已提交
52 53
Paddle Serving provides HTTP and RPC based service for users to access

D
Dong Daxiang 已提交
54
### HTTP service
D
Dong Daxiang 已提交
55

D
Dong Daxiang 已提交
56
Paddle Serving provides a built-in python module called `paddle_serving_server.serve` that can start a rpc service or a http service with one-line command. If we specify the argument `--name uci`, it means that we will have a HTTP service with a url of `$IP:$PORT/uci/prediction`
D
Dong Daxiang 已提交
57
``` shell
D
Dong Daxiang 已提交
58
python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292 --name uci
D
Dong Daxiang 已提交
59
```
D
Dong Daxiang 已提交
60 61
<center>

D
Dong Daxiang 已提交
62 63
| Argument | Type | Default | Description |
|--------------|------|-----------|--------------------------------|
D
Dong Daxiang 已提交
64
| `thread` | int | `4` | Concurrency of current service |
D
Dong Daxiang 已提交
65
| `port` | int | `9292` | Exposed port of current service to users|
D
Dong Daxiang 已提交
66 67
| `name` | str | `""` | Service name, can be used to generate HTTP request url |
| `model` | str | `""` | Path of paddle model directory to be served |
D
Dong Daxiang 已提交
68

D
Dong Daxiang 已提交
69
Here, we use `curl` to send a HTTP POST request to the service we just started. Users can use any python library to send HTTP POST as well, e.g, [requests](https://requests.readthedocs.io/en/master/).
D
Dong Daxiang 已提交
70 71
</center>

D
Dong Daxiang 已提交
72 73 74
``` shell
curl -H "Content-Type:application/json" -X POST -d '{"x": [0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332], "fetch":["price"]}' http://127.0.0.1:9292/uci/prediction
```
D
Dong Daxiang 已提交
75

D
Dong Daxiang 已提交
76
### RPC service
D
Dong Daxiang 已提交
77

D
Dong Daxiang 已提交
78
A user can also start a rpc service with `paddle_serving_server.serve`. RPC service is usually faster than HTTP service, although a user needs to do some coding based on Paddle Serving's python client API. Note that we do not specify `--name` here. 
D
Dong Daxiang 已提交
79 80 81
``` shell
python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292
```
D
Dong Daxiang 已提交
82

D
Dong Daxiang 已提交
83
``` python
D
Dong Daxiang 已提交
84
# A user can visit rpc service through paddle_serving_client API
D
Dong Daxiang 已提交
85 86 87
from paddle_serving_client import Client

client = Client()
D
Dong Daxiang 已提交
88
client.load_client_config("uci_housing_client/serving_client_conf.prototxt")
D
Dong Daxiang 已提交
89
client.connect(["127.0.0.1:9292"])
D
Dong Daxiang 已提交
90
data = [0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727,
D
Dong Daxiang 已提交
91
        -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332]
D
Dong Daxiang 已提交
92
fetch_map = client.predict(feed={"x": data}, fetch=["price"])
D
Dong Daxiang 已提交
93
print(fetch_map)
D
Dong Daxiang 已提交
94 95

```
D
Dong Daxiang 已提交
96
Here, `client.predict` function has two arguments. `feed` is a `python dict` with model input variable alias name and values. `fetch` assigns the prediction variables to be returned from servers. In the example, the name of `"x"` and `"price"` are assigned when the servable model is saved during training.
D
Dong Daxiang 已提交
97

D
Dong Daxiang 已提交
98 99
<h2 align="center"> Pre-built services with Paddle Serving</h2>

D
Dong Daxiang 已提交
100 101
<h3 align="center">Chinese Word Segmentation</h4>

D
Dong Daxiang 已提交
102
- **Description**: 
D
Dong Daxiang 已提交
103
``` shell
D
Dong Daxiang 已提交
104
Chinese word segmentation HTTP service that can be deployed with one line command.
D
Dong Daxiang 已提交
105
```
D
Dong Daxiang 已提交
106

D
Dong Daxiang 已提交
107
- **Download Servable Package**: 
D
Dong Daxiang 已提交
108 109 110
``` shell
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/lac/lac_model_jieba_web.tar.gz
```
D
Dong Daxiang 已提交
111
- **Host web service**: 
D
Dong Daxiang 已提交
112 113 114 115
``` shell
tar -xzf lac_model_jieba_web.tar.gz
python lac_web_service.py jieba_server_model/ lac_workdir 9292
```
D
Dong Daxiang 已提交
116
- **Request sample**: 
D
Dong Daxiang 已提交
117
``` shell
D
Dong Daxiang 已提交
118
curl -H "Content-Type:application/json" -X POST -d '{"words": "我爱北京天安门", "fetch":["word_seg"]}' http://127.0.0.1:9292/lac/prediction
D
Dong Daxiang 已提交
119
```
D
Dong Daxiang 已提交
120
- **Request result**: 
D
Dong Daxiang 已提交
121 122 123 124
``` shell
{"word_seg":"我|爱|北京|天安门"}
```

D
Dong Daxiang 已提交
125
<h3 align="center">Image Classification</h4>
D
Dong Daxiang 已提交
126

D
Dong Daxiang 已提交
127
- **Description**: 
D
Dong Daxiang 已提交
128
``` shell
D
Dong Daxiang 已提交
129
Image classification trained with Imagenet dataset. A label and corresponding probability will be returned.
D
Dong Daxiang 已提交
130
```
D
Dong Daxiang 已提交
131

D
Dong Daxiang 已提交
132
- **Download Servable Package**: 
D
Dong Daxiang 已提交
133 134 135 136 137 138 139 140 141
``` shell
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/imagenet_demo.tar.gz
```
- **Host web service**: 
``` shell
tar -xzf imagenet_demo.tar.gz
python image_classification_service_demo.py resnet50_serving_model
```
- **Request sample**: 
D
Dong Daxiang 已提交
142 143 144 145 146 147 148

<p align="center">
    <br>
<img src='https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg' width = "200" height = "200">
    <br>
<p>
    
D
Dong Daxiang 已提交
149 150 151 152 153 154 155 156 157 158
``` shell
curl -H "Content-Type:application/json" -X POST -d '{"url": "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg", "fetch": ["score"]}' http://127.0.0.1:9292/image/prediction
```
- **Request result**: 
``` shell
{"label":"daisy","prob":0.9341403245925903}
```



D
Dong Daxiang 已提交
159 160


D
Dong Daxiang 已提交
161
<h2 align="center">Document</h2>
D
Dong Daxiang 已提交
162

D
Dong Daxiang 已提交
163
### New to Paddle Serving
D
Dong Daxiang 已提交
164
- [How to save a servable model?](doc/SAVE.md)
D
Dong Daxiang 已提交
165
- [An end-to-end tutorial from training to serving(Chinese)](doc/TRAIN_TO_SERVICE.md)
D
Dong Daxiang 已提交
166
- [Write Bert-as-Service in 10 minutes(Chinese)](doc/BERT_10_MINS.md)
D
Dong Daxiang 已提交
167

D
Dong Daxiang 已提交
168
### Developers
D
Dong Daxiang 已提交
169 170 171 172
- [How to config Serving native operators on server side?](doc/SERVER_DAG.md)
- [How to develop a new Serving operator](doc/NEW_OPERATOR.md)
- [Golang client](doc/IMDB_GO_CLIENT.md)
- [Compile from source code(Chinese)](doc/COMPILE.md)
D
Dong Daxiang 已提交
173

D
Dong Daxiang 已提交
174
### About Efficiency
D
Dong Daxiang 已提交
175
- [How profile serving efficiency?(Chinese)](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/util)
D
Dong Daxiang 已提交
176
- [Benchmarks](doc/BENCHMARK.md)
D
Dong Daxiang 已提交
177

D
Dong Daxiang 已提交
178
### FAQ
D
Dong Daxiang 已提交
179
- [FAQ(Chinese)](doc/FAQ.md)
D
Dong Daxiang 已提交
180

D
Dong Daxiang 已提交
181

D
Dong Daxiang 已提交
182
### Design
D
Dong Daxiang 已提交
183
- [Design Doc(Chinese)](doc/DESIGN_DOC.md)
D
Dong Daxiang 已提交
184
- [Design Doc(English)](doc/DESIGN_DOC_EN.md)
D
Dong Daxiang 已提交
185

D
Dong Daxiang 已提交
186 187 188
<h2 align="center">Community</h2>

### Slack
D
Dong Daxiang 已提交
189

D
Dong Daxiang 已提交
190 191
To connect with other users and contributors, welcome to join our [Slack channel](https://paddleserving.slack.com/archives/CUBPKHKMJ)

D
Dong Daxiang 已提交
192
### Contribution
D
Dong Daxiang 已提交
193

D
Dong Daxiang 已提交
194
If you want to contribute code to Paddle Serving, please reference [Contribution Guidelines](doc/CONTRIBUTE.md)
D
Dong Daxiang 已提交
195 196

### Feedback
D
Dong Daxiang 已提交
197

D
Dong Daxiang 已提交
198 199
For any feedback or to report a bug, please propose a [GitHub Issue](https://github.com/PaddlePaddle/Serving/issues).

D
Dong Daxiang 已提交
200 201
### License

D
Dong Daxiang 已提交
202
[Apache 2.0 License](https://github.com/PaddlePaddle/Serving/blob/develop/LICENSE)