Skip to content

S
Serving

PaddlePaddle / Serving
大约 1 年 前同步成功

通知 185
Star 833
Fork 253
S
Serving

体验新版 GitCode，发现更多精彩内容 >>

未验证 提交 56fde8bb 编写于 作者: S ShiningZhang 提交者： GitHub
浏览文件
操作

Merge branch 'develop' into dev_doc

上级 32ccb13f 1f35ae38
展开全部 隐藏空白更改
内联 并排
Showing with 31470 addition and 1196 deletion
README.md 100644 → 100755
浏览文件 @ 56fde8bb
......@@ -176,8 +176,8 @@ python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --p
| Argument | Type | Default | Description |
| ---------------------------------------------- | ---- | ------- | ----------------------------------------------------- |
| `thread` | int | `2` | Number of brpc service thread |
| `op_num` | int[]| `0` | Thread Number for each model in asynchronous mode |
| `op_max_batch` | int[]| `0` | Batch Number for each model in asynchronous mode |
| `runtime_thread_num` | int[]| `0` | Thread Number for each model in asynchronous mode |
| `batch_infer_size` | int[]| `0` | Batch Number for each model in asynchronous mode |
| `gpu_ids` | str[]| `"-1"` | Gpu card id for each model |
| `port` | int | `9292` | Exposed port of current service to users |
| `model` | str[]| `""` | Path of paddle model directory to be served |
......@@ -197,8 +197,8 @@ python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --p
In asynchronous mode, each model will start n threads of the number you specify, and each thread contains a model instance. In other words, each model is equivalent to a thread pool containing N threads, and the task is taken from the task queue of the thread pool to execute.
In asynchronous mode, each RPC server thread is only responsible for putting the request into the task queue of the model thread pool. After the task is executed, the completed task is removed from the task queue.
In the above table, the number of RPC server threads is specified by --thread, and the default value is 2.
--op_num specifies the number of threads in the thread pool of each model. The default value is 0, indicating that asynchronous mode is not used.
--op_max_batch specifies the number of batches for each model. The default value is 32. It takes effect when --op_num is not 0.
--runtime_thread_num specifies the number of threads in the thread pool of each model. The default value is 0, indicating that asynchronous mode is not used.
--batch_infer_size specifies the number of batches for each model. The default value is 32. It takes effect when --runtime_thread_num is not 0.
#### When you want a model to use multiple GPU cards.
python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292 --gpu_ids 0,1,2
#### When you want 2 models.
......@@ -206,7 +206,7 @@ python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_m
#### When you want 2 models, and want each of them use multiple GPU cards.
python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_model_2 --thread 10 --port 9292 --gpu_ids 0,1 1,2
#### When a service contains two models, and each model needs to specify multiple GPU cards, and needs asynchronous mode, each model specifies different concurrency number.
python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_model_2 --thread 10 --port 9292 --gpu_ids 0,1 1,2 --op_num 4 8
python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_model_2 --thread 10 --port 9292 --gpu_ids 0,1 1,2 --runtime_thread_num 4 8
</center>
```python
......
README_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
......@@ -175,8 +175,8 @@ python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --p
| Argument | Type | Default | Description |
| ---------------------------------------------- | ---- | ------- | ----------------------------------------------------- |
| `thread` | int | `2` | Number of brpc service thread |
| `op_num` | int[]| `0` | Thread Number for each model in asynchronous mode |
| `op_max_batch` | int[]| `32` | Batch Number for each model in asynchronous mode |
| `runtime_thread_num` | int[]| `0` | Thread Number for each model in asynchronous mode |
| `batch_infer_size` | int[]| `32` | Batch Number for each model in asynchronous mode |
| `gpu_ids` | str[]| `"-1"` | Gpu card id for each model |
| `port` | int | `9292` | Exposed port of current service to users |
| `model` | str[]| `""` | Path of paddle model directory to be served |
......@@ -195,8 +195,8 @@ python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --p
异步模式有助于提高Service服务的吞吐（QPS），但对于单次请求而言，时延会有少量增加。
异步模式中，每个模型会启动您指定个数的N个线程，每个线程中包含一个模型实例，换句话说每个模型相当于包含N个线程的线程池，从线程池的任务队列中取任务来执行。
异步模式中，各个RPC Server的线程只负责将Request请求放入模型线程池的任务队列中，等任务被执行完毕后，再从任务队列中取出已完成的任务。
上表中通过 --thread 10 指定的是RPC Server的线程数量，默认值为2，--op_num 指定的是各个模型的线程池中线程数N，默认值为0，表示不使用异步模式。
--op_max_batch 指定的各个模型的batch数量，默认值为32，该参数只有当--op_num不为0时才生效。
上表中通过 --thread 10 指定的是RPC Server的线程数量，默认值为2，--runtime_thread_num 指定的是各个模型的线程池中线程数N，默认值为0，表示不使用异步模式。
--batch_infer_size 指定的各个模型的batch数量，默认值为32，该参数只有当--runtime_thread_num不为0时才生效。
#### 当您的某个模型想使用多张GPU卡部署时.
python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292 --gpu_ids 0,1,2
......@@ -205,7 +205,7 @@ python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_m
#### 当您的一个服务包含两个模型，且每个模型都需要指定多张GPU卡部署时.
python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_model_2 --thread 10 --port 9292 --gpu_ids 0,1 1,2
#### 当您的一个服务包含两个模型，且每个模型都需要指定多张GPU卡，且需要异步模式每个模型指定不同的并发数时.
python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_model_2 --thread 10 --port 9292 --gpu_ids 0,1 1,2 --op_num 4 8
python3 -m paddle_serving_server.serve --model uci_housing_model_1 uci_housing_model_2 --thread 10 --port 9292 --gpu_ids 0,1 1,2 --runtime_thread_num 4 8
</center>
......
doc/cpp_server/ABTEST_IN_PADDLE_SERVING_CN.md → doc/C++Serving/ABTest_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
# 如何使用Paddle Serving做ABTEST
(简体中文|[English](./ABTEST_IN_PADDLE_SERVING.md))
(简体中文|[English](./ABTest_EN.md))
该文档将会用一个基于IMDB数据集的文本分类任务的例子，介绍如何使用Paddle Serving搭建A/B Test框架，例中的Client端、Server端结构如下图所示。
<img src="images/abtest.png" style="zoom:33%;" />
<img src="../images/abtest.png" style="zoom:33%;" />
需要注意的是：A/B Test只适用于RPC模式，不适用于WEB模式。
......@@ -24,13 +24,13 @@ pip install Shapely
````
您可以直接运行下面的命令来处理数据。
[python abtest_get_data.py](../python/examples/imdb/abtest_get_data.py)
[python abtest_get_data.py](../../examples/C++/imdb/abtest_get_data.py)
文件中的Python代码将处理`test_data/part-0`的数据，并将处理后的数据生成并写入`processed.data`文件中。
### 启动Server端
这里采用[Docker方式](RUN_IN_DOCKER_CN.md)启动Server端服务。
这里采用[Docker方式](../RUN_IN_DOCKER_CN.md)启动Server端服务。
首先启动BOW Server，该服务启用`8000`端口：
......@@ -62,7 +62,7 @@ exit
您可以直接使用下面的命令，进行ABTEST预测。
[python abtest_client.py](../python/examples/imdb/abtest_client.py)
[python abtest_client.py](../../examples/C++/imdb/abtest_client.py)
```python
from paddle_serving_client import Client
......
doc/cpp_server/ABTEST_IN_PADDLE_SERVING.md → doc/C++Serving/ABTest_EN.md 100644 → 100755
浏览文件 @ 56fde8bb
# ABTEST in Paddle Serving
([简体中文](./ABTEST_IN_PADDLE_SERVING_CN.md)|English)
([简体中文](./ABTest_CN.md)|English)
This document will use an example of text classification task based on IMDB dataset to show how to build a A/B Test framework using Paddle Serving. The structure relationship between the client and servers in the example is shown in the figure below.
<img src="images/abtest.png" style="zoom:25%;" />
<img src="../images/abtest.png" style="zoom:25%;" />
Note that: A/B Test is only applicable to RPC mode, not web mode.
......@@ -25,13 +25,13 @@ pip install Shapely
You can directly run the following command to process the data.
[python abtest_get_data.py](../python/examples/imdb/abtest_get_data.py)
[python abtest_get_data.py](../../examples/C++/imdb/abtest_get_data.py)
The Python code in the file will process the data `test_data/part-0` and write to the `processed.data` file.
### Start Server
Here, we [use docker](RUN_IN_DOCKER.md) to start the server-side service.
Here, we [use docker](../RUN_IN_DOCKER.md) to start the server-side service.
First, start the BOW server, which enables the `8000` port:
......@@ -63,7 +63,7 @@ Before running, use `pip install paddle-serving-client` to install the paddle-se
You can directly use the following command to make abtest prediction.
[python abtest_client.py](../python/examples/imdb/abtest_client.py)
[python abtest_client.py](../../examples/C++/imdb/abtest_client.py)
[//file]:#abtest_client.py
``` python
......
doc/C++Serving/Benchmark_CN.md 0 → 100755
浏览文件 @ 56fde8bb
# C++ Serving vs TensorFlow Serving 性能对比
# 1. 测试环境和说明
1) GPU型号：Tesla P4(7611 Mib)
2) Cuda版本：11.0
3) 模型：ResNet_v2_50
4) 为了测试异步合并batch的效果，测试数据中batch=1
5) [使用的测试代码和使用的数据集](../../examples/C++/PaddleClas/resnet_v2_50)
6) 下图中蓝色是C++ Serving，灰色为TF-Serving。
7) 折线图为QPS，数值越大表示每秒钟处理的请求数量越大，性能就越好。
8) 柱状图为平均处理时延，数值越大表示单个请求处理时间越长，性能就越差。
# 2. 同步模式
均使用同步模式，默认参数配置。
可以看出同步模型默认参数配置情况下，C++Serving QPS和平均时延指标均优于TF-Serving。
<p align="center">
<br>
<img src='../images/syn_benchmark.png'">
<br>
<p>
|client_num | model_name | qps(samples/s) | mean(ms) | model_name | qps(samples/s) | mean(ms) |
| --- | --- | --- | --- | --- | --- | --- |
| 10 | pd-serving | 111.336 | 89.787| tf-serving| 84.632| 118.13|
|30 |pd-serving |165.928 |180.761 |tf-serving |106.572 |281.473|
|50| pd-serving| 207.244| 241.211| tf-serving| 80.002 |624.959|
|70 |pd-serving |214.769 |325.894 |tf-serving |105.17 |665.561|
|100| pd-serving| 235.405| 424.759| tf-serving| 93.664 |1067.619|
|150 |pd-serving |239.114 |627.279 |tf-serving |86.312 |1737.848|
# 3. 异步模式
均使用异步模式，最大batch=32，异步线程数=2。
可以看出异步模式情况下，两者性能接近，但当Client端并发数达到70的时候，TF-Serving服务直接超时，而C++Serving能够正常返回结果。
同时，对比同步和异步模式可以看出，异步模式在请求batch数较小时，通过合并batch能够有效提高QPS和平均处理时延。
<p align="center">
<br>
<img src='../images/asyn_benchmark.png'">
<br>
<p>
|client_num | model_name | qps(samples/s) | mean(ms) | model_name | qps(samples/s) | mean(ms) |
| --- | --- | --- | --- | --- | --- | --- |
|10| pd-serving| 130.631| 76.502| tf-serving |172.64 |57.916|
|30| pd-serving| 201.062| 149.168| tf-serving| 241.669| 124.128|
|50| pd-serving| 286.01| 174.764| tf-serving |278.744 |179.367|
|70| pd-serving| 313.58| 223.187| tf-serving| 298.241| 234.7|
|100| pd-serving| 323.369| 309.208| tf-serving| 0| ∞|
|150| pd-serving| 328.248| 456.933| tf-serving| 0| ∞|
doc/cpp_server/CREATING.md → doc/C++Serving/Creat_C++Serving_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
......@@ -75,9 +75,9 @@ service ImageClassifyService {
#### 2.2.2 示例配置
关于Serving端的配置的详细信息，可以参考[Serving端配置](SERVING_CONFIGURE.md)
关于Serving端的配置的详细信息，可以参考[Serving端配置](../SERVING_CONFIGURE_CN.md)
以下配置文件将ReaderOP, ClassifyOP和WriteJsonOP串联成一个workflow (关于OP/workflow等概念，可参考[设计文档](C++DESIGN_CN.md))
以下配置文件将ReaderOP, ClassifyOP和WriteJsonOP串联成一个workflow (关于OP/workflow等概念，可参考[OP介绍](OP_CN.md)和[DAG介绍](DAG_CN.md))
- 配置文件示例：
......@@ -310,7 +310,7 @@ api.thrd_finalize();
api.destroy();
```
具体实现可参考paddle Serving提供的例子sdk-cpp/demo/ximage.cpp
具体实现可参考C++Serving提供的例子。sdk-cpp/demo/ximage.cpp
### 3.3 链接
......@@ -392,4 +392,4 @@ predictors {
}
}
```
关于客户端的详细配置选项，可参考[CLIENT CONFIGURATION](CLIENT_CONFIGURE.md)
关于客户端的详细配置选项，可参考[CLIENT CONFIGURATION](Client_Configure_CN.md)
doc/cpp_server/SERVER_DAG_CN.md → doc/C++Serving/DAG_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
# Server端的计算图
(简体中文|[English](./SERVER_DAG.md))
(简体中文|[English](DAG_EN.md))
本文档显示了Server端上计算图的概念。 如何使用PaddleServing内置运算符定义计算图。 还显示了一些顺序执行逻辑的示例。
......@@ -9,7 +9,7 @@
深度神经网络通常在输入数据上有一些预处理步骤，而在模型推断分数上有一些后处理步骤。 由于深度学习框架现在非常灵活，因此可以在训练计算图之外进行预处理和后处理。 如果要在服务器端进行输入数据预处理和推理结果后处理，则必须在服务器上添加相应的计算逻辑。 此外，如果用户想在多个模型上使用相同的输入进行推理，则最好的方法是在仅提供一个客户端请求的情况下在服务器端同时进行推理，这样我们可以节省一些网络计算开销。 由于以上两个原因，自然而然地将有向无环图（DAG）视为服务器推理的主要计算方法。 DAG的一个示例如下：
<center>
<img src='images/server_dag.png' width = "450" height = "500" align="middle"/>
<img src='../images/server_dag.png' width = "450" height = "500" align="middle"/>
</center>
## 如何定义节点
......@@ -18,7 +18,7 @@
PaddleServing在框架中具有一些预定义的计算节点。 一种非常常用的计算图是简单的reader-infer-response模式，可以涵盖大多数单一模型推理方案。 示例图和相应的DAG定义代码如下。
<center>
<img src='images/simple_dag.png' width = "260" height = "370" align="middle"/>
<img src='../images/simple_dag.png' width = "260" height = "370" align="middle"/>
</center>
``` python
......@@ -47,10 +47,10 @@ python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --po
### 包含多个输入的节点
在[Paddle Serving中的集成预测](./deprecated/MODEL_ENSEMBLE_IN_PADDLE_SERVING_CN.md)文档中给出了一个包含多个输入节点的样例，示意图和代码如下。
在[Paddle Serving中的集成预测](Model_Ensemble_CN.md)文档中给出了一个包含多个输入节点的样例，示意图和代码如下。
<center>
<img src='images/complex_dag.png' width = "480" height = "400" align="middle"/>
<img src='../images/complex_dag.png' width = "480" height = "400" align="middle"/>
</center>
```python
......
doc/cpp_server/SERVER_DAG.md → doc/C++Serving/DAG_EN.md 100644 → 100755
浏览文件 @ 56fde8bb
# Computation Graph On Server
([简体中文](./SERVER_DAG_CN.md)|English)
([简体中文](./DAG_CN.md)|English)
This document shows the concept of computation graph on server. How to define computation graph with PaddleServing built-in operators. Examples for some sequential execution logics are shown as well.
......@@ -9,7 +9,7 @@ This document shows the concept of computation graph on server. How to define co
Deep neural nets often have some preprocessing steps on input data, and postprocessing steps on model inference scores. Since deep learning frameworks are now very flexible, it is possible to do preprocessing and postprocessing outside the training computation graph. If we want to do input data preprocessing and inference result postprocess on server side, we have to add the corresponding computation logics on server. Moreover, if a user wants to do inference with the same inputs on more than one model, the best way is to do the inference concurrently on server side given only one client request so that we can save some network computation overhead. For the above two reasons, it is naturally to think of a Directed Acyclic Graph(DAG) as the main computation method for server inference. One example of DAG is as follows:
<center>
<img src='images/server_dag.png' width = "450" height = "500" align="middle"/>
<img src='../images/server_dag.png' width = "450" height = "500" align="middle"/>
</center>
## How to define Node
......@@ -19,7 +19,7 @@ Deep neural nets often have some preprocessing steps on input data, and postproc
PaddleServing has some predefined Computation Node in the framework. A very commonly used Computation Graph is the simple reader-inference-response mode that can cover most of the single model inference scenarios. A example graph and the corresponding DAG definition code is as follows.
<center>
<img src='images/simple_dag.png' width = "260" height = "370" align="middle"/>
<img src='../images/simple_dag.png' width = "260" height = "370" align="middle"/>
</center>
``` python
......@@ -48,10 +48,10 @@ python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --po
### Nodes with multiple inputs
An example containing multiple input nodes is given in the [MODEL_ENSEMBLE_IN_PADDLE_SERVING](./deprecated/MODEL_ENSEMBLE_IN_PADDLE_SERVING.md). A example graph and the corresponding DAG definition code is as follows.
An example containing multiple input nodes is given in the [Model_Ensemble](Model_Ensemble_EN.md). A example graph and the corresponding DAG definition code is as follows.
<center>
<img src='images/complex_dag.png' width = "480" height = "400" align="middle"/>
<img src='../images/complex_dag.png' width = "480" height = "400" align="middle"/>
</center>
```python
......
doc/cpp_server/ENCRYPTION_CN.md → doc/C++Serving/Encryption_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
# 加密模型预测
(简体中文|[English](ENCRYPTION.md))
(简体中文|[English](Encryption_EN.md))
Padle Serving提供了模型加密预测功能，本文档显示了详细信息。
......@@ -12,7 +12,7 @@ Padle Serving提供了模型加密预测功能，本文档显示了详细信息
普通的模型和参数可以理解为一个字符串，通过对其使用加密算法（参数是您的密钥），普通模型和参数就变成了一个加密的模型和参数。
我们提供了一个简单的演示来加密模型。请参阅[`python/examples/encryption/encrypt.py`](../python/examples/encryption/encrypt.py)。
我们提供了一个简单的演示来加密模型。请参阅[examples/C++/encryption/encrypt.py](../../examples/C++/encryption/encrypt.py)。
### 启动加密服务
......@@ -40,5 +40,4 @@ python -m paddle_serving_server.serve --model encrypt_server/ --port 9300 --use_
### 模型加密推理示例
模型加密推理示例, 请参见[`/python/examples/encryption/`](../python/examples/encryption/)。
模型加密推理示例, 请参见[examples/C++/encryption/](../../examples/C++/encryption/)。
doc/cpp_server/ENCRYPTION.md → doc/C++Serving/Encryption_EN.md 100644 → 100755
浏览文件 @ 56fde8bb
# MOEDL ENCRYPTION INFERENCE
([简体中文](ENCRYPTION_CN.md)|English)
([简体中文](Encryption_CN.md)|English)
Paddle Serving provides model encryption inference, This document shows the details.
......@@ -12,7 +12,7 @@ We use symmetric encryption algorithm to encrypt the model. Symmetric encryption
Normal model and parameters can be understood as a string, by using the encryption algorithm (parameter is your key) on them, the normal model and parameters become an encrypted one.
We provide a simple demo to encrypt the model. See the [python/examples/encryption/encrypt.py](../python/examples/encryption/encrypt.py)。
We provide a simple demo to encrypt the model. See the [examples/C++/encryption/encrypt.py](../../examples/C++/encryption/encrypt.py)。
### Start Encryption Service
......@@ -40,5 +40,4 @@ Once the server gets the key, it uses the key to parse the model and starts the
### Example of Model Encryption Inference
Example of model encryption inference, See the [`/python/examples/encryption/`](../python/examples/encryption/)。
Example of model encryption inference, See the [examples/C++/encryption/](../../examples/C++/encryption/)。
doc/C++Serving/Frame_Performance_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
doc/cpp_server/HOT_LOADING_IN_SERVING_CN.md → doc/C++Serving/Hot_Loading_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
# Paddle Serving中的模型热加载
(简体中文|[English](HOT_LOADING_IN_SERVING.md))
(简体中文|[English](Hot_Loading_EN.md))
## 背景
......
doc/cpp_server/HOT_LOADING_IN_SERVING.md → doc/C++Serving/Hot_Loading_EN.md 100644 → 100755
浏览文件 @ 56fde8bb
# Hot Loading in Paddle Serving
([简体中文](HOT_LOADING_IN_SERVING_CN.md)|English)
([简体中文](Hot_Loading_CN.md)|English)
## Background
......
doc/HTTP_SERVICE_CN.md → doc/C++Serving/Http_Service_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
......@@ -14,12 +14,10 @@ BRPC-Server会尝试去JSON字符串中再去反序列化出Proto格式的数据
各种语言都提供了对ProtoBuf的支持，如果您对此比较熟悉，您也可以先将数据使用ProtoBuf序列化，再将序列化后的数据放入Http请求数据体中，然后指定Content-Type: application/proto，从而使用http/h2+protobuf二进制串访问服务。
实测随着数据量的增大，使用JSON方式的Http的数据量和反序列化的耗时会大幅度增加，推荐当您的数据量较大时，使用Http+protobuf方式，目前已经在Java和Python的Client端提供了支持。
**理论上讲，序列化/反序列化的性能从高到底排序为：protobuf > http/h2+protobuf > http**
## 示例
我们将以python/examples/fit_a_line为例，讲解如何通过Http访问Server端。
我们将以examples/C++/fit_a_line为例，讲解如何通过Http访问Server端。
### 获取模型
......
doc/C++Serving/Introduction_CN.md 0 → 100755
浏览文件 @ 56fde8bb
# C++ Serving 简要介绍
## 适用场景
C++ Serving主打性能，如果您想搭建企业级的高性能线上推理服务，对高并发、低延时有一定的要求。C++ Serving框架可能会更适合您。目前无论是使用同步/异步模型，[C++ Serving与TensorFlow Serving性能对比](Benchmark_CN.md)均有优势。
C++ Serving网络框架使用brpc，核心执行引擎是基于C/C++编写，并且提供强大的工业级应用能力，包括模型热加载、模型加密部署、A/B Test、多模型组合、同步/异步模式、支持多语言多协议Client等功能。
## 1.网络框架(BRPC)
C++ Serving采用[brpc框架](https://github.com/apache/incubator-brpc)进行Client/Server端的通信。brpc是百度开源的一款PRC网络框架，具有高并发、低延时等特点，已经支持了包括百度在内上百万在线预估实例、上千个在线预估服务，稳定可靠。与gRPC网络框架相比，具有更低的延时，更高的并发性能，且底层支持<mark>**brpc/grpc/http+json/http+proto**</mark>等多种协议；缺点是跨操作系统平台能力不足。详细的框架性能开销见[C++ Serving框架性能测试](Frame_Performance_CN.md)。
## 2.核心执行引擎
C++ Serving的核心执行引擎是一个有向无环图(也称作[DAG图](DAG_CN.md))，DAG图中的每个节点(在PaddleServing中,借用模型中operator算子的概念，将DAG图中的节点也称为[OP](OP_CN.md))代表预估服务的一个环节，DAG图支持多个OP按照串并联的方式进行组合，从而实现在一个服务中完成多个模型的预测整合最终产出结果。整个框架原理如下图所示，可分为Client Side 和 Server Side。
<p align="center">
<br>
<img src='../images/design_doc.png'">
<br>
<p>
### 2.1 Client Side
如图所示，Client端通过Pybind API接口将Request请求，按照ProtoBuf协议进行序列化后，经由BRPC网络框架Client端发送给Server端。此时，Client端等待Server端的返回数据并反序列化为正常的数据，之后将结果返给Client调用方。
### 2.2 Server Side
Server端接收到序列化的Request请求后，反序列化正常数据，进入图执行引擎，按照定义好的DAG图结构，执行每个OP环节的操作（每个OP环节的处理由用户定义，即可以只是单纯的数据处理，也可以是调用预测引擎用不同的模型对输入数据进行预测），当DAG图中所有OP环节均执行完成后，将结果数据序列化后返回给Client端。
### 2.3 通信数据格式ProtoBuf
Protocol Buffers(简称Protobuf) ，是Google出品的序列化框架，与开发语言无关，和平台无关，具有良好的可扩展性。Protobuf和所有的序列化框架一样，都可以用于数据存储、通讯协议。Protobuf支持生成代码的语言包括Java、Python、C++、Go、JavaNano、Ruby、C#。Portobuf的序列化的结果体积要比XML、JSON小很多，速度比XML、JSON快很多。
在C++ Serving中定义了Client Side 和 Server Side之间通信的ProtoBuf，详细的字段的介绍见《[C++ Serving ProtoBuf简介](Inference_Protocols_CN.md)》。
## 3.Server端特性
### 3.1 启动Server端
Server端的核心是一个由项目代码编译产生的名称为serving的二进制可执行文件，启动serving时需要用户指定一些参数(<mark>**例如，网络IP和Port端口、brpc线程数、使用哪个显卡、模型文件路径、模型是否开启trt、XPU推理、模型精度设置等等**</mark>)，有些参数是通过命令行直接传入的，还有一些是写在指定的配置文件中配置文件中。
为了方便用户快速的启动C++ Serving的Server端，除了用户自行修改配置文件并通过命令行传参运行serving二进制可执行文件以外，我们也提供了另外一种通过python脚本启动的方式。python脚本启动本质上仍是运行serving二进制可执行文件，但python脚本中会自动完成两件事：1、配置文件的生成；2、根据需要配置的参数，生成命令行，通过命令行的方式，传入参数信息并运行serving二进制可执行文件。
更多详细说明和示例，请参考[C++ Serving 参数配置和启动的详细说明](../SERVING_CONFIGURE_CN.md)。
### 3.2 同步/异步模式
同步模式比较简单直接，适用于模型预测时间短，单个Request请求的batch已经比较大的情况。
同步模型下，Server端线程数N = 模型预测引擎数N = 同时处理Request请求数N，超发的Request请求需要等待当前线程处理结束后才能得到响应和处理。
<p align="center">
<img src='../images/syn_mode.png' width = "350" height = "300">
<p>
异步模型主要适用于模型支持多batch(最大batch数M可通过配置选项指定),单个Request请求的batch较小(batch << M)，单次预测时间较长的情况。
异步模型下，Server端N个线程只负责接收Request请求，实际调用预测引擎是在异步框架的线程中，异步框架的线程数可以由配置选项来指定。为了方便理解，我们假设每个Request请求的batch均为1，此时异步框架会尽可能多得从请求池中取n(n≤M)个Request并将其拼装为1个Request(batch=n)，调用1次预测引擎，得到1个Response(batch = n)，再将其对应拆分为n个Response作为返回结果。
<p align="center">
<img src='../images/asyn_mode.png'">
<p>
更多关于模式参数配置以及性能调优的介绍见《[C++ Serving性能调优](Performance_Tuning_CN.md)》。
### 3.3 多模型组合
当用户需要多个模型组合处理结果来作为一个服务接口对外暴露时，通常的解决办法是搭建内外两层服务，内层服务负责跑模型预测，外层服务负责串联和前后处理。当传输的数据量不大时，这样做的性能开销并不大，但当输出的数据量较大时，因为网络传输而带来的性能开销不容忽视(实测单次传输40MB数据时，RPC耗时为160-170ms)。
<p align="center">
<br>
<img src='../images/multi_model.png'>
<br>
<p>
C++ Serving框架支持[自定义DAG图](Model_Ensemble_CN.md)的方式来表示多模型之间串并联组合关系，也支持用户[使用C++开发自定义OP节点](OP_CN.md)。相比于使用内外两层服务来提供多模型组合处理的方式，由于节省了一次RPC网络传输的开销，把多模型在一个服务中处理性能上会有一定的提升，尤其当RPC通信传输的数据量较大时。
### 3.4 模型管理与热加载
C++ Serving的引擎支持模型管理功能，支持多种模型和模型不同版本的管理。为了保证在模型更换期间推理服务的可用性，需要在服务不中断的情况下对模型进行热加载。C++ Serving对该特性进行了支持，并提供了一个监控产出模型更新本地模型的工具，具体例子请参考《[C++ Serving中的模型热加载](Hot_Loading_CN.md)》。
### 3.5 模型加解密
C++ Serving采用对称加密算法对模型进行加密，在服务加载模型过程中在内存中解密。目前，提供基础的模型安全能力，并不保证模型绝对安全性，用户可根据我们的设计加以完善，实现更高级别的安全性。说明文档参考《[C++ Serving加密模型预测](Encryption_CN.md)》。
## 4.Client端特性
### 4.1 A/B Test
在对模型进行充分的离线评估后，通常需要进行在线A/B测试，来决定是否大规模上线服务。下图为使用Paddle Serving做A/B测试的基本结构，Client端做好相应的配置后，自动将流量分发给不同的Server，从而完成A/B测试。具体例子请参考《[如何使用Paddle Serving做ABTEST](ABTest_CN.md)》。
<p align="center">
<br>
<img src='../images/abtest.png' width = "345" height = "230">
<br>
<p>
### 4.2 多语言多协议Client
BRPC网络框架支持[多种底层通信协议](#1网络框架(BRPC))，即使用目前的C++ Serving框架的Server端，各种语言的Client端，甚至使用curl的方式，只要按照上述协议（具体支持的协议见[brpc官网](https://github.com/apache/incubator-brpc)）封装数据并发送，Server端就能够接收、处理和返回结果。
对于支持的各种协议我们提供了部分的Client SDK示例供用户参考和使用，用户也可以根据自己的需求去开发新的Client SDK，也欢迎用户添加其他语言/协议（例如GRPC-Go、GRPC-C++ HTTP2-Go、HTTP2-Java等）Client SDK到我们的仓库供其他开发者借鉴和参考。
| 通信协议 | 速度 | 是否支持 | 是否提供Client SDK |
|-------------|-----|---------|-------------------|
| BRPC | 最快 | 支持 | [C++](../../core/general-client/README_CN.md)、[Python(Pybind方式)](../../examples/C++/fit_a_line/README_CN.md) |
| HTTP2+Proto | 快 | 支持 | coming soon |
| GRPC | 快 | 支持 | [Java](../../java/README_CN.md)、[Python](../../examples/C++/fit_a_line/README_CN.md) |
| HTTP1+Proto | 一般 | 支持 | [Java](../../java/README_CN.md)、[Python](../../examples/C++/fit_a_line/README_CN.md) |
| HTTP1+Json | 慢 | 支持 | [Java](../../java/README_CN.md)、[Python](../../examples/C++/fit_a_line/README_CN.md)、[Curl](Http_Service_CN.md) |
doc/C++Serving/Model_Ensemble_CN.md 0 → 100755
浏览文件 @ 56fde8bb
# Paddle Serving中的集成预测
(简体中文|[English](Model_Ensemble_EN.md))
在一些场景中，可能使用多个相同输入的模型并行集成预测以获得更好的预测效果，Paddle Serving提供了这项功能。
下面将以文本分类任务为例，来展示Paddle Serving的集成预测功能（暂时还是串行预测，我们会尽快支持并行化）。
## 集成预测样例
该样例中（见下图），Server端在一项服务中并行预测相同输入的BOW和CNN模型，Client端获取两个模型的预测结果并进行后处理，得到最终的预测结果。
![simple example](../images/model_ensemble_example.png)
需要注意的是，目前只支持在同一个服务中使用多个相同格式输入输出的模型。在该例子中，CNN模型和BOW模型的输入输出格式是相同的。
样例中用到的代码保存在`examples/C++/imdb`路径下：
```shell
.
├── get_data.sh
├── imdb_reader.py
├── test_ensemble_client.py
└── test_ensemble_server.py
```
### 数据准备
通过下面命令获取预训练的CNN和BOW模型（您也可以直接运行`get_data.sh`脚本）：
```shell
wget --no-check-certificate https://fleet.bj.bcebos.com/text_classification_data.tar.gz
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz
tar -zxvf text_classification_data.tar.gz
tar -zxvf imdb_model.tar.gz
```
### 启动Server
通过下面的Python代码启动Server端（您也可以直接运行`test_ensemble_server.py`脚本）：
```python
from paddle_serving_server import OpMaker
from paddle_serving_server import OpGraphMaker
from paddle_serving_server import Server
op_maker = OpMaker()
read_op = op_maker.create('general_reader')
cnn_infer_op = op_maker.create(
'general_infer', engine_name='cnn', inputs=[read_op])
bow_infer_op = op_maker.create(
'general_infer', engine_name='bow', inputs=[read_op])
response_op = op_maker.create(
'general_response', inputs=[cnn_infer_op, bow_infer_op])
op_graph_maker = OpGraphMaker()
op_graph_maker.add_op(read_op)
op_graph_maker.add_op(cnn_infer_op)
op_graph_maker.add_op(bow_infer_op)
op_graph_maker.add_op(response_op)
server = Server()
server.set_op_graph(op_graph_maker.get_op_graph())
model_config = {cnn_infer_op: 'imdb_cnn_model', bow_infer_op: 'imdb_bow_model'}
server.load_model_config(model_config)
server.prepare_server(workdir="work_dir1", port=9393, device="cpu")
server.run_server()
```
与普通预测服务不同的是，这里我们需要用DAG来描述Server端的运行逻辑。
在创建Op的时候需要指定当前Op的前继（在该例子中，`cnn_infer_op`与`bow_infer_op`的前继均是`read_op`，`response_op`的前继是`cnn_infer_op`和`bow_infer_op`），对于预测Op`infer_op`还需要定义预测引擎名称`engine_name`（也可以使用默认值，建议设置该值方便Client端获取预测结果）。
同时在配置模型路径时，需要以预测Op为key，对应的模型路径为value，创建模型配置字典，来告知Serving每个预测Op使用哪个模型。
### 启动Client
通过下面的Python代码运行Client端（您也可以直接运行`test_ensemble_client.py`脚本）：
```python
from paddle_serving_client import Client
from imdb_reader import IMDBDataset
client = Client()
# If you have more than one model, make sure that the input
# and output of more than one model are the same.
client.load_client_config('imdb_bow_client_conf/serving_client_conf.prototxt')
client.connect(["127.0.0.1:9393"])
# you can define any english sentence or dataset here
# This example reuses imdb reader in training, you
# can define your own data preprocessing easily.
imdb_dataset = IMDBDataset()
imdb_dataset.load_resource('imdb.vocab')
for i in range(3):
line = 'i am very sad | 0'
word_ids, label = imdb_dataset.get_words_and_label(line)
feed = {"words": word_ids}
fetch = ["acc", "cost", "prediction"]
fetch_maps = client.predict(feed=feed, fetch=fetch)
if len(fetch_maps) == 1:
print("step: {}, res: {}".format(i, fetch_maps['prediction'][0][1]))
else:
for model, fetch_map in fetch_maps.items():
print("step: {}, model: {}, res: {}".format(i, model, fetch_map[
'prediction'][0][1]))
```
Client端与普通预测服务没有发生太大的变化。当使用多个模型预测时，预测服务将返回一个key为Server端定义的引擎名称`engine_name`，value为对应的模型预测结果的字典。
### 预期结果
```txt
step: 0, model: cnn, res: 0.560272455215
step: 0, model: bow, res: 0.633530199528
step: 1, model: cnn, res: 0.560272455215
step: 1, model: bow, res: 0.633530199528
step: 2, model: cnn, res: 0.560272455215
step: 2, model: bow, res: 0.633530199528
```
doc/C++Serving/Model_Ensemble_EN.md 0 → 100755
浏览文件 @ 56fde8bb
# Model Ensemble in Paddle Serving
([简体中文](Model_Ensemble_CN.md)|English)
In some scenarios, multiple models with the same input may be used to predict in parallel and integrate predicted results for better prediction effect. Paddle Serving also supports this feature.
Next, we will take the text classification task as an example to show model ensemble in Paddle Serving (This feature is still serial prediction for the time being. We will support parallel prediction as soon as possible).
## Simple example
In this example (see the figure below), the server side predict the bow and CNN models with the same input in a service in parallel, The client side fetchs the prediction results of the two models, and processes the prediction results to get the final predict results.
![simple example](../images/model_ensemble_example.png)
It should be noted that at present, only multiple models with the same format input and output in the same service are supported. In this example, the input and output formats of CNN and BOW model are the same.
The code used in the example is saved in the `examples/C++/imdb` path:
```shell
.
├── get_data.sh
├── imdb_reader.py
├── test_ensemble_client.py
└── test_ensemble_server.py
```
### Prepare data
Get the pre-trained CNN and BOW models by the following command (you can also run the `get_data.sh` script):
```shell
wget --no-check-certificate https://fleet.bj.bcebos.com/text_classification_data.tar.gz
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz
tar -zxvf text_classification_data.tar.gz
tar -zxvf imdb_model.tar.gz
```
### Start server
Start server by the following Python code (you can also run the `test_ensemble_server.py` script):
```python
from paddle_serving_server import OpMaker
from paddle_serving_server import OpGraphMaker
from paddle_serving_server import Server
op_maker = OpMaker()
read_op = op_maker.create('general_reader')
cnn_infer_op = op_maker.create(
'general_infer', engine_name='cnn', inputs=[read_op])
bow_infer_op = op_maker.create(
'general_infer', engine_name='bow', inputs=[read_op])
response_op = op_maker.create(
'general_response', inputs=[cnn_infer_op, bow_infer_op])
op_graph_maker = OpGraphMaker()
op_graph_maker.add_op(read_op)
op_graph_maker.add_op(cnn_infer_op)
op_graph_maker.add_op(bow_infer_op)
op_graph_maker.add_op(response_op)
server = Server()
server.set_op_graph(op_graph_maker.get_op_graph())
model_config = {cnn_infer_op: 'imdb_cnn_model', bow_infer_op: 'imdb_bow_model'}
server.load_model_config(model_config)
server.prepare_server(workdir="work_dir1", port=9393, device="cpu")
server.run_server()
```
Different from the normal prediction service, here we need to use DAG to describe the logic of the server side.
When creating an Op, you need to specify the predecessor of the current Op (in this example, the predecessor of `cnn_infer_op` and `bow_infer_op` is `read_op`, and the predecessor of `response_op` is `cnn_infer_op` and `bow_infer_op`. For the infer Op `infer_op`, you need to define the prediction engine name `engine_name` (You can also use the default value. It is recommended to set the value to facilitate the client side to obtain the order of prediction results).
At the same time, when configuring the model path, you need to create a model configuration dictionary with the infer Op as the key and the corresponding model path as value to inform Serving which model each infer OP uses.
### Start client
Start client by the following Python code (you can also run the `test_ensemble_client.py` script):
```python
from paddle_serving_client import Client
from imdb_reader import IMDBDataset
client = Client()
# If you have more than one model, make sure that the input
# and output of more than one model are the same.
client.load_client_config('imdb_bow_client_conf/serving_client_conf.prototxt')
client.connect(["127.0.0.1:9393"])
# you can define any english sentence or dataset here
# This example reuses imdb reader in training, you
# can define your own data preprocessing easily.
imdb_dataset = IMDBDataset()
imdb_dataset.load_resource('imdb.vocab')
for i in range(3):
line = 'i am very sad | 0'
word_ids, label = imdb_dataset.get_words_and_label(line)
feed = {"words": word_ids}
fetch = ["acc", "cost", "prediction"]
fetch_maps = client.predict(feed=feed, fetch=fetch)
if len(fetch_maps) == 1:
print("step: {}, res: {}".format(i, fetch_maps['prediction'][0][1]))
else:
for model, fetch_map in fetch_maps.items():
print("step: {}, model: {}, res: {}".format(i, model, fetch_map[
'prediction'][0][1]))
```
Compared with the normal prediction service, the client side has not changed much. When multiple model predictions are used, the prediction service will return a dictionary with engine name `engine_name`(the value is defined on the server side) as the key, and the corresponding model prediction results as the value.
### Expected result
```shell
step: 0, model: cnn, res: 0.560272455215
step: 0, model: bow, res: 0.633530199528
step: 1, model: cnn, res: 0.560272455215
step: 1, model: bow, res: 0.633530199528
step: 2, model: cnn, res: 0.560272455215
step: 2, model: bow, res: 0.633530199528
```
doc/cpp_server/NEW_OPERATOR_CN.md → doc/C++Serving/OP_CN.md 100644 → 100755
浏览文件 @ 56fde8bb
# 如何开发一个新的General Op?
(简体中文|[English](./NEW_OPERATOR.md))
(简体中文|[English](OP_EN.md))
在本文档中，我们主要集中于如何为Paddle Serving开发新的服务器端运算符。 在开始编写新运算符之前，让我们看一些示例代码以获得为服务器编写新运算符的基本思想。 我们假设您已经知道Paddle Serving服务器端的基本计算逻辑。 下面的代码您可以在 Serving代码库下的 `core/general-server/op` 目录查阅。
......
doc/cpp_server/NEW_OPERATOR.md → doc/C++Serving/OP_EN.md 100644 → 100755
浏览文件 @ 56fde8bb
# How to write an general operator?
([简体中文](./NEW_OPERATOR_CN.md)|English)
([简体中文](OP_CN.md)|English)
In this document, we mainly focus on how to develop a new server side operator for PaddleServing. Before we start to write a new operator, let's look at some sample code to get the basic idea of writing a new operator for server. We assume you have known the basic computation logic on server side of PaddleServing, please reference to []() if you do not know much about it. The following code can be visited at `core/general-server/op` of Serving repo.
......
doc/Model_Zoo_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# Model Zoo
本页面展示了Paddle Serving目前支持的预训练模型以及下载链接
若您想为Paddle Serving提供新的模型，可通过[pull request](https://github.com/PaddlePaddle/Serving/pulls)提交PR
特别感谢[Padddle wholechain](https://www.paddlepaddle.org.cn/wholechain)以及[PaddleHub](https://www.paddlepaddle.org.cn/hub)为Paddle Serving提供的部分预训练模型
| 模型 | 类型 | 部署方式 | 下载 | 服务端 |
| --- | --- | --- | ---- | --- |
| resnet_v2_50_imagenet | PaddleClas | [单模型](../examples/PaddleClas/resnet_v2_50)</br>[多模型](../examples/pipeline/PaddleClas/ResNet_V2_50) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ImageClassification/resnet_v2_50_imagenet.tar.gz) | Pipeline Serving, C++ Serving|
| mobilenet_v2_imagenet | PaddleClas | [单模型](../examples/PaddleClas/mobilenet) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ImageClassification/mobilenet_v2_imagenet.tar.gz) |C++ Serving|
| resnet50_vd | PaddleClas | [单模型](../examples/PaddleClas/imagenet)</br>[多模型](../examples/pipeline/PaddleClas/ResNet50_vd) | [.tar.gz](https://paddle-serving.bj.bcebos.com/ResNet50_vd.tar) |Pipeline Serving, C++ Serving|
| ResNet50_vd_KL | PaddleClas | [多模型](../examples/pipeline/PaddleClas/ResNet50_vd_KL) | [.tar](https://paddle-serving.bj.bcebos.com/model/ResNet50_vd_KL.tar) |Pipeline Serving|
| DarkNet53 | PaddleClas | [多模型](../examples/pipeline/PaddleClas/DarkNet53) | [.tar](https://paddle-serving.bj.bcebos.com/model/DarkNet53.tar) |Pipeline Serving|
| MobileNetV1 | PaddleClas | [多模型](../examples/pipeline/PaddleClas/MobileNetV1) | [.tar](https://paddle-serving.bj.bcebos.com/model/MobileNetV1.tar) |Pipeline Serving|
| MobileNetV2 | PaddleClas | [多模型](../examples/pipeline/PaddleClas/MobileNetV2) | [.tar](https://paddle-serving.bj.bcebos.com/model/MobileNetV2.tar) |Pipeline Serving|
| MobileNetV3_large_x1_0 | PaddleClas | [多模型](../examples/pipeline/PaddleClas/MobileNetV3_large_x1_0) | [.tar](https://paddle-serving.bj.bcebos.com/model/MobileNetV3_large_x1_0.tar) |Pipeline Serving|
| ResNet50_vd_FPGM | PaddleClas | [多模型](../examples/pipeline/PaddleClas/ResNet50_vd_FPGM) | [.tar](https://paddle-serving.bj.bcebos.com/model/ResNet50_vd_FPGM.tar) |Pipeline Serving|
| ResNet50_vd_PACT | PaddleClas | [多模型](../examples/pipeline/PaddleClas/ResNet50_vd_PACT) | [.tar](https://paddle-serving.bj.bcebos.com/model/ResNet50_vd_PACT.tar) |Pipeline Serving|
| ResNeXt101_vd_64x4d | PaddleClas | [多模型](../examples/pipeline/PaddleClas/ResNeXt101_vd_64x4d) | [.tar](https://paddle-serving.bj.bcebos.com/model/ResNeXt101_vd_64x4d.tar) |Pipeline Serving|
| HRNet_W18_C | PaddleClas | [多模型](../examples/pipeline/PaddleClas/HRNet_W18_C) | [.tar](https://paddle-serving.bj.bcebos.com/model/HRNet_W18_C.tar) |Pipeline Serving|
| ShuffleNetV2_x1_0 | PaddleClas | [多模型](../examples/pipeline/PaddleClas/ShuffleNetV2_x1_0) | [.tar](https://paddle-serving.bj.bcebos.com/model/ShuffleNetV2_x1_0.tar) |Pipeline Serving|
| bert_chinese_L-12_H-768_A-12 | PaddleNLP | [单模型](../examples/PaddleNLP/bert)</br>[多模型](../examples/pipeline/bert) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SemanticModel/bert_chinese_L-12_H-768_A-12.tar.gz) |Pipeline Serving, C++ Serving|
| senta_bilstm | PaddleNLP | [单模型](../examples/PaddleNLP/senta) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SentimentAnalysis/senta_bilstm.tar.gz) |C++ Serving|
| lac | PaddleNLP | [单模型](../examples/PaddleNLP/lac) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/LexicalAnalysis/lac.tar.gz) | C++ Serving|
| transformer | PaddleNLP | [多模型](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/examples/machine_translation/transformer/deploy/serving/README.md) | [model](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/machine_translation/transformer) | Pipeline Serving|
| criteo_ctr | PaddleRec | [单模型](../examples/PaddleRec/criteo_ctr) | [.tar.gz](https://paddle-serving.bj.bcebos.com/criteo_ctr_example/criteo_ctr_demo_model.tar.gz) | C++ Serving |
| criteo_ctr_with_cube | PaddleRec | [单模型](../examples/PaddleRec/criteo_ctr_with_cube) | [.tar.gz](https://paddle-serving.bj.bcebos.com/unittest/ctr_cube_unittest.tar.gz) |C++ Serving|
| wide&deep | PaddleRec | [单模型](https://github.com/PaddlePaddle/PaddleRec/blob/release/2.1.0/doc/serving.md) | [model](https://github.com/PaddlePaddle/PaddleRec/blob/release/2.1.0/models/rank/wide_deep/README.md) |C++ Serving|
| blazeface | PaddleDetection | [单模型](../examples/PaddleDetection/blazeface) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ObjectDetection/blazeface.tar.gz) |C++ Serving|
| cascade_mask_rcnn_r50_vd_fpn_ssld_2x_coco | PaddleDetection | [单模型](../examples/PaddleDetection/cascade_rcnn) | [.tar.gz](https://paddle-serving.bj.bcebos.com/pddet_demo/cascade_mask_rcnn_r50_vd_fpn_ssld_2x_coco_serving.tar.gz) |C++ Serving|
| yolov4 | PaddleDetection | [单模型](../examples/PaddleDetection/yolov4) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ObjectDetection/yolov4.tar.gz) |C++ Serving|
| faster_rcnn_hrnetv2p_w18_1x | PaddleDetection | [单模型](../examples/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x) | [.tar.gz](https://paddle-serving.bj.bcebos.com/pddet_demo/faster_rcnn_hrnetv2p_w18_1x.tar.gz) |C++ Serving|
| fcos_dcn_r50_fpn_1x_coco | PaddleDetection | [单模型](../examples/PaddleDetection/fcos_dcn_r50_fpn_1x_coco) | [.tar.gz](https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/fcos_dcn_r50_fpn_1x_coco.tar) |C++ Serving|
| ssd_vgg16_300_240e_voc | PaddleDetection | [单模型](../examples/PaddleDetection/ssd_vgg16_300_240e_voc) | [.tar](https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ssd_vgg16_300_240e_voc.tar) |C++ Serving |
| yolov3_darknet53_270e_coco | PaddleDetection | [单模型](../examples/PaddleDetection/yolov3_darknet53_270e_coco)</br>[多模型](../examples/pipeline/PaddleDetection/yolov3) | [.tar](https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/yolov3_darknet53_270e_coco.tar) |Pipeline Serving, C++ Serving |
| faster_rcnn_r50_fpn_1x_coco | PaddleDetection | [单模型](../examples/PaddleDetection/faster_rcnn_r50_fpn_1x_coco)</br>[多模型](../examples/pipeline/PaddleDetection/faster_rcnn) | [.tar](https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/faster_rcnn_r50_fpn_1x_coco.tar) |Pipeline Serving, C++ Serving |
| ppyolo_r50vd_dcn_1x_coco | PaddleDetection | [单模型](../examples/PaddleDetection/ppyolo_r50vd_dcn_1x_coco) | [.tar](https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_r50vd_dcn_1x_coco.tar) |C++ Serving |
| ppyolo_mbv3_large_coco | PaddleDetection | [多模型](../examples/pipeline/PaddleDetection/ppyolo_mbv3) | [.tar](https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_mbv3_large_coco.tar) |Pipeline Serving |
| ttfnet_darknet53_1x_coco | PaddleDetection | [单模型](../examples/PaddleDetection/ttfnet_darknet53_1x_coco) | [.tar](https://paddle-serving.bj.bcebos.com/pddet_demo/ttfnet_darknet53_1x_coco.tar) |C++ Serving |
| YOLOv3-DarkNet | PaddleDetection | [单模型](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.3/deploy/serving) | [.pdparams](https://paddledet.bj.bcebos.com/models/yolov3_darknet53_270e_coco.pdparams)</br>[.yml](https://github.com/PaddlePaddle/PaddleDetection/blob/develop/configs/yolov3/yolov3_darknet53_270e_coco.yml) |C++ Serving |
| ocr_rec | PaddleOCR | [单模型](../examples/PaddleOCR/ocr_rec_det)</br>[多模型](../examples/pipeline/ocr) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/OCR/ocr_rec.tar.gz) |Pipeline Serving, C++ Serving |
| ocr_det | PaddleOCR | [单模型](../examples/PaddleOCR/ocr_rec_det)</br>[多模型](../examples/pipeline/ocr) | [.tar.gz](https://paddle-serving.bj.bcebos.com/ocr/ocr_det.tar.gz) |Pipeline Serving, C++ Serving |
| ch_ppocr_mobile_v2.0_det | PaddleOCR | [多模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/deploy/pdserving/README.md) | [model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar)</br>[.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml) |Pipeline Serving |
| ch_ppocr_server_v2.0_det | PaddleOCR | [多模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/deploy/pdserving/README.md) | [model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_infer.tar)</br>[.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/configs/det/ch_ppocr_v2.0/ch_det_res18_db_v2.0.yml) |Pipeline Serving |
| ch_ppocr_mobile_v2.0_rec | PaddleOCR | [多模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/deploy/pdserving/README.md) | [model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar)</br>[.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml) |Pipeline Serving |
| ch_ppocr_server_v2.0_rec | PaddleOCR | [多模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/deploy/pdserving/README.md) | [model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_infer.tar)</br>[.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/configs/rec/ch_ppocr_v2.0/rec_chinese_common_train_v2.0.yml) |Pipeline Serving |
| ch_ppocr_mobile_v2.0 | PaddleOCR | [多模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/deploy/pdserving/README.md) | [model](https://github.com/PaddlePaddle/PaddleOCR) |Pipeline Serving |
| ch_ppocr_server_v2.0 | PaddleOCR | [多模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/deploy/pdserving/README.md) | [model](https://github.com/PaddlePaddle/PaddleOCR) |Pipeline Serving |
| deeplabv3 | PaddleSeg | [单模型](../examples/PaddleSeg/deeplabv3) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ImageSegmentation/deeplabv3.tar.gz) | C++ Serving |
| unet | PaddleSeg | [单模型](../examples/PaddleSeg/unet_for_image_seg) | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ImageSegmentation/unet.tar.gz) |C++ Serving |
- 注意事项
- 多模型部署示例均在pipeline文件夹下
- 单模型采用C++ Serving，多模型采用Pipeline Serving
- 请参考 [example](../examples) 查看详情
- 更多模型请参考[wholechain](https://www.paddlepaddle.org.cn/wholechain)
doc/cpp_server/C++DESIGN.md 已删除 100644 → 0
浏览文件 @ 32ccb13f
# C++ Serving Design
([简体中文](./C++DESIGN_CN.md)|English)
## 1. Background
PaddlePaddle is the Baidu's open source machine learning framework, which supports a wide range of customized development of deep learning models; Paddle serving is the online prediction framework of Paddle, which seamlessly connects with Paddle model training, and provides cloud services for machine learning prediction. This article will describe the Paddle Serving design from the bottom up, from the model, service, and access levels.
1. The model is the core of Paddle Serving prediction, including the management of model data and inference calculations;
2. Prediction framework encapsulation model for inference calculations, providing external RPC interface to connect different upstream
3. The prediction service SDK provides a set of access frameworks
The result is a complete serving solution.
## 2. Terms explanation
- **baidu-rpc**: Baidu's official open source RPC framework, supports multiple common communication protocols, and provides a custom interface experience based on protobuf
- **Variant**: Paddle Serving architecture is an abstraction of a minimal prediction cluster, which is characterized by all internal instances (replicas) being completely homogeneous and logically corresponding to a fixed version of a model
- **Endpoint**: Multiple Variants form an Endpoint. Logically, Endpoint represents a model, and Variants within the Endpoint represent different versions.
- **OP**: PaddlePaddle is used to encapsulate a numerical calculation operator, Paddle Serving is used to represent a basic business operation operator, and the core interface is inference. OP configures its dependent upstream OP to connect multiple OPs into a workflow
- **Channel**: An abstraction of all request-level intermediate data of the OP; data exchange between OPs through Channels
- **Bus**: manages all channels in a thread, and schedules the access relationship between the two sets of OP and Channel according to the DAG dependency graph between DAGs
- **Stage**: Workflow according to the topology diagram described by DAG, a collection of OPs that belong to the same link and can be executed in parallel
- **Node**: An OP operator instance composed of an OP operator class combined with parameter configuration, which is also an execution unit in Workflow
- **Workflow**: executes the inference interface of each OP in order according to the topology described by DAG
- **DAG/Workflow**: consists of several interdependent Nodes. Each Node can obtain the Request object through a specific interface. The node Op obtains the output object of its pre-op through the dependency relationship. The output of the last Node is the Response object by default.
- **Service**: encapsulates a pv request, can configure several Workflows, reuse the current PV's Request object with each other, and then execute each in parallel/serial execution, and finally write the Response to the corresponding output slot; a Paddle-serving process Multiple sets of Service interfaces can be configured. The upstream determines the Service interface currently accessed based on the ServiceName.
## 3. Python Interface Design
### 3.1 Core Targets:
A set of Paddle Serving dynamic library, support the remote estimation service of the common model saved by Paddle, and call the various underlying functions of PaddleServing through the Python Interface.
### 3.2 General Model:
Models that can be predicted using the Paddle Inference Library, models saved during training, including Feed Variable and Fetch Variable
### 3.3 Overall design:
- The user starts the Client and Server through the Python Client. The Python API has a function to check whether the interconnection and the models to be accessed match.
- The Python API calls the pybind corresponding to the client and server functions implemented by Paddle Serving, and the information transmitted through RPC is implemented through RPC.
- The Client Python API currently has two simple functions, load_inference_conf and predict, which are used to perform loading of the model to be predicted and prediction, respectively.
- The Server Python API is mainly responsible for loading the inference model and generating various configurations required by Paddle Serving, including engines, workflow, resources, etc.
### 3.4 Server Inferface
![Server Interface](images/server_interface.png)
### 3.5 Client Interface
<img src='images/client_inferface.png' width = "600" height = "200">
### 3.6 Client io used during Training
PaddleServing is designed to saves the model interface that can be used during the training process, which is basically the same as the Paddle save inference model interface, feed_var_dict and fetch_var_dict
You can alias the input and output variables. The configuration that needs to be read when the serving starts is saved in the client and server storage directories.
``` python
def save_model(server_model_folder,
client_config_folder,
feed_var_dict,
fetch_var_dict,
main_program=None)
```
## 4. Paddle Serving Underlying Framework
![Paddle-Serging Overall Architecture](images/framework.png)
**Model Management Framework**: Connects model files of multiple machine learning platforms and provides a unified inference interface
**Business Scheduling Framework**: Abstracts the calculation logic of various different inference models, provides a general DAG scheduling framework, and connects different operators through DAG diagrams to complete a prediction service together. This abstract model allows users to conveniently implement their own calculation logic, and at the same time facilitates operator sharing. (Users build their own forecasting services. A large part of their work is to build DAGs and provide operators.)
**Predict Service**: Encapsulation of the externally provided prediction service interface. Define communication fields with the client through protobuf.
### 4.1 Model Management Framework
The model management framework is responsible for managing the models trained by the machine learning framework. It can be abstracted into three levels: model loading, model data, and model reasoning.
#### Model Loading
Load model from disk to memory, support multi-version, hot-load, incremental update, etc.
#### Model data
Model data structure in memory, integrated fluid inference lib
#### inferencer
it provided united inference interface for upper layers
```C++
class FluidFamilyCore {
virtual bool Run(const void* in_data, void* out_data);
virtual int create(const std::string& data_path);
virtual int clone(void* origin_core);
};
```
### 4.2 Business Scheduling Framework
#### 4.2.1 Inference Service
With reference to the abstract idea of model calculation of the TensorFlow framework, the business logic is abstracted into a DAG diagram, driven by configuration, generating a workflow, and skipping C ++ code compilation. Each specific step of the service corresponds to a specific OP. The OP can configure the upstream OP that it depends on. Unified message passing between OPs is achieved by the thread-level bus and channel mechanisms. For example, the service process of a simple prediction service can be abstracted into 3 steps including reading request data-> calling the prediction interface-> writing back the prediction result, and correspondingly implemented to 3 OP: ReaderOp-> ClassifyOp-> WriteOp
![Infer Service](images/predict-service.png)
Regarding the dependencies between OPs, and the establishment of workflows through OPs, you can refer to [从零开始写一个预测服务](CREATING.md) (simplified Chinese Version)
Server instance perspective
![Server instance perspective](images/server-side.png)
#### 4.2.2 Paddle Serving Multi-Service Mechanism
![Paddle Serving multi-service](images/multi-service.png)
Paddle Serving instances can load multiple models at the same time, and each model uses a Service (and its configured workflow) to undertake services. You can refer to [service configuration file in Demo example](../tools/cpp_examples/demo-serving/conf/service.prototxt) to learn how to configure multiple services for the serving instance
#### 4.2.3 Hierarchical relationship of business scheduling
From the client's perspective, a Paddle Serving service can be divided into three levels: Service, Endpoint, and Variant from top to bottom.
![Call hierarchy relationship](images/multi-variants.png)
One Service corresponds to one inference model, and there is one endpoint under the model. Different versions of the model are implemented through multiple variant concepts under endpoint:
The same model prediction service can configure multiple variants, and each variant has its own downstream IP list. The client code can configure relative weights for each variant to achieve the relationship of adjusting the traffic ratio (refer to the description of variant_weight_list in [Client Configuration](CLIENT_CONFIGURE.md) section 3.2).
![Client-side proxy function](images/client-side-proxy.png)
## 5. User Interface
Under the premise of meeting certain interface specifications, the service framework does not make any restrictions on user data fields to meet different business interfaces of various forecast services. Baidu-rpc inherits the interface of Protobuf serice, and the user describes the Request and Response business interfaces according to the Protobuf syntax specification. Paddle Serving is built on the Baidu-rpc framework and supports this feature by default.
No matter how the communication protocol changes, the framework only needs to ensure that the communication protocol between the client and server and the format of the business data are synchronized to ensure normal communication. This information can be broken down as follows:
-Protocol: Header information agreed in advance between Server and Client to ensure mutual recognition of data format. Paddle Serving uses Protobuf as the basic communication format
-Data: Used to describe the interface of Request and Response, such as the sample data to be predicted, and the score returned by the prediction. include:
   -Data fields: Field definitions included in the two data structures of Request and Return.
   -Description interface: similar to the protocol interface, it supports Protobuf by default
### 5.1 Data Compression Method
Baidu-rpc has built-in data compression methods such as snappy, gzip, zlib, which can be configured in the configuration file (refer to [Client Configuration](CLIENT_CONFIGURE.md) Section 3.1 for an introduction to compress_type)
### 5.2 C ++ SDK API Interface
```C++
class PredictorApi {
public:
int create(const char* path, const char* file);
int thrd_initialize();
int thrd_clear();
int thrd_finalize();
void destroy();
Predictor* fetch_predictor(std::string ep_name);
int free_predictor(Predictor* predictor);
};
class Predictor {
public:
// synchronize interface
virtual int inference(google::protobuf::Message* req,
google::protobuf::Message* res) = 0;
// asynchronize interface
virtual int inference(google::protobuf::Message* req,
google::protobuf::Message* res,
DoneType done,
brpc::CallId* cid = NULL) = 0;
// synchronize interface
virtual int debug(google::protobuf::Message* req,
google::protobuf::Message* res,
butil::IOBufBuilder* debug_os) = 0;
};
```
### 5.3 Inferfaces related to Op
```C++
class Op {
// ------Getters for Channel/Data/Message of dependent OP-----
// Get the Channel object of dependent OP
Channel* mutable_depend_channel(const std::string& op);
// Get the Channel object of dependent OP
const Channel* get_depend_channel(const std::string& op) const;
template <typename T>
T* mutable_depend_argument(const std::string& op);
template <typename T>
const T* get_depend_argument(const std::string& op) const;
// -----Getters for Channel/Data/Message of current OP----
// Get pointer to the progobuf message of current OP
google::protobuf::Message* mutable_message();
// Get pointer to the protobuf message of current OP
const google::protobuf::Message* get_message() const;
// Get the template class data object of current OP
template <typename T>
T* mutable_data();
// Get the template class data object of current OP
template <typename T>
const T* get_data() const;
// ---------------- Other base class members ----------------
int init(Bus* bus,
Dag* dag,
uint32_t id,
const std::string& name,
const std::string& type,
void* conf);
int deinit();
int process(bool debug);
// Get the input object
const google::protobuf::Message* get_request_message();
const std::string& type() const;
uint32_t id() const;
// ------------------ OP Interface -------------------
// Get the derived Channel object of current OP
virtual Channel* mutable_channel() = 0;
// Get the derived Channel object of current OP
virtual const Channel* get_channel() const = 0;
// Release the derived Channel object of current OP
virtual int release_channel() = 0;
// Inference interface
virtual int inference() = 0;
// ------------------ Conf Interface -------------------
virtual void* create_config(const configure::DAGNode& conf) { return NULL; }
virtual void delete_config(void* conf) {}
virtual void set_config(void* conf) { return; }
// ------------------ Metric Interface -------------------
virtual void regist_metric() { return; }
};
```
### 5.4 Interfaces related to framework
Service
```C++
class InferService {
public:
static const char* tag() { return "service"; }
int init(const configure::InferService& conf);
int deinit() { return 0; }
int reload();
const std::string& name() const;
const std::string& full_name() const { return _infer_service_format; }
// Execute each workflow serially
virtual int inference(const google::protobuf::Message* request,
google::protobuf::Message* response,
butil::IOBufBuilder* debug_os = NULL);
int debug(const google::protobuf::Message* request,
google::protobuf::Message* response,
butil::IOBufBuilder* debug_os);
};
class ParallelInferService : public InferService {
public:
// Execute workflows in parallel
int inference(const google::protobuf::Message* request,
google::protobuf::Message* response,
butil::IOBufBuilder* debug_os) {
return 0;
}
};
```
ServerManager
```C++
class ServerManager {
public:
typedef google::protobuf::Service Service;
ServerManager();
static ServerManager& instance() {
static ServerManager server;
return server;
}
static bool reload_starting() { return _s_reload_starting; }
static void stop_reloader() { _s_reload_starting = false; }
int add_service_by_format(const std::string& format);
int start_and_wait();
};
```
DAG
```C++
class Dag {
public:
EdgeMode parse_mode(std::string& mode); // NOLINT
int init(const char* path, const char* file, const std::string& name);
int init(const configure::Workflow& conf, const std::string& name);
int deinit();
uint32_t nodes_size();
const DagNode* node_by_id(uint32_t id);
const DagNode* node_by_id(uint32_t id) const;
const DagNode* node_by_name(std::string& name); // NOLINT
const DagNode* node_by_name(const std::string& name) const;
uint32_t stage_size();
const DagStage* stage_by_index(uint32_t index);
const std::string& name() const { return _dag_name; }
const std::string& full_name() const { return _dag_name; }
void regist_metric(const std::string& service_name);
};
```
Workflow
```C++
class Workflow {
public:
Workflow() {}
static const char* tag() { return "workflow"; }
// Each workflow object corresponds to an independent
// configure file, so you can share the object between
// different apps.
int init(const configure::Workflow& conf);
DagView* fetch_dag_view(const std::string& service_name);
int deinit() { return 0; }
void return_dag_view(DagView* view);
int reload();
const std::string& name() { return _name; }
const std::string& full_name() { return _name; }
};
```
doc/cpp_server/C++DESIGN_CN.md 已删除 100644 → 0
浏览文件 @ 32ccb13f
# C++ Serving设计方案
(简体中文|[English](./C++DESIGN.md))
注意本页内容有已经过期，请查看：[设计文档](DESIGN_DOC_CN.md)
## 1. 项目背景
PaddlePaddle是百度开源的机器学习框架，广泛支持各种深度学习模型的定制化开发; Paddle Serving是Paddle的在线预测部分，与Paddle模型训练环节无缝衔接，提供机器学习预测云服务。本文将从模型、服务、接入等层面，自底向上描述Paddle Serving设计方案。
1. 模型是Paddle Serving预测的核心，包括模型数据和推理计算的管理；
2. 预测框架封装模型推理计算，对外提供RPC接口，对接不同上游；
3. 预测服务SDK提供一套接入框架
最终形成一套完整的serving解决方案。
## 2. 名词解释
- **baidu-rpc**: 百度官方开源RPC框架，支持多种常见通信协议，提供基于protobuf的自定义接口体验
- **Variant**: Paddle Serving架构对一个最小预测集群的抽象，其特点是内部所有实例（副本）完全同质，逻辑上对应一个model的一个固定版本
- **Endpoint**: 多个Variant组成一个Endpoint，逻辑上看，Endpoint代表一个model，Endpoint内部的Variant代表不同的版本
- **OP**: PaddlePaddle用来封装一种数值计算的算子，Paddle Serving用来表示一种基础的业务操作算子，核心接口是inference。OP通过配置其依赖的上游OP，将多个OP串联成一个workflow
- **Channel**: 一个OP所有请求级中间数据的抽象；OP之间通过Channel进行数据交互
- **Bus**: 对一个线程中所有channel的管理，以及根据DAG之间的DAG依赖图对OP和Channel两个集合间的访问关系进行调度
- **Stage**: Workflow按照DAG描述的拓扑图中，属于同一个环节且可并行执行的OP集合
- **Node**: 由某个OP算子类结合参数配置组成的OP算子实例，也是Workflow中的一个执行单元
- **Workflow**: 按照DAG描述的拓扑，有序执行每个OP的inference接口
- **DAG/Workflow**: 由若干个相互依赖的Node组成，每个Node均可通过特定接口获得Request对象，节点OP通过依赖关系获得其前置OP的输出对象，最后一个Node的输出默认就是Response对象
- **Service**: 对一次PV的请求封装，可配置若干条Workflow，彼此之间复用当前PV的Request对象，然后各自并行/串行执行，最后将Response写入对应的输出slot中；一个Paddle-serving进程可配置多套Service接口，上游根据ServiceName决定当前访问的Service接口。
## 3. Python Interface设计
### 3.1 核心目标：
完成一整套Paddle Serving的动态库，支持Paddle保存的通用模型的远程预估服务，通过Python Interface调用PaddleServing底层的各种功能。
### 3.2 通用模型：
能够使用Paddle Inference Library进行预测的模型，在训练过程中保存的模型，包含Feed Variable和Fetch Variable
### 3.3 整体设计：
- 用户通过Python Client启动Client和Server，Python API有检查互联和待访问模型是否匹配的功能
- Python API背后调用的是Paddle Serving实现的client和server对应功能的pybind，互传的信息通过RPC实现
- Client Python API当前有两个简单的功能，load_inference_conf和predict，分别用来执行加载待预测的模型和预测
- Server Python API主要负责加载预估模型，以及生成Paddle Serving需要的各种配置，包括engines，workflow，resource等
### 3.4 Server Inferface
![Server Interface](images/server_interface.png)
### 3.5 Client Interface
<img src='images/client_inferface.png' width = "600" height = "200">
### 3.6 训练过程中使用的Client io
PaddleServing设计可以在训练过程中使用的保存模型接口，与Paddle保存inference model的接口基本一致，feed_var_dict与fetch_var_dict
可以为输入和输出变量起别名，serving启动需要读取的配置会保存在client端和server端的保存目录中。
``` python
def save_model(server_model_folder,
client_config_folder,
feed_var_dict,
fetch_var_dict,
main_program=None)
```
## 4. Paddle Serving底层框架
![Paddle-Serging总体框图](images/framework.png)
**模型管理框架**：对接多种机器学习平台的模型文件，向上提供统一的inference接口
**业务调度框架**：对各种不同预测模型的计算逻辑进行抽象，提供通用的DAG调度框架，通过DAG图串联不同的算子，共同完成一次预测服务。该抽象模型使用户可以方便的实现自己的计算逻辑，同时便于算子共用。（用户搭建自己的预测服务，很大一部分工作是搭建DAG和提供算子的实现）
**PredictService**:对外部提供的预测服务接口封装。通过protobuf定义与客户端的通信字段。
### 4.1 模型管理框架
模型管理框架负责管理机器学习框架训练出来的模型，总体可抽象成模型加载、模型数据和模型推理等3个层次。
#### 模型加载
将模型从磁盘加载到内存，支持多版本、热加载、增量更新等功能
#### 模型数据
模型在内存中的数据结构，集成fluid预测lib
#### inferencer
向上为预测服务提供统一的预测接口
```C++
class FluidFamilyCore {
virtual bool Run(const void* in_data, void* out_data);
virtual int create(const std::string& data_path);
virtual int clone(void* origin_core);
};
```
### 4.2 业务调度框架
#### 4.2.1 预测服务Service
参考TF框架的模型计算的抽象思想，将业务逻辑抽象成DAG图，由配置驱动，生成workflow，跳过C++代码编译。业务的每个具体步骤，对应一个具体的OP，OP可配置自己依赖的上游OP。OP之间消息传递统一由线程级Bus和channel机制实现。例如，一个简单的预测服务的服务过程，可以抽象成读请求数据->调用预测接口->写回预测结果等3个步骤，相应的实现到3个OP: ReaderOp->ClassifyOp->WriteOp
![预测服务Service](images/predict-service.png)
关于OP之间的依赖关系，以及通过OP组建workflow，可以参考[从零开始写一个预测服务](CREATING.md)的相关章节
服务端实例透视图
![服务端实例透视图](images/server-side.png)
#### 4.2.2 Paddle Serving的多服务机制
![Paddle Serving的多服务机制](images/multi-service.png)
Paddle Serving实例可以同时加载多个模型，每个模型用一个Service（以及其所配置的workflow）承接服务。可以参考[Demo例子中的service配置文件](../tools/cpp_examples/demo-serving/conf/service.prototxt)了解如何为serving实例配置多个service
#### 4.2.3 业务调度层级关系
从客户端看，一个Paddle Serving service从顶向下可分为Service, Endpoint, Variant等3个层级
![调用层级关系](images/multi-variants.png)
一个Service对应一个预测模型，模型下有1个endpoint。模型的不同版本，通过endpoint下多个variant概念实现：
同一个模型预测服务，可以配置多个variant，每个variant有自己的下游IP列表。客户端代码可以对各个variant配置相对权重，以达到调节流量比例的关系（参考[客户端配置](CLIENT_CONFIGURE.md)第3.2节中关于variant_weight_list的说明）。
![Client端proxy功能](images/client-side-proxy.png)
## 5. 用户接口
在满足一定的接口规范前提下，服务框架不对用户数据字段做任何约束，以应对各种预测服务的不同业务接口。Baidu-rpc继承了Protobuf serice的接口，用户按照Protobuf语法规范描述Request和Response业务接口。Paddle Serving基于Baidu-rpc框架搭建，默认支持该特性。
无论通信协议如何变化，框架只需确保Client和Server间通信协议和业务数据两种信息的格式同步，即可保证正常通信。这些信息又可细分如下：
- 协议：Server和Client之间事先约定的、确保相互识别数据格式的包头信息。Paddle Serving用Protobuf作为基础通信格式
- 数据：用来描述Request和Response的接口，例如待预测样本数据，和预测返回的打分。包括：
- 数据字段：请求包Request和返回包Response两种数据结构包含的字段定义
- 描述接口：跟协议接口类似，默认支持Protobuf
### 5.1 数据压缩方法
Baidu-rpc内置了snappy, gzip, zlib等数据压缩方法，可在配置文件中配置（参考[客户端配置](CLIENT_CONFIGURE.md)第3.1节关于compress_type的介绍）
### 5.2 C++ SDK API接口
```C++
class PredictorApi {
public:
int create(const char* path, const char* file);
int thrd_initialize();
int thrd_clear();
int thrd_finalize();
void destroy();
Predictor* fetch_predictor(std::string ep_name);
int free_predictor(Predictor* predictor);
};
class Predictor {
public:
// synchronize interface
virtual int inference(google::protobuf::Message* req,
google::protobuf::Message* res) = 0;
// asynchronize interface
virtual int inference(google::protobuf::Message* req,
google::protobuf::Message* res,
DoneType done,
brpc::CallId* cid = NULL) = 0;
// synchronize interface
virtual int debug(google::protobuf::Message* req,
google::protobuf::Message* res,
butil::IOBufBuilder* debug_os) = 0;
};
```
### 5.3 OP相关接口
```C++
class Op {
// ------Getters for Channel/Data/Message of dependent OP-----
// Get the Channel object of dependent OP
Channel* mutable_depend_channel(const std::string& op);
// Get the Channel object of dependent OP
const Channel* get_depend_channel(const std::string& op) const;
template <typename T>
T* mutable_depend_argument(const std::string& op);
template <typename T>
const T* get_depend_argument(const std::string& op) const;
// -----Getters for Channel/Data/Message of current OP----
// Get pointer to the progobuf message of current OP
google::protobuf::Message* mutable_message();
// Get pointer to the protobuf message of current OP
const google::protobuf::Message* get_message() const;
// Get the template class data object of current OP
template <typename T>
T* mutable_data();
// Get the template class data object of current OP
template <typename T>
const T* get_data() const;
// ---------------- Other base class members ----------------
int init(Bus* bus,
Dag* dag,
uint32_t id,
const std::string& name,
const std::string& type,
void* conf);
int deinit();
int process(bool debug);
// Get the input object
const google::protobuf::Message* get_request_message();
const std::string& type() const;
uint32_t id() const;
// ------------------ OP Interface -------------------
// Get the derived Channel object of current OP
virtual Channel* mutable_channel() = 0;
// Get the derived Channel object of current OP
virtual const Channel* get_channel() const = 0;
// Release the derived Channel object of current OP
virtual int release_channel() = 0;
// Inference interface
virtual int inference() = 0;
// ------------------ Conf Interface -------------------
virtual void* create_config(const configure::DAGNode& conf) { return NULL; }
virtual void delete_config(void* conf) {}
virtual void set_config(void* conf) { return; }
// ------------------ Metric Interface -------------------
virtual void regist_metric() { return; }
};
```
### 5.4 框架相关接口
Service
```C++
class InferService {
public:
static const char* tag() { return "service"; }
int init(const configure::InferService& conf);
int deinit() { return 0; }
int reload();
const std::string& name() const;
const std::string& full_name() const { return _infer_service_format; }
// Execute each workflow serially
virtual int inference(const google::protobuf::Message* request,
google::protobuf::Message* response,
butil::IOBufBuilder* debug_os = NULL);
int debug(const google::protobuf::Message* request,
google::protobuf::Message* response,
butil::IOBufBuilder* debug_os);
};
class ParallelInferService : public InferService {
public:
// Execute workflows in parallel
int inference(const google::protobuf::Message* request,
google::protobuf::Message* response,
butil::IOBufBuilder* debug_os) {
return 0;
}
};
```
ServerManager
```C++
class ServerManager {
public:
typedef google::protobuf::Service Service;
ServerManager();
static ServerManager& instance() {
static ServerManager server;
return server;
}
static bool reload_starting() { return _s_reload_starting; }
static void stop_reloader() { _s_reload_starting = false; }
int add_service_by_format(const std::string& format);
int start_and_wait();
};
```
DAG
```C++
class Dag {
public:
EdgeMode parse_mode(std::string& mode); // NOLINT
int init(const char* path, const char* file, const std::string& name);
int init(const configure::Workflow& conf, const std::string& name);
int deinit();
uint32_t nodes_size();
const DagNode* node_by_id(uint32_t id);
const DagNode* node_by_id(uint32_t id) const;
const DagNode* node_by_name(std::string& name); // NOLINT
const DagNode* node_by_name(const std::string& name) const;
uint32_t stage_size();
const DagStage* stage_by_index(uint32_t index);
const std::string& name() const { return _dag_name; }
const std::string& full_name() const { return _dag_name; }
void regist_metric(const std::string& service_name);
};
```
Workflow
```C++
class Workflow {
public:
Workflow() {}
static const char* tag() { return "workflow"; }
// Each workflow object corresponds to an independent
// configure file, so you can share the object between
// different apps.
int init(const configure::Workflow& conf);
DagView* fetch_dag_view(const std::string& service_name);
int deinit() { return 0; }
void return_dag_view(DagView* view);
int reload();
const std::string& name() { return _name; }
const std::string& full_name() { return _name; }
};
```
doc/cpp_server/NEW_WEB_SERVICE.md 已删除 100644 → 0
浏览文件 @ 32ccb13f
# How to develop a new Web service?
([简体中文](NEW_WEB_SERVICE_CN.md)|English)
This document will take Uci service as an example to introduce how to develop a new Web Service. You can check out the complete code [here](../python/examples/pipeline/simple_web_service/web_service.py).
## Op base class
In some services, a single model may not meet business needs, requiring multiple models to be concatenated or parallel to complete the entire service. We call a single model operation Op and provide a simple set of interfaces to implement the complex logic of Op concatenation or parallelism.
Data between Ops is passed as a dictionary, Op can be started as threads or process, and Op can be configured for the number of concurrencies, etc.
Typically, you need to inherit the Op base class and override its `init_op`, `preprocess` and `postprocess` methods, which are implemented by default as follows:
```python
class Op(object):
def init_op(self):
pass
def preprocess(self, input_dicts):
# multiple previous Op
if len(input_dicts) != 1:
_LOGGER.critical(
"Failed to run preprocess: this Op has multiple previous "
"inputs. Please override this func.")
os._exit(-1)
(_, input_dict), = input_dicts.items()
return input_dict
def postprocess(self, input_dicts, fetch_dict):
return fetch_dict
```
### init_op
This method is used to load user-defined resources such as dictionaries. A separator is loaded in the [UciOp](../python/examples/pipeline/simple_web_service/web_service.py).
**Note**: If Op is launched in threaded mode, different threads of the same Op execute `init_op` only once and share `init_op` loaded resources when Op is multi-concurrent.
### preprocess
This method is used to preprocess the data before model prediction. It has an `input_dicts` parameter, `input_dicts` is a dictionary, key is the `name` of the previous Op, and value is the data transferred from the corresponding previous op (the data is also in dictionary format).
The `preprocess` method needs to process the data into a ndarray dictionary (key is the feed variable name, and value is the corresponding ndarray value). Op will take the return value as the input of the model prediction and pass the output to the `postprocess` method.
**Note**: if Op does not have a model configuration file, the return value of `preprocess` will be directly passed to `postprocess`.
### postprocess
This method is used for data post-processing after model prediction. It has two parameters, `input_dicts` and `fetch_dict`.
Where the `input_dicts` parameter is consistent with the parameter in `preprocess` method, and `fetch_dict` is the output of the model prediction (key is the name of the fetch variable, and value is the corresponding ndarray value). Op will take the return value of `postprocess` as the input of subsequent Op `preprocess`.
**Note**: if Op does not have a model configuration file, `fetch_dict` will be the return value of `preprocess`.
Here is the op of the UCI example:
```python
class UciOp(Op):
def init_op(self):
self.separator = ","
def preprocess(self, input_dicts):
(_, input_dict), = input_dicts.items()
x_value = input_dict["x"]
if isinstance(x_value, (str, unicode)):
input_dict["x"] = np.array(
[float(x.strip()) for x in x_value.split(self.separator)])
return input_dict
def postprocess(self, input_dicts, fetch_dict):
fetch_dict["price"] = str(fetch_dict["price"][0][0])
return fetch_dict
```
## WebService base class
Paddle Serving implements the [WebService](https://github.com/PaddlePaddle/Serving/blob/develop/python/paddle_serving_server/web_service.py#L23) base class. You need to override its `get_pipeline_response` method to define the topological relationship between Ops. The default implementation is as follows:
```python
class WebService(object):
def get_pipeline_response(self, read_op):
return None
```
Where `read_op` serves as the entry point of the topology map of the whole service (that is, the first op defined by the user is followed by `read_op`).
For single Op service (single model), take Uci service as an example (there is only one Uci prediction model in the whole service):
```python
class UciService(WebService):
def get_pipeline_response(self, read_op):
uci_op = UciOp(name="uci", input_ops=[read_op])
return uci_op
```
For multiple Op services (multiple models), take Ocr service as an example (the whole service is completed in series by Det model and Rec model):
```python
class OcrService(WebService):
def get_pipeline_response(self, read_op):
det_op = DetOp(name="det", input_ops=[read_op])
rec_op = RecOp(name="rec", input_ops=[det_op])
return rec_op
```
WebService objects need to load a yaml configuration file through the `prepare_pipeline_config` to configure each Op and the entire service. The simplest configuration file is as follows (Uci example):
```yaml
http_port: 18080
op:
uci:
local_service_conf:
model_config: uci_housing_model # path
```
All field names of yaml file are as follows:
```yaml
rpc_port: 18080 # gRPC port
build_dag_each_worker: false # Whether to use process server or not. The default is false
worker_num: 1 # gRPC thread pool size (the number of processes in the process version servicer). The default is 1
http_port: 0 # HTTP service port. Do not start HTTP service when the value is less or equals 0. The default value is 0.
dag:
is_thread_op: true # Whether to use the thread version of OP. The default is true
client_type: brpc # Use brpc or grpc client. The default is brpc
retry: 1 # The number of times DAG executor retries after failure. The default value is 1, that is, no retrying
use_profile: false # Whether to print the log on the server side. The default is false
tracer:
interval_s: -1 # Monitoring time interval of Tracer (in seconds). Do not start monitoring when the value is less than 1. The default value is -1
op:
<op_name>: # op name, corresponding to the one defined in the program
concurrency: 1 # op concurrency number, the default is 1
timeout: -1 # predict timeout in milliseconds. The default value is -1, that is, no timeout
retry: 1 # timeout retransmissions. The default value is 1, that is, do not try again
batch_size: 1 # If this field is set, Op will merge multiple request outputs into a single batch
auto_batching_timeout: -1 # auto-batching timeout in milliseconds. The default value is -1, that is, no timeout
local_service_conf:
model_config: # the path of the corresponding model file. There is no default value(None). If this item is not configured, the model file will not be loaded.
workdir: "" # working directory of corresponding model
thread_num: 2 # the corresponding model is started with thread_num threads
devices: "" # on which device does the model launched. You can specify the GPU card number(such as "0,1,2"), which is CPU by default
mem_optim: true # mem optimization option, the default is true
ir_optim: false # ir optimization option, the default is false
```
All fields of Op can be defined when Op is created in the program (which will override yaml fields).
doc/cpp_server/NEW_WEB_SERVICE_CN.md 已删除 100644 → 0
浏览文件 @ 32ccb13f
# 如何开发一个新的Web Service？
(简体中文|[English](NEW_WEB_SERVICE.md))
本文档将以 Uci 房价预测服务为例，来介绍如何开发一个新的Web Service。您可以在[这里](../python/examples/pipeline/simple_web_service/web_service.py)查阅完整的代码。
## Op 基类
在一些服务中，单个模型可能无法满足需求，需要多个模型串联或并联来完成整个服务。我们将单个模型操作称为 Op，并提供了一套简单的接口来实现 Op 串联或并联的复杂逻辑。
Op 间数据是以字典形式进行传递的，Op 可以以线程或进程方式启动，同时可以对 Op 的并发数等进行配置。
通常情况下，您需要继承 Op 基类，重写它的 `init_op`、`preprocess` 和 `postprocess` 方法，默认实现如下：
```python
class Op(object):
def init_op(self):
pass
def preprocess(self, input_dicts):
# multiple previous Op
if len(input_dicts) != 1:
_LOGGER.critical(
"Failed to run preprocess: this Op has multiple previous "
"inputs. Please override this func.")
os._exit(-1)
(_, input_dict), = input_dicts.items()
return input_dict
def postprocess(self, input_dicts, fetch_dict):
return fetch_dict
```
### init_op 方法
该方法用于加载用户自定义资源（如字典等），在 [UciOp](../python/examples/pipeline/simple_web_service/web_service.py) 中加载了一个分隔符。
**注意**：如果 Op 是以线程模式加载的，那么在 Op 多并发时，同种 Op 的不同线程只执行一次 `init_op`，且共用 `init_op` 加载的资源。
### preprocess 方法
该方法用于模型预测前对数据的预处理，它有一个 `input_dicts` 参数，`input_dicts` 是一个字典，key 为前继 Op 的 `name`，value 为对应前继 Op 传递过来的数据（数据同样是字典格式）。
`preprocess` 方法需要将数据处理成 ndarray 字典（key 为 feed 变量名，value 为对应的 ndarray 值），Op 会将该返回值作为模型预测的输入，并将输出传递给 `postprocess` 方法。
**注意**：如果 Op 没有配置模型，则 `preprocess` 的返回值会直接传递给 `postprocess`。
### postprocess 方法
该方法用于模型预测后对数据的后处理，它有两个参数，`input_dicts` 和 `fetch_dict`。
其中，`input_dicts` 与 `preprocess` 的参数相同，`fetch_dict` 为模型预测的输出（key 为 fetch 变量名，value 为对应的 ndarray 值）。Op 会将 `postprocess` 的返回值作为后继 Op `preprocess` 的输入。
**注意**：如果 Op 没有配置模型，则 `fetch_dict` 将为 `preprocess` 的返回值。
下面是 Uci 例子的 Op：
```python
class UciOp(Op):
def init_op(self):
self.separator = ","
def preprocess(self, input_dicts):
(_, input_dict), = input_dicts.items()
x_value = input_dict["x"]
if isinstance(x_value, (str, unicode)):
input_dict["x"] = np.array(
[float(x.strip()) for x in x_value.split(self.separator)])
return input_dict
def postprocess(self, input_dicts, fetch_dict):
fetch_dict["price"] = str(fetch_dict["price"][0][0])
return fetch_dict
```
## WebService 基类
Paddle Serving 实现了 [WebService](https://github.com/PaddlePaddle/Serving/blob/develop/python/paddle_serving_server/web_service.py#L28) 基类，您需要重写它的 `get_pipeline_response` 方法来定义 Op 间的拓扑关系，并返回作为 Response 的 Op，默认实现如下：
```python
class WebService(object):
def get_pipeline_response(self, read_op):
return None
```
其中，`read_op` 作为整个服务拓扑图的入口（即用户自定义的第一个 Op 的前继为 `read_op`）。
对于单 Op 服务（单模型），以 Uci 服务为例（整个服务中只有一个 Uci 房价预测模型）：
```python
class UciService(WebService):
def get_pipeline_response(self, read_op):
uci_op = UciOp(name="uci", input_ops=[read_op])
return uci_op
```
对于多 Op 服务（多模型），以 Ocr 服务为例（整个服务由 Det 模型和 Rec 模型串联完成）：
```python
class OcrService(WebService):
def get_pipeline_response(self, read_op):
det_op = DetOp(name="det", input_ops=[read_op])
rec_op = RecOp(name="rec", input_ops=[det_op])
return rec_op
```
WebService 对象需要通过 `prepare_pipeline_config` 加载一个 yaml 配置文件，用来对各个 Op 以及整个服务进行配置，最简单的配置文件如下（Uci 例子）：
```yaml
http_port: 18080
op:
uci:
local_service_conf:
model_config: uci_housing_model # 路径
```
yaml 文件的所有字段名详见下面：
```yaml
rpc_port: 18080 # gRPC端口号
build_dag_each_worker: false # 是否使用进程版 Servicer，默认为 false
worker_num: 1 # gRPC线程池大小（进程版 Servicer 中为进程数），默认为 1
http_port: 0 # HTTP 服务的端口号，若该值小于或等于 0 则不开启 HTTP 服务，默认为 0
dag:
is_thread_op: true # 是否使用线程版Op，默认为 true
client_type: brpc # 使用 brpc 或 grpc client，默认为 brpc
retry: 1 # DAG Executor 在失败后重试次数，默认为 1，即不重试
use_profile: false # 是否在 Server 端打印日志，默认为 false
tracer:
interval_s: -1 # Tracer 监控的时间间隔，单位为秒。当该值小于 1 时不启动监控，默认为 -1
op:
<op_name>: # op 名，与程序中定义的相对应
concurrency: 1 # op 并发数，默认为 1
timeout: -1 # 预测超时时间，单位为毫秒。默认为 -1 即不超时
retry: 1 # 超时重发次数。默认为 1 即不重试
batch_size: 1 # auto-batching 中的 batch_size，若设置该字段则 Op 会将多个请求输出合并为一个 batch
auto_batching_timeout: -1 # auto-batching 超时时间，单位为毫秒。默认为 -1 即不超时
local_service_conf:
model_config: # 对应模型文件的路径，无默认值（None）。若不配置该项则不会加载模型文件。
workdir: "" # 对应模型的工作目录
thread_num: 2 # 对应模型用几个线程启动
devices: "" # 模型启动在哪个设备上，可以指定 gpu 卡号（如 "0,1,2"），默认为 cpu
mem_optim: true # mem 优化选项，默认为 true
ir_optim: false # ir 优化选项，默认为 false
```
其中，Op 的所有字段均可以在程序中创建 Op 时定义（会覆盖 yaml 的字段）。
doc/model_zoo.md 已删除 100644 → 0
浏览文件 @ 32ccb13f
# Model Zoo
本页面展示了Paddle Serving目前支持的预训练模型以及下载链接
若您想为Paddle Serving提供新的模型，可通过[pull requese](https://github.com/PaddlePaddle/Serving/pulls)提交PR
*特别感谢[PadddlePaddle 全链条](https://www.paddlepaddle.org.cn/wholechain)以及[PaddleHub](https://www.paddlepaddle.org.cn/hub)为Paddle Serving提供的部分预训练模型*
| Model | Type | Dataset | Size | Download | Sample Input| Model mode |
| --- | --- | --- | --- | --- | --- | --- |
| ResNet_V2_50 | PaddleClas | ImageNet | 90.78 MB | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ImageClassification/resnet_v2_50_imagenet.tar.gz) | [daisy.jpg](../examples/PaddleClas/resnet_v2_50/daisy.jpg) |Eager|
| MobileNet_v2 | PaddleClas | ImageNet | 8.06 MB | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/ImageClassification/mobilenet_v2_imagenet.tar.gz) | [daisy.jpg](../examples/PaddleClas/mobilenet/daisy.jpg) |Eager|
| Bert | PaddleNLP | zhwiki | 361.96 MB | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SemanticModel/bert_chinese_L-12_H-768_A-12.tar.gz) | [data-c.txt](../examples/PaddleNLP/data-c.txt) |Eager|
| Senta | PaddleNLP | Baidu | 578.37 MB | [.tar.gz](https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SentimentAnalysis/senta_bilstm.tar.gz) | |Eager|
| Squeezenet 1_1 | Image Classification | ImageNet | 4.4 MB | [.mar](https://torchserve.pytorch.org/mar_files/squeezenet1_1.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Eager|
| MNIST digit classifier | Image Classification | MNIST | 4.3 MB | [.mar](https://torchserve.pytorch.org/mar_files/mnist_v2.mar) | [0.png](../examples/image_classifier/mnist/test_data/0.png) |Eager|
| Resnet 152 |Image Classification | ImageNet | 214 MB | [.mar](https://torchserve.pytorch.org/mar_files/resnet-152-batch_v2.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Eager|
| Faster RCNN | Object Detection | COCO | 148 MB | [.mar](https://torchserve.pytorch.org/mar_files/fastrcnn.mar) | [persons.jpg](../examples/object_detector/persons.jpg) |Eager|
| MASK RCNN | Object Detection | COCO | 158 MB | [.mar](https://torchserve.pytorch.org/mar_files/maskrcnn.mar) | [persons.jpg](../examples/object_detector/persons.jpg) |Eager|
| Text classifier | Text Classification | AG_NEWS | 169 MB | [.mar](https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar) | [sample_text.txt](../examples/text_classification/sample_text.txt) |Eager|
| FCN ResNet 101 | Image Segmentation | COCO | 193 MB | [.mar](https://torchserve.pytorch.org/mar_files/fcn_resnet_101.mar) | [persons.jpg](../examples/image_segmenter/persons.jpg) |Eager|
| DeepLabV3 ResNet 101 | Image Segmentation | COCO | 217 MB | [.mar](https://torchserve.pytorch.org/mar_files/deeplabv3_resnet_101_eager.mar) | [persons.jpg](https://github.com/pytorch/serve/blob/master/examples/image_segmenter/persons.jpg) |Eager|
| AlexNet Scripted | Image Classification | ImageNet | 216 MB | [.mar](https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Torchscripted |
| Densenet161 Scripted| Image Classification | ImageNet | 105 MB | [.mar](https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Torchscripted |
| Resnet18 Scripted| Image Classification | ImageNet | 42 MB | [.mar](https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Torchscripted |
| VGG16 Scripted| Image Classification | ImageNet | 489 MB | [.mar](https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Torchscripted |
| Squeezenet 1_1 Scripted | Image Classification | ImageNet | 4.4 MB | [.mar](https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Torchscripted |
| MNIST digit classifier Scripted | Image Classification | MNIST | 4.3 MB | [.mar](https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar) | [0.png](../examples/image_classifier/mnist/test_data/0.png) |Torchscripted |
| Resnet 152 Scripted |Image Classification | ImageNet | 215 MB | [.mar](https://torchserve.pytorch.org/mar_files/resnet-152-scripted_v2.mar) | [kitten.jpg](../examples/image_classifier/kitten.jpg) |Torchscripted |
| Text classifier Scripted | Text Classification | AG_NEWS | 169 MB | [.mar](https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar) | [sample_text.txt](../examples/text_classification/sample_text.txt) |Torchscripted |
| FCN ResNet 101 Scripted | Image Segmentation | COCO | 193 MB | [.mar](https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar) | [persons.jpg](../examples/image_segmenter/persons.jpg) |Torchscripted |
| DeepLabV3 ResNet 101 Scripted | Image Segmentation | COCO | 217 MB | [.mar](https://torchserve.pytorch.org/mar_files/deeplabv3_resnet_101_scripted.mar) | [persons.jpg](https://github.com/pytorch/serve/blob/master/examples/image_segmenter/persons.jpg) |Torchscripted |
Refer [example](../examples) for more details on above models.
examples/C++/PaddleClas/imagenet/README.md 0 → 100755
浏览文件 @ 56fde8bb
## Image Classification
([简体中文](./README_CN.md)|English)
The example uses the ResNet50_vd model to perform the imagenet 1000 classification task.
### Get model config and sample dataset
```
sh get_model.sh
```
### Install preprocess module
```
pip3 install paddle_serving_app
```
### Inference Service(Support BRPC-Client/GRPC-Client/Http-Client)
launch server side
```
python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 #cpu inference service
```
```
python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 --gpu_ids 0 #gpu inference service
```
### BRPC-Client
client send inference request
```
python3 resnet50_rpc_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
```
*the port of server side in this example is 9696
### GRPC-Client/Http-Client
client send inference request
```
python3 resnet50_http_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
```
examples/C++/PaddleClas/imagenet/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
## 图像分类示例
(简体中文|[English](./README.md))
示例中采用ResNet50_vd模型执行imagenet 1000分类任务。
### 获取模型配置文件和样例数据
```
sh get_model.sh
```
### 安装数据预处理模块
```
pip3 install paddle_serving_app
```
### 启动服务端（支持BRPC-Client、GRPC-Client、Http-Client）
启动server端
```
python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 #cpu预测服务
```
```
python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 --gpu_ids 0 #gpu预测服务
```
### BRPC-Client预测
client端进行预测
```
python3 resnet50_rpc_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
```
*server端示例中服务端口为9696端口
### GRPC-Client/Http-Client预测
client端进行预测
```
python3 resnet50_http_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
```
examples/C++/PaddleClas/imagenet/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
# -*- coding: utf-8 -*-
#
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# pylint: disable=doc-string-missing
from __future__ import unicode_literals, absolute_import
import os
import sys
import time
import requests
import json
import base64
from paddle_serving_client import Client
from paddle_serving_client.utils import MultiThreadRunner
from paddle_serving_client.utils import benchmark_args, show_latency
from paddle_serving_app.reader import Sequential, File2Image, Resize
from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
args = benchmark_args()
seq_preprocess = Sequential([
File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
def single_func(idx, resource):
file_list = []
turns = resource["turns"]
latency_flags = False
if os.getenv("FLAGS_serving_latency"):
latency_flags = True
latency_list = []
for file_name in os.listdir("./image_data/n01440764"):
file_list.append(file_name)
img_list = []
for i in range(1000):
img_list.append("./image_data/n01440764/" + file_list[i])
profile_flags = False
if "FLAGS_profile_client" in os.environ and os.environ[
"FLAGS_profile_client"]:
profile_flags = True
if args.request == "rpc":
fetch = ["score"]
client = Client()
client.load_client_config(args.model)
client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
start = time.time()
for i in range(turns):
if args.batch_size >= 1:
l_start = time.time()
feed_batch = []
i_start = time.time()
for bi in range(args.batch_size):
img = seq_preprocess(img_list[i])
feed_batch.append({"image": img})
i_end = time.time()
if profile_flags:
print("PROFILE\tpid:{}\timage_pre_0:{} image_pre_1:{}".
format(os.getpid(),
int(round(i_start * 1000000)),
int(round(i_end * 1000000))))
result = client.predict(feed=feed_batch, fetch=fetch)
l_end = time.time()
if latency_flags:
latency_list.append(l_end * 1000 - l_start * 1000)
else:
print("unsupport batch size {}".format(args.batch_size))
elif args.request == "http":
py_version = sys.version_info[0]
server = "http://" + resource["endpoint"][idx % len(resource[
"endpoint"])] + "/image/prediction"
start = time.time()
for i in range(turns):
if py_version == 2:
image = base64.b64encode(
open("./image_data/n01440764/" + file_list[i]).read())
else:
image_path = "./image_data/n01440764/" + file_list[i]
image = base64.b64encode(open(image_path, "rb").read()).decode(
"utf-8")
req = json.dumps({"feed": [{"image": image}], "fetch": ["score"]})
r = requests.post(
server, data=req, headers={"Content-Type": "application/json"})
end = time.time()
if latency_flags:
return [[end - start], latency_list]
return [[end - start]]
if __name__ == '__main__':
multi_thread_runner = MultiThreadRunner()
endpoint_list = [
"127.0.0.1:9292", "127.0.0.1:9293", "127.0.0.1:9294", "127.0.0.1:9295"
]
turns = 100
start = time.time()
result = multi_thread_runner.run(
single_func, args.thread, {"endpoint": endpoint_list,
"turns": turns})
#result = single_func(0, {"endpoint": endpoint_list})
end = time.time()
total_cost = end - start
avg_cost = 0
for i in range(args.thread):
avg_cost += result[0][i]
avg_cost = avg_cost / args.thread
print("total cost: {}s".format(end - start))
print("each thread cost: {}s.".format(avg_cost))
print("qps: {}samples/s".format(args.batch_size * args.thread * turns /
total_cost))
if os.getenv("FLAGS_serving_latency"):
show_latency(result[1])
examples/C++/PaddleClas/imagenet/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
rm profile_log*
export CUDA_VISIBLE_DEVICES=0,1,2,3
export FLAGS_profile_server=1
export FLAGS_profile_client=1
python -m paddle_serving_server.serve --model $1 --port 9292 --thread 4 --gpu_ids 0,1,2,3 --mem_optim --ir_optim 2> elog > stdlog &
sleep 5
gpu_id=0
#save cpu and gpu utilization log
if [ -d utilization ];then
rm -rf utilization
else
mkdir utilization
fi
#warm up
$PYTHONROOT/bin/python3 benchmark.py --thread 4 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
echo -e "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
for thread_num in 1 4 8 16
do
for batch_size in 1 4 16 64
do
job_bt=`date '+%Y%m%d%H%M%S'`
nvidia-smi --id=0 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_use.log 2>&1 &
nvidia-smi --id=0 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
gpu_memory_pid=$!
$PYTHONROOT/bin/python benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
kill ${gpu_memory_pid}
kill `ps -ef|grep used_memory|awk '{print $2}'`
echo "model name :" $1
echo "thread num :" $thread_num
echo "batch size :" $batch_size
echo "=================Done===================="
echo "model name :$1" >> profile_log
echo "batch size :$batch_size" >> profile_log
job_et=`date '+%Y%m%d%H%M%S'`
awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$1
awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$1
rm -rf gpu_use.log gpu_utilization.log
$PYTHONROOT/bin/python ../util/show_profile.py profile $thread_num >> profile_log
tail -n 8 profile >> profile_log
echo "" >> profile_log_$1
done
done
#Divided log
awk 'BEGIN{RS="\n\n"}{i++}{print > "ResNet_log_"i}' profile_log_$1
mkdir $1_log && mv ResNet_log_* $1_log
ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
examples/C++/PaddleClas/imagenet/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet50_vd.tar.gz
tar -xzvf ResNet50_vd.tar.gz
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet101_vd.tar.gz
tar -xzvf ResNet101_vd.tar.gz
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
tar -xzvf image_data.tar.gz
examples/C++/PaddleClas/imagenet/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleClas/imagenet/resnet50_http_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from paddle_serving_client import HttpClient
from paddle_serving_app.reader import Sequential, URL2Image, Resize
from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
import time
client = HttpClient()
client.load_client_config(sys.argv[1])
'''
if you want use GRPC-client, set_use_grpc_client(True)
or you can directly use client.grpc_client_predict(...)
as for HTTP-client,set_use_grpc_client(False)(which is default)
or you can directly use client.http_client_predict(...)
'''
#client.set_use_grpc_client(True)
'''
if you want to enable Encrypt Module,uncommenting the following line
'''
#client.use_key("./key")
'''
if you want to compress,uncommenting the following line
'''
#client.set_response_compress(True)
#client.set_request_compress(True)
'''
we recommend use Proto data format in HTTP-body, set True(which is default)
if you want use JSON data format in HTTP-body, set False
'''
#client.set_http_proto(True)
client.connect(["127.0.0.1:9696"])
label_dict = {}
label_idx = 0
with open("imagenet.label") as fin:
for line in fin:
label_dict[label_idx] = line.strip()
label_idx += 1
seq = Sequential([
URL2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
start = time.time()
image_file = "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"
for i in range(10):
img = seq(image_file)
fetch_map = client.predict(
feed={"image": img}, fetch=["score"], batch=False)
print(fetch_map)
end = time.time()
print(end - start)
examples/C++/PaddleClas/imagenet/resnet50_rpc_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from paddle_serving_client import Client
from paddle_serving_app.reader import Sequential, URL2Image, Resize
from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
import time
client = Client()
client.load_client_config(sys.argv[1])
client.connect(["127.0.0.1:9696"])
label_dict = {}
label_idx = 0
with open("imagenet.label") as fin:
for line in fin:
label_dict[label_idx] = line.strip()
label_idx += 1
seq = Sequential([
URL2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
start = time.time()
image_file = "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"
for i in range(10):
img = seq(image_file)
fetch_map = client.predict(
feed={"image": img}, fetch=["score"], batch=False)
prob = max(fetch_map["score"][0])
label = label_dict[fetch_map["score"][0].tolist().index(prob)].strip(
).replace(",", "")
print("prediction: {}, probability: {}".format(label, prob))
end = time.time()
print(end - start)
examples/C++/PaddleClas/imagenet/test_image_reader.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from paddle_serving_app.reader.image_reader import String2Image, Base64ToImage, Sequential
import base64
def test_String2Image():
with open("./daisy.jpg") as f:
img_str = f.read()
seq = Sequential([String2Image()])
img = seq(img_str)
assert (img.shape == (563, 500, 3))
def test_Base64ToImage():
with open("./daisy.jpg") as f:
img_str = f.read()
seq = Sequential([Base64ToImage()])
img = seq(base64.b64encode(img_str))
assert (img.shape == (563, 500, 3))
if __name__ == "__main__":
test_String2Image()
test_Base64ToImage()
examples/C++/PaddleClas/mobilenet/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Image Classification
## Get Model
```
python3 -m paddle_serving_app.package --get_model mobilenet_v2_imagenet
tar -xzvf mobilenet_v2_imagenet.tar.gz
```
## RPC Service
### Start Service
```
python3 -m paddle_serving_server.serve --model mobilenet_v2_imagenet_model --gpu_ids 0 --port 9393
```
### Client Prediction
```
python3 mobilenet_tutorial.py
```
examples/C++/PaddleClas/mobilenet/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 图像分类
## 获取模型
```
python3 -m paddle_serving_app.package --get_model mobilenet_v2_imagenet
tar -xzvf mobilenet_v2_imagenet.tar.gz
```
## RPC 服务
### 启动服务端
```
python3 -m paddle_serving_server.serve --model mobilenet_v2_imagenet_model --gpu_ids 0 --port 9393
```
### 客户端预测
```
python3 mobilenet_tutorial.py
```
examples/C++/PaddleClas/mobilenet/mobilenet_tutorial.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from paddle_serving_client import Client
from paddle_serving_app.reader import Sequential, File2Image, Resize
from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
client = Client()
client.load_client_config(
"mobilenet_v2_imagenet_client/serving_client_conf.prototxt")
client.connect(["127.0.0.1:9393"])
seq = Sequential([
File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
image_file = "daisy.jpg"
img = seq(image_file)
fetch_map = client.predict(feed={"image": img}, fetch=["feature_map"])
print(fetch_map["feature_map"].reshape(-1))
examples/C++/PaddleClas/resnet_v2_50/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Image Classification
## Get Model
```
python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
tar -xzvf resnet_v2_50_imagenet.tar.gz
```
## RPC Service
### Start Service
```
python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --gpu_ids 0 --port 9393
```
### Client Prediction
```
python3 resnet50_v2_tutorial.py
```
examples/C++/PaddleClas/resnet_v2_50/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 图像分类
## 获取模型
```
python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
tar -xzvf resnet_v2_50_imagenet.tar.gz
```
## RPC 服务
### 启动服务端
```
python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --gpu_ids 0 --port 9393
```
### 客户端预测
```
python3 resnet50_v2_tutorial.py
```
examples/C++/PaddleClas/resnet_v2_50/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
# -*- coding: utf-8 -*-
#
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# pylint: disable=doc-string-missing
from __future__ import unicode_literals, absolute_import
import os
import sys
import time
import json
import requests
import numpy as np
from paddle_serving_client import Client
from paddle_serving_client.utils import MultiThreadRunner
from paddle_serving_client.utils import benchmark_args, show_latency
from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
args = benchmark_args()
def single_func(idx, resource):
total_number = 0
profile_flags = False
latency_flags = False
if os.getenv("FLAGS_profile_client"):
profile_flags = True
if os.getenv("FLAGS_serving_latency"):
latency_flags = True
latency_list = []
if args.request == "rpc":
client = Client()
client.load_client_config(args.model)
client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
start = time.time()
for i in range(turns):
if args.batch_size >= 1:
l_start = time.time()
seq = Sequential([
File2Image(), Resize(256), CenterCrop(224), RGB2BGR(),
Transpose((2, 0, 1)), Div(255), Normalize(
[0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
image_file = "daisy.jpg"
img = seq(image_file)
feed_data = np.array(img)
feed_data = np.expand_dims(feed_data, 0).repeat(
args.batch_size, axis=0)
result = client.predict(
feed={"image": feed_data},
fetch=["save_infer_model/scale_0.tmp_0"],
batch=True)
l_end = time.time()
if latency_flags:
latency_list.append(l_end * 1000 - l_start * 1000)
total_number = total_number + 1
else:
print("unsupport batch size {}".format(args.batch_size))
else:
raise ValueError("not implemented {} request".format(args.request))
end = time.time()
if latency_flags:
return [[end - start], latency_list, [total_number]]
else:
return [[end - start]]
if __name__ == '__main__':
multi_thread_runner = MultiThreadRunner()
endpoint_list = ["127.0.0.1:9393"]
turns = 1
start = time.time()
result = multi_thread_runner.run(
single_func, args.thread, {"endpoint": endpoint_list,
"turns": turns})
end = time.time()
total_cost = end - start
total_number = 0
avg_cost = 0
for i in range(args.thread):
avg_cost += result[0][i]
total_number += result[2][i]
avg_cost = avg_cost / args.thread
print("total cost-include init: {}s".format(total_cost))
print("each thread cost: {}s. ".format(avg_cost))
print("qps: {}samples/s".format(args.batch_size * total_number / (
avg_cost * args.thread)))
print("qps(request): {}samples/s".format(total_number / (avg_cost *
args.thread)))
print("total count: {} ".format(total_number))
if os.getenv("FLAGS_serving_latency"):
show_latency(result[1])
examples/C++/PaddleClas/resnet_v2_50/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
rm profile_log*
rm -rf resnet_log*
export CUDA_VISIBLE_DEVICES=0,1,2,3
export FLAGS_profile_server=1
export FLAGS_profile_client=1
export FLAGS_serving_latency=1
gpu_id=3
#save cpu and gpu utilization log
if [ -d utilization ];then
rm -rf utilization
else
mkdir utilization
fi
#start server
python3.6 -m paddle_serving_server.serve --model $1 --port 9393 --thread 10 --gpu_ids $gpu_id --use_trt --ir_optim > elog 2>&1 &
sleep 15
#warm up
python3.6 benchmark.py --thread 1 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
echo -e "import psutil\nimport time\nwhile True:\n\tcpu_res = psutil.cpu_percent()\n\twith open('cpu.txt', 'a+') as f:\n\t\tf.write(f'{cpu_res}\\\n')\n\ttime.sleep(0.1)" > cpu.py
for thread_num in 1 2 4 8 16
do
for batch_size in 1 4 8 16 32
do
job_bt=`date '+%Y%m%d%H%M%S'`
nvidia-smi --id=$gpu_id --query-compute-apps=used_memory --format=csv -lms 100 > gpu_memory_use.log 2>&1 &
nvidia-smi --id=$gpu_id --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
rm -rf cpu.txt
python3.6 cpu.py &
gpu_memory_pid=$!
python3.6 benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
kill `ps -ef|grep used_memory|awk '{print $2}'` > /dev/null
kill `ps -ef|grep utilization.gpu|awk '{print $2}'` > /dev/null
kill `ps -ef|grep cpu.py|awk '{print $2}'` > /dev/null
echo "model_name:" $1
echo "thread_num:" $thread_num
echo "batch_size:" $batch_size
echo "=================Done===================="
echo "model_name:$1" >> profile_log_$1
echo "batch_size:$batch_size" >> profile_log_$1
job_et=`date '+%Y%m%d%H%M%S'`
awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "CPU_UTILIZATION:", max}' cpu.txt >> profile_log_$1
#awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_memory_use.log >> profile_log_$1
#awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$1
grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_memory_use.log >> profile_log_$1
awk -F" " '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$1
rm -rf gpu_memory_use.log gpu_utilization.log gpu_utilization.log.tmp
python3.6 ../util/show_profile.py profile $thread_num >> profile_log_$1
tail -n 10 profile >> profile_log_$1
echo "" >> profile_log_$1
done
done
#Divided log
awk 'BEGIN{RS="\n\n"}{i++}{print > "resnet_log_"i}' profile_log_$1
mkdir resnet_log && mv resnet_log_* resnet_log
ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
examples/C++/PaddleClas/resnet_v2_50/resnet50_debug.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
from paddle_serving_app.local_predict import LocalPredictor
import sys
debugger = LocalPredictor()
debugger.load_model_config(sys.argv[1], gpu=True)
seq = Sequential([
File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
image_file = "daisy.jpg"
img = seq(image_file)
fetch_map = debugger.predict(feed={"image": img}, fetch=["feature_map"])
print(fetch_map["feature_map"].reshape(-1))
examples/C++/PaddleClas/resnet_v2_50/resnet50_v2_tutorial.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from paddle_serving_client import Client
from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
client = Client()
client.load_client_config(
"resnet_v2_50_imagenet_client/serving_client_conf.prototxt")
client.connect(["127.0.0.1:9393"])
seq = Sequential([
File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
])
image_file = "daisy.jpg"
img = seq(image_file)
fetch_map = client.predict(feed={"image": img}, fetch=["score"])
print(fetch_map["score"].reshape(-1))
examples/C++/PaddleClas/resnet_v2_50/run_benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
if [ ! -x "ResNet50.tar.gz"]; then
wget https://paddle-inference-dist.bj.bcebos.com/AI-Rank/models/Paddle/ResNet50.tar.gz
fi
tar -xzvf ResNet50.tar.gz
python3.6 -m paddle_serving_client.convert --dirname ./ResNet50 --model_filename model --params_filename params
bash benchmark.sh serving_server serving_client
examples/C++/PaddleDetection/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Serve models from Paddle Detection
(English|[简体中文](./README_CN.md))
### Introduction
PaddleDetection flying paddle target detection development kit is designed to help developers complete the whole development process of detection model formation, training, optimization and deployment faster and better. For details, see [Github](https://github.com/PaddlePaddle/PaddleDetection/tree/master)
This article mainly introduces the deployment of Paddle Detection's dynamic graph model on Serving.
Paddle Detection provides a large number of [Model Zoo](https://github.com/PaddlePaddle/PaddleDetection/blob/master/docs/MODEL_ZOO_cn.md), these model libraries can be used in Paddle Serving with export tools Model. For the export tutorial, please refer to [Paddle Detection Export Model Tutorial (Simplified Chinese)](https://github.com/PaddlePaddle/PaddleDetection/blob/master/deploy/EXPORT_MODEL.md).
### Serving example
Several examples of PaddleDetection models used in Serving are given in this folder
All examples support TensorRT.
- [Faster RCNN](./faster_rcnn_r50_fpn_1x_coco)
- [PPYOLO](./ppyolo_r50vd_dcn_1x_coco)
- [TTFNet](./ttfnet_darknet53_1x_coco)
- [YOLOv3](./yolov3_darknet53_270e_coco)
- [HRNet](./faster_rcnn_hrnetv2p_w18_1x)
- [Fcos](./fcos_dcn_r50_fpn_1x_coco)
- [SSD](./ssd_vgg16_300_240e_voc/)
examples/C++/PaddleDetection/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
## 使用Paddle Detection模型
([English](./README.md)|简体中文)
### 简介
PaddleDetection飞桨目标检测开发套件，旨在帮助开发者更快更好地完成检测模型的组建、训练、优化及部署等全开发流程。详情参见[Github](https://github.com/PaddlePaddle/PaddleDetection/tree/master)
本文主要是介绍Paddle Detection的动态图模型在Serving上的部署。
### 导出模型
Paddle Detection提供了大量的[模型库](https://github.com/PaddlePaddle/PaddleDetection/blob/master/docs/MODEL_ZOO_cn.md), 这些模型库配合导出工具都可以得到可以用于Paddle Serving的模型。导出教程参见[Paddle Detection模型导出教程](https://github.com/PaddlePaddle/PaddleDetection/blob/master/deploy/EXPORT_MODEL.md)。
### Serving示例
本文件夹下给出了多个PaddleDetection模型用于Serving的范例
- [Faster RCNN](./faster_rcnn_r50_fpn_1x_coco)
- [PPYOLO](./ppyolo_r50vd_dcn_1x_coco)
- [TTFNet](./ttfnet_darknet53_1x_coco)
- [YOLOv3](./yolov3_darknet53_270e_coco)
- [HRNet](./faster_rcnn_hrnetv2p_w18_1x)
- [Fcos](./fcos_dcn_r50_fpn_1x_coco)
- [SSD](./ssd_vgg16_300_240e_voc/)
examples/C++/PaddleDetection/blazeface/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Blazeface
## Get Model
```
python3 -m paddle_serving_app.package --get_model blazeface
tar -xf blazeface.tar.gz
```
## RPC Service
### Start Service
```
python3 -m paddle_serving_server.serve --model serving_server --port 9494
```
### Client Prediction
```
python3 test_client.py serving_client/serving_client_conf.prototxt test.jpg
```
the result is in `output` folder, including a json file and image file with bounding boxes.
examples/C++/PaddleDetection/blazeface/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from paddle_serving_client import Client
from paddle_serving_app.reader import *
import sys
import numpy as np
from paddle_serving_app.reader import BlazeFacePostprocess
preprocess = Sequential([
File2Image(),
Normalize([104, 117, 123], [127.502231, 127.502231, 127.502231], False)
])
postprocess = BlazeFacePostprocess("label_list.txt", "output")
client = Client()
client.load_client_config(sys.argv[1])
client.connect(['127.0.0.1:9494'])
im_0 = preprocess(sys.argv[2])
tmp = Transpose((2, 0, 1))
im = tmp(im_0)
fetch_map = client.predict(
feed={"image": im}, fetch=["detection_output_0.tmp_0"])
fetch_map["image"] = sys.argv[2]
fetch_map["im_shape"] = im_0.shape
postprocess(fetch_map)
examples/C++/PaddleDetection/cascade_rcnn/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Cascade RCNN model on Paddle Serving
([简体中文](./README_CN.md)|English)
### Get The Cascade RCNN Model
```
sh get_data.sh
```
If you want to have more detection models, please refer to [Paddle Detection Model Zoo](https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/MODEL_ZOO_cn.md)
### Start the service
```
python3 -m paddle_serving_server.serve --model serving_server --port 9292 --gpu_id 0
```
### Perform prediction
```
python3 test_client.py 000000570688.jpg
```
Image with bounding boxes and json result would be saved in `output` folder.
examples/C++/PaddleDetection/cascade_rcnn/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 使用Paddle Serving部署Cascade RCNN模型
(简体中文|[English](./README.md))
## 获得Cascade RCNN模型
```
sh get_data.sh
```
如果你想要更多的检测模型，请参考[Paddle检测模型库](https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/MODEL_ZOO_cn.md)
### 启动服务
```
python3 -m paddle_serving_server.serve --model serving_server --port 9292 --gpu_id 0
```
### 执行预测
```
python3 test_client.py 000000570688.jpg
```
客户端已经为图片做好了后处理，在`output`文件夹下存放各个框的json格式信息还有后处理结果图片。
examples/C++/PaddleDetection/cascade_rcnn/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/cascade_mask_rcnn_r50_vd_fpn_ssld_2x_coco_serving.tar.gz
tar xf cascade_mask_rcnn_r50_vd_fpn_ssld_2x_coco_serving.tar.gz
examples/C++/PaddleDetection/cascade_rcnn/label_list.txt 0 → 100644
浏览文件 @ 56fde8bb
person
bicycle
car
motorcycle
airplane
bus
train
truck
boat
traffic light
fire hydrant
stop sign
parking meter
bench
bird
cat
dog
horse
sheep
cow
elephant
bear
zebra
giraffe
backpack
umbrella
handbag
tie
suitcase
frisbee
skis
snowboard
sports ball
kite
baseball bat
baseball glove
skateboard
surfboard
tennis racket
bottle
wine glass
cup
fork
knife
spoon
bowl
banana
apple
sandwich
orange
broccoli
carrot
hot dog
pizza
donut
cake
chair
couch
potted plant
bed
dining table
toilet
tv
laptop
mouse
remote
keyboard
cell phone
microwave
oven
toaster
sink
refrigerator
book
clock
vase
scissors
teddy bear
hair drier
toothbrush
examples/C++/PaddleDetection/cascade_rcnn/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import numpy as np
from paddle_serving_client import Client
from paddle_serving_app.reader import *
import cv2
preprocess = DetectionSequential([
DetectionFile2Image(), DetectionResize(
(800, 1333), True, interpolation=2),
DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
DetectionTranspose((2, 0, 1)), DetectionPadStride(32)
])
postprocess = RCNNPostprocess("label_list.txt", "output")
client = Client()
client.load_client_config("serving_client/serving_client_conf.prototxt")
client.connect(['127.0.0.1:9292'])
im, im_info = preprocess(sys.argv[1])
fetch_map = client.predict(
feed={
"image": im,
"im_shape": np.array(list(im.shape[1:])).reshape(-1),
"scale_factor": im_info['scale_factor'],
},
fetch=["save_infer_model/scale_0.tmp_1"],
batch=False)
print(fetch_map)
fetch_map["image"] = sys.argv[1]
postprocess(fetch_map)
examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Faster RCNN HRNet model on Paddle Serving
([简体中文](./README_CN.md)|English)
### Get The Faster RCNN HRNet Model
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/faster_rcnn_hrnetv2p_w18_1x.tar.gz
```
### Start the service
```
tar xf faster_rcnn_hrnetv2p_w18_1x.tar.gz
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
This model support TensorRT, if you want a faster inference, please use `--use_trt`. But you need to do some extra work.
Please reference to https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
### Prediction
```
python3 test_client.py 000000570688.jpg
```
examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 使用Paddle Serving部署Faster RCNN HRNet模型
(简体中文|[English](./README.md))
## 获得Faster RCNN HRNet模型
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/faster_rcnn_hrnetv2p_w18_1x.tar.gz
```
### 启动服务
```
tar xf faster_rcnn_hrnetv2p_w18_1x.tar.gz
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项,但此时需要额外设置子图的TRT变长最大最小最优shape.
请参考https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
### 执行预测
```
python3 test_client.py 000000570688.jpg
```
examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/label_list.txt 0 → 100644
浏览文件 @ 56fde8bb
person
bicycle
car
motorcycle
airplane
bus
train
truck
boat
traffic light
fire hydrant
stop sign
parking meter
bench
bird
cat
dog
horse
sheep
cow
elephant
bear
zebra
giraffe
backpack
umbrella
handbag
tie
suitcase
frisbee
skis
snowboard
sports ball
kite
baseball bat
baseball glove
skateboard
surfboard
tennis racket
bottle
wine glass
cup
fork
knife
spoon
bowl
banana
apple
sandwich
orange
broccoli
carrot
hot dog
pizza
donut
cake
chair
couch
potted plant
bed
dining table
toilet
tv
laptop
mouse
remote
keyboard
cell phone
microwave
oven
toaster
sink
refrigerator
book
clock
vase
scissors
teddy bear
hair drier
toothbrush
examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import numpy as np
from paddle_serving_client import Client
from paddle_serving_app.reader import *
import cv2
preprocess = DetectionSequential([
DetectionFile2Image(), DetectionResize(
(800, 1333), True, interpolation=2),
DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
DetectionTranspose((2, 0, 1)), DetectionPadStride(32)
])
postprocess = RCNNPostprocess("label_list.txt", "output")
client = Client()
client.load_client_config("serving_client/serving_client_conf.prototxt")
client.connect(['127.0.0.1:9494'])
im, im_info = preprocess(sys.argv[1])
fetch_map = client.predict(
feed={
"image": im,
"im_shape": np.array(list(im.shape[1:])).reshape(-1),
"scale_factor": im_info['scale_factor'],
},
fetch=["save_infer_model/scale_0.tmp_1"],
batch=False)
print(fetch_map)
fetch_map["image"] = sys.argv[1]
postprocess(fetch_map)
examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README.md 0 → 100644
浏览文件 @ 56fde8bb
# Faster RCNN model on Paddle Serving
([简体中文](./README_CN.md)|English)
### Get The Faster RCNN Model
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/faster_rcnn_r50_fpn_1x_coco.tar
```
### Start the service
```
tar xf faster_rcnn_r50_fpn_1x_coco.tar
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
This model support TensorRT, if you want a faster inference, please use `--use_trt`. But you need to do some extra work.
Please reference to https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
### Perform prediction
```
python3 test_client.py 000000570688.jpg
```
## 3. Result analysis
<p align = "center">
    <br>
<img src = '000000570688.jpg'>
    <br>
<p>
This is the input picture
  
<p align = "center">
    <br>
<img src = '000000570688_bbox.jpg'>
    <br>
<p>
This is the picture after adding bbox. You can see that the client has done post-processing for the picture. In addition, the output/bbox.json also has the number and coordinate information of each box.
examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 使用Paddle Serving部署Faster RCNN模型
(简体中文|[English](./README.md))
## 获得Faster RCNN模型
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/faster_rcnn_r50_fpn_1x_coco.tar
```
### 启动服务
```
tar xf faster_rcnn_r50_fpn_1x_coco.tar
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项,但此时需要额外设置子图的TRT变长最大最小最优shape.
请参考https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
### 执行预测
```
python3 test_client.py 000000570688.jpg
```
## 3. 结果分析
<p align="center">
<br>
<img src='000000570688.jpg' >
<br>
<p>
这是输入图片
<p align="center">
<br>
<img src='000000570688_bbox.jpg' >
<br>
<p>
这是实现添加了bbox之后的图片，可以看到客户端已经为图片做好了后处理，此外在output/bbox.json也有各个框的编号和坐标信息。
examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/label_list.txt 0 → 100644
浏览文件 @ 56fde8bb
person
bicycle
car
motorcycle
airplane
bus
train
truck
boat
traffic light
fire hydrant
stop sign
parking meter
bench
bird
cat
dog
horse
sheep
cow
elephant
bear
zebra
giraffe
backpack
umbrella
handbag
tie
suitcase
frisbee
skis
snowboard
sports ball
kite
baseball bat
baseball glove
skateboard
surfboard
tennis racket
bottle
wine glass
cup
fork
knife
spoon
bowl
banana
apple
sandwich
orange
broccoli
carrot
hot dog
pizza
donut
cake
chair
couch
potted plant
bed
dining table
toilet
tv
laptop
mouse
remote
keyboard
cell phone
microwave
oven
toaster
sink
refrigerator
book
clock
vase
scissors
teddy bear
hair drier
toothbrush
examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import numpy as np
from paddle_serving_client import Client
from paddle_serving_app.reader import *
import cv2
preprocess = DetectionSequential([
DetectionFile2Image(),
DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
DetectionResize(
(800, 1333), True, interpolation=cv2.INTER_LINEAR), DetectionTranspose(
(2, 0, 1)), DetectionPadStride(128)
])
postprocess = RCNNPostprocess("label_list.txt", "output")
client = Client()
client.load_client_config("serving_client/serving_client_conf.prototxt")
client.connect(['127.0.0.1:9494'])
im, im_info = preprocess(sys.argv[1])
fetch_map = client.predict(
feed={
"image": im,
"im_shape": np.array(list(im.shape[1:])).reshape(-1),
"scale_factor": im_info['scale_factor'],
},
fetch=["save_infer_model/scale_0.tmp_1"],
batch=False)
fetch_map["image"] = sys.argv[1]
postprocess(fetch_map)
examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README.md 0 → 100644
浏览文件 @ 56fde8bb
# FCOS model on Paddle Serving
([简体中文](./README_CN.md)|English)
### Get Model
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/fcos_dcn_r50_fpn_1x_coco.tar
```
### Start the service
```
tar xf fcos_dcn_r50_fpn_1x_coco.tar
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
This model support TensorRT, if you want a faster inference, please use `--use_trt`.
### Perform prediction
```
python3 test_client.py 000000014439.jpg
```
examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 使用Paddle Serving部署FCOS模型
(简体中文|[English](./README.md))
## 获得模型
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/fcos_dcn_r50_fpn_1x_coco.tar
```
### 启动服务
```
tar xf fcos_dcn_r50_fpn_1x_coco.tar
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
### 执行预测
```
python3 test_client.py 000000014439.jpg
```
examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/label_list.txt 0 → 100644
浏览文件 @ 56fde8bb
person
bicycle
car
motorcycle
airplane
bus
train
truck
boat
traffic light
fire hydrant
stop sign
parking meter
bench
bird
cat
dog
horse
sheep
cow
elephant
bear
zebra
giraffe
backpack
umbrella
handbag
tie
suitcase
frisbee
skis
snowboard
sports ball
kite
baseball bat
baseball glove
skateboard
surfboard
tennis racket
bottle
wine glass
cup
fork
knife
spoon
bowl
banana
apple
sandwich
orange
broccoli
carrot
hot dog
pizza
donut
cake
chair
couch
potted plant
bed
dining table
toilet
tv
laptop
mouse
remote
keyboard
cell phone
microwave
oven
toaster
sink
refrigerator
book
clock
vase
scissors
teddy bear
hair drier
toothbrush
examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import numpy as np
from paddle_serving_client import Client
from paddle_serving_app.reader import *
import cv2
preprocess = DetectionSequential([
DetectionFile2Image(),
DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
DetectionResize(
(800, 1333), True, interpolation=cv2.INTER_LINEAR), DetectionTranspose(
(2, 0, 1)), DetectionPadStride(128)
])
postprocess = RCNNPostprocess("label_list.txt", "output")
client = Client()
client.load_client_config("serving_client/serving_client_conf.prototxt")
client.connect(['127.0.0.1:9494'])
im, im_info = preprocess(sys.argv[1])
fetch_map = client.predict(
feed={
"image": im,
"scale_factor": im_info['scale_factor'],
},
fetch=["save_infer_model/scale_0.tmp_1"],
batch=False)
print(fetch_map)
fetch_map["image"] = sys.argv[1]
postprocess(fetch_map)
examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README.md 0 → 100644
浏览文件 @ 56fde8bb
# PP-YOLO model on Paddle Serving
([简体中文](./README_CN.md)|English)
### Get The Model
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_r50vd_dcn_1x_coco.tar
```
### Start the service
```
tar xf ppyolo_r50vd_dcn_1x_coco.tar
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
This model support TensorRT, if you want a faster inference, please use `--use_trt`.
### Perform prediction
```
python3 test_client.py 000000570688.jpg
```
examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
# 使用Paddle Serving部署PP-YOLO模型
(简体中文|[English](./README.md))
## 获得模型
```
wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_r50vd_dcn_1x_coco.tar
```
### 启动服务
```
tar xf ppyolo_r50vd_dcn_1x_coco.tar
python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
```
该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
### 执行预测
```
python3 test_client.py 000000570688.jpg
```
examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import numpy as np
from paddle_serving_client import Client
from paddle_serving_app.reader import *
import cv2
preprocess = DetectionSequential([
DetectionFile2Image(),
DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
DetectionResize(
(608, 608), False, interpolation=2), DetectionTranspose((2, 0, 1))
])
postprocess = RCNNPostprocess("label_list.txt", "output")
client = Client()
client.load_client_config("serving_client/serving_client_conf.prototxt")
client.connect(['127.0.0.1:9494'])
im, im_info = preprocess(sys.argv[1])
fetch_map = client.predict(
feed={
"image": im,
"im_shape": np.array(list(im.shape[1:])).reshape(-1),
"scale_factor": im_info['scale_factor'],
},
fetch=["save_infer_model/scale_0.tmp_1"],
batch=False)
fetch_map["image"] = sys.argv[1]
postprocess(fetch_map)
examples/C++/PaddleDetection/yolov4/000000570688.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleDetection/yolov4/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleDetection/yolov4/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleDetection/yolov4/label_list.txt 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleDetection/yolov4/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/README.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/batching.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/benchmark_with_profile.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/bert_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/bert_gpu_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/bert_httpclient.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/bert_reader.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/bert_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/prepare_model.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/test_multi_fetch_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/bert/tokenization.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/README.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/lac_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/lac_http_client.py 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/lac_reader.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/lac/utils.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/senta/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/senta/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/senta/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleNLP/senta/senta_web_service.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/README.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/det_debugger_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/det_web_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/imgs/1.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/ocr_cpp_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/ocr_debugger_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/ocr_web_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/ocr_web_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/rec_debugger_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/rec_img/ch_doc3.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/rec_web_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleOCR/ocr/rec_web_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/args.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/benchmark_batch.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/benchmark_batch.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/clean.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/local_train.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/network_conf.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr/test_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr_with_cube/README.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr_with_cube/cube/keys 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleRec/criteo_ctr_with_cube/get_data.sh 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/deeplabv3/N0060.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/deeplabv3/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/deeplabv3/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/deeplabv3/deeplabv3_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/unet_for_image_seg/N0060.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/unet_for_image_seg/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/unet_for_image_seg/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/PaddleSeg/unet_for_image_seg/seg_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/encryption/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/encryption/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/encryption/encrypt.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/encryption/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/encryption/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/local_train.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/test_client.py 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/test_httpclient.py 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/test_multi_process_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/fit_a_line/test_server.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/README.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/abtest_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/abtest_get_data.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/clean_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/imdb_reader.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/local_train.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/nets.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/imdb/test_http_client.py 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/low_precision/resnet50/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/low_precision/resnet50/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/low_precision/resnet50/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/low_precision/resnet50/resnet50_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/util/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/util/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/util/get_acc.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/util/show_profile.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/util/timeline_trace.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/bert/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/bert/bert_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/bert/chinese_bert_reader.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/bert/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/ernie/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/ernie/chinese_ernie_reader.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/ernie/ernie_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/ernie/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/fit_a_line_xpu/README.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/fit_a_line_xpu/README_CN.md 0 → 100755
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/fit_a_line_xpu/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/fit_a_line_xpu/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/fit_a_line_xpu/local_train.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/fit_a_line_xpu/test_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/resnet_v2_50_xpu/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/resnet_v2_50_xpu/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/resnet_v2_50_xpu/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/resnet_v2_50_xpu/localpredict.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/resnet_v2_50_xpu/resnet50_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/vgg19/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/vgg19/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/C++/xpu/vgg19/vgg19_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/DarkNet53/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/HRNet_W18_C/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV1/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/MobileNetV2/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_KL/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_KL/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_KL/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet50_vd_PACT/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet_V2_50/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet_V2_50/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet_V2_50/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ResNet_V2_50/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/imagenet/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/imagenet/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/imagenet/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/imagenet/daisy.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/imagenet/get_model.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleClas/imagenet/imagenet.label 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/faster_rcnn/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/ppyolo_mbv3/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/000000570688.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/label_list.txt 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleDetection/yolov3/web_service.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/pipeline_rpc_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleNLP/bert/web_service.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/imgs/1.jpg 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/PaddleOCR/ocr/web_service.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/imdb_model_ensemble/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/imdb_model_ensemble/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/imdb_model_ensemble/analyse.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/imdb_model_ensemble/analyse.yaml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/imdb_model_ensemble/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/imdb_model_ensemble/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/README.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/README_CN.md 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/benchmark.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/benchmark.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/config.yml 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/get_data.sh 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
examples/Pipeline/simple_web_service/web_service.py 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
tools/Dockerfile.cuda11.2-cudnn8.devel 0 → 100644
浏览文件 @ 56fde8bb
此差异已折叠。
tools/Dockerfile.devel
浏览文件 @ 56fde8bb
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或