+
+
+
+Server 端线程数 N = 模型预测引擎数 N = 同时处理 Request 请求数 N，超发的 Request 请求需要等待当前线程处理结束后才能得到响应和处理。
+
+
+
+**二.异步调度线程**
+
+为了提高计算芯片吞吐和计算资源利用率，C++ Serving 在调度层实现异步多线程并发合并请求，实现动态批量推理。异步模型主要适用于模型支持批量，单个 Request 请求的无批量或较小，单次预测时间较长的情况。
+
+
+
+
+异步模式下，Server 端 N 个线程只负责接收 Request 请求，实际调用预测引擎是在异步框架的线程池中，异步框架的线程数可以由配置选项来指定。为了方便理解，我们假设每个 Request 请求批量均为1，此时异步框架会尽可能多得从请求池中取 n(n≤M)个 Request 并将其拼装为1个 Request(batch=n)，调用1次预测引擎，得到1个 Response(batch = n)，再将其对应拆分为 n 个 Response 作为返回结果。
+
+
+
+**三.动态批量**
+
+通常，异步框架合并多个请求的前提是所有请求的 `feed var` 的维度除 batch 维度外必须是相同的。例如，以 OCR 文字识别案例中检测模型为例，A 请求的 `x` 变量的 shape 是 [1, 3, 960, 960]，B 请求的 `x` 变量的 shape 是 [2, 3, 960, 960]，虽然第一个维度值不相同，但第一个维度属于 `batch` 维度，因此，请求 A 和 请求 B 可以合并。C 请求的 `x` 变量的 shape 是 [1, 3, 640, 480]，由于除了 `batch` 维度外还有2个维度值不同，A 和 C 不能直接合并。
+
+从经验来看，当2个请求的同一个变量 shape 维度的数量相等时，通过 `padding` 补0的方式按最大 shape 值对齐即可。即 C 请求的 shape 补齐到 [1, 3, 960, 960]，那么就可以与 A 和 B 请求合并了。Paddle Serving 框架实现了动态 Padding 功能补齐 shape。
+
+当多个将要合并的请求中有一个 shape 值很大时，所有请求的 shape 都要按最大补齐，导致计算量成倍增长。Paddle Serving 设计了一套合并策略，满足任何一个条件均可合并:
+
+- 条件 1：绝对值差的字节数小于 **1024** 字节，评估补齐绝对长度
+- 条件 2：相似度的乘积大于 **50%**，评估相似度，评估补齐绝对值整体数据量比例
+
+场景1：`Shape-1 = [batch, 500, 500], Shape-2 = [batch, 400, 400]`。此时，`绝对值差 = 500*500 - 400*400 = 90000` 字节，`相对误差= (400/500) * (400/500) = 0.8*0.8 = 0.64`，满足条件1，不满足条件2，触发动态 Padding。
+
+场景2：`Shape-1 = [batch, 1, 1], Shape-2 = [batch, 2, 2]`。此时，`绝对值差 = 2*2 - 1*1 = 3`字节，`相对误差 = (1/2) * (1/2) = 0.5*0.5 = 0.25`，满足条件2，不满足条件1，触发动态 Padding。
+
+场景3：`Shape-1 = [batch, 3, 320, 320], Shape-2 = [batch, 3, 960, 960]`。此时，`绝对值差 = 3*960*960 - 3*320*320 = 2457600`字节，`相对误差 = (3/3) * (320/960) * (320/960) = 0.3*0.3 = 0.09`，条件1和条件2均不满足，未触发动态 Padding。
+
+
+
+## 使用案例
+
+
+
+**一.开启同步模式**
+
+启动命令不使用 `--runtime_thread_num` 和 `--batch_infer_size` 时，属于同步处理模式，未开启异步模式。`--thread 16` 表示启动16个同步网络处理线程。
+```
+python3 -m paddle_serving_server.serve --model uci_housing_model --thread 16 --port 9292
+```
+
+
+
+**二.开启异步模式**
+
+启动命令使用 `--runtime_thread_num 2` 和 `--batch_infer_size 32` 开启异步模式，Serving 框架会启动2个异步线程，单次合并最大批量为32，自动开启动态 Padding。
+```
+python3 -m paddle_serving_server.serve --model uci_housing_model --thread 16 --port 9292 --runtime_thread_num 4 --batch_infer_size 32 --ir_optim --gpu_multi_stream --gpu_ids 0
+```
+
+
+
+## 性能测试
+
+
+- GPU：Tesla P4 7611 MiB
+- CUDA：cuda11.2-cudnn8-trt8
+- Python 版本：python3.7
+- 模型：ResNet_v2_50
+- 测试数据：构造全1输入，单client请求100次，shape 范围(1, 224 ± 50, 224 ± 50)
+
+同步模式启动命令:
+```
+python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --port 9393 --thread 8 --ir_optim --gpu_multi_stream --gpu_ids 1 --enable_prometheus --prometheus_port 1939
+```
+
+异步模式启动命令:
+```
+python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --port 9393 --thread 64 --runtime_thread_num 8 --ir_optim --gpu_multi_stream --gpu_ids 1 --enable_prometheus --prometheus_port 19393
+```
+
+
+
+**一.测试结果**
+
+使用异步模式，并开启动态批量后，并发测试不同 shape 数据时，吞吐性能大幅提升。
+
+
+
+
+异步模式情况下，两者性能接近，但当 Client 并发数达到70的时候，TF-Serving 服务直接超时，而 C++ Serving 能够正常返回结果。
+
+
+
+
+## 6.合作案例
+
+## 7.资源汇总
+
+## 8.开发者贡献&社区
diff --git a/doc/Offical_Docs/images/6-1_Cpp_Asynchronous_Framwork_CN_1.png b/doc/Offical_Docs/images/6-1_Cpp_Asynchronous_Framwork_CN_1.png
new file mode 100644
index 0000000000000000000000000000000000000000..88c36f7688719d7081598e5177cd64042ab5c9de
Binary files /dev/null and b/doc/Offical_Docs/images/6-1_Cpp_Asynchronous_Framwork_CN_1.png differ
diff --git a/doc/Offical_Docs/images/6-5_Cpp_ABTest_CN_1.png b/doc/Offical_Docs/images/6-5_Cpp_ABTest_CN_1.png
new file mode 100644
index 0000000000000000000000000000000000000000..5e8f8980dffb46f4960390e6edb281968ae8bd83
Binary files /dev/null and b/doc/Offical_Docs/images/6-5_Cpp_ABTest_CN_1.png differ
diff --git a/doc/Run_On_NPU_CN.md b/doc/Run_On_NPU_CN.md
index 2919ae7290071b409852634e4274911d8f46992b..74a438bca1e4bfa0016c4165ab1edb372ef62268 100644
--- a/doc/Run_On_NPU_CN.md
+++ b/doc/Run_On_NPU_CN.md
@@ -75,7 +75,7 @@ go env -w GOPROXY=https://goproxy.cn,direct
go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway@v1.15.2
go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger@v1.15.2
go install github.com/golang/protobuf/protoc-gen-go@v1.4.3
-go install google.golang.org/grpc@v1.33.0
+go install google.golang.org/grpc@v1.33.1
go env -w GO111MODULE=auto
```
@@ -193,4 +193,4 @@ python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --thr
## 其他说明
### NPU芯片支持相关参考资料
-* [昇腾NPU芯片运行飞桨](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/09_hardware_support/npu_docs/paddle_install_cn.html)
\ No newline at end of file
+* [昇腾NPU芯片运行飞桨](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/09_hardware_support/npu_docs/paddle_install_cn.html)
diff --git a/doc/Save_CN.md b/doc/Save_CN.md
index 29b260bfb029333dff8da7db83f651aa6b8bb8af..6c8122ea4517d3b4f731b8f27c311cbdec2f58db 100644
--- a/doc/Save_CN.md
+++ b/doc/Save_CN.md
@@ -2,6 +2,12 @@
(简体中文|[English](./Save_EN.md))
+## 保存用于 Serving 部署模型的意义
+
+
+
+
+
## 从已保存的模型文件中导出
如果已使用Paddle 的`save_inference_model`接口保存出预测要使用的模型，你可以使用Paddle Serving提供的名为`paddle_serving_client.convert`的内置模块进行转换。
```python
diff --git a/doc/TensorRT_Dynamic_Shape_CN.md b/doc/TensorRT_Dynamic_Shape_CN.md
index 7ffc8693a4b8070c6395ad0c0fe200d646fc1df6..7a9759c33ce6e9b5658af91d0deed3919856d340 100644
--- a/doc/TensorRT_Dynamic_Shape_CN.md
+++ b/doc/TensorRT_Dynamic_Shape_CN.md
@@ -33,6 +33,7 @@ python -m paddle_serving_server.serve \
**二. C++ Serving 设置动态 shape**
+1. 方法一：
在`**/paddle_inference/paddle/include/paddle_engine.h` 修改如下代码
```
@@ -127,6 +128,55 @@ python -m paddle_serving_server.serve \
}
```
+2. 方法二：
+在`**/python/paddle_serving_server/serve.py` 参考如下代码生成配置信息，
+并使用`server.set_trt_dynamic_shape_info(info)`方法进行设置
+
+```
+def set_ocr_dynamic_shape_info():
+ info = []
+ min_input_shape = {
+ "x": [1, 3, 50, 50],
+ "conv2d_182.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_2.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_3.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_4.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_5.tmp_0": [1, 1, 20, 20]
+ }
+ max_input_shape = {
+ "x": [1, 3, 1536, 1536],
+ "conv2d_182.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_2.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_3.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_4.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_5.tmp_0": [20, 200, 960, 960],
+ }
+ opt_input_shape = {
+ "x": [1, 3, 960, 960],
+ "conv2d_182.tmp_0": [3, 96, 240, 240],
+ "nearest_interp_v2_2.tmp_0": [3, 96, 240, 240],
+ "nearest_interp_v2_3.tmp_0": [3, 24, 240, 240],
+ "nearest_interp_v2_4.tmp_0": [3, 24, 240, 240],
+ "nearest_interp_v2_5.tmp_0": [3, 24, 240, 240],
+ }
+ det_info = {
+ "min_input_shape": min_input_shape,
+ "max_input_shape": max_input_shape,
+ "opt_input_shape": opt_input_shape,
+ }
+ info.append(det_info)
+ min_input_shape = {"x": [1, 3, 32, 10], "lstm_1.tmp_0": [1, 1, 128]}
+ max_input_shape = {"x": [50, 3, 32, 1000], "lstm_1.tmp_0": [500, 50, 128]}
+ opt_input_shape = {"x": [6, 3, 32, 100], "lstm_1.tmp_0": [25, 5, 128]}
+ rec_info = {
+ "min_input_shape": min_input_shape,
+ "max_input_shape": max_input_shape,
+ "opt_input_shape": opt_input_shape,
+ }
+ info.append(rec_info)
+ return info
+```
+
## Pipeline Serving
diff --git a/doc/TensorRT_Dynamic_Shape_EN.md b/doc/TensorRT_Dynamic_Shape_EN.md
index c9c9aabf2025028cf0aa22e9e86187cdc8f8cfad..47a6704971eda2172e19b460cdd22eea2d16279f 100644
--- a/doc/TensorRT_Dynamic_Shape_EN.md
+++ b/doc/TensorRT_Dynamic_Shape_EN.md
@@ -16,6 +16,8 @@ The following is the dynamic shape api
For detail, please refer to API doc [C++](https://paddleinference.paddlepaddle.org.cn/api_reference/cxx_api_doc/Config/GPUConfig.html#tensorrt)/[Python](https://paddleinference.paddlepaddle.org.cn/api_reference/python_api_doc/Config/GPUConfig.html#tensorrt)
### C++ Serving
+
+1. Method 1:
Modify the following code in `**/paddle_inference/paddle/include/paddle_engine.h`
```
@@ -110,6 +112,54 @@ Modify the following code in `**/paddle_inference/paddle/include/paddle_engine.h
}
```
+2. Method 2:
+Refer to the code of `**/python/paddle_serving_server/serve.py` below to generate the configuration information,
+and using method `server.set_trt_dynamic_shape_info(info)` to set information.
+
+```
+def set_ocr_dynamic_shape_info():
+ info = []
+ min_input_shape = {
+ "x": [1, 3, 50, 50],
+ "conv2d_182.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_2.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_3.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_4.tmp_0": [1, 1, 20, 20],
+ "nearest_interp_v2_5.tmp_0": [1, 1, 20, 20]
+ }
+ max_input_shape = {
+ "x": [1, 3, 1536, 1536],
+ "conv2d_182.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_2.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_3.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_4.tmp_0": [20, 200, 960, 960],
+ "nearest_interp_v2_5.tmp_0": [20, 200, 960, 960],
+ }
+ opt_input_shape = {
+ "x": [1, 3, 960, 960],
+ "conv2d_182.tmp_0": [3, 96, 240, 240],
+ "nearest_interp_v2_2.tmp_0": [3, 96, 240, 240],
+ "nearest_interp_v2_3.tmp_0": [3, 24, 240, 240],
+ "nearest_interp_v2_4.tmp_0": [3, 24, 240, 240],
+ "nearest_interp_v2_5.tmp_0": [3, 24, 240, 240],
+ }
+ det_info = {
+ "min_input_shape": min_input_shape,
+ "max_input_shape": max_input_shape,
+ "opt_input_shape": opt_input_shape,
+ }
+ info.append(det_info)
+ min_input_shape = {"x": [1, 3, 32, 10], "lstm_1.tmp_0": [1, 1, 128]}
+ max_input_shape = {"x": [50, 3, 32, 1000], "lstm_1.tmp_0": [500, 50, 128]}
+ opt_input_shape = {"x": [6, 3, 32, 100], "lstm_1.tmp_0": [25, 5, 128]}
+ rec_info = {
+ "min_input_shape": min_input_shape,
+ "max_input_shape": max_input_shape,
+ "opt_input_shape": opt_input_shape,
+ }
+ info.append(rec_info)
+ return info
+```
### Pipeline Serving
@@ -151,4 +201,4 @@ if use_trt:
names[3]: [10, head_number, 60, 60]
})
-```
\ No newline at end of file
+```
diff --git a/doc/images/kubernetes_design.png b/doc/images/kubernetes_design.png
new file mode 100644
index 0000000000000000000000000000000000000000..05ace675ba636f7d271cb9db8cdba741db494026
Binary files /dev/null and b/doc/images/kubernetes_design.png differ
diff --git a/doc/images/low_precision_profile.png b/doc/images/low_precision_profile.png
new file mode 100644
index 0000000000000000000000000000000000000000..82356df740bf4a31bc47ae2017103e7fca3792be
Binary files /dev/null and b/doc/images/low_precision_profile.png differ
diff --git a/doc/images/wechat_group_1.jpeg b/doc/images/wechat_group_1.jpeg
index 4e9e65b14c5dcd594ecd95e7ea56add14d0f92d4..80e8acc728faaae7bcb254e4fed93dfaffd84d59 100644
Binary files a/doc/images/wechat_group_1.jpeg and b/doc/images/wechat_group_1.jpeg differ
diff --git a/doc/wechat_group_1.jpeg b/doc/wechat_group_1.jpeg
new file mode 100644
index 0000000000000000000000000000000000000000..443a1549d5c79e86b26038d7eb2e704ed5f9213f
Binary files /dev/null and b/doc/wechat_group_1.jpeg differ
diff --git a/examples/C++/PaddleClas/resnet_50_vd/README_CN.md b/examples/C++/PaddleClas/resnet_50_vd/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..0034b5b4c03e572b7a27ff6296e185fed713eabb
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_50_vd/README_CN.md
@@ -0,0 +1,69 @@
+# 图像分类
+
+## 1.获取模型
+
+```
+wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/ResNet50_vd_infer.tar && tar xf ResNet50_vd_infer.tar
+
+```
+
+## 2.用 paddle_serving_client 把下载的推理模型保存用于 Serving 部署的模型参数
+```
+# 保存 ResNet50_vd 模型参数
+python3 -m paddle_serving_client.convert --dirname ./ResNet50_vd_infer/ \
+ --model_filename inference.pdmodel \
+ --params_filename inference.pdiparams \
+ --serving_server ./ResNet50_vd_serving/ \
+ --serving_client ./ResNet50_vd_client/
+```
+会在当前文件夹多出 `ResNet50_vd_serving` 和 `ResNet50_vd_client` 的文件夹
+
+
+保存参数后，会在当前文件夹多出 `ResNet50_vd_serving` 和 `ResNet50_vd_client` 的文件夹：
+```
+├── daisy.jpg
+├── http_client.py
+├── imagenet.label
+├── ResNet50_vd_client
+│ ├── serving_client_conf.prototxt
+│ └── serving_client_conf.stream.prototxt
+├── ResNet50_vd_infer
+│ ├── inference.pdiparams
+│ ├── inference.pdiparams.info
+│ └── inference.pdmodel
+├── ResNet50_vd_serving
+│ ├── fluid_time_file
+│ ├── inference.pdiparams
+│ ├── inference.pdmodel
+│ ├── serving_server_conf.prototxt
+│ └── serving_server_conf.stream.prototxt
+├── rpc_client.py
+```
+
+**三.启动服务**
+
+C++ Serving 服务可以指定一个网络端口同时接收 HTTP、gRPC 和 bRPC 请求。命令参数 `--model` 指定模型路径，`--gpu_ids` 指定 GPU 卡，`--port` 指定端口。
+
+```
+python3 -m paddle_serving_server.serve --model ResNet50_vd_serving --gpu_ids 0 --port 9394
+```
+
+**四.启动客户端**
+
+1. `rpc_client.py` 封装了 HTTP 请求客户端
+
+```
+python3 http_client.py
+```
+
+2. `http_client.py` 封装了 gRPC 请求客户端
+
+```
+python3 rpc_client.py
+```
+
+成功运行后，模型预测的结果会打印如下：
+
+```
+prediction: daisy, probability: 0.9341399073600769
+```
diff --git a/examples/C++/PaddleClas/resnet_50_vd/daisy.jpg b/examples/C++/PaddleClas/resnet_50_vd/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/PaddleClas/resnet_50_vd/daisy.jpg differ
diff --git a/examples/C++/PaddleClas/resnet_50_vd/http_client.py b/examples/C++/PaddleClas/resnet_50_vd/http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..722f489e30d5e7e98408f287540ff1b6c1cf0cfc
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_50_vd/http_client.py
@@ -0,0 +1,82 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import HttpClient
+
+#app
+from paddle_serving_app.reader import Sequential, URL2Image, Resize
+from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
+import time
+
+client = HttpClient()
+client.load_client_config("./ResNet50_vd_client/serving_client_conf.prototxt")
+'''
+if you want use GRPC-client, set_use_grpc_client(True)
+or you can directly use client.grpc_client_predict(...)
+as for HTTP-client,set_use_grpc_client(False)(which is default)
+or you can directly use client.http_client_predict(...)
+'''
+#client.set_use_grpc_client(True)
+'''
+if you want to enable Encrypt Module,uncommenting the following line
+'''
+#client.use_key("./key")
+'''
+if you want to compress,uncommenting the following line
+'''
+#client.set_response_compress(True)
+#client.set_request_compress(True)
+'''
+we recommend use Proto data format in HTTP-body, set True(which is default)
+if you want use JSON data format in HTTP-body, set False
+'''
+#client.set_http_proto(True)
+client.connect(["127.0.0.1:9394"])
+
+label_dict = {}
+label_idx = 0
+with open("imagenet.label") as fin:
+ for line in fin:
+ label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+#preprocess
+seq = Sequential([
+ URL2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+start = time.time()
+image_file = "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"
+for i in range(1):
+ img = seq(image_file)
+ res = client.predict(feed={"inputs": img}, fetch=[], batch=False)
+
+ if res is None:
+ raise ValueError("predict error")
+
+ if res.err_no != 0:
+ raise ValueError("predict error. Response : {}".format(res))
+
+ max_val = res.outputs[0].tensor[0].float_data[0]
+ max_idx = 0
+ for one_data in enumerate(res.outputs[0].tensor[0].float_data):
+ if one_data[1] > max_val:
+ max_val = one_data[1]
+ max_idx = one_data[0]
+ label = label_dict[max_idx].strip().replace(",", "")
+ print("prediction: {}, probability: {}".format(label, max_val))
+end = time.time()
+print(end - start)
diff --git a/examples/C++/PaddleClas/resnet_50_vd/imagenet.label b/examples/C++/PaddleClas/resnet_50_vd/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_50_vd/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/C++/PaddleClas/resnet_50_vd/rpc_client.py b/examples/C++/PaddleClas/resnet_50_vd/rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..5e4c42841fccc257b057fd09c06ae495b0fa77bc
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_50_vd/rpc_client.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import Client
+
+#app
+from paddle_serving_app.reader import Sequential, URL2Image, Resize
+from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
+import time
+
+client = Client()
+client.load_client_config("./ResNet50_vd_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9394"])
+
+label_dict = {}
+label_idx = 0
+with open("imagenet.label") as fin:
+ for line in fin:
+ label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+#preprocess
+seq = Sequential([
+ URL2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+start = time.time()
+image_file = "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"
+for i in range(1):
+ img = seq(image_file)
+ fetch_map = client.predict(feed={"inputs": img}, fetch=[], batch=False)
+
+ prob = max(fetch_map["save_infer_model/scale_0.tmp_1"][0])
+ label = label_dict[fetch_map["save_infer_model/scale_0.tmp_1"][0].tolist()
+ .index(prob)].strip().replace(",", "")
+ print("prediction: {}, probability: {}".format(label, prob))
+end = time.time()
+print(end - start)
diff --git a/examples/C++/PaddleNLP/bert/bert_gpu_server.py b/examples/C++/PaddleNLP/bert/bert_gpu_server.py
old mode 100644
new mode 100755
index 7708a078636fd876c40e88d1441bc711d599f8a6..1d683e59cf05062065d1277827d4bc456e1031ea
--- a/examples/C++/PaddleNLP/bert/bert_gpu_server.py
+++ b/examples/C++/PaddleNLP/bert/bert_gpu_server.py
@@ -19,9 +19,9 @@ from paddle_serving_server import OpSeqMaker
from paddle_serving_server import Server
op_maker = OpMaker()
-read_op = op_maker.create('general_reader')
-general_infer_op = op_maker.create('general_infer')
-general_response_op = op_maker.create('general_response')
+read_op = op_maker.create('GeneralReaderOp')
+general_infer_op = op_maker.create('GeneralInferOp')
+general_response_op = op_maker.create('GeneralResponseOp')
op_seq_maker = OpSeqMaker()
op_seq_maker.add_op(read_op)
diff --git a/examples/C++/PaddleNLP/bert/bert_server.py b/examples/C++/PaddleNLP/bert/bert_server.py
old mode 100644
new mode 100755
index 35d38be0cac50b899b58085c7f103f32537859c4..c2ccd7ef2d662534b91c4d6acd255b237c05f26e
--- a/examples/C++/PaddleNLP/bert/bert_server.py
+++ b/examples/C++/PaddleNLP/bert/bert_server.py
@@ -19,9 +19,9 @@ from paddle_serving_server import OpSeqMaker
from paddle_serving_server import Server
op_maker = OpMaker()
-read_op = op_maker.create('general_reader')
-general_infer_op = op_maker.create('general_infer')
-general_response_op = op_maker.create('general_response')
+read_op = op_maker.create('GeneralReaderOp')
+general_infer_op = op_maker.create('GeneralInferOp')
+general_response_op = op_maker.create('GeneralResponseOp')
op_seq_maker = OpSeqMaker()
op_seq_maker.add_op(read_op)
diff --git a/examples/C++/PaddleOCR/ocr/README.md b/examples/C++/PaddleOCR/ocr/README.md
index 28849bf3278ace80340344a1661c620d70c7b062..12f0fd6cd5f06a97b0f5e846037217a40e37fe89 100755
--- a/examples/C++/PaddleOCR/ocr/README.md
+++ b/examples/C++/PaddleOCR/ocr/README.md
@@ -4,9 +4,9 @@
## Get Model
```
-wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/OCR/ocr_rec.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_rec
tar -xzvf ocr_rec.tar.gz
-wget https://paddle-serving.bj.bcebos.com/ocr/ocr_det.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_det
tar -xzvf ocr_det.tar.gz
```
@@ -108,7 +108,7 @@ python3 rec_web_client.py
When a service starts the concatenation of two models, it only needs to pass in the relative path of the model folder in order after `--model`, and the custom C++ OP class name after `--op`. The order of the model after `--model` and the class name after `--OP` needs to correspond. Here, it is assumed that we have defined the two OPs as GeneralDetectionOp and GeneralRecOp respectively, The script code is as follows:
```python
#One service starts the concatenation of two models
-python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --op GeneralDetectionOp GeneralRecOp --port 9293
+python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --op GeneralDetectionOp GeneralInferOp --port 9293
#ocr_det_model correspond to GeneralDetectionOp, ocr_rec_model correspond to GeneralRecOp
```
diff --git a/examples/C++/PaddleOCR/ocr/README_CN.md b/examples/C++/PaddleOCR/ocr/README_CN.md
index d9671115e0c4970af48fcd224551a96526be1356..a737531d778a25ef225c46240736e1bbc7381f35 100755
--- a/examples/C++/PaddleOCR/ocr/README_CN.md
+++ b/examples/C++/PaddleOCR/ocr/README_CN.md
@@ -4,9 +4,9 @@
## 获取模型
```
-wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/image/OCR/ocr_rec.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_rec
tar -xzvf ocr_rec.tar.gz
-wget https://paddle-serving.bj.bcebos.com/ocr/ocr_det.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_det
tar -xzvf ocr_det.tar.gz
```
## 获取数据集（可选）
@@ -106,7 +106,7 @@ python3 rec_web_client.py
一个服务启动两个模型串联，只需要在`--model后依次按顺序传入模型文件夹的相对路径`，且需要在`--op后依次传入自定义C++OP类名称`，其中--model后面的模型与--op后面的类名称的顺序需要对应，`这里假设我们已经定义好了两个OP分别为GeneralDetectionOp和GeneralRecOp`，则脚本代码如下：
```python
#一个服务启动多模型串联
-python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --op GeneralDetectionOp GeneralRecOp --port 9293
+python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --op GeneralDetectionOp GeneralInferOp --port 9293
#多模型串联 ocr_det_model对应GeneralDetectionOp ocr_rec_model对应GeneralRecOp
```
diff --git a/examples/C++/PaddleOCR/ocr/det_debugger_server.py b/examples/C++/PaddleOCR/ocr/det_debugger_server.py
index 5b40fe9372a56b2b663c1bfeff02619a8ec9730b..6679ee0f4e0f18000ee33e5331a3d2e44197c846 100644
--- a/examples/C++/PaddleOCR/ocr/det_debugger_server.py
+++ b/examples/C++/PaddleOCR/ocr/det_debugger_server.py
@@ -47,18 +47,18 @@ class OCRService(WebService):
})
def preprocess(self, feed=[], fetch=[]):
- data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = base64.b64decode(feed[0]["x"].encode('utf8'))
data = np.fromstring(data, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
self.ori_h, self.ori_w, _ = im.shape
det_img = self.det_preprocess(im)
_, self.new_h, self.new_w = det_img.shape
return {
- "image": det_img[np.newaxis, :].copy()
- }, ["concat_1.tmp_0"], True
+ "x": det_img[np.newaxis, :].copy()
+ }, ["save_infer_model/scale_0.tmp_1"], True
def postprocess(self, feed={}, fetch=[], fetch_map=None):
- det_out = fetch_map["concat_1.tmp_0"]
+ det_out = fetch_map["save_infer_model/scale_0.tmp_1"]
ratio_list = [
float(self.new_h) / self.ori_h, float(self.new_w) / self.ori_w
]
diff --git a/examples/C++/PaddleOCR/ocr/det_web_server.py b/examples/C++/PaddleOCR/ocr/det_web_server.py
index d38686e5a86c4f2df45db7f495a8c08a72270919..d52f4b447006e220ed2f6362afee253c0b9eb69d 100644
--- a/examples/C++/PaddleOCR/ocr/det_web_server.py
+++ b/examples/C++/PaddleOCR/ocr/det_web_server.py
@@ -47,17 +47,17 @@ class OCRService(WebService):
})
def preprocess(self, feed=[], fetch=[]):
- data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = base64.b64decode(feed[0]["x"].encode('utf8'))
data = np.fromstring(data, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
self.ori_h, self.ori_w, _ = im.shape
det_img = self.det_preprocess(im)
_, self.new_h, self.new_w = det_img.shape
print(det_img)
- return {"image": det_img}, ["concat_1.tmp_0"], False
+ return {"x": det_img}, ["save_infer_model/scale_0.tmp_1"], False
def postprocess(self, feed={}, fetch=[], fetch_map=None):
- det_out = fetch_map["concat_1.tmp_0"]
+ det_out = fetch_map["save_infer_model/scale_0.tmp_1"]
ratio_list = [
float(self.new_h) / self.ori_h, float(self.new_w) / self.ori_w
]
diff --git a/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py b/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py
index b3187f50e6f0d677d5377dac1735bd6e679b4755..507971c36c0e900cce471ceb2c636f4dd6232ccd 100644
--- a/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py
+++ b/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py
@@ -42,13 +42,11 @@ for img_file in os.listdir(test_img_dir):
image_data = file.read()
image = cv2_to_base64(image_data)
fetch_map = client.predict(
- feed={"image": image},
- fetch=["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"],
+ feed={"x": image},
+ fetch=["save_infer_model/scale_0.tmp_1"],
batch=True)
result = {}
- result["score"] = fetch_map["softmax_0.tmp_0"]
- del fetch_map["softmax_0.tmp_0"]
- rec_res = OCRReader().postprocess(fetch_map, with_score=False)
+ rec_res = OCRReader().postprocess_ocrv2(fetch_map, with_score=False)
res_lst = []
for res in rec_res:
res_lst.append(res[0])
diff --git a/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py b/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py
index 88dd94a8224fc5c9c6f972b96d81af60ce518763..bb10dba44d4baf3a9ed1e6b1f2f9af02178c1261 100644
--- a/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py
+++ b/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py
@@ -48,7 +48,7 @@ class OCRService(WebService):
self.ocr_reader = OCRReader()
def preprocess(self, feed=[], fetch=[]):
- data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = base64.b64decode(feed[0]["x"].encode('utf8'))
data = np.fromstring(data, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
ori_h, ori_w, _ = im.shape
@@ -57,7 +57,7 @@ class OCRService(WebService):
det_img = det_img[np.newaxis, :]
det_img = det_img.copy()
det_out = self.det_client.predict(
- feed={"image": det_img}, fetch=["concat_1.tmp_0"], batch=True)
+ feed={"x": det_img}, fetch=["save_infer_model/scale_0.tmp_1"], batch=True)
filter_func = FilterBoxes(10, 10)
post_func = DBPostProcess({
"thresh": 0.3,
@@ -68,7 +68,7 @@ class OCRService(WebService):
})
sorted_boxes = SortedBoxes()
ratio_list = [float(new_h) / ori_h, float(new_w) / ori_w]
- dt_boxes_list = post_func(det_out["concat_1.tmp_0"], [ratio_list])
+ dt_boxes_list = post_func(det_out["save_infer_model/scale_0.tmp_1"], [ratio_list])
dt_boxes = filter_func(dt_boxes_list[0], [ori_h, ori_w])
dt_boxes = sorted_boxes(dt_boxes)
get_rotate_crop_image = GetRotateCropImage()
@@ -88,12 +88,12 @@ class OCRService(WebService):
for id, img in enumerate(img_list):
norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
imgs[id] = norm_img
- feed = {"image": imgs.copy()}
- fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ feed = {"x": imgs.copy()}
+ fetch = ["save_infer_model/scale_0.tmp_1"]
return feed, fetch, True
def postprocess(self, feed={}, fetch=[], fetch_map=None):
- rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ rec_res = self.ocr_reader.postprocess_ocrv2(fetch_map, with_score=True)
res_lst = []
for res in rec_res:
res_lst.append(res[0])
diff --git a/examples/C++/PaddleOCR/ocr/ocr_web_client.py b/examples/C++/PaddleOCR/ocr/ocr_web_client.py
index ce96a8bbcd585f37368d70070d649e25a0129029..91620e8ed9b0973a6bb31b09afbdce7b99aac8b6 100644
--- a/examples/C++/PaddleOCR/ocr/ocr_web_client.py
+++ b/examples/C++/PaddleOCR/ocr/ocr_web_client.py
@@ -34,7 +34,7 @@ for img_file in os.listdir(test_img_dir):
with open(os.path.join(test_img_dir, img_file), 'rb') as file:
image_data1 = file.read()
image = cv2_to_base64(image_data1)
- data = {"feed": [{"image": image}], "fetch": ["res"]}
+ data = {"feed": [{"x": image}], "fetch": ["res"]}
r = requests.post(url=url, headers=headers, data=json.dumps(data))
print(r)
print(r.json())
diff --git a/examples/C++/PaddleOCR/ocr/ocr_web_server.py b/examples/C++/PaddleOCR/ocr/ocr_web_server.py
index 58fc850c94a5e8d2f37ae5d03f14b60d343a2203..2273c7e1d2d90a1f4d3dd0a14a8469974a09ba98 100644
--- a/examples/C++/PaddleOCR/ocr/ocr_web_server.py
+++ b/examples/C++/PaddleOCR/ocr/ocr_web_server.py
@@ -44,13 +44,13 @@ class OCRService(WebService):
self.ocr_reader = OCRReader()
def preprocess(self, feed=[], fetch=[]):
- data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = base64.b64decode(feed[0]["x"].encode('utf8'))
data = np.fromstring(data, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
ori_h, ori_w, _ = im.shape
det_img = self.det_preprocess(im)
det_out = self.det_client.predict(
- feed={"image": det_img}, fetch=["concat_1.tmp_0"], batch=False)
+ feed={"x": det_img}, fetch=["save_infer_model/scale_0.tmp_1"], batch=False)
_, new_h, new_w = det_img.shape
filter_func = FilterBoxes(10, 10)
post_func = DBPostProcess({
@@ -62,7 +62,7 @@ class OCRService(WebService):
})
sorted_boxes = SortedBoxes()
ratio_list = [float(new_h) / ori_h, float(new_w) / ori_w]
- dt_boxes_list = post_func(det_out["concat_1.tmp_0"], [ratio_list])
+ dt_boxes_list = post_func(det_out["save_infer_model/scale_0.tmp_1"], [ratio_list])
dt_boxes = filter_func(dt_boxes_list[0], [ori_h, ori_w])
dt_boxes = sorted_boxes(dt_boxes)
get_rotate_crop_image = GetRotateCropImage()
@@ -78,12 +78,12 @@ class OCRService(WebService):
for img in img_list:
norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
feed_list.append(norm_img[np.newaxis, :])
- feed_batch = {"image": np.concatenate(feed_list, axis=0)}
- fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ feed_batch = {"x": np.concatenate(feed_list, axis=0)}
+ fetch = ["save_infer_model/scale_0.tmp_1"]
return feed_batch, fetch, True
def postprocess(self, feed={}, fetch=[], fetch_map=None):
- rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ rec_res = self.ocr_reader.postprocess_ocrv2(fetch_map, with_score=True)
res_lst = []
for res in rec_res:
res_lst.append(res[0])
diff --git a/examples/C++/PaddleOCR/ocr/rec_debugger_server.py b/examples/C++/PaddleOCR/ocr/rec_debugger_server.py
index f84463238af859a00983f515e405686c00fdf9fa..cb096ac1a7ab9475da0d61fb396bdec471fc2f2d 100644
--- a/examples/C++/PaddleOCR/ocr/rec_debugger_server.py
+++ b/examples/C++/PaddleOCR/ocr/rec_debugger_server.py
@@ -38,7 +38,7 @@ class OCRService(WebService):
def preprocess(self, feed=[], fetch=[]):
img_list = []
for feed_data in feed:
- data = base64.b64decode(feed_data["image"].encode('utf8'))
+ data = base64.b64decode(feed_data["x"].encode('utf8'))
data = np.fromstring(data, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
img_list.append(im)
@@ -53,12 +53,12 @@ class OCRService(WebService):
for i, img in enumerate(img_list):
norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
imgs[i] = norm_img
- feed = {"image": imgs.copy()}
- fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ feed = {"x": imgs.copy()}
+ fetch = ["save_infer_model/scale_0.tmp_1"]
return feed, fetch, True
def postprocess(self, feed={}, fetch=[], fetch_map=None):
- rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ rec_res = self.ocr_reader.postprocess_ocrv2(fetch_map, with_score=True)
res_lst = []
for res in rec_res:
res_lst.append(res[0])
diff --git a/examples/C++/PaddleOCR/ocr/rec_web_client.py b/examples/C++/PaddleOCR/ocr/rec_web_client.py
index 312a2148886d6f084a1c077d84e907cb28c0652a..e78145b60d0c2d6c6032acbdba679bbdf89df51b 100644
--- a/examples/C++/PaddleOCR/ocr/rec_web_client.py
+++ b/examples/C++/PaddleOCR/ocr/rec_web_client.py
@@ -36,6 +36,6 @@ for img_file in os.listdir(test_img_dir):
image_data1 = file.read()
image = cv2_to_base64(image_data1)
#data = {"feed": [{"image": image}], "fetch": ["res"]}
- data = {"feed": [{"image": image}] * 3, "fetch": ["res"]}
+ data = {"feed": [{"x": image}] * 3, "fetch": ["res"]}
r = requests.post(url=url, headers=headers, data=json.dumps(data))
print(r.json())
diff --git a/examples/C++/PaddleOCR/ocr/rec_web_server.py b/examples/C++/PaddleOCR/ocr/rec_web_server.py
index 2db6e398d3a025e739761fabd50c5bb8a6609f07..1a6e45812c42280100fb14b029a3a16508c3b9a5 100644
--- a/examples/C++/PaddleOCR/ocr/rec_web_server.py
+++ b/examples/C++/PaddleOCR/ocr/rec_web_server.py
@@ -39,7 +39,7 @@ class OCRService(WebService):
# TODO: to handle batch rec images
img_list = []
for feed_data in feed:
- data = base64.b64decode(feed_data["image"].encode('utf8'))
+ data = base64.b64decode(feed_data["x"].encode('utf8'))
data = np.fromstring(data, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
img_list.append(im)
@@ -55,12 +55,12 @@ class OCRService(WebService):
norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
imgs[i] = norm_img
- feed = {"image": imgs.copy()}
- fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ feed = {"x": imgs.copy()}
+ fetch = ["save_infer_model/scale_0.tmp_1"]
return feed, fetch, True
def postprocess(self, feed={}, fetch=[], fetch_map=None):
- rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ rec_res = self.ocr_reader.postprocess_ocrv2(fetch_map, with_score=True)
res_lst = []
for res in rec_res:
res_lst.append(res[0])
diff --git a/examples/C++/PaddleRec/criteo_ctr/test_server.py b/examples/C++/PaddleRec/criteo_ctr/test_server.py
old mode 100644
new mode 100755
index 34f859daab4c808aa9d50d2109a81a69eed96df6..ca4c6afb0ddc0da3e24540797f242c748d4b128f
--- a/examples/C++/PaddleRec/criteo_ctr/test_server.py
+++ b/examples/C++/PaddleRec/criteo_ctr/test_server.py
@@ -20,14 +20,14 @@ from paddle_serving_server import OpSeqMaker
from paddle_serving_server import Server
op_maker = OpMaker()
-read_op = op_maker.create('general_reader')
-general_infer_op = op_maker.create('general_infer')
-response_op = op_maker.create('general_response')
+read_op = op_maker.create('GeneralReaderOp')
+general_infer_op = op_maker.create('GeneralInferOp')
+general_response_op = op_maker.create('GeneralResponseOp')
op_seq_maker = OpSeqMaker()
op_seq_maker.add_op(read_op)
op_seq_maker.add_op(general_infer_op)
-op_seq_maker.add_op(response_op)
+op_seq_maker.add_op(general_response_op)
server = Server()
server.set_op_sequence(op_seq_maker.get_op_sequence())
diff --git a/examples/C++/imdb/README_CN.md b/examples/C++/imdb/README_CN.md
index a1fecc8af35dcd2f5a38f47480b9b80b3cf96054..42841c6b8d0b4d1f064a33b54503193b88eed635 100755
--- a/examples/C++/imdb/README_CN.md
+++ b/examples/C++/imdb/README_CN.md
@@ -1,4 +1,4 @@
-## IMDB评论情绪预测服务
+## IMDB 评论情绪预测 ABTest 服务
(简体中文|[English](./README.md))
@@ -11,16 +11,24 @@ sh get_data.sh
### 启动预测服务(支持BRPC-Client/GRPC-Client/Http-Client)
+```python
+## 启动 bow 模型服务
+python3 -m paddle_serving_server.serve --model imdb_bow_model/ --port 9297 >/dev/null 2>&1 &
+
+## 启动 cnn 模型服务
+python3 -m paddle_serving_server.serve --model imdb_cnn_model/ --port 9298 >/dev/null 2>&1 &
+
+## 启动 lstm 模型服务
+python3 -m paddle_serving_server.serve --model imdb_lstm_model/ --port 9299 >/dev/null 2>&1 &
```
-python3 -m paddle_serving_server.serve --model imdb_cnn_model/ --port 9292
-```
-### BRPC-Client预测
+
+### ABTest 预测
```
-head test_data/part-0 | python3 test_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
+head test_data/part-0 | python3 abtest_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
```
预测test_data/part-0的前十个样例。
-### BRPC-Client预测
+### http预测
```
head test_data/part-0 | python3 test_http_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
```
diff --git a/examples/C++/imdb/abtest_client.py b/examples/C++/imdb/abtest_client.py
index 1a14c87c355552248394fa504d37f54a4c58132a..e0f910e37c7cf5a99c0b5ec1249f0ceed68f21ae 100644
--- a/examples/C++/imdb/abtest_client.py
+++ b/examples/C++/imdb/abtest_client.py
@@ -11,35 +11,35 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
+# pylint: disable=doc-string-missing
from paddle_serving_client import Client
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+import sys
import numpy as np
client = Client()
-client.load_client_config('imdb_bow_client_conf/serving_client_conf.prototxt')
-client.add_variant("bow", ["127.0.0.1:8000"], 10)
-client.add_variant("lstm", ["127.0.0.1:9000"], 90)
+client.load_client_config(sys.argv[1])
+client.add_variant("bow", ["127.0.0.1:9297"], 10)
+client.add_variant("cnn", ["127.0.0.1:9298"], 30)
+client.add_variant("lstm", ["127.0.0.1:9299"], 60)
client.connect()
-print('please wait for about 10s')
-with open('processed.data') as f:
- cnt = {"bow": {'acc': 0, 'total': 0}, "lstm": {'acc': 0, 'total': 0}}
- for line in f:
- word_ids, label = line.split(';')
- word_ids = [int(x) for x in word_ids.split(',')]
- word_len = len(word_ids)
- feed = {
- "words": np.array(word_ids).reshape(word_len, 1),
- "words.lod": [0, word_len]
- }
- fetch = ["acc", "cost", "prediction"]
- [fetch_map, tag] = client.predict(
- feed=feed, fetch=fetch, need_variant_tag=True, batch=True)
- if (float(fetch_map["prediction"][0][1]) - 0.5) * (float(label[0]) - 0.5
- ) > 0:
- cnt[tag]['acc'] += 1
- cnt[tag]['total'] += 1
+# you can define any english sentence or dataset here
+# This example reuses imdb reader in training, you
+# can define your own data preprocessing easily.
+imdb_dataset = IMDBDataset()
+imdb_dataset.load_resource(sys.argv[2])
- for tag, data in cnt.items():
- print('[{}](total: {}) acc: {}'.format(tag, data[
- 'total'], float(data['acc']) / float(data['total'])))
+for line in sys.stdin:
+ word_ids, label = imdb_dataset.get_words_and_label(line)
+ word_len = len(word_ids)
+ feed = {
+ "words": np.array(word_ids).reshape(word_len, 1),
+ "words.lod": [0, word_len]
+ }
+ #print(feed)
+ fetch = ["prediction"]
+ fetch_map = client.predict(
+ feed=feed, fetch=fetch, batch=True, need_variant_tag=True)
+ print("server_tag={} prediction={} ".format(fetch_map[1], fetch_map[0][
+ "prediction"][0]))
diff --git a/examples/C++/imdb/test_http_client.py b/examples/C++/imdb/test_http_client.py
index e3cc705150ccc197ab1be24bf11e0a92e1d62380..d22b92d7850097130e24d3cf86857ddcc8caef2f 100755
--- a/examples/C++/imdb/test_http_client.py
+++ b/examples/C++/imdb/test_http_client.py
@@ -40,7 +40,7 @@ we recommend use Proto data format in HTTP-body, set True(which is default)
if you want use JSON data format in HTTP-body, set False
'''
#client.set_http_proto(True)
-client.connect(["127.0.0.1:9292"])
+client.connect(["127.0.0.1:9297"])
# you can define any english sentence or dataset here
# This example reuses imdb reader in training, you
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/README.md b/examples/Pipeline/LowPrecision/ResNet50_Slim/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..8b115baee975c2a96eb6c1f4b43dcf8059fe4fea
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/README.md
@@ -0,0 +1,25 @@
+# Low precsion examples of python pipeline
+
+Here we take the ResNet50 quantization model as an example to introduce the low-precision deployment case of Python Pipline.
+
+## 1.Get model
+```
+wget https://paddle-inference-dist.bj.bcebos.com/inference_demo/python/resnet50/ResNet50_quant.tar.gz
+tar zxvf ResNet50_quant.tar.gz
+```
+
+## 2.Save model var for serving
+```
+python3 -m paddle_serving_client.convert --dirname ResNet50_quant --serving_server serving_server --serving_client serving_client
+```
+
+## 3.Start server
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 4.Test
+```
+python3 pipeline_rpc_client.py
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/README_CN.md b/examples/Pipeline/LowPrecision/ResNet50_Slim/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..1396b29858710eed8bc54a9a27f05022ef3a443f
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/README_CN.md
@@ -0,0 +1,25 @@
+# Python Pipeline 低精度部署案例
+
+这里以 ResNet50 量化模型为例，介绍 Python Pipline 低精度量化模型部署案例。
+
+## 1.获取模型
+```
+wget https://paddle-inference-dist.bj.bcebos.com/inference_demo/python/resnet50/ResNet50_quant.tar.gz
+tar zxvf ResNet50_quant.tar.gz
+```
+
+## 2.保存模型参数
+```
+python3 -m paddle_serving_client.convert --dirname ResNet50_quant --serving_server serving_server --serving_client serving_client
+```
+
+## 3.启动服务
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 4.测试
+```
+python3 pipeline_rpc_client.py
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/benchmark.py b/examples/Pipeline/LowPrecision/ResNet50_Slim/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..4b0336f97c2c520a46d596bf5e435c2b9e3094a9
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18000/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/benchmark.sh b/examples/Pipeline/LowPrecision/ResNet50_Slim/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..14c82dfcca801bc00bec57ef972f2260dd1d844a
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNet_v2_50"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/config.yml b/examples/Pipeline/LowPrecision/ResNet50_Slim/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..eb5e918b9b0ec8780e833c2ef01150fafcbc312d
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/config.yml
@@ -0,0 +1,44 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 10
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: serving_server/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 2
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["save_infer_model/scale_0.tmp_0"]
+
+ #precsion, 预测精度，降低预测精度可提升预测速度
+ #GPU 支持: "fp32"(default), "fp16", "int8"；
+ #CPU 支持: "fp32"(default), "fp16", "bf16"(mkldnn); 不支持: "int8"
+ precision: "int8"
+
+ #开启 TensorRT calibration, 量化模型要设置 use_calib: False, 非量化模型离线生成int8需要开启 use_calib: True
+ use_calib: False
+
+ #开启 ir_optim
+ ir_optim: True
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/daisy.jpg b/examples/Pipeline/LowPrecision/ResNet50_Slim/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/LowPrecision/ResNet50_Slim/daisy.jpg differ
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/imagenet.label b/examples/Pipeline/LowPrecision/ResNet50_Slim/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/pipeline_http_client.py b/examples/Pipeline/LowPrecision/ResNet50_Slim/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..119a412113dac95621ef8fcad06398fed40a3da0
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/pipeline_http_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(1):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/pipeline_rpc_client.py b/examples/Pipeline/LowPrecision/ResNet50_Slim/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/LowPrecision/ResNet50_Slim/resnet50_web_service.py b/examples/Pipeline/LowPrecision/ResNet50_Slim/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..0a9c5f97e3536944ee5c7f0bbae1448a19bf4eb7
--- /dev/null
+++ b/examples/Pipeline/LowPrecision/ResNet50_Slim/resnet50_web_service.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["save_infer_model/scale_0.tmp_0"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleNLP/bert/README.md b/examples/Pipeline/PaddleNLP/bert/README.md
index c396b77c9d2b9198d0474540872cb1c4dcdce5b1..0fb8739b430f49f8bc066eb9a357ee92850b89d3 100644
--- a/examples/Pipeline/PaddleNLP/bert/README.md
+++ b/examples/Pipeline/PaddleNLP/bert/README.md
@@ -1,6 +1,6 @@
-# Imagenet Pipeline WebService
+# Bert Pipeline WebService
-This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+This document will takes Bert service as an example to introduce how to use Pipeline WebService.
## Get model
```
diff --git a/examples/Pipeline/PaddleNLP/bert/README_CN.md b/examples/Pipeline/PaddleNLP/bert/README_CN.md
index 841abdadf5a3848fcf1e042d8e73c051610eefaa..eb93951d50ffb0c3bd899d816294ffde66e646a9 100644
--- a/examples/Pipeline/PaddleNLP/bert/README_CN.md
+++ b/examples/Pipeline/PaddleNLP/bert/README_CN.md
@@ -1,6 +1,6 @@
-# Imagenet Pipeline WebService
+# Bert Pipeline WebService
-这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+这里以 Bert 服务为例来介绍 Pipeline WebService 的使用。
## 获取模型
```
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/README.md b/examples/Pipeline/PaddleNLP/semantic_indexing/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..20e0bc04a2b0de6c3fb21355b8636de73c625d42
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/README.md
@@ -0,0 +1,201 @@
+# In-batch Negatives
+
+ **目录**
+
+* [模型下载](#模型下载)
+* [模型部署](#模型部署)
+
+
+
+
+## 1. 语义索引模型
+
+**语义索引训练模型下载链接：**
+
+以下模型结构参数为: `TrasformerLayer:12, Hidden:768, Heads:12, OutputEmbSize: 256`
+
+|Model|训练参数配置|硬件|MD5|
+| ------------ | ------------ | ------------ |-----------|
+|[batch_neg](https://bj.bcebos.com/v1/paddlenlp/models/inbatch_model.zip)|
diff --git a/cmake/paddlepaddle.cmake b/cmake/paddlepaddle.cmake
index ad8df0f1844e902a632f7c7df390ec3c2a783345..c9d4a583944072dcddc25f6d8544f47b110acab5 100644
--- a/cmake/paddlepaddle.cmake
+++ b/cmake/paddlepaddle.cmake
@@ -30,7 +30,7 @@ message( "WITH_GPU = ${WITH_GPU}")
# Paddle Version should be one of:
# latest: latest develop build
# version number like 1.5.2
-SET(PADDLE_VERSION "2.2.2")
+SET(PADDLE_VERSION "2.3.0")
if (WITH_GPU)
message("CUDA: ${CUDA_VERSION}, CUDNN_MAJOR_VERSION: ${CUDNN_MAJOR_VERSION}")
# cuda 11.0 is not supported, 11.2 would be added.
@@ -53,6 +53,7 @@ else()
set(WITH_TRT OFF)
endif()
if (WITH_GPU)
+ SET(PADDLE_VERSION "2.3.0-no-ort")
SET(PADDLE_LIB_VERSION "${PADDLE_VERSION}/cxx_c/Linux/GPU/${CUDA_SUFFIX}")
elseif (WITH_LITE)
message("cpu arch: ${CMAKE_SYSTEM_PROCESSOR}")
@@ -85,6 +86,7 @@ elseif (WITH_ASCEND_CL)
endif()
else()
if (WITH_AVX)
+ SET(PADDLE_VERSION "2.3.0-no-ort")
if (WITH_MKLML)
SET(PADDLE_LIB_VERSION "${PADDLE_VERSION}/cxx_c/Linux/CPU/gcc8.2_avx_mkl")
else()
@@ -100,7 +102,7 @@ endif()
if(WITH_LITE)
if (WITH_XPU)
- SET(PADDLE_LIB_PATH "https://paddle-inference-lib.bj.bcebos.com/${PADDLE_LIB_VERSION}/paddle_inference_install_dir.tar.gz ")
+ SET(PADDLE_LIB_PATH "https://paddle-serving.bj.bcebos.com/inferlib/${PADDLE_LIB_VERSION}/paddle_inference_install_dir.tar.gz ")
elseif (WITH_ASCEND_CL)
SET(PADDLE_LIB_PATH "http://paddle-serving.bj.bcebos.com/inferlib/${PADDLE_LIB_VERSION}/paddle_inference_install_dir.tgz ")
endif()
@@ -113,7 +115,7 @@ else()
endif()
MESSAGE(STATUS "PADDLE_LIB_PATH=${PADDLE_LIB_PATH}")
-if (WITH_GPU OR WITH_MKLML)
+if ((WITH_GPU OR WITH_MKLML) AND NOT WITH_JETSON)
if (WITH_TRT)
ExternalProject_Add(
"extern_paddle"
@@ -171,14 +173,27 @@ LINK_DIRECTORIES(${PADDLE_INSTALL_DIR}/third_party/install/mklml/lib)
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}" "${PADDLE_INSTALL_DIR}/third_party/install/mkldnn/lib")
LINK_DIRECTORIES(${PADDLE_INSTALL_DIR}/third_party/install/mkldnn/lib)
+#SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}" "${PADDLE_INSTALL_DIR}/third_party/install/paddle2onnx/lib")
+#LINK_DIRECTORIES(${PADDLE_INSTALL_DIR}/third_party/install/paddle2onnx/lib)
+
+#SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}" "${PADDLE_INSTALL_DIR}/third_party/install/onnxruntime/lib")
+#LINK_DIRECTORIES(${PADDLE_INSTALL_DIR}/third_party/install/onnxruntime/lib)
+
if (NOT WITH_MKLML)
ADD_LIBRARY(openblas STATIC IMPORTED GLOBAL)
SET_PROPERTY(TARGET openblas PROPERTY IMPORTED_LOCATION ${PADDLE_INSTALL_DIR}/third_party/install/openblas/lib/libopenblas.a)
endif()
+#ADD_LIBRARY(paddle2onnx STATIC IMPORTED GLOBAL)
+#SET_PROPERTY(TARGET paddle2onnx PROPERTY IMPORTED_LOCATION ${PADDLE_INSTALL_DIR}/third_party/install/paddle2onnx/lib/libpaddle2onnx.so)
+
+#ADD_LIBRARY(onnxruntime STATIC IMPORTED GLOBAL)
+#SET_PROPERTY(TARGET onnxruntime PROPERTY IMPORTED_LOCATION ${PADDLE_INSTALL_DIR}/third_party/install/onnxruntime/lib/libonnxruntime.so.1.10.0)
+
ADD_LIBRARY(paddle_inference STATIC IMPORTED GLOBAL)
SET_PROPERTY(TARGET paddle_inference PROPERTY IMPORTED_LOCATION ${PADDLE_INSTALL_DIR}/lib/libpaddle_inference.a)
-if (WITH_ASCEND_CL)
+
+if (WITH_ASCEND_CL OR WITH_XPU)
SET_PROPERTY(TARGET paddle_inference PROPERTY IMPORTED_LOCATION ${PADDLE_INSTALL_DIR}/lib/libpaddle_inference.so)
endif()
diff --git a/core/configure/proto/general_model_service.proto b/core/configure/proto/general_model_service.proto
index c2deab2f69ea6f6ca5e77354ec955bf679f9a3d6..b4f1ce0cfbb1f407a217e1a72440b55489e8a1c3 100644
--- a/core/configure/proto/general_model_service.proto
+++ b/core/configure/proto/general_model_service.proto
@@ -90,11 +90,12 @@ message Request {
message Response {
repeated ModelOutput outputs = 1;
repeated int64 profile_time = 2;
+ bool profile_server = 3;
+ uint64 log_id = 4;
// Error code
- int32 err_no = 3;
-
+ int32 err_no = 5;
// Error messages
- string err_msg = 4;
+ string err_msg = 6;
};
message ModelOutput {
diff --git a/core/configure/proto/server_configure.proto b/core/configure/proto/server_configure.proto
old mode 100755
new mode 100644
index c974f010737a8836d5de83d737ee0f9b9519462f..4f49aa3c959a63afbb623853bcc1f0c14cbd52e3
--- a/core/configure/proto/server_configure.proto
+++ b/core/configure/proto/server_configure.proto
@@ -49,6 +49,17 @@ message EngineDesc {
optional bool gpu_multi_stream = 20;
optional bool use_ascend_cl = 21;
+ /*
+ * "gpu_memory_mb": allocate gpu memory by config.EnableUseGpu()
+ * "cpu_math_thread_num": set thread numbers of cpu math by config.SetCpuMathLibraryNumThreads()
+ * "trt_workspace_size": set TensorRT workspace size by config.EnableTensorRtEngine(), 1 << 25 default
+ * "trt_use_static": If true, save the optimization information of the TRT serialized to the disk, and load from the disk.
+ */
+ optional int32 gpu_memory_mb = 22 [default = 100];
+ optional int32 cpu_math_thread_num = 23 [default = 1];
+ optional int32 trt_workspace_size = 24 [default = 33554432];
+ optional bool trt_use_static = 25 [default = false];
+
/*
* "runtime_thread_num": n == 0 means don`t use Asynchronous task scheduling
* mode.
@@ -65,6 +76,28 @@ message EngineDesc {
optional int32 batch_infer_size = 31 [ default = 32 ];
optional bool enable_overrun = 32 [ default = false ];
optional bool allow_split_request = 33 [ default = true ];
+ optional int32 min_subgraph_size = 34 [ default = 3 ];
+ map
+
+
+
+
+
+**二.测试数据**
+
+1. 同步模式
+
+| client_num | batch_size |CPU_util_pre(%) |CPU_util(%) |GPU_memory(mb) |GPU_util(%) |qps(samples/s) |total count |mean(ms) |median(ms) |80 percent(ms) |90 percent(ms) |99 percent(ms) |total cost(s) |each cost(s)|infer_count_total|infer_cost_total(ms)|infer_cost_avg(ms)|
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| 1 |1 |1.30 |18.90 |2066 |71.56 |22.938 |100 |43.594 |23.516 |78.118 |78.323 |133.544 |4.4262 |4.3596 |7100.0000 |1666392.70 | 41.1081 |
+| 5 |1 |2.00 |28.20 |3668 |92.57 |33.630 |500 |148.673 |39.531 |373.231 |396.306 |419.088 |15.0606 |14.8676 |7600.0000 |1739372.7480| 145.9601 |
+|10 |1 |1.90 |29.80 |4202 |91.98 |34.303 |1000 |291.512 |76.728 |613.963 |632.736 |1217.863 |29.8004 |29.1516 |8600.0000 |1974147.7420| 234.7750 |
+|20 |1 |4.70 |49.60 |4736 |92.63 |34.359 |2000 |582.089 |154.952 |1239.115 |1813.371 |1858.128 |59.7303 |58.2093 |12100.0000 |2798459.6330 |235.6248 |
+|30 |1 |5.70 |65.70 |4736 |92.60 |34.162 |3000 |878.164 |231.121 |2391.687 |2442.744 |2499.963 |89.6546 |87.8168 |17600.0000 |4100408.9560 |236.6877 |
+|40 |1 |5.40 |74.40 |5270 |92.44 |34.090 |4000 |1173.373 |306.244 |3037.038 |3070.198 |3134.894 |119.4162 |117.3377 |21600.0000 |5048139.2170 |236.9326|
+|50 |1 |1.40 |64.70 |5270 |92.37 |34.031 |5000 |1469.250 |384.327 |3676.812 |3784.330 |4366.862 |149.7041 |146.9254 |26600.0000 |6236269.4230 |237.6260|
+|70 |1 |3.70 |79.70 |5270 |91.89 |33.976 |7000 |2060.246 |533.439 |5429.255 |5552.704 |5661.492 |210.1008 |206.0250 |33600.0000 |7905005.9940 |238.3909|
+
+
+2. 异步模式 - 未开启动态批量
+
+| client_num | batch_size |CPU_util_pre(%) |CPU_util(%) |GPU_memory(mb) |GPU_util(%) |qps(samples/s) |total count |mean(ms) |median(ms) |80 percent(ms) |90 percent(ms) |99 percent(ms) |total cost(s) |each cost(s)|infer_count_total|infer_cost_total(ms)|infer_cost_avg(ms)|
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| 1 |1 |6.20 |13.60 |5170 |71.11 |22.894 |100 |43.677 |23.992 |78.285 |78.788 |123.542 |4.4253 |4.3679 |3695.0000 |745061.9120 |40.6655 |
+| 5 |1 |6.10 |32.20 |7306 |89.54 |33.532 |500 |149.109 |43.906 |376.889 |401.999 |422.753 |15.1623 |14.9113 |4184.0000 |816834.2250 |146.7736|
+|10 |1 |4.90 |43.60 |7306 |91.55 |38.136 |1000 |262.216 |75.393 |575.788 |632.016 |1247.775 |27.1019 |26.2220 |5107.0000 |1026490.3950 |227.1464|
+|20 |1 |5.70 |39.60 |7306 |91.36 |58.601 |2000 |341.287 |145.774 |646.824 |994.748 |1132.979 |38.3915 |34.1291 |7461.0000 |1555234.6260 |229.9113|
+|30 |1 |1.30 |45.40 |7484 |91.10 |69.008 |3000 |434.728 |204.347 |959.184 |1092.181 |1661.289 |46.3822 |43.4732 |10289.0000 |2269499.9730 |249.4257|
+|40 |1 |3.10 |73.00 |7562 |91.83 |80.956 |4000 |494.091 |272.889 |966.072 |1310.011 |1851.887 |52.0609 |49.4095 |12102.0000 |2678878.2010 |225.8016|
+|50 |1 |0.80 |68.00 |7522 |91.10 |83.018 |5000 |602.276 |364.064 |1058.261 |1473.051 |1671.025 |72.9869 |60.2280 |14225.0000 |3256628.2820 |272.1385|
+|70 |1 |6.10 |78.40 |7584 |92.02 |65.069 |7000 |1075.777 |474.014 |2411.296 |2705.863 |3409.085 |111.6653 |107.5781 |17974.0000 |4139377.4050 |235.4626
+
+
+
+3. 异步模式 - 开启动态批量
+
+
+| client_num | batch_size |CPU_util_pre(%) |CPU_util(%) |GPU_memory(mb) |GPU_util(%) |qps(samples/s) |total count |mean(ms) |median(ms) |80 percent(ms) |90 percent(ms) |99 percent(ms) |total cost(s) |each cost(s)|infer_count_total|infer_cost_total(ms)|infer_cost_avg(ms)|
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| 1 |1 |1.20 |13.30 |6048 |70.07 |22.417 |100 |44.606 |24.486 |78.365 |78.707 |139.349 |4.5201 |4.4608 |1569.0000 |462418.6390 |41.7646 |
+| 5 |1 |1.20 |50.80 |7116 |87.37 |31.106 |500 |160.740 |42.506 |414.903 |458.841 |481.112 |16.3525 |16.0743 |2059.0000 |539439.3300 |157.1851
+|10 |1 |0.80 |26.20 |7264 |88.74 |37.417 |1000 |267.254 |79.452 |604.451 |686.477 |1345.528 |27.9848 |26.7258 |2950.0000 |752428.0570 |239.0446|
+|20 |1 |1.50 |32.80 |7264 |89.52 |70.641 |2000 |283.117 |133.441 |516.066 |652.089 |1274.957 |33.0280 |28.3121 |4805.0000 |1210814.5610 |260.5873|
+|30 |1 |0.90 |59.10 |7348 |89.57 |84.894 |3000 |353.380 |217.385 |613.587 |757.829 |1277.283 |40.7093 |35.3384 |6924.0000 |1817515.1710 |276.3695|
+|40 |1 |1.30 |57.30 |7356 |89.30 |99.853 |4000 |400.584 |204.425 |666.015 |1031.186 |1380.650 |49.4807 |40.0588 |8104.0000 |2200137.0060 |324.2558|
+|50 |1 |1.50 |50.60 |7578 |89.04 |121.545 |5000 |411.364 |331.118 |605.809 |874.543 |1285.650 |48.2343 |41.1369 |9350.0000 |2568777.6400 |295.8593|
+|70 |1 |3.80 |83.20 |7602 |89.59 |133.568 |7000 |524.073 |382.653 |799.463 |1202.179 |1576.809 |57.2885 |52.4077 |10761.0000 |3013600.9670 |315.2540|
+
+
+
diff --git a/doc/Offical_Docs/6-1_Inference_Protocols_CN.md b/doc/Offical_Docs/6-1_Inference_Protocols_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..094dcbb0be4d5e4675c8b72cd92be3361647c4bb
--- /dev/null
+++ b/doc/Offical_Docs/6-1_Inference_Protocols_CN.md
@@ -0,0 +1,245 @@
+# Inference Protocols
+
+C++ Serving 基于 BRPC 进行服务构建，支持 BRPC、GRPC、RESTful 请求。请求数据为 protobuf 格式，详见 `core/general-server/proto/general_model_service.proto`。本文介绍构建请求以及解析结果的方法。
+
+## Tensor
+
+**一.Tensor 定义**
+
+Tensor 可以装载多种类型的数据，是 Request 和 Response 的基础单元。Tensor 的具体定义如下：
+
+```protobuf
+message Tensor {
+ // VarType: INT64
+ repeated int64 int64_data = 1;
+
+ // VarType: FP32
+ repeated float float_data = 2;
+
+ // VarType: INT32
+ repeated int32 int_data = 3;
+
+ // VarType: FP64
+ repeated double float64_data = 4;
+
+ // VarType: UINT32
+ repeated uint32 uint32_data = 5;
+
+ // VarType: BOOL
+ repeated bool bool_data = 6;
+
+ // (No support)VarType: COMPLEX64, 2x represents the real part, 2x+1
+ // represents the imaginary part
+ repeated float complex64_data = 7;
+
+ // (No support)VarType: COMPLEX128, 2x represents the real part, 2x+1
+ // represents the imaginary part
+ repeated double complex128_data = 8;
+
+ // VarType: STRING
+ repeated string data = 9;
+
+ // Element types:
+ // 0 => INT64
+ // 1 => FP32
+ // 2 => INT32
+ // 3 => FP64
+ // 4 => INT16
+ // 5 => FP16
+ // 6 => BF16
+ // 7 => UINT8
+ // 8 => INT8
+ // 9 => BOOL
+ // 10 => COMPLEX64
+ // 11 => COMPLEX128
+ // 20 => STRING
+ int32 elem_type = 10;
+
+ // Shape of the tensor, including batch dimensions.
+ repeated int32 shape = 11;
+
+ // Level of data(LOD), support variable length data, only for fetch tensor
+ // currently.
+ repeated int32 lod = 12;
+
+ // Correspond to the variable 'name' in the model description prototxt.
+ string name = 13;
+
+ // Correspond to the variable 'alias_name' in the model description prototxt.
+ string alias_name = 14; // get from the Model prototxt
+
+ // VarType: FP16, INT16, INT8, BF16, UINT8
+ bytes tensor_content = 15;
+};
+```
+
+- elem_type：数据类型，当前支持 FLOAT32, INT64, INT32, UINT8, INT8, FLOAT16
+
+|elem_type|类型|
+|---------|----|
+|0|INT64|
+|1|FLOAT32|
+|2|INT32|
+|3|FP64|
+|4|INT16|
+|5|FP16|
+|6|BF16|
+|7|UINT8|
+|8|INT8|
+
+- shape：数据维度
+- lod：lod 信息，LoD(Level-of-Detail) Tensor 是 Paddle 的高级特性，是对 Tensor 的一种扩充，用于支持更自由的数据输入。Lod 相关原理介绍，请参考[相关文档](../LOD_CN.md)
+- name/alias_name: 名称及别名，与模型配置对应
+
+**二.构建 Tensor 数据**
+
+1. FLOAT32 类型 Tensor
+
+```C
+// 原始数据
+std::vector
+
`fs_name` Hadoop fs_name，默认为空
`fs_ugi` Hadoop fs_ugi，默认为空 |
+| ftp | 远端为 FTP，通过 `ftplib` 进行相关访问（使用该 Monitor，您可能需要执行 `pip install ftplib` 下载 `ftplib`） | `ftp_host` FTP host
`ftp_port` FTP port
`ftp_username` FTP username，默认为空
`ftp_password` FTP password，默认为空 |
+
+| Monitor通用选项 | 描述 | 默认值 |
+| :--------------------: | :----------------------------------------------------------: | :--------------------: |
+| `type` | 指定 Monitor 类型 | 无 |
+| `remote_path` | 指定远端的基础路径 | 无 |
+| `remote_model_name` | 指定远端需要拉取的模型名 | 无 |
+| `remote_donefile_name` | 指定远端标志模型更新完毕的 donefile 文件名 | 无 |
+| `local_path` | 指定本地工作路径 | 无 |
+| `local_model_name` | 指定本地模型名 | 无 |
+| `local_timestamp_file` | 指定本地用于热加载的时间戳文件，该文件被认为在 `local_path/local_model_name` 下。 | `fluid_time_file` |
+| `local_tmp_path` | 指定本地存放临时文件的文件夹路径，若不存在则自动创建。 | `_serving_monitor_tmp` |
+| `interval` | 指定轮询间隔时间，单位为秒。 | `10` |
+| `unpacked_filename` | Monitor 支持 tarfile 打包的远程模型。如果远程模型是打包格式，则需要设置该选项来告知 Monitor 解压后的文件名。 | `None` |
+| `debug` | 如果添加 `--debug` 选项，则输出更详细的中间信息。 | 默认不添加该选项 |
+
+下面通过 HadoopMonitor 示例来展示 Paddle Serving 的模型热加载功能。
+
+## HadoopMonitor 示例
+
+示例中在 `product_path` 中生产模型上传至 hdfs，在 `server_path` 中模拟服务端模型热加载：
+
+```shell
+.
+├── product_path
+└── server_path
+```
+
+**一.生产模型**
+
+在 `product_path` 下运行下面的 Python 代码生产模型（运行前需要修改 hadoop 相关的参数），每隔 60 秒会产出 Boston 房价预测模型的打包文件 `uci_housing.tar.gz` 并上传至 hdfs 的`/`路径下，上传完毕后更新时间戳文件 `donefile` 并上传至 hdfs 的`/`路径下。
+
+```python
+import os
+import sys
+import time
+import tarfile
+import paddle
+import paddle.fluid as fluid
+import paddle_serving_client.io as serving_io
+
+train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=16)
+
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=16)
+
+x = fluid.data(name='x', shape=[None, 13], dtype='float32')
+y = fluid.data(name='y', shape=[None, 1], dtype='float32')
+
+y_predict = fluid.layers.fc(input=x, size=1, act=None)
+cost = fluid.layers.square_error_cost(input=y_predict, label=y)
+avg_loss = fluid.layers.mean(cost)
+sgd_optimizer = fluid.optimizer.SGD(learning_rate=0.01)
+sgd_optimizer.minimize(avg_loss)
+
+place = fluid.CPUPlace()
+feeder = fluid.DataFeeder(place=place, feed_list=[x, y])
+exe = fluid.Executor(place)
+exe.run(fluid.default_startup_program())
+
+def push_to_hdfs(local_file_path, remote_path):
+ afs = 'afs://***.***.***.***:***' # User needs to change
+ uci = '***,***' # User needs to change
+ hadoop_bin = '/path/to/haddop/bin' # User needs to change
+ prefix = '{} fs -Dfs.default.name={} -Dhadoop.job.ugi={}'.format(hadoop_bin, afs, uci)
+ os.system('{} -rmr {}/{}'.format(
+ prefix, remote_path, local_file_path))
+ os.system('{} -put {} {}'.format(
+ prefix, local_file_path, remote_path))
+
+name = "uci_housing"
+for pass_id in range(30):
+ for data_train in train_reader():
+ avg_loss_value, = exe.run(fluid.default_main_program(),
+ feed=feeder.feed(data_train),
+ fetch_list=[avg_loss])
+ # Simulate the production model every other period of time
+ time.sleep(60)
+ model_name = "{}_model".format(name)
+ client_name = "{}_client".format(name)
+ serving_io.save_model(model_name, client_name,
+ {"x": x}, {"price": y_predict},
+ fluid.default_main_program())
+ # Packing model
+ tar_name = "{}.tar.gz".format(name)
+ tar = tarfile.open(tar_name, 'w:gz')
+ tar.add(model_name)
+ tar.close()
+
+ # Push packaged model file to hdfs
+ push_to_hdfs(tar_name, '/')
+
+ # Generate donefile
+ donefile_name = 'donefile'
+ os.system('touch {}'.format(donefile_name))
+
+ # Push donefile to hdfs
+ push_to_hdfs(donefile_name, '/')
+```
+
+hdfs 上的文件如下列所示：
+
+```bash
+# hadoop fs -ls /
+Found 2 items
+-rw-r--r-- 1 root supergroup 0 2020-04-02 02:54 /donefile
+-rw-r--r-- 1 root supergroup 2101 2020-04-02 02:54 /uci_housing.tar.gz
+```
+
+**二.服务端加载模型**
+
+进入 `server_path` 文件夹。
+
+1. 用初始模型启动 Server 端
+
+这里使用预训练的 Boston 房价预测模型作为初始模型：
+
+```shell
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
+tar -xzf uci_housing.tar.gz
+```
+
+启动 Server 端：
+
+```shell
+python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292
+```
+
+2. 执行监控程序
+
+用下面的命令来执行 HDFS 监控程序：
+
+```shell
+python -m paddle_serving_server.monitor \
+ --type='hdfs' --hadoop_bin='/hadoop-3.1.2/bin/hadoop' \
+ --remote_path='/' --remote_model_name='uci_housing.tar.gz' \
+ --remote_donefile_name='donefile' --local_path='.' \
+ --local_model_name='uci_housing_model' --local_timestamp_file='fluid_time_file' \
+ --local_tmp_path='_tmp' --unpacked_filename='uci_housing_model' --debug
+```
+
+上面代码通过轮询方式监控远程 HDFS 地址`/`的时间戳文件`/donefile`，当时间戳变更则认为远程模型已经更新，将远程打包模型`/uci_housing.tar.gz`拉取到本地临时路径`./_tmp/uci_housing.tar.gz`下，解包出模型文件`./_tmp/uci_housing_model`后，更新本地模型`./uci_housing_model`以及Paddle Serving的时间戳文件`./uci_housing_model/fluid_time_file`。
+
+预计输出如下：
+
+```shell
+2020-04-02 10:12 INFO [monitor.py:85] _hadoop_bin: /hadoop-3.1.2/bin/hadoop
+2020-04-02 10:12 INFO [monitor.py:85] _fs_name:
+2020-04-02 10:12 INFO [monitor.py:85] _fs_ugi:
+2020-04-02 10:12 INFO [monitor.py:209] AFS prefix cmd: /hadoop-3.1.2/bin/hadoop fs
+2020-04-02 10:12 INFO [monitor.py:85] _remote_path: /
+2020-04-02 10:12 INFO [monitor.py:85] _remote_model_name: uci_housing.tar.gz
+2020-04-02 10:12 INFO [monitor.py:85] _remote_donefile_name: donefile
+2020-04-02 10:12 INFO [monitor.py:85] _local_model_name: uci_housing_model
+2020-04-02 10:12 INFO [monitor.py:85] _local_path: .
+2020-04-02 10:12 INFO [monitor.py:85] _local_timestamp_file: fluid_time_file
+2020-04-02 10:12 INFO [monitor.py:85] _local_tmp_path: _tmp
+2020-04-02 10:12 INFO [monitor.py:85] _interval: 10
+2020-04-02 10:12 DEBUG [monitor.py:214] check cmd: /hadoop-3.1.2/bin/hadoop fs -ls /donefile 2>/dev/null
+2020-04-02 10:12 DEBUG [monitor.py:216] resp: -rw-r--r-- 1 root supergroup 0 2020-04-02 10:11 /donefile
+2020-04-02 10:12 INFO [monitor.py:138] doneilfe(donefile) changed.
+2020-04-02 10:12 DEBUG [monitor.py:233] pull cmd: /hadoop-3.1.2/bin/hadoop fs -get /uci_housing.tar.gz _tmp/uci_housing.tar.gz 2>/dev/null
+2020-04-02 10:12 INFO [monitor.py:144] pull remote model(uci_housing.tar.gz).
+2020-04-02 10:12 INFO [monitor.py:98] unpack remote file(uci_housing.tar.gz).
+2020-04-02 10:12 DEBUG [monitor.py:108] remove packed file(uci_housing.tar.gz).
+2020-04-02 10:12 INFO [monitor.py:110] using unpacked filename: uci_housing_model.
+2020-04-02 10:12 DEBUG [monitor.py:175] update model cmd: cp -r _tmp/uci_housing_model/* ./uci_housing_model
+2020-04-02 10:12 INFO [monitor.py:152] update local model(uci_housing_model).
+2020-04-02 10:12 DEBUG [monitor.py:184] update timestamp cmd: touch ./uci_housing_model/fluid_time_file
+2020-04-02 10:12 INFO [monitor.py:157] update model timestamp(fluid_time_file).
+2020-04-02 10:12 INFO [monitor.py:161] sleep 10s.
+2020-04-02 10:12 DEBUG [monitor.py:214] check cmd: /hadoop-3.1.2/bin/hadoop fs -ls /donefile 2>/dev/null
+2020-04-02 10:12 DEBUG [monitor.py:216] resp: -rw-r--r-- 1 root supergroup 0 2020-04-02 10:11 /donefile
+2020-04-02 10:12 INFO [monitor.py:161] sleep 10s.
+```
+
+3. 查看 Server 日志
+
+通过下面命令查看 Server 的运行日志：
+
+```shell
+tail -f log/serving.INFO
+```
+
+日志中显示模型已经被热加载：
+
+```shell
+I0330 09:38:40.087316 7361 server.cpp:150] Begin reload framework...
+W0330 09:38:40.087399 7361 infer.h:656] Succ reload version engine: 18446744073709551615
+I0330 09:38:40.087414 7361 manager.h:131] Finish reload 1 workflow(s)
+I0330 09:38:50.087535 7361 server.cpp:150] Begin reload framework...
+W0330 09:38:50.087641 7361 infer.h:250] begin reload model[uci_housing_model].
+I0330 09:38:50.087972 7361 infer.h:66] InferEngineCreationParams: model_path = uci_housing_model, enable_memory_optimization = 0, static_optimization = 0, force_update_static_cache = 0
+I0330 09:38:50.088027 7361 analysis_predictor.cc:88] Profiler is deactivated, and no profiling report will be generated.
+I0330 09:38:50.088393 7361 analysis_predictor.cc:841] MODEL VERSION: 1.7.1
+I0330 09:38:50.088413 7361 analysis_predictor.cc:843] PREDICTOR VERSION: 1.6.3
+I0330 09:38:50.089519 7361 graph_pattern_detector.cc:96] --- detected 1 subgraphs
+I0330 09:38:50.090925 7361 analysis_predictor.cc:470] ======= optimize end =======
+W0330 09:38:50.090986 7361 infer.h:472] Succ load common model[0x7fc83c06abd0], path[uci_housing_model].
+I0330 09:38:50.091022 7361 analysis_predictor.cc:88] Profiler is deactivated, and no profiling report will be generated.
+W0330 09:38:50.091050 7361 infer.h:509] td_core[0x7fc83c0ad770] clone model from pd_core[0x7fc83c06abd0] succ, cur_idx[0].
+...
+W0330 09:38:50.091784 7361 infer.h:489] Succ load clone model, path[uci_housing_model]
+W0330 09:38:50.091794 7361 infer.h:656] Succ reload version engine: 18446744073709551615
+I0330 09:38:50.091820 7361 manager.h:131] Finish reload 1 workflow(s)
+I0330 09:39:00.091987 7361 server.cpp:150] Begin reload framework...
+W0330 09:39:00.092161 7361 infer.h:656] Succ reload version engine: 18446744073709551615
+I0330 09:39:00.092177 7361 manager.h:131] Finish reload 1 workflow(s)
+```
diff --git a/doc/Offical_Docs/6-5_Cpp_ABTest_CN.md b/doc/Offical_Docs/6-5_Cpp_ABTest_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..ff5c7145f8a3680e66e1bf7b82292c8a78059be4
--- /dev/null
+++ b/doc/Offical_Docs/6-5_Cpp_ABTest_CN.md
@@ -0,0 +1,198 @@
+# C++ Serving ABTest
+
+- [功能设计](#1)
+- [使用案例](#2)
+ - [1.1 安装 Paddle Serving Wheels](#2.1)
+ - [1.2 下载多个模型并保存模型参数](#2.2)
+ - [1.3 启动 A，B，C 3个服务](#2.3)
+ - [1.4 客户端注册 A，B，C 服务端地址](#2.4)
+ - [1.5 启动客户端并验证结果](#2.5)
+
+ABTest 是一种功能测试方案，一般是为同一个产品目标制定多种方案，让一部分用户使用 A 方案，另一部分用户使用 B 或 C 方案，根据测试效果，如点击率、转化率等来评价方案的优劣。
+
+模型服务化部署框架中，ABTest 属于一个重要的基础功能，为模型迭代升级提供实验环境。Paddle Serving 的 PYTHON SDK 中实现 ABTest 功能，为用户提供简单易用功能测试环境。
+
+
+
+## 功能设计
+
+Paddle Serving 的 ABTest 功能是基于 PYTHON SDK 和 多个服务端构成。每个服务端加载不同模型，在客户端上注册多个服务端地址和访问比例，最终确定访问。
+
+
+
+
+## 使用案例
+
+以 [imdb](https://github.com/PaddlePaddle/Serving/tree/develop/examples/C%2B%2B/imdb) 示例为例，介绍 ABTest 的使用，部署有5个步骤：
+
+1. 安装 Paddle Serving Wheels
+2. 下载多个模型并保存模型参数
+3. 启动 A，B，C 3个服务
+4. 客户端注册 A，B，C 服务端地址
+5. 启动客户端并验证结果
+
+
+
+**一.安装 Paddle Serving Wheels**
+
+使用 ABTest 功能的前提是使用 PYTHON SDK，因此需要安装 `paddle_serving_client` 的 wheel 包。[安装方法](./2-1_Docker_Images_CN.md) 如下：
+
+```
+pip3 install paddle-serving-client==0.8.3 -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
+
+
+
+**二.下载多个模型并保存模型参数**
+
+本示例已提供了一键下载脚本 `sh get_data.sh`，下载自训练的模型 `bow`、`cnn`和`lstm` 3种不同方式训练的模型。
+
+```
+sh get_data.sh
+```
+
+3种模型的所有文件如下所示，已为用户提前保存模型参数，无需执行保存操作。
+```
+├── imdb_bow_client_conf
+│ ├── serving_client_conf.prototxt
+│ └── serving_client_conf.stream.prototxt
+├── imdb_bow_model
+│ ├── embedding_0.w_0
+│ ├── fc_0.b_0
+│ ├── fc_0.w_0
+│ ├── fc_1.b_0
+│ ├── fc_1.w_0
+│ ├── fc_2.b_0
+│ ├── fc_2.w_0
+│ ├── fluid_time_file
+│ ├── __model__
+│ ├── serving_server_conf.prototxt
+│ └── serving_server_conf.stream.prototxt
+├── imdb_cnn_client_conf
+│ ├── serving_client_conf.prototxt
+│ └── serving_client_conf.stream.prototxt
+├── imdb_cnn_model
+│ ├── embedding_0.w_0
+│ ├── fc_0.b_0
+│ ├── fc_0.w_0
+│ ├── fc_1.b_0
+│ ├── fc_1.w_0
+│ ├── fluid_time_file
+│ ├── __model__
+│ ├── sequence_conv_0.b_0
+│ ├── sequence_conv_0.w_0
+│ ├── serving_server_conf.prototxt
+│ └── serving_server_conf.stream.prototxt
+├── imdb_lstm_client_conf
+│ ├── serving_client_conf.prototxt
+│ └── serving_client_conf.stream.prototxt
+├── imdb_lstm_model
+│ ├── embedding_0.w_0
+│ ├── fc_0.b_0
+│ ├── fc_0.w_0
+│ ├── fc_1.b_0
+│ ├── fc_1.w_0
+│ ├── fc_2.b_0
+│ ├── fc_2.w_0
+│ ├── lstm_0.b_0
+│ ├── lstm_0.w_0
+│ ├── __model__
+│ ├── serving_server_conf.prototxt
+│ └── serving_server_conf.stream.prototxt
+```
+
+虽然3个模型的网络结构不同，但是 `feed var` 和 `fetch_var` 都是相同的便于做 ABTest。
+```
+feed_var {
+ name: "words"
+ alias_name: "words"
+ is_lod_tensor: true
+ feed_type: 0
+ shape: -1
+}
+fetch_var {
+ name: "fc_2.tmp_2"
+ alias_name: "prediction"
+ is_lod_tensor: false
+ fetch_type: 1
+ shape: 2
+}
+```
+
+
+
+**三.启动 A，B，C 3个服务**
+
+后台启动 `bow`、`cnn` 和 `lstm` 模型服务:
+
+```python
+## 启动 bow 模型服务
+python3 -m paddle_serving_server.serve --model imdb_bow_model/ --port 9297 >/dev/null 2>&1 &
+
+## 启动 cnn 模型服务
+python3 -m paddle_serving_server.serve --model imdb_cnn_model/ --port 9298 >/dev/null 2>&1 &
+
+## 启动 lstm 模型服务
+python3 -m paddle_serving_server.serve --model imdb_lstm_model/ --port 9299 >/dev/null 2>&1 &
+```
+
+
+
+**四.客户端注册 A，B，C 服务端地址**
+
+使用 `paddle_serving_client` 中 `Client::add_variant(self, tag, cluster, variant_weight)` 接口注册服务标签、服务地址和权重。框架会将所有权重求和后计算每个服务的比例。本示例中，bow 服务的权重是10，cnn 服务的权重是30, lstm的权重是60，每次请求分别请求到3个服务的比例是10%、30%和60%。
+
+```
+from paddle_serving_client import Client
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+import sys
+import numpy as np
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.add_variant("bow", ["127.0.0.1:9297"], 10)
+client.add_variant("cnn", ["127.0.0.1:9298"], 30)
+client.add_variant("lstm", ["127.0.0.1:9299"], 60)
+client.connect()
+```
+如要在结果中打印请求到了哪个服务，在 `client.predict(feed, fetch, batch, need_variant_tag, logid)` 中设置 `need_variant_tag=True`。
+
+
+
+**五.启动客户端并验证结果**
+
+运行命令:
+```
+head test_data/part-0 | python3.7 abtest_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
+```
+
+运行结果如下，10次请求中，bow 服务2次，cnn 服务3次，lstm 服务5次，与设置的比例基本相近。
+```
+I0506 04:02:46.720135 44567 naming_service_thread.cpp:202] brpc::policy::ListNamingService("127.0.0.1:9297"): added 1
+I0506 04:02:46.722630 44567 naming_service_thread.cpp:202] brpc::policy::ListNamingService("127.0.0.1:9298"): added 1
+I0506 04:02:46.723577 44567 naming_service_thread.cpp:202] brpc::policy::ListNamingService("127.0.0.1:9299"): added 1
+I0506 04:02:46.814075 44567 general_model.cpp:490] [client]logid=0,client_cost=9.889ms,server_cost=6.283ms.
+server_tag=lstm prediction=[0.500398 0.49960205]
+I0506 04:02:46.826339 44567 general_model.cpp:490] [client]logid=0,client_cost=10.261ms,server_cost=9.503ms.
+server_tag=lstm prediction=[0.5007235 0.49927652]
+I0506 04:02:46.828992 44567 general_model.cpp:490] [client]logid=0,client_cost=1.667ms,server_cost=0.741ms.
+server_tag=bow prediction=[0.25859657 0.74140346]
+I0506 04:02:46.843299 44567 general_model.cpp:490] [client]logid=0,client_cost=13.402ms,server_cost=12.827ms.
+server_tag=lstm prediction=[0.50039905 0.4996009 ]
+I0506 04:02:46.850219 44567 general_model.cpp:490] [client]logid=0,client_cost=5.129ms,server_cost=4.332ms.
+server_tag=cnn prediction=[0.6369219 0.36307803]
+I0506 04:02:46.854203 44567 general_model.cpp:490] [client]logid=0,client_cost=2.804ms,server_cost=0.782ms.
+server_tag=bow prediction=[0.15088597 0.849114 ]
+I0506 04:02:46.858268 44567 general_model.cpp:490] [client]logid=0,client_cost=3.292ms,server_cost=2.677ms.
+server_tag=cnn prediction=[0.4608788 0.5391212]
+I0506 04:02:46.869217 44567 general_model.cpp:490] [client]logid=0,client_cost=10.13ms,server_cost=9.556ms.
+server_tag=lstm prediction=[0.5000269 0.49997318]
+I0506 04:02:46.883790 44567 general_model.cpp:490] [client]logid=0,client_cost=13.312ms,server_cost=12.822ms.
+server_tag=lstm prediction=[0.50083774 0.49916226]
+I0506 04:02:46.887256 44567 general_model.cpp:490] [client]logid=0,client_cost=2.432ms,server_cost=1.812ms.
+server_tag=cnn prediction=[0.47895813 0.52104187]
+
+```
diff --git a/doc/Offical_Docs/7-0_Python_Pipeline_Int_CN.md b/doc/Offical_Docs/7-0_Python_Pipeline_Int_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..71df7d608db65bde046e959b12f81b7cf379926f
--- /dev/null
+++ b/doc/Offical_Docs/7-0_Python_Pipeline_Int_CN.md
@@ -0,0 +1,13 @@
+# Python Pipeline 框架
+
+在许多深度学习框架中，模型服务化部署通常用于单模型的一键部署。但在 AI 工业大生产的背景下，端到端的单一深度学习模型不能解决复杂问题，多个深度学习模型组合使用是解决现实复杂问题的常规手段，如文字识别 OCR 服务至少需要检测和识别2种模型；视频理解服务一般需要视频抽帧、切词、音频处理、分类等多种模型组合实现。当前，通用多模型组合服务的设计和实现是非常复杂的，既要能实现复杂的模型拓扑关系，又要保证服务的高并发、高可用和易于开发和维护等。
+
+Paddle Serving 实现了一套通用的多模型组合服务编程框架 Python Pipeline，不仅解决上述痛点，同时还能大幅提高 GPU 利用率，并易于开发和维护。
+
+Python Pipeline 使用案例请阅读[Python Pipeline 快速部署案例](./3-2_QuickStart_Pipeline_OCR_CN.md)
+
+通过阅读以下内容掌握 Python Pipeline 核心功能和使用方法、高阶功能用法和性能优化指南等。
+- [Python Pipeline 框架设计](7-1_Python_Pipeline_Design_CN.md)
+- [Python Pipeline 核心功能](7-2_Python_Pipeline_Senior_CN.md)
+- [Python Pipeline 优化指南](7-3_Python_Pipeline_Optimize_CN.md)
+- [Python Pipeline 性能指标](7-4_Python_Pipeline_Benchmark_CN.md)
diff --git a/doc/Offical_Docs/7-1_Python_Pipeline_Design_CN.md b/doc/Offical_Docs/7-1_Python_Pipeline_Design_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..24cbbf0d03d7df0bca7841782d1bfaefce165003
--- /dev/null
+++ b/doc/Offical_Docs/7-1_Python_Pipeline_Design_CN.md
@@ -0,0 +1,514 @@
+# Python Pipeline 框架设计
+
+- [目标](#1)
+- [框架设计](#2)
+ - [2.1 网络层设计](#2.1)
+ - [2.2 图执行引擎层](#2.2)
+ - [2.3 服务日志](#2.3)
+ - [2.4 错误信息](#2.4)
+- [自定义信息](#3)
+ - [3.1 自定义 Web 服务 URL](#3.1)
+ - [3.2 自定义服务输入和输出结构](#3.2)
+ - [3.3 自定义服务并发和模型配置](#3.3)
+ - [3.4 自定义推理过程](#3.4)
+ - [3.5 自定义业务错误类型](#3.5)
+
+
+
+
+## 目标
+为了解决多个深度学习模型组合的复杂问题，Paddle Serving 团队设计了一个通用端到端多模型组合框架，其核心特点包括:
+
+1. 通用性：框架既要满足通用模型的输入类型，又要满足模型组合的复杂拓扑关系。
+2. 高性能：与常见互联网后端服务不同，深度学习模型的推理程序属于计算密集型程序，同时 GPU 等计算芯片价格昂贵，因此在平均响应时间不苛刻的场景下，计算资源占用和吞吐量指标格外重要。
+3. 高可用性：高可用的架构依赖每个服务的健壮性，服务状态可查询、异常可监控和管理是必备条件。
+4. 易于开发与调试：使用 Python 语言开发可大幅提升研发效率，运行的错误信息准确帮助开发者快速定位问题。
+
+
+
+## 框架设计
+Python Pipeline 框架分为网络服务层和图执行引擎2部分，网络服务层处理多种网络协议请求和通用输入参数问题，图执行引擎层解决复杂拓扑关系。如下图所示
+
+
+
+
+
+
+
+**一.网络服务层**
+
+网络服务层包括了 gRPC-gateway 和 gRPC Server。gPRC gateway 接收 HTTP 请求，打包成 proto 格式后转发给 gRPC Server，一套处理程序可同时处理 HTTP、gRPC 2种类型请求。
+
+另外，在支持多种模型的输入输出数据类型上，使用统一的 service.proto 结构，具有更好的通用性。
+
+```proto
+message Request {
+ repeated string key = 1;
+ repeated string value = 2;
+ optional string name = 3;
+ optional string method = 4;
+ optional int64 logid = 5;
+ optional string clientip = 6;
+};
+
+message Response {
+ optional int32 err_no = 1;
+ optional string err_msg = 2;
+ repeated string key = 3;
+ repeated string value = 4;
+};
+```
+Request 是输入结构，`key` 与 `value` 是配对的 string 数组。 `name` 与 `method` 对应 URL://{ip}:{port}/{name}/{method}。`logid` 和 `clientip` 便于用户串联服务级请求和自定义策略。
+
+Response 是输出结构，`err_no` 和 `err_msg` 表达处理结果的正确性和错误信息，`key` 和 `value` 为结果。
+
+Pipeline 服务包装了继承于 WebService 类，以 [OCR 示例](https://github.com/PaddlePaddle/Serving/tree/develop/examples/Pipeline/PaddleOCR/ocr)为例，派生出 OcrService 类，get_pipeline_response 函数内实现 DAG 拓扑关系，默认服务入口为 read_op，函数返回的 Op 为最后一个处理，此处要求最后返回的 Op 必须唯一。
+
+所有服务和模型的所有配置信息在 `config.yml` 中记录，URL 的 name 字段由 OcrService 初始化定义；run_service 函数启动服务。
+
+```python
+class OcrService(WebService):
+ def get_pipeline_response(self, read_op):
+ det_op = DetOp(name="det", input_ops=[read_op])
+ rec_op = RecOp(name="rec", input_ops=[det_op])
+ return rec_op
+
+ocr_service = OcrService(name="ocr")
+ocr_service.prepare_pipeline_config("config.yml")
+ocr_service.run_service()
+```
+
+与网络框架相关的配置在 `config.yml` 中设置。其中 `worker_num` 表示框架主线程 gRPC 线程池工作线程数，可理解成网络同步线程并发数。
+
+其次，`rpc_port` 和 `http_port` 是服务端口，可同时开启，不允许同时为空。
+```
+worker_num: 10
+
+# http 和 gRPC 服务端口
+rpc_port: 9988
+http_port: 18089
+```
+
+
+
+**二.图执行引擎层**
+
+图执行引擎的设计思路是基于有向无环图实现多模型组合的复杂拓扑关系，有向无环图由单节点或多节点串联、并联结构构成。
+
+
+
+
+
+图执行引擎抽象归纳出2种数据结构 Op 节点和 Channel 有向边，构建一条异步流水线工作流。核心概念和设计思路如下：
+- Op 节点： 可理解成1个推理模型、一个处理方法，甚至是训练前向代码，可独立运行，独立设置并发度。每个 Op 节点的计算结果放入其绑定的 Channel 中。
+- Channel 数据管道： 可理解为一个单向缓冲队列。每个 Channel 只接收上游 Op 节点的计算输出，作为下游 Op 节点的输入。
+- 工作流：根据用户定义的节点依赖关系，图执行引擎自动生成有向无环图。每条用户请求到达图执行引擎时会生成一个唯一自增 ID，通过这种唯一性绑定关系标记流水线中的不同请求。
+
+Op 的设计原则：
+- 单个 Op 默认的功能是根据输入的 Channel 数据，访问一个 Paddle Serving 的单模型服务，并将结果存在输出的 Channel
+- 单个 Op 可以支持用户自定义，包括 preprocess，process，postprocess 三个函数都可以由用户继承和实现
+- 单个 Op 可以控制并发数，从而增加处理并发数
+- 单个 Op 可以获取多个不同 RPC 请求的数据，以实现 Auto-Batching
+- Op 可以由线程或进程启动
+
+其构造函数如下：
+
+```python
+def __init__(name=None,
+ input_ops=[],
+ server_endpoints=[],
+ fetch_list=[],
+ client_config=None,
+ client_type=None,
+ concurrency=1,
+ timeout=-1,
+ retry=1,
+ batch_size=1,
+ auto_batching_timeout=None,
+ local_service_handler=None)
+```
+
+各参数含义如下：
+
+| 参数名 | 类型 | 含义 |
+| :-------------------: | :---------: |:------------------------------------------------: |
+| name | （str） | 用于标识 Op 类型的字符串，该字段必须全局唯一。 |
+| input_ops | （list） | 当前 Op 的所有前继 Op 的列表。 |
+| server_endpoints | （list） |远程 Paddle Serving Service 的 endpoints 列表。如果不设置该参数，认为是local_precditor模式，从local_service_conf中读取配置。 |
+| fetch_list | （list） |远程 Paddle Serving Service 的 fetch 列表。 |
+| client_config | （str） |Paddle Serving Service 对应的 Client 端配置文件路径。 |
+| client_type | (str) |可选择brpc、grpc或local_predictor。local_predictor不启动Serving服务，进程内预测。 |
+| concurrency | （int） | Op 的并发数。 |
+| timeout | （int） |process 操作的超时时间，单位为毫秒。若该值小于零，则视作不超时。 |
+| retry | （int） |超时重试次数。当该值为 1 时，不进行重试。 |
+| batch_size | （int） |进行 Auto-Batching 的期望 batch_size 大小，由于构建 batch 可能超时，实际 batch_size 可能小于设定值，默认为 1。 |
+| auto_batching_timeout | （float） |进行 Auto-Batching 构建 batch 的超时时间，单位为毫秒。batch_size > 1时，要设置auto_batching_timeout，否则请求数量不足batch_size时会阻塞等待。 |
+| local_service_handler | (object) |local predictor handler，Op init() 入参赋值或在 Op init() 中创建|
+
+对于 Op 之间需要传输过大数据的情况，可以考虑 RAM DB 外存进行全局存储，通过在 Channel 中传递索引的 Key 来进行数据传输
+
+
+Channel的设计原则：
+- Channel 是 Op 之间共享数据的数据结构，负责共享数据或者共享数据状态信息
+- Channel 可以支持多个OP的输出存储在同一个 Channel，同一个 Channel 中的数据可以被多个 Op 使用
+
+下图为图执行引擎中 Channel 的设计，采用 input buffer 和 output buffer 进行多 Op 输入或多 Op 输出的数据对齐，中间采用一个 Queue 进行缓冲
+
+
+
+
+
+
+
+**三.服务日志**
+
+Pipeline 服务日志在当前目录的 `PipelineServingLogs` 目录下，有3种类型日志，分别是 `pipeline.log`、`pipeline.log.wf`、`pipeline.tracer`。
+- `pipeline.log` : 记录 debug & info日志信息
+- `pipeline.log.wf` : 记录 warning & error日志
+- `pipeline.tracer` : 统计各个阶段耗时、channel 堆积信息
+
+```
+├── config.yml
+├── get_data.sh
+├── PipelineServingLogs
+│ ├── pipeline.log
+│ ├── pipeline.log.wf
+│ └── pipeline.tracer
+├── README_CN.md
+├── README.md
+├── uci_housing_client
+│ ├── serving_client_conf.prototxt
+│ └── serving_client_conf.stream.prototxt
+├── uci_housing_model
+│ ├── fc_0.b_0
+│ ├── fc_0.w_0
+│ ├── __model__
+│ ├── serving_server_conf.prototxt
+│ └── serving_server_conf.stream.prototxt
+├── web_service_java.py
+└── web_service.py
+```
+
+在服务发生异常时，错误信息会记录在 pipeline.log.wf 日志中。打印 tracer 日志要求在 config.yml 的 DAG 属性中添加 tracer 配置。
+
+1. 日志与请求的唯一标识
+Pipeline 中有2种 id 用以串联请求，分别是 data_id 和 log_id，二者区别如下：
+- data_id : Pipeline 框架生成的自增 ID，标记请求唯一性标识
+- log_id : 上游模块传入的标识，跟踪多个服务间串联关系，由于用户可不传入或不保证唯一性，因此不能作为唯一性标识
+
+通常，Pipeline 框架打印的日志会同时带上 data_id 和 log_id。开启 auto-batching 后，会使用批量中的第一个 data_id 标记 batch 整体，同时框架会在一条日志中打印批量中所有 data_id。
+
+2. 日志滚动
+Pipeline 的日志模块在 `logger.py` 中定义，使用了 `logging.handlers.RotatingFileHandler` 支持磁盘日志文件的轮换。根据不同文件级别和日质量分别设置了 `maxBytes` 和 `backupCount`，当即将超出预定大小时，将关闭旧文件并打开一个新文件用于输出。
+
+```python
+"handlers": {
+ "f_pipeline.log": {
+ "class": "logging.handlers.RotatingFileHandler",
+ "level": "INFO",
+ "formatter": "normal_fmt",
+ "filename": os.path.join(log_dir, "pipeline.log"),
+ "maxBytes": 512000000,
+ "backupCount": 20,
+ },
+ "f_pipeline.log.wf": {
+ "class": "logging.handlers.RotatingFileHandler",
+ "level": "WARNING",
+ "formatter": "normal_fmt",
+ "filename": os.path.join(log_dir, "pipeline.log.wf"),
+ "maxBytes": 512000000,
+ "backupCount": 10,
+ },
+ "f_tracer.log": {
+ "class": "logging.handlers.RotatingFileHandler",
+ "level": "INFO",
+ "formatter": "tracer_fmt",
+ "filename": os.path.join(log_dir, "pipeline.tracer"),
+ "maxBytes": 512000000,
+ "backupCount": 5,
+ },
+}
+
+```
+
+
+
+**四. 错误信息**
+
+框架提供的错误信息如下所示， 完整信息在 `error_catch.py` 中 `CustomExceptionCode` 类中定义。
+
+| 错误码 | 说明 |
+| :---: | :-------------: |
+| 0 | 成功 |
+| 50 ~ 999 | 产品错误 |
+| 3000 ~ 3999 | 框架内部服务错误 |
+| 4000 ~ 4999 | 配置错误 |
+| 5000 ~ 5999 | 用户输入错误 |
+| 6000 ~ 6999 | 超时错误 |
+| 7000 ~ 7999 | 类型检查错误 |
+| 8000 ~ 8999 | 内部通讯错误 |
+| 9000 ~ 9999 | 推理错误 |
+| 10000 ~ | 其他错误 |
+
+具体错误信息如下：
+
+```
+class CustomExceptionCode(enum.Enum):
+ OK = 0
+ PRODUCT_ERROR = 50
+
+ NOT_IMPLEMENTED = 3000
+ CLOSED_ERROR = 3001
+ NO_SERVICE = 3002
+ INIT_ERROR = 3003
+ CONF_ERROR = 4000
+ INPUT_PARAMS_ERROR = 5000
+ TIMEOUT = 6000
+ TYPE_ERROR = 7000
+ RPC_PACKAGE_ERROR = 8000
+ CLIENT_ERROR = 9000
+ UNKNOW = 10000
+```
+
+
+
+## 自定义信息
+
+提供给开发者提供以下自定义信息，包括自定义 Web 服务、自定义服务输入和输出结构、自定义服务并发和模型配置和自定义推理过程
+- 自定义 Web 服务 URL
+- 自定义服务输入和输出结构
+- 自定义服务并发和模型配置
+- 自定义推理过程
+- 自定义业务错误类型
+
+
+
+**一.自定义 Web 服务 URL**
+
+在 Web 服务中自定义服务名称是常见操作，尤其是将已有服务迁移到新框架。URL 中核心字段包括 `ip`、`port`、`name` 和 `method`，根据最新部署的环境信息设置前2个字段，重点介绍如何设置 `name` 和 `method`，框架提供默认的 `methon` 是 `prediciton`，如 `http://127.0.0.1:9999/ocr/prediction` 。
+
+框架有2处代码与此相关，分别是 gRPC Gateway 的配置文件 `python/pipeline/gateway/proto/gateway.proto` 和 服务启动文件 `web_server.py`。
+
+业务场景中通过设置 `name` 和 验证 `method` 解决问题。以 [OCR 示例]()为例，服务启动文件 `web_server.py` 通过类 `OcrService` 构造函数的 `name` 字段设置 URL 中 `name` 字段；
+```
+ocr_service = OcrService(name="ocr")
+ocr_service.prepare_pipeline_config("config.yml")
+ocr_service.run_service()
+```
+
+框架提供默认的 `methon` 是 `prediciton`，通过重载 `RequestOp::unpack_request_package` 来验证 `method`。
+```
+def unpack_request_package(self, request):
+ dict_data = {}
+ log_id = None
+ if request is None:
+ _LOGGER.critical("request is None")
+ raise ValueError("request is None")
+ if request.method is not "prediction":
+ _LOGGER.critical("request method error")
+ raise ValueError("request method error")
+ ...
+```
+
+在 `python/pipeline/gateway/proto/gateway.proto` 文件可以对 `name` 和 `method` 做严格限制，一般不需要修改，如需要特殊指定修改后，需要重新编译 Paddle Serving，[编译方法]()
+
+```proto
+service PipelineService {
+ rpc inference(Request) returns (Response) {
+ option (google.api.http) = {
+ post : "/{name=*}/{method=*}"
+ body : "*"
+ };
+ }
+};
+```
+
+
+**二.自定义服务输入和输出结构**
+
+输入和输出结构包括 proto 中 Request 和 Response 结构，以及 Op 前后处理返回。
+
+当默认 proto 结构不满足业务需求时，同时下面2个文件的 proto 的 Request 和 Response message 结构，保持一致。
+- pipeline/gateway/proto/gateway.proto
+- pipeline/proto/pipeline_service.proto
+
+修改后，需要[重新编译]()
+
+
+
+**三.自定义服务并发和模型配置**
+
+完整的配置信息可参考[配置信息]()
+
+
+
+**四.自定义推理过程**
+
+推理 Op 为开发者提供3个外部函数接口：
+
+| 变量或接口 | 说明 |
+| :----------------------------------------------: | :----------------------------------------------------------: |
+| def preprocess(self, input_dicts) | 对从 Channel 中获取的数据进行处理，处理完的数据将作为 **process** 函数的输入。（该函数对一个 **sample** 进行处理） |
+| def process(self, feed_dict_list, typical_logid) | 基于 Paddle Serving Client 进行 RPC 预测，处理完的数据将作为 **postprocess** 函数的输入。（该函数对一个 **batch** 进行处理） |
+| def postprocess(self, input_dicts, fetch_dict) | 处理预测结果，处理完的数据将被放入后继 Channel 中，以被后继 Op 获取。（该函数对一个 **sample** 进行处理） |
+| def init_op(self) | 用于加载资源（如字典等）。 |
+| self.concurrency_idx | 当前进程（非线程）的并发数索引（不同种类的 Op 单独计算）。 |
+
+Op 在一个运行周期中会依次执行 preprocess，process，postprocess 三个操作（当不设置 `server_endpoints` 参数时，不执行 process 操作），用户可以对这三个函数进行重写，默认实现如下：
+
+```python
+def preprocess(self, input_dicts):
+ # multiple previous Op
+ if len(input_dicts) != 1:
+ raise NotImplementedError(
+ 'this Op has multiple previous inputs. Please override this func.'
+ ）
+ (_, input_dict), = input_dicts.items()
+ return input_dict
+
+def process(self, feed_dict_list, typical_logid):
+ err, err_info = ChannelData.check_batch_npdata(feed_dict_list)
+ if err != 0:
+ raise NotImplementedError(
+ "{} Please override preprocess func.".format(err_info))
+ call_result = self.client.predict(
+ feed=feed_dict_list, fetch=self._fetch_names, log_id=typical_logid)
+ if isinstance(self.client, MultiLangClient):
+ if call_result is None or call_result["serving_status_code"] != 0:
+ return None
+ call_result.pop("serving_status_code")
+ return call_result
+
+def postprocess(self, input_dicts, fetch_dict):
+ return fetch_dict
+```
+
+**preprocess** 的参数是前继 Channel 中的数据 `input_dicts`，该变量（作为一个 **sample**）是一个以前继 Op 的 name 为 Key，对应 Op 的输出为 Value 的字典。
+
+**process** 的参数是 Paddle Serving Client 预测接口的输入变量 `fetch_dict_list`（preprocess 函数的返回值的列表），该变量（作为一个 **batch**）是一个列表，列表中的元素为以 feed_name 为 Key，对应 ndarray 格式的数据为 Value 的字典。`typical_logid` 作为向 PaddleServingService 穿透的 logid。
+
+**postprocess** 的参数是 `input_dicts` 和 `fetch_dict`，`input_dicts` 与 preprocess 的参数一致，`fetch_dict` （作为一个 **sample**）是 process 函数的返回 batch 中的一个 sample（如果没有执行 process ，则该值为 preprocess 的返回值）。
+
+用户还可以对 **init_op** 函数进行重写，已加载自定义的一些资源（比如字典等），默认实现如下：
+
+```python
+def init_op(self):
+ pass
+```
+
+RequestOp 和 ResponseOp 是 Python Pipeline 的中2个特殊 Op，分别是用分解 RPC 数据加入到图执行引擎中，和拿到图执行引擎的预测结果并打包 RPC 数据到客户端。
+RequestOp 类的设计如下所示，核心是在 unpack_request_package 函数中解析请求数据，因此，当修改 Request 结构后重写此函数实现全新的解包处理。
+
+| 接口 | 说明 |
+| :---------------------------------------: | :----------------------------------------: |
+| init_op(self) | OP初始化，设置默认名称@DAGExecutor |
+| unpack_request_package(self, request) | 解析请求数据 |
+
+```python
+class RequestOp(Op):
+ def __init__(self):
+ # PipelineService.name = "@DAGExecutor"
+ super(RequestOp, self).__init__(name="@DAGExecutor", input_ops=[])
+ # init op
+ try:
+ self.init_op()
+ except Exception as e:
+ _LOGGER.critical("Op(Request) Failed to init: {}".format(e))
+ os._exit(-1)
+
+ def unpack_request_package(self, request):
+ dict_data = {}
+ log_id = None
+ if request is None:
+ _LOGGER.critical("request is None")
+ raise ValueError("request is None")
+
+ for idx, key in enumerate(request.key):
+ dict_data[key] = request.value[idx]
+ log_id = request.logid
+ _LOGGER.info("RequestOp unpack one request. log_id:{}, clientip:{} \
+ name:{}, method:{}".format(log_id, request.clientip, request.name,
+ request.method))
+
+ return dict_data, log_id, None, ""
+```
+
+ResponseOp 类的设计如下所示，核心是在 pack_response_package 中打包返回结构，因此修改 Response 结构后重写此函数实现全新的打包格式。
+
+| 接口 | 说明 |
+| :------------------------------------------: | :-----------------------------------------: |
+| init_op(self) | Op 初始化，设置默认名称 @DAGExecutor |
+| pack_response_package(self, channeldata) | 处理接收的 RPC 数据 |
+
+```python
+class ResponseOp(Op):
+ def __init__(self, input_ops):
+ super(ResponseOp, self).__init__(
+ name="@DAGExecutor", input_ops=input_ops)
+ # init op
+ try:
+ self.init_op()
+ except Exception as e:
+ _LOGGER.critical("Op(ResponseOp) Failed to init: {}".format(
+ e, exc_info=True))
+ os._exit(-1)
+
+ def pack_response_package(self, channeldata):
+ resp = pipeline_service_pb2.Response()
+ error_code = channeldata.error_code
+ error_info = ""
+ ...
+
+ # pack results
+ if error_code is None:
+ error_code = 0
+ resp.err_no = error_code
+ resp.err_msg = error_info
+
+ return resp
+```
+
+
+**五.自定义业务错误类型**
+
+用户可根据业务场景自定义错误码，继承 ProductErrCode，在 Op 的 preprocess 或 postprocess 中返回列表中返回，下一阶段处理会根据自定义错误码跳过后置OP处理。
+```python
+class ProductErrCode(enum.Enum):
+ """
+ ProductErrCode is a base class for recording business error code.
+ product developers inherit this class and extend more error codes.
+ """
+ pass
+```
+
+其使用方法如下所示，定义了一种错误类型 `Product_Error` ，在 `preprocess` 函数返回值中设置错误信息，在 `postprocess` 函数中也可以设置。
+```python
+
+class ProductErrCode(enum.Enum):
+ """
+ ProductErrCode is a base class for recording business error code.
+ product developers inherit this class and extend more error codes.
+ """
+ Product_Error = 100001,
+
+def preprocess(self, input_dicts, data_id, log_id):
+ """
+ In preprocess stage, assembling data for process stage. users can
+ override this function for model feed features.
+ Args:
+ input_dicts: input data to be preprocessed
+ data_id: inner unique id
+ log_id: global unique id for RTT
+ Return:
+ input_dict: data for process stage
+ is_skip_process: skip process stage or not, False default
+ prod_errcode: None default, otherwise, product errores occured.
+ It is handled in the same way as exception.
+ prod_errinfo: "" default
+ """
+ (_, input_dict), = input_dicts.items()
+ if input_dict.get_key("product_error"):
+ return input_dict, False, Product_Error, "Product Error Occured"
+ return input_dict, False, None, ""
+
+```
diff --git a/doc/Offical_Docs/7-2_Python_Pipeline_Senior_CN.md b/doc/Offical_Docs/7-2_Python_Pipeline_Senior_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c372d0800407bd14455bbcab1cbabe928fd5f659
--- /dev/null
+++ b/doc/Offical_Docs/7-2_Python_Pipeline_Senior_CN.md
@@ -0,0 +1,628 @@
+# Python Pipeline 核心功能
+
+从设计上，Python Pipeline 框架实现轻量级的服务化部署，提供了丰富的核心功能，既能满足服务基本使用，又能满足特性需求。
+
+- [安装与环境检查](#1)
+- [服务启动与关闭](#2)
+- [本地与远程推理](#3)
+- [批量推理](#4)
+ - [4.1 客户端打包批量数据](#4.1)
+ - [4.2 服务端合并多个请求动态合并批量](#4.2)
+ - [4.3 Mini-Batch](#4.3)
+- [单机多卡推理](#5)
+- [多种计算芯片上推理](#6)
+- [TensorRT 推理加速](#7)
+- [MKLDNN 推理加速](#8)
+- [低精度推理](#9)
+ - [9.1 CPU 低精度推理](#9.1)
+ - [9.2 GPU 和 TensorRT 低精度推理](#9.2)
+ - [9.3 性能测试](#9.3)
+- [复杂图结构 DAG 跳过某个 Op 运行](#10)
+
+
+
+## 安装与环境检查
+
+在运行 Python Pipeline 服务前，确保当前环境下可部署且通过[安装指南](./2-0_Index_CN.md)已完成安装。其次，`v0.8.0`及以上版本提供了环境检查功能，检验环境是否安装正确。
+
+输入以下命令，进入环境检查程序。
+```python
+python3 -m paddle_serving_server.serve check
+```
+
+在环境检验程序中输入多条指令来检查，例如 `check_pipeline`，`check_all`等，完整指令列表如下。
+
+| 指令 | 描述|
+|---------|----|
+|check_all | 检查 Paddle Inference、Pipeline Serving、C++ Serving。只打印检测结果，不记录日志|
+|check_pipeline | 检查 Pipeline Serving，只打印检测结果，不记录日志|
+|check_cpp | 检查 C++ Serving，只打印检测结果，不记录日志|
+|check_inference | 检查 Paddle Inference 是否安装正确，只打印检测结果，不记录日志|
+|debug | 发生报错后，该命令将打印提示日志到屏幕，并记录详细日志文件|
+|exit | 退出|
+
+
+程序会分别运行 cpu 和 gpu 示例。运行成功则打印 `Pipeline cpu environment running success
+` 和 `Pipeline gpu environment running success`。
+
+```
+/usr/local/lib/python3.7/runpy.py:125: RuntimeWarning: 'paddle_serving_server.serve' found in sys.modules after import of package 'paddle_serving_server', but prior to execution of 'paddle_serving_server.serve'; this may result in unpredictable behaviour
+ warn(RuntimeWarning(msg))
+Welcome to the check env shell.Type help to list commands.
+
+(Cmd) check_pipeline
+Pipeline cpu environment running success
+Pipeline gpu environment running success
+```
+
+运行失败时，错误信息会记录到当前目录下 `stderr.log` 文件 和 `Pipeline_test_cpu/PipelineServingLogs` 目录下。用户可根据错误信息调试。
+
+```
+(Cmd) check_all
+PaddlePaddle inference environment running success
+C++ cpu environment running success
+C++ gpu environment running failure, if you need this environment, please refer to https://github.com/PaddlePaddle/Serving/blob/develop/doc/Install_CN.md
+Traceback (most recent call last):
+ File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
+ "__main__", mod_spec)
+ File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
+ exec(code, run_globals)
+ File "/usr/local/lib/python3.7/site-packages/paddle_serving_server/serve.py", line 541, in 
+
+
+
+## 复杂图结构 DAG 跳过某个 Op 运行
+
+此应用场景一般在 Op 前后处理中有 if 条件判断时，不满足条件时，跳过后面处理。实际做法是在跳过此 Op 的 process 阶段，只要在 preprocess 做好判断，跳过 process 阶段，在和 postprocess 后直接返回即可。
+preprocess 返回结果列表的第二个结果是 `is_skip_process=True` 表示是否跳过当前 Op 的 process 阶段，直接进入 postprocess 处理。
+
+```python
+## Op::preprocess() 函数实现
+def preprocess(self, input_dicts, data_id, log_id):
+ """
+ In preprocess stage, assembling data for process stage. users can
+ override this function for model feed features.
+ Args:
+ input_dicts: input data to be preprocessed
+ data_id: inner unique id
+ log_id: global unique id for RTT
+ Return:
+ input_dict: data for process stage
+ is_skip_process: skip process stage or not, False default
+ prod_errcode: None default, otherwise, product errores occured.
+ It is handled in the same way as exception.
+ prod_errinfo: "" default
+ """
+ # multiple previous Op
+ if len(input_dicts) != 1:
+ _LOGGER.critical(
+ self._log(
+ "Failed to run preprocess: this Op has multiple previous "
+ "inputs. Please override this func."))
+ os._exit(-1)
+ (_, input_dict), = input_dicts.items()
+ return input_dict, False, None, ""
+
+```
+以下示例 Jump::preprocess() 重载了原函数，返回了 True 字段
+```python
+class JumpOp(Op):
+ ## Overload func JumpOp::preprocess
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ if input_dict.has_key("jump"):
+ return input_dict, True, None, ""
+ else
+ return input_dict, False, None, ""
+```
diff --git a/doc/Offical_Docs/7-3_Python_Pipeline_Optimize_CN.md b/doc/Offical_Docs/7-3_Python_Pipeline_Optimize_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..ef2380193413f564775649e9012a5d642cd044a4
--- /dev/null
+++ b/doc/Offical_Docs/7-3_Python_Pipeline_Optimize_CN.md
@@ -0,0 +1,214 @@
+# Python Pipeline 优化指南
+
+- [优化响应时长](#1)
+ - [1.1 分析响应时长](#1.1)
+ - [Pipeline Trace Tool](#1.1.1)
+ - [Pipeline Profile Tool](#1.1.2)
+ - [1.2 优化思路](#1.2)
+- [优化服务吞吐](#2)
+ - [2.1 分析吞吐瓶颈](#2.1)
+ - [2.2 优化思路](#2.2)
+ - [增加 Op 并发](#2.2.1)
+ - [动态批量](#2.2.2)
+ - [CPU 与 GPU 处理分离](#2.2.3)
+
+
+通常，服务的性能优化是基于耗时分析，首先要掌握服务运行的各阶段耗时信息，从中找到耗时最长的性能瓶颈再做针对性优化。对于模型推理服务化不仅要关注耗时，由于 GPU 芯片昂贵，更要关注服务吞吐，从而提升 GPU 利用率实现降本增效。因此，模型推理服务化可总结为：
+- 优化响应时长
+- 优化服务吞吐
+
+经过分析和调优后，各个阶段实现整体服务的性能最优。
+
+
+
+## 优化响应时长
+
+首先，优化响应时长的主要思路首先要掌握各阶段耗时，并分析出性能瓶颈或者耗时占比较高的阶段，再针对性能瓶颈做专项优化。
+
+Paddle Serving 提供2种耗时分析工具，`Pipeline Trace Tool` 和 `Pipeline Profile Tool`。2个工具的特点如下：
+- Pipeline Trace Tool : 统计服务端所有进程各个阶段的平均耗时，包括每个 `Op` 和 `Channel`，用于定量分析。
+- Pipeline Profile Tool : 是可视化 Trace View 工具，生成多进程并发效果图，用定性和定量分析执行和并发效果。
+
+
+
+**一.耗时分析**
+
+
+
+1.Pipeline Trace Tool
+
+`Pipeline Trace Tool` 统计每个 `Op` 和 `Channel` 中各阶段的处理耗时，
+
+开启方法在配置文件 `config.yml` 的 `dag` 区段内添加 `tracer` 字段，框架会每隔 `interval_s` 时间生成 Trace 信息。
+```
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: True
+
+ #tracer, 跟踪框架吞吐，每个OP和channel的工作情况。无tracer时不生成数据
+ tracer:
+ #每次trace的时间间隔，单位秒/s
+ interval_s: 10
+```
+
+生成的 Trace 信息保存在 `./PipelineServingLogs/pipeline.tracer` 日志中。如下图所示
+```
+==================== TRACER ======================
+ Op(uci):
+ in[8473.507333333333 ms]： # 等待前置 Channel 中数据放入 Op 的耗时，如长时间无请求，此值会变大
+ prep[0.6753333333333333 ms] # 推理前处理 preprocess 阶段耗时
+ midp[26.476333333333333 ms] # 推理 process 阶段耗时
+ postp[1.8616666666666666 ms] # 推理后处理 postprocess 阶段耗时
+ out[1.3236666666666668 ms] # 后处理结果放入后置 channel 耗时
+ idle[0.9965882097324374] # 框架自循环耗时，间隔 1 ms，如此值很大说明系统负载高，调度变慢
+ DAGExecutor:
+ Query count[30] # interval_s 间隔时间内请求数量
+ QPS[27.35 q/s] # interval_s 间隔时间内服务 QPS
+ Succ[1.0] # interval_s 间隔时间内请求成功率
+ Error req[] # 异常请求信息
+ Latency:
+ ave[36.55233333333334 ms] # 平均延时
+ .50[8.702 ms] # 50分位延时
+ .60[8.702 ms] # 60分位延时
+ .70[92.346 ms] # 70分位延时
+ .80[92.346 ms] # 70分位延时
+ .90[92.346 ms] # 90分位延时
+ .95[92.346 ms] # 95分位延时
+ .99[92.346 ms] # 99分位延时
+ Channel (server worker num[1]):
+ chl0(In: ['@DAGExecutor'], Out: ['uci']) size[0/0] # 框架 RequestOp 与 uci Op 之间 Channel 中堆积请求数。此值较大，说明下游 uci Op 消费能力不足。
+ chl1(In: ['uci'], Out: ['@DAGExecutor']) size[0/0] # uci Op 与 框架 ResponseOp 之间 Channel 中堆积的请求数。此值较大，说明下游 ReponseOp 消费能力不足。
+ ==================== TRACER ======================
+```
+
+
+2.Pipeline Profile Tool
+
+```
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: True
+
+ #使用性能分析, 默认为 False，imeline性能数据，对性能有一定影响
+ use_profile: True,
+```
+
+开启后，Server 端在预测的过程中会将对应的日志信息打印到`标准输出`，为了更直观地展现各阶段的耗时，因此服务启动要使用如下命令：
+```
+python3.7 web_service.py > profile.txt 2>&1
+```
+
+服务接收请求后，输出 Profile 信息到 `profile.txt` 文件中。再粘贴如下代码到 `trace.py`， 使用框架提供 Analyst 模块对日志文件做进一步的分析处理。
+```
+from paddle_serving_server.pipeline import Analyst
+import json
+import sys
+
+if __name__ == "__main__":
+ log_filename = "profile.txt"
+ trace_filename = "trace"
+ analyst = Analyst(log_filename)
+ analyst.save_trace(trace_filename)
+```
+
+运行命令，脚本将日志中的时间打点信息转换成 json 格式保存到 `trace` 文件。
+```
+python3.7 trace.py
+```
+
+`trace` 文件可以通过 `chrome` 浏览器的 `tracing` 功能进行可视化。
+```
+打开 chrome 浏览器，在地址栏输入 chrome://tracing/ ，跳转至 tracing 页面，点击 load 按钮，打开保存的 trace 文件，即可将预测服务的各阶段时间信息可视化。
+```
+
+通过图示中并发请求的处理流程可观测到推理阶段的流水线状态，以及多个请求在推理阶段的`间隔`信息，进行优化。
+
+
+
+**二.降低响应时长优化思路**
+
+根据 `Pipeline Trace Tool` 输出结果在不同阶段耗时长的问题，常见场景的优化方法如下：
+- Op 推理阶段(midp) 耗时长:
+ - 增加 Op 并发度
+ - 开启 auto-batching (前提是多个请求的 shape 一致)
+ - 若批量数据中某条数据的 shape 很大，padding 很大导致推理很慢，可参考 OCR 示例中 mini-batch 方法。
+ - 开启 TensorRT/MKL-DNN 优化
+ - 开启低精度推理
+- Op 前处理阶段(prep) 或 后处理阶段耗时长:
+ - 增加 OP 并发度
+ - 优化前后处理逻辑
+- in/out 耗时长（channel 堆积>5）
+ - 检查 channel 传递的数据大小，可能为传输的数据大导致延迟大。
+ - 优化传入数据，不传递数据或压缩后再传入
+ - 增加 Op 并发度
+ - 减少上游 Op 并发度
+
+根据 `Pipeline Profile Tool` 输出结果优化流水行并发的效果
+- 增加 Op 并发度，或调整不同 Op 的并发度
+- 开启 auto-batching
+
+此外，还有一些优化思路，如将 CPU 处理较慢的过程转换到 GPU 上处理等，客户端与服务端传输较大数据时，可使用共享内存方式传递内存或显存地址等。
+
+
+
+## 优化服务吞吐
+
+
+
+**一.分析吞吐瓶颈**
+
+服务的吞吐量受到多种多因素条件制约，如 Op 处理时长、传输数据耗时、并发数和 DAG 图结构等，可以将这些因素进一步拆解，当传输数据不是极端庞大的时候，最重要因素是流水线中`最慢 Op 的处理时长和并发数`。
+```
+Op 处理时长：
+op_cost = process(pre + mid + post)
+
+服务吞吐量：
+service_throughput = 1 / 最慢 op_cost * 并发数
+
+服务平响：
+service_avg_cost = ∑op_concurrency 【关键路径】
+
+批量预测平均耗时：
+avg_batch_cost = (N * pre + mid + post) / N
+```
+
+
+**二.优化思路**
+
+优化吞吐的主要方法是 `增大 Op 并发数`、`自动批量` 和 `CPU 与 GPU 处理分离`
+
+
+
+1.增加 Op 并发**
+
+调整 Op 的并发数量通过设置 `is_thread_op: False` 进程类型 Op 和 `uci` Op 的 `concurrency` 字段
+```
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ uci:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 10
+```
+Op 的进程数量不是越大越好，受到机器 CPU 核数、内存和显存大小的限制，推荐设置 Op 的并发数不超过系统 CPU 核数。
+
+
+
+2.动态批量
+
+动态批量是增加吞吐的有一种方法，开启方式可参考[Python Pipeline 核心功能](./7-2_Python_Pipeline_Senior_CN.md#批量推理)
+
+
+
+3.CPU 与 GPU 处理分离
+
+在 `CV` 模型中，对图片或视频的前后处理成为主要瓶颈时，可考虑此方案，即将前后处理过程独立成一个 Op 并独立设置并发度。
+
+将 CPU 前后处理和 GPU 推理过程比例调整到服务最佳配比。以 OCR 为例，原有流水线设计为 `RequestOp -> DetOp -> RecOp -> ResponseOp`。
+
+根据耗时分析，`DetOp` 和 `RecOp` 的前处理耗时很长，因此，将2个模型前处理分离成独立 Op，最新的流水线设计为:
+
+`RequestOp -> PreDetOp -> DetOp -> PreRecOp -> RecOp -> ResponseOp`，并调大 `PreDetOp` 和 `PreRecOp`的并发度，从而获得 20% 的性能提升。
+
+由于增加了2次数据传递，单条请求的处理延时会增加。
diff --git a/doc/Offical_Docs/7-4_Python_Pipeline_Benchmark_CN.md b/doc/Offical_Docs/7-4_Python_Pipeline_Benchmark_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..9d29abf823fd24b98176df093bbb0258456f200f
--- /dev/null
+++ b/doc/Offical_Docs/7-4_Python_Pipeline_Benchmark_CN.md
@@ -0,0 +1,59 @@
+# Python Pipeline 性能测试
+
+- [测试环境](#1)
+- [性能指标与结论](#2)
+
+
+
+## 测试环境
+
+测试环境如下表所示：
+| | GPU | 显存 | CPU | 内存 |
+|----------|---------|----------|----------------------------------------------|------|
+| Serving端 | 4x Tesla P4-8GB | 7611MiB | Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 | 216G |
+| Client端 | 4x Tesla P4-8GB | 7611MiB | Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 | 216G |
+
+使用单卡GPU，未开启TensorRT。
+模型：ResNet_v2_50
+
+
+
+## 性能指标与结论
+
+通过测试，使用 Python Pipeline 模式通过多进程并发，充分利用 GPU 显卡，具有较好的吞吐性能。
+
+
+测试数据如下：
+
+|model_name |thread_num |batch_size |CPU_util(%) |GPU_memory(mb) |GPU_util(%) |qps(samples/s) |total count |mean(ms) |median(ms) |80 percent(ms) |90 percent(ms) |99 percent(ms) |total cost(s) |each cost(s)|
+|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--
+|ResNet_v2_50 |1 |1 |2.2 |3327 |17.25 |17.633658869240787 |355 |56.428481238996476 |38.646728515625 |39.496826171875 |39.98369140625 |1273.1911083984373 |20.131953477859497 |20.033540725708008|
+|ResNet_v2_50 |1 |4 |2.7 |3617 |28.122 |53.50748430453522 |268 |74.71539215543378 |74.6181640625 |75.3138671875 |75.6051025390625 |77.85322998046874 |20.03458046913147 |20.024930953979492|
+|ResNet_v2_50 |1 |8 |1.7 |3877 |25.7869 |59.60582783086999 |150 |133.5897119140625 |132.7503662109375 |134.968310546875 |136.470703125 |140.79039062499996 |20.132259607315063 |20.03933620452881|
+|ResNet_v2_50 |1 |16 |7.0 |4777 |27.0175 |63.2627646819339 |80 |252.30162048339844 |251.8448486328125 |253.046630859375 |253.91142578125 |263.361640625 |20.233070850372314 |20.18476152420044|
+|ResNet_v2_50 |1 |32 |7.5 |6567 |38.532 |62.945314687348024 |40 |506.8969482421875 |507.3531494140625 |510.562353515625 |511.421240234375 |536.8068920898437 |20.335111618041992 |20.276386737823486|
+|ResNet_v2_50 |2 |1 |4.7 |6567 |49.4828 |50.40600094376044 |1010 |39.63352195815285 |39.5345458984375 |40.452880859375 |41.1375 |42.940522460937494 |20.037296772003174 |20.01696753501892|
+|ResNet_v2_50 |2 |4 |2.7 |6567 |44.4744 |83.4255836891382 |420 |95.38548002697172 |95.7069091796875 |97.599951171875 |98.098583984375 |102.39680908203125 |20.137707471847534 |20.03199553489685|
+|ResNet_v2_50 |2 |8 |2.2 |6567 |42.898 |91.3727510505176 |230 |174.89108568274457 |175.0452880859375 |175.82001953125 |176.7634033203125 |178.64064453125002 |20.13729453086853 |20.1132071018219|
+|ResNet_v2_50 |2 |16 |2.2 |6567 |45 |97.5591285698611 |124 |327.16720088835683 |328.6126708984375 |329.75185546875 |330.386962890625 |336.86397460937496 |20.336385011672974 |20.284939169883728|
+|ResNet_v2_50 |2 |32 |3.2 |6567 |59.5714 |100.70765418116333 |64 |633.9812698364258 |637.8568115234375 |648.103515625 |650.7439697265625 |659.2212915039062 |20.336090803146362 |20.28787398338318|
+|ResNet_v2_50 |4 |1 |3.1 |6567 |64.3333 |80.27845081929433 |1617 |49.56464230756223 |49.4873046875 |51.5537109375 |52.693408203125 |55.207568359374996 |20.142391681671143 |20.038144528865814|
+|ResNet_v2_50 |4 |4 |3.3 |6567 |70.4563 |136.62061939701394 |688 |116.51574919944586 |121.8629150390625 |129.8181640625 |133.384423828125 |142.69500732421875 |20.143372297286987 |20.041599333286285|
+|ResNet_v2_50 |4 |8 |3.0 |6567 |70.896 |158.46554975132275 |399 |201.30669079926378 |210.69775390625 |228.51748046875 |236.427294921875 |252.24822753906233 |20.143179416656494 |20.081032752990723|
+|ResNet_v2_50 |4 |16 |3.2 |6567 |66.3832 |156.4935247130092 |197 |407.6668608224937 |423.974609375 |450.368212890625 |464.45986328125 |482.93658203125 |20.141408443450928 |20.078101694583893|
+|ResNet_v2_50 |4 |32 |3.3 |6567 |72.4791 |162.01742190796557 |104 |785.5079204852765 |813.0341796875 |887.107958984375 |909.6556640625 |935.3334838867188 |20.541000843048096 |20.423666059970856|
+|ResNet_v2_50 |8 |1 |3.5 |6567 |93.977 |115.9749228558386 |2337 |68.5580409078145 |65.45849609375 |76.13930664062501 |83.542041015625 |91.45666015624998 |20.15090799331665 |20.028797417879105|
+|ResNet_v2_50 |8 |4 |4.2 |6567 |90.0952 |175.58748591910316 |889 |180.7330482920592 |170.5810546875 |218.99931640625 |240.06337890625002 |254.413759765625 |20.252012729644775 |20.084695398807526|
+|ResNet_v2_50 |8 |8 |2.6 |6567 |93.8693 |206.76595246418208 |526 |306.52158695119414 |303.043212890625 |321.0791015625 |350.5477294921875 |400.32452392578125 |20.351513147354126 |20.15437400341034|
+|ResNet_v2_50 |8 |16 |3.2 |6567 |85.7273 |205.31850043117367 |265 |614.1745522553066 |552.372314453125 |775.89169921875 |802.022607421875 |902.2763183593761 |20.650842428207397 |20.345011442899704|
+|ResNet_v2_50 |8 |32 |5.0 |6567 |89.8717 |219.8410273718835 |146 |1138.4533474020761 |1039.640869140625 |1364.289794921875 |1474.6744384765625 |1788.2614379882834 |21.251720190048218 |20.777225106954575|
+|ResNet_v2_50 |12 |1 |5.0 |6567 |89.4762 |110.00858327847862 |2218 |108.50048552943953 |103.015625 |121.09404296875003 |137.1392333984375 |151.80401123046872 |20.162063121795654 |20.055511037508648|
+|ResNet_v2_50 |12 |4 |4.1 |6567 |77.7619 |153.7824464757549 |779 |309.68895575507463 |285.585205078125 |378.07421875 |413.481640625 |424.70853515625 |20.262390613555908 |20.104551911354065|
+|ResNet_v2_50 |12 |8 |3.6 |6567 |72.6977 |165.36021780846013 |425 |571.1991590073529 |510.995849609375 |731.9383300781251 |747.6568359375 |757.304716796875 |20.56117272377014 |20.230452219645183|
+|ResNet_v2_50 |12 |16 |1.5 |6567 |76.2222 |189.6414991568285 |252 |987.7153136238219 |926.00390625 |1080.99130859375 |1249.4956298828126 |1434.4802392578124 |21.26116919517517 |20.74245794614156|
+|ResNet_v2_50 |12 |32 |2.8 |6567 |84.25 |203.868228281784 |138 |1811.640237559443 |1764.2760009765625 |1855.28046875 |2023.56826171875 |2586.8038134765625 |21.66105055809021 |20.834286351998646|
+|ResNet_v2_50 |16 |1 |4.8 |6567 |94.3333 |116.34927733312234 |2347 |136.7957122373642 |135.959716796875 |144.1568359375 |146.105517578125 |175.05707519531248 |20.172020435333252 |20.067057371139526|
+|ResNet_v2_50 |16 |4 |15.4 |6567 |83.6364 |160.59012047270738 |822 |393.3079394412447 |396.446533203125 |426.272216796875 |429.777734375 |564.1119360351562 |20.47448492050171 |20.206754431128502|
+|ResNet_v2_50 |16 |8 |6.8 |6567 |81.0233 |169.95774070621547 |437 |741.5512622684854 |751.521484375 |763.199169921875 |948.8041992187501 |1001.156142578125 |20.56981921195984 |20.254074171185493|
+|ResNet_v2_50 |16 |16 |3.5 |6567 |77.8706 |186.56600081516 |248 |1332.1007946383568 |1365.2745361328125 |1399.212255859375 |1432.4037353515625 |1771.4374853515626 |21.26861262321472 |20.64799252152443|
+|ResNet_v2_50 |16 |32 |4.3 |6567 |83.6371 |201.1293408638195 |140 |2419.3400198800223 |2561.09228515625 |2616.081103515625 |2642.0835205078124 |2883.8197412109366 |22.274224042892456 |21.169659316539764|
diff --git a/doc/Offical_Docs/9-0_Kubernetes_Int_CN.md b/doc/Offical_Docs/9-0_Kubernetes_Int_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..cd06680961ae7272524bbf6852f30b5b45e6f401
--- /dev/null
+++ b/doc/Offical_Docs/9-0_Kubernetes_Int_CN.md
@@ -0,0 +1,7 @@
+Kubernetes 集群部署
+
+服务部署经历从物理机、虚拟机、容器化、云原生4个阶段。云原生，提供集装箱组合模式的乐高生态，Docker、Kubernetes 已称为云原生时代基础设施，推动应用程序大发展。Kubernetes 的可扩展性和分布式架构一直是人工智能和机器学习的绝佳选择，随着解决方案不断成熟，推动机器学习大规模工程落地。
+
+本章节介绍 Kubernetes 上集群化部署 Paddle Serving 方案以及企业级安全网关部署案例。
+- [Kubernetes 集群部署方案](./9-1_Kubernetes_CN.md)
+- [Kubernetes 安全网关部署案例]()
diff --git a/doc/Offical_Docs/9-1_Kubernetes_CN.md b/doc/Offical_Docs/9-1_Kubernetes_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..96fb5473fb4602c11d69200016f37183665efb73
--- /dev/null
+++ b/doc/Offical_Docs/9-1_Kubernetes_CN.md
@@ -0,0 +1,205 @@
+# Kubernetes 集群部署
+
+Kubernetes 是一个基于容器技术的分布式架构的解决方案，是云原生容器集群管理系统，提供服务发现与负载均衡、存储编排、自动部署和回滚、资源管理、自动恢复以及密钥和配置管理。Paddle Serving 支持 Kubenetes 集群部署方案，为企业级用户提供集群部署示例。
+
+## 部署方案
+
+为了解决 Pod 迁移、Node Pod 端口、域名动态分配等问题，选择使用 Ingress 解决方案，对外提供可访问的 URL、负载均衡、SSL、基于名称的虚拟主机等功能。在众多 Ingress 插件中选用 Kong 作为微服务的 API 网关，因其具备以下优势：
+- 拥有丰富的微服务功能，如 API认证、鉴权、DDos保护和灰度部署等
+- 提供一些 API、服务的定义，可抽象成 Kubernetes 的 CRD，通过 Kubernetes Ingress 配置实现同步状态到 Kong 集群
+- 集群配置信息存储在 postgres 数据库，配置信息实现全局节点共享和实时同步
+- 有成熟的第三方管理 UI，实现可视化管理 Kong 配置
+
+Paddle Serving 的 Kubernetes 集群部署方案设计如下图所示，用户流量通过 Kong Ingress 转发到 Kubernetes 集群。Kubernetes 集群负责管理 Service 和 Pod 实例。
+
+
+
+
+
+
+
+
+
+margin:0.2 scale:30 epoch:3 lr:5E-5 bs:64 max_len:64 |4卡 v100-16g|f3e5c7d7b0b718c2530c5e1b136b2d74|
+
+```
+wget https://bj.bcebos.com/v1/paddlenlp/models/inbatch_model.zip
+unzip inbatch_model.zip -d checkpoints
+```
+
+
+
+## 2. 模型部署
+
+### 2.1 动转静导出
+
+首先把动态图模型转换为静态图：
+
+```
+python export_model.py --params_path checkpoints/model_40/model_state.pdparams --output_path=./output
+```
+也可以运行下面的bash脚本：
+
+```
+sh scripts/export_model.sh
+```
+
+### 2.2 Paddle Inference预测
+
+预测既可以抽取向量也可以计算两个文本的相似度。
+
+修改id2corpus的样本：
+
+```
+# 抽取向量
+id2corpus={0:'国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据'}
+# 计算相似度
+corpus_list=[['中西方语言与文化的差异','中西方文化差异以及语言体现中西方文化,差异,语言体现'],
+ ['中西方语言与文化的差异','飞桨致力于让深度学习技术的创新与应用更简单']]
+
+```
+
+然后使用PaddleInference
+
+```
+python deploy/python/predict.py --model_dir=./output
+```
+也可以运行下面的bash脚本：
+
+```
+sh deploy.sh
+```
+最终输出的是256维度的特征向量和句子对的预测概率：
+
+```
+(1, 256)
+[[-0.0394925 -0.04474756 -0.065534 0.00939134 0.04359895 0.14659195
+ -0.0091779 -0.07303623 0.09413272 -0.01255222 -0.08685658 0.02762237
+ 0.10138468 0.00962821 0.10888419 0.04553023 0.05898942 0.00694253
+ ....
+
+[0.959269642829895, 0.04725276678800583]
+```
+
+### 2.3 Paddle Serving部署
+
+Paddle Serving 的详细文档请参考 [Pipeline_Design](https://github.com/PaddlePaddle/Serving/blob/v0.7.0/doc/Python_Pipeline/Pipeline_Design_CN.md)和[Serving_Design](https://github.com/PaddlePaddle/Serving/blob/v0.7.0/doc/Serving_Design_CN.md),首先把静态图模型转换成Serving的格式：
+
+```
+python export_to_serving.py \
+ --dirname "output" \
+ --model_filename "inference.get_pooled_embedding.pdmodel" \
+ --params_filename "inference.get_pooled_embedding.pdiparams" \
+ --server_path "./serving_server" \
+ --client_path "./serving_client" \
+ --fetch_alias_names "output_embedding"
+
+```
+
+参数含义说明
+* `dirname`: 需要转换的模型文件存储路径，Program 结构文件和参数文件均保存在此目录。
+* `model_filename`： 存储需要转换的模型 Inference Program 结构的文件名称。如果设置为 None ，则使用 `__model__` 作为默认的文件名
+* `params_filename`: 存储需要转换的模型所有参数的文件名称。当且仅当所有模型参数被保>存在一个单独的二进制文件中，它才需要被指定。如果模型参数是存储在各自分离的文件中，设置它的值为 None
+* `server_path`: 转换后的模型文件和配置文件的存储路径。默认值为 serving_server
+* `client_path`: 转换后的客户端配置文件存储路径。默认值为 serving_client
+* `fetch_alias_names`: 模型输出的别名设置，比如输入的 input_ids 等，都可以指定成其他名字，默认不指定
+* `feed_alias_names`: 模型输入的别名设置，比如输出 pooled_out 等，都可以重新指定成其他模型，默认不指定
+
+也可以运行下面的 bash 脚本：
+```
+sh scripts/export_to_serving.sh
+```
+
+Paddle Serving的部署有两种方式，第一种方式是Pipeline的方式，第二种是C++的方式，下面分别介绍这两种方式的用法：
+
+#### 2.3.1 Pipeline方式
+
+启动 Pipeline Server:
+
+```
+python web_service.py
+```
+
+启动客户端调用 Server。
+
+首先修改rpc_client.py中需要预测的样本：
+
+```
+list_data = [
+ "国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据",
+ "试论翻译过程中的文化差异与语言空缺翻译过程,文化差异,语言空缺,文化对比"
+]
+```
+然后运行：
+
+```
+python rpc_client.py
+```
+模型的输出为：
+
+```
+{'0': '国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据', '1': '试论翻译过程中的文化差异与语言空缺翻译过程,文化差异,语言空缺,文化对比'}
+PipelineClient::predict pack_data time:1641450851.3752182
+PipelineClient::predict before time:1641450851.375738
+['output_embedding']
+(2, 256)
+[[ 0.07830612 -0.14036864 0.03433796 -0.14967982 -0.03386067 0.06630666
+ 0.01357943 0.03531194 0.02411093 0.02000859 0.05724002 -0.08119463
+ ......
+```
+
+可以看到客户端发送了2条文本，返回了2个 embedding 向量
+
+#### 2.3.2 C++的方式
+
+启动C++的Serving：
+
+```
+python -m paddle_serving_server.serve --model serving_server --port 9393 --gpu_id 2 --thread 5 --ir_optim True --use_trt --precision FP16
+```
+也可以使用脚本：
+
+```
+sh deploy/C++/start_server.sh
+```
+Client 可以使用 http 或者 rpc 两种方式，rpc 的方式为：
+
+```
+python deploy/C++/rpc_client.py
+```
+运行的输出为：
+```
+I0209 20:40:07.978225 20896 general_model.cpp:490] [client]logid=0,client_cost=395.695ms,server_cost=392.559ms.
+time to cost :0.3960278034210205 seconds
+{'output_embedding': array([[ 9.01343748e-02, -1.21870913e-01, 1.32834800e-02,
+ -1.57673359e-01, -2.60387752e-02, 6.98455423e-02,
+ 1.58108603e-02, 3.89952064e-02, 3.22783105e-02,
+ 3.49135026e-02, 7.66086206e-02, -9.12970975e-02,
+ 6.25643134e-02, 7.21886680e-02, 7.03565404e-02,
+ 5.44054210e-02, 3.25332815e-03, 5.01751155e-02,
+......
+```
+可以看到服务端返回了向量
+
+或者使用 http 的客户端访问模式：
+
+```
+python deploy/C++/http_client.py
+```
+运行的输出为：
+
+```
+(2, 64)
+(2, 64)
+outputs {
+ tensor {
+ float_data: 0.09013437479734421
+ float_data: -0.12187091261148453
+ float_data: 0.01328347995877266
+ float_data: -0.15767335891723633
+......
+```
+可以看到服务端返回了向量
+
+
+
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/base_model.py b/examples/Pipeline/PaddleNLP/semantic_indexing/base_model.py
new file mode 100644
index 0000000000000000000000000000000000000000..c471d126c2649fee7554fa8f026284c7300ada2f
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/base_model.py
@@ -0,0 +1,187 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import abc
+import sys
+
+import numpy as np
+
+import paddle
+import paddle.nn as nn
+import paddle.nn.functional as F
+
+
+class SemanticIndexBase(nn.Layer):
+ def __init__(self, pretrained_model, dropout=None, output_emb_size=None):
+ super().__init__()
+ self.ptm = pretrained_model
+ self.dropout = nn.Dropout(dropout if dropout is not None else 0.1)
+
+ # if output_emb_size is not None, then add Linear layer to reduce embedding_size,
+ # we recommend set output_emb_size = 256 considering the trade-off beteween
+ # recall performance and efficiency
+
+ self.output_emb_size = output_emb_size
+ if output_emb_size > 0:
+ weight_attr = paddle.ParamAttr(
+ initializer=paddle.nn.initializer.TruncatedNormal(std=0.02))
+ self.emb_reduce_linear = paddle.nn.Linear(
+ 768, output_emb_size, weight_attr=weight_attr)
+
+ @paddle.jit.to_static(input_spec=[
+ paddle.static.InputSpec(
+ shape=[None, None], dtype='int64'), paddle.static.InputSpec(
+ shape=[None, None], dtype='int64')
+ ])
+ def get_pooled_embedding(self,
+ input_ids,
+ token_type_ids=None,
+ position_ids=None,
+ attention_mask=None):
+ _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,
+ attention_mask)
+
+ if self.output_emb_size > 0:
+ cls_embedding = self.emb_reduce_linear(cls_embedding)
+ cls_embedding = self.dropout(cls_embedding)
+ cls_embedding = F.normalize(cls_embedding, p=2, axis=-1)
+
+ return cls_embedding
+
+ def get_semantic_embedding(self, data_loader):
+ self.eval()
+ with paddle.no_grad():
+ for batch_data in data_loader:
+ input_ids, token_type_ids = batch_data
+ input_ids = paddle.to_tensor(input_ids)
+ token_type_ids = paddle.to_tensor(token_type_ids)
+
+ text_embeddings = self.get_pooled_embedding(
+ input_ids, token_type_ids=token_type_ids)
+
+ yield text_embeddings
+
+ def cosine_sim(self,
+ query_input_ids,
+ title_input_ids,
+ query_token_type_ids=None,
+ query_position_ids=None,
+ query_attention_mask=None,
+ title_token_type_ids=None,
+ title_position_ids=None,
+ title_attention_mask=None):
+
+ query_cls_embedding = self.get_pooled_embedding(
+ query_input_ids, query_token_type_ids, query_position_ids,
+ query_attention_mask)
+
+ title_cls_embedding = self.get_pooled_embedding(
+ title_input_ids, title_token_type_ids, title_position_ids,
+ title_attention_mask)
+
+ cosine_sim = paddle.sum(query_cls_embedding * title_cls_embedding,
+ axis=-1)
+ return cosine_sim
+
+ @abc.abstractmethod
+ def forward(self):
+ pass
+
+
+class SemanticIndexBaseStatic(nn.Layer):
+ def __init__(self, pretrained_model, dropout=None, output_emb_size=None):
+ super().__init__()
+ self.ptm = pretrained_model
+ self.dropout = nn.Dropout(dropout if dropout is not None else 0.1)
+
+ # if output_emb_size is not None, then add Linear layer to reduce embedding_size,
+ # we recommend set output_emb_size = 256 considering the trade-off beteween
+ # recall performance and efficiency
+
+ self.output_emb_size = output_emb_size
+ if output_emb_size > 0:
+ weight_attr = paddle.ParamAttr(
+ initializer=paddle.nn.initializer.TruncatedNormal(std=0.02))
+ self.emb_reduce_linear = paddle.nn.Linear(
+ 768, output_emb_size, weight_attr=weight_attr)
+
+ @paddle.jit.to_static(input_spec=[
+ paddle.static.InputSpec(
+ shape=[None, None], dtype='int64'), paddle.static.InputSpec(
+ shape=[None, None], dtype='int64')
+ ])
+ def get_pooled_embedding(self,
+ input_ids,
+ token_type_ids=None,
+ position_ids=None,
+ attention_mask=None):
+ _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,
+ attention_mask)
+
+ if self.output_emb_size > 0:
+ cls_embedding = self.emb_reduce_linear(cls_embedding)
+ cls_embedding = self.dropout(cls_embedding)
+ cls_embedding = F.normalize(cls_embedding, p=2, axis=-1)
+
+ return cls_embedding
+
+ def get_semantic_embedding(self, data_loader):
+ self.eval()
+ with paddle.no_grad():
+ for batch_data in data_loader:
+ input_ids, token_type_ids = batch_data
+ input_ids = paddle.to_tensor(input_ids)
+ token_type_ids = paddle.to_tensor(token_type_ids)
+
+ text_embeddings = self.get_pooled_embedding(
+ input_ids, token_type_ids=token_type_ids)
+
+ yield text_embeddings
+
+ def cosine_sim(self,
+ query_input_ids,
+ title_input_ids,
+ query_token_type_ids=None,
+ query_position_ids=None,
+ query_attention_mask=None,
+ title_token_type_ids=None,
+ title_position_ids=None,
+ title_attention_mask=None):
+
+ query_cls_embedding = self.get_pooled_embedding(
+ query_input_ids, query_token_type_ids, query_position_ids,
+ query_attention_mask)
+
+ title_cls_embedding = self.get_pooled_embedding(
+ title_input_ids, title_token_type_ids, title_position_ids,
+ title_attention_mask)
+
+ cosine_sim = paddle.sum(query_cls_embedding * title_cls_embedding,
+ axis=-1)
+ return cosine_sim
+
+ def forward(self,
+ input_ids,
+ token_type_ids=None,
+ position_ids=None,
+ attention_mask=None):
+ _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,
+ attention_mask)
+
+ if self.output_emb_size > 0:
+ cls_embedding = self.emb_reduce_linear(cls_embedding)
+ cls_embedding = self.dropout(cls_embedding)
+ cls_embedding = F.normalize(cls_embedding, p=2, axis=-1)
+
+ return cls_embedding
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/http_client.py b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..a976ad9fc33b06ce7148adc7153d4b35183e31c0
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/http_client.py
@@ -0,0 +1,81 @@
+# coding:utf-8
+# pylint: disable=doc-string-missing
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import time
+import numpy as np
+import requests
+import json
+
+from paddle_serving_client import HttpClient
+import paddlenlp as ppnlp
+
+
+def convert_example(example,
+ tokenizer,
+ max_seq_length=512,
+ pad_to_max_seq_len=True):
+ list_input_ids = []
+ list_token_type_ids = []
+ for text in example:
+ encoded_inputs = tokenizer(
+ text=text,
+ max_seq_len=max_seq_length,
+ pad_to_max_seq_len=pad_to_max_seq_len)
+ input_ids = encoded_inputs["input_ids"]
+ token_type_ids = encoded_inputs["token_type_ids"]
+
+ list_input_ids.append(input_ids)
+ list_token_type_ids.append(token_type_ids)
+ return list_input_ids, list_token_type_ids
+
+
+# 启动python客户端
+endpoint_list = ['127.0.0.1:9393']
+client = HttpClient()
+client.load_client_config('serving_client')
+client.connect(endpoint_list)
+feed_names = client.feed_names_
+fetch_names = client.fetch_names_
+print(feed_names)
+print(fetch_names)
+
+# 创建tokenizer
+tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained('ernie-1.0')
+max_seq_len = 64
+
+# 数据预处理
+
+list_data = ['国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据.', '面向生态系统服务的生态系统分类方案研发与应用']
+# for i in range(5):
+# list_data.extend(list_data)
+# print(len(list_data))
+examples = convert_example(list_data, tokenizer, max_seq_length=max_seq_len)
+print(examples)
+
+feed_dict = {}
+feed_dict['input_ids'] = np.array(examples[0])
+feed_dict['token_type_ids'] = np.array(examples[1])
+
+print(feed_dict['input_ids'].shape)
+print(feed_dict['token_type_ids'].shape)
+
+# batch设置为True表示的是批量预测
+b_start = time.time()
+result = client.predict(feed=feed_dict, fetch=fetch_names, batch=True)
+b_end = time.time()
+print(result)
+print("time to cost :{} seconds".format(b_end - b_start))
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/rpc_client.py b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..9ea4c245f2a10256166a512f9282282e69d9997b
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/rpc_client.py
@@ -0,0 +1,77 @@
+# coding:utf-8
+# pylint: disable=doc-string-missing
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import time
+import numpy as np
+
+from paddle_serving_client import Client
+import paddlenlp as ppnlp
+
+
+def convert_example(example,
+ tokenizer,
+ max_seq_length=512,
+ pad_to_max_seq_len=True):
+ list_input_ids = []
+ list_token_type_ids = []
+ for text in example:
+ encoded_inputs = tokenizer(
+ text=text,
+ max_seq_len=max_seq_length,
+ pad_to_max_seq_len=pad_to_max_seq_len)
+ input_ids = encoded_inputs["input_ids"]
+ token_type_ids = encoded_inputs["token_type_ids"]
+ list_input_ids.append(input_ids)
+ list_token_type_ids.append(token_type_ids)
+ return list_input_ids, list_token_type_ids
+
+
+# 启动python客户端
+endpoint_list = ['127.0.0.1:9393']
+client = Client()
+client.load_client_config('serving_client')
+client.connect(endpoint_list)
+feed_names = client.feed_names_
+fetch_names = client.fetch_names_
+print(feed_names)
+print(fetch_names)
+
+# 创建tokenizer
+tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained('ernie-1.0')
+max_seq_len = 64
+
+# 数据预处理
+
+list_data = ['国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据.', '面向生态系统服务的生态系统分类方案研发与应用']
+# for i in range(5):
+# list_data.extend(list_data)
+# print(len(list_data))
+examples = convert_example(list_data, tokenizer, max_seq_length=max_seq_len)
+print(examples)
+
+feed_dict = {}
+feed_dict['input_ids'] = np.array(examples[0])
+feed_dict['token_type_ids'] = np.array(examples[1])
+
+print(feed_dict['input_ids'].shape)
+print(feed_dict['token_type_ids'].shape)
+# batch设置为True表示的是批量预测
+b_start = time.time()
+result = client.predict(feed=feed_dict, fetch=fetch_names, batch=True)
+b_end = time.time()
+print("time to cost :{} seconds".format(b_end - b_start))
+print(result)
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/start_server.sh b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/start_server.sh
new file mode 100644
index 0000000000000000000000000000000000000000..55d380d6f87396887675a008c54bb8544ce2a793
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/C++/start_server.sh
@@ -0,0 +1 @@
+python -m paddle_serving_server.serve --model serving_server --port 9393 --gpu_id 2 --thread 5 --ir_optim True --use_trt --precision FP16
\ No newline at end of file
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/config_nlp.yml b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/config_nlp.yml
new file mode 100644
index 0000000000000000000000000000000000000000..d896adbfa1f9671cb569137637cf5f3ec169ef69
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/config_nlp.yml
@@ -0,0 +1,34 @@
+# worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+# 当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
+# build_dag_each_worker, False，框架在进程内创建一条DAG；True，框架会每个进程内创建多个独立的DAG
+build_dag_each_worker: false
+
+dag:
+ # op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+ # 使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ tracer:
+ interval_s: 10
+# http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18082
+# rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 8088
+op:
+ ernie:
+ # 并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+ # 当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+ # client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+ #ir_optim
+ ir_optim: True
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+ # 计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: '2'
+ # Fetch结果列表，以client_config中fetch_var的alias_name为准, 如果没有设置则全部返回
+ fetch_list: ['output_embedding']
+ # 模型路径
+ model_config: ../../serving_server/
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/deploy.sh b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/deploy.sh
new file mode 100644
index 0000000000000000000000000000000000000000..fe8f071e0a47a47f5dc24d84ea4eaaf8e7503c06
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/deploy.sh
@@ -0,0 +1 @@
+python predict.py --model_dir=../../output
\ No newline at end of file
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/predict.py b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/predict.py
new file mode 100644
index 0000000000000000000000000000000000000000..0e81dbb5092ce6178587f5aa8f40d758f4446a42
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/predict.py
@@ -0,0 +1,292 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+import sys
+
+import numpy as np
+import paddle
+import paddlenlp as ppnlp
+from scipy.special import softmax
+from scipy import spatial
+from paddle import inference
+from paddlenlp.data import Stack, Tuple, Pad
+from paddlenlp.datasets import load_dataset
+from paddlenlp.utils.log import logger
+
+sys.path.append('.')
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_dir", type=str, required=True,
+ help="The directory to static model.")
+
+parser.add_argument("--max_seq_length", default=128, type=int,
+ help="The maximum total input sequence length after tokenization. Sequences "
+ "longer than this will be truncated, sequences shorter will be padded.")
+parser.add_argument("--batch_size", default=15, type=int,
+ help="Batch size per GPU/CPU for training.")
+parser.add_argument('--device', choices=['cpu', 'gpu', 'xpu'], default="gpu",
+ help="Select which device to train model, defaults to gpu.")
+
+parser.add_argument('--use_tensorrt', default=False, type=eval, choices=[True, False],
+ help='Enable to use tensorrt to speed up.')
+parser.add_argument("--precision", default="fp32", type=str, choices=["fp32", "fp16", "int8"],
+ help='The tensorrt precision.')
+
+parser.add_argument('--cpu_threads', default=10, type=int,
+ help='Number of threads to predict when using cpu.')
+parser.add_argument('--enable_mkldnn', default=False, type=eval, choices=[True, False],
+ help='Enable to use mkldnn to speed up when using cpu.')
+
+parser.add_argument("--benchmark", type=eval, default=False,
+ help="To log some information about environment and running.")
+parser.add_argument("--save_log_path", type=str, default="./log_output/",
+ help="The file path to save log.")
+args = parser.parse_args()
+# yapf: enable
+
+
+def convert_example(example,
+ tokenizer,
+ max_seq_length=512,
+ pad_to_max_seq_len=False):
+ """
+ Builds model inputs from a sequence.
+
+ A BERT sequence has the following format:
+
+ - single sequence: ``[CLS] X [SEP]``
+
+ Args:
+ example(obj:`list(str)`): The list of text to be converted to ids.
+ tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
+ which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+ max_seq_len(obj:`int`): The maximum total input sequence length after tokenization.
+ Sequences longer than this will be truncated, sequences shorter will be padded.
+ is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+
+ Returns:
+ input_ids(obj:`list[int]`): The list of query token ids.
+ token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+ """
+
+ result = []
+ for key, text in example.items():
+ encoded_inputs = tokenizer(
+ text=text,
+ max_seq_len=max_seq_length,
+ pad_to_max_seq_len=pad_to_max_seq_len)
+ input_ids = encoded_inputs["input_ids"]
+ token_type_ids = encoded_inputs["token_type_ids"]
+ result += [input_ids, token_type_ids]
+ return result
+
+
+class Predictor(object):
+ def __init__(self,
+ model_dir,
+ device="gpu",
+ max_seq_length=128,
+ batch_size=32,
+ use_tensorrt=False,
+ precision="fp32",
+ cpu_threads=10,
+ enable_mkldnn=False):
+ self.max_seq_length = max_seq_length
+ self.batch_size = batch_size
+
+ model_file = model_dir + "/inference.pdmodel"
+ params_file = model_dir + "/inference.pdiparams"
+ if not os.path.exists(model_file):
+ raise ValueError("not find model file path {}".format(model_file))
+ if not os.path.exists(params_file):
+ raise ValueError("not find params file path {}".format(params_file))
+ config = paddle.inference.Config(model_file, params_file)
+
+ if device == "gpu":
+ # set GPU configs accordingly
+ # such as intialize the gpu memory, enable tensorrt
+ config.enable_use_gpu(100, 0)
+ precision_map = {
+ "fp16": inference.PrecisionType.Half,
+ "fp32": inference.PrecisionType.Float32,
+ "int8": inference.PrecisionType.Int8
+ }
+ precision_mode = precision_map[precision]
+
+ if args.use_tensorrt:
+ config.enable_tensorrt_engine(
+ max_batch_size=batch_size,
+ min_subgraph_size=30,
+ precision_mode=precision_mode)
+ elif device == "cpu":
+ # set CPU configs accordingly,
+ # such as enable_mkldnn, set_cpu_math_library_num_threads
+ config.disable_gpu()
+ if args.enable_mkldnn:
+ # cache 10 different shapes for mkldnn to avoid memory leak
+ config.set_mkldnn_cache_capacity(10)
+ config.enable_mkldnn()
+ config.set_cpu_math_library_num_threads(args.cpu_threads)
+ elif device == "xpu":
+ # set XPU configs accordingly
+ config.enable_xpu(100)
+
+ config.switch_use_feed_fetch_ops(False)
+ self.predictor = paddle.inference.create_predictor(config)
+ self.input_handles = [
+ self.predictor.get_input_handle(name)
+ for name in self.predictor.get_input_names()
+ ]
+ self.output_handle = self.predictor.get_output_handle(
+ self.predictor.get_output_names()[0])
+
+ if args.benchmark:
+ import auto_log
+ pid = os.getpid()
+ self.autolog = auto_log.AutoLogger(
+ model_name="ernie-1.0",
+ model_precision=precision,
+ batch_size=self.batch_size,
+ data_shape="dynamic",
+ save_path=args.save_log_path,
+ inference_config=config,
+ pids=pid,
+ process_name=None,
+ gpu_ids=0,
+ time_keys=[
+ 'preprocess_time', 'inference_time', 'postprocess_time'
+ ],
+ warmup=0,
+ logger=logger)
+
+ def extract_embedding(self, data, tokenizer):
+ """
+ Predicts the data labels.
+
+ Args:
+ data (obj:`List(str)`): The batch data whose each element is a raw text.
+ tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
+ which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+
+ Returns:
+ results(obj:`dict`): All the feature vectors.
+ """
+ if args.benchmark:
+ self.autolog.times.start()
+
+ examples = []
+ for text in data:
+ input_ids, segment_ids = convert_example(text, tokenizer)
+ examples.append((input_ids, segment_ids))
+
+ batchify_fn = lambda samples, fn=Tuple(
+ Pad(axis=0, pad_val=tokenizer.pad_token_id), # input
+ Pad(axis=0, pad_val=tokenizer.pad_token_id), # segment
+ ): fn(samples)
+
+ if args.benchmark:
+ self.autolog.times.stamp()
+
+ input_ids, segment_ids = batchify_fn(examples)
+ self.input_handles[0].copy_from_cpu(input_ids)
+ self.input_handles[1].copy_from_cpu(segment_ids)
+ self.predictor.run()
+ logits = self.output_handle.copy_to_cpu()
+ if args.benchmark:
+ self.autolog.times.stamp()
+
+ if args.benchmark:
+ self.autolog.times.end(stamp=True)
+
+ return logits
+
+ def predict(self, data, tokenizer):
+ """
+ Predicts the data labels.
+
+ Args:
+ data (obj:`List(str)`): The batch data whose each element is a raw text.
+ tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
+ which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+
+ Returns:
+ results(obj:`dict`): All the predictions probs.
+ """
+ if args.benchmark:
+ self.autolog.times.start()
+
+ examples = []
+ for idx, text in enumerate(data):
+ input_ids, segment_ids = convert_example({idx: text[0]}, tokenizer)
+ title_ids, title_segment_ids = convert_example({
+ idx: text[1]
+ }, tokenizer)
+ examples.append(
+ (input_ids, segment_ids, title_ids, title_segment_ids))
+
+ batchify_fn = lambda samples, fn=Tuple(
+ Pad(axis=0, pad_val=tokenizer.pad_token_id), # input
+ Pad(axis=0, pad_val=tokenizer.pad_token_id), # segment
+ Pad(axis=0, pad_val=tokenizer.pad_token_id), # segment
+ Pad(axis=0, pad_val=tokenizer.pad_token_id), # segment
+ ): fn(samples)
+
+ if args.benchmark:
+ self.autolog.times.stamp()
+
+ query_ids, query_segment_ids, title_ids, title_segment_ids = batchify_fn(
+ examples)
+ self.input_handles[0].copy_from_cpu(query_ids)
+ self.input_handles[1].copy_from_cpu(query_segment_ids)
+ self.predictor.run()
+ query_logits = self.output_handle.copy_to_cpu()
+
+ self.input_handles[0].copy_from_cpu(title_ids)
+ self.input_handles[1].copy_from_cpu(title_segment_ids)
+ self.predictor.run()
+ title_logits = self.output_handle.copy_to_cpu()
+
+ if args.benchmark:
+ self.autolog.times.stamp()
+
+ if args.benchmark:
+ self.autolog.times.end(stamp=True)
+ result = [
+ float(1 - spatial.distance.cosine(arr1, arr2))
+ for arr1, arr2 in zip(query_logits, title_logits)
+ ]
+ return result
+
+
+if __name__ == "__main__":
+ # Define predictor to do prediction.
+ predictor = Predictor(args.model_dir, args.device, args.max_seq_length,
+ args.batch_size, args.use_tensorrt, args.precision,
+ args.cpu_threads, args.enable_mkldnn)
+
+ # ErnieTinyTokenizer is special for ernie-tiny pretained model.
+ output_emb_size = 256
+ tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained('ernie-1.0')
+ id2corpus = {0: '国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据'}
+ corpus_list = [{idx: text} for idx, text in id2corpus.items()]
+ res = predictor.extract_embedding(corpus_list, tokenizer)
+ print(res.shape)
+ print(res)
+ corpus_list = [['中西方语言与文化的差异', '中西方文化差异以及语言体现中西方文化,差异,语言体现'],
+ ['中西方语言与文化的差异', '飞桨致力于让深度学习技术的创新与应用更简单']]
+ res = predictor.predict(corpus_list, tokenizer)
+ print(res)
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/rpc_client.py b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..03863db6114b7c381dae17ee3bf33f00f15d8f4a
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/rpc_client.py
@@ -0,0 +1,39 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import time
+import numpy as np
+
+from paddle_serving_server.pipeline import PipelineClient
+
+client = PipelineClient()
+client.connect(['127.0.0.1:8088'])
+
+list_data = [
+ "国有企业引入非国有资本对创新绩效的影响——基于制造业国有上市公司的经验证据",
+ "试论翻译过程中的文化差异与语言空缺翻译过程,文化差异,语言空缺,文化对比"
+]
+feed = {}
+for i, item in enumerate(list_data):
+ feed[str(i)] = item
+
+print(feed)
+start_time = time.time()
+ret = client.predict(feed_dict=feed)
+end_time = time.time()
+print("time to cost :{} seconds".format(end_time - start_time))
+
+result = np.array(eval(ret.value[0]))
+print(ret.key)
+print(result.shape)
+print(result)
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/web_service.py b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..1ad12032b3c92d72a5297f15d732b7dfbd19589e
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/deploy/python/web_service.py
@@ -0,0 +1,82 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import numpy as np
+import sys
+
+from paddle_serving_server.web_service import WebService, Op
+
+_LOGGER = logging.getLogger()
+
+
+def convert_example(example,
+ tokenizer,
+ max_seq_length=512,
+ pad_to_max_seq_len=False):
+ result = []
+ for text in example:
+ encoded_inputs = tokenizer(
+ text=text,
+ max_seq_len=max_seq_length,
+ pad_to_max_seq_len=pad_to_max_seq_len)
+ input_ids = encoded_inputs["input_ids"]
+ token_type_ids = encoded_inputs["token_type_ids"]
+ result += [input_ids, token_type_ids]
+ return result
+
+
+class ErnieOp(Op):
+ def init_op(self):
+ import paddlenlp as ppnlp
+ self.tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained(
+ 'ernie-1.0')
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ from paddlenlp.data import Stack, Tuple, Pad
+
+ (_, input_dict), = input_dicts.items()
+ print("input dict", input_dict)
+ batch_size = len(input_dict.keys())
+ examples = []
+ for i in range(batch_size):
+ input_ids, segment_ids = convert_example([input_dict[str(i)]],
+ self.tokenizer)
+ examples.append((input_ids, segment_ids))
+ batchify_fn = lambda samples, fn=Tuple(
+ Pad(axis=0, pad_val=self.tokenizer.pad_token_id), # input
+ Pad(axis=0, pad_val=self.tokenizer.pad_token_id), # segment
+ ): fn(samples)
+ input_ids, segment_ids = batchify_fn(examples)
+ feed_dict = {}
+ feed_dict['input_ids'] = input_ids
+ feed_dict['token_type_ids'] = segment_ids
+ return feed_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ new_dict = {}
+ new_dict["output_embedding"] = str(fetch_dict["output_embedding"]
+ .tolist())
+ return new_dict, None, ""
+
+
+class ErnieService(WebService):
+ def get_pipeline_response(self, read_op):
+ ernie_op = ErnieOp(name="ernie", input_ops=[read_op])
+ return ernie_op
+
+
+ernie_service = ErnieService(name="ernie")
+ernie_service.prepare_pipeline_config("config_nlp.yml")
+ernie_service.run_service()
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/export_model.py b/examples/Pipeline/PaddleNLP/semantic_indexing/export_model.py
new file mode 100644
index 0000000000000000000000000000000000000000..da468ea7b2c3af6eff093eef98a3e4f9393f9b3d
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/export_model.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+from functools import partial
+
+import numpy as np
+import paddle
+import paddle.nn.functional as F
+import paddlenlp as ppnlp
+from paddlenlp.data import Stack, Tuple, Pad
+
+from base_model import SemanticIndexBase, SemanticIndexBaseStatic
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--params_path", type=str, required=True,
+ default='./checkpoint/model_900/model_state.pdparams', help="The path to model parameters to be loaded.")
+parser.add_argument("--output_path", type=str, default='./output',
+ help="The path of model parameter in static graph to be saved.")
+args = parser.parse_args()
+# yapf: enable
+
+if __name__ == "__main__":
+ # If you want to use ernie1.0 model, plesace uncomment the following code
+ output_emb_size = 256
+
+ pretrained_model = ppnlp.transformers.ErnieModel.from_pretrained(
+ "ernie-1.0")
+
+ tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained('ernie-1.0')
+ model = SemanticIndexBaseStatic(
+ pretrained_model, output_emb_size=output_emb_size)
+
+ if args.params_path and os.path.isfile(args.params_path):
+ state_dict = paddle.load(args.params_path)
+ model.set_dict(state_dict)
+ print("Loaded parameters from %s" % args.params_path)
+
+ model.eval()
+
+ # Convert to static graph with specific input description
+ model = paddle.jit.to_static(
+ model,
+ input_spec=[
+ paddle.static.InputSpec(
+ shape=[None, None], dtype="int64"), # input_ids
+ paddle.static.InputSpec(
+ shape=[None, None], dtype="int64") # segment_ids
+ ])
+ # Save in static graph model.
+ save_path = os.path.join(args.output_path, "inference")
+ paddle.jit.save(model, save_path)
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/export_to_serving.py b/examples/Pipeline/PaddleNLP/semantic_indexing/export_to_serving.py
new file mode 100644
index 0000000000000000000000000000000000000000..c24f931510e5662ae1b824049d1ac35c4ef34076
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/export_to_serving.py
@@ -0,0 +1,47 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import paddle_serving_client.io as serving_io
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--dirname", type=str, required=True,
+ default='./output', help="Path of saved model files. Program file and parameter files are saved in this directory.")
+parser.add_argument("--model_filename", type=str, required=True,
+ default='inference.get_pooled_embedding.pdmodel', help="The name of file to load the inference program. If it is None, the default filename __model__ will be used.")
+parser.add_argument("--params_filename", type=str, required=True,
+ default='inference.get_pooled_embedding.pdiparams', help="The name of file to load all parameters. It is only used for the case that all parameters were saved in a single binary file. If parameters were saved in separate files, set it as None. Default: None.")
+parser.add_argument("--server_path", type=str, default='./serving_server',
+ help="The path of server parameter in static graph to be saved.")
+parser.add_argument("--client_path", type=str, default='./serving_client',
+ help="The path of client parameter in static graph to be saved.")
+parser.add_argument("--feed_alias_names", type=str, default=None,
+ help='set alias names for feed vars, split by comma \',\', you should run --show_proto to check the number of feed vars')
+parser.add_argument("--fetch_alias_names", type=str, default=None,
+ help='set alias names for feed vars, split by comma \',\', you should run --show_proto to check the number of fetch vars')
+parser.add_argument("--show_proto", type=bool, default=False,
+ help='If yes, you can preview the proto and then determine your feed var alias name and fetch var alias name.')
+# yapf: enable
+
+if __name__ == "__main__":
+ args = parser.parse_args()
+ serving_io.inference_model_to_serving(
+ dirname=args.dirname,
+ serving_server=args.server_path,
+ serving_client=args.client_path,
+ model_filename=args.model_filename,
+ params_filename=args.params_filename,
+ show_proto=args.show_proto,
+ feed_alias_names=args.feed_alias_names,
+ fetch_alias_names=args.fetch_alias_names)
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/scripts/export_model.sh b/examples/Pipeline/PaddleNLP/semantic_indexing/scripts/export_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..7c79266219cea03e16968ed0d00a3755615c7432
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/scripts/export_model.sh
@@ -0,0 +1 @@
+python export_model.py --params_path checkpoints/model_40/model_state.pdparams --output_path=./output
\ No newline at end of file
diff --git a/examples/Pipeline/PaddleNLP/semantic_indexing/scripts/export_to_serving.sh b/examples/Pipeline/PaddleNLP/semantic_indexing/scripts/export_to_serving.sh
new file mode 100644
index 0000000000000000000000000000000000000000..b0d7a422551fd09eb1a28cfacdf47237a8efc795
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/semantic_indexing/scripts/export_to_serving.sh
@@ -0,0 +1,7 @@
+python export_to_serving.py \
+ --dirname "output" \
+ --model_filename "inference.get_pooled_embedding.pdmodel" \
+ --params_filename "inference.get_pooled_embedding.pdiparams" \
+ --server_path "serving_server" \
+ --client_path "serving_client" \
+ --fetch_alias_names "output_embedding"
diff --git a/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py b/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py
index 9952271430b2e260be292b63f39eaf1105b06521..f60d357df7b1ad4e2ce0e4e3f327733090db2947 100644
--- a/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py
+++ b/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py
@@ -29,9 +29,26 @@ test_img_dir = "imgs/"
for img_file in os.listdir(test_img_dir):
with open(os.path.join(test_img_dir, img_file), 'rb') as file:
image_data1 = file.read()
+ # print file name
+ print('{}{}{}'.format('*' * 10, img_file, '*' * 10))
image = cv2_to_base64(image_data1)
-for i in range(4):
data = {"key": ["image"], "value": [image]}
r = requests.post(url=url, data=json.dumps(data))
- print(r.json())
+ result = r.json()
+ print("erro_no:{}, err_msg:{}".format(result["err_no"], result["err_msg"]))
+ # check success
+ if result["err_no"] == 0:
+ ocr_result = result["value"][0]
+ try:
+ for item in eval(ocr_result):
+ # return transcription and points
+ print("{}, {}".format(item[0], item[1]))
+ except Exception as e:
+ print("No results")
+ continue
+
+ else:
+ print(
+ "For details about error message, see PipelineServingLogs/pipeline.log.wf"
+ )
diff --git a/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py b/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py
index 66faa0428cfcbd97a5fb0d340d4ee07be828cda2..e37182359fa66005e112a9fc04e66b2fb4326718 100644
--- a/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py
+++ b/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py
@@ -34,8 +34,24 @@ test_img_dir = "imgs/"
for img_file in os.listdir(test_img_dir):
with open(os.path.join(test_img_dir, img_file), 'rb') as file:
image_data = file.read()
+ # print file name
+ print('{}{}{}'.format('*' * 10, img_file, '*' * 10))
image = cv2_to_base64(image_data)
-for i in range(1):
- ret = client.predict(feed_dict={"image": image}, fetch=["res"])
- print(ret)
+ result = client.predict(feed_dict={"image": image}, fetch=["res"])
+ print("erro_no:{}, err_msg:{}".format(result.err_no, result.err_msg))
+ # check success
+ if result.err_no == 0:
+ ocr_result = result.value[0]
+ try:
+ for item in eval(ocr_result):
+ # return transcription and points
+ print("{}, {}".format(item[0], item[1]))
+ except Exception as e:
+ print("No results")
+ continue
+
+ else:
+ print(
+ "For details about error message, see PipelineServingLogs/pipeline.log.wf"
+ )
diff --git a/examples/Pipeline/PaddleOCR/ocr/web_service.py b/examples/Pipeline/PaddleOCR/ocr/web_service.py
index 5cb1ada570287f98340d9456175b58558f71e8f6..0ef748cedf45aecdfa3193329ac440ef4ab1f58a 100644
--- a/examples/Pipeline/PaddleOCR/ocr/web_service.py
+++ b/examples/Pipeline/PaddleOCR/ocr/web_service.py
@@ -14,6 +14,7 @@
from paddle_serving_server.web_service import WebService, Op
import logging
import numpy as np
+import copy
import cv2
import base64
from paddle_serving_app.reader import OCRReader
@@ -34,17 +35,18 @@ class DetOp(Op):
self.filter_func = FilterBoxes(10, 10)
self.post_func = DBPostProcess({
"thresh": 0.3,
- "box_thresh": 0.5,
+ "box_thresh": 0.6,
"max_candidates": 1000,
"unclip_ratio": 1.5,
"min_size": 3
})
-
+
"""
when opening tensorrt(configure in config.yml) and each time the input shape
for inferring is different, using this method for configuring tensorrt
dynamic shape to infer in each op model
"""
+
def set_dynamic_shape_info(self):
min_input_shape = {
"x": [1, 3, 50, 50],
@@ -74,7 +76,7 @@ class DetOp(Op):
"min_input_shape": min_input_shape,
"max_input_shape": max_input_shape,
"opt_input_shape": opt_input_shape,
- }
+ }
def preprocess(self, input_dicts, data_id, log_id):
(_, input_dict), = input_dicts.items()
@@ -107,25 +109,20 @@ class RecOp(Op):
self.ocr_reader = OCRReader()
self.get_rotate_crop_image = GetRotateCropImage()
self.sorted_boxes = SortedBoxes()
-
+
"""
when opening tensorrt(configure in config.yml) and each time the input shape
for inferring is different, using this method for configuring tensorrt
dynamic shape to infer in each op model
"""
+
def set_dynamic_shape_info(self):
- min_input_shape = {
- "x": [1, 3, 32, 10],
- "lstm_1.tmp_0": [1, 1, 128]
- }
+ min_input_shape = {"x": [1, 3, 32, 10], "lstm_1.tmp_0": [1, 1, 128]}
max_input_shape = {
"x": [50, 3, 32, 1000],
"lstm_1.tmp_0": [500, 50, 128]
}
- opt_input_shape = {
- "x": [6, 3, 32, 100],
- "lstm_1.tmp_0": [25, 5, 128]
- }
+ opt_input_shape = {"x": [6, 3, 32, 100], "lstm_1.tmp_0": [25, 5, 128]}
self.dynamic_shape_info = {
"min_input_shape": min_input_shape,
"max_input_shape": max_input_shape,
@@ -137,8 +134,10 @@ class RecOp(Op):
raw_im = input_dict["image"]
data = np.frombuffer(raw_im, np.uint8)
im = cv2.imdecode(data, cv2.IMREAD_COLOR)
- dt_boxes = input_dict["dt_boxes"]
- dt_boxes = self.sorted_boxes(dt_boxes)
+ self.dt_list = input_dict["dt_boxes"]
+ self.dt_list = self.sorted_boxes(self.dt_list)
+ # deepcopy to save origin dt_boxes
+ dt_boxes = copy.deepcopy(self.dt_list)
feed_list = []
img_list = []
max_wh_ratio = 0
@@ -205,26 +204,31 @@ class RecOp(Op):
imgs[id] = norm_img
feed = {"x": imgs.copy()}
feed_list.append(feed)
- #_LOGGER.info("feed_list : {}".format(feed_list))
return feed_list, False, None, ""
def postprocess(self, input_dicts, fetch_data, data_id, log_id):
- res_list = []
+ rec_list = []
+ dt_num = len(self.dt_list)
if isinstance(fetch_data, dict):
if len(fetch_data) > 0:
rec_batch_res = self.ocr_reader.postprocess_ocrv2(
fetch_data, with_score=True)
for res in rec_batch_res:
- res_list.append(res[0])
+ rec_list.append(res)
elif isinstance(fetch_data, list):
for one_batch in fetch_data:
one_batch_res = self.ocr_reader.postprocess_ocrv2(
one_batch, with_score=True)
for res in one_batch_res:
- res_list.append(res[0])
-
- res = {"res": str(res_list)}
+ rec_list.append(res)
+
+ result_list = []
+ for i in range(dt_num):
+ text = rec_list[i]
+ dt_box = self.dt_list[i]
+ result_list.append([text, dt_box.tolist()])
+ res = {"result": str(result_list)}
return res, None, ""
diff --git a/java/src/main/proto/general_model_service.proto b/java/src/main/proto/general_model_service.proto
index aa06d388a468d71e968aa53b19f25c55f8c42ee1..6b0fc7e0d585d45ca45f26cf9083a7f116a25d33 100644
--- a/java/src/main/proto/general_model_service.proto
+++ b/java/src/main/proto/general_model_service.proto
@@ -89,11 +89,13 @@ message Request {
message Response {
repeated ModelOutput outputs = 1;
repeated int64 profile_time = 2;
- // Error code
- int32 err_no = 3;
+ bool profile_server = 3;
+ uint64 log_id = 4;
+ // Error code
+ int32 err_no = 5;
// Error messages
- string err_msg = 4;
+ string err_msg = 6;
};
message ModelOutput {
diff --git a/paddle_inference/paddle/include/paddle_engine.h b/paddle_inference/paddle/include/paddle_engine.h
index bf8c98ede60bc4266965d1aa12e2627dd0d0647a..2d76730555acb6ed0408584db1334e842db126c3 100644
--- a/paddle_inference/paddle/include/paddle_engine.h
+++ b/paddle_inference/paddle/include/paddle_engine.h
@@ -37,6 +37,7 @@ using paddle_infer::PrecisionType;
using paddle_infer::Predictor;
using paddle_infer::Tensor;
using paddle_infer::CreatePredictor;
+using paddle_infer::DistConfig;
DECLARE_int32(gpuid);
DECLARE_string(precision);
@@ -206,11 +207,44 @@ class PaddleInferenceEngine : public EngineCore {
config.SetModel(model_path);
}
+ // Enable distributed model inferencing
+ DistConfig distCfg;
+ if (engine_conf.has_enable_dist_model() &&
+ engine_conf.enable_dist_model()) {
+ int ep_size = engine_conf.dist_endpoints_size();
+ int cur_index = engine_conf.dist_subgraph_index();
+ if (ep_size <= cur_index) {
+ LOG(ERROR) << "create paddle predictor failed, Distributed model error."
+ << " dist_endpoints_size=" << ep_size
+ << " is not bigger than dist_subgraph_index=" << cur_index;
+ return -1;
+ }
+ std::vector