Merge branch 'develop' into v0.8.0_doc

README.md

@@ -62,7 +62,7 @@ This chapter guides you through the installation and deployment steps. It is str
 - [Build Paddle Serving from Source with Docker](doc/Compile_EN.md)
 - [Deploy Paddle Serving on Kubernetes(Chinese)](doc/Run_On_Kubernetes_CN.md)
 - [Deploy Paddle Serving with Security gateway(Chinese)](doc/Serving_Auth_Docker_CN.md)
-- Deploy on more hardwares[[百度昆仑](doc/Run_On_XPU_CN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
+- Deploy on more hardwares[[ARM CPU、百度昆仑](doc/Run_On_XPU_EN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
 - [Docker Images](doc/Docker_Images_EN.md)
 - [Latest Wheel packages](doc/Latest_Packages_CN.md)
 

README_CN.md

@@ -58,7 +58,7 @@ Paddle Serving依托深度学习框架PaddlePaddle旨在帮助深度学习开发
 - [源码编译安装Paddle Serving](doc/Compile_CN.md)
 - [在Kuberntes集群上部署Paddle Serving](doc/Run_On_Kubernetes_CN.md)
 - [部署Paddle Serving安全网关](doc/Serving_Auth_Docker_CN.md)
-- 异构硬件部署[[百度昆仑](doc/Run_On_XPU_CN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
+- 异构硬件部署[[ARM CPU、百度昆仑](doc/Run_On_XPU_CN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
 - [Docker镜像](doc/Docker_Images_CN.md)
 - [最新Wheel开发包(English)](doc/Latest_Packages_CN.md)
 

doc/Install_EN.md

@@ -62,6 +62,8 @@ pip3 install -r python/requirements.txt
 
 Install the service whl package. There are three types of client, app and server. The server is divided into CPU and GPU. Choose one installation according to the environment. 
 - GPU with CUDA10.2 + Cudnn7 + TensorRT6(Recommended)
+- post101 = CUDA10.1 + TensorRT6
+- post112 = CUDA11.2 + TensorRT8
 ```shell
 pip3 install paddle-serving-client==0.8.2 -i https://pypi.tuna.tsinghua.edu.cn/simple
 pip3 install paddle-serving-app==0.8.2 -i https://pypi.tuna.tsinghua.edu.cn/simple

examples/Pipeline/PaddleClas/DarkNet53-encryption/README.md

+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+python encrypt.py
+```
+
+## Start server
+
+```
+python -m paddle_serving_server.serve --model encrypt_server/ --port 9400  --encryption_rpc_port 9401  --use_encryption_model &
+python web_service.py &>log.txt &
+```
+
+## client test
+```
+python http_client.py
+```
+
+if you configure the api gateway, you can use `https_client.py`

examples/Pipeline/PaddleClas/DarkNet53-encryption/README_CN.md

+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+python encrypt.py
+```
+
+## 启动服务
+
+```
+python -m paddle_serving_server.serve --model encrypt_server/ --port 9400  --encryption_rpc_port 9401  --use_encryption_model &
+python web_service.py &>log.txt &
+```
+
+## 测试
+```
+python http_client.py
+```
+如果您已经配置好了api gateway， 您可以使用 `https_client.py`
+
+~

examples/Pipeline/PaddleClas/DarkNet53-encryption/config.yml

+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+    #op资源类型, True, 为线程模型；False，为进程模型
+    is_thread_op: False
+op:
+    imagenet:
+        #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+        concurrency: 1
+        client_type: brpc
+        retry: 1
+        timeout: 3000
+        server_endpoints: ["127.0.0.1:9400"]
+        client_config: "encrypt_client"
+        fetch_list: ["save_infer_model/scale_0.tmp_0"]
+        batch_size: 1
+        auto_batching_timeout: 2000
+        use_encryption_model: True
+        encryption_key: "./key" 

examples/Pipeline/PaddleClas/DarkNet53-encryption/encrypt.py

+from paddle_serving_client.io import inference_model_to_serving
+
+
+def serving_encryption():
+    inference_model_to_serving(
+        dirname="./DarkNet53/ppcls_model/",
+        model_filename="__model__",
+        params_filename="./__params__",
+        serving_server="encrypt_server",
+        serving_client="encrypt_client",
+        encryption=True)
+
+
+if __name__ == "__main__":
+    serving_encryption()

examples/Pipeline/PaddleClas/DarkNet53-encryption/get_model.sh

+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/DarkNet53.tar
+tar -xf DarkNet53.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz

examples/Pipeline/PaddleClas/DarkNet53-encryption/http_client.py

+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+    return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+    url = "http://127.0.0.1:18080/imagenet/prediction"
+    with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+        image_data1 = file.read()
+    image = cv2_to_base64(image_data1)
+    data = {"key": ["image"], "value": [image]}
+    for i in range(1):
+        r = requests.post(url=url, data=json.dumps(data))
+        print(r.json())

examples/Pipeline/PaddleClas/DarkNet53-encryption/https_client.py

+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+    return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+    url = "https://10.21.8.132:8443/image-clas/imagenet/prediction"
+    with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+        image_data1 = file.read()
+    image = cv2_to_base64(image_data1)
+    headers = {"Content-Type":"application/json", "apikey":"BlfvO08Z9mQpFjcMagl2dxOIA8h2UVdp", "X-INSTANCE-ID" : "kong_ins10"}
+    data = {"key": ["image"], "value": [image]}
+    for i in range(1):
+        r = requests.post(url=url, headers=headers, data=json.dumps(data),verify=False)
+        print(r.json())

examples/Pipeline/PaddleClas/DarkNet53-encryption/web_service.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+    def init_op(self):
+        self.seq = Sequential([
+            Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+            Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+                                True)
+        ])
+        self.label_dict = {}
+        label_idx = 0
+        with open("imagenet.label") as fin:
+            for line in fin:
+                self.label_dict[label_idx] = line.strip()
+                label_idx += 1
+
+    def preprocess(self, input_dicts, data_id, log_id):
+        (_, input_dict), = input_dicts.items()
+        batch_size = len(input_dict.keys())
+        imgs = []
+        for key in input_dict.keys():
+            data = base64.b64decode(input_dict[key].encode('utf8'))
+            data = np.fromstring(data, np.uint8)
+            im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+            img = self.seq(im)
+            imgs.append(img[np.newaxis, :].copy())
+        input_imgs = np.concatenate(imgs, axis=0)
+        return {"image": input_imgs}, False, None, ""
+
+    def postprocess(self, input_dicts, fetch_dict, data_id=0, log_id=0):
+        score_list = fetch_dict["save_infer_model/scale_0.tmp_0"]
+        result = {"label": [], "prob": []}
+        for score in score_list:
+            score = score.tolist()
+            max_score = max(score)
+            result["label"].append(self.label_dict[score.index(max_score)]
+                                   .strip().replace(",", ""))
+            result["prob"].append(max_score)
+        result["label"] = str(result["label"])
+        result["prob"] = str(result["prob"])
+        return result, None, ""
+
+
+class ImageService(WebService):
+    def get_pipeline_response(self, read_op):
+        image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+        return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()

examples/Pipeline/PaddleVideo/PPTSN_K400/README_CN.md

+# PP-TSN 视频分类模型
+
+## 模型简介
+PP-TSN模型是对TSN模型进行改进，在不增加参数量和计算量的情况下，得到了的更高精度的2D实用视频分类模型。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 部署模型并测试
+
+### 启动 server
+```
+python3 web_service.py &>log.txt &
+```
+
+## 启动 client
+
+**一. http client**
+```
+python3 pipeline_http_client.py
+```
+**二. rpc client**
+
+```
+python3 pipeline_rpc_client.py
+```

examples/Pipeline/PaddleVideo/PPTSN_K400/pipeline_http_client.py

@@ -16,7 +16,7 @@ import requests
 import json
 
 url = "http://127.0.0.1:9999/ppTSN/prediction"
-video_url = "https://paddle-serving.bj.bcebos.com/huangjianhui04/example.avi"
+video_url = "https://paddle-serving.bj.bcebos.com/model/PaddleVideo/example.avi"
 for i in range(4):
     data = {"key": ["filename"], "value": [video_url]}
     r = requests.post(url=url, data=json.dumps(data))

examples/Pipeline/PaddleVideo/PPTSN_K400/pipeline_rpc_client.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+try:
+    from paddle_serving_server.pipeline import PipelineClient
+except ImportError:
+    from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:18090'])
+
+video_url = "https://paddle-serving.bj.bcebos.com/model/PaddleVideo/example.avi"
+for i in range(1):
+    ret = client.predict(feed_dict={"video_url": video_url}, fetch=["res"])
+    print(ret)

python/paddle_serving_server/serve.py

@@ -184,6 +184,12 @@ def serve_args():
         default=False,
         action="store_true",
         help="Use encryption model")
+    parser.add_argument(
+        "--encryption_rpc_port", 
+        type=int, 
+        required=False, 
+        default=12000, 
+        help="Port of encryption model, only valid for arg.use_encryption_model")
     parser.add_argument(
         "--use_trt", default=False, action="store_true", help="Use TensorRT")
     parser.add_argument(
@@ -352,8 +358,11 @@ def start_multi_card(args, serving_port=None):  # pylint: disable=doc-string-mis
 
 
 class MainService(BaseHTTPRequestHandler):
+    #def __init__(self):
+    #    print("MainService ___init________\n")
     def get_available_port(self):
-        default_port = 12000
+        global encryption_rpc_port
+        default_port = encryption_rpc_port
         for i in range(1000):
             if port_is_available(default_port + i):
                 return default_port + i
@@ -553,7 +562,8 @@ if __name__ == "__main__":
         p_flag = False
         p = None
         serving_port = 0
-        server = HTTPServer(('0.0.0.0', int(args.port)), MainService)
+        encryption_rpc_port = args.encryption_rpc_port
+        server = HTTPServer(('localhost', int(args.port)), MainService)
         print(
             'Starting encryption server, waiting for key from client, use <Ctrl-C> to stop'
         )

python/pipeline/operator.py

@@ -102,6 +102,8 @@ class Op(object):
         self._retry = max(1, retry)
         self._batch_size = batch_size
         self._auto_batching_timeout = auto_batching_timeout
+        self._use_encryption_model = None
+        self._encryption_key = ""
 
         self._input = None
         self._outputs = []
@@ -161,6 +163,11 @@ class Op(object):
             self._fetch_names = conf.get("fetch_list")
         if self._client_config is None:
             self._client_config = conf.get("client_config")
+        if self._use_encryption_model is None:
+            print ("config use_encryption model here", conf.get("use_encryption_model"))
+            self._use_encryption_model = conf.get("use_encryption_model")
+            if self._encryption_key is None or self._encryption_key=="":
+                self._encryption_key = conf.get("encryption_key")
         if self._timeout is None:
             self._timeout = conf["timeout"]
         if self._timeout > 0:
@@ -409,7 +416,12 @@ class Op(object):
             self._fetch_names = client.fetch_names_
             _LOGGER.info("Op({}) has no fetch name set. So fetch all vars")
         if self.client_type != "local_predictor":
-            client.connect(server_endpoints)
+            if self._use_encryption_model is None or self._use_encryption_model is False:
+               client.connect(server_endpoints)
+            else:
+               print("connect to encryption rpc client")
+               client.use_key(self._encryption_key)
+               client.connect(server_endpoints, encryption=True)
         _LOGGER.info("init_client, feed_list:{}, fetch_list: {}".format(self.right_feed_names, self.right_fetch_names))
         return client