rm origin file; and add resnet5_vd paddleserving examples

deploy/paddleserving/README.md

+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python pipeline_rpc_client.py
+```

deploy/paddleserving/README_CN.md

+# PaddleClas 服务化部署
+
+([English](./README.md)|简体中文)
+
+PaddleClas提供2种服务部署方式：
+- 基于PaddleHub Serving的部署：代码路径为"`./deploy/hubserving`"，使用方法参考[文档](../../deploy/hubserving/readme.md)；
+- 基于PaddleServing的部署：代码路径为"`./deploy/paddleserving`"，按照本教程使用。
+
+# 基于PaddleServing的服务部署
+
+本文档以经典的ResNet50_vd模型为例，介绍如何使用[PaddleServing](https://github.com/PaddlePaddle/Serving/blob/develop/README_CN.md)工具部署PaddleClas
+动态图模型的pipeline在线服务。
+
+相比较于hubserving部署，PaddleServing具备以下优点：
+- 支持客户端和服务端之间高并发和高效通信
+- 支持 工业级的服务能力 例如模型管理，在线加载，在线A/B测试等
+- 支持 多种编程语言 开发客户端，例如C++, Python和Java
+
+更多有关PaddleServing服务化部署框架介绍和使用教程参考[文档](https://github.com/PaddlePaddle/Serving/blob/develop/README_CN.md)。
+
+## 目录
+- [环境准备](#环境准备)
+- [模型转换](#模型转换)
+- [Paddle Serving pipeline部署](#部署)
+- [FAQ](#FAQ)
+
+<a name="环境准备"></a>
+## 环境准备
+
+需要准备PaddleClas的运行环境和Paddle Serving的运行环境。
+
+- 准备PaddleOCR的运行环境[链接](../../doc/doc_ch/installation.md)
+  根据环境下载对应的paddle whl包，推荐安装2.0.1版本
+
+- 准备PaddleServing的运行环境，步骤如下
+
+1. 安装serving，用于启动服务
+    ```
+    pip3 install paddle-serving-server==0.6.1 # for CPU
+    pip3 install paddle-serving-server-gpu==0.6.1 # for GPU
+    # 其他GPU环境需要确认环境再选择执行如下命令
+    pip3 install paddle-serving-server-gpu==0.6.1.post101 # GPU with CUDA10.1 + TensorRT6
+    pip3 install paddle-serving-server-gpu==0.6.1.post11 # GPU with CUDA11 + TensorRT7
+    ```
+
+2. 安装client，用于向服务发送请求
+    在[下载链接](https://github.com/PaddlePaddle/Serving/blob/develop/doc/LATEST_PACKAGES.md)中找到对应python版本的client安装包，这里推荐python3.7版本：
+
+    ```
+    wget https://paddle-serving.bj.bcebos.com/test-dev/whl/paddle_serving_client-0.0.0-cp37-none-any.whl
+    pip3 install paddle_serving_client-0.0.0-cp37-none-any.whl
+    ```
+
+3. 安装serving-app
+    ```
+    pip3 install paddle-serving-app==0.6.1
+    ```
+    **Note:** 如果要安装最新版本的PaddleServing参考[链接](https://github.com/PaddlePaddle/Serving/blob/develop/doc/LATEST_PACKAGES.md)。
+
+<a name="模型转换"></a>
+## 模型转换
+
+使用PaddleServing做服务化部署时，需要将保存的inference模型转换为serving易于部署的模型。
+
+首先，下载ResNet50_vd的[inference模型](https://github.com/PaddlePaddle/PaddleOCR#pp-ocr-20-series-model-listupdate-on-dec-15)
+```
+# 下载并解压ResNet50_vd模型
+wget "https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/ResNet50_vd_infer.tar" && tar xf ResNet50_vd_infer.tar
+```
+
+接下来，用安装的paddle_serving_client把下载的inference模型转换成易于server部署的模型格式。
+
+```
+# 转换ResNet50_vd模型
+python3 -m paddle_serving_client.convert --dirname ./inference/ \
+                                         --model_filename inference.pdmodel  \        \
+                                         --params_filename inference.pdiparams \       \
+                                         --serving_server ./ResNet50_vd_serving/ \
+                                         --serving_client ./ResNet50_vd_client/
+```
+检测模型转换完成后，会在当前文件夹多出`ResNet50_vd_serving` 和`ResNet50_vd_client`的文件夹，具备如下格式：
+```
+|- ResNet50_vd_client/
+  |- __model__  
+  |- __params__
+  |- serving_server_conf.prototxt  
+  |- serving_server_conf.stream.prototxt
+
+|- ResNet50_vd_client
+  |- serving_client_conf.prototxt  
+  |- serving_client_conf.stream.prototxt
+
+```
+
+<a name="部署"></a>
+## Paddle Serving pipeline部署
+
+1. 下载PaddleClas代码，若已下载可跳过此步骤
+    ```
+    git clone https://github.com/PaddlePaddle/PaddleClas
+
+    # 进入到工作目录
+    cd PaddleOCR/deploy/paddleserving/
+    ```
+    pdserver目录包含启动pipeline服务和发送预测请求的代码，包括：
+    ```
+    __init__.py
+    config.yml                 # 启动服务的配置文件
+    pipeline_http_client.py    # http方式发送pipeline预测请求的脚本
+    pipeline_rpc_client.py     # rpc方式发送pipeline预测请求的脚本
+    resnet50_web_service.py    # 启动pipeline服务端的脚本
+    ```
+
+2. 启动服务可运行如下命令：
+    ```
+    # 启动服务，运行日志保存在log.txt
+    python3 resnet50_web_service.py &>log.txt &
+    ```
+    成功启动服务后，log.txt中会打印类似如下日志
+    ![](./imgs/start_server.png)
+
+3. 发送服务请求：
+    ```
+    python3 pipeline_http_client.py
+    ```
+    成功运行后，模型预测的结果会打印在cmd窗口中，结果示例为：
+    ![](./imgs/results.png)
+
+    调整 config.yml 中的并发个数获得最大的QPS, 一般检测和识别的并发数为2：1
+    ```
+    op:
+        #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+        concurrency: 8
+        ...
+    有需要的话可以同时发送多个服务请求
+
+    预测性能数据会被自动写入 `PipelineServingLogs/pipeline.tracer` 文件中。
+
+<a name="FAQ"></a>
+## FAQ
+**Q1**： 发送请求后没有结果返回或者提示输出解码报错
+
+**A1**： 启动服务和发送请求时不要设置代理，可以在启动服务前和发送请求前关闭代理，关闭代理的命令是：
+```
+unset https_proxy
+unset http_proxy
+```

deploy/paddleserving/config.yml

+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+    #op资源类型, True, 为线程模型；False，为进程模型
+    is_thread_op: False
+op:
+    imagenet:
+        #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+        concurrency: 1
+
+        #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+        local_service_conf:
+
+            #uci模型路径
+            model_config: ResNet50_vd/ppcls_model/
+
+            #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+            device_type: 1
+
+            #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+            devices: "0" # "0,1"
+
+            #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+            client_type: local_predictor
+
+            #Fetch结果列表，以client_config中fetch_var的alias_name为准
+            fetch_list: ["prediction"] 

deploy/paddleserving/cpu_utilization.py

+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+

deploy/paddleserving/image_service_cpu.py

-# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sys
-import base64
-from paddle_serving_server.web_service import WebService
-import utils
-
-
-class ImageService(WebService):
-    def __init__(self, name):
-        super(ImageService, self).__init__(name=name)
-        self.operators = self.create_operators()
-
-    def create_operators(self):
-        size = 224
-        img_mean = [0.485, 0.456, 0.406]
-        img_std = [0.229, 0.224, 0.225]
-        img_scale = 1.0 / 255.0
-        decode_op = utils.DecodeImage()
-        resize_op = utils.ResizeImage(resize_short=256)
-        crop_op = utils.CropImage(size=(size, size))
-        normalize_op = utils.NormalizeImage(
-            scale=img_scale, mean=img_mean, std=img_std)
-        totensor_op = utils.ToTensor()
-        return [decode_op, resize_op, crop_op, normalize_op, totensor_op]
-
-    def _process_image(self, data, ops):
-        for op in ops:
-            data = op(data)
-        return data
-
-    def preprocess(self, feed={}, fetch=[]):
-        feed_batch = []
-        for ins in feed:
-            if "image" not in ins:
-                raise ("feed data error!")
-            sample = base64.b64decode(ins["image"])
-            img = self._process_image(sample, self.operators)
-            feed_batch.append({"image": img})
-        return feed_batch, fetch
-
-
-image_service = ImageService(name="image")
-image_service.load_model_config(sys.argv[1])
-image_service.prepare_server(
-    workdir=sys.argv[2], port=int(sys.argv[3]), device="cpu")
-image_service.run_server()
-image_service.run_flask()

deploy/paddleserving/image_service_gpu.py

-# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sys
-import base64
-from paddle_serving_server_gpu.web_service import WebService
-
-import utils
-
-
-class ImageService(WebService):
-    def __init__(self, name):
-        super(ImageService, self).__init__(name=name)
-        self.operators = self.create_operators()
-
-    def create_operators(self):
-        size = 224
-        img_mean = [0.485, 0.456, 0.406]
-        img_std = [0.229, 0.224, 0.225]
-        img_scale = 1.0 / 255.0
-        decode_op = utils.DecodeImage()
-        resize_op = utils.ResizeImage(resize_short=256)
-        crop_op = utils.CropImage(size=(size, size))
-        normalize_op = utils.NormalizeImage(
-            scale=img_scale, mean=img_mean, std=img_std)
-        totensor_op = utils.ToTensor()
-        return [decode_op, resize_op, crop_op, normalize_op, totensor_op]
-
-    def _process_image(self, data, ops):
-        for op in ops:
-            data = op(data)
-        return data
-
-    def preprocess(self, feed={}, fetch=[]):
-        feed_batch = []
-        for ins in feed:
-            if "image" not in ins:
-                raise ("feed data error!")
-            sample = base64.b64decode(ins["image"])
-            img = self._process_image(sample, self.operators)
-            feed_batch.append({"image": img})
-        return feed_batch, fetch
-
-
-image_service = ImageService(name="image")
-image_service.load_model_config(sys.argv[1])
-image_service.set_gpus("0")
-image_service.prepare_server(
-    workdir=sys.argv[2], port=int(sys.argv[3]), device="gpu")
-image_service.run_server()
-image_service.run_flask()

deploy/paddleserving/pipeline_http_client.py

+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+    return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+    url = "http://127.0.0.1:18080/imagenet/prediction"
+    with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+        image_data1 = file.read()
+    image = cv2_to_base64(image_data1)
+    data = {"key": ["image"], "value": [image]}
+    for i in range(100):
+        r = requests.post(url=url, data=json.dumps(data))
+        print(r.json())

deploy/paddleserving/image_http_client.py → deploy/paddleserving/pipeline_rpc_client.py

@@ -11,36 +11,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
+try:
+    from paddle_serving_server_gpu.pipeline import PipelineClient
+except ImportError:
+    from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
 import requests
-import base64
 import json
-import sys
-import numpy as np
-
-py_version = sys.version_info[0]
-
-
-def predict(image_path, server):
+import cv2
+import base64
+import os
 
-    with open(image_path, "rb") as f:
-        image = base64.b64encode(f.read()).decode("utf-8")
-    req = json.dumps({"feed": [{"image": image}], "fetch": ["prediction"]})
-    r = requests.post(
-        server, data=req, headers={"Content-Type": "application/json"})
-    try:
-        pred = r.json()["result"]["prediction"][0]
-        cls_id = np.argmax(pred)
-        score = pred[cls_id]
-        pred = {"cls_id": cls_id, "score": score}
-        return pred
-    except ValueError:
-        print(r.text)
-    return r
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
 
+def cv2_to_base64(image):
+    return base64.b64encode(image).decode('utf8')
 
 if __name__ == "__main__":
-    server = "http://127.0.0.1:{}/image/prediction".format(sys.argv[1])
-    image_file = sys.argv[2]
-    res = predict(image_file, server)
-    print("res:", res)
+    with open("daisy.jpg", 'rb') as file:
+        image_data = file.read()
+    image = cv2_to_base64(image_data)
+
+    for i in range(1):
+        ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+        print(ret)

deploy/paddleserving/resnet50_web_service.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+try:
+    from paddle_serving_server_gpu.web_service import WebService, Op
+except ImportError:
+    from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+class ImagenetOp(Op):
+    def init_op(self):
+        self.seq = Sequential([
+            Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+            Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+                                True)
+        ])
+        self.label_dict = {}
+        label_idx = 0
+        with open("imagenet.label") as fin:
+            for line in fin:
+                self.label_dict[label_idx] = line.strip()
+                label_idx += 1
+
+    def preprocess(self, input_dicts, data_id, log_id):
+        (_, input_dict), = input_dicts.items()
+        batch_size = len(input_dict.keys())
+        imgs = []
+        for key in input_dict.keys():
+            data = base64.b64decode(input_dict[key].encode('utf8'))
+            data = np.fromstring(data, np.uint8)
+            im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+            img = self.seq(im)
+            imgs.append(img[np.newaxis, :].copy())
+        input_imgs = np.concatenate(imgs, axis=0)
+        return {"image": input_imgs}, False, None, ""
+
+    def postprocess(self, input_dicts, fetch_dict, log_id):
+        score_list = fetch_dict["prediction"]
+        result = {"label": [], "prob": []}
+        for score in score_list:
+            score = score.tolist()
+            max_score = max(score)
+            result["label"].append(self.label_dict[score.index(max_score)]
+                                   .strip().replace(",", ""))
+            result["prob"].append(max_score)
+        result["label"] = str(result["label"])
+        result["prob"] = str(result["prob"])
+        return result, None, ""
+
+
+class ImageService(WebService):
+    def get_pipeline_response(self, read_op):
+        image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+        return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()

deploy/paddleserving/utils.py

-# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import cv2
-import numpy as np
-
-
-class DecodeImage(object):
-    def __init__(self, to_rgb=True):
-        self.to_rgb = to_rgb
-
-    def __call__(self, img):
-        data = np.frombuffer(img, dtype='uint8')
-        img = cv2.imdecode(data, 1)
-        if self.to_rgb:
-            assert img.shape[2] == 3, 'invalid shape of image[%s]' % (
-                img.shape)
-            img = img[:, :, ::-1]
-
-        return img
-
-
-class ResizeImage(object):
-    def __init__(self, resize_short=None):
-        self.resize_short = resize_short
-
-    def __call__(self, img):
-        img_h, img_w = img.shape[:2]
-        percent = float(self.resize_short) / min(img_w, img_h)
-        w = int(round(img_w * percent))
-        h = int(round(img_h * percent))
-        return cv2.resize(img, (w, h))
-
-
-class CropImage(object):
-    def __init__(self, size):
-        if type(size) is int:
-            self.size = (size, size)
-        else:
-            self.size = size
-
-    def __call__(self, img):
-        w, h = self.size
-        img_h, img_w = img.shape[:2]
-        w_start = (img_w - w) // 2
-        h_start = (img_h - h) // 2
-
-        w_end = w_start + w
-        h_end = h_start + h
-        return img[h_start:h_end, w_start:w_end, :]
-
-
-class NormalizeImage(object):
-    def __init__(self, scale=None, mean=None, std=None):
-        self.scale = np.float32(scale if scale is not None else 1.0 / 255.0)
-        mean = mean if mean is not None else [0.485, 0.456, 0.406]
-        std = std if std is not None else [0.229, 0.224, 0.225]
-
-        shape = (1, 1, 3)
-        self.mean = np.array(mean).reshape(shape).astype('float32')
-        self.std = np.array(std).reshape(shape).astype('float32')
-
-    def __call__(self, img):
-        return (img.astype('float32') * self.scale - self.mean) / self.std
-
-
-class ToTensor(object):
-    def __init__(self):
-        pass
-
-    def __call__(self, img):
-        img = img.transpose((2, 0, 1))
-        return img