Merge branch 'PaddlePaddle:develop' into develop

README.md

@@ -62,7 +62,7 @@ This chapter guides you through the installation and deployment steps. It is str
 - [Build Paddle Serving from Source with Docker](doc/Compile_EN.md)
 - [Deploy Paddle Serving on Kubernetes(Chinese)](doc/Run_On_Kubernetes_CN.md)
 - [Deploy Paddle Serving with Security gateway(Chinese)](doc/Serving_Auth_Docker_CN.md)
-- Deploy on more hardwares[[百度昆仑](doc/Run_On_XPU_CN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
+- Deploy on more hardwares[[ARM CPU、百度昆仑](doc/Run_On_XPU_EN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
 - [Docker Images](doc/Docker_Images_EN.md)
 - [Download Wheel packages](doc/Latest_Packages_EN.md)
 

README_CN.md

@@ -34,7 +34,7 @@ Paddle Serving依托深度学习框架PaddlePaddle旨在帮助深度学习开发
 - 集成Intel MKLDNN、Nvidia TensorRT加速库，以及低精度和量化推理
 - 提供一套模型安全部署解决方案，包括加密模型部署、鉴权校验、HTTPs安全网关，并在实际项目中应用
 - 支持云端部署，提供百度云智能云kubernetes集群部署Paddle Serving案例
-- 提供丰富的经典预模型部署示例，如PaddleOCR、PaddleClas、PaddleDetection、PaddleSeg、PaddleNLP、PaddleRec等套件，共计40+个预训练精品模型
+- 提供丰富的经典模型部署示例，如PaddleOCR、PaddleClas、PaddleDetection、PaddleSeg、PaddleNLP、PaddleRec等套件，共计40+个预训练精品模型
 - 支持大规模稀疏参数索引模型分布式部署，具有多表、多分片、多副本、本地高频cache等特性、可单机或云端部署
 - 支持服务监控，提供基于普罗米修斯的性能数据统计及端口访问
 
@@ -58,7 +58,7 @@ Paddle Serving依托深度学习框架PaddlePaddle旨在帮助深度学习开发
 - [源码编译安装Paddle Serving](doc/Compile_CN.md)
 - [在Kuberntes集群上部署Paddle Serving](doc/Run_On_Kubernetes_CN.md)
 - [部署Paddle Serving安全网关](doc/Serving_Auth_Docker_CN.md)
-- 异构硬件部署[[百度昆仑](doc/Run_On_XPU_CN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
+- 异构硬件部署[[ARM CPU、百度昆仑](doc/Run_On_XPU_CN.md)、[华为昇腾](doc/Run_On_NPU_CN.md)、[海光DCU](doc/Run_On_DCU_CN.md)、[Jetson](doc/Run_On_JETSON_CN.md)]
 - [Docker镜像](doc/Docker_Images_CN.md)
 - [下载Wheel包](doc/Latest_Packages_CN.md)
 

doc/Install_EN.md

@@ -62,6 +62,8 @@ pip3 install -r python/requirements.txt
 
 Install the service whl package. There are three types of client, app and server. The server is divided into CPU and GPU. Choose one installation according to the environment. 
 - GPU with CUDA10.2 + Cudnn7 + TensorRT6(Recommended)
+- post101 = CUDA10.1 + TensorRT6
+- post112 = CUDA11.2 + TensorRT8
 ```shell
 pip3 install paddle-serving-client==0.8.2 -i https://pypi.tuna.tsinghua.edu.cn/simple
 pip3 install paddle-serving-app==0.8.2 -i https://pypi.tuna.tsinghua.edu.cn/simple

doc/Serving_Configure_EN.md

-# Serving Configuration
-
+# Serving Configuration & Startup Params
 ([简体中文](./Serving_Configure_CN.md)|English)
 
 ## Overview

examples/Pipeline/PaddleVideo/PPTSN_K400/Kinetics-400_label_list.txt

+0 abseiling
+1 air_drumming
+2 answering_questions
+3 applauding
+4 applying_cream
+5 archery
+6 arm_wrestling
+7 arranging_flowers
+8 assembling_computer
+9 auctioning
+10 baby_waking_up
+11 baking_cookies
+12 balloon_blowing
+13 bandaging
+14 barbequing
+15 bartending
+16 beatboxing
+17 bee_keeping
+18 belly_dancing
+19 bench_pressing
+20 bending_back
+21 bending_metal
+22 biking_through_snow
+23 blasting_sand
+24 blowing_glass
+25 blowing_leaves
+26 blowing_nose
+27 blowing_out_candles
+28 bobsledding
+29 bookbinding
+30 bouncing_on_trampoline
+31 bowling
+32 braiding_hair
+33 breading_or_breadcrumbing
+34 breakdancing
+35 brush_painting
+36 brushing_hair
+37 brushing_teeth
+38 building_cabinet
+39 building_shed
+40 bungee_jumping
+41 busking
+42 canoeing_or_kayaking
+43 capoeira
+44 carrying_baby
+45 cartwheeling
+46 carving_pumpkin
+47 catching_fish
+48 catching_or_throwing_baseball
+49 catching_or_throwing_frisbee
+50 catching_or_throwing_softball
+51 celebrating
+52 changing_oil
+53 changing_wheel
+54 checking_tires
+55 cheerleading
+56 chopping_wood
+57 clapping
+58 clay_pottery_making
+59 clean_and_jerk
+60 cleaning_floor
+61 cleaning_gutters
+62 cleaning_pool
+63 cleaning_shoes
+64 cleaning_toilet
+65 cleaning_windows
+66 climbing_a_rope
+67 climbing_ladder
+68 climbing_tree
+69 contact_juggling
+70 cooking_chicken
+71 cooking_egg
+72 cooking_on_campfire
+73 cooking_sausages
+74 counting_money
+75 country_line_dancing
+76 cracking_neck
+77 crawling_baby
+78 crossing_river
+79 crying
+80 curling_hair
+81 cutting_nails
+82 cutting_pineapple
+83 cutting_watermelon
+84 dancing_ballet
+85 dancing_charleston
+86 dancing_gangnam_style
+87 dancing_macarena
+88 deadlifting
+89 decorating_the_christmas_tree
+90 digging
+91 dining
+92 disc_golfing
+93 diving_cliff
+94 dodgeball
+95 doing_aerobics
+96 doing_laundry
+97 doing_nails
+98 drawing
+99 dribbling_basketball
+100 drinking
+101 drinking_beer
+102 drinking_shots
+103 driving_car
+104 driving_tractor
+105 drop_kicking
+106 drumming_fingers
+107 dunking_basketball
+108 dying_hair
+109 eating_burger
+110 eating_cake
+111 eating_carrots
+112 eating_chips
+113 eating_doughnuts
+114 eating_hotdog
+115 eating_ice_cream
+116 eating_spaghetti
+117 eating_watermelon
+118 egg_hunting
+119 exercising_arm
+120 exercising_with_an_exercise_ball
+121 extinguishing_fire
+122 faceplanting
+123 feeding_birds
+124 feeding_fish
+125 feeding_goats
+126 filling_eyebrows
+127 finger_snapping
+128 fixing_hair
+129 flipping_pancake
+130 flying_kite
+131 folding_clothes
+132 folding_napkins
+133 folding_paper
+134 front_raises
+135 frying_vegetables
+136 garbage_collecting
+137 gargling
+138 getting_a_haircut
+139 getting_a_tattoo
+140 giving_or_receiving_award
+141 golf_chipping
+142 golf_driving
+143 golf_putting
+144 grinding_meat
+145 grooming_dog
+146 grooming_horse
+147 gymnastics_tumbling
+148 hammer_throw
+149 headbanging
+150 headbutting
+151 high_jump
+152 high_kick
+153 hitting_baseball
+154 hockey_stop
+155 holding_snake
+156 hopscotch
+157 hoverboarding
+158 hugging
+159 hula_hooping
+160 hurdling
+161 hurling_(sport)
+162 ice_climbing
+163 ice_fishing
+164 ice_skating
+165 ironing
+166 javelin_throw
+167 jetskiing
+168 jogging
+169 juggling_balls
+170 juggling_fire
+171 juggling_soccer_ball
+172 jumping_into_pool
+173 jumpstyle_dancing
+174 kicking_field_goal
+175 kicking_soccer_ball
+176 kissing
+177 kitesurfing
+178 knitting
+179 krumping
+180 laughing
+181 laying_bricks
+182 long_jump
+183 lunge
+184 making_a_cake
+185 making_a_sandwich
+186 making_bed
+187 making_jewelry
+188 making_pizza
+189 making_snowman
+190 making_sushi
+191 making_tea
+192 marching
+193 massaging_back
+194 massaging_feet
+195 massaging_legs
+196 massaging_person's_head
+197 milking_cow
+198 mopping_floor
+199 motorcycling
+200 moving_furniture
+201 mowing_lawn
+202 news_anchoring
+203 opening_bottle
+204 opening_present
+205 paragliding
+206 parasailing
+207 parkour
+208 passing_American_football_(in_game)
+209 passing_American_football_(not_in_game)
+210 peeling_apples
+211 peeling_potatoes
+212 petting_animal_(not_cat)
+213 petting_cat
+214 picking_fruit
+215 planting_trees
+216 plastering
+217 playing_accordion
+218 playing_badminton
+219 playing_bagpipes
+220 playing_basketball
+221 playing_bass_guitar
+222 playing_cards
+223 playing_cello
+224 playing_chess
+225 playing_clarinet
+226 playing_controller
+227 playing_cricket
+228 playing_cymbals
+229 playing_didgeridoo
+230 playing_drums
+231 playing_flute
+232 playing_guitar
+233 playing_harmonica
+234 playing_harp
+235 playing_ice_hockey
+236 playing_keyboard
+237 playing_kickball
+238 playing_monopoly
+239 playing_organ
+240 playing_paintball
+241 playing_piano
+242 playing_poker
+243 playing_recorder
+244 playing_saxophone
+245 playing_squash_or_racquetball
+246 playing_tennis
+247 playing_trombone
+248 playing_trumpet
+249 playing_ukulele
+250 playing_violin
+251 playing_volleyball
+252 playing_xylophone
+253 pole_vault
+254 presenting_weather_forecast
+255 pull_ups
+256 pumping_fist
+257 pumping_gas
+258 punching_bag
+259 punching_person_(boxing)
+260 push_up
+261 pushing_car
+262 pushing_cart
+263 pushing_wheelchair
+264 reading_book
+265 reading_newspaper
+266 recording_music
+267 riding_a_bike
+268 riding_camel
+269 riding_elephant
+270 riding_mechanical_bull
+271 riding_mountain_bike
+272 riding_mule
+273 riding_or_walking_with_horse
+274 riding_scooter
+275 riding_unicycle
+276 ripping_paper
+277 robot_dancing
+278 rock_climbing
+279 rock_scissors_paper
+280 roller_skating
+281 running_on_treadmill
+282 sailing
+283 salsa_dancing
+284 sanding_floor
+285 scrambling_eggs
+286 scuba_diving
+287 setting_table
+288 shaking_hands
+289 shaking_head
+290 sharpening_knives
+291 sharpening_pencil
+292 shaving_head
+293 shaving_legs
+294 shearing_sheep
+295 shining_shoes
+296 shooting_basketball
+297 shooting_goal_(soccer)
+298 shot_put
+299 shoveling_snow
+300 shredding_paper
+301 shuffling_cards
+302 side_kick
+303 sign_language_interpreting
+304 singing
+305 situp
+306 skateboarding
+307 ski_jumping
+308 skiing_(not_slalom_or_crosscountry)
+309 skiing_crosscountry
+310 skiing_slalom
+311 skipping_rope
+312 skydiving
+313 slacklining
+314 slapping
+315 sled_dog_racing
+316 smoking
+317 smoking_hookah
+318 snatch_weight_lifting
+319 sneezing
+320 sniffing
+321 snorkeling
+322 snowboarding
+323 snowkiting
+324 snowmobiling
+325 somersaulting
+326 spinning_poi
+327 spray_painting
+328 spraying
+329 springboard_diving
+330 squat
+331 sticking_tongue_out
+332 stomping_grapes
+333 stretching_arm
+334 stretching_leg
+335 strumming_guitar
+336 surfing_crowd
+337 surfing_water
+338 sweeping_floor
+339 swimming_backstroke
+340 swimming_breast_stroke
+341 swimming_butterfly_stroke
+342 swing_dancing
+343 swinging_legs
+344 swinging_on_something
+345 sword_fighting
+346 tai_chi
+347 taking_a_shower
+348 tango_dancing
+349 tap_dancing
+350 tapping_guitar
+351 tapping_pen
+352 tasting_beer
+353 tasting_food
+354 testifying
+355 texting
+356 throwing_axe
+357 throwing_ball
+358 throwing_discus
+359 tickling
+360 tobogganing
+361 tossing_coin
+362 tossing_salad
+363 training_dog
+364 trapezing
+365 trimming_or_shaving_beard
+366 trimming_trees
+367 triple_jump
+368 tying_bow_tie
+369 tying_knot_(not_on_a_tie)
+370 tying_tie
+371 unboxing
+372 unloading_truck
+373 using_computer
+374 using_remote_controller_(not_gaming)
+375 using_segway
+376 vault
+377 waiting_in_line
+378 walking_the_dog
+379 washing_dishes
+380 washing_feet
+381 washing_hair
+382 washing_hands
+383 water_skiing
+384 water_sliding
+385 watering_plants
+386 waxing_back
+387 waxing_chest
+388 waxing_eyebrows
+389 waxing_legs
+390 weaving_basket
+391 welding
+392 whistling
+393 windsurfing
+394 wrapping_present
+395 wrestling
+396 writing
+397 yawning
+398 yoga
+399 zumba

examples/Pipeline/PaddleVideo/PPTSN_K400/config.yml

+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 18090
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 9999
+
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
+
+#build_dag_each_worker, False，框架在进程内创建一条DAG；True，框架会每个进程内创建多个独立的DAG
+build_dag_each_worker: false
+
+dag:
+    #op资源类型, True, 为线程模型；False，为进程模型
+    is_thread_op: False
+
+    #重试次数
+    retry: 1
+
+    #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+    use_profile: false
+    tracer:
+        interval_s: 10
+
+op:
+    ppTSN:
+        #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+        concurrency: 1
+
+        #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+        local_service_conf:
+            #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+            client_type: local_predictor
+
+            #det模型路径
+            model_config: serving_server
+
+            #Fetch结果列表，以client_config中fetch_var的alias_name为准
+            fetch_list: ["linear_1.tmp_1"]
+            
+            # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+            device_type: 1
+
+            #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+            devices: "0"
+
+            #use_mkldnn
+            #use_mkldnn: True
+
+            #thread_num
+            thread_num: 2
+
+            #ir_optim
+            ir_optim: True

examples/Pipeline/PaddleVideo/PPTSN_K400/get_model.sh

+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/PaddleVideo/PPTSN_K400.tar
+tar -xvf PPTSN_K400.tar
+

examples/Pipeline/PaddleVideo/PPTSN_K400/pipeline_http_client.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# from paddle_serving_server.pipeline import PipelineClient
+import requests
+import json
+
+url = "http://127.0.0.1:9999/ppTSN/prediction"
+video_url = "https://paddle-serving.bj.bcebos.com/huangjianhui04/example.avi"
+for i in range(4):
+    data = {"key": ["filename"], "value": [video_url]}
+    r = requests.post(url=url, data=json.dumps(data))
+    print(r.json())

examples/Pipeline/PaddleVideo/PPTSN_K400/web_service.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import cv2
+import base64
+from paddle_serving_app.reader import VideoDecoder, Sampler, Scale, TenCrop, Image2Array, Normalization
+import urllib.request
+_LOGGER = logging.getLogger()
+
+
+class ppTSNOp(Op):
+    def init_op(self,
+                num_seg=25,
+                seg_len=1,
+                short_size=256,
+                target_size=224,
+                top_k=1):
+        self.top_k = top_k
+        img_mean = [0.485, 0.456, 0.406]
+        img_std = [0.229, 0.224, 0.225]
+        self.ops = [
+            VideoDecoder(),
+            Sampler(num_seg, seg_len, valid_mode=True, select_left=True),
+            Scale(short_size, fixed_ratio=True, do_round=True, backend='cv2'),
+            TenCrop(target_size),
+            Image2Array(),
+            Normalization(img_mean, img_std)                                
+        ]
+        self.label_dict = {}
+        with open("Kinetics-400_label_list.txt") as fin:
+            for line in fin:
+                label_list = line.strip().split(" ")
+                index = int(label_list[0])
+                label = label_list[1]
+                self.label_dict[index] = label
+
+    def preprocess(self, input_dicts, data_id, log_id):
+        (_, input_dict), = input_dicts.items()
+        self.input_file = []
+        for key in input_dict.keys():
+            try:
+                filename = urllib.request.urlretrieve(input_dict[key], key)
+                self.input_file.append(filename[0])
+                print("download video success")
+            except:
+                print("download video failed")
+        batched_inputs = []
+        for filename in self.input_file:
+            results = {'filename': filename}
+            for op in self.ops:
+                results = op(results)
+            res = np.expand_dims(results['imgs'], axis=0).copy()
+            batched_inputs.append(res)
+        batched_inputs = [
+            np.concatenate([item[i] for item in batched_inputs])
+            for i in range(len(batched_inputs[0]))
+        ]
+        return {"data_batch_0": batched_inputs[0]}, False, None, ""
+
+    def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+        output = fetch_dict["linear_1.tmp_1"]
+        if not isinstance(self.input_file, list):
+            self.input_file = [
+                self.input_file,
+            ]
+        N = len(self.input_file)
+        if output.shape[0] != N:
+            output = output.reshape([N] + [output.shape[0] // N] +
+                                    list(output.shape[1:]))  # [N, T, C]
+            output = output.mean(axis=1)  # [N, C]
+        import paddle
+        import paddle.nn.functional as F
+        output = F.softmax(paddle.to_tensor(output), axis=-1).numpy()
+        for i in range(N):
+            classes = np.argpartition(output[i], -self.top_k)[-self.top_k:]
+            classes = classes[np.argsort(-output[i, classes])]
+            labels = [self.label_dict[label] for label in classes.tolist()]
+            scores = output[i, classes]
+        res = {"res" :"class: {} score: {}".format(labels, scores)}
+        return res, None, ""
+
+class ppTSNService(WebService):
+    def get_pipeline_response(self, read_op):
+        ppTSN_op = ppTSNOp(name="ppTSN", input_ops=[read_op])
+        return ppTSN_op
+
+pptsn_service = ppTSNService(name="ppTSN")
+pptsn_service.prepare_pipeline_config("config.yml")
+pptsn_service.run_service()

python/paddle_serving_app/reader/__init__.py

@@ -21,3 +21,4 @@ from .lac_reader import LACReader
 from .senta_reader import SentaReader
 #from .imdb_reader import IMDBDataset
 from .ocr_reader import OCRReader
+from .pptsn_reader import VideoDecoder, Sampler, Scale, TenCrop, Image2Array, Normalization

python/requirements.txt

@@ -23,3 +23,6 @@ opencv-python; platform_machine == "aarch64"
 pytest==7.0.1
 prometheus-client==0.12.0
 pillow==8.4.0
+av==8.0.3
+decord==0.4.2
+SimpleITK