: # op 名，与程序中定义的相对应
- concurrency: 1 # op 并发数，默认为 1
- timeout: -1 # 预测超时时间，单位为毫秒。默认为 -1 即不超时
- retry: 1 # 超时重发次数。默认为 1 即不重试
- batch_size: 1 # auto-batching 中的 batch_size，若设置该字段则 Op 会将多个请求输出合并为一个 batch
- auto_batching_timeout: -1 # auto-batching 超时时间，单位为毫秒。默认为 -1 即不超时
- local_service_conf:
- model_config: # 对应模型文件的路径，无默认值（None）。若不配置该项则不会加载模型文件。
- workdir: "" # 对应模型的工作目录
- thread_num: 2 # 对应模型用几个线程启动
- devices: "" # 模型启动在哪个设备上，可以指定 gpu 卡号（如 "0,1,2"），默认为 cpu
- mem_optim: true # mem 优化选项，默认为 true
- ir_optim: false # ir 优化选项，默认为 false
-```
-
-其中，Op 的所有字段均可以在程序中创建 Op 时定义（会覆盖 yaml 的字段）。
diff --git a/doc/deprecated/CLUSTERING.md b/doc/deprecated/CLUSTERING.md
deleted file mode 100644
index b9da2e56165f17ee07116c46dbbdfefd806cb56f..0000000000000000000000000000000000000000
--- a/doc/deprecated/CLUSTERING.md
+++ /dev/null
@@ -1,142 +0,0 @@
-# 搭建预测服务集群
-
-从[客户端配置](../CLIENT_CONFIGURE.md)中我们已经知道，通过在客户端SDK的配置文件predictors.prototxt适当配置，可以搭建多副本和多Variant的预测集群。以下以图像分类任务为例，在单机上模拟搭建单Variant的多副本、和多Variant的预测集群
-
-## 1. 单Variant多副本的预测集群
-
-### 1.1 在本机创建一个serving副本
-
-首先复制一个sering目录
-
-```shell
-$ cd /path/to/paddle-serving/build/output/demo
-$ cp -r serving/ serving_new/
-$ cd serving_new/
-
-```
-
-在serving_new目录中，在conf/gflags.conf中增加如下一行，修改其启动端口为8011，这是为了让该副本监听不同端口
-
-```shell
---port=8011
-```
-
-然后启动新副本
-
-```shell
-$ bin/serving&
-```
-
-### 1.2 修改client端配置，将新副本地址加入ip列表：
-
-```shell
-$ cd /path/to/paddle-serving/build/output/demo/client/image_classification
-```
-
-修改conf/predictors.prototxt ImageClassifyService部分如下所示
-
-```JSON
-predictors {
- name: "ximage"
- service_name: "baidu.paddle_serving.predictor.image_classification.ImageClassifyService"
- endpoint_router: "WeightedRandomRender"
- weighted_random_render_conf {
- variant_weight_list: "50"
- }
- variants {
- tag: "var1"
- naming_conf {
- cluster: "list://127.0.0.1:8010, 127.0.0.1:8011" # 在这里增加一个新的副本地址
- }
- }
-}
-```
-
-重启client端
-
-```shell
-$ bin/ximage&
-```
-
-查看2个serving副本目录下是否均有收到请求：
-
-```shell
-$ cd /path/to/paddle-serving/build/output/demo/serving
-$ tail -f log/serving.INFO
-
-$ cd /path/to/paddle-serving/build/output/demo/serving_new
-$ tail -f log/serving.INFO
-```
-
-## 2. 多Variant
-
-### 2.1 本机创建新的serving副本
-
-步骤同1.1节，略过
-
-### 2.2 修改client配置，增加一个Variant
-
-```shell
-$ cd /path/to/paddle-serving/build/output/demo/client/image_classification
-```
-
-修改conf/predictors.prototxt ImageClassifyService部分如下所示
-
-```JSON
-predictors {
- name: "ximage"
- service_name: "baidu.paddle_serving.predictor.image_classification.ImageClassifyService"
- endpoint_router: "WeightedRandomRender"
- weighted_random_render_conf {
- variant_weight_list: "50 | 50" # 一共2个variant，代表模型的2个版本。这里的权重代表调度的流量比例关系
- }
- variants {
- tag: "var1"
- naming_conf {
- cluster: "list://127.0.0.1:8010"
- }
- }
- variants { # 增加一个variant
- tag: "var2"
- naming_conf {
- cluster: "list://127.0.0.1:8011"
- }
- }
-}
-```
-
-重启client端
-
-```shell
-$ bin/ximage&
-```
-
-查看2个serving副本目录下是否均有收到请求：
-
-```shell
-$ cd /path/to/paddle-serving/build/output/demo/serving
-$ tail -f log/serving.INFO
-
-$ cd /path/to/paddle-serving/build/output/demo/serving_new
-$ tail -f log/serving.INFO
-```
-
-查看client端是否有收到来自Variant1和Variant2的响应
-
-```shell
-$ cd /path/to/paddle-serving/build/output/demo/client/image_classification
-$ tail -f log/ximage.INFO
-
-```
-
-以下是正常的输出
-
-```
-I0307 17:54:22.862087 24719 ximage.cpp:172] Debug string:
-I0307 17:54:22.862650 24719 ximage.cpp:110] sample-0's classify result: n02112018,博美犬, prop: 0.522815
-I0307 17:54:22.862666 24719 ximage.cpp:114] Succ call predictor[ximage], the tag is: var1, elapse_ms: 333
-
-I0307 17:54:23.194780 24719 ximage.cpp:172] Debug string:
-I0307 17:54:23.195322 24719 ximage.cpp:110] sample-0's classify result: n02112018,博美犬, prop: 0.522815
-I0307 17:54:23.195334 24719 ximage.cpp:114] Succ call predictor[ximage], the tag is: var2, elapse_ms: 332
-```
diff --git a/doc/deprecated/CTR_PREDICTION.md b/doc/deprecated/CTR_PREDICTION.md
deleted file mode 100644
index acfb6c08e61a8a393d3978dc4ca29b9cda631d20..0000000000000000000000000000000000000000
--- a/doc/deprecated/CTR_PREDICTION.md
+++ /dev/null
@@ -1,334 +0,0 @@
-# CTR预估模型
-
-## 1. 背景
-
-在搜索、推荐、在线广告等业务场景中，embedding参数的规模常常非常庞大，达到数百GB甚至T级别；训练如此规模的模型需要用到多机分布式训练能力，将参数分片更新和保存；另一方面，训练好的模型，要应用于在线业务，也难以单机加载。Paddle Serving提供大规模稀疏参数读写服务，用户可以方便地将超大规模的稀疏参数以kv形式托管到参数服务，在线预测只需将所需要的参数子集从参数服务读取回来，再执行后续的预测流程。
-
-我们以CTR预估模型为例，演示Paddle Serving中如何使用大规模稀疏参数服务。关于模型细节请参考[原始模型](https://github.com/PaddlePaddle/models/tree/v1.5/PaddleRec/ctr)
-
-根据[对数据集的描述](https://www.kaggle.com/c/criteo-display-ad-challenge/data)，该模型原始输入为13维integer features和26维categorical features。在我们的模型中，13维integer feature作为dense feature整体feed到一个data layer，而26维categorical features各自作为一个feature分别feed到一个data layer。除此之外，为计算auc指标，还将label作为一个feature输入。
-
-若按缺省训练参数，本模型的embedding dim为100w，size为10，也就是参数矩阵为1000000 x 10的float型矩阵，实际占用内存共1000000 x 10 x sizeof(float) = 39MB；**实际场景中，embedding参数要大的多；因此该demo仅为演示使用**。
-
-
-## 2. 模型裁剪
-
-在写本文档时([v1.5](https://github.com/PaddlePaddle/models/tree/v1.5))，训练脚本用PaddlePaddle py_reader加速样例读取速度，program中带有py_reader相关OP，且训练过程中只保存了模型参数，没有保存program，保存的参数没法直接用预测库加载；另外原始网络中最终输出的tensor是auc和batch_auc，而实际模型用于预测时只需要每个样例的predict，需要改掉模型的输出tensor为predict。再有，为了演示稀疏参数服务的使用，我们要有意将embedding layer包含的lookup_table OP从预测program中拿掉，以embedding layer的output variable作为网络的输入，然后再添加对应的feed OP，使得我们能够在预测时从稀疏参数服务获取到embedding向量后，将数据直接feed到各个embedding的output variable。
-
-基于以上几方面考虑，我们需要对原始program进行裁剪。大致过程为：
-
-1) 去掉py_reader相关代码，改为用fluid自带的reader和DataFeed
-2) 修改原始网络配置，将predict变量作为fetch target
-3) 修改原始网络配置，将26个稀疏参数的embedding layer的output作为feed target，以与后续稀疏参数服务配合使用
-4) 修改后的网络，本地train 1个batch后，调用`fluid.io.save_inference_model()`，获得裁剪后的模型program
-5) 裁剪后的program，用python再次处理，去掉embedding layer的lookup_table OP。这是因为，当前Paddle Fluid在第4步`save_inference_model()`时没有裁剪干净，还保留了embedding的lookup_table OP；如果这些OP不去除掉，那么embedding的output variable就会有2个输入OP：一个是feed OP（我们要添加的），一个是lookup_table；而lookup_table又没有输入，它的输出会与feed OP的输出互相覆盖，导致错乱。另外网络中还保留了SparseFeatFactors这个variable(全局共享的embedding矩阵对应的变量)，这个variable也要去掉，否则网络加载时还会尝试从磁盘读取embedding参数，就失去了我们这个demo的意义。
-6) 第4步拿到的program，与分布式训练保存的模型参数（除embedding之外）保存到一起，形成完整的预测模型
-
-第1) - 第5)步裁剪完毕后的模型网络配置如下：
-
-
-
-
-整个裁剪过程具体说明如下：
-
-### 2.1 网络配置中去除py_reader
-
-Inference program调用ctr_dnn_model()函数时添加`user_py_reader=False`参数。这会在ctr_dnn_model定义中将py_reader相关的代码去掉
-
-修改前：
-```python
-def train():
- args = parse_args()
-
- if not os.path.isdir(args.model_output_dir):
- os.mkdir(args.model_output_dir)
-
- loss, auc_var, batch_auc_var, py_reader, _ = ctr_dnn_model(args.embedding_size, args.sparse_feature_dim)
- ...
-```
-
-修改后：
-```python
-def train():
- args = parse_args()
-
- if not os.path.isdir(args.model_output_dir):
- os.mkdir(args.model_output_dir)
-
- loss, auc_var, batch_auc_var, py_reader, _ = ctr_dnn_model(args.embedding_size, args.sparse_feature_dim, use_py_reader=False)
- ...
-```
-
-
-### 2.2 网络配置中修改feed targets和fetch targets
-
-如第2节开头所述，为了使program适合于演示稀疏参数的使用，我们要裁剪program，将`ctr_dnn_model`中feed variable list和fetch variable分别改掉：
-
-1) Inference program中26维稀疏特征的输入改为每个特征的embedding layer的output variable
-2) fetch targets中返回的是predict，取代auc_var和batch_auc_var
-
-截至写本文时，原始的网络配置 (network_conf.py中)`ctr_dnn_model`定义如下：
-
-```python
-def ctr_dnn_model(embedding_size, sparse_feature_dim, use_py_reader=True):
-
- def embedding_layer(input):
- emb = fluid.layers.embedding(
- input=input,
- is_sparse=True,
- # you need to patch https://github.com/PaddlePaddle/Paddle/pull/14190
- # if you want to set is_distributed to True
- is_distributed=False,
- size=[sparse_feature_dim, embedding_size],
- param_attr=fluid.ParamAttr(name="SparseFeatFactors",
- initializer=fluid.initializer.Uniform()))
- return fluid.layers.sequence_pool(input=emb, pool_type='average') # 需修改1
-
- dense_input = fluid.layers.data(
- name="dense_input", shape=[dense_feature_dim], dtype='float32')
-
- sparse_input_ids = [
- fluid.layers.data(name="C" + str(i), shape=[1], lod_level=1, dtype='int64')
- for i in range(1, 27)]
-
- label = fluid.layers.data(name='label', shape=[1], dtype='int64')
-
- words = [dense_input] + sparse_input_ids + [label]
-
- py_reader = None
- if use_py_reader:
- py_reader = fluid.layers.create_py_reader_by_data(capacity=64,
- feed_list=words,
- name='py_reader',
- use_double_buffer=True)
- words = fluid.layers.read_file(py_reader)
-
- sparse_embed_seq = list(map(embedding_layer, words[1:-1])) # 需修改2
- concated = fluid.layers.concat(sparse_embed_seq + words[0:1], axis=1)
-
- fc1 = fluid.layers.fc(input=concated, size=400, act='relu',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(concated.shape[1]))))
- fc2 = fluid.layers.fc(input=fc1, size=400, act='relu',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(fc1.shape[1]))))
- fc3 = fluid.layers.fc(input=fc2, size=400, act='relu',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(fc2.shape[1]))))
- predict = fluid.layers.fc(input=fc3, size=2, act='softmax',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(fc3.shape[1]))))
-
- cost = fluid.layers.cross_entropy(input=predict, label=words[-1])
- avg_cost = fluid.layers.reduce_sum(cost)
- accuracy = fluid.layers.accuracy(input=predict, label=words[-1])
- auc_var, batch_auc_var, auc_states = \
- fluid.layers.auc(input=predict, label=words[-1], num_thresholds=2 ** 12, slide_steps=20)
-
- return avg_cost, auc_var, batch_auc_var, py_reader, words # 需修改3
-```
-
-修改后
-
-```python
-def ctr_dnn_model(embedding_size, sparse_feature_dim, use_py_reader=True):
- def embedding_layer(input):
- emb = fluid.layers.embedding(
- input=input,
- is_sparse=True,
- # you need to patch https://github.com/PaddlePaddle/Paddle/pull/14190
- # if you want to set is_distributed to True
- is_distributed=False,
- size=[sparse_feature_dim, embedding_size],
- param_attr=fluid.ParamAttr(name="SparseFeatFactors",
- initializer=fluid.initializer.Uniform()))
- seq = fluid.layers.sequence_pool(input=emb, pool_type='average')
- return emb, seq # 对应上文修改处1
- dense_input = fluid.layers.data(
- name="dense_input", shape=[dense_feature_dim], dtype='float32')
- sparse_input_ids = [
- fluid.layers.data(name="C" + str(i), shape=[1], lod_level=1, dtype='int64')
- for i in range(1, 27)]
- label = fluid.layers.data(name='label', shape=[1], dtype='int64')
- words = [dense_input] + sparse_input_ids + [label]
- sparse_embed_and_seq = list(map(embedding_layer, words[1:-1]))
-
- emb_list = [x[0] for x in sparse_embed_and_seq] # 对应上文修改处2
- sparse_embed_seq = [x[1] for x in sparse_embed_and_seq]
-
- concated = fluid.layers.concat(sparse_embed_seq + words[0:1], axis=1)
-
- train_feed_vars = words # 对应上文修改处2
- inference_feed_vars = emb_list + words[0:1]
-
- fc1 = fluid.layers.fc(input=concated, size=400, act='relu',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(concated.shape[1]))))
- fc2 = fluid.layers.fc(input=fc1, size=400, act='relu',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(fc1.shape[1]))))
- fc3 = fluid.layers.fc(input=fc2, size=400, act='relu',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(fc2.shape[1]))))
- predict = fluid.layers.fc(input=fc3, size=2, act='softmax',
- param_attr=fluid.ParamAttr(initializer=fluid.initializer.Normal(
- scale=1 / math.sqrt(fc3.shape[1]))))
- cost = fluid.layers.cross_entropy(input=predict, label=words[-1])
- avg_cost = fluid.layers.reduce_sum(cost)
- accuracy = fluid.layers.accuracy(input=predict, label=words[-1])
- auc_var, batch_auc_var, auc_states = \
- fluid.layers.auc(input=predict, label=words[-1], num_thresholds=2 ** 12, slide_steps=20)
- fetch_vars = [predict]
-
- # 对应上文修改处3
- return avg_cost, auc_var, batch_auc_var, train_feed_vars, inference_feed_vars, fetch_vars
-```
-
-说明：
-
-1) 修改处1，我们将embedding layer的输出变量返回
-2) 修改处2，我们将embedding layer的输出变量保存到`emb_list`，后者进一步保存到`inference_feed_vars`，用来将来在`save_inference_model()`时指定feed variable list。
-3) 修改处3，我们将`words`变量作为训练时的feed variable list (`train_feed_vars`)，将embedding layer的output variable作为infer时的feed variable list (`inference_feed_vars`)，将`predict`作为fetch target (`fetch_vars`)，分别返回。`inference_feed_vars`和`fetch_vars`用于`fluid.io.save_inference_model()`时指定feed variable list和fetch target list
-
-
-### 2.3 fluid.io.save_inference_model()保存裁剪后的program
-
-`fluid.io.save_inference_model()`不仅保存模型参数，还能够根据feed variable list和fetch target list参数，对program进行裁剪，形成适合inference用的program。大致原理是，根据前向网络配置，从fetch target list开始，反向查找其所依赖的OP列表，并将每个OP的输入加入目标variable list，再次递归地反向找到所有依赖OP和variable list。
-
-在2.2节中我们已经拿到所需的`inference_feed_vars`和`fetch_vars`，接下来只要在训练过程中每次保存模型参数时改为调用`fluid.io.save_inference_model()`：
-
-修改前：
-
-```python
-def train_loop(args, train_program, py_reader, loss, auc_var, batch_auc_var,
- trainer_num, trainer_id):
-
-...省略
- for pass_id in range(args.num_passes):
- pass_start = time.time()
- batch_id = 0
- py_reader.start()
-
- try:
- while True:
- loss_val, auc_val, batch_auc_val = pe.run(fetch_list=[loss.name, auc_var.name, batch_auc_var.name])
- loss_val = np.mean(loss_val)
- auc_val = np.mean(auc_val)
- batch_auc_val = np.mean(batch_auc_val)
-
- logger.info("TRAIN --> pass: {} batch: {} loss: {} auc: {}, batch_auc: {}"
- .format(pass_id, batch_id, loss_val/args.batch_size, auc_val, batch_auc_val))
- if batch_id % 1000 == 0 and batch_id != 0:
- model_dir = args.model_output_dir + '/batch-' + str(batch_id)
- if args.trainer_id == 0:
- fluid.io.save_persistables(executor=exe, dirname=model_dir,
- main_program=fluid.default_main_program())
- batch_id += 1
- except fluid.core.EOFException:
- py_reader.reset()
- print("pass_id: %d, pass_time_cost: %f" % (pass_id, time.time() - pass_start))
-...省略
-```
-
-修改后
-
-```python
-def train_loop(args,
- train_program,
- train_feed_vars,
- inference_feed_vars, # 裁剪program用的feed variable list
- fetch_vars, # 裁剪program用的fetch variable list
- loss,
- auc_var,
- batch_auc_var,
- trainer_num,
- trainer_id):
- # 因为已经将py_reader去掉，这里用fluid自带的DataFeeder
- dataset = reader.CriteoDataset(args.sparse_feature_dim)
- train_reader = paddle.batch(
- paddle.reader.shuffle(
- dataset.train([args.train_data_path], trainer_num, trainer_id),
- buf_size=args.batch_size * 100),
- batch_size=args.batch_size)
-
- inference_feed_var_names = [var.name for var in inference_feed_vars]
-
- place = fluid.CPUPlace()
- exe = fluid.Executor(place)
- exe.run(fluid.default_startup_program())
- total_time = 0
- pass_id = 0
- batch_id = 0
-
- feed_var_names = [var.name for var in feed_vars]
- feeder = fluid.DataFeeder(feed_var_names, place)
-
- for data in train_reader():
- loss_val, auc_val, batch_auc_val = exe.run(fluid.default_main_program(),
- feed = feeder.feed(data),
- fetch_list=[loss.name, auc_var.name, batch_auc_var.name])
- fluid.io.save_inference_model(model_dir,
- inference_feed_var_names,
- fetch_vars,
- exe,
- fluid.default_main_program())
- break # 我们只要裁剪后的program，不需要模型参数，因此只train一个batch就停止了
- loss_val = np.mean(loss_val)
- auc_val = np.mean(auc_val)
- batch_auc_val = np.mean(batch_auc_val)
- logger.info("TRAIN --> pass: {} batch: {} loss: {} auc: {}, batch_auc: {}"
- .format(pass_id, batch_id, loss_val/args.batch_size, auc_val, batch_auc_val))
-```
-
-### 2.4 用python再次处理inference program，去除lookup_table OP和SparseFeatFactors变量
-
-这一步是因为`fluid.io.save_inference_model()`裁剪出的program没有将lookup_table OP去除。未来如果`save_inference_model`接口完善，本节可跳过
-
-主要代码：
-
-```python
-def prune_program():
- args = parse_args()
-
- # 从磁盘打开网络配置文件并反序列化成protobuf message
- model_dir = args.model_output_dir + "/inference_only"
- model_file = model_dir + "/__model__"
- with open(model_file, "rb") as f:
- protostr = f.read()
- f.close()
- proto = framework_pb2.ProgramDesc.FromString(six.binary_type(protostr))
-
- # 去除lookup_table OP
- block = proto.blocks[0]
- kept_ops = [op for op in block.ops if op.type != "lookup_table"]
- del block.ops[:]
- block.ops.extend(kept_ops)
-
- # 去除SparseFeatFactors var
- kept_vars = [var for var in block.vars if var.name != "SparseFeatFactors"]
- del block.vars[:]
- block.vars.extend(kept_vars)
-
- # 写回磁盘文件
- with open(model_file + ".pruned", "wb") as f:
- f.write(proto.SerializePartialToString())
- f.close()
- with open(model_file + ".prototxt.pruned", "w") as f:
- f.write(text_format.MessageToString(proto))
- f.close()
-```
-
-### 2.5 裁剪过程串到一起
-
-我们提供了完整的裁剪CTR预估模型的脚本文件save_program.py，同[CTR分布式训练和Serving流程化部署](https://github.com/PaddlePaddle/Serving/blob/master/doc/DEPLOY.md)一起发布，可以在trainer和pserver容器的训练脚本目录下找到，也可以在[这里](https://github.com/PaddlePaddle/Serving/tree/master/doc/resource)下载。
-
-## 3. 整个预测计算流程
-
-Client端：
-1) Dense feature: 从dataset每条样例读取13个integer features，形成1个dense feature
-2) Sparse feature: 从dataset每条样例读取26个categorical feature，分别经过hash(str(feature_index) + feature_string)签名，得到每个feature的id，形成26个sparse feature
-
-Serving端:
-1) Dense feature: dense feature共13个float型数字，一起feed到网络dense_input这个variable对应的LodTensor
-2) Sparse feature: 26个sparse feature id，分别访问kv服务获取对应的embedding向量，feed到对应的26个embedding layer的output variable。在我们裁剪出来的网络中，这些variable分别对应的变量名为embedding_0.tmp_0, embedding_1.tmp_0, ... embedding_25.tmp_0
-3) 执行预测，获取预测结果。
diff --git a/doc/deprecated/FAQ.md b/doc/deprecated/FAQ.md
deleted file mode 100644
index 2ba9ec9d5e0a5d7c8f0ccc3ebfc480f21170751d..0000000000000000000000000000000000000000
--- a/doc/deprecated/FAQ.md
+++ /dev/null
@@ -1,26 +0,0 @@
-# FAQ
-## 1. 如何修改端口配置？
-使用该框架搭建的服务需要申请一个端口，可以通过以下方式修改端口号：
-
-- 如果在inferservice_file里指定了port:xxx，那么就去申请该端口号；
-- 否则，如果在gflags.conf里指定了--port:xxx，那就去申请该端口号；
-- 否则，使用程序里指定的默认端口号：8010。
-
-## 2. GPU预测中为何请求的响应时间波动会非常大？
-PaddleServing依托PaddlePaddle预测库执行预测计算；在GPU设备上，由于同一个进程内目前共用1个GPU stream，进程内的多个请求的预测计算会被严格串行。所以如果有2个请求同时到达某个Serving实例，不管该实例启动时创建了多少个worker线程，都不能起到加速作用，后到的请求会被排队，直到前面请求计算完成。
-
-## 3. 如何充分利用GPU卡的计算能力？
-如问题2所说，由于预测库的限制，单个Serving进程只能绑定单张GPU卡，且进程内共用1个GPU stream，所有请求必须串行计算。
-
-为提高GPU卡使用率，目前可以想到的方法是：在单张GPU卡上启动多个Serving进程，每个进程绑定一个GPU stream，多个stream并行计算。这种方法是否能起到加速作用，受限于多个因素，主要有：
-
-1. 单个stream占用GPU算力；假如单个stream已经将GPU算力占用超过50%，那么增加stream很可能会导致2个stream的job分别排队，拖慢各自的响应时间
-2. GPU显存：Serving进程需要将模型参数加载到显存中，并且计算时要在GPU显存池分配临时变量；假如单个Serving进程已经用掉超过50%的显存，则增加Serving进程会造成显存不足，导致进程报错退出
-
-为此，可采用如下步骤，进行测试：
-
-1. 加载模型时，在model_toolkit.prototxt中，model type选择FLUID_GPU_ANALYSIS或FLUID_GPU_ANALYSIS_DIR；会对模型进行静态分析，进行一定程度显存优化
-2. 在步骤1完成后，启动单个Serving进程，启动参数:`--gpuid=N --bthread_concurrency=4 --bthread_min_concurrency=4`；启动一个client，进行并发度为1的压力测试，batch size从小到大，记下平响；由于算力的限制，当batch size增大到一定程度，应该会出现响应时间明显变大；或虽然没有明显变大，但已经不满足系统需求
-3. 再启动1个Serving进程，与步骤2启动时使用相同的参数略有不同: `--gpuid=N --bthread_concurrency=4 --bthread_min_concurrency=4 --port=8011` 其中--port=8011用来让新启动的进程使用一个新的服务端口；然后同时对这2个Serving进程进行压测，继续观察batch size从小到大时平均响应时间的变化，直到取得batch size和响应时间的折中
-4. 重复步骤2-3
-5. 以2-4步的测试，来决定：单张GPU卡可以由多少个Serving进程共用; 实际部署时，就在一张GPU卡上启动这么多个Serving进程同时提供服务
diff --git a/doc/deprecated/HTTP_INTERFACE.md b/doc/deprecated/HTTP_INTERFACE.md
deleted file mode 100644
index 96df2edc7b98aaa995e93fcd806cded01d044bd7..0000000000000000000000000000000000000000
--- a/doc/deprecated/HTTP_INTERFACE.md
+++ /dev/null
@@ -1,131 +0,0 @@
-# HTTP Inferface
-
-Paddle Serving服务均可以通过HTTP接口访问，客户端只需按照Service定义的Request消息格式构造json字符串即可。客户端构造HTTP请求，将json格式数据以POST请求发给serving端，serving端**自动**按Service定义的Protobuf消息格式，将json数据转换成protobuf消息。
-
-本文档介绍以python和PHP语言访问Serving的HTTP服务接口的用法。
-
-## 1. 访问地址
-
-访问Serving节点的HTTP服务与C++服务使用同一个端口（例如8010），访问URL规则为：
-
-```
-http://127.0.0.1:8010/ServiceName/inference
-http://127.0.0.1:8010/ServiceName/debug
-```
-
-其中ServiceName应该与Serving的配置文件`conf/services.prototxt`中配置的一致，假如有如下2个service：
-
-```protobuf
-services {
- name: "BuiltinTestEchoService"
- workflows: "workflow3"
-}
-
-services {
- name: "TextClassificationService"
- workflows: "workflow6"
-}
-```
-
-则访问上述2个Serving服务的HTTP URL分别为：
-
-```
-http://127.0.0.1:8010/BuiltinTestEchoService/inference
-http://127.0.0.1:8010/BuiltinTestEchoService/debug
-
-http://127.0.0.1:8010/TextClassificationService/inference
-http://127.0.0.1:8010/TextClassificationService/debug
-```
-
-## 2. Python访问HTTP Serving
-
-Python语言访问HTTP Serving，关键在于构造json格式的请求数据，可以通过以下步骤完成：
-
-1) 按照Service定义的Request消息格式构造python object
-2) `json.dump()` / `json.dumps()` 等函数将python object转换成json格式字符串
-
-以TextClassificationService为例，关键代码如下：
-
-```python
-# Connect to server
-conn = httplib.HTTPConnection("127.0.0.1", 8010)
-
-# samples是一个list，其中每个元素是一个ids字典：
-# samples[0] = [190, 1, 70, 382, 914, 5146, 190...]
-for i in range(0, len(samples) - BATCH_SIZE, BATCH_SIZE):
- # 构建批量预测数据
- batch = samples[i: i + BATCH_SIZE]
- ids = []
- for x in batch:
- ids.append({"ids" : x})
- ids = {"instances": ids}
-
- # python object转成json
- request_json = json.dumps(ids)
-
- # 请求HTTP服务，打印response
- try:
- conn.request('POST', "/TextClassificationService/inference", request_json, {"Content-Type": "application/json"})
- response = conn.getresponse()
- print response.read()
- except httplib.HTTPException as e:
- print e.reason
-```
-
-完整示例请参考[text_classification.py](https://github.com/PaddlePaddle/Serving/blob/develop/tools/cpp_examples/demo-client/python/text_classification.py)
-
-## 3. PHP访问HTTP Serving
-
-PHP语言构造json格式字符串的步骤如下：
-
-1) 按照Service定义的Request消息格式，构造PHP array
-2) `json_encode()`函数将PHP array转换成json字符串
-
-以TextCLassificationService为例，关键代码如下：
-
-```PHP
-function http_post(&$ch, $data) {
- // array to json string
- $data_string = json_encode($data);
-
- // post data 封装
- curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
-
- // set header
- curl_setopt($ch,
- CURLOPT_HTTPHEADER,
- array(
- 'Content-Length: ' . strlen($data_string)
- )
- );
-
- // 执行
- $result = curl_exec($ch);
- return $result;
-}
-
-$ch = &http_connect('http://127.0.0.1:8010/TextClassificationService/inference');
-
-$count = 0;
-
-# $samples是一个2层array，其中每个元素是一个如下array：
-# $samples[0] = array(
-# "ids" => array(
-# [0] => int(190),
-# [1] => int(1),
-# [2] => int(70),
-# [3] => int(382),
-# [4] => int(914),
-# [5] => int(5146),
-# [6] => int(190)...)
-# )
-
-for ($i = 0; $i < count($samples) - BATCH_SIZE; $i += BATCH_SIZE) {
- $instances = array_slice($samples, $i, BATCH_SIZE);
- echo http_post($ch, array("instances" => $instances)) . "\n";
-}
-
-curl_close($ch);
-```
-
-完整代码请参考[text_classification.php](https://github.com/PaddlePaddle/Serving/blob/develop/tools/cpp_examples/demo-client/php/text_classification.php)
diff --git a/doc/images/asyn_benchmark.png b/doc/images/asyn_benchmark.png
new file mode 100644
index 0000000000000000000000000000000000000000..13b1f356e6117a0036fd964e0c038dd28711ba49
Binary files /dev/null and b/doc/images/asyn_benchmark.png differ
diff --git a/doc/images/asyn_mode.png b/doc/images/asyn_mode.png
new file mode 100644
index 0000000000000000000000000000000000000000..711f3a242b0adb9e1ce20d67734ff1b7dbf99e87
Binary files /dev/null and b/doc/images/asyn_mode.png differ
diff --git a/doc/images/detection.png b/doc/images/detection.png
new file mode 100644
index 0000000000000000000000000000000000000000..f6d5e4e4d5ed071d19b844ae3d393d730f4485ec
Binary files /dev/null and b/doc/images/detection.png differ
diff --git a/doc/images/multi_model.png b/doc/images/multi_model.png
new file mode 100644
index 0000000000000000000000000000000000000000..369410dcbb4663229bdb7f06118073536f85a9da
Binary files /dev/null and b/doc/images/multi_model.png differ
diff --git a/doc/images/qps-threads-bow.png b/doc/images/qps-threads-bow.png
new file mode 100755
index 0000000000000000000000000000000000000000..8123b71ee154ca86ea8b028fba4bceecac363007
Binary files /dev/null and b/doc/images/qps-threads-bow.png differ
diff --git a/doc/images/qps-threads-cnn.png b/doc/images/qps-threads-cnn.png
new file mode 100755
index 0000000000000000000000000000000000000000..f983d4882be0a923c0bbfbb41bbca56d684d4035
Binary files /dev/null and b/doc/images/qps-threads-cnn.png differ
diff --git a/doc/images/serving-timings.png b/doc/images/serving-timings.png
new file mode 100755
index 0000000000000000000000000000000000000000..32bab31a57799f7f1d4b80f7e809e284df98d332
Binary files /dev/null and b/doc/images/serving-timings.png differ
diff --git a/doc/images/syn_benchmark.png b/doc/images/syn_benchmark.png
new file mode 100644
index 0000000000000000000000000000000000000000..ad42e187eb1fa44fa88efc135425218112a91cbd
Binary files /dev/null and b/doc/images/syn_benchmark.png differ
diff --git a/doc/images/syn_mode.png b/doc/images/syn_mode.png
new file mode 100644
index 0000000000000000000000000000000000000000..9bae50d01dd6add790c559d50b90f36fce5cf5db
Binary files /dev/null and b/doc/images/syn_mode.png differ
diff --git a/examples/C++/PaddleClas/imagenet/README.md b/examples/C++/PaddleClas/imagenet/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..eaff522a5ae31eab08786489cbce0fa83f85e91d
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/README.md
@@ -0,0 +1,41 @@
+## Image Classification
+
+([简体中文](./README_CN.md)|English)
+
+The example uses the ResNet50_vd model to perform the imagenet 1000 classification task.
+
+### Get model config and sample dataset
+```
+sh get_model.sh
+```
+
+### Install preprocess module
+
+```
+pip3 install paddle_serving_app
+```
+
+
+### Inference Service(Support BRPC-Client/GRPC-Client/Http-Client)
+
+launch server side
+```
+python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 #cpu inference service
+```
+
+```
+python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 --gpu_ids 0 #gpu inference service
+```
+
+### BRPC-Client
+client send inference request
+```
+python3 resnet50_rpc_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
+```
+*the port of server side in this example is 9696
+
+### GRPC-Client/Http-Client
+client send inference request
+```
+python3 resnet50_http_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
+```
diff --git a/examples/C++/PaddleClas/imagenet/README_CN.md b/examples/C++/PaddleClas/imagenet/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..642bee3d0cbab98a48f2f09284ea887751752667
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/README_CN.md
@@ -0,0 +1,41 @@
+## 图像分类示例
+
+(简体中文|[English](./README.md))
+
+示例中采用ResNet50_vd模型执行imagenet 1000分类任务。
+
+### 获取模型配置文件和样例数据
+```
+sh get_model.sh
+```
+
+### 安装数据预处理模块
+
+```
+pip3 install paddle_serving_app
+```
+
+### 启动服务端（支持BRPC-Client、GRPC-Client、Http-Client）
+
+启动server端
+```
+python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 #cpu预测服务
+```
+
+```
+python3 -m paddle_serving_server.serve --model ResNet50_vd_model --port 9696 --gpu_ids 0 #gpu预测服务
+```
+
+### BRPC-Client预测
+client端进行预测
+```
+python3 resnet50_rpc_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
+```
+*server端示例中服务端口为9696端口
+
+
+### GRPC-Client/Http-Client预测
+client端进行预测
+```
+python3 resnet50_http_client.py ResNet50_vd_client_config/serving_client_conf.prototxt
+```
diff --git a/examples/C++/PaddleClas/imagenet/benchmark.py b/examples/C++/PaddleClas/imagenet/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..12b013bd2554f24430ad1810f971a340c4b6903e
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/benchmark.py
@@ -0,0 +1,127 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import unicode_literals, absolute_import
+import os
+import sys
+import time
+import requests
+import json
+import base64
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+from paddle_serving_app.reader import Sequential, File2Image, Resize
+from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
+
+args = benchmark_args()
+
+seq_preprocess = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+
+def single_func(idx, resource):
+ file_list = []
+ turns = resource["turns"]
+ latency_flags = False
+ if os.getenv("FLAGS_serving_latency"):
+ latency_flags = True
+ latency_list = []
+ for file_name in os.listdir("./image_data/n01440764"):
+ file_list.append(file_name)
+ img_list = []
+ for i in range(1000):
+ img_list.append("./image_data/n01440764/" + file_list[i])
+ profile_flags = False
+ if "FLAGS_profile_client" in os.environ and os.environ[
+ "FLAGS_profile_client"]:
+ profile_flags = True
+ if args.request == "rpc":
+ fetch = ["score"]
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
+ start = time.time()
+ for i in range(turns):
+ if args.batch_size >= 1:
+ l_start = time.time()
+ feed_batch = []
+ i_start = time.time()
+ for bi in range(args.batch_size):
+ img = seq_preprocess(img_list[i])
+ feed_batch.append({"image": img})
+ i_end = time.time()
+ if profile_flags:
+ print("PROFILE\tpid:{}\timage_pre_0:{} image_pre_1:{}".
+ format(os.getpid(),
+ int(round(i_start * 1000000)),
+ int(round(i_end * 1000000))))
+
+ result = client.predict(feed=feed_batch, fetch=fetch)
+ l_end = time.time()
+ if latency_flags:
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ elif args.request == "http":
+ py_version = sys.version_info[0]
+ server = "http://" + resource["endpoint"][idx % len(resource[
+ "endpoint"])] + "/image/prediction"
+ start = time.time()
+ for i in range(turns):
+ if py_version == 2:
+ image = base64.b64encode(
+ open("./image_data/n01440764/" + file_list[i]).read())
+ else:
+ image_path = "./image_data/n01440764/" + file_list[i]
+ image = base64.b64encode(open(image_path, "rb").read()).decode(
+ "utf-8")
+ req = json.dumps({"feed": [{"image": image}], "fetch": ["score"]})
+ r = requests.post(
+ server, data=req, headers={"Content-Type": "application/json"})
+ end = time.time()
+ if latency_flags:
+ return [[end - start], latency_list]
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = [
+ "127.0.0.1:9292", "127.0.0.1:9293", "127.0.0.1:9294", "127.0.0.1:9295"
+ ]
+ turns = 100
+ start = time.time()
+ result = multi_thread_runner.run(
+ single_func, args.thread, {"endpoint": endpoint_list,
+ "turns": turns})
+ #result = single_func(0, {"endpoint": endpoint_list})
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ avg_cost = avg_cost / args.thread
+ print("total cost: {}s".format(end - start))
+ print("each thread cost: {}s.".format(avg_cost))
+ print("qps: {}samples/s".format(args.batch_size * args.thread * turns /
+ total_cost))
+ if os.getenv("FLAGS_serving_latency"):
+ show_latency(result[1])
diff --git a/examples/C++/PaddleClas/imagenet/benchmark.sh b/examples/C++/PaddleClas/imagenet/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5287337d09550881fcf534df521012ff920d036f
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/benchmark.sh
@@ -0,0 +1,50 @@
+rm profile_log*
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+export FLAGS_profile_server=1
+export FLAGS_profile_client=1
+python -m paddle_serving_server.serve --model $1 --port 9292 --thread 4 --gpu_ids 0,1,2,3 --mem_optim --ir_optim 2> elog > stdlog &
+
+sleep 5
+gpu_id=0
+#save cpu and gpu utilization log
+if [ -d utilization ];then
+ rm -rf utilization
+else
+ mkdir utilization
+fi
+
+#warm up
+$PYTHONROOT/bin/python3 benchmark.py --thread 4 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+echo -e "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+
+for thread_num in 1 4 8 16
+do
+for batch_size in 1 4 16 64
+do
+ job_bt=`date '+%Y%m%d%H%M%S'`
+ nvidia-smi --id=0 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_use.log 2>&1 &
+ nvidia-smi --id=0 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ gpu_memory_pid=$!
+ $PYTHONROOT/bin/python benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+ kill ${gpu_memory_pid}
+ kill `ps -ef|grep used_memory|awk '{print $2}'`
+ echo "model name :" $1
+ echo "thread num :" $thread_num
+ echo "batch size :" $batch_size
+ echo "=================Done===================="
+ echo "model name :$1" >> profile_log
+ echo "batch size :$batch_size" >> profile_log
+ job_et=`date '+%Y%m%d%H%M%S'`
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$1
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$1
+ rm -rf gpu_use.log gpu_utilization.log
+ $PYTHONROOT/bin/python ../../../util/show_profile.py profile $thread_num >> profile_log
+ tail -n 8 profile >> profile_log
+ echo "" >> profile_log_$1
+done
+done
+
+#Divided log
+awk 'BEGIN{RS="\n\n"}{i++}{print > "ResNet_log_"i}' profile_log_$1
+mkdir $1_log && mv ResNet_log_* $1_log
+ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
diff --git a/examples/C++/PaddleClas/imagenet/daisy.jpg b/examples/C++/PaddleClas/imagenet/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/PaddleClas/imagenet/daisy.jpg differ
diff --git a/examples/C++/PaddleClas/imagenet/data/n01440764_10026.JPEG b/examples/C++/PaddleClas/imagenet/data/n01440764_10026.JPEG
new file mode 100644
index 0000000000000000000000000000000000000000..b985769e1c1f09585e67291a4926537186a40e49
Binary files /dev/null and b/examples/C++/PaddleClas/imagenet/data/n01440764_10026.JPEG differ
diff --git a/examples/C++/PaddleClas/imagenet/flower.jpg b/examples/C++/PaddleClas/imagenet/flower.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..903f812c4ad87e7f608e895a8e6d26d596cc0b48
Binary files /dev/null and b/examples/C++/PaddleClas/imagenet/flower.jpg differ
diff --git a/examples/C++/PaddleClas/imagenet/get_model.sh b/examples/C++/PaddleClas/imagenet/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..e017cc5101771c30f0c83e17f203ac5bff8d8570
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/get_model.sh
@@ -0,0 +1,7 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet50_vd.tar.gz
+tar -xzvf ResNet50_vd.tar.gz
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet101_vd.tar.gz
+tar -xzvf ResNet101_vd.tar.gz
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/C++/PaddleClas/imagenet/imagenet.label b/examples/C++/PaddleClas/imagenet/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/C++/PaddleClas/imagenet/resnet50_http_client.py b/examples/C++/PaddleClas/imagenet/resnet50_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..77782671b72a1fa41e65ca02b3edeb2a7753face
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/resnet50_http_client.py
@@ -0,0 +1,67 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import HttpClient
+from paddle_serving_app.reader import Sequential, URL2Image, Resize
+from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
+import time
+
+client = HttpClient()
+client.load_client_config(sys.argv[1])
+'''
+if you want use GRPC-client, set_use_grpc_client(True)
+or you can directly use client.grpc_client_predict(...)
+as for HTTP-client,set_use_grpc_client(False)(which is default)
+or you can directly use client.http_client_predict(...)
+'''
+#client.set_use_grpc_client(True)
+'''
+if you want to enable Encrypt Module,uncommenting the following line
+'''
+#client.use_key("./key")
+'''
+if you want to compress,uncommenting the following line
+'''
+#client.set_response_compress(True)
+#client.set_request_compress(True)
+'''
+we recommend use Proto data format in HTTP-body, set True(which is default)
+if you want use JSON data format in HTTP-body, set False
+'''
+#client.set_http_proto(True)
+client.connect(["127.0.0.1:9696"])
+
+label_dict = {}
+label_idx = 0
+with open("imagenet.label") as fin:
+ for line in fin:
+ label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+seq = Sequential([
+ URL2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+start = time.time()
+image_file = "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"
+for i in range(10):
+ img = seq(image_file)
+ fetch_map = client.predict(
+ feed={"image": img}, fetch=["score"], batch=False)
+ print(fetch_map)
+
+end = time.time()
+print(end - start)
diff --git a/examples/C++/PaddleClas/imagenet/resnet50_rpc_client.py b/examples/C++/PaddleClas/imagenet/resnet50_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..b23f99175b97a011c3b1c72d3b7358b646c54e68
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/resnet50_rpc_client.py
@@ -0,0 +1,49 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, URL2Image, Resize
+from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
+import time
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9696"])
+
+label_dict = {}
+label_idx = 0
+with open("imagenet.label") as fin:
+ for line in fin:
+ label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+seq = Sequential([
+ URL2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+start = time.time()
+image_file = "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"
+for i in range(10):
+ img = seq(image_file)
+ fetch_map = client.predict(
+ feed={"image": img}, fetch=["score"], batch=False)
+ prob = max(fetch_map["score"][0])
+ label = label_dict[fetch_map["score"][0].tolist().index(prob)].strip(
+ ).replace(",", "")
+ print("prediction: {}, probability: {}".format(label, prob))
+
+end = time.time()
+print(end - start)
diff --git a/examples/C++/PaddleClas/imagenet/test_image_reader.py b/examples/C++/PaddleClas/imagenet/test_image_reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..b3e1aac786360838304e03ec284076ea834ae888
--- /dev/null
+++ b/examples/C++/PaddleClas/imagenet/test_image_reader.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_app.reader.image_reader import String2Image, Base64ToImage, Sequential
+import base64
+
+
+def test_String2Image():
+ with open("./daisy.jpg") as f:
+ img_str = f.read()
+ seq = Sequential([String2Image()])
+ img = seq(img_str)
+ assert (img.shape == (563, 500, 3))
+
+
+def test_Base64ToImage():
+ with open("./daisy.jpg") as f:
+ img_str = f.read()
+ seq = Sequential([Base64ToImage()])
+ img = seq(base64.b64encode(img_str))
+ assert (img.shape == (563, 500, 3))
+
+
+if __name__ == "__main__":
+ test_String2Image()
+ test_Base64ToImage()
diff --git a/examples/C++/PaddleClas/mobilenet/README.md b/examples/C++/PaddleClas/mobilenet/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..1a16b749220bdf8e6db0dd8950fc505620cbc8fc
--- /dev/null
+++ b/examples/C++/PaddleClas/mobilenet/README.md
@@ -0,0 +1,22 @@
+# Image Classification
+
+## Get Model
+
+```
+python3 -m paddle_serving_app.package --get_model mobilenet_v2_imagenet
+tar -xzvf mobilenet_v2_imagenet.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model mobilenet_v2_imagenet_model --gpu_ids 0 --port 9393
+```
+
+### Client Prediction
+
+```
+python3 mobilenet_tutorial.py
+```
diff --git a/examples/C++/PaddleClas/mobilenet/README_CN.md b/examples/C++/PaddleClas/mobilenet/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..68474e5d80afdec183cb5bac0e9ebfc13a7f9ac6
--- /dev/null
+++ b/examples/C++/PaddleClas/mobilenet/README_CN.md
@@ -0,0 +1,22 @@
+# 图像分类
+
+## 获取模型
+
+```
+python3 -m paddle_serving_app.package --get_model mobilenet_v2_imagenet
+tar -xzvf mobilenet_v2_imagenet.tar.gz
+```
+
+## RPC 服务
+
+### 启动服务端
+
+```
+python3 -m paddle_serving_server.serve --model mobilenet_v2_imagenet_model --gpu_ids 0 --port 9393
+```
+
+### 客户端预测
+
+```
+python3 mobilenet_tutorial.py
+```
diff --git a/examples/C++/PaddleClas/mobilenet/daisy.jpg b/examples/C++/PaddleClas/mobilenet/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/PaddleClas/mobilenet/daisy.jpg differ
diff --git a/examples/C++/PaddleClas/mobilenet/mobilenet_tutorial.py b/examples/C++/PaddleClas/mobilenet/mobilenet_tutorial.py
new file mode 100644
index 0000000000000000000000000000000000000000..9550a5ff705d23d3f6a97d8498d5a8b1e4f152b7
--- /dev/null
+++ b/examples/C++/PaddleClas/mobilenet/mobilenet_tutorial.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize
+from paddle_serving_app.reader import CenterCrop, RGB2BGR, Transpose, Div, Normalize
+
+client = Client()
+client.load_client_config(
+ "mobilenet_v2_imagenet_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9393"])
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = client.predict(feed={"image": img}, fetch=["feature_map"])
+print(fetch_map["feature_map"].reshape(-1))
diff --git a/examples/C++/PaddleClas/resnet_v2_50/README.md b/examples/C++/PaddleClas/resnet_v2_50/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..12144b0ea9836c9eb647fa6482db244f1030354b
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/README.md
@@ -0,0 +1,22 @@
+# Image Classification
+
+## Get Model
+
+```
+python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
+tar -xzvf resnet_v2_50_imagenet.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --gpu_ids 0 --port 9393
+```
+
+### Client Prediction
+
+```
+python3 resnet50_v2_tutorial.py
+```
diff --git a/examples/C++/PaddleClas/resnet_v2_50/README_CN.md b/examples/C++/PaddleClas/resnet_v2_50/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..fee0e01f3cbac29052e4ae931027574ab6f778a0
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/README_CN.md
@@ -0,0 +1,22 @@
+# 图像分类
+
+## 获取模型
+
+```
+python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
+tar -xzvf resnet_v2_50_imagenet.tar.gz
+```
+
+## RPC 服务
+
+### 启动服务端
+
+```
+python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --gpu_ids 0 --port 9393
+```
+
+### 客户端预测
+
+```
+python3 resnet50_v2_tutorial.py
+```
diff --git a/examples/C++/PaddleClas/resnet_v2_50/benchmark.py b/examples/C++/PaddleClas/resnet_v2_50/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c232d84ed603d441885fdccb5230581632b3daa4
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/benchmark.py
@@ -0,0 +1,107 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import unicode_literals, absolute_import
+import os
+import sys
+import time
+import json
+import requests
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ total_number = 0
+ profile_flags = False
+ latency_flags = False
+ if os.getenv("FLAGS_profile_client"):
+ profile_flags = True
+ if os.getenv("FLAGS_serving_latency"):
+ latency_flags = True
+ latency_list = []
+
+ if args.request == "rpc":
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
+ start = time.time()
+ for i in range(turns):
+ if args.batch_size >= 1:
+ l_start = time.time()
+ seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(),
+ Transpose((2, 0, 1)), Div(255), Normalize(
+ [0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+ ])
+ image_file = "daisy.jpg"
+ img = seq(image_file)
+ feed_data = np.array(img)
+ feed_data = np.expand_dims(feed_data, 0).repeat(
+ args.batch_size, axis=0)
+ result = client.predict(
+ feed={"image": feed_data},
+ fetch=["save_infer_model/scale_0.tmp_0"],
+ batch=True)
+ l_end = time.time()
+ if latency_flags:
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_number = total_number + 1
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ else:
+ raise ValueError("not implemented {} request".format(args.request))
+ end = time.time()
+ if latency_flags:
+ return [[end - start], latency_list, [total_number]]
+ else:
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = ["127.0.0.1:9393"]
+ turns = 1
+ start = time.time()
+ result = multi_thread_runner.run(
+ single_func, args.thread, {"endpoint": endpoint_list,
+ "turns": turns})
+ end = time.time()
+ total_cost = end - start
+ total_number = 0
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / args.thread
+
+ print("total cost-include init: {}s".format(total_cost))
+ print("each thread cost: {}s. ".format(avg_cost))
+ print("qps: {}samples/s".format(args.batch_size * total_number / (
+ avg_cost * args.thread)))
+ print("qps(request): {}samples/s".format(total_number / (avg_cost *
+ args.thread)))
+ print("total count: {} ".format(total_number))
+ if os.getenv("FLAGS_serving_latency"):
+ show_latency(result[1])
diff --git a/examples/C++/PaddleClas/resnet_v2_50/benchmark.sh b/examples/C++/PaddleClas/resnet_v2_50/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..c098f8b1a63a89526d233a4696f4f5a843276364
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/benchmark.sh
@@ -0,0 +1,58 @@
+rm profile_log*
+rm -rf resnet_log*
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+export FLAGS_profile_server=1
+export FLAGS_profile_client=1
+export FLAGS_serving_latency=1
+gpu_id=3
+#save cpu and gpu utilization log
+if [ -d utilization ];then
+ rm -rf utilization
+else
+ mkdir utilization
+fi
+#start server
+python3.6 -m paddle_serving_server.serve --model $1 --port 9393 --thread 10 --gpu_ids $gpu_id --use_trt --ir_optim > elog 2>&1 &
+sleep 15
+
+#warm up
+python3.6 benchmark.py --thread 1 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+echo -e "import psutil\nimport time\nwhile True:\n\tcpu_res = psutil.cpu_percent()\n\twith open('cpu.txt', 'a+') as f:\n\t\tf.write(f'{cpu_res}\\\n')\n\ttime.sleep(0.1)" > cpu.py
+for thread_num in 1 2 4 8 16
+do
+for batch_size in 1 4 8 16 32
+do
+ job_bt=`date '+%Y%m%d%H%M%S'`
+ nvidia-smi --id=$gpu_id --query-compute-apps=used_memory --format=csv -lms 100 > gpu_memory_use.log 2>&1 &
+ nvidia-smi --id=$gpu_id --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ rm -rf cpu.txt
+ python3.6 cpu.py &
+ gpu_memory_pid=$!
+ python3.6 benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+ kill `ps -ef|grep used_memory|awk '{print $2}'` > /dev/null
+ kill `ps -ef|grep utilization.gpu|awk '{print $2}'` > /dev/null
+ kill `ps -ef|grep cpu.py|awk '{print $2}'` > /dev/null
+ echo "model_name:" $1
+ echo "thread_num:" $thread_num
+ echo "batch_size:" $batch_size
+ echo "=================Done===================="
+ echo "model_name:$1" >> profile_log_$1
+ echo "batch_size:$batch_size" >> profile_log_$1
+ job_et=`date '+%Y%m%d%H%M%S'`
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "CPU_UTILIZATION:", max}' cpu.txt >> profile_log_$1
+ #awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_memory_use.log >> profile_log_$1
+ #awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$1
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_memory_use.log >> profile_log_$1
+ awk -F" " '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$1
+ rm -rf gpu_memory_use.log gpu_utilization.log gpu_utilization.log.tmp
+ python3.6 ../../../util/show_profile.py profile $thread_num >> profile_log_$1
+ tail -n 10 profile >> profile_log_$1
+ echo "" >> profile_log_$1
+done
+done
+
+#Divided log
+awk 'BEGIN{RS="\n\n"}{i++}{print > "resnet_log_"i}' profile_log_$1
+mkdir resnet_log && mv resnet_log_* resnet_log
+ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
diff --git a/examples/C++/PaddleClas/resnet_v2_50/daisy.jpg b/examples/C++/PaddleClas/resnet_v2_50/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/PaddleClas/resnet_v2_50/daisy.jpg differ
diff --git a/examples/C++/PaddleClas/resnet_v2_50/resnet50_debug.py b/examples/C++/PaddleClas/resnet_v2_50/resnet50_debug.py
new file mode 100644
index 0000000000000000000000000000000000000000..6919b4903686817cdfbb89932396e6db28552ab3
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/resnet50_debug.py
@@ -0,0 +1,31 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+from paddle_serving_app.local_predict import LocalPredictor
+import sys
+
+debugger = LocalPredictor()
+debugger.load_model_config(sys.argv[1], gpu=True)
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = debugger.predict(feed={"image": img}, fetch=["feature_map"])
+print(fetch_map["feature_map"].reshape(-1))
diff --git a/examples/C++/PaddleClas/resnet_v2_50/resnet50_v2_tutorial.py b/examples/C++/PaddleClas/resnet_v2_50/resnet50_v2_tutorial.py
new file mode 100644
index 0000000000000000000000000000000000000000..b249d2a6df85f87258f66c96aaa779eb2e299613
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/resnet50_v2_tutorial.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+
+client = Client()
+client.load_client_config(
+ "resnet_v2_50_imagenet_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9393"])
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = client.predict(feed={"image": img}, fetch=["score"])
+print(fetch_map["score"].reshape(-1))
diff --git a/examples/C++/PaddleClas/resnet_v2_50/run_benchmark.sh b/examples/C++/PaddleClas/resnet_v2_50/run_benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..e63be7ed4153c7008ecdb7758d88f73f0444fbdd
--- /dev/null
+++ b/examples/C++/PaddleClas/resnet_v2_50/run_benchmark.sh
@@ -0,0 +1,6 @@
+if [ ! -x "ResNet50.tar.gz"]; then
+ wget https://paddle-inference-dist.bj.bcebos.com/AI-Rank/models/Paddle/ResNet50.tar.gz
+fi
+tar -xzvf ResNet50.tar.gz
+python3.6 -m paddle_serving_client.convert --dirname ./ResNet50 --model_filename model --params_filename params
+bash benchmark.sh serving_server serving_client
diff --git a/examples/C++/PaddleDetection/README.md b/examples/C++/PaddleDetection/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..e6d4da87aadd005756c151358869ba8632333118
--- /dev/null
+++ b/examples/C++/PaddleDetection/README.md
@@ -0,0 +1,23 @@
+# Serve models from Paddle Detection
+
+(English|[简体中文](./README_CN.md))
+
+### Introduction
+
+PaddleDetection flying paddle target detection development kit is designed to help developers complete the whole development process of detection model formation, training, optimization and deployment faster and better. For details, see [Github](https://github.com/PaddlePaddle/PaddleDetection/tree/master)
+
+This article mainly introduces the deployment of Paddle Detection's dynamic graph model on Serving.
+
+Paddle Detection provides a large number of [Model Zoo](https://github.com/PaddlePaddle/PaddleDetection/blob/master/docs/MODEL_ZOO_cn.md), these model libraries can be used in Paddle Serving with export tools Model. For the export tutorial, please refer to [Paddle Detection Export Model Tutorial (Simplified Chinese)](https://github.com/PaddlePaddle/PaddleDetection/blob/master/deploy/EXPORT_MODEL.md).
+
+### Serving example
+Several examples of PaddleDetection models used in Serving are given in this folder
+All examples support TensorRT.
+
+- [Faster RCNN](./faster_rcnn_r50_fpn_1x_coco)
+- [PPYOLO](./ppyolo_r50vd_dcn_1x_coco)
+- [TTFNet](./ttfnet_darknet53_1x_coco)
+- [YOLOv3](./yolov3_darknet53_270e_coco)
+- [HRNet](./faster_rcnn_hrnetv2p_w18_1x)
+- [Fcos](./fcos_dcn_r50_fpn_1x_coco)
+- [SSD](./ssd_vgg16_300_240e_voc/)
diff --git a/examples/C++/PaddleDetection/README_CN.md b/examples/C++/PaddleDetection/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..f5e62d7a7a0f682aa6e8b6a07ff9b304328d222e
--- /dev/null
+++ b/examples/C++/PaddleDetection/README_CN.md
@@ -0,0 +1,24 @@
+## 使用Paddle Detection模型
+
+([English](./README.md)|简体中文)
+
+### 简介
+
+PaddleDetection飞桨目标检测开发套件，旨在帮助开发者更快更好地完成检测模型的组建、训练、优化及部署等全开发流程。详情参见[Github](https://github.com/PaddlePaddle/PaddleDetection/tree/master)
+
+本文主要是介绍Paddle Detection的动态图模型在Serving上的部署。
+
+### 导出模型
+
+Paddle Detection提供了大量的[模型库](https://github.com/PaddlePaddle/PaddleDetection/blob/master/docs/MODEL_ZOO_cn.md), 这些模型库配合导出工具都可以得到可以用于Paddle Serving的模型。导出教程参见[Paddle Detection模型导出教程](https://github.com/PaddlePaddle/PaddleDetection/blob/master/deploy/EXPORT_MODEL.md)。
+
+### Serving示例
+本文件夹下给出了多个PaddleDetection模型用于Serving的范例
+
+- [Faster RCNN](./faster_rcnn_r50_fpn_1x_coco)
+- [PPYOLO](./ppyolo_r50vd_dcn_1x_coco)
+- [TTFNet](./ttfnet_darknet53_1x_coco)
+- [YOLOv3](./yolov3_darknet53_270e_coco)
+- [HRNet](./faster_rcnn_hrnetv2p_w18_1x)
+- [Fcos](./fcos_dcn_r50_fpn_1x_coco)
+- [SSD](./ssd_vgg16_300_240e_voc/)
diff --git a/examples/C++/PaddleDetection/blazeface/README.md b/examples/C++/PaddleDetection/blazeface/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..29e3026b4d972e141eabcc1a180d7a5cdb804a52
--- /dev/null
+++ b/examples/C++/PaddleDetection/blazeface/README.md
@@ -0,0 +1,23 @@
+# Blazeface
+
+## Get Model
+```
+python3 -m paddle_serving_app.package --get_model blazeface
+tar -xf blazeface.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 9494
+```
+
+### Client Prediction
+
+```
+python3 test_client.py serving_client/serving_client_conf.prototxt test.jpg
+```
+
+the result is in `output` folder, including a json file and image file with bounding boxes.
diff --git a/examples/C++/PaddleDetection/blazeface/test_client.py b/examples/C++/PaddleDetection/blazeface/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..5e22cb866e34cba9fbd38c415215b8985b1584b2
--- /dev/null
+++ b/examples/C++/PaddleDetection/blazeface/test_client.py
@@ -0,0 +1,39 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import sys
+import numpy as np
+from paddle_serving_app.reader import BlazeFacePostprocess
+
+preprocess = Sequential([
+ File2Image(),
+ Normalize([104, 117, 123], [127.502231, 127.502231, 127.502231], False)
+])
+
+postprocess = BlazeFacePostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config(sys.argv[1])
+client.connect(['127.0.0.1:9494'])
+
+im_0 = preprocess(sys.argv[2])
+tmp = Transpose((2, 0, 1))
+im = tmp(im_0)
+fetch_map = client.predict(
+ feed={"image": im}, fetch=["detection_output_0.tmp_0"])
+fetch_map["image"] = sys.argv[2]
+fetch_map["im_shape"] = im_0.shape
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/cascade_rcnn/000000570688.jpg b/examples/C++/PaddleDetection/cascade_rcnn/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/cascade_rcnn/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/cascade_rcnn/README.md b/examples/C++/PaddleDetection/cascade_rcnn/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..8029f39a11fcbadefe7f7c77ad709b4a0080707e
--- /dev/null
+++ b/examples/C++/PaddleDetection/cascade_rcnn/README.md
@@ -0,0 +1,21 @@
+# Cascade RCNN model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get The Cascade RCNN Model
+```
+sh get_data.sh
+```
+If you want to have more detection models, please refer to [Paddle Detection Model Zoo](https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/MODEL_ZOO_cn.md)
+
+### Start the service
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 9292 --gpu_id 0
+```
+
+### Perform prediction
+```
+python3 test_client.py 000000570688.jpg
+```
+
+Image with bounding boxes and json result would be saved in `output` folder.
diff --git a/examples/C++/PaddleDetection/cascade_rcnn/README_CN.md b/examples/C++/PaddleDetection/cascade_rcnn/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..828aba8a9546465c89ef673625b8b2b5140f96a6
--- /dev/null
+++ b/examples/C++/PaddleDetection/cascade_rcnn/README_CN.md
@@ -0,0 +1,21 @@
+# 使用Paddle Serving部署Cascade RCNN模型
+
+(简体中文|[English](./README.md))
+
+## 获得Cascade RCNN模型
+```
+sh get_data.sh
+```
+如果你想要更多的检测模型，请参考[Paddle检测模型库](https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/docs/MODEL_ZOO_cn.md)
+
+### 启动服务
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 9292 --gpu_id 0
+```
+
+### 执行预测
+```
+python3 test_client.py 000000570688.jpg
+```
+
+客户端已经为图片做好了后处理，在`output`文件夹下存放各个框的json格式信息还有后处理结果图片。
diff --git a/examples/C++/PaddleDetection/cascade_rcnn/get_data.sh b/examples/C++/PaddleDetection/cascade_rcnn/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..204ae1a269e00a0156141946db7cfed37475564f
--- /dev/null
+++ b/examples/C++/PaddleDetection/cascade_rcnn/get_data.sh
@@ -0,0 +1,2 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/cascade_mask_rcnn_r50_vd_fpn_ssld_2x_coco_serving.tar.gz
+tar xf cascade_mask_rcnn_r50_vd_fpn_ssld_2x_coco_serving.tar.gz
diff --git a/examples/C++/PaddleDetection/cascade_rcnn/label_list.txt b/examples/C++/PaddleDetection/cascade_rcnn/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/cascade_rcnn/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/cascade_rcnn/test_client.py b/examples/C++/PaddleDetection/cascade_rcnn/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..6ddb8a79e40c00539aaa232d5aae04e579335822
--- /dev/null
+++ b/examples/C++/PaddleDetection/cascade_rcnn/test_client.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(), DetectionResize(
+ (800, 1333), True, interpolation=2),
+ DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
+ DetectionTranspose((2, 0, 1)), DetectionPadStride(32)
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9292'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+print(fetch_map)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/000000570688.jpg b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README.md b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..3c0fb8dbee6c0d6eac7b09cb16428679cb8b9e5d
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README.md
@@ -0,0 +1,23 @@
+# Faster RCNN HRNet model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get The Faster RCNN HRNet Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/faster_rcnn_hrnetv2p_w18_1x.tar.gz
+```
+
+### Start the service
+```
+tar xf faster_rcnn_hrnetv2p_w18_1x.tar.gz
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+This model support TensorRT, if you want a faster inference, please use `--use_trt`. But you need to do some extra work.
+Please reference to https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
+
+
+### Prediction
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README_CN.md b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..11dcbd85fe62f4dae5a4714ad3996424499024c0
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/README_CN.md
@@ -0,0 +1,22 @@
+# 使用Paddle Serving部署Faster RCNN HRNet模型
+
+(简体中文|[English](./README.md))
+
+## 获得Faster RCNN HRNet模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/faster_rcnn_hrnetv2p_w18_1x.tar.gz
+```
+
+
+### 启动服务
+```
+tar xf faster_rcnn_hrnetv2p_w18_1x.tar.gz
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项,但此时需要额外设置子图的TRT变长最大最小最优shape.
+请参考https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
+
+### 执行预测
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/label_list.txt b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/test_client.py b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..329f6effb4cb8a8a163cada106f6aaacc1cc3857
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_hrnetv2p_w18_1x/test_client.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(),
+ DetectionResize((800, 1333), True, interpolation=2),
+ DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
+ DetectionTranspose((2,0,1)),
+ DetectionPadStride(32)
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+print(fetch_map)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/000000570688.jpg b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README.md b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..d56aa416b9e54114646f9271c27f6afde7d41259
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README.md
@@ -0,0 +1,39 @@
+# Faster RCNN model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get The Faster RCNN Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/faster_rcnn_r50_fpn_1x_coco.tar
+```
+
+### Start the service
+```
+tar xf faster_rcnn_r50_fpn_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+This model support TensorRT, if you want a faster inference, please use `--use_trt`. But you need to do some extra work.
+Please reference to https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
+
+
+### Perform prediction
+```
+python3 test_client.py 000000570688.jpg
+```
+
+## 3. Result analysis
+
+
+
+
+
+This is the input picture
+
+
+
+
+
+
+
+This is the picture after adding bbox. You can see that the client has done post-processing for the picture. In addition, the output/bbox.json also has the number and coordinate information of each box.
diff --git a/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README_CN.md b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..f8475daf029ae2230432871237281970052fe3e3
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/README_CN.md
@@ -0,0 +1,37 @@
+# 使用Paddle Serving部署Faster RCNN模型
+
+(简体中文|[English](./README.md))
+
+## 获得Faster RCNN模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/faster_rcnn_r50_fpn_1x_coco.tar
+```
+
+
+### 启动服务
+```
+tar xf faster_rcnn_r50_fpn_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项,但此时需要额外设置子图的TRT变长最大最小最优shape.
+请参考https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/c%2B%2B/paddle-trt/trt_dynamic_shape_test.cc#L40
+
+### 执行预测
+```
+python3 test_client.py 000000570688.jpg
+```
+
+## 3. 结果分析
+
+
+
+
+
+这是输入图片
+
+
+
+
+
+
+这是实现添加了bbox之后的图片，可以看到客户端已经为图片做好了后处理，此外在output/bbox.json也有各个框的编号和坐标信息。
diff --git a/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/label_list.txt b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/test_client.py b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..b6b2c534b0609692fea34bafcf4059222738debd
--- /dev/null
+++ b/examples/C++/PaddleDetection/faster_rcnn_r50_fpn_1x_coco/test_client.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(),
+ DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
+ DetectionResize(
+ (800, 1333), True, interpolation=cv2.INTER_LINEAR),
+ DetectionTranspose((2,0,1)),
+ DetectionPadStride(128)
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/000000014439.jpg b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/000000014439.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..0abbdab06eb5950b93908cc91adfa640e8a3ac78
Binary files /dev/null and b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/000000014439.jpg differ
diff --git a/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README.md b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..58d13e53fe9ac3b177a3b6e6661a1370efa796b9
--- /dev/null
+++ b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README.md
@@ -0,0 +1,20 @@
+# FCOS model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/fcos_dcn_r50_fpn_1x_coco.tar
+```
+
+### Start the service
+```
+tar xf fcos_dcn_r50_fpn_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+This model support TensorRT, if you want a faster inference, please use `--use_trt`.
+
+### Perform prediction
+```
+python3 test_client.py 000000014439.jpg
+```
diff --git a/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README_CN.md b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..af2fd8753cc56ef9c732c21020712674313ac4fa
--- /dev/null
+++ b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/README_CN.md
@@ -0,0 +1,22 @@
+# 使用Paddle Serving部署FCOS模型
+
+(简体中文|[English](./README.md))
+
+## 获得模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/fcos_dcn_r50_fpn_1x_coco.tar
+```
+
+
+### 启动服务
+```
+tar xf fcos_dcn_r50_fpn_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
+
+### 执行预测
+```
+python3 test_client.py 000000014439.jpg
+```
diff --git a/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/label_list.txt b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/test_client.py b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..0cacceb8610e58512af83ec389c805584bfcd664
--- /dev/null
+++ b/examples/C++/PaddleDetection/fcos_dcn_r50_fpn_1x_coco/test_client.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(),
+ DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
+ DetectionResize(
+ (800, 1333), True, interpolation=cv2.INTER_LINEAR), DetectionTranspose(
+ (2, 0, 1)), DetectionPadStride(128)
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+print(fetch_map)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/000000570688.jpg b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README.md b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..8060e087107e54bc401849fd576497e9fc9cd421
--- /dev/null
+++ b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README.md
@@ -0,0 +1,21 @@
+# PP-YOLO model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get The Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_r50vd_dcn_1x_coco.tar
+```
+
+### Start the service
+```
+tar xf ppyolo_r50vd_dcn_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+This model support TensorRT, if you want a faster inference, please use `--use_trt`.
+
+### Perform prediction
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README_CN.md b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..3071db7b124fd998d15901be7a78a67018d0de0f
--- /dev/null
+++ b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/README_CN.md
@@ -0,0 +1,22 @@
+# 使用Paddle Serving部署PP-YOLO模型
+
+(简体中文|[English](./README.md))
+
+## 获得模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_r50vd_dcn_1x_coco.tar
+```
+
+
+### 启动服务
+```
+tar xf ppyolo_r50vd_dcn_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
+
+### 执行预测
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/label_list.txt b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/test_client.py b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..6e6c5dd65ddb5840040290c3c019044fe6b10fd2
--- /dev/null
+++ b/examples/C++/PaddleDetection/ppyolo_r50vd_dcn_1x_coco/test_client.py
@@ -0,0 +1,44 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(),
+ DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
+ DetectionResize(
+ (608, 608), False, interpolation=2), DetectionTranspose((2, 0, 1))
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/000000014439.jpg b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/000000014439.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..0abbdab06eb5950b93908cc91adfa640e8a3ac78
Binary files /dev/null and b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/000000014439.jpg differ
diff --git a/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/README.md b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..8a9a766c7b24d8468cbc72d6affd90263e86b013
--- /dev/null
+++ b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/README.md
@@ -0,0 +1,20 @@
+# SSD model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ssd_vgg16_300_240e_voc.tar
+```
+
+### Start the service
+```
+tar xf ssd_vgg16_300_240e_voc.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+This model support TensorRT, if you want a faster inference, please use `--use_trt`.
+
+### Perform prediction
+```
+python3 test_client.py 000000014439.jpg
+```
diff --git a/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/README_CN.md b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..d3df37d774bd1a478af0a41a9fca9f238ca69aac
--- /dev/null
+++ b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/README_CN.md
@@ -0,0 +1,22 @@
+# 使用Paddle Serving部署SSD模型
+
+(简体中文|[English](./README.md))
+
+## 获得模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ssd_vgg16_300_240e_voc.tar
+```
+
+
+### 启动服务
+```
+tar xf ssd_vgg16_300_240e_voc.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
+
+### 执行预测
+```
+python3 test_client.py 000000014439.jpg
+```
diff --git a/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/label_list.txt b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8420ab35ede7400974f25836a6bb543024686a0e
--- /dev/null
+++ b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/label_list.txt
@@ -0,0 +1,20 @@
+aeroplane
+bicycle
+bird
+boat
+bottle
+bus
+car
+cat
+chair
+cow
+diningtable
+dog
+horse
+motorbike
+person
+pottedplant
+sheep
+sofa
+train
+tvmonitor
diff --git a/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/test_client.py b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..d270b0fb5f0001f542b8877d0ae3059a0b33e781
--- /dev/null
+++ b/examples/C++/PaddleDetection/ssd_vgg16_300_240e_voc/test_client.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(),
+ DetectionResize((300, 300), False, interpolation=cv2.INTER_LINEAR),
+ DetectionNormalize([104.0, 117.0, 123.0], [1.0, 1.0, 1.0], False),
+ DetectionTranspose((2, 0, 1)),
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+print(fetch_map)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/000000570688.jpg b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/README.md b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..adf5de2abd39c3b440ac43ab9b1c1c58aba69c51
--- /dev/null
+++ b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/README.md
@@ -0,0 +1,20 @@
+# TTF-Net model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/ttfnet_darknet53_1x_coco.tar
+```
+
+### Start the service
+```
+tar xf ttfnet_darknet53_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+This model support TensorRT, if you want a faster inference, please use `--use_trt`.
+
+### Perform prediction
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/README_CN.md b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..7a2c860967643585023ce0f644a36e9c056c21a2
--- /dev/null
+++ b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/README_CN.md
@@ -0,0 +1,22 @@
+# 使用Paddle Serving部署TTF-Net模型
+
+(简体中文|[English](./README.md))
+
+## 获得模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/ttfnet_darknet53_1x_coco.tar
+```
+
+
+### 启动服务
+```
+tar xf ttfnet_darknet53_1x_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
+
+### 执行预测
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/label_list.txt b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/test_client.py b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..f957bff60c9e27b344703fe54516cac4817ba27c
--- /dev/null
+++ b/examples/C++/PaddleDetection/ttfnet_darknet53_1x_coco/test_client.py
@@ -0,0 +1,43 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(), DetectionResize(
+ (512, 512), False, interpolation=cv2.INTER_LINEAR),
+ DetectionNormalize([123.675, 116.28, 103.53], [58.395, 57.12, 57.375],
+ False), DetectionTranspose((2, 0, 1))
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+print(fetch_map)
diff --git a/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/000000570688.jpg b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/README.md b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..32670748db42336053d01e61bf087d00c03c7e06
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/README.md
@@ -0,0 +1,21 @@
+# YOLOv3 model on Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/yolov3_darknet53_270e_coco.tar
+```
+
+### Start the service
+```
+tar xf yolov3_darknet53_270e_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+This model support TensorRT, if you want a faster inference, please use `--use_trt`.
+
+### Perform prediction
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/README_CN.md b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..4185e0fe4963113ed0f9c0ea865705fd33226d1b
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/README_CN.md
@@ -0,0 +1,22 @@
+# 使用Paddle Serving部署YOLOv3模型
+
+(简体中文|[English](./README.md))
+
+## 获得模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/yolov3_darknet53_270e_coco.tar
+```
+
+
+### 启动服务
+```
+tar xf yolov3_darknet53_270e_coco.tar
+python3 -m paddle_serving_server.serve --model serving_server --port 9494 --gpu_ids 0
+```
+
+该模型支持TensorRT，如果想要更快的预测速度，可以开启`--use_trt`选项。
+
+### 执行预测
+```
+python3 test_client.py 000000570688.jpg
+```
diff --git a/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/label_list.txt b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/test_client.py b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..a5433339935a9b9c190fadf815416b811693b457
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov3_darknet53_270e_coco/test_client.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+
+preprocess = DetectionSequential([
+ DetectionFile2Image(),
+ DetectionResize(
+ (608, 608), False, interpolation=2),
+ DetectionNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True),
+ DetectionTranspose((2, 0, 1)),
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output")
+client = Client()
+
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9494'])
+
+im, im_info = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_shape": np.array(list(im.shape[1:])).reshape(-1),
+ "scale_factor": im_info['scale_factor'],
+ },
+ fetch=["save_infer_model/scale_0.tmp_1"],
+ batch=False)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleDetection/yolov4/000000570688.jpg b/examples/C++/PaddleDetection/yolov4/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/C++/PaddleDetection/yolov4/000000570688.jpg differ
diff --git a/examples/C++/PaddleDetection/yolov4/README.md b/examples/C++/PaddleDetection/yolov4/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..0c7cfa7c0ffb4938456aa908015aff2daf367727
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov4/README.md
@@ -0,0 +1,23 @@
+# Yolov4 Detection Service
+
+([简体中文](README_CN.md)|English)
+
+## Get Model
+
+```
+python3 -m paddle_serving_app.package --get_model yolov4
+tar -xzvf yolov4.tar.gz
+```
+
+## Start RPC Service
+
+```
+python3 -m paddle_serving_server.serve --model yolov4_model --port 9393 --gpu_ids 0
+```
+
+## Prediction
+
+```
+python3 test_client.py 000000570688.jpg
+```
+After the prediction is completed, a json file to save the prediction result and a picture with the detection result box will be generated in the `./outpu folder.
diff --git a/examples/C++/PaddleDetection/yolov4/README_CN.md b/examples/C++/PaddleDetection/yolov4/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..1c773033418b9d072a7096a91d47b665b465c322
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov4/README_CN.md
@@ -0,0 +1,24 @@
+# Yolov4 检测服务
+
+(简体中文|[English](README.md))
+
+## 获取模型
+
+```
+python3 -m paddle_serving_app.package --get_model yolov4
+tar -xzvf yolov4.tar.gz
+```
+
+## 启动RPC服务
+
+```
+python3 -m paddle_serving_server.serve --model yolov4_model --port 9393 --gpu_ids 0
+```
+
+## 预测
+
+```
+python3 test_client.py 000000570688.jpg
+```
+
+预测完成会在`./output`文件夹下生成保存预测结果的json文件以及标出检测结果框的图片。
diff --git a/examples/C++/PaddleDetection/yolov4/label_list.txt b/examples/C++/PaddleDetection/yolov4/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov4/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/C++/PaddleDetection/yolov4/test_client.py b/examples/C++/PaddleDetection/yolov4/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..dfcd58610c3b8df1a1579350c6bb756119cf6940
--- /dev/null
+++ b/examples/C++/PaddleDetection/yolov4/test_client.py
@@ -0,0 +1,41 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_app.reader import *
+import cv2
+preprocess = Sequential([
+ File2Image(), BGR2RGB(), Resize(
+ (608, 608), interpolation=cv2.INTER_LINEAR), Div(255.0), Transpose(
+ (2, 0, 1))
+])
+
+postprocess = RCNNPostprocess("label_list.txt", "output", [608, 608])
+client = Client()
+
+client.load_client_config("yolov4_client/serving_client_conf.prototxt")
+client.connect(['127.0.0.1:9393'])
+
+im = preprocess(sys.argv[1])
+fetch_map = client.predict(
+ feed={
+ "image": im,
+ "im_size": np.array(list(im.shape[1:])),
+ },
+ fetch=["save_infer_model/scale_0.tmp_0"],
+ batch=False)
+fetch_map["image"] = sys.argv[1]
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleNLP/bert/README.md b/examples/C++/PaddleNLP/bert/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..5d3242837f6d8be08f321d68890587e4bba725e8
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/README.md
@@ -0,0 +1,80 @@
+Http## Bert as service
+
+([简体中文](./README_CN.md)|English)
+
+In the example, a BERT model is used for semantic understanding prediction, and the text is represented as a vector, which can be used for further analysis and prediction.
+If your python version is 3.X, replace the 'pip' field in the following command with 'pip3',replace 'python' with 'python3'.
+
+### Getting Model
+method 1:
+This example use model [BERT Chinese Model](https://www.paddlepaddle.org.cn/hubdetail?name=bert_chinese_L-12_H-768_A-12&en_category=SemanticModel) from [Paddlehub](https://github.com/PaddlePaddle/PaddleHub).
+
+Install paddlehub first
+```
+pip3 install paddlehub
+```
+
+run
+```
+python3 prepare_model.py 128
+```
+
+**PaddleHub only support Python 3.5+**
+
+the 128 in the command above means max_seq_len in BERT model, which is the length of sample after preprocessing.
+the config file and model file for server side are saved in the folder bert_seq128_model.
+the config file generated for client side is saved in the folder bert_seq128_client.
+
+method 2:
+You can also download the above model from BOS(max_seq_len=128). After decompression, the config file and model file for server side are stored in the bert_chinese_L-12_H-768_A-12_model folder, and the config file generated for client side is stored in the bert_chinese_L-12_H-768_A-12_client folder:
+```shell
+wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SemanticModel/bert_chinese_L-12_H-768_A-12.tar.gz
+tar -xzf bert_chinese_L-12_H-768_A-12.tar.gz
+mv bert_chinese_L-12_H-768_A-12_model bert_seq128_model
+mv bert_chinese_L-12_H-768_A-12_client bert_seq128_client
+```
+if your model is bert_chinese_L-12_H-768_A-12_model, replace the 'bert_seq128_model' field in the following command with 'bert_chinese_L-12_H-768_A-12_model',replace 'bert_seq128_client' with 'bert_chinese_L-12_H-768_A-12_client'.
+
+### Getting Dict and Sample Dataset
+
+```
+sh get_data.sh
+```
+this script will download Chinese Dictionary File vocab.txt and Chinese Sample Data data-c.txt
+
+### Inference Service(Support BRPC-Client、GRPC-Client、Http-Client)
+start cpu inference service,Run
+```
+python3 -m paddle_serving_server.serve --model bert_seq128_model/ --port 9292 #cpu inference service
+```
+Or,start gpu inference service,Run
+```
+python3 -m paddle_serving_server.serve --model bert_seq128_model/ --port 9292 --gpu_ids 0 #launch gpu inference service at GPU 0
+```
+
+### BRPC-Client Inference
+
+before prediction we should install paddle_serving_app. This module provides data preprocessing for BERT model.
+```
+pip3 install paddle_serving_app
+```
+Run
+```
+head data-c.txt | python3 bert_client.py --model bert_seq128_client/serving_client_conf.prototxt
+```
+
+the client reads data from data-c.txt and send prediction request, the prediction is given by word vector. (Due to massive data in the word vector, we do not print it).
+
+#### GRPC-Client/HTTP-Client
+Run
+```
+head data-c.txt | python3 bert_httpclient.py --model bert_seq128_client/serving_client_conf.prototxt
+
+```
+
+
+## Benchmark
+``` shell
+bash benchmark.sh bert_seq128_model bert_seq128_client
+```
+The output log file of benchmark named `profile_log_bert_seq128_model`
diff --git a/examples/C++/PaddleNLP/bert/README_CN.md b/examples/C++/PaddleNLP/bert/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..42bc3ffab0ad51e304b11a78634b5a90415d1ace
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/README_CN.md
@@ -0,0 +1,85 @@
+## 语义理解预测服务
+
+(简体中文|[English](./README.md))
+
+示例中采用BERT模型进行语义理解预测，将文本表示为向量的形式，可以用来做进一步的分析和预测。
+
+若使用python的版本为3.X, 将以下命令中的pip 替换为pip3, python替换为python3.
+### 获取模型
+方法1：
+示例中采用[Paddlehub](https://github.com/PaddlePaddle/PaddleHub)中的[BERT中文模型](https://www.paddlepaddle.org.cn/hubdetail?name=bert_chinese_L-12_H-768_A-12&en_category=SemanticModel)。
+请先安装paddlehub
+```
+pip3 install paddlehub
+```
+执行
+```
+python3 prepare_model.py 128
+```
+参数128表示BERT模型中的max_seq_len，即预处理后的样本长度。
+生成server端配置文件与模型文件，存放在bert_seq128_model文件夹。
+生成client端配置文件，存放在bert_seq128_client文件夹。
+
+方法2：
+您也可以从bos上直接下载上述模型（max_seq_len=128），解压后server端配置文件与模型文件存放在bert_chinese_L-12_H-768_A-12_model文件夹，client端配置文件存放在bert_chinese_L-12_H-768_A-12_client文件夹：
+```shell
+wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SemanticModel/bert_chinese_L-12_H-768_A-12.tar.gz
+tar -xzf bert_chinese_L-12_H-768_A-12.tar.gz
+mv bert_chinese_L-12_H-768_A-12_model bert_seq128_model
+mv bert_chinese_L-12_H-768_A-12_client bert_seq128_client
+```
+若使用bert_chinese_L-12_H-768_A-12_model模型，将下面命令中的bert_seq128_model字段替换为bert_chinese_L-12_H-768_A-12_model，bert_seq128_client字段替换为bert_chinese_L-12_H-768_A-12_client.
+
+
+
+
+### 获取词典和样例数据
+
+```
+sh get_data.sh
+```
+脚本将下载中文词典vocab.txt和中文样例数据data-c.txt
+
+### 启动预测服务（支持BRPC-Client、GRPC-Client、HTTP-Client三种方式访问）
+启动cpu预测服务，执行
+```
+python3 -m paddle_serving_server.serve --model bert_seq128_model/ --port 9292 #启动cpu预测服务
+
+```
+或者，启动gpu预测服务，执行
+```
+python3 -m paddle_serving_server.serve --model bert_seq128_model/ --port 9292 --gpu_ids 0 #在gpu 0上启动gpu预测服务
+
+```
+
+### 执行预测
+
+执行预测前需要安装paddle_serving_app，模块中提供了BERT模型的数据预处理方法。
+```
+pip3 install paddle_serving_app
+```
+
+#### BRPC-Client
+执行
+```
+head data-c.txt | python3 bert_client.py --model bert_seq128_client/serving_client_conf.prototxt
+
+```
+启动client读取data-c.txt中的数据进行预测，预测结果为文本的向量表示（由于数据较多，脚本中没有将输出进行打印），server端的地址在脚本中修改。
+
+#### GRPC-Client/HTTP-Client
+执行
+```
+head data-c.txt | python3 bert_httpclient.py --model bert_seq128_client/serving_client_conf.prototxt
+
+```
+
+## 性能测试
+``` shell
+bash benchmark.sh bert_seq128_model bert_seq128_client
+```
+性能测试的日志文件为profile_log_bert_seq128_model
+
+如需修改性能测试用例的参数，请修改benchmark.sh中的配置信息。
+
+注意:bert_seq128_model和bert_seq128_client路径后不要加'/'符号,示例需要在GPU机器上运行。
diff --git a/examples/C++/PaddleNLP/bert/batching.py b/examples/C++/PaddleNLP/bert/batching.py
new file mode 100644
index 0000000000000000000000000000000000000000..5ec5f320cf5ec7bd0ab4624d9b39ef936553c774
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/batching.py
@@ -0,0 +1,126 @@
+#coding:utf-8
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Mask, padding and batching."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+
+
+def prepare_batch_data(insts,
+ total_token_num,
+ max_seq_len=128,
+ pad_id=None,
+ cls_id=None,
+ sep_id=None,
+ mask_id=None,
+ return_input_mask=True,
+ return_max_len=True,
+ return_num_token=False):
+ """
+ 1. generate Tensor of data
+ 2. generate Tensor of position
+ 3. generate self attention mask, [shape: batch_size * max_len * max_len]
+ """
+
+ batch_src_ids = [inst[0] for inst in insts]
+ batch_sent_ids = [inst[1] for inst in insts]
+ batch_pos_ids = [inst[2] for inst in insts]
+ labels_list = []
+ # compatible with squad, whose example includes start/end positions,
+ # or unique id
+
+ for i in range(3, len(insts[0]), 1):
+ labels = [inst[i] for inst in insts]
+ labels = np.array(labels).astype("int64").reshape([-1, 1])
+ labels_list.append(labels)
+
+ out = batch_src_ids
+ # Second step: padding
+ src_id, self_input_mask = pad_batch_data(
+ out, pad_idx=pad_id, max_seq_len=max_seq_len, return_input_mask=True)
+ pos_id = pad_batch_data(
+ batch_pos_ids,
+ pad_idx=pad_id,
+ max_seq_len=max_seq_len,
+ return_pos=False,
+ return_input_mask=False)
+ sent_id = pad_batch_data(
+ batch_sent_ids,
+ pad_idx=pad_id,
+ max_seq_len=max_seq_len,
+ return_pos=False,
+ return_input_mask=False)
+
+ return_list = [src_id, pos_id, sent_id, self_input_mask] + labels_list
+
+ return return_list if len(return_list) > 1 else return_list[0]
+
+
+def pad_batch_data(insts,
+ pad_idx=0,
+ max_seq_len=128,
+ return_pos=False,
+ return_input_mask=False,
+ return_max_len=False,
+ return_num_token=False,
+ return_seq_lens=False):
+ """
+ Pad the instances to the max sequence length in batch, and generate the
+ corresponding position data and input mask.
+ """
+ return_list = []
+ #max_len = max(len(inst) for inst in insts)
+ max_len = max_seq_len
+ # Any token included in dict can be used to pad, since the paddings' loss
+ # will be masked out by weights and make no effect on parameter gradients.
+
+ inst_data = np.array([
+ list(inst) + list([pad_idx] * (max_len - len(inst))) for inst in insts
+ ])
+ return_list += [inst_data.astype("int64").reshape([-1, max_len, 1])]
+
+ # position data
+ if return_pos:
+ inst_pos = np.array([
+ list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))
+ for inst in insts
+ ])
+
+ return_list += [inst_pos.astype("int64").reshape([-1, max_len, 1])]
+
+ if return_input_mask:
+ # This is used to avoid attention on paddings.
+ input_mask_data = np.array(
+ [[1] * len(inst) + [0] * (max_len - len(inst)) for inst in insts])
+ input_mask_data = np.expand_dims(input_mask_data, axis=-1)
+ return_list += [input_mask_data.astype("float32")]
+
+ if return_max_len:
+ return_list += [max_len]
+
+ if return_num_token:
+ num_token = 0
+ for inst in insts:
+ num_token += len(inst)
+ return_list += [num_token]
+
+ if return_seq_lens:
+ seq_lens = np.array([len(inst) for inst in insts])
+ return_list += [seq_lens.astype("int64").reshape([-1, 1])]
+
+ return return_list if len(return_list) > 1 else return_list[0]
diff --git a/examples/C++/PaddleNLP/bert/benchmark.py b/examples/C++/PaddleNLP/bert/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..bdef982830cea34f5a9ea925e6759b48b86ce7a7
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/benchmark.py
@@ -0,0 +1,144 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import unicode_literals, absolute_import
+import os
+import sys
+import time
+import json
+import requests
+import numpy as np
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+from paddle_serving_app.reader import ChineseBertReader
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ fin = open("data-c.txt")
+ dataset = []
+ for line in fin:
+ dataset.append(line.strip())
+
+ profile_flags = False
+ latency_flags = False
+ if os.getenv("FLAGS_profile_client"):
+ profile_flags = True
+ if os.getenv("FLAGS_serving_latency"):
+ latency_flags = True
+ latency_list = []
+
+ if args.request == "rpc":
+ reader = ChineseBertReader({"max_seq_len": 128})
+ fetch = ["pooled_output"]
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
+ start = time.time()
+ for i in range(turns):
+ if args.batch_size >= 1:
+ l_start = time.time()
+ feed_batch = []
+ b_start = time.time()
+ for bi in range(args.batch_size):
+ feed_dict = reader.process(dataset[bi])
+ for key in feed_dict.keys():
+ feed_dict[key] = np.array(feed_dict[key]).reshape(
+ (1, 128, 1))
+ feed_batch.append(feed_dict)
+ b_end = time.time()
+
+ if profile_flags:
+ sys.stderr.write(
+ "PROFILE\tpid:{}\tbert_pre_0:{} bert_pre_1:{}\n".format(
+ os.getpid(),
+ int(round(b_start * 1000000)),
+ int(round(b_end * 1000000))))
+ result = client.predict(
+ feed=feed_batch, fetch=fetch, batch=True)
+
+ l_end = time.time()
+ if latency_flags:
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ elif args.request == "http":
+ reader = ChineseBertReader({"max_seq_len": 128})
+ fetch = ["pooled_output"]
+ server = "http://" + resource["endpoint"][idx % len(resource[
+ "endpoint"])] + "/bert/prediction"
+ start = time.time()
+ for i in range(turns):
+ if args.batch_size >= 1:
+ l_start = time.time()
+ feed_batch = []
+ b_start = time.time()
+ for bi in range(args.batch_size):
+ feed_batch.append({"words": dataset[bi]})
+ req = json.dumps({"feed": feed_batch, "fetch": fetch})
+ b_end = time.time()
+
+ if profile_flags:
+ sys.stderr.write(
+ "PROFILE\tpid:{}\tbert_pre_0:{} bert_pre_1:{}\n".format(
+ os.getpid(),
+ int(round(b_start * 1000000)),
+ int(round(b_end * 1000000))))
+ result = requests.post(
+ server,
+ data=req,
+ headers={"Content-Type": "application/json"})
+ l_end = time.time()
+ if latency_flags:
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ else:
+ raise ValueError("not implemented {} request".format(args.request))
+ end = time.time()
+ if latency_flags:
+ return [[end - start], latency_list]
+ else:
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = ["127.0.0.1:9292", "127.0.0.1:9293"]
+ turns = 100
+ start = time.time()
+ result = multi_thread_runner.run(
+ single_func, args.thread, {"endpoint": endpoint_list,
+ "turns": turns})
+ end = time.time()
+ total_cost = end - start
+
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ avg_cost = avg_cost / args.thread
+
+ print("total cost: {}s".format(total_cost))
+ print("each thread cost: {}s. ".format(avg_cost))
+ print("qps: {}samples/s".format(args.batch_size * args.thread * turns /
+ total_cost))
+ if os.getenv("FLAGS_serving_latency"):
+ show_latency(result[1])
diff --git a/examples/C++/PaddleNLP/bert/benchmark.sh b/examples/C++/PaddleNLP/bert/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..a898ef83253dc3130c9847efd2120c84942cc9a7
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/benchmark.sh
@@ -0,0 +1,55 @@
+rm profile_log*
+export CUDA_VISIBLE_DEVICES=0,1
+export FLAGS_profile_server=1
+export FLAGS_profile_client=1
+export FLAGS_serving_latency=1
+
+gpu_id=0
+#save cpu and gpu utilization log
+if [ -d utilization ];then
+ rm -rf utilization
+else
+ mkdir utilization
+fi
+#start server
+$PYTHONROOT/bin/python3 -m paddle_serving_server.serve --model $1 --port 9292 --thread 4 --gpu_ids 0,1 --mem_optim --ir_optim > elog 2>&1 &
+sleep 5
+
+#warm up
+$PYTHONROOT/bin/python3 benchmark.py --thread 4 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+echo -e "import psutil\nimport time\nwhile True:\n\tcpu_res = psutil.cpu_percent()\n\twith open('cpu.txt', 'a+') as f:\n\t\tf.write(f'{cpu_res}\\\n')\n\ttime.sleep(0.1)" > cpu.py
+for thread_num in 1 4 8 16
+do
+for batch_size in 1 4 16 64
+do
+ job_bt=`date '+%Y%m%d%H%M%S'`
+ nvidia-smi --id=0 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_memory_use.log 2>&1 &
+ nvidia-smi --id=0 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ rm -rf cpu.txt
+ $PYTHONROOT/bin/python3 cpu.py &
+ gpu_memory_pid=$!
+ $PYTHONROOT/bin/python3 benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+ kill `ps -ef|grep used_memory|awk '{print $2}'` > /dev/null
+ kill `ps -ef|grep utilization.gpu|awk '{print $2}'` > /dev/null
+ kill `ps -ef|grep cpu.py|awk '{print $2}'` > /dev/null
+ echo "model_name:" $1
+ echo "thread_num:" $thread_num
+ echo "batch_size:" $batch_size
+ echo "=================Done===================="
+ echo "model_name:$1" >> profile_log_$1
+ echo "batch_size:$batch_size" >> profile_log_$1
+ job_et=`date '+%Y%m%d%H%M%S'`
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "CPU_UTILIZATION:", max}' cpu.txt >> profile_log_$1
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_memory_use.log >> profile_log_$1
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$1
+ rm -rf gpu_use.log gpu_utilization.log
+ $PYTHONROOT/bin/python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$1
+ tail -n 8 profile >> profile_log_$1
+ echo "" >> profile_log_$1
+done
+done
+
+#Divided log
+awk 'BEGIN{RS="\n\n"}{i++}{print > "bert_log_"i}' profile_log_$1
+mkdir bert_log && mv bert_log_* bert_log
+ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
diff --git a/examples/C++/PaddleNLP/bert/benchmark_with_profile.sh b/examples/C++/PaddleNLP/bert/benchmark_with_profile.sh
new file mode 100644
index 0000000000000000000000000000000000000000..f36fbbce917d2956195d08e7638e06d84caf961a
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/benchmark_with_profile.sh
@@ -0,0 +1,10 @@
+export CUDA_VISIBLE_DEVICES=0,1
+python -m paddle_serving_server.serve --model bert_seq20_model/ --port 9295 --thread 4 --gpu_ids 0,1 2> elog > stdlog &
+export FLAGS_profile_client=1
+export FLAGS_profile_server=1
+sleep 5
+thread_num=4
+python benchmark_batch.py --thread ${thread_num} --batch_size 64 --model serving_client_conf/serving_client_conf.prototxt 2> profile
+
+python show_profile.py profile ${thread_num}
+python timeline_trace.py profile trace
diff --git a/examples/C++/PaddleNLP/bert/bert_client.py b/examples/C++/PaddleNLP/bert/bert_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..4111589b3ddfde980e415fbac1a5f38f4abafada
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/bert_client.py
@@ -0,0 +1,37 @@
+# coding:utf-8
+# pylint: disable=doc-string-missing
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import Client
+from paddle_serving_client.utils import benchmark_args
+from paddle_serving_app.reader import ChineseBertReader
+import numpy as np
+args = benchmark_args()
+
+reader = ChineseBertReader({"max_seq_len": 128})
+fetch = ["pooled_output"]
+endpoint_list = ['127.0.0.1:9292']
+client = Client()
+client.load_client_config(args.model)
+client.connect(endpoint_list)
+
+for line in sys.stdin:
+ feed_dict = reader.process(line)
+ for key in feed_dict.keys():
+ feed_dict[key] = np.array(feed_dict[key]).reshape((128, 1))
+ #print(feed_dict)
+ result = client.predict(feed=feed_dict, fetch=fetch, batch=False)
+print(result)
diff --git a/examples/C++/PaddleNLP/bert/bert_gpu_server.py b/examples/C++/PaddleNLP/bert/bert_gpu_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..7708a078636fd876c40e88d1441bc711d599f8a6
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/bert_gpu_server.py
@@ -0,0 +1,42 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+from paddle_serving_server import OpMaker
+from paddle_serving_server import OpSeqMaker
+from paddle_serving_server import Server
+
+op_maker = OpMaker()
+read_op = op_maker.create('general_reader')
+general_infer_op = op_maker.create('general_infer')
+general_response_op = op_maker.create('general_response')
+
+op_seq_maker = OpSeqMaker()
+op_seq_maker.add_op(read_op)
+op_seq_maker.add_op(general_infer_op)
+op_seq_maker.add_op(general_response_op)
+
+server = Server()
+server.set_op_sequence(op_seq_maker.get_op_sequence())
+server.set_num_threads(8)
+server.set_memory_optimize(True)
+server.set_gpuid(1)
+
+server.load_model_config(sys.argv[1])
+port = int(sys.argv[2])
+gpuid = sys.argv[3]
+server.set_gpuid(gpuid)
+server.prepare_server(workdir="work_dir1", port=port, device="gpu")
+server.run_server()
diff --git a/examples/C++/PaddleNLP/bert/bert_httpclient.py b/examples/C++/PaddleNLP/bert/bert_httpclient.py
new file mode 100644
index 0000000000000000000000000000000000000000..255c78ec0ca7e33ddd1486f05cf6d9d225a5f406
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/bert_httpclient.py
@@ -0,0 +1,58 @@
+# coding:utf-8
+# pylint: disable=doc-string-missing
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import HttpClient
+from paddle_serving_client.utils import benchmark_args
+from paddle_serving_app.reader import ChineseBertReader
+import numpy as np
+args = benchmark_args()
+
+reader = ChineseBertReader({"max_seq_len": 128})
+fetch = ["pooled_output"]
+endpoint_list = ['127.0.0.1:9292']
+client = HttpClient()
+client.load_client_config(args.model)
+'''
+if you want use GRPC-client, set_use_grpc_client(True)
+or you can directly use client.grpc_client_predict(...)
+as for HTTP-client,set_use_grpc_client(False)(which is default)
+or you can directly use client.http_client_predict(...)
+'''
+#client.set_use_grpc_client(True)
+'''
+if you want to enable Encrypt Module,uncommenting the following line
+'''
+#client.use_key("./key")
+'''
+if you want to compress,uncommenting the following line
+'''
+#client.set_response_compress(True)
+#client.set_request_compress(True)
+'''
+we recommend use Proto data format in HTTP-body, set True(which is default)
+if you want use JSON data format in HTTP-body, set False
+'''
+#client.set_http_proto(True)
+client.connect(endpoint_list)
+
+for line in sys.stdin:
+ feed_dict = reader.process(line)
+ for key in feed_dict.keys():
+ feed_dict[key] = np.array(feed_dict[key]).reshape((128, 1))
+ #print(feed_dict)
+ result = client.predict(feed=feed_dict, fetch=fetch, batch=False)
+print(result)
diff --git a/examples/C++/PaddleNLP/bert/bert_reader.py b/examples/C++/PaddleNLP/bert/bert_reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..366c19b5dc7de7e14979124272329b8ba00fdb3c
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/bert_reader.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+from batching import pad_batch_data
+import tokenization
+
+
+class BertReader():
+ def __init__(self, vocab_file="", max_seq_len=128):
+ self.vocab_file = vocab_file
+ self.tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file)
+ self.max_seq_len = max_seq_len
+ self.vocab = self.tokenizer.vocab
+ self.pad_id = self.vocab["[PAD]"]
+ self.cls_id = self.vocab["[CLS]"]
+ self.sep_id = self.vocab["[SEP]"]
+ self.mask_id = self.vocab["[MASK]"]
+
+ def pad_batch(self, token_ids, text_type_ids, position_ids):
+ batch_token_ids = [token_ids]
+ batch_text_type_ids = [text_type_ids]
+ batch_position_ids = [position_ids]
+
+ padded_token_ids, input_mask = pad_batch_data(
+ batch_token_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id,
+ return_input_mask=True)
+ padded_text_type_ids = pad_batch_data(
+ batch_text_type_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id)
+ padded_position_ids = pad_batch_data(
+ batch_position_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id)
+ return padded_token_ids, padded_position_ids, padded_text_type_ids, input_mask
+
+ def process(self, sent):
+ text_a = tokenization.convert_to_unicode(sent)
+ tokens_a = self.tokenizer.tokenize(text_a)
+ if len(tokens_a) > self.max_seq_len - 2:
+ tokens_a = tokens_a[0:(self.max_seq_len - 2)]
+ tokens = []
+ text_type_ids = []
+ tokens.append("[CLS]")
+ text_type_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ text_type_ids.append(0)
+ token_ids = self.tokenizer.convert_tokens_to_ids(tokens)
+ position_ids = list(range(len(token_ids)))
+ p_token_ids, p_pos_ids, p_text_type_ids, input_mask = \
+ self.pad_batch(token_ids, text_type_ids, position_ids)
+ feed_result = {
+ "input_ids": p_token_ids.reshape(-1).tolist(),
+ "position_ids": p_pos_ids.reshape(-1).tolist(),
+ "segment_ids": p_text_type_ids.reshape(-1).tolist(),
+ "input_mask": input_mask.reshape(-1).tolist()
+ }
+ return feed_result
diff --git a/examples/C++/PaddleNLP/bert/bert_server.py b/examples/C++/PaddleNLP/bert/bert_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..35d38be0cac50b899b58085c7f103f32537859c4
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/bert_server.py
@@ -0,0 +1,38 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+from paddle_serving_server import OpMaker
+from paddle_serving_server import OpSeqMaker
+from paddle_serving_server import Server
+
+op_maker = OpMaker()
+read_op = op_maker.create('general_reader')
+general_infer_op = op_maker.create('general_infer')
+general_response_op = op_maker.create('general_response')
+
+op_seq_maker = OpSeqMaker()
+op_seq_maker.add_op(read_op)
+op_seq_maker.add_op(general_infer_op)
+op_seq_maker.add_op(general_response_op)
+
+server = Server()
+server.set_op_sequence(op_seq_maker.get_op_sequence())
+server.set_num_threads(4)
+
+server.load_model_config(sys.argv[1])
+port = int(sys.argv[2])
+server.prepare_server(workdir="work_dir1", port=port, device="cpu")
+server.run_server()
diff --git a/examples/C++/PaddleNLP/bert/get_data.sh b/examples/C++/PaddleNLP/bert/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5e17d10d144c5c9e202a5cb9c4b90a62caeeed9a
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/get_data.sh
@@ -0,0 +1,2 @@
+wget https://paddle-serving.bj.bcebos.com/bert_example/data-c.txt --no-check-certificate
+wget https://paddle-serving.bj.bcebos.com/bert_example/vocab.txt --no-check-certificate
diff --git a/examples/C++/PaddleNLP/bert/prepare_model.py b/examples/C++/PaddleNLP/bert/prepare_model.py
new file mode 100644
index 0000000000000000000000000000000000000000..e883b6b15746946a7f3412fe64c5933c4cfb37ab
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/prepare_model.py
@@ -0,0 +1,49 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+import paddlehub as hub
+import paddle.fluid as fluid
+import sys
+import paddle_serving_client.io as serving_io
+import paddle
+
+paddle.enable_static()
+model_name = "bert_chinese_L-12_H-768_A-12"
+module = hub.Module(name=model_name)
+inputs, outputs, program = module.context(
+ trainable=True, max_seq_len=int(sys.argv[1]))
+place = fluid.core_avx.CPUPlace()
+exe = fluid.Executor(place)
+input_ids = inputs["input_ids"]
+position_ids = inputs["position_ids"]
+segment_ids = inputs["segment_ids"]
+input_mask = inputs["input_mask"]
+pooled_output = outputs["pooled_output"]
+sequence_output = outputs["sequence_output"]
+
+feed_var_names = [
+ input_ids.name, position_ids.name, segment_ids.name, input_mask.name
+]
+
+target_vars = [pooled_output, sequence_output]
+
+serving_io.save_model(
+ "bert_seq{}_model".format(sys.argv[1]),
+ "bert_seq{}_client".format(sys.argv[1]), {
+ "input_ids": input_ids,
+ "position_ids": position_ids,
+ "segment_ids": segment_ids,
+ "input_mask": input_mask,
+ }, {"pooled_output": pooled_output,
+ "sequence_output": sequence_output}, program)
diff --git a/examples/C++/PaddleNLP/bert/test_multi_fetch_client.py b/examples/C++/PaddleNLP/bert/test_multi_fetch_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..1ee540097c32429348fbeb504278fb986bd3a9e7
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/test_multi_fetch_client.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import ChineseBertReader
+import sys
+import numpy as np
+
+client = Client()
+client.load_client_config("./bert_seq32_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9292"])
+
+reader = ChineseBertReader({"max_seq_len": 32})
+fetch = ["sequence_10", "sequence_12", "pooled_output"]
+expected_shape = {
+ "sequence_10": (4, 32, 768),
+ "sequence_12": (4, 32, 768),
+ "pooled_output": (4, 768)
+}
+batch_size = 4
+feed_batch = {}
+
+batch_len = 0
+for line in sys.stdin:
+ feed = reader.process(line)
+ if batch_len == 0:
+ for key in feed.keys():
+ val_len = len(feed[key])
+ feed_batch[key] = np.array(feed[key]).reshape((1, val_len, 1))
+ continue
+ if len(feed_batch) < batch_size:
+ for key in feed.keys():
+ np.concatenate([
+ feed_batch[key], np.array(feed[key]).reshape((1, val_len, 1))
+ ])
+ else:
+ fetch_map = client.predict(feed=feed_batch, fetch=fetch)
+ feed_batch = []
+ for var_name in fetch:
+ if fetch_map[var_name].shape != expected_shape[var_name]:
+ print("fetch var {} shape error.".format(var_name))
+ sys.exit(1)
diff --git a/examples/C++/PaddleNLP/bert/tokenization.py b/examples/C++/PaddleNLP/bert/tokenization.py
new file mode 100644
index 0000000000000000000000000000000000000000..0d84ed38468207e853e5270a59179b4274900cb0
--- /dev/null
+++ b/examples/C++/PaddleNLP/bert/tokenization.py
@@ -0,0 +1,441 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import io
+import unicodedata
+import six
+import sentencepiece as spm
+import pickle
+
+
+def convert_to_unicode(text): # pylint: disable=doc-string-with-all-args
+ """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
+ if six.PY3:
+ if isinstance(text, str):
+ return text
+ elif isinstance(text, bytes):
+ return text.decode("utf-8", "ignore")
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ elif six.PY2:
+ if isinstance(text, str):
+ return text.decode("utf-8", "ignore")
+ elif isinstance(text, unicode): # noqa
+ return text
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ else:
+ raise ValueError("Not running on Python2 or Python 3?")
+
+
+def printable_text(text): # pylint: disable=doc-string-with-all-args
+ """Returns text encoded in a way suitable for print or `tf.logging`."""
+
+ # These functions want `str` for both Python2 and Python3, but in one case
+ # it's a Unicode string and in the other it's a byte string.
+ if six.PY3:
+ if isinstance(text, str):
+ return text
+ elif isinstance(text, bytes):
+ return text.decode("utf-8", "ignore")
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ elif six.PY2:
+ if isinstance(text, str):
+ return text
+ elif isinstance(text, unicode): # noqa
+ return text.encode("utf-8")
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ else:
+ raise ValueError("Not running on Python2 or Python 3?")
+
+
+def load_vocab(vocab_file): # pylint: disable=doc-string-with-all-args, doc-string-with-returns
+ """Loads a vocabulary file into a dictionary."""
+ vocab = collections.OrderedDict()
+ fin = io.open(vocab_file, "r", encoding="UTF-8")
+ for num, line in enumerate(fin):
+ items = convert_to_unicode(line.strip()).split("\t")
+ if len(items) > 2:
+ break
+ token = items[0]
+ index = items[1] if len(items) == 2 else num
+ token = token.strip()
+ vocab[token] = int(index)
+ fin.close()
+ return vocab
+
+
+def convert_by_vocab(vocab, items):
+ """Converts a sequence of [tokens|ids] using the vocab."""
+ output = []
+ for item in items:
+ output.append(vocab[item])
+ return output
+
+
+def convert_tokens_to_ids(vocab, tokens):
+ return convert_by_vocab(vocab, tokens)
+
+
+def convert_ids_to_tokens(inv_vocab, ids):
+ return convert_by_vocab(inv_vocab, ids)
+
+
+def whitespace_tokenize(text):
+ """Runs basic whitespace cleaning and splitting on a peice of text."""
+ text = text.strip()
+ if not text:
+ return []
+ tokens = text.split()
+ return tokens
+
+
+class FullTokenizer(object):
+ """Runs end-to-end tokenziation."""
+
+ def __init__(self,
+ vocab_file,
+ do_lower_case=True,
+ use_sentence_piece_vocab=False):
+ self.vocab = load_vocab(vocab_file)
+ self.inv_vocab = {v: k for k, v in self.vocab.items()}
+ self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
+ self.use_sentence_piece_vocab = use_sentence_piece_vocab
+ self.wordpiece_tokenizer = WordpieceTokenizer(
+ vocab=self.vocab,
+ use_sentence_piece_vocab=self.use_sentence_piece_vocab)
+
+ def tokenize(self, text):
+ split_tokens = []
+ for token in self.basic_tokenizer.tokenize(text):
+ for sub_token in self.wordpiece_tokenizer.tokenize(token):
+ split_tokens.append(sub_token)
+
+ return split_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ return convert_by_vocab(self.vocab, tokens)
+
+ def convert_ids_to_tokens(self, ids):
+ return convert_by_vocab(self.inv_vocab, ids)
+
+
+class CharTokenizer(object):
+ """Runs end-to-end tokenziation."""
+
+ def __init__(self, vocab_file, do_lower_case=True):
+ self.vocab = load_vocab(vocab_file)
+ self.inv_vocab = {v: k for k, v in self.vocab.items()}
+ self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
+
+ def tokenize(self, text):
+ split_tokens = []
+ for token in text.lower().split(" "):
+ for sub_token in self.wordpiece_tokenizer.tokenize(token):
+ split_tokens.append(sub_token)
+
+ return split_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ return convert_by_vocab(self.vocab, tokens)
+
+ def convert_ids_to_tokens(self, ids):
+ return convert_by_vocab(self.inv_vocab, ids)
+
+
+class WSSPTokenizer(object): # pylint: disable=doc-string-missing
+ def __init__(self, vocab_file, sp_model_dir, word_dict, ws=True,
+ lower=True):
+ self.vocab = load_vocab(vocab_file)
+ self.inv_vocab = {v: k for k, v in self.vocab.items()}
+ self.ws = ws
+ self.lower = lower
+ self.dict = pickle.load(open(word_dict, 'rb'))
+ self.sp_model = spm.SentencePieceProcessor()
+ self.window_size = 5
+ self.sp_model.Load(sp_model_dir)
+
+ def cut(self, chars): # pylint: disable=doc-string-missing
+ words = []
+ idx = 0
+ while idx < len(chars):
+ matched = False
+ for i in range(self.window_size, 0, -1):
+ cand = chars[idx:idx + i]
+ if cand in self.dict:
+ words.append(cand)
+ matched = True
+ break
+ if not matched:
+ i = 1
+ words.append(chars[idx])
+ idx += i
+ return words
+
+ def tokenize(self, text, unk_token="[UNK]"): # pylint: disable=doc-string-missing
+ text = convert_to_unicode(text)
+ if self.ws:
+ text = [s for s in self.cut(text) if s != ' ']
+ else:
+ text = text.split(' ')
+ if self.lower:
+ text = [s.lower() for s in text]
+ text = ' '.join(text)
+ tokens = self.sp_model.EncodeAsPieces(text)
+ in_vocab_tokens = []
+ for token in tokens:
+ if token in self.vocab:
+ in_vocab_tokens.append(token)
+ else:
+ in_vocab_tokens.append(unk_token)
+ return in_vocab_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ return convert_by_vocab(self.vocab, tokens)
+
+ def convert_ids_to_tokens(self, ids):
+ return convert_by_vocab(self.inv_vocab, ids)
+
+
+class BasicTokenizer(object):
+ """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
+
+ def __init__(self, do_lower_case=True):
+ """Constructs a BasicTokenizer.
+
+ Args:
+ do_lower_case: Whether to lower case the input.
+ """
+ self.do_lower_case = do_lower_case
+
+ def tokenize(self, text): # pylint: disable=doc-string-with-all-args, doc-string-with-returns
+ """Tokenizes a piece of text."""
+ text = convert_to_unicode(text)
+ text = self._clean_text(text)
+
+ # This was added on November 1st, 2018 for the multilingual and Chinese
+ # models. This is also applied to the English models now, but it doesn't
+ # matter since the English models were not trained on any Chinese data
+ # and generally don't have any Chinese data in them (there are Chinese
+ # characters in the vocabulary because Wikipedia does have some Chinese
+ # words in the English Wikipedia.).
+ text = self._tokenize_chinese_chars(text)
+
+ orig_tokens = whitespace_tokenize(text)
+ split_tokens = []
+ for token in orig_tokens:
+ if self.do_lower_case:
+ token = token.lower()
+ token = self._run_strip_accents(token)
+ split_tokens.extend(self._run_split_on_punc(token))
+
+ output_tokens = whitespace_tokenize(" ".join(split_tokens))
+ return output_tokens
+
+ def _run_strip_accents(self, text):
+ """Strips accents from a piece of text."""
+ text = unicodedata.normalize("NFD", text)
+ output = []
+ for char in text:
+ cat = unicodedata.category(char)
+ if cat == "Mn":
+ continue
+ output.append(char)
+ return "".join(output)
+
+ def _run_split_on_punc(self, text):
+ """Splits punctuation on a piece of text."""
+ chars = list(text)
+ i = 0
+ start_new_word = True
+ output = []
+ while i < len(chars):
+ char = chars[i]
+ if _is_punctuation(char):
+ output.append([char])
+ start_new_word = True
+ else:
+ if start_new_word:
+ output.append([])
+ start_new_word = False
+ output[-1].append(char)
+ i += 1
+
+ return ["".join(x) for x in output]
+
+ def _tokenize_chinese_chars(self, text):
+ """Adds whitespace around any CJK character."""
+ output = []
+ for char in text:
+ cp = ord(char)
+ if self._is_chinese_char(cp):
+ output.append(" ")
+ output.append(char)
+ output.append(" ")
+ else:
+ output.append(char)
+ return "".join(output)
+
+ def _is_chinese_char(self, cp):
+ """Checks whether CP is the codepoint of a CJK character."""
+ # This defines a "chinese character" as anything in the CJK Unicode block:
+ # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
+ #
+ # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
+ # despite its name. The modern Korean Hangul alphabet is a different block,
+ # as is Japanese Hiragana and Katakana. Those alphabets are used to write
+ # space-separated words, so they are not treated specially and handled
+ # like the all of the other languages.
+ if ((cp >= 0x4E00 and cp <= 0x9FFF) or #
+ (cp >= 0x3400 and cp <= 0x4DBF) or #
+ (cp >= 0x20000 and cp <= 0x2A6DF) or #
+ (cp >= 0x2A700 and cp <= 0x2B73F) or #
+ (cp >= 0x2B740 and cp <= 0x2B81F) or #
+ (cp >= 0x2B820 and cp <= 0x2CEAF) or
+ (cp >= 0xF900 and cp <= 0xFAFF) or #
+ (cp >= 0x2F800 and cp <= 0x2FA1F)): #
+ return True
+
+ return False
+
+ def _clean_text(self, text):
+ """Performs invalid character removal and whitespace cleanup on text."""
+ output = []
+ for char in text:
+ cp = ord(char)
+ if cp == 0 or cp == 0xfffd or _is_control(char):
+ continue
+ if _is_whitespace(char):
+ output.append(" ")
+ else:
+ output.append(char)
+ return "".join(output)
+
+
+class WordpieceTokenizer(object):
+ """Runs WordPiece tokenziation."""
+
+ def __init__(self,
+ vocab,
+ unk_token="[UNK]",
+ max_input_chars_per_word=100,
+ use_sentence_piece_vocab=False):
+ self.vocab = vocab
+ self.unk_token = unk_token
+ self.max_input_chars_per_word = max_input_chars_per_word
+ self.use_sentence_piece_vocab = use_sentence_piece_vocab
+
+ def tokenize(self, text): # pylint: disable=doc-string-with-all-args
+ """Tokenizes a piece of text into its word pieces.
+
+ This uses a greedy longest-match-first algorithm to perform tokenization
+ using the given vocabulary.
+
+ For example:
+ input = "unaffable"
+ output = ["un", "##aff", "##able"]
+
+ Args:
+ text: A single token or whitespace separated tokens. This should have
+ already been passed through `BasicTokenizer.
+
+ Returns:
+ A list of wordpiece tokens.
+ """
+
+ text = convert_to_unicode(text)
+
+ output_tokens = []
+ for token in whitespace_tokenize(text):
+ chars = list(token)
+ if len(chars) > self.max_input_chars_per_word:
+ output_tokens.append(self.unk_token)
+ continue
+
+ is_bad = False
+ start = 0
+ sub_tokens = []
+ while start < len(chars):
+ end = len(chars)
+ cur_substr = None
+ while start < end:
+ substr = "".join(chars[start:end])
+ if start == 0 and self.use_sentence_piece_vocab:
+ substr = u'\u2581' + substr
+ if start > 0 and not self.use_sentence_piece_vocab:
+ substr = "##" + substr
+ if substr in self.vocab:
+ cur_substr = substr
+ break
+ end -= 1
+ if cur_substr is None:
+ is_bad = True
+ break
+ sub_tokens.append(cur_substr)
+ start = end
+
+ if is_bad:
+ output_tokens.append(self.unk_token)
+ else:
+ output_tokens.extend(sub_tokens)
+ return output_tokens
+
+
+def _is_whitespace(char):
+ """Checks whether `chars` is a whitespace character."""
+ # \t, \n, and \r are technically contorl characters but we treat them
+ # as whitespace since they are generally considered as such.
+ if char == " " or char == "\t" or char == "\n" or char == "\r":
+ return True
+ cat = unicodedata.category(char)
+ if cat == "Zs":
+ return True
+ return False
+
+
+def _is_control(char):
+ """Checks whether `chars` is a control character."""
+ # These are technically control characters but we count them as whitespace
+ # characters.
+ if char == "\t" or char == "\n" or char == "\r":
+ return False
+ cat = unicodedata.category(char)
+ if cat.startswith("C"):
+ return True
+ return False
+
+
+def _is_punctuation(char):
+ """Checks whether `chars` is a punctuation character."""
+ cp = ord(char)
+ # We treat all non-letter/number ASCII as punctuation.
+ # Characters such as "^", "$", and "`" are not in the Unicode
+ # Punctuation class but we treat them as punctuation anyways, for
+ # consistency.
+ if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
+ (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
+ return True
+ cat = unicodedata.category(char)
+ if cat.startswith("P"):
+ return True
+ return False
diff --git a/examples/C++/PaddleNLP/lac/README.md b/examples/C++/PaddleNLP/lac/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..108d5051b50b2b639e28c023364d36ec9a0a0a44
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/README.md
@@ -0,0 +1,26 @@
+## Chinese Word Segmentation
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+python3 -m paddle_serving_app.package --get_model lac
+tar -xzvf lac.tar.gz
+```
+
+#### Start inference service(Support BRPC-Client/GRPC-Client/Http-Client)
+
+```
+python3 -m paddle_serving_server.serve --model lac_model/ --port 9292
+```
+### BRPC Infer
+```
+echo "我爱北京天安门" | python3 lac_client.py lac_client/serving_client_conf.prototxt
+```
+
+It will get the segmentation result.
+
+### GRPC/Http Infer
+```
+echo "我爱北京天安门" | python3 lac_http_client.py lac_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/PaddleNLP/lac/README_CN.md b/examples/C++/PaddleNLP/lac/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..5634128c80c23126836677f4cb434df68dde9056
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/README_CN.md
@@ -0,0 +1,26 @@
+## 中文分词模型
+
+(简体中文|[English](./README.md))
+
+### 获取模型
+```
+python3 -m paddle_serving_app.package --get_model lac
+tar -xzvf lac.tar.gz
+```
+
+#### 开启预测服务(支持BRPC-Client/GRPC-Client/Http-Client)
+
+```
+python3 -m paddle_serving_server.serve --model lac_model/ --port 9292
+```
+### 执行BRPC预测
+```
+echo "我爱北京天安门" | python3 lac_client.py lac_client/serving_client_conf.prototxt
+```
+
+我们就能得到分词结果
+
+### 执行GRPC/Http预测
+```
+echo "我爱北京天安门" | python3 lac_http_client.py lac_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/PaddleNLP/lac/benchmark.py b/examples/C++/PaddleNLP/lac/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..64e935a608477d5841df1b64abf7b6eb35dd1a4b
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/benchmark.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import sys
+import time
+import requests
+from paddle_serving_app.reader import LACReader
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ reader = LACReader()
+ start = time.time()
+ if args.request == "rpc":
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([args.endpoint])
+ fin = open("jieba_test.txt")
+ for line in fin:
+ feed_data = reader.process(line)
+ fetch_map = client.predict(
+ feed={"words": feed_data}, fetch=["crf_decode"])
+ elif args.request == "http":
+ fin = open("jieba_test.txt")
+ for line in fin:
+ req_data = {"words": line.strip(), "fetch": ["crf_decode"]}
+ r = requests.post(
+ "http://{}/lac/prediction".format(args.endpoint),
+ data={"words": line.strip(),
+ "fetch": ["crf_decode"]})
+ end = time.time()
+ return [[end - start]]
+
+
+multi_thread_runner = MultiThreadRunner()
+result = multi_thread_runner.run(single_func, args.thread, {})
+print(result)
diff --git a/examples/C++/PaddleNLP/lac/lac_client.py b/examples/C++/PaddleNLP/lac/lac_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..568b08d8b3af86fd7aa7b20660aeb4acbf060e04
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/lac_client.py
@@ -0,0 +1,49 @@
+# encoding=utf-8
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import LACReader
+import sys
+import os
+import io
+import numpy as np
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9292"])
+
+reader = LACReader()
+for line in sys.stdin:
+ if len(line) <= 0:
+ continue
+ feed_data = reader.process(line)
+ if len(feed_data) <= 0:
+ continue
+ print(feed_data)
+ #fetch_map = client.predict(feed={"words": np.array(feed_data).reshape(len(feed_data), 1), "words.lod": [0, len(feed_data)]}, fetch=["crf_decode"], batch=True)
+ fetch_map = client.predict(
+ feed={
+ "words": np.array(feed_data + feed_data).reshape(
+ len(feed_data) * 2, 1),
+ "words.lod": [0, len(feed_data), 2 * len(feed_data)]
+ },
+ fetch=["crf_decode"],
+ batch=True)
+ print(fetch_map)
+ begin = fetch_map['crf_decode.lod'][0]
+ end = fetch_map['crf_decode.lod'][1]
+ segs = reader.parse_result(line, fetch_map["crf_decode"][begin:end])
+ print("word_seg: " + "|".join(str(words) for words in segs))
diff --git a/examples/C++/PaddleNLP/lac/lac_http_client.py b/examples/C++/PaddleNLP/lac/lac_http_client.py
new file mode 100755
index 0000000000000000000000000000000000000000..5cdfaf1df46a43d04b7e09f0f6376364a9dcb89f
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/lac_http_client.py
@@ -0,0 +1,66 @@
+# encoding=utf-8
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import HttpClient
+from paddle_serving_app.reader import LACReader
+import sys
+import os
+import io
+import numpy as np
+
+client = HttpClient()
+client.load_client_config(sys.argv[1])
+'''
+if you want use GRPC-client, set_use_grpc_client(True)
+or you can directly use client.grpc_client_predict(...)
+as for HTTP-client,set_use_grpc_client(False)(which is default)
+or you can directly use client.http_client_predict(...)
+'''
+#client.set_use_grpc_client(True)
+'''
+if you want to enable Encrypt Module,uncommenting the following line
+'''
+#client.use_key("./key")
+'''
+if you want to compress,uncommenting the following line
+'''
+#client.set_response_compress(True)
+#client.set_request_compress(True)
+'''
+we recommend use Proto data format in HTTP-body, set True(which is default)
+if you want use JSON data format in HTTP-body, set False
+'''
+#client.set_http_proto(True)
+client.connect(["127.0.0.1:9292"])
+
+reader = LACReader()
+for line in sys.stdin:
+ if len(line) <= 0:
+ continue
+ feed_data = reader.process(line)
+ if len(feed_data) <= 0:
+ continue
+ print(feed_data)
+ #fetch_map = client.predict(feed={"words": np.array(feed_data).reshape(len(feed_data), 1), "words.lod": [0, len(feed_data)]}, fetch=["crf_decode"], batch=True)
+ fetch_map = client.predict(
+ feed={
+ "words": np.array(feed_data + feed_data).reshape(
+ len(feed_data) * 2, 1),
+ "words.lod": [0, len(feed_data), 2 * len(feed_data)]
+ },
+ fetch=["crf_decode"],
+ batch=True)
+ print(fetch_map)
diff --git a/examples/C++/PaddleNLP/lac/lac_reader.py b/examples/C++/PaddleNLP/lac/lac_reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..488e7ced1ce27f914f299c45295e82f33c68d6d0
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/lac_reader.py
@@ -0,0 +1,126 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+import sys
+py_version = sys.version_info[0]
+if py_version == 2:
+ reload(sys)
+ sys.setdefaultencoding('utf-8')
+import os
+import io
+
+
+def load_kv_dict(dict_path,
+ reverse=False,
+ delimiter="\t",
+ key_func=None,
+ value_func=None):
+ result_dict = {}
+ for line in io.open(dict_path, "r", encoding="utf8"):
+ terms = line.strip("\n").split(delimiter)
+ if len(terms) != 2:
+ continue
+ if reverse:
+ value, key = terms
+ else:
+ key, value = terms
+ if key in result_dict:
+ raise KeyError("key duplicated with [%s]" % (key))
+ if key_func:
+ key = key_func(key)
+ if value_func:
+ value = value_func(value)
+ result_dict[key] = value
+ return result_dict
+
+
+class LACReader(object):
+ """data reader"""
+
+ def __init__(self, dict_folder):
+ # read dict
+ #basepath = os.path.abspath(__file__)
+ #folder = os.path.dirname(basepath)
+ word_dict_path = os.path.join(dict_folder, "word.dic")
+ label_dict_path = os.path.join(dict_folder, "tag.dic")
+ self.word2id_dict = load_kv_dict(
+ word_dict_path, reverse=True, value_func=int)
+ self.id2word_dict = load_kv_dict(word_dict_path)
+ self.label2id_dict = load_kv_dict(
+ label_dict_path, reverse=True, value_func=int)
+ self.id2label_dict = load_kv_dict(label_dict_path)
+
+ @property
+ def vocab_size(self):
+ """vocabulary size"""
+ return max(self.word2id_dict.values()) + 1
+
+ @property
+ def num_labels(self):
+ """num_labels"""
+ return max(self.label2id_dict.values()) + 1
+
+ def word_to_ids(self, words):
+ """convert word to word index"""
+ word_ids = []
+ idx = 0
+ try:
+ words = unicode(words, 'utf-8')
+ except:
+ pass
+ for word in words:
+ if word not in self.word2id_dict:
+ word = "OOV"
+ word_id = self.word2id_dict[word]
+ word_ids.append(word_id)
+ return word_ids
+
+ def label_to_ids(self, labels):
+ """convert label to label index"""
+ label_ids = []
+ for label in labels:
+ if label not in self.label2id_dict:
+ label = "O"
+ label_id = self.label2id_dict[label]
+ label_ids.append(label_id)
+ return label_ids
+
+ def process(self, sent):
+ words = sent.strip()
+ word_ids = self.word_to_ids(words)
+ return word_ids
+
+ def parse_result(self, words, crf_decode):
+ tags = [self.id2label_dict[str(x[0])] for x in crf_decode]
+
+ sent_out = []
+ tags_out = []
+ partial_word = ""
+ for ind, tag in enumerate(tags):
+ if partial_word == "":
+ partial_word = self.id2word_dict[str(words[ind])]
+ tags_out.append(tag.split('-')[0])
+ continue
+ if tag.endswith("-B") or (tag == "O" and tag[ind - 1] != "O"):
+ sent_out.append(partial_word)
+ tags_out.append(tag.split('-')[0])
+ partial_word = self.id2word_dict[str(words[ind])]
+ continue
+ partial_word += self.id2word_dict[str(words[ind])]
+
+ if len(sent_out) < len(tags_out):
+ sent_out.append(partial_word)
+
+ return sent_out
diff --git a/examples/C++/PaddleNLP/lac/utils.py b/examples/C++/PaddleNLP/lac/utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..64602902f362cc847c705a3e18d3e76255961314
--- /dev/null
+++ b/examples/C++/PaddleNLP/lac/utils.py
@@ -0,0 +1,141 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+util tools
+"""
+from __future__ import print_function
+import os
+import sys
+import numpy as np
+import paddle.fluid as fluid
+import io
+
+
+def str2bool(v):
+ """
+ argparse does not support True or False in python
+ """
+ return v.lower() in ("true", "t", "1")
+
+
+def parse_result(words, crf_decode, dataset):
+ """ parse result """
+ offset_list = (crf_decode.lod())[0]
+ words = np.array(words)
+ crf_decode = np.array(crf_decode)
+ batch_size = len(offset_list) - 1
+
+ for sent_index in range(batch_size):
+ begin, end = offset_list[sent_index], offset_list[sent_index + 1]
+ sent = []
+ for id in words[begin:end]:
+ if dataset.id2word_dict[str(id[0])] == 'OOV':
+ sent.append(' ')
+ else:
+ sent.append(dataset.id2word_dict[str(id[0])])
+ tags = [
+ dataset.id2label_dict[str(id[0])] for id in crf_decode[begin:end]
+ ]
+
+ sent_out = []
+ tags_out = []
+ parital_word = ""
+ for ind, tag in enumerate(tags):
+ # for the first word
+ if parital_word == "":
+ parital_word = sent[ind]
+ tags_out.append(tag.split('-')[0])
+ continue
+
+ # for the beginning of word
+ if tag.endswith("-B") or (tag == "O" and tags[ind - 1] != "O"):
+ sent_out.append(parital_word)
+ tags_out.append(tag.split('-')[0])
+ parital_word = sent[ind]
+ continue
+
+ parital_word += sent[ind]
+
+ # append the last word, except for len(tags)=0
+ if len(sent_out) < len(tags_out):
+ sent_out.append(parital_word)
+ return sent_out, tags_out
+
+
+def parse_padding_result(words, crf_decode, seq_lens, dataset):
+ """ parse padding result """
+ words = np.squeeze(words)
+ batch_size = len(seq_lens)
+
+ batch_out = []
+ for sent_index in range(batch_size):
+
+ sent = []
+ for id in words[begin:end]:
+ if dataset.id2word_dict[str(id[0])] == 'OOV':
+ sent.append(' ')
+ else:
+ sent.append(dataset.id2word_dict[str(id[0])])
+ tags = [
+ dataset.id2label_dict[str(id)]
+ for id in crf_decode[sent_index][1:seq_lens[sent_index] - 1]
+ ]
+
+ sent_out = []
+ tags_out = []
+ parital_word = ""
+ for ind, tag in enumerate(tags):
+ # for the first word
+ if parital_word == "":
+ parital_word = sent[ind]
+ tags_out.append(tag.split('-')[0])
+ continue
+
+ # for the beginning of word
+ if tag.endswith("-B") or (tag == "O" and tags[ind - 1] != "O"):
+ sent_out.append(parital_word)
+ tags_out.append(tag.split('-')[0])
+ parital_word = sent[ind]
+ continue
+
+ parital_word += sent[ind]
+
+ # append the last word, except for len(tags)=0
+ if len(sent_out) < len(tags_out):
+ sent_out.append(parital_word)
+
+ batch_out.append([sent_out, tags_out])
+ return batch_out
+
+
+def init_checkpoint(exe, init_checkpoint_path, main_program):
+ """
+ Init CheckPoint
+ """
+ assert os.path.exists(
+ init_checkpoint_path), "[%s] cann't be found." % init_checkpoint_path
+
+ def existed_persitables(var):
+ """
+ If existed presitabels
+ """
+ if not fluid.io.is_persistable(var):
+ return False
+ return os.path.exists(os.path.join(init_checkpoint_path, var.name))
+
+ fluid.io.load_vars(
+ exe,
+ init_checkpoint_path,
+ main_program=main_program,
+ predicate=existed_persitables)
diff --git a/examples/C++/PaddleNLP/senta/README.md b/examples/C++/PaddleNLP/senta/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..9a159133eeb20832c1870bb949136a59ae461901
--- /dev/null
+++ b/examples/C++/PaddleNLP/senta/README.md
@@ -0,0 +1,23 @@
+# Chinese Sentence Sentiment Classification
+([简体中文](./README_CN.md)|English)
+
+## Get Model
+```
+python3 -m paddle_serving_app.package --get_model senta_bilstm
+python3 -m paddle_serving_app.package --get_model lac
+tar -xzvf senta_bilstm.tar.gz
+tar -xzvf lac.tar.gz
+```
+
+## Start HTTP Service
+```
+python3 -m paddle_serving_server.serve --model lac_model --port 9300
+python3 senta_web_service.py
+```
+In the Chinese sentiment classification task, the Chinese word segmentation needs to be done through [LAC task] (../lac).
+In this demo, the LAC task is placed in the preprocessing part of the HTTP prediction service of the sentiment classification task.
+
+## Client prediction
+```
+curl -H "Content-Type:application/json" -X POST -d '{"feed":[{"words": "天气不错"}], "fetch":["class_probs"]}' http://127.0.0.1:9393/senta/prediction
+```
diff --git a/examples/C++/PaddleNLP/senta/README_CN.md b/examples/C++/PaddleNLP/senta/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..a09fd117767cbdd01847d6cdef06992caf4a9715
--- /dev/null
+++ b/examples/C++/PaddleNLP/senta/README_CN.md
@@ -0,0 +1,23 @@
+# 中文语句情感分类
+(简体中文|[English](./README.md))
+
+## 获取模型文件
+```
+python3 -m paddle_serving_app.package --get_model senta_bilstm
+python3 -m paddle_serving_app.package --get_model lac
+tar -xzvf lac.tar.gz
+tar -xzvf senta_bilstm.tar.gz
+```
+
+## 启动HTTP服务
+```
+python3 -m paddle_serving_server.serve --model lac_model --port 9300
+python3 senta_web_service.py
+```
+中文情感分类任务中需要先通过[LAC任务](../lac)进行中文分词。
+示例中将LAC任务放在情感分类任务的HTTP预测服务的预处理部分。
+
+## 客户端预测
+```
+curl -H "Content-Type:application/json" -X POST -d '{"feed":[{"words": "天气不错"}], "fetch":["class_probs"]}' http://127.0.0.1:9393/senta/prediction
+```
diff --git a/examples/C++/PaddleNLP/senta/get_data.sh b/examples/C++/PaddleNLP/senta/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..7fd5c3e21880c44f27c4f4a037be87dc24790bc4
--- /dev/null
+++ b/examples/C++/PaddleNLP/senta/get_data.sh
@@ -0,0 +1,7 @@
+wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SentimentAnalysis/senta_bilstm.tar.gz --no-check-certificate
+tar -xzvf senta_bilstm.tar.gz
+wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/LexicalAnalysis/lac.tar.gz --no-check-certificate
+tar -xzvf lac.tar.gz
+wget https://paddle-serving.bj.bcebos.com/reader/lac/lac_dict.tar.gz --no-check-certificate
+tar -xzvf lac_dict.tar.gz
+wget https://paddle-serving.bj.bcebos.com/reader/senta/vocab.txt --no-check-certificate
diff --git a/examples/C++/PaddleNLP/senta/senta_web_service.py b/examples/C++/PaddleNLP/senta/senta_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..1e872f0eae0e9ecbfae820367e26db9e94f3cf86
--- /dev/null
+++ b/examples/C++/PaddleNLP/senta/senta_web_service.py
@@ -0,0 +1,73 @@
+#encoding=utf-8
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+import numpy as np
+from paddle_serving_server.web_service import WebService
+from paddle_serving_client import Client
+from paddle_serving_app.reader import LACReader, SentaReader
+
+
+class SentaService(WebService):
+ #初始化lac模型预测服务
+ def init_lac_client(self, lac_port, lac_client_config):
+ self.lac_reader = LACReader()
+ self.senta_reader = SentaReader()
+ self.lac_client = Client()
+ self.lac_client.load_client_config(lac_client_config)
+ self.lac_client.connect(["127.0.0.1:{}".format(lac_port)])
+
+ #定义senta模型预测服务的预处理，调用顺序：lac reader->lac模型预测->预测结果后处理->senta reader
+ def preprocess(self, feed=[], fetch=[]):
+ feed_batch = []
+ is_batch = True
+ words_lod = [0]
+ for ins in feed:
+ if "words" not in ins:
+ raise ("feed data error!")
+ feed_data = self.lac_reader.process(ins["words"])
+ words_lod.append(words_lod[-1] + len(feed_data))
+ feed_batch.append(np.array(feed_data).reshape(len(feed_data), 1))
+ words = np.concatenate(feed_batch, axis=0)
+
+ lac_result = self.lac_client.predict(
+ feed={"words": words,
+ "words.lod": words_lod},
+ fetch=["crf_decode"],
+ batch=True)
+ result_lod = lac_result["crf_decode.lod"]
+ feed_batch = []
+ words_lod = [0]
+ for i in range(len(feed)):
+ segs = self.lac_reader.parse_result(
+ feed[i]["words"],
+ lac_result["crf_decode"][result_lod[i]:result_lod[i + 1]])
+ feed_data = self.senta_reader.process(segs)
+ feed_batch.append(np.array(feed_data).reshape(len(feed_data), 1))
+ words_lod.append(words_lod[-1] + len(feed_data))
+ return {
+ "words": np.concatenate(feed_batch),
+ "words.lod": words_lod
+ }, fetch, is_batch
+
+
+senta_service = SentaService(name="senta")
+senta_service.load_model_config("senta_bilstm_model")
+senta_service.prepare_server(workdir="workdir")
+senta_service.init_lac_client(
+ lac_port=9300, lac_client_config="lac_model/serving_server_conf.prototxt")
+senta_service.run_rpc_service()
+senta_service.run_web_service()
diff --git a/examples/C++/PaddleOCR/ocr/README.md b/examples/C++/PaddleOCR/ocr/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..95cc210a7e68d5582e68460f2eec89419bf7fd7c
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/README.md
@@ -0,0 +1,127 @@
+# OCR
+
+(English|[简体中文](./README_CN.md))
+
+## Get Model
+```
+python3 -m paddle_serving_app.package --get_model ocr_rec
+tar -xzvf ocr_rec.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_det
+tar -xzvf ocr_det.tar.gz
+```
+
+## Get Dataset (Optional)
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/ocr/test_imgs.tar
+tar xf test_imgs.tar
+```
+
+## Web Service
+
+### Start Service
+
+```
+#choose one of cpu/gpu commands as following
+#for cpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model --port 9293
+python3 ocr_web_server.py cpu
+#for gpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model --port 9293 --gpu_ids 0
+python3 ocr_web_server.py gpu
+```
+
+### Client Prediction
+```
+python3 ocr_web_client.py
+```
+If you want a faster web service, please try Web LocalPredictor Service
+
+## Web LocalPredictor Service
+```
+#choose one of cpu/gpu commands as following
+#for cpu user
+python3 ocr_debugger_server.py cpu
+#for gpu user
+python3 ocr_debugger_server.py gpu
+```
+
+## Web LocalPredictor Client Prediction
+```
+python3 ocr_web_client.py
+```
+
+## Benchmark
+
+CPU: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz * 40
+
+GPU: Nvidia Tesla V100 * 1
+
+Dataset: RCTW 500 sample images
+
+| engine | client read image(ms) | client-server tras time(ms) | server read image（ms） | det pre(ms) | det infer(ms) | det post(ms) | rec pre(ms) | rec infer(ms) | rec post(ms) | server-client trans time(ms) | server side time consumption(ms) | server side overhead(ms) | total time（ms) |
+|------------------------------|----------------|----------------------------|------------------|--------------------|------------------|--------------------|--------------------|------------------|--------------------|--------------------------|--------------------|--------------|---------------|
+| Serving web service | 8.69 | 13.41 | 109.97 | 2.82 | 87.76 | 4.29 | 3.98 | 78.51 | 3.66 | 4.12 | 181.02 | 136.49 | 317.51 |
+| Serving LocalPredictor web service | 8.73 | 16.42 | 115.27 | 2.93 | 20.63 | 3.97 | 4.48 | 13.84 | 3.60 | 6.91 | 49.45 | 147.33 | 196.78 |
+
+## Appendix: For Users who want to launch Det or Rec only
+if you are going to detect images not recognize it or directly recognize the words from images. We also provide Det and Rec server for you.
+
+### Det Server
+
+```
+python3 det_web_server.py cpu #for cpu user
+python3 det_web_server.py gpu #for gpu user
+#or
+python3 det_debugger_server.py cpu #for cpu user
+python3 det_debugger_server.py gpu #for gpu user
+```
+
+### Det Client
+
+```
+# also use ocr_web_client.py
+python3 ocr_web_client.py
+```
+
+### Rec Server
+
+```
+python3 rec_web_server.py cpu #for cpu user
+python3 rec_web_server.py gpu #for gpu user
+#or
+python3 rec_debugger_server.py cpu #for cpu user
+python3 rec_debugger_server.py gpu #for gpu user
+```
+
+### Rec Client
+
+```
+python3 rec_web_client.py
+```
+
+## C++ OCR Service
+
+**Notice：** If you need to concatenate det model and rec model, and do pre-processing and post-processing in Paddle Serving C++ framework, you need to use the C++ server compiled with WITH_OPENCV option，see the [COMPILE.md](../../../doc/COMPILE.md)
+
+### Start Service
+Select a startup mode according to CPU / GPU device
+
+After the -- model parameter, the folder path of multiple model files is passed in to start the prediction service of multiple model concatenation.
+```
+#for cpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --port 9293
+#for gpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --port 9293 --gpu_ids 0
+```
+
+### Client Prediction
+The pre-processing and post-processing is in the C + + server part, the image's Base64 encoded string is passed into the C + + server.
+
+so the value of parameter `feed_var` which is in the file `ocr_det_client/serving_client_conf.prototxt` should be changed.
+
+for this case, `feed_type` should be 20(which means the data type is string),`shape` should be 1.
+
+By passing in multiple client folder paths, the client can be started for multi model prediction.
+```
+python3 ocr_cpp_client.py ocr_det_client ocr_rec_client
+```
diff --git a/examples/C++/PaddleOCR/ocr/README_CN.md b/examples/C++/PaddleOCR/ocr/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..5c0734c94aa6d61e1fdb9e8f87d5ee187c805ff0
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/README_CN.md
@@ -0,0 +1,126 @@
+# OCR 服务
+
+([English](./README.md)|简体中文)
+
+## 获取模型
+```
+python3 -m paddle_serving_app.package --get_model ocr_rec
+tar -xzvf ocr_rec.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_det
+tar -xzvf ocr_det.tar.gz
+```
+## 获取数据集（可选）
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/ocr/test_imgs.tar
+tar xf test_imgs.tar
+```
+
+## Web Service服务
+
+### 启动服务
+
+```
+#根据CPU/GPU设备选择一种启动方式
+#for cpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model --port 9293
+python3 ocr_web_server.py cpu
+#for gpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model --port 9293 --gpu_ids 0
+python3 ocr_web_server.py gpu
+```
+
+### 启动客户端
+```
+python3 ocr_web_client.py
+```
+
+如果用户需要更快的执行速度，请尝试LocalPredictor版Web服务
+## 启动LocalPredictor版Web服务
+```
+#根据CPU/GPU设备选择一种启动方式
+#for cpu user
+python3 ocr_debugger_server.py cpu
+#for gpu user
+python3 ocr_debugger_server.py gpu
+```
+
+## 启动客户端
+```
+python3 ocr_web_client.py
+```
+
+## 性能指标
+
+CPU: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz * 40
+
+GPU: Nvidia Tesla V100单卡
+
+数据集：RCTW 500张测试数据集
+
+| engine | 客户端读图(ms) | 客户端发送请求到服务端(ms) | 服务端读图（ms） | 检测预处理耗时(ms) | 检测模型耗时(ms) | 检测后处理耗时(ms) | 识别预处理耗时(ms) | 识别模型耗时(ms) | 识别后处理耗时(ms) | 服务端回传客户端时间(ms) | 服务端整体耗时(ms) | 空跑耗时(ms) | 整体耗时（ms) |
+|------------------------------|----------------|----------------------------|------------------|--------------------|------------------|--------------------|--------------------|------------------|--------------------|--------------------------|--------------------|--------------|---------------|
+| Serving web service | 8.69 | 13.41 | 109.97 | 2.82 | 87.76 | 4.29 | 3.98 | 78.51 | 3.66 | 4.12 | 181.02 | 136.49 | 317.51 |
+| Serving LocalPredictor web service | 8.73 | 16.42 | 115.27 | 2.93 | 20.63 | 3.97 | 4.48 | 13.84 | 3.60 | 6.91 | 49.45 | 147.33 | 196.78 |
+
+
+## 附录： 检测/识别单服务启动
+如果您想单独启动检测或者识别服务，我们也提供了启动单服务的代码
+
+### 启动检测服务
+
+```
+python3 det_web_server.py cpu #for cpu user
+python3 det_web_server.py gpu #for gpu user
+#or
+python3 det_debugger_server.py cpu #for cpu user
+python3 det_debugger_server.py gpu #for gpu user
+```
+
+### 检测服务客户端
+
+```
+# also use ocr_web_client.py
+python3 ocr_web_client.py
+```
+
+### 启动识别服务
+
+```
+python3 rec_web_server.py cpu #for cpu user
+python3 rec_web_server.py gpu #for gpu user
+#or
+python3 rec_debugger_server.py cpu #for cpu user
+python3 rec_debugger_server.py gpu #for gpu user
+```
+
+### 识别服务客户端
+
+```
+python3 rec_web_client.py
+```
+## C++ OCR Service服务
+
+**注意：** 若您需要使用Paddle Serving C++框架串联det模型和rec模型，并进行前后处理，您需要使用开启WITH_OPENCV选项编译的C++ Server，详见[COMPILE.md](../../../doc/COMPILE.md)
+
+### 启动服务
+根据CPU/GPU设备选择一种启动方式
+
+通过--model后，指定多个模型文件的文件夹路径来启动多模型串联的预测服务。
+```
+#for cpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --port 9293
+#for gpu user
+python3 -m paddle_serving_server.serve --model ocr_det_model ocr_rec_model --port 9293 --gpu_ids 0
+```
+
+### 启动客户端
+由于需要在C++Server部分进行前后处理，传入C++Server的仅仅是图片的base64编码的字符串，故第一个模型的Client配置需要修改
+
+即`ocr_det_client/serving_client_conf.prototxt`中`feed_var`字段
+
+对于本示例而言，`feed_type`应修改为20(数据类型为string),`shape`为1.
+
+通过在客户端启动后加入多个client模型的client配置文件夹路径，启动client进行预测。
+```
+python3 ocr_cpp_client.py ocr_det_client ocr_rec_client
+```
diff --git a/examples/C++/PaddleOCR/ocr/det_debugger_server.py b/examples/C++/PaddleOCR/ocr/det_debugger_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..5b40fe9372a56b2b663c1bfeff02619a8ec9730b
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/det_debugger_server.py
@@ -0,0 +1,79 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+import cv2
+import sys
+import numpy as np
+import os
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes
+if sys.argv[1] == 'gpu':
+ from paddle_serving_server.web_service import WebService
+elif sys.argv[1] == 'cpu':
+ from paddle_serving_server.web_service import WebService
+import time
+import re
+import base64
+
+
+class OCRService(WebService):
+ def init_det(self):
+ self.det_preprocess = Sequential([
+ ResizeByFactor(32, 960), Div(255),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), Transpose(
+ (2, 0, 1))
+ ])
+ self.filter_func = FilterBoxes(10, 10)
+ self.post_func = DBPostProcess({
+ "thresh": 0.3,
+ "box_thresh": 0.5,
+ "max_candidates": 1000,
+ "unclip_ratio": 1.5,
+ "min_size": 3
+ })
+
+ def preprocess(self, feed=[], fetch=[]):
+ data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ self.ori_h, self.ori_w, _ = im.shape
+ det_img = self.det_preprocess(im)
+ _, self.new_h, self.new_w = det_img.shape
+ return {
+ "image": det_img[np.newaxis, :].copy()
+ }, ["concat_1.tmp_0"], True
+
+ def postprocess(self, feed={}, fetch=[], fetch_map=None):
+ det_out = fetch_map["concat_1.tmp_0"]
+ ratio_list = [
+ float(self.new_h) / self.ori_h, float(self.new_w) / self.ori_w
+ ]
+ dt_boxes_list = self.post_func(det_out, [ratio_list])
+ dt_boxes = self.filter_func(dt_boxes_list[0], [self.ori_h, self.ori_w])
+ return {"dt_boxes": dt_boxes.tolist()}
+
+
+ocr_service = OCRService(name="ocr")
+ocr_service.load_model_config("ocr_det_model")
+if sys.argv[1] == 'gpu':
+ ocr_service.set_gpus("0")
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="gpu")
+elif sys.argv[1] == 'cpu':
+ ocr_service.prepare_server(workdir="workdir", port=9292)
+ocr_service.init_det()
+ocr_service.run_debugger_service()
+ocr_service.run_web_service()
diff --git a/examples/C++/PaddleOCR/ocr/det_web_server.py b/examples/C++/PaddleOCR/ocr/det_web_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..d38686e5a86c4f2df45db7f495a8c08a72270919
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/det_web_server.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+import cv2
+import sys
+import numpy as np
+import os
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes
+if sys.argv[1] == 'gpu':
+ from paddle_serving_server.web_service import WebService
+elif sys.argv[1] == 'cpu':
+ from paddle_serving_server.web_service import WebService
+import time
+import re
+import base64
+
+
+class OCRService(WebService):
+ def init_det(self):
+ self.det_preprocess = Sequential([
+ ResizeByFactor(32, 960), Div(255),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), Transpose(
+ (2, 0, 1))
+ ])
+ self.filter_func = FilterBoxes(10, 10)
+ self.post_func = DBPostProcess({
+ "thresh": 0.3,
+ "box_thresh": 0.5,
+ "max_candidates": 1000,
+ "unclip_ratio": 1.5,
+ "min_size": 3
+ })
+
+ def preprocess(self, feed=[], fetch=[]):
+ data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ self.ori_h, self.ori_w, _ = im.shape
+ det_img = self.det_preprocess(im)
+ _, self.new_h, self.new_w = det_img.shape
+ print(det_img)
+ return {"image": det_img}, ["concat_1.tmp_0"], False
+
+ def postprocess(self, feed={}, fetch=[], fetch_map=None):
+ det_out = fetch_map["concat_1.tmp_0"]
+ ratio_list = [
+ float(self.new_h) / self.ori_h, float(self.new_w) / self.ori_w
+ ]
+ dt_boxes_list = self.post_func(det_out, [ratio_list])
+ dt_boxes = self.filter_func(dt_boxes_list[0], [self.ori_h, self.ori_w])
+ return {"dt_boxes": dt_boxes.tolist()}
+
+
+ocr_service = OCRService(name="ocr")
+ocr_service.load_model_config("ocr_det_model")
+if sys.argv[1] == 'gpu':
+ ocr_service.set_gpus("0")
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="gpu")
+elif sys.argv[1] == 'cpu':
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="cpu")
+ocr_service.init_det()
+ocr_service.run_rpc_service()
+ocr_service.run_web_service()
diff --git a/examples/C++/PaddleOCR/ocr/imgs/1.jpg b/examples/C++/PaddleOCR/ocr/imgs/1.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..08010177fed2ee8c3709912c06c0b161ba546313
Binary files /dev/null and b/examples/C++/PaddleOCR/ocr/imgs/1.jpg differ
diff --git a/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py b/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..aba8f7bbf2365bb251d043aa5628e0118785ea5d
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/ocr_cpp_client.py
@@ -0,0 +1,48 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+import sys
+import numpy as np
+import base64
+import os
+import cv2
+from paddle_serving_app.reader import Sequential, URL2Image, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+
+client = Client()
+# TODO:load_client need to load more than one client model.
+# this need to figure out some details.
+client.load_client_config(sys.argv[1:])
+client.connect(["127.0.0.1:9293"])
+
+import paddle
+test_img_dir = "imgs/"
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image) #data.tostring()).decode('utf8')
+
+
+for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ fetch_map = client.predict(
+ feed={"image": image},
+ fetch=["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"],
+ batch=True)
+ #print("{} {}".format(fetch_map["price"][0], data[0][1][0]))
+ print(fetch_map)
diff --git a/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py b/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..88dd94a8224fc5c9c6f972b96d81af60ce518763
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/ocr_debugger_server.py
@@ -0,0 +1,113 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import OCRReader
+import cv2
+import sys
+import numpy as np
+import os
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, URL2Image, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes, GetRotateCropImage, SortedBoxes
+if sys.argv[1] == 'gpu':
+ from paddle_serving_server.web_service import WebService
+elif sys.argv[1] == 'cpu':
+ from paddle_serving_server.web_service import WebService
+from paddle_serving_app.local_predict import LocalPredictor
+import time
+import re
+import base64
+
+
+class OCRService(WebService):
+ def init_det_debugger(self, det_model_config):
+ self.det_preprocess = Sequential([
+ ResizeByFactor(32, 960), Div(255),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), Transpose(
+ (2, 0, 1))
+ ])
+ self.det_client = LocalPredictor()
+ if sys.argv[1] == 'gpu':
+ self.det_client.load_model_config(
+ det_model_config, use_gpu=True, gpu_id=1)
+ elif sys.argv[1] == 'cpu':
+ self.det_client.load_model_config(det_model_config)
+ self.ocr_reader = OCRReader()
+
+ def preprocess(self, feed=[], fetch=[]):
+ data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ ori_h, ori_w, _ = im.shape
+ det_img = self.det_preprocess(im)
+ _, new_h, new_w = det_img.shape
+ det_img = det_img[np.newaxis, :]
+ det_img = det_img.copy()
+ det_out = self.det_client.predict(
+ feed={"image": det_img}, fetch=["concat_1.tmp_0"], batch=True)
+ filter_func = FilterBoxes(10, 10)
+ post_func = DBPostProcess({
+ "thresh": 0.3,
+ "box_thresh": 0.5,
+ "max_candidates": 1000,
+ "unclip_ratio": 1.5,
+ "min_size": 3
+ })
+ sorted_boxes = SortedBoxes()
+ ratio_list = [float(new_h) / ori_h, float(new_w) / ori_w]
+ dt_boxes_list = post_func(det_out["concat_1.tmp_0"], [ratio_list])
+ dt_boxes = filter_func(dt_boxes_list[0], [ori_h, ori_w])
+ dt_boxes = sorted_boxes(dt_boxes)
+ get_rotate_crop_image = GetRotateCropImage()
+ img_list = []
+ max_wh_ratio = 0
+ for i, dtbox in enumerate(dt_boxes):
+ boximg = get_rotate_crop_image(im, dt_boxes[i])
+ img_list.append(boximg)
+ h, w = boximg.shape[0:2]
+ wh_ratio = w * 1.0 / h
+ max_wh_ratio = max(max_wh_ratio, wh_ratio)
+ if len(img_list) == 0:
+ return [], []
+ _, w, h = self.ocr_reader.resize_norm_img(img_list[0],
+ max_wh_ratio).shape
+ imgs = np.zeros((len(img_list), 3, w, h)).astype('float32')
+ for id, img in enumerate(img_list):
+ norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
+ imgs[id] = norm_img
+ feed = {"image": imgs.copy()}
+ fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ return feed, fetch, True
+
+ def postprocess(self, feed={}, fetch=[], fetch_map=None):
+ rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ res_lst = []
+ for res in rec_res:
+ res_lst.append(res[0])
+ res = {"res": res_lst}
+ return res
+
+
+ocr_service = OCRService(name="ocr")
+ocr_service.load_model_config("ocr_rec_model")
+ocr_service.prepare_server(workdir="workdir", port=9292)
+ocr_service.init_det_debugger(det_model_config="ocr_det_model")
+if sys.argv[1] == 'gpu':
+ ocr_service.set_gpus("0")
+ ocr_service.run_debugger_service(gpu=True)
+elif sys.argv[1] == 'cpu':
+ ocr_service.run_debugger_service()
+ocr_service.run_web_service()
diff --git a/examples/C++/PaddleOCR/ocr/ocr_web_client.py b/examples/C++/PaddleOCR/ocr/ocr_web_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..ce96a8bbcd585f37368d70070d649e25a0129029
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/ocr_web_client.py
@@ -0,0 +1,40 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -*- coding: utf-8 -*-
+
+import requests
+import json
+import cv2
+import base64
+import os, sys
+import time
+
+
+def cv2_to_base64(image):
+ #data = cv2.imencode('.jpg', image)[1]
+ return base64.b64encode(image).decode(
+ 'utf8') #data.tostring()).decode('utf8')
+
+
+headers = {"Content-type": "application/json"}
+url = "http://127.0.0.1:9292/ocr/prediction"
+test_img_dir = "imgs/"
+for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"feed": [{"image": image}], "fetch": ["res"]}
+ r = requests.post(url=url, headers=headers, data=json.dumps(data))
+ print(r)
+ print(r.json())
diff --git a/examples/C++/PaddleOCR/ocr/ocr_web_server.py b/examples/C++/PaddleOCR/ocr/ocr_web_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..58fc850c94a5e8d2f37ae5d03f14b60d343a2203
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/ocr_web_server.py
@@ -0,0 +1,105 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import OCRReader
+import cv2
+import sys
+import numpy as np
+import os
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, URL2Image, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes, GetRotateCropImage, SortedBoxes
+if sys.argv[1] == 'gpu':
+ from paddle_serving_server.web_service import WebService
+elif sys.argv[1] == 'cpu':
+ from paddle_serving_server.web_service import WebService
+import time
+import re
+import base64
+
+
+class OCRService(WebService):
+ def init_det_client(self, det_port, det_client_config):
+ self.det_preprocess = Sequential([
+ ResizeByFactor(32, 960), Div(255),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), Transpose(
+ (2, 0, 1))
+ ])
+ self.det_client = Client()
+ self.det_client.load_client_config(det_client_config)
+ self.det_client.connect(["127.0.0.1:{}".format(det_port)])
+ self.ocr_reader = OCRReader()
+
+ def preprocess(self, feed=[], fetch=[]):
+ data = base64.b64decode(feed[0]["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ ori_h, ori_w, _ = im.shape
+ det_img = self.det_preprocess(im)
+ det_out = self.det_client.predict(
+ feed={"image": det_img}, fetch=["concat_1.tmp_0"], batch=False)
+ _, new_h, new_w = det_img.shape
+ filter_func = FilterBoxes(10, 10)
+ post_func = DBPostProcess({
+ "thresh": 0.3,
+ "box_thresh": 0.5,
+ "max_candidates": 1000,
+ "unclip_ratio": 1.5,
+ "min_size": 3
+ })
+ sorted_boxes = SortedBoxes()
+ ratio_list = [float(new_h) / ori_h, float(new_w) / ori_w]
+ dt_boxes_list = post_func(det_out["concat_1.tmp_0"], [ratio_list])
+ dt_boxes = filter_func(dt_boxes_list[0], [ori_h, ori_w])
+ dt_boxes = sorted_boxes(dt_boxes)
+ get_rotate_crop_image = GetRotateCropImage()
+ feed_list = []
+ img_list = []
+ max_wh_ratio = 0
+ for i, dtbox in enumerate(dt_boxes):
+ boximg = get_rotate_crop_image(im, dt_boxes[i])
+ img_list.append(boximg)
+ h, w = boximg.shape[0:2]
+ wh_ratio = w * 1.0 / h
+ max_wh_ratio = max(max_wh_ratio, wh_ratio)
+ for img in img_list:
+ norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
+ feed_list.append(norm_img[np.newaxis, :])
+ feed_batch = {"image": np.concatenate(feed_list, axis=0)}
+ fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ return feed_batch, fetch, True
+
+ def postprocess(self, feed={}, fetch=[], fetch_map=None):
+ rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ res_lst = []
+ for res in rec_res:
+ res_lst.append(res[0])
+ res = {"res": res_lst}
+ return res
+
+
+ocr_service = OCRService(name="ocr")
+ocr_service.load_model_config("ocr_rec_model")
+if sys.argv[1] == 'gpu':
+ ocr_service.set_gpus("0")
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="gpu")
+elif sys.argv[1] == 'cpu':
+ ocr_service.prepare_server(workdir="workdir", port=9292)
+ocr_service.init_det_client(
+ det_port=9293,
+ det_client_config="ocr_det_client/serving_client_conf.prototxt")
+ocr_service.run_rpc_service()
+ocr_service.run_web_service()
diff --git a/examples/C++/PaddleOCR/ocr/rec_debugger_server.py b/examples/C++/PaddleOCR/ocr/rec_debugger_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..f84463238af859a00983f515e405686c00fdf9fa
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/rec_debugger_server.py
@@ -0,0 +1,79 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import OCRReader
+import cv2
+import sys
+import numpy as np
+import os
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, URL2Image, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes, GetRotateCropImage, SortedBoxes
+if sys.argv[1] == 'gpu':
+ from paddle_serving_server.web_service import WebService
+elif sys.argv[1] == 'cpu':
+ from paddle_serving_server.web_service import WebService
+import time
+import re
+import base64
+
+
+class OCRService(WebService):
+ def init_rec(self):
+ self.ocr_reader = OCRReader()
+
+ def preprocess(self, feed=[], fetch=[]):
+ img_list = []
+ for feed_data in feed:
+ data = base64.b64decode(feed_data["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img_list.append(im)
+ max_wh_ratio = 0
+ for i, boximg in enumerate(img_list):
+ h, w = boximg.shape[0:2]
+ wh_ratio = w * 1.0 / h
+ max_wh_ratio = max(max_wh_ratio, wh_ratio)
+ _, w, h = self.ocr_reader.resize_norm_img(img_list[0],
+ max_wh_ratio).shape
+ imgs = np.zeros((len(img_list), 3, w, h)).astype('float32')
+ for i, img in enumerate(img_list):
+ norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
+ imgs[i] = norm_img
+ feed = {"image": imgs.copy()}
+ fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ return feed, fetch, True
+
+ def postprocess(self, feed={}, fetch=[], fetch_map=None):
+ rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ res_lst = []
+ for res in rec_res:
+ res_lst.append(res[0])
+ res = {"res": res_lst}
+ return res
+
+
+ocr_service = OCRService(name="ocr")
+ocr_service.load_model_config("ocr_rec_model")
+if sys.argv[1] == 'gpu':
+ ocr_service.set_gpus("0")
+ ocr_service.init_rec()
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="gpu")
+elif sys.argv[1] == 'cpu':
+ ocr_service.init_rec()
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="cpu")
+ocr_service.run_debugger_service()
+ocr_service.run_web_service()
diff --git a/examples/C++/PaddleOCR/ocr/rec_img/ch_doc3.jpg b/examples/C++/PaddleOCR/ocr/rec_img/ch_doc3.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..c0c2053643c6211b9c2017e305c5fa05bba0cc66
Binary files /dev/null and b/examples/C++/PaddleOCR/ocr/rec_img/ch_doc3.jpg differ
diff --git a/examples/C++/PaddleOCR/ocr/rec_web_client.py b/examples/C++/PaddleOCR/ocr/rec_web_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..312a2148886d6f084a1c077d84e907cb28c0652a
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/rec_web_client.py
@@ -0,0 +1,41 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -*- coding: utf-8 -*-
+
+import requests
+import json
+import cv2
+import base64
+import os, sys
+import time
+
+
+def cv2_to_base64(image):
+ #data = cv2.imencode('.jpg', image)[1]
+ return base64.b64encode(image).decode(
+ 'utf8') #data.tostring()).decode('utf8')
+
+
+headers = {"Content-type": "application/json"}
+url = "http://127.0.0.1:9292/ocr/prediction"
+test_img_dir = "rec_img/"
+
+for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ #data = {"feed": [{"image": image}], "fetch": ["res"]}
+ data = {"feed": [{"image": image}] * 3, "fetch": ["res"]}
+ r = requests.post(url=url, headers=headers, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/C++/PaddleOCR/ocr/rec_web_server.py b/examples/C++/PaddleOCR/ocr/rec_web_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..2db6e398d3a025e739761fabd50c5bb8a6609f07
--- /dev/null
+++ b/examples/C++/PaddleOCR/ocr/rec_web_server.py
@@ -0,0 +1,80 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import OCRReader
+import cv2
+import sys
+import numpy as np
+import os
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, URL2Image, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes, GetRotateCropImage, SortedBoxes
+if sys.argv[1] == 'gpu':
+ from paddle_serving_server.web_service import WebService
+elif sys.argv[1] == 'cpu':
+ from paddle_serving_server.web_service import WebService
+import time
+import re
+import base64
+
+
+class OCRService(WebService):
+ def init_rec(self):
+ self.ocr_reader = OCRReader()
+
+ def preprocess(self, feed=[], fetch=[]):
+ # TODO: to handle batch rec images
+ img_list = []
+ for feed_data in feed:
+ data = base64.b64decode(feed_data["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img_list.append(im)
+ max_wh_ratio = 0
+ for i, boximg in enumerate(img_list):
+ h, w = boximg.shape[0:2]
+ wh_ratio = w * 1.0 / h
+ max_wh_ratio = max(max_wh_ratio, wh_ratio)
+ _, w, h = self.ocr_reader.resize_norm_img(img_list[0],
+ max_wh_ratio).shape
+ imgs = np.zeros((len(img_list), 3, w, h)).astype('float32')
+ for i, img in enumerate(img_list):
+ norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
+ imgs[i] = norm_img
+
+ feed = {"image": imgs.copy()}
+ fetch = ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ return feed, fetch, True
+
+ def postprocess(self, feed={}, fetch=[], fetch_map=None):
+ rec_res = self.ocr_reader.postprocess(fetch_map, with_score=True)
+ res_lst = []
+ for res in rec_res:
+ res_lst.append(res[0])
+ res = {"res": res_lst}
+ return res
+
+
+ocr_service = OCRService(name="ocr")
+ocr_service.load_model_config("ocr_rec_model")
+ocr_service.init_rec()
+if sys.argv[1] == 'gpu':
+ ocr_service.set_gpus("0")
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="gpu")
+elif sys.argv[1] == 'cpu':
+ ocr_service.prepare_server(workdir="workdir", port=9292, device="cpu")
+ocr_service.run_rpc_service()
+ocr_service.run_web_service()
diff --git a/examples/C++/PaddleRec/criteo_ctr/README.md b/examples/C++/PaddleRec/criteo_ctr/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6c1d79e7362a0240a49a9f0243f3de3340119ce3
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/README.md
@@ -0,0 +1,31 @@
+## CTR Prediction Service
+
+([简体中文](./README_CN.md)|English)
+
+### download criteo dataset
+```
+sh get_data.sh
+```
+
+### download inference model
+```
+wget https://paddle-serving.bj.bcebos.com/criteo_ctr_example/criteo_ctr_demo_model.tar.gz
+tar xf criteo_ctr_demo_model.tar.gz
+mv models/ctr_client_conf .
+mv models/ctr_serving_model .
+```
+the directories like `ctr_serving_model` and `ctr_client_conf` will appear.
+
+### Start RPC Inference Service
+
+```
+python3 -m paddle_serving_server.serve --model ctr_serving_model/ --port 9292 #CPU RPC Service
+python3 -m paddle_serving_server.serve --model ctr_serving_model/ --port 9292 --gpu_ids 0 #RPC Service on GPU 0
+```
+
+### RPC Infer
+
+```
+python3 test_client.py ctr_client_conf/serving_client_conf.prototxt raw_data/part-0
+```
+the latency will display in the end.
diff --git a/examples/C++/PaddleRec/criteo_ctr/README_CN.md b/examples/C++/PaddleRec/criteo_ctr/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c5b1da76055e64bd08bcf2a00dffe537bc931ee9
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/README_CN.md
@@ -0,0 +1,31 @@
+## CTR预测服务
+
+(简体中文|[English](./README.md))
+
+### 获取样例数据
+```
+sh get_data.sh
+```
+
+### 下载模型
+```
+wget https://paddle-serving.bj.bcebos.com/criteo_ctr_example/criteo_ctr_demo_model.tar.gz
+tar xf criteo_ctr_demo_model.tar.gz
+mv models/ctr_client_conf .
+mv models/ctr_serving_model .
+```
+会在当前目录出现`ctr_serving_model` 和 `ctr_client_conf`文件夹。
+
+### 启动RPC预测服务
+
+```
+python3 -m paddle_serving_server.serve --model ctr_serving_model/ --port 9292 #启动CPU预测服务
+python3 -m paddle_serving_server.serve --model ctr_serving_model/ --port 9292 --gpu_ids 0 #在GPU 0上启动预测服务
+```
+
+### 执行预测
+
+```
+python3 test_client.py ctr_client_conf/serving_client_conf.prototxt raw_data/part-0
+```
+预测完毕会输出预测过程的耗时。
diff --git a/examples/C++/PaddleRec/criteo_ctr/args.py b/examples/C++/PaddleRec/criteo_ctr/args.py
new file mode 100644
index 0000000000000000000000000000000000000000..30124d4ebd9cd27cdb4580e654a8a47c55b178bf
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/args.py
@@ -0,0 +1,105 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+import argparse
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(description="PaddlePaddle CTR example")
+ parser.add_argument(
+ '--train_data_path',
+ type=str,
+ default='./data/raw/train.txt',
+ help="The path of training dataset")
+ parser.add_argument(
+ '--sparse_only',
+ type=bool,
+ default=False,
+ help="Whether we use sparse features only")
+ parser.add_argument(
+ '--test_data_path',
+ type=str,
+ default='./data/raw/valid.txt',
+ help="The path of testing dataset")
+ parser.add_argument(
+ '--batch_size',
+ type=int,
+ default=1000,
+ help="The size of mini-batch (default:1000)")
+ parser.add_argument(
+ '--embedding_size',
+ type=int,
+ default=10,
+ help="The size for embedding layer (default:10)")
+ parser.add_argument(
+ '--num_passes',
+ type=int,
+ default=10,
+ help="The number of passes to train (default: 10)")
+ parser.add_argument(
+ '--model_output_dir',
+ type=str,
+ default='models',
+ help='The path for model to store (default: models)')
+ parser.add_argument(
+ '--sparse_feature_dim',
+ type=int,
+ default=1000001,
+ help='sparse feature hashing space for index processing')
+ parser.add_argument(
+ '--is_local',
+ type=int,
+ default=1,
+ help='Local train or distributed train (default: 1)')
+ parser.add_argument(
+ '--cloud_train',
+ type=int,
+ default=0,
+ help='Local train or distributed train on paddlecloud (default: 0)')
+ parser.add_argument(
+ '--async_mode',
+ action='store_true',
+ default=False,
+ help='Whether start pserver in async mode to support ASGD')
+ parser.add_argument(
+ '--no_split_var',
+ action='store_true',
+ default=False,
+ help='Whether split variables into blocks when update_method is pserver')
+ parser.add_argument(
+ '--role',
+ type=str,
+ default='pserver', # trainer or pserver
+ help='The path for model to store (default: models)')
+ parser.add_argument(
+ '--endpoints',
+ type=str,
+ default='127.0.0.1:6000',
+ help='The pserver endpoints, like: 127.0.0.1:6000,127.0.0.1:6001')
+ parser.add_argument(
+ '--current_endpoint',
+ type=str,
+ default='127.0.0.1:6000',
+ help='The path for model to store (default: 127.0.0.1:6000)')
+ parser.add_argument(
+ '--trainer_id',
+ type=int,
+ default=0,
+ help='The path for model to store (default: models)')
+ parser.add_argument(
+ '--trainers',
+ type=int,
+ default=1,
+ help='The num of trianers, (default: 1)')
+ return parser.parse_args()
diff --git a/examples/C++/PaddleRec/criteo_ctr/benchmark.py b/examples/C++/PaddleRec/criteo_ctr/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..8be7387d6ef32d656f676d55c21e25052e403f16
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/benchmark.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import unicode_literals, absolute_import
+import os
+import sys
+import time
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args
+import requests
+import json
+import criteo_reader as criteo
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ batch = 1
+ buf_size = 100
+ dataset = criteo.CriteoDataset()
+ dataset.setup(1000001)
+ test_filelists = [
+ "./raw_data/part-%d" % x for x in range(len(os.listdir("./raw_data")))
+ ]
+ reader = dataset.infer_reader(test_filelists[len(test_filelists) - 40:],
+ batch, buf_size)
+ if args.request == "rpc":
+ fetch = ["prob"]
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
+
+ start = time.time()
+ for i in range(1000):
+ if args.batch_size == 1:
+ data = reader().next()
+ feed_dict = {}
+ for i in range(1, 27):
+ feed_dict["sparse_{}".format(i - 1)] = data[0][i]
+ result = client.predict(feed=feed_dict, fetch=fetch)
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ elif args.request == "http":
+ raise ("Not support http service.")
+ end = time.time()
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = ["127.0.0.1:9292"]
+ #endpoint_list = endpoint_list + endpoint_list + endpoint_list
+ result = multi_thread_runner.run(single_func, args.thread,
+ {"endpoint": endpoint_list})
+ #result = single_func(0, {"endpoint": endpoint_list})
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ avg_cost = avg_cost / args.thread
+ print("average total cost {} s.".format(avg_cost))
diff --git a/examples/C++/PaddleRec/criteo_ctr/benchmark.sh b/examples/C++/PaddleRec/criteo_ctr/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d15fd560f90e808e724e159e7ffb125b9c06b0a7
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/benchmark.sh
@@ -0,0 +1,9 @@
+rm profile_log
+for thread_num in 1 2 4 8 16
+do
+ $PYTHONROOT/bin/python benchmark.py --thread $thread_num --model ctr_client_conf/serving_client_conf.prototxt --request rpc > profile 2>&1
+ echo "========================================"
+ echo "batch size : $batch_size" >> profile_log
+ $PYTHONROOT/bin/python ../../../util/show_profile.py profile $thread_num >> profile_log
+ tail -n 1 profile >> profile_log
+done
diff --git a/examples/C++/PaddleRec/criteo_ctr/benchmark_batch.py b/examples/C++/PaddleRec/criteo_ctr/benchmark_batch.py
new file mode 100644
index 0000000000000000000000000000000000000000..1e4348c99dc0d960b1818ea6f0eb1ae2f5bd2ccb
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/benchmark_batch.py
@@ -0,0 +1,79 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import unicode_literals, absolute_import
+import os
+import sys
+import time
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args
+import requests
+import json
+import criteo_reader as criteo
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ batch = 1
+ buf_size = 100
+ dataset = criteo.CriteoDataset()
+ dataset.setup(1000001)
+ test_filelists = [
+ "./raw_data/part-%d" % x for x in range(len(os.listdir("./raw_data")))
+ ]
+ reader = dataset.infer_reader(test_filelists[len(test_filelists) - 40:],
+ batch, buf_size)
+ if args.request == "rpc":
+ fetch = ["prob"]
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([resource["endpoint"][idx % len(resource["endpoint"])]])
+
+ start = time.time()
+ for i in range(1000):
+ if args.batch_size >= 1:
+ feed_batch = []
+ for bi in range(args.batch_size):
+ feed_dict = {}
+ data = reader().next()
+ for i in range(1, 27):
+ feed_dict["sparse_{}".format(i - 1)] = data[0][i]
+ feed_batch.append(feed_dict)
+ result = client.predict(feed=feed_batch, fetch=fetch)
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ elif args.request == "http":
+ raise ("no batch predict for http")
+ end = time.time()
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = ["127.0.0.1:9292"]
+ #endpoint_list = endpoint_list + endpoint_list + endpoint_list
+ result = multi_thread_runner.run(single_func, args.thread,
+ {"endpoint": endpoint_list})
+ #result = single_func(0, {"endpoint": endpoint_list})
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ avg_cost = avg_cost / args.thread
+ print("average total cost {} s.".format(avg_cost))
diff --git a/examples/C++/PaddleRec/criteo_ctr/benchmark_batch.sh b/examples/C++/PaddleRec/criteo_ctr/benchmark_batch.sh
new file mode 100644
index 0000000000000000000000000000000000000000..71ce4764e7e341997934d6b6c608813155b9eb6f
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/benchmark_batch.sh
@@ -0,0 +1,12 @@
+rm profile_log
+for thread_num in 1 2 4 8 16
+do
+for batch_size in 1 2 4 8 16 32 64 128 256 512
+do
+ $PYTHONROOT/bin/python benchmark_batch.py --thread $thread_num --batch_size $batch_size --model serving_client_conf/serving_client_conf.prototxt --request rpc > profile 2>&1
+ echo "========================================"
+ echo "batch size : $batch_size" >> profile_log
+ $PYTHONROOT/bin/python ../../../util/show_profile.py profile $thread_num >> profile_log
+ tail -n 1 profile >> profile_log
+done
+done
diff --git a/examples/C++/PaddleRec/criteo_ctr/clean.sh b/examples/C++/PaddleRec/criteo_ctr/clean.sh
new file mode 100644
index 0000000000000000000000000000000000000000..78703636bf2b5b037e75d07055ca377abb3123c4
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/clean.sh
@@ -0,0 +1 @@
+rm -rf *pyc kvdb raw_data ctr_client_conf ctr_serving_model ctr_data.tar.gz *~
diff --git a/examples/C++/PaddleRec/criteo_ctr/get_data.sh b/examples/C++/PaddleRec/criteo_ctr/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..1f244b3a4aa81488bb493825576ba30c4b3bba22
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/get_data.sh
@@ -0,0 +1,2 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/data/ctr_prediction/ctr_data.tar.gz
+tar -zxvf ctr_data.tar.gz
diff --git a/examples/C++/PaddleRec/criteo_ctr/local_train.py b/examples/C++/PaddleRec/criteo_ctr/local_train.py
new file mode 100644
index 0000000000000000000000000000000000000000..bbc940750c5f608b47300a9a33f9e48bfb4344b1
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/local_train.py
@@ -0,0 +1,88 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import print_function
+
+from args import parse_args
+import os
+import paddle.fluid as fluid
+import sys
+from network_conf import dnn_model
+
+dense_feature_dim = 13
+
+
+def train():
+ args = parse_args()
+ sparse_only = args.sparse_only
+ if not os.path.isdir(args.model_output_dir):
+ os.mkdir(args.model_output_dir)
+ dense_input = fluid.layers.data(
+ name="dense_input", shape=[dense_feature_dim], dtype='float32')
+ sparse_input_ids = [
+ fluid.layers.data(
+ name="C" + str(i), shape=[1], lod_level=1, dtype="int64")
+ for i in range(1, 27)
+ ]
+ label = fluid.layers.data(name='label', shape=[1], dtype='int64')
+
+ #nn_input = None if sparse_only else dense_input
+ nn_input = dense_input
+ predict_y, loss, auc_var, batch_auc_var = dnn_model(
+ nn_input, sparse_input_ids, label, args.embedding_size,
+ args.sparse_feature_dim)
+
+ optimizer = fluid.optimizer.SGD(learning_rate=1e-4)
+ optimizer.minimize(loss)
+
+ exe = fluid.Executor(fluid.CPUPlace())
+ exe.run(fluid.default_startup_program())
+ dataset = fluid.DatasetFactory().create_dataset("InMemoryDataset")
+ dataset.set_use_var([dense_input] + sparse_input_ids + [label])
+
+ python_executable = "python"
+ pipe_command = "{} criteo_reader.py {}".format(python_executable,
+ args.sparse_feature_dim)
+
+ dataset.set_pipe_command(pipe_command)
+ dataset.set_batch_size(128)
+ thread_num = 10
+ dataset.set_thread(thread_num)
+
+ whole_filelist = [
+ "raw_data/part-%d" % x for x in range(len(os.listdir("raw_data")))
+ ]
+
+ dataset.set_filelist(whole_filelist[:thread_num])
+ dataset.load_into_memory()
+
+ epochs = 1
+ for i in range(epochs):
+ exe.train_from_dataset(
+ program=fluid.default_main_program(), dataset=dataset, debug=True)
+ print("epoch {} finished".format(i))
+
+ import paddle_serving_client.io as server_io
+ feed_var_dict = {}
+ for i, sparse in enumerate(sparse_input_ids):
+ feed_var_dict["sparse_{}".format(i)] = sparse
+ fetch_var_dict = {"prob": predict_y}
+
+ server_io.save_model("ctr_serving_model", "ctr_client_conf", feed_var_dict,
+ fetch_var_dict, fluid.default_main_program())
+
+
+if __name__ == '__main__':
+ train()
diff --git a/examples/C++/PaddleRec/criteo_ctr/network_conf.py b/examples/C++/PaddleRec/criteo_ctr/network_conf.py
new file mode 100644
index 0000000000000000000000000000000000000000..ec5eb7d55ab59965e474842015301b3a9088d91e
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/network_conf.py
@@ -0,0 +1,74 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import paddle.fluid as fluid
+import math
+
+
+def dnn_model(dense_input, sparse_inputs, label, embedding_size,
+ sparse_feature_dim):
+ def embedding_layer(input):
+ emb = fluid.layers.embedding(
+ input=input,
+ is_sparse=True,
+ is_distributed=False,
+ size=[sparse_feature_dim, embedding_size],
+ param_attr=fluid.ParamAttr(
+ name="SparseFeatFactors",
+ initializer=fluid.initializer.Uniform()))
+ return fluid.layers.sequence_pool(input=emb, pool_type='sum')
+
+ def mlp_input_tensor(emb_sums, dense_tensor):
+ if isinstance(dense_tensor, fluid.Variable):
+ return fluid.layers.concat(emb_sums, axis=1)
+ else:
+ return fluid.layers.concat(emb_sums + [dense_tensor], axis=1)
+
+ def mlp(mlp_input):
+ fc1 = fluid.layers.fc(input=mlp_input,
+ size=400,
+ act='relu',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(mlp_input.shape[1]))))
+ fc2 = fluid.layers.fc(input=fc1,
+ size=400,
+ act='relu',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(fc1.shape[1]))))
+ fc3 = fluid.layers.fc(input=fc2,
+ size=400,
+ act='relu',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(fc2.shape[1]))))
+ pre = fluid.layers.fc(input=fc3,
+ size=2,
+ act='softmax',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(fc3.shape[1]))))
+ return pre
+
+ emb_sums = list(map(embedding_layer, sparse_inputs))
+ mlp_in = mlp_input_tensor(emb_sums, dense_input)
+ predict = mlp(mlp_in)
+ cost = fluid.layers.cross_entropy(input=predict, label=label)
+ avg_cost = fluid.layers.reduce_sum(cost)
+ accuracy = fluid.layers.accuracy(input=predict, label=label)
+ auc_var, batch_auc_var, auc_states = \
+ fluid.layers.auc(input=predict, label=label, num_thresholds=2 ** 12, slide_steps=20)
+ return predict, avg_cost, auc_var, batch_auc_var
diff --git a/examples/C++/PaddleRec/criteo_ctr/test_client.py b/examples/C++/PaddleRec/criteo_ctr/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..c1c1ea685b4d28c7f11a7c5a456e11a36377e22d
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/test_client.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+import sys
+import os
+import time
+from paddle_serving_client.metric import auc
+import numpy as np
+import sys
+
+
+class CriteoReader(object):
+ def __init__(self, sparse_feature_dim):
+ self.cont_min_ = [0, -3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+ self.cont_max_ = [
+ 20, 600, 100, 50, 64000, 500, 100, 50, 500, 10, 10, 10, 50
+ ]
+ self.cont_diff_ = [
+ 20, 603, 100, 50, 64000, 500, 100, 50, 500, 10, 10, 10, 50
+ ]
+ self.hash_dim_ = sparse_feature_dim
+ # here, training data are lines with line_index < train_idx_
+ self.train_idx_ = 41256555
+ self.continuous_range_ = range(1, 14)
+ self.categorical_range_ = range(14, 40)
+
+ def process_line(self, line):
+ features = line.rstrip('\n').split('\t')
+ dense_feature = []
+ sparse_feature = []
+ for idx in self.continuous_range_:
+ if features[idx] == '':
+ dense_feature.append(0.0)
+ else:
+ dense_feature.append((float(features[idx]) - self.cont_min_[idx - 1]) / \
+ self.cont_diff_[idx - 1])
+ for idx in self.categorical_range_:
+ sparse_feature.append(
+ [hash(str(idx) + features[idx]) % self.hash_dim_])
+
+ return sparse_feature
+
+
+py_version = sys.version_info[0]
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9292"])
+reader = CriteoReader(1000001)
+batch = 1
+buf_size = 100
+label_list = []
+prob_list = []
+start = time.time()
+f = open(sys.argv[2], 'r')
+for ei in range(10):
+ data = reader.process_line(f.readline())
+ feed_dict = {}
+ for i in range(1, 27):
+ feed_dict["sparse_{}".format(i - 1)] = np.array(data[i - 1]).reshape(-1)
+ feed_dict["sparse_{}.lod".format(i - 1)] = [0, len(data[i - 1])]
+ fetch_map = client.predict(feed=feed_dict, fetch=["prob"])
+ print(fetch_map)
+end = time.time()
+f.close()
diff --git a/examples/C++/PaddleRec/criteo_ctr/test_server.py b/examples/C++/PaddleRec/criteo_ctr/test_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..34f859daab4c808aa9d50d2109a81a69eed96df6
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr/test_server.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import os
+import sys
+from paddle_serving_server import OpMaker
+from paddle_serving_server import OpSeqMaker
+from paddle_serving_server import Server
+
+op_maker = OpMaker()
+read_op = op_maker.create('general_reader')
+general_infer_op = op_maker.create('general_infer')
+response_op = op_maker.create('general_response')
+
+op_seq_maker = OpSeqMaker()
+op_seq_maker.add_op(read_op)
+op_seq_maker.add_op(general_infer_op)
+op_seq_maker.add_op(response_op)
+
+server = Server()
+server.set_op_sequence(op_seq_maker.get_op_sequence())
+server.load_model_config(sys.argv[1])
+server.prepare_server(workdir="work_dir1", port=9292, device="cpu")
+server.run_server()
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/README.md b/examples/C++/PaddleRec/criteo_ctr_with_cube/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..4492b398add170104a7cd17adff6dc5b83368dbe
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/README.md
@@ -0,0 +1,72 @@
+## Criteo CTR with Sparse Parameter Indexing Service
+
+([简体中文](./README_CN.md)|English)
+
+### Get Sample Dataset
+
+go to directory `python/examples/criteo_ctr_with_cube`
+```
+sh get_data.sh
+```
+
+### Download Model and Sparse Parameter Sequence Files
+```
+wget https://paddle-serving.bj.bcebos.com/unittest/ctr_cube_unittest.tar.gz
+tar xf ctr_cube_unittest.tar.gz
+mv models/ctr_client_conf ./
+mv models/ctr_serving_model_kv ./
+mv models/data ./cube/
+```
+the model will be in ./ctr_server_model_kv and ./ctr_client_config.
+
+### Start Sparse Parameter Indexing Service
+```
+wget https://paddle-serving.bj.bcebos.com/others/cube_app.tar.gz
+tar xf cube_app.tar.gz
+mv cube_app/cube* ./cube/
+sh cube_prepare.sh &
+```
+
+Here, the sparse parameter is loaded by cube sparse parameter indexing service Cube.
+
+### Start RPC Predictor, the number of serving thread is 4（configurable in test_server.py）
+
+```
+python3 test_server.py ctr_serving_model_kv
+```
+
+### Run Prediction
+
+```
+python3 test_client.py ctr_client_conf/serving_client_conf.prototxt ./raw_data
+```
+
+### Benchmark
+
+CPU ：Intel(R) Xeon(R) CPU 6148 @ 2.40GHz
+
+Model ：[Criteo CTR](https://github.com/PaddlePaddle/Serving/blob/develop/python/examples/criteo_ctr_with_cube/network_conf.py)
+
+server core/thread num ： 4/8
+
+Run
+```
+bash benchmark.sh
+```
+1000 batches will be sent by every client
+
+| client thread num | prepro | client infer | op0 | op1 | op2 | postpro | avg_latency | qps |
+| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- | ----- |
+| 1 | 0.035 | 1.596 | 0.021 | 0.518 | 0.0024 | 0.0025 | 6.774 | 147.7 |
+| 2 | 0.034 | 1.780 | 0.027 | 0.463 | 0.0020 | 0.0023 | 6.931 | 288.3 |
+| 4 | 0.038 | 2.954 | 0.025 | 0.455 | 0.0019 | 0.0027 | 8.378 | 477.5 |
+| 8 | 0.044 | 8.230 | 0.028 | 0.464 | 0.0023 | 0.0034 | 14.191 | 563.8 |
+| 16 | 0.048 | 21.037 | 0.028 | 0.455 | 0.0025 | 0.0041 | 27.236 | 587.5 |
+
+the average latency of threads
+
+
+
+The QPS is
+
+
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/README_CN.md b/examples/C++/PaddleRec/criteo_ctr_with_cube/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..8c8d51d918410f5fa8a681dacab2000cdc0192bd
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/README_CN.md
@@ -0,0 +1,70 @@
+## 带稀疏参数索引服务的CTR预测服务
+(简体中文|[English](./README.md))
+
+### 获取样例数据
+进入目录 `python/examples/criteo_ctr_with_cube`
+```
+sh get_data.sh
+```
+
+### 下载模型和稀疏参数序列文件
+```
+wget https://paddle-serving.bj.bcebos.com/unittest/ctr_cube_unittest.tar.gz
+tar xf ctr_cube_unittest.tar.gz
+mv models/ctr_client_conf ./
+mv models/ctr_serving_model_kv ./
+mv models/data ./cube/
+```
+执行脚本后会在当前目录有ctr_server_model_kv和ctr_client_config文件夹。
+
+### 启动稀疏参数索引服务
+```
+wget https://paddle-serving.bj.bcebos.com/others/cube_app.tar.gz
+tar xf cube_app.tar.gz
+mv cube_app/cube* ./cube/
+sh cube_prepare.sh &
+```
+
+此处，模型当中的稀疏参数会被存放在稀疏参数索引服务Cube当中。
+
+### 启动RPC预测服务，服务端线程数为4（可在test_server.py配置）
+
+```
+python3 test_server.py ctr_serving_model_kv
+```
+
+### 执行预测
+
+```
+python3 test_client.py ctr_client_conf/serving_client_conf.prototxt ./raw_data
+```
+
+### Benchmark
+
+设备 ：Intel(R) Xeon(R) CPU 6148 @ 2.40GHz
+
+模型 ：[Criteo CTR](https://github.com/PaddlePaddle/Serving/blob/develop/python/examples/criteo_ctr_with_cube/network_conf.py)
+
+server core/thread num ： 4/8
+
+执行
+```
+bash benchmark.sh
+```
+客户端每个线程会发送1000个batch
+
+| client thread num | prepro | client infer | op0 | op1 | op2 | postpro | avg_latency | qps |
+| ------------------ | ------ | ------------ | ------ | ----- | ------ | ------- | ----- | ----- |
+| 1 | 0.035 | 1.596 | 0.021 | 0.518 | 0.0024 | 0.0025 | 6.774 | 147.7 |
+| 2 | 0.034 | 1.780 | 0.027 | 0.463 | 0.0020 | 0.0023 | 6.931 | 288.3 |
+| 4 | 0.038 | 2.954 | 0.025 | 0.455 | 0.0019 | 0.0027 | 8.378 | 477.5 |
+| 8 | 0.044 | 8.230 | 0.028 | 0.464 | 0.0023 | 0.0034 | 14.191 | 563.8 |
+| 16 | 0.048 | 21.037 | 0.028 | 0.455 | 0.0025 | 0.0041 | 27.236 | 587.5 |
+
+平均每个线程耗时图如下
+
+
+
+每个线程QPS耗时如下
+
+
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/criteo_reader.py b/examples/C++/PaddleRec/criteo_ctr_with_cube/criteo_reader.py
new file mode 100755
index 0000000000000000000000000000000000000000..2a80af78a9c2033bf246f703ca70a817ab786af3
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/criteo_reader.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import sys
+import paddle.fluid.incubate.data_generator as dg
+
+
+class CriteoDataset(dg.MultiSlotDataGenerator):
+ def setup(self, sparse_feature_dim):
+ self.cont_min_ = [0, -3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
+ self.cont_max_ = [
+ 20, 600, 100, 50, 64000, 500, 100, 50, 500, 10, 10, 10, 50
+ ]
+ self.cont_diff_ = [
+ 20, 603, 100, 50, 64000, 500, 100, 50, 500, 10, 10, 10, 50
+ ]
+ self.hash_dim_ = sparse_feature_dim
+ # here, training data are lines with line_index < train_idx_
+ self.train_idx_ = 41256555
+ self.continuous_range_ = range(1, 14)
+ self.categorical_range_ = range(14, 40)
+
+ def _process_line(self, line):
+ features = line.rstrip('\n').split('\t')
+ dense_feature = []
+ sparse_feature = []
+ for idx in self.continuous_range_:
+ if features[idx] == '':
+ dense_feature.append(0.0)
+ else:
+ dense_feature.append((float(features[idx]) - self.cont_min_[idx - 1]) / \
+ self.cont_diff_[idx - 1])
+ for idx in self.categorical_range_:
+ sparse_feature.append(
+ [hash(str(idx) + features[idx]) % self.hash_dim_])
+
+ return dense_feature, sparse_feature, [int(features[0])]
+
+ def infer_reader(self, filelist, batch, buf_size):
+ def local_iter():
+ for fname in filelist:
+ with open(fname.strip(), "r") as fin:
+ for line in fin:
+ dense_feature, sparse_feature, label = self._process_line(
+ line)
+ #yield dense_feature, sparse_feature, label
+ yield [dense_feature] + sparse_feature + [label]
+
+ import paddle
+ batch_iter = paddle.batch(
+ paddle.reader.shuffle(
+ local_iter, buf_size=buf_size),
+ batch_size=batch)
+ return batch_iter
+
+ def generate_sample(self, line):
+ def data_iter():
+ dense_feature, sparse_feature, label = self._process_line(line)
+ feature_name = ["dense_input"]
+ for idx in self.categorical_range_:
+ feature_name.append("C" + str(idx - 13))
+ feature_name.append("label")
+ yield zip(feature_name, [dense_feature] + sparse_feature + [label])
+
+ return data_iter
+
+
+if __name__ == "__main__":
+ criteo_dataset = CriteoDataset()
+ criteo_dataset.setup(int(sys.argv[1]))
+ criteo_dataset.run_from_stdin()
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/conf/cube.conf b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/conf/cube.conf
new file mode 100755
index 0000000000000000000000000000000000000000..b70f6e34247e410f9b80054010338d3c8f452ec6
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/conf/cube.conf
@@ -0,0 +1,13 @@
+[{
+ "dict_name": "test_dict",
+ "shard": 1,
+ "dup": 1,
+ "timeout": 200,
+ "retry": 3,
+ "backup_request": 100,
+ "type": "ipport_list",
+ "load_balancer": "rr",
+ "nodes": [{
+ "ipport_list": "list://127.0.0.1:8027"
+ }]
+}]
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/conf/gflags.conf b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/conf/gflags.conf
new file mode 100755
index 0000000000000000000000000000000000000000..21c7bddebd8f22b91d0ba26a6121007f96a4380b
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/conf/gflags.conf
@@ -0,0 +1,4 @@
+--port=8027
+--dict_split=1
+--in_mem=true
+--log_dir=./log/
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/keys b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/keys
new file mode 100755
index 0000000000000000000000000000000000000000..f00c965d8307308469e537302baa73048488f162
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube/keys
@@ -0,0 +1,10 @@
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/cube_prepare.sh b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube_prepare.sh
new file mode 100755
index 0000000000000000000000000000000000000000..773baba4d91b02b244e766cd8ebf899cc740dbbc
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/cube_prepare.sh
@@ -0,0 +1,20 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+#! /bin/bash
+
+mkdir -p cube_model
+mkdir -p cube/data
+./cube/cube-builder -dict_name=test_dict -job_mode=base -last_version=0 -cur_version=0 -depend_version=0 -input_path=./cube_model -output_path=${PWD}/cube/data -shard_num=1 -only_build=false
+cd cube && ./cube
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/get_data.sh b/examples/C++/PaddleRec/criteo_ctr_with_cube/get_data.sh
new file mode 100755
index 0000000000000000000000000000000000000000..1f244b3a4aa81488bb493825576ba30c4b3bba22
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/get_data.sh
@@ -0,0 +1,2 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/data/ctr_prediction/ctr_data.tar.gz
+tar -zxvf ctr_data.tar.gz
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/local_train.py b/examples/C++/PaddleRec/criteo_ctr_with_cube/local_train.py
new file mode 100755
index 0000000000000000000000000000000000000000..27ed67852140873f6ae245a19c1bad88c61ba00f
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/local_train.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from __future__ import print_function
+
+from args import parse_args
+import os
+import paddle.fluid as fluid
+import paddle
+import sys
+from network_conf import dnn_model
+
+dense_feature_dim = 13
+
+paddle.enable_static()
+
+
+def train():
+ args = parse_args()
+ sparse_only = args.sparse_only
+ if not os.path.isdir(args.model_output_dir):
+ os.mkdir(args.model_output_dir)
+ dense_input = fluid.layers.data(
+ name="dense_input", shape=[dense_feature_dim], dtype='float32')
+ sparse_input_ids = [
+ fluid.layers.data(
+ name="C" + str(i), shape=[1], lod_level=1, dtype="int64")
+ for i in range(1, 27)
+ ]
+ label = fluid.layers.data(name='label', shape=[1], dtype='int64')
+
+ #nn_input = None if sparse_only else dense_input
+ nn_input = dense_input
+ predict_y, loss, auc_var, batch_auc_var, infer_vars = dnn_model(
+ nn_input, sparse_input_ids, label, args.embedding_size,
+ args.sparse_feature_dim)
+
+ optimizer = fluid.optimizer.SGD(learning_rate=1e-4)
+ optimizer.minimize(loss)
+
+ exe = fluid.Executor(fluid.CPUPlace())
+ exe.run(fluid.default_startup_program())
+ dataset = fluid.DatasetFactory().create_dataset("InMemoryDataset")
+ dataset.set_use_var([dense_input] + sparse_input_ids + [label])
+
+ python_executable = "python3.6"
+ pipe_command = "{} criteo_reader.py {}".format(python_executable,
+ args.sparse_feature_dim)
+
+ dataset.set_pipe_command(pipe_command)
+ dataset.set_batch_size(128)
+ thread_num = 10
+ dataset.set_thread(thread_num)
+
+ whole_filelist = [
+ "raw_data/part-%d" % x for x in range(len(os.listdir("raw_data")))
+ ]
+
+ print(whole_filelist)
+ dataset.set_filelist(whole_filelist[:100])
+ dataset.load_into_memory()
+ fluid.layers.Print(auc_var)
+ epochs = 1
+ for i in range(epochs):
+ exe.train_from_dataset(
+ program=fluid.default_main_program(), dataset=dataset, debug=True)
+ print("epoch {} finished".format(i))
+
+ import paddle_serving_client.io as server_io
+ feed_var_dict = {}
+ feed_var_dict['dense_input'] = dense_input
+ for i, sparse in enumerate(sparse_input_ids):
+ feed_var_dict["embedding_{}.tmp_0".format(i)] = sparse
+ fetch_var_dict = {"prob": predict_y}
+
+ feed_kv_dict = {}
+ feed_kv_dict['dense_input'] = dense_input
+ for i, emb in enumerate(infer_vars):
+ feed_kv_dict["embedding_{}.tmp_0".format(i)] = emb
+ fetch_var_dict = {"prob": predict_y}
+
+ server_io.save_model("ctr_serving_model", "ctr_client_conf", feed_var_dict,
+ fetch_var_dict, fluid.default_main_program())
+
+ server_io.save_model("ctr_serving_model_kv", "ctr_client_conf_kv",
+ feed_kv_dict, fetch_var_dict,
+ fluid.default_main_program())
+
+
+if __name__ == '__main__':
+ train()
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/network_conf.py b/examples/C++/PaddleRec/criteo_ctr_with_cube/network_conf.py
new file mode 100755
index 0000000000000000000000000000000000000000..2975533a72ad21d6dd5896446fd06c1f9bdfe8b4
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/network_conf.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import paddle.fluid as fluid
+import math
+
+
+def dnn_model(dense_input, sparse_inputs, label, embedding_size,
+ sparse_feature_dim):
+ def embedding_layer(input):
+ emb = fluid.layers.embedding(
+ input=input,
+ is_sparse=True,
+ is_distributed=False,
+ size=[sparse_feature_dim, embedding_size],
+ param_attr=fluid.ParamAttr(
+ name="SparseFeatFactors",
+ initializer=fluid.initializer.Uniform()))
+ x = fluid.layers.sequence_pool(input=emb, pool_type='sum')
+ return emb, x
+
+ def mlp_input_tensor(emb_sums, dense_tensor):
+ #if isinstance(dense_tensor, fluid.Variable):
+ # return fluid.layers.concat(emb_sums, axis=1)
+ #else:
+ return fluid.layers.concat(emb_sums + [dense_tensor], axis=1)
+
+ def mlp(mlp_input):
+ fc1 = fluid.layers.fc(input=mlp_input,
+ size=400,
+ act='relu',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(mlp_input.shape[1]))))
+ fc2 = fluid.layers.fc(input=fc1,
+ size=400,
+ act='relu',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(fc1.shape[1]))))
+ fc3 = fluid.layers.fc(input=fc2,
+ size=400,
+ act='relu',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(fc2.shape[1]))))
+ pre = fluid.layers.fc(input=fc3,
+ size=2,
+ act='softmax',
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(
+ scale=1 / math.sqrt(fc3.shape[1]))))
+ return pre
+
+ emb_pair_sums = list(map(embedding_layer, sparse_inputs))
+ emb_sums = [x[1] for x in emb_pair_sums]
+ infer_vars = [x[0] for x in emb_pair_sums]
+ mlp_in = mlp_input_tensor(emb_sums, dense_input)
+ predict = mlp(mlp_in)
+ cost = fluid.layers.cross_entropy(input=predict, label=label)
+ avg_cost = fluid.layers.reduce_sum(cost)
+ accuracy = fluid.layers.accuracy(input=predict, label=label)
+ auc_var, batch_auc_var, auc_states = \
+ fluid.layers.auc(input=predict, label=label, num_thresholds=2 ** 12, slide_steps=20)
+ return predict, avg_cost, auc_var, batch_auc_var, infer_vars
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/test_client.py b/examples/C++/PaddleRec/criteo_ctr_with_cube/test_client.py
new file mode 100755
index 0000000000000000000000000000000000000000..e9d517e0d9961a771e54af8062b958631da68bc8
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/test_client.py
@@ -0,0 +1,56 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+import sys
+import os
+import criteo_reader as criteo
+import time
+from paddle_serving_client.metric import auc
+import numpy as np
+py_version = sys.version_info[0]
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9292"])
+
+batch = 1
+buf_size = 100
+dataset = criteo.CriteoDataset()
+dataset.setup(1000001)
+test_filelists = ["{}/part-0".format(sys.argv[2])]
+reader = dataset.infer_reader(test_filelists, batch, buf_size)
+label_list = []
+prob_list = []
+start = time.time()
+for ei in range(100):
+ if py_version == 2:
+ data = reader().next()
+ else:
+ data = reader().__next__()
+ feed_dict = {}
+ feed_dict['dense_input'] = np.array(data[0][0]).reshape(1, len(data[0][0]))
+
+ for i in range(1, 27):
+ feed_dict["embedding_{}.tmp_0".format(i - 1)] = np.array(data[0][
+ i]).reshape(len(data[0][i]))
+ feed_dict["embedding_{}.tmp_0.lod".format(i - 1)] = [0, len(data[0][i])]
+ fetch_map = client.predict(feed=feed_dict, fetch=["prob"], batch=True)
+ print(fetch_map)
+ prob_list.append(fetch_map['prob'][0][1])
+ label_list.append(data[0][-1][0])
+
+end = time.time()
+print(end - start)
diff --git a/examples/C++/PaddleRec/criteo_ctr_with_cube/test_server.py b/examples/C++/PaddleRec/criteo_ctr_with_cube/test_server.py
new file mode 100755
index 0000000000000000000000000000000000000000..479c602910b5afa52b35a66d00316f54905c0741
--- /dev/null
+++ b/examples/C++/PaddleRec/criteo_ctr_with_cube/test_server.py
@@ -0,0 +1,41 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import os
+import sys
+from paddle_serving_server import OpMaker
+from paddle_serving_server import OpSeqMaker
+from paddle_serving_server import Server
+
+op_maker = OpMaker()
+read_op = op_maker.create('general_reader')
+general_dist_kv_infer_op = op_maker.create('general_dist_kv_infer')
+response_op = op_maker.create('general_response')
+
+op_seq_maker = OpSeqMaker()
+op_seq_maker.add_op(read_op)
+op_seq_maker.add_op(general_dist_kv_infer_op)
+op_seq_maker.add_op(response_op)
+
+server = Server()
+server.set_op_sequence(op_seq_maker.get_op_sequence())
+server.set_num_threads(4)
+server.load_model_config(sys.argv[1])
+server.prepare_server(
+ workdir="work_dir1",
+ port=9292,
+ device="cpu",
+ cube_conf="./cube/conf/cube.conf")
+server.run_server()
diff --git a/examples/C++/PaddleSeg/deeplabv3/N0060.jpg b/examples/C++/PaddleSeg/deeplabv3/N0060.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..feac2837eaa5ae5db414d9769a0c5a830dde268d
Binary files /dev/null and b/examples/C++/PaddleSeg/deeplabv3/N0060.jpg differ
diff --git a/examples/C++/PaddleSeg/deeplabv3/README.md b/examples/C++/PaddleSeg/deeplabv3/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..08022618fcec5220667ca19bfb803cba36519c7b
--- /dev/null
+++ b/examples/C++/PaddleSeg/deeplabv3/README.md
@@ -0,0 +1,22 @@
+# Image Segmentation
+
+## Get Model
+
+```
+python3 -m paddle_serving_app.package --get_model deeplabv3
+tar -xzvf deeplabv3.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model deeplabv3_server --gpu_ids 0 --port 9494
+```
+
+### Client Prediction
+
+```
+python3 deeplabv3_client.py
+```
diff --git a/examples/C++/PaddleSeg/deeplabv3/README_CN.md b/examples/C++/PaddleSeg/deeplabv3/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..16f11daba354349f1b73f8bba00cac8ff5c88864
--- /dev/null
+++ b/examples/C++/PaddleSeg/deeplabv3/README_CN.md
@@ -0,0 +1,21 @@
+# 图像分割
+
+## 获取模型
+
+```
+python3 -m paddle_serving_app.package --get_model deeplabv3
+tar -xzvf deeplabv3.tar.gz
+```
+
+## RPC 服务
+
+### 启动服务端
+
+```
+python3 -m paddle_serving_server.serve --model deeplabv3_server --gpu_ids 0 --port 9494
+```
+
+### 客户端预测
+
+```
+python3 deeplabv3_client.py
diff --git a/examples/C++/PaddleSeg/deeplabv3/deeplabv3_client.py b/examples/C++/PaddleSeg/deeplabv3/deeplabv3_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..77e25d5f5a24d0aa1dad8939c1e7845eaf5e4122
--- /dev/null
+++ b/examples/C++/PaddleSeg/deeplabv3/deeplabv3_client.py
@@ -0,0 +1,34 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize, Transpose, BGR2RGB, SegPostprocess
+import sys
+import cv2
+
+client = Client()
+client.load_client_config("deeplabv3_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9494"])
+
+preprocess = Sequential(
+ [File2Image(), Resize(
+ (512, 512), interpolation=cv2.INTER_LINEAR)])
+
+postprocess = SegPostprocess(2)
+
+filename = "N0060.jpg"
+im = preprocess(filename)
+fetch_map = client.predict(feed={"image": im}, fetch=["output"])
+fetch_map["filename"] = filename
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/N0060.jpg b/examples/C++/PaddleSeg/unet_for_image_seg/N0060.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..feac2837eaa5ae5db414d9769a0c5a830dde268d
Binary files /dev/null and b/examples/C++/PaddleSeg/unet_for_image_seg/N0060.jpg differ
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/README.md b/examples/C++/PaddleSeg/unet_for_image_seg/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..59004712bd76f5388d6e57947f70ce22562f8dbe
--- /dev/null
+++ b/examples/C++/PaddleSeg/unet_for_image_seg/README.md
@@ -0,0 +1,22 @@
+# Image Segmentation
+
+## Get Model
+
+```
+python3 -m paddle_serving_app.package --get_model unet
+tar -xzvf unet.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model unet_model --gpu_ids 0 --port 9494
+```
+
+### Client Prediction
+
+```
+python3 seg_client.py
+```
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/README_CN.md b/examples/C++/PaddleSeg/unet_for_image_seg/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..53c2f1893a879d5585cea0b77103fc1461086784
--- /dev/null
+++ b/examples/C++/PaddleSeg/unet_for_image_seg/README_CN.md
@@ -0,0 +1,22 @@
+# 图像分割
+
+## 获取模型
+
+```
+python3 -m paddle_serving_app.package --get_model unet
+tar -xzvf unet.tar.gz
+```
+
+## RPC 服务
+
+### 启动服务端
+
+```
+python3 -m paddle_serving_server.serve --model unet_model --gpu_ids 0 --port 9494
+```
+
+### 客户端预测
+
+```
+python3 seg_client.py
+```
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/seg_client.py b/examples/C++/PaddleSeg/unet_for_image_seg/seg_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..44f634b6090159ee1bd37c176eebb7d2b7f37065
--- /dev/null
+++ b/examples/C++/PaddleSeg/unet_for_image_seg/seg_client.py
@@ -0,0 +1,34 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize, Transpose, BGR2RGB, SegPostprocess
+import sys
+import cv2
+
+client = Client()
+client.load_client_config("unet_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9494"])
+
+preprocess = Sequential(
+ [File2Image(), Resize(
+ (512, 512), interpolation=cv2.INTER_LINEAR)])
+
+postprocess = SegPostprocess(2)
+
+filename = "N0060.jpg"
+im = preprocess(filename)
+fetch_map = client.predict(feed={"image": im}, fetch=["output"])
+fetch_map["filename"] = filename
+postprocess(fetch_map)
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/README.md b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..edb2af5864db746dc3368423dd7414575ed7b675
--- /dev/null
+++ b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/README.md
@@ -0,0 +1,8 @@
+#UNET_BENCHMARK 使用说明
+## 功能
+* benchmark测试
+## 注意事项
+* 示例图片（可以有多张）请放置于与img_data路径中，支持jpg，jpeg
+* 图片张数应该大于等于并发数量
+## TODO
+* http benchmark
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/img_data/N0060.jpg b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/img_data/N0060.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..feac2837eaa5ae5db414d9769a0c5a830dde268d
Binary files /dev/null and b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/img_data/N0060.jpg differ
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/launch_benckmark.sh b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/launch_benckmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..59c2293e34b11dd2efd088c97a3c8de0dc62cf6f
--- /dev/null
+++ b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/launch_benckmark.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+python unet_benchmark.py --thread 1 --batch_size 1 --model ../unet_client/serving_client_conf.prototxt
+# thread/batch can be modified as you wish
diff --git a/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/unet_benchmark.py b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/unet_benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..172643e364c5462aeed59ebe5e7b45bee7abf8ef
--- /dev/null
+++ b/examples/C++/PaddleSeg/unet_for_image_seg/unet_benchmark/unet_benchmark.py
@@ -0,0 +1,159 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+ unet bench mark script
+ 20201130 first edition by cg82616424
+"""
+from __future__ import unicode_literals, absolute_import
+import os
+import time
+import json
+import requests
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+from paddle_serving_app.reader import Sequential, File2Image, Resize, Transpose, BGR2RGB, SegPostprocess
+args = benchmark_args()
+
+
+def get_img_names(path):
+ """
+ Brief:
+ get img files(jpg) under this path
+ if any exception happened return None
+ Args:
+ path (string): image file path
+ Returns:
+ list: images names under this folder
+ """
+ if not os.path.exists(path):
+ return None
+ if not os.path.isdir(path):
+ return None
+ list_name = []
+ for f_handler in os.listdir(path):
+ file_path = os.path.join(path, f_handler)
+ if os.path.isdir(file_path):
+ continue
+ else:
+ if not file_path.endswith(".jpeg") and not file_path.endswith(
+ ".jpg"):
+ continue
+ list_name.append(file_path)
+ return list_name
+
+
+def preprocess_img(img_list):
+ """
+ Brief:
+ prepare img data for benchmark
+ Args:
+ img_list(list): list for img file path
+ Returns:
+ image content binary list after preprocess
+ """
+ preprocess = Sequential([File2Image(), Resize((512, 512))])
+ result_list = []
+ for img in img_list:
+ img_tmp = preprocess(img)
+ result_list.append(img_tmp)
+ return result_list
+
+
+def benckmark_worker(idx, resource):
+ """
+ Brief:
+ benchmark single worker for unet
+ Args:
+ idx(int): worker idx ,use idx to select backend unet service
+ resource(dict): unet serving endpoint dict
+ Returns:
+ latency
+ TODO:
+ http benckmarks
+ """
+ profile_flags = False
+ latency_flags = False
+ postprocess = SegPostprocess(2)
+ if os.getenv("FLAGS_profile_client"):
+ profile_flags = True
+ if os.getenv("FLAGS_serving_latency"):
+ latency_flags = True
+ latency_list = []
+ client_handler = Client()
+ client_handler.load_client_config(args.model)
+ client_handler.connect(
+ [resource["endpoint"][idx % len(resource["endpoint"])]])
+ start = time.time()
+ turns = resource["turns"]
+ img_list = resource["img_list"]
+ for i in range(turns):
+ if args.batch_size >= 1:
+ l_start = time.time()
+ feed_batch = []
+ b_start = time.time()
+ for bi in range(args.batch_size):
+ feed_batch.append({"image": img_list[bi]})
+ b_end = time.time()
+ if profile_flags:
+ sys.stderr.write(
+ "PROFILE\tpid:{}\tunt_pre_0:{} unet_pre_1:{}\n".format(
+ os.getpid(),
+ int(round(b_start * 1000000)),
+ int(round(b_end * 1000000))))
+ result = client_handler.predict(
+ feed={"image": img_list[bi]}, fetch=["output"])
+ #result["filename"] = "./img_data/N0060.jpg" % (os.getpid(), idx, time.time())
+ #postprocess(result) # if you want to measure post process time, you have to uncomment this line
+ l_end = time.time()
+ if latency_flags:
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+ end = time.time()
+ if latency_flags:
+ return [[end - start], latency_list]
+ else:
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ """
+ usage:
+ """
+ img_file_list = get_img_names("./img_data")
+ img_content_list = preprocess_img(img_file_list)
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = ["127.0.0.1:9494"]
+ turns = 1
+ start = time.time()
+ result = multi_thread_runner.run(benckmark_worker, args.thread, {
+ "endpoint": endpoint_list,
+ "turns": turns,
+ "img_list": img_content_list
+ })
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ avg_cost = avg_cost / args.thread
+ print("total cost: {}s".format(total_cost))
+ print("each thread cost: {}s. ".format(avg_cost))
+ print("qps: {}samples/s".format(args.batch_size * args.thread * turns /
+ total_cost))
+ if os.getenv("FLAGS_serving_latency"):
+ show_latency(result[1])
diff --git a/examples/C++/encryption/README.md b/examples/C++/encryption/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..3120422ebfaa2a88851eda18c42e7740fe29e884
--- /dev/null
+++ b/examples/C++/encryption/README.md
@@ -0,0 +1,48 @@
+# Encryption Model Prediction
+
+([简体中文](README_CN.md)|English)
+
+## Get Origin Model
+
+The example uses the model file of the fit_a_line example as a origin model
+
+```
+sh get_data.sh
+```
+
+## Encrypt Model
+
+The `paddlepaddle` package is used in this example, you may need to download the corresponding package(`pip3 install paddlepaddle`).
+
+[python3 encrypt.py](./encrypt.py)
+
+[//file]:#encrypt.py
+``` python
+def serving_encryption():
+ inference_model_to_serving(
+ dirname="./uci_housing_model",
+ params_filename=None,
+ serving_server="encrypt_server",
+ serving_client="encrypt_client",
+ encryption=True)
+```
+dirname is the folder path where the model is located. If the parameter is discrete, it is unnecessary to specify params_filename, else you need to set `params_filename="__params__"`.
+
+The key is stored in the `key` file, and the encrypted model file and server-side configuration file are stored in the `encrypt_server` directory.
+client-side configuration file are stored in the `encrypt_client` directory.
+
+**Notice：** When encryption prediction is used, the model configuration and parameter folder loaded by server and client should be encrypt_server/ and encrypt_client/
+## Start Encryption Service
+CPU Service
+```
+python3 -m paddle_serving_server.serve --model encrypt_server/ --port 9393 --use_encryption_model
+```
+GPU Service
+```
+python3 -m paddle_serving_server.serve --model encrypt_server/ --port 9393 --use_encryption_model --gpu_ids 0
+```
+
+## Prediction
+```
+python3 test_client.py encrypt_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/encryption/README_CN.md b/examples/C++/encryption/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..ad82d49b61cb70093a9423ad83dbc30663b6d4f1
--- /dev/null
+++ b/examples/C++/encryption/README_CN.md
@@ -0,0 +1,49 @@
+# 加密模型预测
+
+(简体中文|[English](README.md))
+
+## 获取明文模型
+
+示例中使用fit_a_line示例的模型文件作为明文模型
+
+```
+sh get_data.sh
+```
+
+## 模型加密
+本示例中使用了`paddlepaddle`包中的模块，需要进行下载（`pip3 install paddlepaddle`）。
+
+运行[python3 encrypt.py](./encrypt.py)进行模型加密
+
+[//file]:#encrypt.py
+``` python
+def serving_encryption():
+ inference_model_to_serving(
+ dirname="./uci_housing_model",
+ params_filename=None,
+ serving_server="encrypt_server",
+ serving_client="encrypt_client",
+ encryption=True)
+```
+其中dirname为模型所在的文件夹路径
+
+当参数为离散参数时，无须指定params_filename，当参数为__params__时，需指定`params_filename="__params__"`
+
+密钥保存在`key`文件中，加密模型文件以及server端配置文件保存在`encrypt_server`目录下，client端配置文件保存在`encrypt_client`目录下。
+
+**注意：** 当使用加密预测时，服务端和客户端启动加载的模型配置和参数文件夹是encrypt_server/和encrypt_client/
+
+## 启动加密预测服务
+CPU预测服务
+```
+python3 -m paddle_serving_server.serve --model encrypt_server/ --port 9393 --use_encryption_model
+```
+GPU预测服务
+```
+python3 -m paddle_serving_server.serve --model encrypt_server/ --port 9393 --use_encryption_model --gpu_ids 0
+```
+
+## 预测
+```
+python3 test_client.py encrypt_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/encryption/encrypt.py b/examples/C++/encryption/encrypt.py
new file mode 100644
index 0000000000000000000000000000000000000000..e233784390f0899cd81ec7862ceef0d506bbcd1f
--- /dev/null
+++ b/examples/C++/encryption/encrypt.py
@@ -0,0 +1,28 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client.io import inference_model_to_serving
+
+
+def serving_encryption():
+ inference_model_to_serving(
+ dirname="./uci_housing_model",
+ params_filename=None,
+ serving_server="encrypt_server",
+ serving_client="encrypt_client",
+ encryption=True)
+
+
+if __name__ == "__main__":
+ serving_encryption()
diff --git a/examples/C++/encryption/get_data.sh b/examples/C++/encryption/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..c3cd5c236f5643d53c3a30bf0ffd367853ffaf13
--- /dev/null
+++ b/examples/C++/encryption/get_data.sh
@@ -0,0 +1,4 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing_example/encrypt.tar.gz
+tar -xzf encrypt.tar.gz
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
+tar -xzf uci_housing.tar.gz
diff --git a/examples/C++/encryption/test_client.py b/examples/C++/encryption/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..33816e741c9a6ffeda0685d5fd3bda6774a5f186
--- /dev/null
+++ b/examples/C++/encryption/test_client.py
@@ -0,0 +1,33 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+import sys
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.use_key("./key")
+client.connect(["0.0.0.0:9393"], encryption=True)
+fetch_list = client.get_fetch_names()
+
+import paddle
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=1)
+
+for data in test_reader():
+ fetch_map = client.predict(feed={"x": data[0][0]}, fetch=fetch_list)
+ print(fetch_map)
diff --git a/examples/C++/fit_a_line/README.md b/examples/C++/fit_a_line/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..9586cd670240eb43e4a706ff89ea435b7a8c6d1c
--- /dev/null
+++ b/examples/C++/fit_a_line/README.md
@@ -0,0 +1,41 @@
+# Fit a line prediction example
+
+([简体中文](./README_CN.md)|English)
+
+## Get data
+
+```shell
+sh get_data.sh
+```
+
+
+
+## RPC service
+
+### Start server
+
+```shell
+python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9393
+```
+
+## Client prediction
+
+### RPC Client
+The `paddlepaddle` package is used in `test_client.py`, and you may need to download the corresponding package(`pip3 install paddlepaddle`).
+
+``` shell
+python3 test_client.py uci_housing_client/serving_client_conf.prototxt
+```
+
+### Http Client
+
+``` shell
+python3 test_httpclient.py uci_housing_client/serving_client_conf.prototxt
+```
+
+
+## Benchmark
+``` shell
+bash benchmark.sh uci_housing_model uci_housing_client
+```
+The log file of benchmark named `profile_log_uci_housing_model`
diff --git a/examples/C++/fit_a_line/README_CN.md b/examples/C++/fit_a_line/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..d1cace5e2c5b5cee2195deaa1667af68e5f1f067
--- /dev/null
+++ b/examples/C++/fit_a_line/README_CN.md
@@ -0,0 +1,43 @@
+# 线性回归预测服务示例
+
+(简体中文|[English](./README.md))
+
+## 获取数据
+
+```shell
+sh get_data.sh
+```
+
+
+## 开启服务端（支持BRPC-Client/GRPC Client/Http-Client）
+
+```shell
+python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9393
+```
+
+## 客户端预测
+
+### BRPC-Client
+
+`test_client.py`中使用了`paddlepaddle`包，需要进行下载（`pip3 install paddlepaddle`）。
+
+``` shell
+python3 test_client.py uci_housing_client/serving_client_conf.prototxt
+```
+
+### GRPC-Client/Http-Client
+
+``` shell
+python3 test_httpclient.py uci_housing_client/serving_client_conf.prototxt
+```
+
+
+## 性能测试
+``` shell
+bash benchmark.sh uci_housing_model uci_housing_client
+```
+性能测试的日志文件为profile_log_uci_housing_model
+
+如需修改性能测试用例的参数，请修改benchmark.sh中的配置信息。
+
+注意:uci_housing_model和uci_housing_client路径后不要加'/'符号,示例需要在GPU机器上运行。
diff --git a/examples/C++/fit_a_line/benchmark.py b/examples/C++/fit_a_line/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..7c4e4b4c582361f2f0f5d48fb374b2e7899c65b2
--- /dev/null
+++ b/examples/C++/fit_a_line/benchmark.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+import time
+import paddle
+import sys
+import requests
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=1)
+ total_number = sum(1 for _ in train_reader())
+ latency_list = []
+
+ if args.request == "rpc":
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([args.endpoint])
+ start = time.time()
+ for data in train_reader():
+ l_start = time.time()
+ fetch_map = client.predict(feed={"x": data[0][0]}, fetch=["price"])
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ end = time.time()
+ return [[end - start], latency_list, [total_number]]
+ elif args.request == "http":
+ train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=1)
+ start = time.time()
+ for data in train_reader():
+ l_start = time.time()
+ r = requests.post(
+ 'http://{}/uci/prediction'.format(args.endpoint),
+ data={"x": data[0]})
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ end = time.time()
+ return [[end - start], latency_list, [total_number]]
+
+
+start = time.time()
+multi_thread_runner = MultiThreadRunner()
+result = multi_thread_runner.run(single_func, args.thread, {})
+end = time.time()
+total_cost = end - start
+avg_cost = 0
+for i in range(args.thread):
+ avg_cost += result[0][i]
+avg_cost = avg_cost / args.thread
+
+print("total cost: {}s".format(total_cost))
+print("each thread cost: {}s. ".format(avg_cost))
+print("qps: {}samples/s".format(args.batch_size * args.thread / total_cost))
+show_latency(result[1])
diff --git a/examples/C++/fit_a_line/benchmark.sh b/examples/C++/fit_a_line/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..7e23f57cd0073fce1969cd7c4eeaf567e73c29af
--- /dev/null
+++ b/examples/C++/fit_a_line/benchmark.sh
@@ -0,0 +1,55 @@
+rm profile_log*
+export CUDA_VISIBLE_DEVICES=0,1
+export FLAGS_profile_server=1
+export FLAGS_profile_client=1
+export FLAGS_serving_latency=1
+
+gpu_id=0
+#save cpu and gpu utilization log
+if [ -d utilization ];then
+ rm -rf utilization
+else
+ mkdir utilization
+fi
+#start server
+$PYTHONROOT/bin/python3 -m paddle_serving_server.serve --model $1 --port 9292 --thread 4 --gpu_ids 0,1 --mem_optim --ir_optim > elog 2>&1 &
+sleep 5
+
+#warm up
+$PYTHONROOT/bin/python3 benchmark.py --thread 4 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+echo -e "import psutil\nimport time\nwhile True:\n\tcpu_res = psutil.cpu_percent()\n\twith open('cpu.txt', 'a+') as f:\n\t\tf.write(f'{cpu_res}\\\n')\n\ttime.sleep(0.1)" > cpu.py
+for thread_num in 1 4 8 16
+do
+for batch_size in 1 4 16 64
+do
+ job_bt=`date '+%Y%m%d%H%M%S'`
+ nvidia-smi --id=0 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_memory_use.log 2>&1 &
+ nvidia-smi --id=0 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ rm -rf cpu.txt
+ $PYTHONROOT/bin/python3 cpu.py &
+ gpu_memory_pid=$!
+ $PYTHONROOT/bin/python3 benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+ kill `ps -ef|grep used_memory|awk '{print $2}'` > /dev/null
+ kill `ps -ef|grep utilization.gpu|awk '{print $2}'` > /dev/null
+ kill `ps -ef|grep cpu.py|awk '{print $2}'` > /dev/null
+ echo "model_name:" $1
+ echo "thread_num:" $thread_num
+ echo "batch_size:" $batch_size
+ echo "=================Done===================="
+ echo "model_name:$1" >> profile_log_$1
+ echo "batch_size:$batch_size" >> profile_log_$1
+ job_et=`date '+%Y%m%d%H%M%S'`
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "CPU_UTILIZATION:", max}' cpu.txt >> profile_log_$1
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "MAX_GPU_MEMORY:", max}' gpu_memory_use.log >> profile_log_$1
+ awk 'BEGIN {max = 0} {if(NR>1){if ($1 > max) max=$1}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$1
+ rm -rf gpu_use.log gpu_utilization.log
+ $PYTHONROOT/bin/python3 ../../util/show_profile.py profile $thread_num >> profile_log_$1
+ tail -n 8 profile >> profile_log_$1
+ echo "" >> profile_log_$1
+done
+done
+
+#Divided log
+awk 'BEGIN{RS="\n\n"}{i++}{print > "bert_log_"i}' profile_log_$1
+mkdir bert_log && mv bert_log_* bert_log
+ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
diff --git a/examples/C++/fit_a_line/get_data.sh b/examples/C++/fit_a_line/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..84a3966a0ef323cef4b146d8e9489c70a7a8ae35
--- /dev/null
+++ b/examples/C++/fit_a_line/get_data.sh
@@ -0,0 +1,2 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
+tar -xzf uci_housing.tar.gz
diff --git a/examples/C++/fit_a_line/local_train.py b/examples/C++/fit_a_line/local_train.py
new file mode 100644
index 0000000000000000000000000000000000000000..3e0f8880a4d006b346712f2592d6c44986882193
--- /dev/null
+++ b/examples/C++/fit_a_line/local_train.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import sys
+import paddle
+import paddle.fluid as fluid
+paddle.enable_static()
+train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=16)
+
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=16)
+
+x = fluid.data(name='x', shape=[None, 13], dtype='float32')
+y = fluid.data(name='y', shape=[None, 1], dtype='float32')
+
+y_predict = fluid.layers.fc(input=x, size=1, act=None)
+cost = fluid.layers.square_error_cost(input=y_predict, label=y)
+avg_loss = fluid.layers.mean(cost)
+sgd_optimizer = fluid.optimizer.SGD(learning_rate=0.01)
+sgd_optimizer.minimize(avg_loss)
+
+place = fluid.CPUPlace()
+feeder = fluid.DataFeeder(place=place, feed_list=[x, y])
+exe = fluid.Executor(place)
+exe.run(fluid.default_startup_program())
+
+import paddle_serving_client.io as serving_io
+
+for pass_id in range(30):
+ for data_train in train_reader():
+ avg_loss_value, = exe.run(fluid.default_main_program(),
+ feed=feeder.feed(data_train),
+ fetch_list=[avg_loss])
+
+serving_io.save_model("uci_housing_model", "uci_housing_client", {"x": x},
+ {"price": y_predict}, fluid.default_main_program())
diff --git a/examples/C++/fit_a_line/test_client.py b/examples/C++/fit_a_line/test_client.py
new file mode 100755
index 0000000000000000000000000000000000000000..d18ece66686520e25a0b9ebbd2d8b29354f4da16
--- /dev/null
+++ b/examples/C++/fit_a_line/test_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+import sys
+import numpy as np
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9393"])
+fetch_list = client.get_fetch_names()
+import paddle
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=1)
+
+for data in test_reader():
+ new_data = np.zeros((1, 13)).astype("float32")
+ new_data[0] = data[0][0]
+ fetch_map = client.predict(
+ feed={"x": new_data}, fetch=fetch_list, batch=True)
+ print(fetch_map)
diff --git a/examples/C++/fit_a_line/test_httpclient.py b/examples/C++/fit_a_line/test_httpclient.py
new file mode 100755
index 0000000000000000000000000000000000000000..c9f785dc99e2699027862fd2a28bd429e8b1a0a5
--- /dev/null
+++ b/examples/C++/fit_a_line/test_httpclient.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client.httpclient import HttpClient
+import sys
+import numpy as np
+import time
+
+client = HttpClient()
+client.load_client_config(sys.argv[1])
+'''
+if you want use GRPC-client, set_use_grpc_client(True)
+or you can directly use client.grpc_client_predict(...)
+as for HTTP-client,set_use_grpc_client(False)(which is default)
+or you can directly use client.http_client_predict(...)
+'''
+#client.set_use_grpc_client(True)
+'''
+if you want to enable Encrypt Module,uncommenting the following line
+'''
+#client.use_key("./key")
+'''
+if you want to compress,uncommenting the following line
+'''
+#client.set_response_compress(True)
+#client.set_request_compress(True)
+'''
+we recommend use Proto data format in HTTP-body, set True(which is default)
+if you want use JSON data format in HTTP-body, set False
+'''
+#client.set_http_proto(True)
+client.connect(["127.0.0.1:9393"])
+fetch_list = client.get_fetch_names()
+
+import paddle
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=1)
+for data in test_reader():
+ new_data = np.zeros((1, 13)).astype("float32")
+ new_data[0] = data[0][0]
+ fetch_map = client.predict(
+ feed={"x": new_data}, fetch=fetch_list, batch=True)
+ print(fetch_map)
+ break
diff --git a/examples/C++/fit_a_line/test_multi_process_client.py b/examples/C++/fit_a_line/test_multi_process_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..e6120266097f8fdd446998741582a9e396cd2efd
--- /dev/null
+++ b/examples/C++/fit_a_line/test_multi_process_client.py
@@ -0,0 +1,42 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+import paddle
+import numpy as np
+
+
+def single_func(idx, resource):
+ client = Client()
+ client.load_client_config(
+ "./uci_housing_client/serving_client_conf.prototxt")
+ client.connect(["127.0.0.1:9293", "127.0.0.1:9292"])
+ x = [
+ 0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584,
+ 0.6283, 0.4919, 0.1856, 0.0795, -0.0332
+ ]
+ x = np.array(x)
+ for i in range(1000):
+ fetch_map = client.predict(feed={"x": x}, fetch=["price"])
+ if fetch_map is None:
+ return [[None]]
+ return [[0]]
+
+
+multi_thread_runner = MultiThreadRunner()
+thread_num = 4
+result = multi_thread_runner.run(single_func, thread_num, {})
+if None in result[0]:
+ exit(1)
diff --git a/examples/C++/fit_a_line/test_server.py b/examples/C++/fit_a_line/test_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..d055b309d7530ccbe928d50e2bcaba23fb1ddaff
--- /dev/null
+++ b/examples/C++/fit_a_line/test_server.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_server.web_service import WebService
+import numpy as np
+
+
+class UciService(WebService):
+ def preprocess(self, feed=[], fetch=[]):
+ feed_batch = []
+ is_batch = True
+ new_data = np.zeros((len(feed), 1, 13)).astype("float32")
+ for i, ins in enumerate(feed):
+ nums = np.array(ins["x"]).reshape(1, 1, 13)
+ new_data[i] = nums
+ feed = {"x": new_data}
+ return feed, fetch, is_batch
+
+
+uci_service = UciService(name="uci")
+uci_service.load_model_config("uci_housing_model")
+uci_service.prepare_server(workdir="workdir", port=9393)
+uci_service.run_rpc_service()
+uci_service.run_web_service()
diff --git a/examples/C++/imdb/README.md b/examples/C++/imdb/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..573ac47db37d23406e66fb1605ac60ea58189ffa
--- /dev/null
+++ b/examples/C++/imdb/README.md
@@ -0,0 +1,28 @@
+## IMDB comment sentiment inference service
+
+([简体中文](./README_CN.md)|English)
+
+### Get model files and sample data
+
+```
+sh get_data.sh
+```
+the package downloaded contains cnn, lstm and bow model config along with their test_data and train_data.
+
+### Start inference service(Support BRPC-Client/GRPC-Client/Http-Client)
+
+```
+python3 -m paddle_serving_server.serve --model imdb_cnn_model/ --port 9292
+```
+### BRPC-Client Infer
+```
+head test_data/part-0 | python3 test_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
+```
+
+it will get predict results of the first 10 test cases.
+
+
+### GRPC-Client/Http-Client Infer
+```
+head test_data/part-0 | python3 test_http_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
+```
diff --git a/examples/C++/imdb/README_CN.md b/examples/C++/imdb/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..a1fecc8af35dcd2f5a38f47480b9b80b3cf96054
--- /dev/null
+++ b/examples/C++/imdb/README_CN.md
@@ -0,0 +1,26 @@
+## IMDB评论情绪预测服务
+
+(简体中文|[English](./README.md))
+
+### 获取模型文件和样例数据
+
+```
+sh get_data.sh
+```
+脚本会下载和解压出cnn、lstm和bow三种模型的配置文文件以及test_data和train_data。
+
+### 启动预测服务(支持BRPC-Client/GRPC-Client/Http-Client)
+
+```
+python3 -m paddle_serving_server.serve --model imdb_cnn_model/ --port 9292
+```
+### BRPC-Client预测
+```
+head test_data/part-0 | python3 test_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
+```
+预测test_data/part-0的前十个样例。
+
+### BRPC-Client预测
+```
+head test_data/part-0 | python3 test_http_client.py imdb_cnn_client_conf/serving_client_conf.prototxt imdb.vocab
+```
diff --git a/examples/C++/imdb/abtest_client.py b/examples/C++/imdb/abtest_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..1a14c87c355552248394fa504d37f54a4c58132a
--- /dev/null
+++ b/examples/C++/imdb/abtest_client.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+import numpy as np
+
+client = Client()
+client.load_client_config('imdb_bow_client_conf/serving_client_conf.prototxt')
+client.add_variant("bow", ["127.0.0.1:8000"], 10)
+client.add_variant("lstm", ["127.0.0.1:9000"], 90)
+client.connect()
+
+print('please wait for about 10s')
+with open('processed.data') as f:
+ cnt = {"bow": {'acc': 0, 'total': 0}, "lstm": {'acc': 0, 'total': 0}}
+ for line in f:
+ word_ids, label = line.split(';')
+ word_ids = [int(x) for x in word_ids.split(',')]
+ word_len = len(word_ids)
+ feed = {
+ "words": np.array(word_ids).reshape(word_len, 1),
+ "words.lod": [0, word_len]
+ }
+ fetch = ["acc", "cost", "prediction"]
+ [fetch_map, tag] = client.predict(
+ feed=feed, fetch=fetch, need_variant_tag=True, batch=True)
+ if (float(fetch_map["prediction"][0][1]) - 0.5) * (float(label[0]) - 0.5
+ ) > 0:
+ cnt[tag]['acc'] += 1
+ cnt[tag]['total'] += 1
+
+ for tag, data in cnt.items():
+ print('[{}](total: {}) acc: {}'.format(tag, data[
+ 'total'], float(data['acc']) / float(data['total'])))
diff --git a/examples/C++/imdb/abtest_get_data.py b/examples/C++/imdb/abtest_get_data.py
new file mode 100644
index 0000000000000000000000000000000000000000..904d23ae0217969c76c29386f3cdfe21cf075a10
--- /dev/null
+++ b/examples/C++/imdb/abtest_get_data.py
@@ -0,0 +1,24 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+imdb_dataset = IMDBDataset()
+imdb_dataset.load_resource('imdb.vocab')
+
+with open('test_data/part-0') as fin:
+ with open('processed.data', 'w') as fout:
+ for line in fin:
+ word_ids, label = imdb_dataset.get_words_and_label(line)
+ fout.write("{};{}\n".format(','.join([str(x) for x in word_ids]),
+ label[0]))
diff --git a/examples/C++/imdb/benchmark.py b/examples/C++/imdb/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..18584f88ea51373ffe2ca2e75946342c94464d76
--- /dev/null
+++ b/examples/C++/imdb/benchmark.py
@@ -0,0 +1,107 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import os
+import sys
+import time
+import requests
+import numpy as np
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import MultiThreadRunner, benchmark_args, show_latency
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ imdb_dataset = IMDBDataset()
+ imdb_dataset.load_resource("./imdb.vocab")
+ dataset = []
+ with open("./test_data/part-0") as fin:
+ for line in fin:
+ dataset.append(line.strip())
+ profile_flags = False
+ latency_flags = False
+ if os.getenv("FLAGS_profile_client"):
+ profile_flags = True
+ if os.getenv("FLAGS_serving_latency"):
+ latency_flags = True
+ latency_list = []
+ start = time.time()
+ if args.request == "rpc":
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([args.endpoint])
+ for i in range(1000):
+ if args.batch_size >= 1:
+ feed_batch = []
+ feed = {"words": [], "words.lod": [0]}
+ for bi in range(args.batch_size):
+ word_ids, label = imdb_dataset.get_words_and_label(dataset[
+ bi])
+ feed["words.lod"].append(feed["words.lod"][-1] + len(
+ word_ids))
+ feed["words"].extend(word_ids)
+ feed["words"] = np.array(feed["words"]).reshape(
+ len(feed["words"]), 1)
+ result = client.predict(
+ feed=feed, fetch=["prediction"], batch=True)
+ if result is None:
+ raise ("predict failed.")
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+
+ elif args.request == "http":
+ if args.batch_size >= 1:
+ feed_batch = []
+ for bi in range(args.batch_size):
+ feed_batch.append({"words": dataset[bi]})
+ r = requests.post(
+ "http://{}/imdb/prediction".format(args.endpoint),
+ json={"feed": feed_batch,
+ "fetch": ["prediction"]})
+ if r.status_code != 200:
+ print('HTTP status code -ne 200')
+ raise ("predict failed.")
+ else:
+ print("unsupport batch size {}".format(args.batch_size))
+ end = time.time()
+ return [[end - start]]
+
+
+if __name__ == '__main__':
+ multi_thread_runner = MultiThreadRunner()
+ endpoint_list = [
+ "127.0.0.1:9292", "127.0.0.1:9293", "127.0.0.1:9294", "127.0.0.1:9295"
+ ]
+ turns = 100
+ start = time.time()
+ result = multi_thread_runner.run(
+ single_func, args.thread, {"endpoint": endpoint_list,
+ "turns": turns})
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ for i in range(args.thread):
+ avg_cost += result[0][i]
+ avg_cost = avg_cost / args.thread
+
+ print("total cost: {}".format(total_cost))
+ print("each thread cost: {}".format(avg_cost))
+ print("qps: {}samples/s".format(args.batch_size * args.thread * turns /
+ total_cost))
+ if os.getenv("FLAGS_serving_latency"):
+ show_latency(result[0])
diff --git a/examples/C++/imdb/benchmark.sh b/examples/C++/imdb/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..33fc5dc836d75fd12f935415a4347c39beecb00a
--- /dev/null
+++ b/examples/C++/imdb/benchmark.sh
@@ -0,0 +1,43 @@
+rm profile_log*
+export FLAGS_profile_server=1
+export FLAGS_profile_client=1
+export FLAGS_serving_latency=1
+$PYTHONROOT/bin/python3 -m paddle_serving_server.serve --model $1 --port 9292 --thread 4 --mem_optim --ir_optim 2> elog > stdlog &
+hostname=`echo $(hostname)|awk -F '.baidu.com' '{print $1}'`
+#save cpu and gpu utilization log
+if [ -d utilization ];then
+ rm -rf utilization
+else
+ mkdir utilization
+fi
+sleep 5
+
+
+#warm up
+$PYTHONROOT/bin/python3 benchmark.py --thread 4 --batch_size 1 --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+echo -e "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+
+for thread_num in 1 4 8 16
+do
+for batch_size in 1 4 16 64
+do
+ job_bt=`date '+%Y%m%d%H%M%S'`
+ $PYTHONROOT/bin/python3 benchmark.py --thread $thread_num --batch_size $batch_size --model $2/serving_client_conf.prototxt --request rpc > profile 2>&1
+ echo "model_name:" $1
+ echo "thread_num:" $thread_num
+ echo "batch_size:" $batch_size
+ echo "=================Done===================="
+ echo "model_name:$1" >> profile_log_$1
+ echo "batch_size:$batch_size" >> profile_log_$1
+ job_et=`date '+%Y%m%d%H%M%S'`
+ $PYTHONROOT/bin/python3 ../../util/show_profile.py profile $thread_num >> profile_log_$1
+ $PYTHONROOT/bin/python3 cpu_utilization.py >> profile_log_$1
+ tail -n 8 profile >> profile_log_$1
+ echo "" >> profile_log_$1
+done
+done
+
+#Divided log
+awk 'BEGIN{RS="\n\n"}{i++}{print > "imdb_log_"i}' profile_log_$1
+mkdir $1_log && mv imdb_log_* $1_log
+ps -ef|grep 'serving'|grep -v grep|cut -c 9-15 | xargs kill -9
diff --git a/examples/C++/imdb/clean_data.sh b/examples/C++/imdb/clean_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..6d2c8d7a7ff195f91796483ce8caf9cc5fa0317f
--- /dev/null
+++ b/examples/C++/imdb/clean_data.sh
@@ -0,0 +1 @@
+rm -rf imdb.vocab kvdb log *.pyc serving_client_conf serving_server_model test_data text_classification_data.tar.gz train_data work_dir1
diff --git a/examples/C++/imdb/get_data.sh b/examples/C++/imdb/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..81d8d5d3b018f133c41e211d1501cf3cd9a3d8a4
--- /dev/null
+++ b/examples/C++/imdb/get_data.sh
@@ -0,0 +1,4 @@
+wget --no-check-certificate https://fleet.bj.bcebos.com/text_classification_data.tar.gz
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz
+tar -zxvf text_classification_data.tar.gz
+tar -zxvf imdb_model.tar.gz
diff --git a/examples/C++/imdb/imdb_reader.py b/examples/C++/imdb/imdb_reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..a4ef3e163a50b0dc244ac2653df1e38d7f91699b
--- /dev/null
+++ b/examples/C++/imdb/imdb_reader.py
@@ -0,0 +1,92 @@
+# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import sys
+import os
+import paddle
+import re
+import paddle.fluid.incubate.data_generator as dg
+
+py_version = sys.version_info[0]
+
+
+class IMDBDataset(dg.MultiSlotDataGenerator):
+ def load_resource(self, dictfile):
+ self._vocab = {}
+ wid = 0
+ if py_version == 2:
+ with open(dictfile) as f:
+ for line in f:
+ self._vocab[line.strip()] = wid
+ wid += 1
+ else:
+ with open(dictfile, encoding="utf-8") as f:
+ for line in f:
+ self._vocab[line.strip()] = wid
+ wid += 1
+ self._unk_id = len(self._vocab)
+ self._pattern = re.compile(r'(;|,|\.|\?|!|\s|\(|\))')
+ self.return_value = ("words", [1, 2, 3, 4, 5, 6]), ("label", [0])
+
+ def get_words_only(self, line):
+ sent = line.lower().replace("
", " ").strip()
+ words = [x for x in self._pattern.split(sent) if x and x != " "]
+ feas = [
+ self._vocab[x] if x in self._vocab else self._unk_id for x in words
+ ]
+ return feas
+
+ def get_words_and_label(self, line):
+ send = '|'.join(line.split('|')[:-1]).lower().replace("
",
+ " ").strip()
+ label = [int(line.split('|')[-1])]
+
+ words = [x for x in self._pattern.split(send) if x and x != " "]
+ feas = [
+ self._vocab[x] if x in self._vocab else self._unk_id for x in words
+ ]
+ return feas, label
+
+ def infer_reader(self, infer_filelist, batch, buf_size):
+ def local_iter():
+ for fname in infer_filelist:
+ with open(fname, "r") as fin:
+ for line in fin:
+ feas, label = self.get_words_and_label(line)
+ yield feas, label
+
+ import paddle
+ batch_iter = paddle.batch(
+ paddle.reader.shuffle(
+ local_iter, buf_size=buf_size),
+ batch_size=batch)
+ return batch_iter
+
+ def generate_sample(self, line):
+ def memory_iter():
+ for i in range(1000):
+ yield self.return_value
+
+ def data_iter():
+ feas, label = self.get_words_and_label(line)
+ yield ("words", feas), ("label", label)
+
+ return data_iter
+
+
+if __name__ == "__main__":
+ imdb = IMDBDataset()
+ imdb.load_resource("imdb.vocab")
+ imdb.run_from_stdin()
diff --git a/examples/C++/imdb/local_train.py b/examples/C++/imdb/local_train.py
new file mode 100644
index 0000000000000000000000000000000000000000..42c867abca68e15a845310a2586e6fb2b024094a
--- /dev/null
+++ b/examples/C++/imdb/local_train.py
@@ -0,0 +1,74 @@
+# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+import os
+import sys
+import paddle
+import logging
+import paddle.fluid as fluid
+
+logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger("fluid")
+logger.setLevel(logging.INFO)
+paddle.enable_static()
+
+
+def load_vocab(filename):
+ vocab = {}
+ with open(filename) as f:
+ wid = 0
+ for line in f:
+ vocab[line.strip()] = wid
+ wid += 1
+ vocab[""] = len(vocab)
+ return vocab
+
+
+if __name__ == "__main__":
+ from nets import lstm_net
+ model_name = "imdb_lstm"
+ vocab = load_vocab('imdb.vocab')
+ dict_dim = len(vocab)
+
+ data = fluid.layers.data(
+ name="words", shape=[1], dtype="int64", lod_level=1)
+ label = fluid.layers.data(name="label", shape=[1], dtype="int64")
+
+ dataset = fluid.DatasetFactory().create_dataset()
+ filelist = ["train_data/%s" % x for x in os.listdir("train_data")]
+ dataset.set_use_var([data, label])
+ pipe_command = "python imdb_reader.py"
+ dataset.set_pipe_command(pipe_command)
+ dataset.set_batch_size(128)
+ dataset.set_filelist(filelist)
+ dataset.set_thread(10)
+ avg_cost, acc, prediction = lstm_net(data, label, dict_dim)
+ optimizer = fluid.optimizer.SGD(learning_rate=0.01)
+ optimizer.minimize(avg_cost)
+
+ exe = fluid.Executor(fluid.CPUPlace())
+ exe.run(fluid.default_startup_program())
+ epochs = 6
+
+ import paddle_serving_client.io as serving_io
+
+ for i in range(epochs):
+ exe.train_from_dataset(
+ program=fluid.default_main_program(), dataset=dataset, debug=False)
+ logger.info("TRAIN --> pass: {}".format(i))
+ if i == 5:
+ serving_io.save_model("{}_model".format(model_name),
+ "{}_client_conf".format(model_name),
+ {"words": data}, {"prediction": prediction},
+ fluid.default_main_program())
diff --git a/examples/C++/imdb/nets.py b/examples/C++/imdb/nets.py
new file mode 100644
index 0000000000000000000000000000000000000000..4f2d2af6a60e97ba6c23897041a987bf26d430c7
--- /dev/null
+++ b/examples/C++/imdb/nets.py
@@ -0,0 +1,134 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing, doc-string-with-all-args, doc-string-with-returns
+
+import sys
+import time
+import numpy as np
+
+import paddle
+import paddle.fluid as fluid
+
+
+def bow_net(data,
+ label,
+ dict_dim,
+ emb_dim=128,
+ hid_dim=128,
+ hid_dim2=96,
+ class_dim=2):
+ """ bow net. """
+ emb = fluid.layers.embedding(
+ input=data, size=[dict_dim, emb_dim], is_sparse=True)
+ bow = fluid.layers.sequence_pool(input=emb, pool_type='sum')
+ bow_tanh = fluid.layers.tanh(bow)
+ fc_1 = fluid.layers.fc(input=bow_tanh, size=hid_dim, act="tanh")
+ fc_2 = fluid.layers.fc(input=fc_1, size=hid_dim2, act="tanh")
+ prediction = fluid.layers.fc(input=[fc_2], size=class_dim, act="softmax")
+ cost = fluid.layers.cross_entropy(input=prediction, label=label)
+ avg_cost = fluid.layers.mean(x=cost)
+ acc = fluid.layers.accuracy(input=prediction, label=label)
+
+ return avg_cost, acc, prediction
+
+
+def cnn_net(data,
+ label,
+ dict_dim,
+ emb_dim=128,
+ hid_dim=128,
+ hid_dim2=96,
+ class_dim=2,
+ win_size=3):
+ """ conv net. """
+ emb = fluid.layers.embedding(
+ input=data, size=[dict_dim, emb_dim], is_sparse=True)
+
+ conv_3 = fluid.nets.sequence_conv_pool(
+ input=emb,
+ num_filters=hid_dim,
+ filter_size=win_size,
+ act="tanh",
+ pool_type="max")
+
+ fc_1 = fluid.layers.fc(input=[conv_3], size=hid_dim2)
+
+ prediction = fluid.layers.fc(input=[fc_1], size=class_dim, act="softmax")
+ cost = fluid.layers.cross_entropy(input=prediction, label=label)
+ avg_cost = fluid.layers.mean(x=cost)
+ acc = fluid.layers.accuracy(input=prediction, label=label)
+
+ return avg_cost, acc, prediction
+
+
+def lstm_net(data,
+ label,
+ dict_dim,
+ emb_dim=128,
+ hid_dim=128,
+ hid_dim2=96,
+ class_dim=2,
+ emb_lr=30.0):
+ """ lstm net. """
+ emb = fluid.layers.embedding(
+ input=data,
+ size=[dict_dim, emb_dim],
+ param_attr=fluid.ParamAttr(learning_rate=emb_lr),
+ is_sparse=True)
+
+ fc0 = fluid.layers.fc(input=emb, size=hid_dim * 4)
+
+ lstm_h, c = fluid.layers.dynamic_lstm(
+ input=fc0, size=hid_dim * 4, is_reverse=False)
+
+ lstm_max = fluid.layers.sequence_pool(input=lstm_h, pool_type='max')
+ lstm_max_tanh = fluid.layers.tanh(lstm_max)
+
+ fc1 = fluid.layers.fc(input=lstm_max_tanh, size=hid_dim2, act='tanh')
+
+ prediction = fluid.layers.fc(input=fc1, size=class_dim, act='softmax')
+
+ cost = fluid.layers.cross_entropy(input=prediction, label=label)
+ avg_cost = fluid.layers.mean(x=cost)
+ acc = fluid.layers.accuracy(input=prediction, label=label)
+
+ return avg_cost, acc, prediction
+
+
+def gru_net(data,
+ label,
+ dict_dim,
+ emb_dim=128,
+ hid_dim=128,
+ hid_dim2=96,
+ class_dim=2,
+ emb_lr=400.0):
+ """ gru net. """
+ emb = fluid.layers.embedding(
+ input=data,
+ size=[dict_dim, emb_dim],
+ param_attr=fluid.ParamAttr(learning_rate=emb_lr))
+
+ fc0 = fluid.layers.fc(input=emb, size=hid_dim * 3)
+ gru_h = fluid.layers.dynamic_gru(input=fc0, size=hid_dim, is_reverse=False)
+ gru_max = fluid.layers.sequence_pool(input=gru_h, pool_type='max')
+ gru_max_tanh = fluid.layers.tanh(gru_max)
+ fc1 = fluid.layers.fc(input=gru_max_tanh, size=hid_dim2, act='tanh')
+ prediction = fluid.layers.fc(input=fc1, size=class_dim, act='softmax')
+
+ cost = fluid.layers.cross_entropy(input=prediction, label=label)
+ avg_cost = fluid.layers.mean(x=cost)
+ acc = fluid.layers.accuracy(input=prediction, label=label)
+
+ return avg_cost, acc, prediction
diff --git a/examples/C++/imdb/test_client.py b/examples/C++/imdb/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..2aeee01a83cde4a66e4bd03ad49c7791c67a287e
--- /dev/null
+++ b/examples/C++/imdb/test_client.py
@@ -0,0 +1,40 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+from paddle_serving_client import Client
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+import sys
+import numpy as np
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9292"])
+
+# you can define any english sentence or dataset here
+# This example reuses imdb reader in training, you
+# can define your own data preprocessing easily.
+imdb_dataset = IMDBDataset()
+imdb_dataset.load_resource(sys.argv[2])
+
+for line in sys.stdin:
+ word_ids, label = imdb_dataset.get_words_and_label(line)
+ word_len = len(word_ids)
+ feed = {
+ "words": np.array(word_ids).reshape(word_len, 1),
+ "words.lod": [0, word_len]
+ }
+ #print(feed)
+ fetch = ["prediction"]
+ fetch_map = client.predict(feed=feed, fetch=fetch, batch=True)
+ print("{} {}".format(fetch_map["prediction"][0], label[0]))
diff --git a/examples/C++/imdb/test_http_client.py b/examples/C++/imdb/test_http_client.py
new file mode 100755
index 0000000000000000000000000000000000000000..e3cc705150ccc197ab1be24bf11e0a92e1d62380
--- /dev/null
+++ b/examples/C++/imdb/test_http_client.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+from paddle_serving_client import HttpClient
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+import sys
+import numpy as np
+
+client = HttpClient()
+client.load_client_config(sys.argv[1])
+'''
+if you want use GRPC-client, set_use_grpc_client(True)
+or you can directly use client.grpc_client_predict(...)
+as for HTTP-client,set_use_grpc_client(False)(which is default)
+or you can directly use client.http_client_predict(...)
+'''
+#client.set_use_grpc_client(True)
+'''
+if you want to enable Encrypt Module,uncommenting the following line
+'''
+#client.use_key("./key")
+'''
+if you want to compress,uncommenting the following line
+'''
+#client.set_response_compress(True)
+#client.set_request_compress(True)
+'''
+we recommend use Proto data format in HTTP-body, set True(which is default)
+if you want use JSON data format in HTTP-body, set False
+'''
+#client.set_http_proto(True)
+client.connect(["127.0.0.1:9292"])
+
+# you can define any english sentence or dataset here
+# This example reuses imdb reader in training, you
+# can define your own data preprocessing easily.
+imdb_dataset = IMDBDataset()
+imdb_dataset.load_resource(sys.argv[2])
+
+for line in sys.stdin:
+ word_ids, label = imdb_dataset.get_words_and_label(line)
+ word_len = len(word_ids)
+ feed = {
+ "words": np.array(word_ids).reshape(word_len, 1),
+ "words.lod": [0, word_len]
+ }
+ #print(feed)
+ fetch = ["prediction"]
+ fetch_map = client.predict(feed=feed, fetch=fetch, batch=True)
+ print(fetch_map)
diff --git a/examples/C++/low_precision/resnet50/README.md b/examples/C++/low_precision/resnet50/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..b4ae2552c3dcd1c30c67b5731d81095e05ca9a86
--- /dev/null
+++ b/examples/C++/low_precision/resnet50/README.md
@@ -0,0 +1,28 @@
+# resnet50 int8 example
+(English|[简体中文](./README_CN.md))
+
+## Obtain the quantized model through PaddleSlim tool
+Train the low-precision models please refer to [PaddleSlim](https://paddleslim.readthedocs.io/zh_CN/latest/tutorials/quant/overview.html).
+
+## Deploy the quantized model from PaddleSlim using Paddle Serving with Nvidia TensorRT int8 mode
+
+Firstly, download the [Resnet50 int8 model](https://paddle-inference-dist.bj.bcebos.com/inference_demo/python/resnet50/ResNet50_quant.tar.gz) and convert to Paddle Serving's saved model。
+```
+wget https://paddle-inference-dist.bj.bcebos.com/inference_demo/python/resnet50/ResNet50_quant.tar.gz
+tar zxvf ResNet50_quant.tar.gz
+
+python3 -m paddle_serving_client.convert --dirname ResNet50_quant
+```
+Start RPC service, specify the GPU id and precision mode
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 9393 --gpu_ids 0 --use_trt --precision int8
+```
+Request the serving service with Client
+```
+python3 resnet50_client.py
+```
+
+## Reference
+* [PaddleSlim](https://github.com/PaddlePaddle/PaddleSlim)
+* [Deploy the quantized model Using Paddle Inference on Intel CPU](https://paddle-inference.readthedocs.io/en/latest/optimize/paddle_x86_cpu_int8.html)
+* [Deploy the quantized model Using Paddle Inference on Nvidia GPU](https://paddle-inference.readthedocs.io/en/latest/optimize/paddle_trt.html)
diff --git a/examples/C++/low_precision/resnet50/README_CN.md b/examples/C++/low_precision/resnet50/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..648b64dd2b0a5089ce8539c42c0222862e89d8f3
--- /dev/null
+++ b/examples/C++/low_precision/resnet50/README_CN.md
@@ -0,0 +1,27 @@
+# resnet50 int8示例
+(简体中文|[English](./README.md))
+
+## 通过PaddleSlim量化生成低精度模型
+详细见[PaddleSlim量化](https://paddleslim.readthedocs.io/zh_CN/latest/tutorials/quant/overview.html)
+
+## 使用TensorRT int8加载PaddleSlim Int8量化模型进行部署
+首先下载Resnet50 [PaddleSlim量化模型](https://paddle-inference-dist.bj.bcebos.com/inference_demo/python/resnet50/ResNet50_quant.tar.gz),并转换为Paddle Serving支持的部署模型格式。
+```
+wget https://paddle-inference-dist.bj.bcebos.com/inference_demo/python/resnet50/ResNet50_quant.tar.gz
+tar zxvf ResNet50_quant.tar.gz
+
+python3 -m paddle_serving_client.convert --dirname ResNet50_quant
+```
+启动rpc服务, 设定所选GPU id、部署模型精度
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 9393 --gpu_ids 0 --use_trt --precision int8
+```
+使用client进行请求
+```
+python3 resnet50_client.py
+```
+
+## 参考文档
+* [PaddleSlim](https://github.com/PaddlePaddle/PaddleSlim)
+* PaddleInference Intel CPU部署量化模型[文档](https://paddle-inference.readthedocs.io/en/latest/optimize/paddle_x86_cpu_int8.html)
+* PaddleInference NV GPU部署量化模型[文档](https://paddle-inference.readthedocs.io/en/latest/optimize/paddle_trt.html)
diff --git a/examples/C++/low_precision/resnet50/daisy.jpg b/examples/C++/low_precision/resnet50/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/low_precision/resnet50/daisy.jpg differ
diff --git a/examples/C++/low_precision/resnet50/resnet50_client.py b/examples/C++/low_precision/resnet50/resnet50_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..1599600dfbd2e24d7337b8f76aa753bc7444d364
--- /dev/null
+++ b/examples/C++/low_precision/resnet50/resnet50_client.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+
+client = Client()
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9393"])
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = client.predict(
+ feed={"image": img}, fetch=["save_infer_model/scale_0.tmp_0"])
+print(fetch_map["save_infer_model/scale_0.tmp_0"].reshape(-1))
diff --git a/examples/C++/xpu/bert/README.md b/examples/C++/xpu/bert/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..676ac361dde376b29ae73cae98bd34a28f55d469
--- /dev/null
+++ b/examples/C++/xpu/bert/README.md
@@ -0,0 +1,36 @@
+
+## Prepare
+### download model and extract
+```
+wget https://paddle-inference-dist.cdn.bcebos.com/PaddleLite/models_and_data_for_unittests/bert_base_chinese.tar.gz
+tar zxvf bert_base_chinese.tar.gz
+```
+### convert model
+```
+python3 -m paddle_serving_client.convert --dirname bert_base_chinese --model_filename bert_base_chinese/model.pdmodel --params_filename bert_base_chinese/model.pdiparams
+```
+### or, you can get the serving saved model directly
+```
+wget https://paddle-serving.bj.bcebos.com/models/xpu/bert.tar.gz
+tar zxvf bert.tar.gz
+```
+### Getting Dict and Sample Dataset
+
+```
+sh get_data.sh
+```
+this script will download Chinese Dictionary File vocab.txt and Chinese Sample Data data-c.txt
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 7703 --use_lite --use_xpu --ir_optim
+```
+
+### Client Prediction
+
+```
+head data-c.txt | python3 bert_client.py --model serving_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/xpu/bert/bert_client.py b/examples/C++/xpu/bert/bert_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..db10425baf1e803d7866dc839bc268cb95882eb0
--- /dev/null
+++ b/examples/C++/xpu/bert/bert_client.py
@@ -0,0 +1,37 @@
+# coding:utf-8
+# pylint: disable=doc-string-missing
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import Client
+from paddle_serving_client.utils import benchmark_args
+from chinese_bert_reader import ChineseBertReader
+import numpy as np
+args = benchmark_args()
+
+reader = ChineseBertReader({"max_seq_len": 128})
+fetch = ["save_infer_model/scale_0.tmp_1"]
+endpoint_list = ['127.0.0.1:7703']
+client = Client()
+client.load_client_config(args.model)
+client.connect(endpoint_list)
+
+for line in sys.stdin:
+ feed_dict = reader.process(line)
+ for key in feed_dict.keys():
+ feed_dict[key] = np.array(feed_dict[key]).reshape((1, 128))
+ #print(feed_dict)
+ result = client.predict(feed=feed_dict, fetch=fetch, batch=True)
+print(result)
diff --git a/examples/C++/xpu/bert/chinese_bert_reader.py b/examples/C++/xpu/bert/chinese_bert_reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..1b5cc06e78275ef2b3d63112f9a7acaf914af830
--- /dev/null
+++ b/examples/C++/xpu/bert/chinese_bert_reader.py
@@ -0,0 +1,121 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# coding=utf-8
+from paddle_serving_app.reader.bert_base_reader import BertBaseReader
+from paddle_serving_app.reader.batching import pad_batch_data
+from paddle_serving_app.reader.tokenization import FullTokenizer, convert_to_unicode
+
+
+class ChineseBertReader(BertBaseReader):
+ """
+ ChineseBertReader handles the most traditional Chinese Bert
+ preprocessing, a user can define the vocab file through initialization
+
+ Examples:
+ from paddle_serving_app import ChineseBertReader
+
+ line = ["this is China"]
+ reader = ChineseBertReader()
+ reader.process(line[0])
+
+ """
+
+ def __init__(self, args={}):
+ super(ChineseBertReader, self).__init__()
+ vocab_file = ""
+ if "vocab_file" in args:
+ vocab_file = args["vocab_file"]
+ else:
+ vocab_file = self._download_or_not()
+
+ self.tokenizer = FullTokenizer(vocab_file=vocab_file)
+ if "max_seq_len" in args:
+ self.max_seq_len = args["max_seq_len"]
+ else:
+ self.max_seq_len = 20
+ self.vocab = self.tokenizer.vocab
+ self.pad_id = self.vocab["[PAD]"]
+ self.cls_id = self.vocab["[CLS]"]
+ self.sep_id = self.vocab["[SEP]"]
+ self.mask_id = self.vocab["[MASK]"]
+ self.feed_keys = ["input_ids", "token_type_ids"]
+
+ """
+ inner function
+ """
+
+ def _download_or_not(self):
+ import os
+ import paddle_serving_app
+ module_path = os.path.dirname(paddle_serving_app.__file__)
+ full_path = "{}/tmp/chinese_bert".format(module_path)
+ os.system("mkdir -p {}".format(full_path))
+ if os.path.exists("{}/vocab.txt".format(full_path)):
+ pass
+ else:
+ url = "https://paddle-serving.bj.bcebos.com/reader/chinese_bert/vocab.txt"
+ r = os.system("wget --no-check-certificate " + url)
+ os.system("mv vocab.txt {}".format(full_path))
+ if r != 0:
+ raise SystemExit('Download failed, please check your network')
+ return "{}/vocab.txt".format(full_path)
+
+ """
+ inner function
+ """
+
+ def _pad_batch(self, token_ids, text_type_ids):
+ batch_token_ids = [token_ids]
+ batch_text_type_ids = [text_type_ids]
+
+ padded_token_ids, input_mask = pad_batch_data(
+ batch_token_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id,
+ return_input_mask=True)
+ padded_text_type_ids = pad_batch_data(
+ batch_text_type_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id)
+ return padded_token_ids, padded_text_type_ids
+
+ """
+ process function deals with a raw Chinese string as a sentence
+ this funtion returns a feed_dict
+ default key of the returned feed_dict: input_ids, position_ids, segment_ids, input_mask
+ """
+
+ def process(self, line):
+ text_a = convert_to_unicode(line)
+ tokens_a = self.tokenizer.tokenize(text_a)
+ if len(tokens_a) > self.max_seq_len - 2:
+ tokens_a = tokens_a[0:(self.max_seq_len - 2)]
+ tokens = []
+ text_type_ids = []
+ tokens.append("[CLS]")
+ text_type_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ text_type_ids.append(0)
+ token_ids = self.tokenizer.convert_tokens_to_ids(tokens)
+ #position_ids = list(range(len(token_ids)))
+ p_token_ids, p_text_type_ids= \
+ self._pad_batch(token_ids, text_type_ids)
+ feed_result = {
+ self.feed_keys[0]: p_token_ids.reshape(-1).tolist(),
+ #self.feed_keys[1]: p_pos_ids.reshape(-1).tolist(),
+ self.feed_keys[1]: p_text_type_ids.reshape(-1).tolist(),
+ #self.feed_keys[3]: input_mask.reshape(-1).tolist()
+ }
+ return feed_result
diff --git a/examples/C++/xpu/bert/get_data.sh b/examples/C++/xpu/bert/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5e17d10d144c5c9e202a5cb9c4b90a62caeeed9a
--- /dev/null
+++ b/examples/C++/xpu/bert/get_data.sh
@@ -0,0 +1,2 @@
+wget https://paddle-serving.bj.bcebos.com/bert_example/data-c.txt --no-check-certificate
+wget https://paddle-serving.bj.bcebos.com/bert_example/vocab.txt --no-check-certificate
diff --git a/examples/C++/xpu/ernie/README.md b/examples/C++/xpu/ernie/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..ef001e807531efa7a15b262f0cfdfd444e6eb66d
--- /dev/null
+++ b/examples/C++/xpu/ernie/README.md
@@ -0,0 +1,36 @@
+
+## Prepare
+### download model and extract
+```
+wget https://paddle-inference-dist.cdn.bcebos.com/PaddleLite/models_and_data_for_unittests/ernie.tar.gz
+tar zxvf ernie.tar.gz
+```
+### convert model
+```
+python3 -m paddle_serving_client.convert --dirname ernie
+```
+### or, you can get the serving saved model directly
+```
+wget https://paddle-serving.bj.bcebos.com/models/xpu/ernie.tar.gz
+tar zxvf ernie.tar.gz
+```
+### Getting Dict and Sample Dataset
+
+```
+sh get_data.sh
+```
+this script will download Chinese Dictionary File vocab.txt and Chinese Sample Data data-c.txt
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 7704 --use_lite --use_xpu --ir_optim
+```
+
+### Client Prediction
+
+```
+head data-c.txt | python3 ernie_client.py --model serving_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/xpu/ernie/chinese_ernie_reader.py b/examples/C++/xpu/ernie/chinese_ernie_reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..13e6aa672b63c6c77714427b2027efbe3b9aa458
--- /dev/null
+++ b/examples/C++/xpu/ernie/chinese_ernie_reader.py
@@ -0,0 +1,130 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# coding=utf-8
+from paddle_serving_app.reader.bert_base_reader import BertBaseReader
+from paddle_serving_app.reader.batching import pad_batch_data
+from paddle_serving_app.reader.tokenization import FullTokenizer, convert_to_unicode
+
+
+class ChineseErnieReader(BertBaseReader):
+ """
+ ChineseErnieReader handles the most traditional Chinese Bert
+ preprocessing, a user can define the vocab file through initialization
+
+ Examples:
+ from paddle_serving_app import ChineseErnieReader
+
+ line = ["this is China"]
+ reader = ChineseErnieReader()
+ reader.process(line[0])
+
+ """
+
+ def __init__(self, args={}):
+ super(ChineseErnieReader, self).__init__()
+ vocab_file = ""
+ if "vocab_file" in args:
+ vocab_file = args["vocab_file"]
+ print("vocab")
+ else:
+ vocab_file = self._download_or_not()
+
+ self.tokenizer = FullTokenizer(vocab_file=vocab_file)
+ print(self.tokenizer)
+ if "max_seq_len" in args:
+ self.max_seq_len = args["max_seq_len"]
+ else:
+ self.max_seq_len = 20
+ self.vocab = self.tokenizer.vocab
+ self.pad_id = self.vocab["[PAD]"]
+ self.cls_id = self.vocab["[CLS]"]
+ self.sep_id = self.vocab["[SEP]"]
+ self.mask_id = self.vocab["[MASK]"]
+ self.feed_keys = [
+ "placeholder_0", "placeholder_1", "placeholder_2", "placeholder_3"
+ ]
+
+ """
+ inner function
+ """
+
+ def _download_or_not(self):
+ import os
+ import paddle_serving_app
+ module_path = os.path.dirname(paddle_serving_app.__file__)
+ full_path = "{}/tmp/chinese_bert".format(module_path)
+ os.system("mkdir -p {}".format(full_path))
+ if os.path.exists("{}/vocab.txt".format(full_path)):
+ pass
+ else:
+ url = "https://paddle-serving.bj.bcebos.com/reader/chinese_bert/vocab.txt"
+ r = os.system("wget --no-check-certificate " + url)
+ os.system("mv vocab.txt {}".format(full_path))
+ if r != 0:
+ raise SystemExit('Download failed, please check your network')
+ return "{}/vocab.txt".format(full_path)
+
+ """
+ inner function
+ """
+
+ def _pad_batch(self, token_ids, text_type_ids, position_ids):
+ batch_token_ids = [token_ids]
+ batch_text_type_ids = [text_type_ids]
+ batch_position_ids = [position_ids]
+
+ padded_token_ids, input_mask = pad_batch_data(
+ batch_token_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id,
+ return_input_mask=True)
+ padded_text_type_ids = pad_batch_data(
+ batch_text_type_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id)
+ padded_position_ids = pad_batch_data(
+ batch_position_ids,
+ max_seq_len=self.max_seq_len,
+ pad_idx=self.pad_id)
+ return padded_token_ids, padded_position_ids, padded_text_type_ids, input_mask
+
+ """
+ process function deals with a raw Chinese string as a sentence
+ this funtion returns a feed_dict
+ default key of the returned feed_dict: input_ids, position_ids, segment_ids, input_mask
+ """
+
+ def process(self, line):
+ text_a = convert_to_unicode(line)
+ tokens_a = self.tokenizer.tokenize(text_a)
+ if len(tokens_a) > self.max_seq_len - 2:
+ tokens_a = tokens_a[0:(self.max_seq_len - 2)]
+ tokens = []
+ text_type_ids = []
+ tokens.append("[CLS]")
+ text_type_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ text_type_ids.append(0)
+ token_ids = self.tokenizer.convert_tokens_to_ids(tokens)
+ position_ids = list(range(len(token_ids)))
+ p_token_ids, p_pos_ids, p_text_type_ids, input_mask = \
+ self._pad_batch(token_ids, text_type_ids, position_ids)
+ feed_result = {
+ self.feed_keys[0]: p_token_ids.reshape(-1).tolist(),
+ self.feed_keys[1]: p_pos_ids.reshape(-1).tolist(),
+ self.feed_keys[2]: p_text_type_ids.reshape(-1).tolist(),
+ self.feed_keys[3]: input_mask.reshape(-1).tolist()
+ }
+ return feed_result
diff --git a/examples/C++/xpu/ernie/ernie_client.py b/examples/C++/xpu/ernie/ernie_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..69b094dff172e9bc40629a90b02d92f93fac7c2c
--- /dev/null
+++ b/examples/C++/xpu/ernie/ernie_client.py
@@ -0,0 +1,37 @@
+# coding:utf-8
+# pylint: disable=doc-string-missing
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+from paddle_serving_client import Client
+from paddle_serving_client.utils import benchmark_args
+from chinese_ernie_reader import ChineseErnieReader
+import numpy as np
+args = benchmark_args()
+
+reader = ChineseErnieReader({"max_seq_len": 128})
+fetch = ["save_infer_model/scale_0"]
+endpoint_list = ['127.0.0.1:7704']
+client = Client()
+client.load_client_config(args.model)
+client.connect(endpoint_list)
+
+for line in sys.stdin:
+ feed_dict = reader.process(line)
+ for key in feed_dict.keys():
+ feed_dict[key] = np.array(feed_dict[key]).reshape((128, 1))
+# print(feed_dict)
+ result = client.predict(feed=feed_dict, fetch=fetch, batch=False)
+ print(result)
diff --git a/examples/C++/xpu/ernie/get_data.sh b/examples/C++/xpu/ernie/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5e17d10d144c5c9e202a5cb9c4b90a62caeeed9a
--- /dev/null
+++ b/examples/C++/xpu/ernie/get_data.sh
@@ -0,0 +1,2 @@
+wget https://paddle-serving.bj.bcebos.com/bert_example/data-c.txt --no-check-certificate
+wget https://paddle-serving.bj.bcebos.com/bert_example/vocab.txt --no-check-certificate
diff --git a/examples/C++/xpu/fit_a_line_xpu/README.md b/examples/C++/xpu/fit_a_line_xpu/README.md
new file mode 100755
index 0000000000000000000000000000000000000000..2640344de82ab5c5f6a9d8e267b0496b603e91f5
--- /dev/null
+++ b/examples/C++/xpu/fit_a_line_xpu/README.md
@@ -0,0 +1,25 @@
+# Fit a line prediction example
+
+([简体中文](./README_CN.md)|English)
+
+## Get data
+
+```shell
+sh get_data.sh
+```
+
+## RPC service
+
+### Start server
+You can use the following code to start the RPC service
+```shell
+python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9393 --use_lite --use_xpu --ir_optim
+```
+
+### Client prediction
+
+The `paddlepaddle` package is used in `test_client.py`, and you may need to download the corresponding package(`pip3 install paddlepaddle`).
+
+``` shell
+python3 test_client.py uci_housing_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/xpu/fit_a_line_xpu/README_CN.md b/examples/C++/xpu/fit_a_line_xpu/README_CN.md
new file mode 100755
index 0000000000000000000000000000000000000000..268acf5a92a54a7c93541a0eaa1d3ae2f2d2656e
--- /dev/null
+++ b/examples/C++/xpu/fit_a_line_xpu/README_CN.md
@@ -0,0 +1,33 @@
+# 线性回归预测服务示例
+
+(简体中文|[English](./README.md))
+
+## 获取数据
+
+```shell
+sh get_data.sh
+```
+
+
+
+## RPC服务
+
+### 开启服务端
+
+``` shell
+python3 test_server.py uci_housing_model/
+```
+
+也可以通过下面的一行代码开启默认RPC服务：
+
+```shell
+python3 -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9393 --use_lite --use_xpu --ir_optim
+```
+
+### 客户端预测
+
+`test_client.py`中使用了`paddlepaddle`包，需要进行下载（`pip3 install paddlepaddle`）。
+
+``` shell
+python3 test_client.py uci_housing_client/serving_client_conf.prototxt
+```
diff --git a/examples/C++/xpu/fit_a_line_xpu/benchmark.py b/examples/C++/xpu/fit_a_line_xpu/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..0ddda2a095eb8542887ea592a79b16665f2daa15
--- /dev/null
+++ b/examples/C++/xpu/fit_a_line_xpu/benchmark.py
@@ -0,0 +1,57 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args
+import time
+import paddle
+import sys
+import requests
+
+args = benchmark_args()
+
+
+def single_func(idx, resource):
+ if args.request == "rpc":
+ client = Client()
+ client.load_client_config(args.model)
+ client.connect([args.endpoint])
+ train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=1)
+ start = time.time()
+ for data in train_reader():
+ fetch_map = client.predict(feed={"x": data[0][0]}, fetch=["price"])
+ end = time.time()
+ return [[end - start]]
+ elif args.request == "http":
+ train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=1)
+ start = time.time()
+ for data in train_reader():
+ r = requests.post(
+ 'http://{}/uci/prediction'.format(args.endpoint),
+ data={"x": data[0]})
+ end = time.time()
+ return [[end - start]]
+
+
+multi_thread_runner = MultiThreadRunner()
+result = multi_thread_runner.run(single_func, args.thread, {})
+print(result)
diff --git a/examples/C++/xpu/fit_a_line_xpu/get_data.sh b/examples/C++/xpu/fit_a_line_xpu/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..84a3966a0ef323cef4b146d8e9489c70a7a8ae35
--- /dev/null
+++ b/examples/C++/xpu/fit_a_line_xpu/get_data.sh
@@ -0,0 +1,2 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
+tar -xzf uci_housing.tar.gz
diff --git a/examples/C++/xpu/fit_a_line_xpu/local_train.py b/examples/C++/xpu/fit_a_line_xpu/local_train.py
new file mode 100644
index 0000000000000000000000000000000000000000..3e0f8880a4d006b346712f2592d6c44986882193
--- /dev/null
+++ b/examples/C++/xpu/fit_a_line_xpu/local_train.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import sys
+import paddle
+import paddle.fluid as fluid
+paddle.enable_static()
+train_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500),
+ batch_size=16)
+
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=16)
+
+x = fluid.data(name='x', shape=[None, 13], dtype='float32')
+y = fluid.data(name='y', shape=[None, 1], dtype='float32')
+
+y_predict = fluid.layers.fc(input=x, size=1, act=None)
+cost = fluid.layers.square_error_cost(input=y_predict, label=y)
+avg_loss = fluid.layers.mean(cost)
+sgd_optimizer = fluid.optimizer.SGD(learning_rate=0.01)
+sgd_optimizer.minimize(avg_loss)
+
+place = fluid.CPUPlace()
+feeder = fluid.DataFeeder(place=place, feed_list=[x, y])
+exe = fluid.Executor(place)
+exe.run(fluid.default_startup_program())
+
+import paddle_serving_client.io as serving_io
+
+for pass_id in range(30):
+ for data_train in train_reader():
+ avg_loss_value, = exe.run(fluid.default_main_program(),
+ feed=feeder.feed(data_train),
+ fetch_list=[avg_loss])
+
+serving_io.save_model("uci_housing_model", "uci_housing_client", {"x": x},
+ {"price": y_predict}, fluid.default_main_program())
diff --git a/examples/C++/xpu/fit_a_line_xpu/test_client.py b/examples/C++/xpu/fit_a_line_xpu/test_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..c480e81c0a4b84db7a36c2a8102f16121009f19c
--- /dev/null
+++ b/examples/C++/xpu/fit_a_line_xpu/test_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+from paddle_serving_client import Client
+import sys
+import numpy as np
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9393"])
+
+import paddle
+test_reader = paddle.batch(
+ paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500),
+ batch_size=1)
+
+for data in test_reader():
+ new_data = np.zeros((1, 1, 13)).astype("float32")
+ new_data[0] = data[0][0]
+ fetch_map = client.predict(
+ feed={"x": new_data}, fetch=["price"], batch=True)
+ print(fetch_map)
diff --git a/examples/C++/xpu/resnet_v2_50_xpu/README.md b/examples/C++/xpu/resnet_v2_50_xpu/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..76b04d614bd4513e806d9a139c38d66b8bce6569
--- /dev/null
+++ b/examples/C++/xpu/resnet_v2_50_xpu/README.md
@@ -0,0 +1,22 @@
+# Image Classification
+
+## Get Model
+
+```
+python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
+tar -xzvf resnet_v2_50_imagenet.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --port 9393 --use_lite --use_xpu --ir_optim
+```
+
+### Client Prediction
+
+```
+python3 resnet50_client.py
+```
diff --git a/examples/C++/xpu/resnet_v2_50_xpu/README_CN.md b/examples/C++/xpu/resnet_v2_50_xpu/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..652c4f672fd82b494a2240f327463e50dca8829c
--- /dev/null
+++ b/examples/C++/xpu/resnet_v2_50_xpu/README_CN.md
@@ -0,0 +1,22 @@
+# 图像分类
+
+## 获取模型
+
+```
+python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
+tar -xzvf resnet_v2_50_imagenet.tar.gz
+```
+
+## RPC 服务
+
+### 启动服务端
+
+```
+python3 -m paddle_serving_server.serve --model resnet_v2_50_imagenet_model --port 9393 --use_lite --use_xpu --ir_optim
+```
+
+### 客户端预测
+
+```
+python3 resnet50_client.py
+```
diff --git a/examples/C++/xpu/resnet_v2_50_xpu/daisy.jpg b/examples/C++/xpu/resnet_v2_50_xpu/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/xpu/resnet_v2_50_xpu/daisy.jpg differ
diff --git a/examples/C++/xpu/resnet_v2_50_xpu/localpredict.py b/examples/C++/xpu/resnet_v2_50_xpu/localpredict.py
new file mode 100644
index 0000000000000000000000000000000000000000..9042328354bc986a4905736b89e88c7139e0a3f3
--- /dev/null
+++ b/examples/C++/xpu/resnet_v2_50_xpu/localpredict.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+from paddle_serving_app.local_predict import LocalPredictor
+import sys
+
+predictor = LocalPredictor()
+predictor.load_model_config(
+ sys.argv[1], use_lite=True, use_xpu=True, ir_optim=True)
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = predictor.predict(feed={"image": img}, fetch=["score"])
+print(fetch_map["score"].reshape(-1))
diff --git a/examples/C++/xpu/resnet_v2_50_xpu/resnet50_client.py b/examples/C++/xpu/resnet_v2_50_xpu/resnet50_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..b249d2a6df85f87258f66c96aaa779eb2e299613
--- /dev/null
+++ b/examples/C++/xpu/resnet_v2_50_xpu/resnet50_client.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+
+client = Client()
+client.load_client_config(
+ "resnet_v2_50_imagenet_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:9393"])
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = client.predict(feed={"image": img}, fetch=["score"])
+print(fetch_map["score"].reshape(-1))
diff --git a/examples/C++/xpu/vgg19/README.md b/examples/C++/xpu/vgg19/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..d8520684f55a9caf88818905f4cc309f55304fe0
--- /dev/null
+++ b/examples/C++/xpu/vgg19/README.md
@@ -0,0 +1,30 @@
+
+## Prepare
+### download model and extract
+```
+wget https://paddle-inference-dist.bj.bcebos.com/PaddleLite/models_and_data_for_unittests/VGG19.tar.gz
+tar zxvf VGG19.tar.gz
+```
+### convert model
+```
+python3 -m paddle_serving_client.convert --dirname VGG19
+```
+### or, you can get the serving saved model directly
+```
+wget https://paddle-serving.bj.bcebos.com/models/xpu/vgg19.tar.gz
+tar zxvf vgg19.tar.gz
+```
+
+## RPC Service
+
+### Start Service
+
+```
+python3 -m paddle_serving_server.serve --model serving_server --port 7702 --use_lite --use_xpu --ir_optim
+```
+
+### Client Prediction
+
+```
+python3 vgg19_client.py
+```
diff --git a/examples/C++/xpu/vgg19/daisy.jpg b/examples/C++/xpu/vgg19/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/C++/xpu/vgg19/daisy.jpg differ
diff --git a/examples/C++/xpu/vgg19/vgg19_client.py b/examples/C++/xpu/vgg19/vgg19_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..913800e1fc96d86f6ee3a0f5dfc3011c02bb0922
--- /dev/null
+++ b/examples/C++/xpu/vgg19/vgg19_client.py
@@ -0,0 +1,33 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_client import Client
+from paddle_serving_app.reader import Sequential, File2Image, Resize, CenterCrop
+from paddle_serving_app.reader import RGB2BGR, Transpose, Div, Normalize
+
+client = Client()
+client.load_client_config("serving_client/serving_client_conf.prototxt")
+client.connect(["127.0.0.1:7702"])
+
+seq = Sequential([
+ File2Image(), Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], True)
+])
+
+image_file = "daisy.jpg"
+img = seq(image_file)
+fetch_map = client.predict(
+ feed={"image": img}, fetch=["save_infer_model/scale_0"])
+#print(fetch_map)
+print(fetch_map["save_infer_model/scale_0"].reshape(-1))
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/README.md b/examples/Pipeline/PaddleClas/DarkNet53/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/README_CN.md b/examples/Pipeline/PaddleClas/DarkNet53/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/benchmark.py b/examples/Pipeline/PaddleClas/DarkNet53/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..3e5db19b69fc8693adfe77a84297436bfb497642
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG_QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/benchmark.sh b/examples/Pipeline/PaddleClas/DarkNet53/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..9a27670fd2e4fb89192fc1eab5f16650c0e1a7c1
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-DarkNet53"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "#----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTIL:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "GPU_MEM:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTIL:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/benchmark_cfg.yaml b/examples/Pipeline/PaddleClas/DarkNet53/benchmark_cfg.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..f4d354e3a6ad919bd1219be20bd27ac45cc85c0f
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/benchmark_cfg.yaml
@@ -0,0 +1,29 @@
+cuda_version: "10.1"
+cudnn_version: "7.6"
+trt_version: "6.0"
+python_version: "3.7"
+gcc_version: "8.2"
+paddle_version: "2.0.1"
+
+cpu: "Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz X12"
+gpu: "T4"
+xpu: "None"
+api: ""
+owner: "cuicheng01"
+
+model_name: "imagenet"
+model_type: "static"
+model_source: "PaddleClas"
+model_url: ""
+
+batch_size: 1
+num_of_samples: 1000
+input_shape: "3,224,224"
+
+runtime_device: "cpu"
+ir_optim: true
+enable_memory_optim: true
+enable_tensorrt: false
+precision: "fp32"
+enable_mkldnn: false
+cpu_math_library_num_threads: ""
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/config.yml b/examples/Pipeline/PaddleClas/DarkNet53/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..19f14e7322e1a04d25ee1bdf931dda7645cd8cc0
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: DarkNet53/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/cpu_utilization.py b/examples/Pipeline/PaddleClas/DarkNet53/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/daisy.jpg b/examples/Pipeline/PaddleClas/DarkNet53/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/DarkNet53/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/get_model.sh b/examples/Pipeline/PaddleClas/DarkNet53/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..b19bd02d24ad82188a788c0a825616d21b6807b8
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/DarkNet53.tar
+tar -xf DarkNet53.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/imagenet.label b/examples/Pipeline/PaddleClas/DarkNet53/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/pipeline_http_client.py b/examples/Pipeline/PaddleClas/DarkNet53/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/DarkNet53/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/DarkNet53/resnet50_web_service.py b/examples/Pipeline/PaddleClas/DarkNet53/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..4fadfd7959ab548c2c88994a36604b2abb7db6d2
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/DarkNet53/resnet50_web_service.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/README.md b/examples/Pipeline/PaddleClas/HRNet_W18_C/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/README_CN.md b/examples/Pipeline/PaddleClas/HRNet_W18_C/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.py b/examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.sh b/examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..9abbd0b362e6b85dc17ac29dd60eac8b88bbbb18
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-HRNet_W18_C"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/config.yml b/examples/Pipeline/PaddleClas/HRNet_W18_C/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..63eb72169f22b1668c63c451ae74715fe5d4f076
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: HRNet_W18_C/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/cpu_utilization.py b/examples/Pipeline/PaddleClas/HRNet_W18_C/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/daisy.jpg b/examples/Pipeline/PaddleClas/HRNet_W18_C/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/HRNet_W18_C/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/get_model.sh b/examples/Pipeline/PaddleClas/HRNet_W18_C/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..73381edb1b09693150867680da9031c7cbf081ec
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/HRNet_W18_C.tar
+tar -xf HRNet_W18_C.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/imagenet.label b/examples/Pipeline/PaddleClas/HRNet_W18_C/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/pipeline_http_client.py b/examples/Pipeline/PaddleClas/HRNet_W18_C/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/HRNet_W18_C/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/HRNet_W18_C/resnet50_web_service.py b/examples/Pipeline/PaddleClas/HRNet_W18_C/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/HRNet_W18_C/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/README.md b/examples/Pipeline/PaddleClas/MobileNetV1/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/README_CN.md b/examples/Pipeline/PaddleClas/MobileNetV1/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/benchmark.py b/examples/Pipeline/PaddleClas/MobileNetV1/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/benchmark.sh b/examples/Pipeline/PaddleClas/MobileNetV1/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d10142449db8b2ad3ea39e6e7531158b833b438a
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-MobileNetV1"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/config.yml b/examples/Pipeline/PaddleClas/MobileNetV1/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..a20d2998748dd11f906a100742e7ee6b8924a083
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: MobileNetV1/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/cpu_utilization.py b/examples/Pipeline/PaddleClas/MobileNetV1/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/daisy.jpg b/examples/Pipeline/PaddleClas/MobileNetV1/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/MobileNetV1/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/get_model.sh b/examples/Pipeline/PaddleClas/MobileNetV1/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..3cc56d3a4edb2d56c1fbf80c023c2eaefffd6dca
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/MobileNetV1.tar
+tar -xf MobileNetV1.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/imagenet.label b/examples/Pipeline/PaddleClas/MobileNetV1/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/pipeline_http_client.py b/examples/Pipeline/PaddleClas/MobileNetV1/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/MobileNetV1/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/MobileNetV1/resnet50_web_service.py b/examples/Pipeline/PaddleClas/MobileNetV1/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV1/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/README.md b/examples/Pipeline/PaddleClas/MobileNetV2/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/README_CN.md b/examples/Pipeline/PaddleClas/MobileNetV2/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/benchmark.py b/examples/Pipeline/PaddleClas/MobileNetV2/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/benchmark.sh b/examples/Pipeline/PaddleClas/MobileNetV2/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..53ede7915316ef23c1208c02b624ad4a327733e6
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-MobileNetV2"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/config.yml b/examples/Pipeline/PaddleClas/MobileNetV2/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..55c2e06d6ecfafdf822b78890f5f184c872eba2e
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: MobileNetV2/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/cpu_utilization.py b/examples/Pipeline/PaddleClas/MobileNetV2/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/daisy.jpg b/examples/Pipeline/PaddleClas/MobileNetV2/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/MobileNetV2/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/get_model.sh b/examples/Pipeline/PaddleClas/MobileNetV2/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..e3262f264a2859ef508581e46e54eef3d8732684
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/MobileNetV2.tar
+tar -xf MobileNetV2.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/imagenet.label b/examples/Pipeline/PaddleClas/MobileNetV2/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/pipeline_http_client.py b/examples/Pipeline/PaddleClas/MobileNetV2/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/MobileNetV2/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/MobileNetV2/resnet50_web_service.py b/examples/Pipeline/PaddleClas/MobileNetV2/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV2/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/README.md b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/README_CN.md b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/benchmark.py b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/benchmark.sh b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..6a431f10ac7bdd05266e59318caafae4c223f427
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-MobileNetV3_large_x1_0"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/config.yml b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..f04e1e90994ab7d80598f95e3f8f7a8077fadf84
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 2
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: MobileNetV3_large_x1_0/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/cpu_utilization.py b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/daisy.jpg b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/get_model.sh b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d2135801fb5b804e37248f17a579d095f8e0afb3
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/MobileNetV3_large_x1_0.tar
+tar -xf MobileNetV3_large_x1_0.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/imagenet.label b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/pipeline_http_client.py b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/resnet50_web_service.py b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/MobileNetV3_large_x1_0/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/README.md b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/README_CN.md b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/benchmark.py b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/benchmark.sh b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..2b9ec90e65a2c58e1b4b6b5c0237c6007e49c71e
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNeXt101_vd_64x4d"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/config.yml b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..0f0facc5d6c799cfef0cd9f9797855684880b958
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ResNeXt101_vd_64x4d/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/cpu_utilization.py b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/daisy.jpg b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/get_model.sh b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..603aa286134516c4956f1078e05fda5a84d57ca1
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/ResNeXt101_vd_64x4d.tar
+tar -xf ResNeXt101_vd_64x4d.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/imagenet.label b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNeXt101_vd_64x4d/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/README.md b/examples/Pipeline/PaddleClas/ResNet50_vd/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/README_CN.md b/examples/Pipeline/PaddleClas/ResNet50_vd/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.py b/examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.sh b/examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..bc1c65244658912f6c700e3c342cec56813de4d3
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNet50_vd"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/config.yml b/examples/Pipeline/PaddleClas/ResNet50_vd/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..104504f41f98654139f5630f350a34ebd3729ba8
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ResNet50_vd/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/cpu_utilization.py b/examples/Pipeline/PaddleClas/ResNet50_vd/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/daisy.jpg b/examples/Pipeline/PaddleClas/ResNet50_vd/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ResNet50_vd/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/get_model.sh b/examples/Pipeline/PaddleClas/ResNet50_vd/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..7b4f0b243542d5c0c5cb81e50bb4b423272c730d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/ResNet50_vd.tar
+tar -xf ResNet50_vd.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/imagenet.label b/examples/Pipeline/PaddleClas/ResNet50_vd/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ResNet50_vd/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README.md b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README_CN.md b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/benchmark.py b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/benchmark.sh b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..17593214deccfa2f1af070ebd71c82775273019c
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNet50_vd_FPGM"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/config.yml b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..26087067dfb92dd95e56fb9475cb15350868e89b
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ResNet50_vd_FPGM/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["save_infer_model/scale_0.tmp_1"]
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/cpu_utilization.py b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/daisy.jpg b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/get_model.sh b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..4e1414e92c19489249414ae0904509796bfe99b6
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/ResNet50_vd_FPGM.tar
+tar -xf ResNet50_vd_FPGM.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/imagenet.label b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..43dac2a27c64d79f85f73011755c418cc6a59f1e
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_FPGM/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["save_infer_model/scale_0.tmp_1"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/README.md b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/README_CN.md b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/benchmark.py b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/benchmark.sh b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..248f9506e346aa15fd347f91f811e95657fdb66b
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNet50_vd_KL"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/config.yml b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..9ea71c67fcb0cc2ff1caef2970de80758c0729aa
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ResNet50_vd_KL/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["save_infer_model/scale_0.tmp_0"]
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/cpu_utilization.py b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/daisy.jpg b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/get_model.sh b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d81b8eb53b0317c364fcff5b9b47f9a7d6075e55
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/ResNet50_vd_KL.tar
+tar -xf ResNet50_vd_KL.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/imagenet.label b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_KL/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..569b15bcfa61a1a1732de303e2980e9b4387c9a0
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_KL/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"inputs": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["save_infer_model/scale_0.tmp_0"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README.md b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README_CN.md b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/benchmark.py b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/benchmark.sh b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..06cbe71821afe257202c5088880d467053d6d762
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNet50_vd_PACT"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/config.yml b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..22472052cbd9f46b5ed1fce0bb41c834e02597d2
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ResNet50_vd_PACT/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["save_infer_model/scale_0.tmp_1"]
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/cpu_utilization.py b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/daisy.jpg b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/get_model.sh b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d02a90c6b5aa97e757623582f5640ee923bfb6e8
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/ResNet50_vd_PACT.tar
+tar -xf ResNet50_vd_PACT.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/imagenet.label b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..debc1753cc9174dd79bf3a0072681b352c8be17b
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet50_vd_PACT/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"inputs": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["save_infer_model/scale_0.tmp_1"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/README.md b/examples/Pipeline/PaddleClas/ResNet_V2_50/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..1297abfb7a649e3eced26ea4c08848e0a51fbdbf
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/README.md
@@ -0,0 +1,20 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
+tar -xzvf resnet_v2_50_imagenet.tar.gz
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/README_CN.md b/examples/Pipeline/PaddleClas/ResNet_V2_50/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..d547b289281cb13a3abb49343b6b77230a2f3d2c
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/README_CN.md
@@ -0,0 +1,20 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+python3 -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
+tar -xzvf resnet_v2_50_imagenet.tar.gz
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.py b/examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..4b0336f97c2c520a46d596bf5e435c2b9e3094a9
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18000/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.sh b/examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..14c82dfcca801bc00bec57ef972f2260dd1d844a
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ResNet_v2_50"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/config.yml b/examples/Pipeline/PaddleClas/ResNet_V2_50/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..ead533a4b639aa84e65bc2a78b07aa95eb0075d9
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: resnet_v2_50_imagenet_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["score"]
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/daisy.jpg b/examples/Pipeline/PaddleClas/ResNet_V2_50/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ResNet_V2_50/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ResNet_V2_50/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..119a412113dac95621ef8fcad06398fed40a3da0
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/pipeline_http_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(1):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ResNet_V2_50/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..82a570244cecc51061a38b64c25602f8dfbe931d
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ResNet_V2_50/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ResNet_V2_50/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..6a7213b7abd0ddf892f64e81f96601205e5b249c
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ResNet_V2_50/resnet50_web_service.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["score"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/README.md b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/README_CN.md b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/benchmark.py b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..c80da12ce36618e75897b33d58e4f4febd382861
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/benchmark.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 1
+ config["op"]["imagenet"]["local_service_conf"]["devices"] = gpu_id
+ else:
+ config["op"]["imagenet"]["local_service_conf"]["device_type"] = 0
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ start = time.time()
+
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ keys, values = [], []
+ for i in range(batch_size):
+ keys.append("image_{}".format(i))
+ values.append(image)
+ data = {"key": keys, "value": values}
+ latency_list = []
+ start_time = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_num += 1
+ if time.time() - start_time > 20:
+ break
+ end = time.time()
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18080'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ start_time = time.time()
+ while True:
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ if time.time() - start_time > 10:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ if device == "gpu":
+ gpu_id = sys.argv[5]
+ else:
+ gpu_id = None
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/benchmark.sh b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..08523c4a5ef929ffb5ce9c30f3c1059799a4f9f0
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="clas-ShuffleNetV2_x1_0"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/config.yml b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..bdfbff08e48a414297ca375753ce8bb335d4a311
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/config.yml
@@ -0,0 +1,33 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18080
+rpc_port: 9993
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ShuffleNetV2_x1_0/ppcls_model/
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/cpu_utilization.py b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/cpu_utilization.py
new file mode 100644
index 0000000000000000000000000000000000000000..984c72370a3551722b21b8e06d418efd832192ef
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/cpu_utilization.py
@@ -0,0 +1,4 @@
+import psutil
+cpu_utilization=psutil.cpu_percent(1,False)
+print('CPU_UTILIZATION:', cpu_utilization)
+
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/daisy.jpg b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/get_model.sh b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..6a0611ce6145454db1c90c5a798d610c5b2d6888
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/model/ShuffleNetV2_x1_0.tar
+tar -xf ShuffleNetV2_x1_0.tar
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/imagenet.label b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/pipeline_http_client.py b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..bc3fab2524cbb23f2f398d3effb667d3629363c7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/pipeline_http_client.py
@@ -0,0 +1,19 @@
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+if __name__ == "__main__":
+ url = "http://127.0.0.1:18080/imagenet/prediction"
+ with open(os.path.join(".", "daisy.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..a816eb8eed49d922d5caf729dfd089fc28936853
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/pipeline_rpc_client.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9993'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/resnet50_web_service.py b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c246e45db331925e47b8d026f4801c5acf5f2ae7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/ShuffleNetV2_x1_0/resnet50_web_service.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ batch_size = len(input_dict.keys())
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ imgs.append(img[np.newaxis, :].copy())
+ input_imgs = np.concatenate(imgs, axis=0)
+ return {"image": input_imgs}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ score_list = fetch_dict["prediction"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleClas/imagenet/README.md b/examples/Pipeline/PaddleClas/imagenet/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..6fbe0c4cf3a635670341d5aee4cee8bcbdc59a88
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_model.sh
+```
+
+## Start server
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/imagenet/README_CN.md b/examples/Pipeline/PaddleClas/imagenet/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c204c3c662825ed26001cf6d444d94f0bab508f7
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_model.sh
+```
+
+## 启动服务
+
+```
+python3 resnet50_web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleClas/imagenet/config.yml b/examples/Pipeline/PaddleClas/imagenet/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..600093b32e3747f778c50a51ccebbcd9507c1a20
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/config.yml
@@ -0,0 +1,41 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18082
+rpc_port: 9999
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+op:
+ imagenet:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 2
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: ResNet50_vd_model
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "0" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["score"]
+
+ #precsion, 预测精度，降低预测精度可提升推理速度
+ #GPU 支持: "fp32"(default), "fp16", "int8"；
+ #CPU 支持: "fp32"(default), "fp16", "bf16"(mkldnn); 不支持: "int8"
+ precision: "fp16"
+
+ #ir_optim开关
+ ir_optim: False
diff --git a/examples/Pipeline/PaddleClas/imagenet/daisy.jpg b/examples/Pipeline/PaddleClas/imagenet/daisy.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..7edeca63e5f32e68550ef720d81f59df58a8eabc
Binary files /dev/null and b/examples/Pipeline/PaddleClas/imagenet/daisy.jpg differ
diff --git a/examples/Pipeline/PaddleClas/imagenet/get_model.sh b/examples/Pipeline/PaddleClas/imagenet/get_model.sh
new file mode 100644
index 0000000000000000000000000000000000000000..1964c79a2e13dbbe373636ca1a06d2967fad7b79
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/get_model.sh
@@ -0,0 +1,5 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/ResNet50_vd.tar.gz
+tar -xzvf ResNet50_vd.tar.gz
+
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imagenet-example/image_data.tar.gz
+tar -xzvf image_data.tar.gz
diff --git a/examples/Pipeline/PaddleClas/imagenet/imagenet.label b/examples/Pipeline/PaddleClas/imagenet/imagenet.label
new file mode 100644
index 0000000000000000000000000000000000000000..d7146735146ea1894173d6d0e20fb90af36be849
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/imagenet.label
@@ -0,0 +1,1000 @@
+tench, Tinca tinca,
+goldfish, Carassius auratus,
+great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias,
+tiger shark, Galeocerdo cuvieri,
+hammerhead, hammerhead shark,
+electric ray, crampfish, numbfish, torpedo,
+stingray,
+cock,
+hen,
+ostrich, Struthio camelus,
+brambling, Fringilla montifringilla,
+goldfinch, Carduelis carduelis,
+house finch, linnet, Carpodacus mexicanus,
+junco, snowbird,
+indigo bunting, indigo finch, indigo bird, Passerina cyanea,
+robin, American robin, Turdus migratorius,
+bulbul,
+jay,
+magpie,
+chickadee,
+water ouzel, dipper,
+kite,
+bald eagle, American eagle, Haliaeetus leucocephalus,
+vulture,
+great grey owl, great gray owl, Strix nebulosa,
+European fire salamander, Salamandra salamandra,
+common newt, Triturus vulgaris,
+eft,
+spotted salamander, Ambystoma maculatum,
+axolotl, mud puppy, Ambystoma mexicanum,
+bullfrog, Rana catesbeiana,
+tree frog, tree-frog,
+tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui,
+loggerhead, loggerhead turtle, Caretta caretta,
+leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea,
+mud turtle,
+terrapin,
+box turtle, box tortoise,
+banded gecko,
+common iguana, iguana, Iguana iguana,
+American chameleon, anole, Anolis carolinensis,
+whiptail, whiptail lizard,
+agama,
+frilled lizard, Chlamydosaurus kingi,
+alligator lizard,
+Gila monster, Heloderma suspectum,
+green lizard, Lacerta viridis,
+African chameleon, Chamaeleo chamaeleon,
+Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis,
+African crocodile, Nile crocodile, Crocodylus niloticus,
+American alligator, Alligator mississipiensis,
+triceratops,
+thunder snake, worm snake, Carphophis amoenus,
+ringneck snake, ring-necked snake, ring snake,
+hognose snake, puff adder, sand viper,
+green snake, grass snake,
+king snake, kingsnake,
+garter snake, grass snake,
+water snake,
+vine snake,
+night snake, Hypsiglena torquata,
+boa constrictor, Constrictor constrictor,
+rock python, rock snake, Python sebae,
+Indian cobra, Naja naja,
+green mamba,
+sea snake,
+horned viper, cerastes, sand viper, horned asp, Cerastes cornutus,
+diamondback, diamondback rattlesnake, Crotalus adamanteus,
+sidewinder, horned rattlesnake, Crotalus cerastes,
+trilobite,
+harvestman, daddy longlegs, Phalangium opilio,
+scorpion,
+black and gold garden spider, Argiope aurantia,
+barn spider, Araneus cavaticus,
+garden spider, Aranea diademata,
+black widow, Latrodectus mactans,
+tarantula,
+wolf spider, hunting spider,
+tick,
+centipede,
+black grouse,
+ptarmigan,
+ruffed grouse, partridge, Bonasa umbellus,
+prairie chicken, prairie grouse, prairie fowl,
+peacock,
+quail,
+partridge,
+African grey, African gray, Psittacus erithacus,
+macaw,
+sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita,
+lorikeet,
+coucal,
+bee eater,
+hornbill,
+hummingbird,
+jacamar,
+toucan,
+drake,
+red-breasted merganser, Mergus serrator,
+goose,
+black swan, Cygnus atratus,
+tusker,
+echidna, spiny anteater, anteater,
+platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus,
+wallaby, brush kangaroo,
+koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus,
+wombat,
+jellyfish,
+sea anemone, anemone,
+brain coral,
+flatworm, platyhelminth,
+nematode, nematode worm, roundworm,
+conch,
+snail,
+slug,
+sea slug, nudibranch,
+chiton, coat-of-mail shell, sea cradle, polyplacophore,
+chambered nautilus, pearly nautilus, nautilus,
+Dungeness crab, Cancer magister,
+rock crab, Cancer irroratus,
+fiddler crab,
+king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica,
+American lobster, Northern lobster, Maine lobster, Homarus americanus,
+spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish,
+crayfish, crawfish, crawdad, crawdaddy,
+hermit crab,
+isopod,
+white stork, Ciconia ciconia,
+black stork, Ciconia nigra,
+spoonbill,
+flamingo,
+little blue heron, Egretta caerulea,
+American egret, great white heron, Egretta albus,
+bittern,
+crane,
+limpkin, Aramus pictus,
+European gallinule, Porphyrio porphyrio,
+American coot, marsh hen, mud hen, water hen, Fulica americana,
+bustard,
+ruddy turnstone, Arenaria interpres,
+red-backed sandpiper, dunlin, Erolia alpina,
+redshank, Tringa totanus,
+dowitcher,
+oystercatcher, oyster catcher,
+pelican,
+king penguin, Aptenodytes patagonica,
+albatross, mollymawk,
+grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus,
+killer whale, killer, orca, grampus, sea wolf, Orcinus orca,
+dugong, Dugong dugon,
+sea lion,
+Chihuahua,
+Japanese spaniel,
+Maltese dog, Maltese terrier, Maltese,
+Pekinese, Pekingese, Peke,
+Shih-Tzu,
+Blenheim spaniel,
+papillon,
+toy terrier,
+Rhodesian ridgeback,
+Afghan hound, Afghan,
+basset, basset hound,
+beagle,
+bloodhound, sleuthhound,
+bluetick,
+black-and-tan coonhound,
+Walker hound, Walker foxhound,
+English foxhound,
+redbone,
+borzoi, Russian wolfhound,
+Irish wolfhound,
+Italian greyhound,
+whippet,
+Ibizan hound, Ibizan Podenco,
+Norwegian elkhound, elkhound,
+otterhound, otter hound,
+Saluki, gazelle hound,
+Scottish deerhound, deerhound,
+Weimaraner,
+Staffordshire bullterrier, Staffordshire bull terrier,
+American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier,
+Bedlington terrier,
+Border terrier,
+Kerry blue terrier,
+Irish terrier,
+Norfolk terrier,
+Norwich terrier,
+Yorkshire terrier,
+wire-haired fox terrier,
+Lakeland terrier,
+Sealyham terrier, Sealyham,
+Airedale, Airedale terrier,
+cairn, cairn terrier,
+Australian terrier,
+Dandie Dinmont, Dandie Dinmont terrier,
+Boston bull, Boston terrier,
+miniature schnauzer,
+giant schnauzer,
+standard schnauzer,
+Scotch terrier, Scottish terrier, Scottie,
+Tibetan terrier, chrysanthemum dog,
+silky terrier, Sydney silky,
+soft-coated wheaten terrier,
+West Highland white terrier,
+Lhasa, Lhasa apso,
+flat-coated retriever,
+curly-coated retriever,
+golden retriever,
+Labrador retriever,
+Chesapeake Bay retriever,
+German short-haired pointer,
+vizsla, Hungarian pointer,
+English setter,
+Irish setter, red setter,
+Gordon setter,
+Brittany spaniel,
+clumber, clumber spaniel,
+English springer, English springer spaniel,
+Welsh springer spaniel,
+cocker spaniel, English cocker spaniel, cocker,
+Sussex spaniel,
+Irish water spaniel,
+kuvasz,
+schipperke,
+groenendael,
+malinois,
+briard,
+kelpie,
+komondor,
+Old English sheepdog, bobtail,
+Shetland sheepdog, Shetland sheep dog, Shetland,
+collie,
+Border collie,
+Bouvier des Flandres, Bouviers des Flandres,
+Rottweiler,
+German shepherd, German shepherd dog, German police dog, alsatian,
+Doberman, Doberman pinscher,
+miniature pinscher,
+Greater Swiss Mountain dog,
+Bernese mountain dog,
+Appenzeller,
+EntleBucher,
+boxer,
+bull mastiff,
+Tibetan mastiff,
+French bulldog,
+Great Dane,
+Saint Bernard, St Bernard,
+Eskimo dog, husky,
+malamute, malemute, Alaskan malamute,
+Siberian husky,
+dalmatian, coach dog, carriage dog,
+affenpinscher, monkey pinscher, monkey dog,
+basenji,
+pug, pug-dog,
+Leonberg,
+Newfoundland, Newfoundland dog,
+Great Pyrenees,
+Samoyed, Samoyede,
+Pomeranian,
+chow, chow chow,
+keeshond,
+Brabancon griffon,
+Pembroke, Pembroke Welsh corgi,
+Cardigan, Cardigan Welsh corgi,
+toy poodle,
+miniature poodle,
+standard poodle,
+Mexican hairless,
+timber wolf, grey wolf, gray wolf, Canis lupus,
+white wolf, Arctic wolf, Canis lupus tundrarum,
+red wolf, maned wolf, Canis rufus, Canis niger,
+coyote, prairie wolf, brush wolf, Canis latrans,
+dingo, warrigal, warragal, Canis dingo,
+dhole, Cuon alpinus,
+African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus,
+hyena, hyaena,
+red fox, Vulpes vulpes,
+kit fox, Vulpes macrotis,
+Arctic fox, white fox, Alopex lagopus,
+grey fox, gray fox, Urocyon cinereoargenteus,
+tabby, tabby cat,
+tiger cat,
+Persian cat,
+Siamese cat, Siamese,
+Egyptian cat,
+cougar, puma, catamount, mountain lion, painter, panther, Felis concolor,
+lynx, catamount,
+leopard, Panthera pardus,
+snow leopard, ounce, Panthera uncia,
+jaguar, panther, Panthera onca, Felis onca,
+lion, king of beasts, Panthera leo,
+tiger, Panthera tigris,
+cheetah, chetah, Acinonyx jubatus,
+brown bear, bruin, Ursus arctos,
+American black bear, black bear, Ursus americanus, Euarctos americanus,
+ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus,
+sloth bear, Melursus ursinus, Ursus ursinus,
+mongoose,
+meerkat, mierkat,
+tiger beetle,
+ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle,
+ground beetle, carabid beetle,
+long-horned beetle, longicorn, longicorn beetle,
+leaf beetle, chrysomelid,
+dung beetle,
+rhinoceros beetle,
+weevil,
+fly,
+bee,
+ant, emmet, pismire,
+grasshopper, hopper,
+cricket,
+walking stick, walkingstick, stick insect,
+cockroach, roach,
+mantis, mantid,
+cicada, cicala,
+leafhopper,
+lacewing, lacewing fly,
+"dragonfly, darning needle, devils darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
+damselfly,
+admiral,
+ringlet, ringlet butterfly,
+monarch, monarch butterfly, milkweed butterfly, Danaus plexippus,
+cabbage butterfly,
+sulphur butterfly, sulfur butterfly,
+lycaenid, lycaenid butterfly,
+starfish, sea star,
+sea urchin,
+sea cucumber, holothurian,
+wood rabbit, cottontail, cottontail rabbit,
+hare,
+Angora, Angora rabbit,
+hamster,
+porcupine, hedgehog,
+fox squirrel, eastern fox squirrel, Sciurus niger,
+marmot,
+beaver,
+guinea pig, Cavia cobaya,
+sorrel,
+zebra,
+hog, pig, grunter, squealer, Sus scrofa,
+wild boar, boar, Sus scrofa,
+warthog,
+hippopotamus, hippo, river horse, Hippopotamus amphibius,
+ox,
+water buffalo, water ox, Asiatic buffalo, Bubalus bubalis,
+bison,
+ram, tup,
+bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis,
+ibex, Capra ibex,
+hartebeest,
+impala, Aepyceros melampus,
+gazelle,
+Arabian camel, dromedary, Camelus dromedarius,
+llama,
+weasel,
+mink,
+polecat, fitch, foulmart, foumart, Mustela putorius,
+black-footed ferret, ferret, Mustela nigripes,
+otter,
+skunk, polecat, wood pussy,
+badger,
+armadillo,
+three-toed sloth, ai, Bradypus tridactylus,
+orangutan, orang, orangutang, Pongo pygmaeus,
+gorilla, Gorilla gorilla,
+chimpanzee, chimp, Pan troglodytes,
+gibbon, Hylobates lar,
+siamang, Hylobates syndactylus, Symphalangus syndactylus,
+guenon, guenon monkey,
+patas, hussar monkey, Erythrocebus patas,
+baboon,
+macaque,
+langur,
+colobus, colobus monkey,
+proboscis monkey, Nasalis larvatus,
+marmoset,
+capuchin, ringtail, Cebus capucinus,
+howler monkey, howler,
+titi, titi monkey,
+spider monkey, Ateles geoffroyi,
+squirrel monkey, Saimiri sciureus,
+Madagascar cat, ring-tailed lemur, Lemur catta,
+indri, indris, Indri indri, Indri brevicaudatus,
+Indian elephant, Elephas maximus,
+African elephant, Loxodonta africana,
+lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens,
+giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca,
+barracouta, snoek,
+eel,
+coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch,
+rock beauty, Holocanthus tricolor,
+anemone fish,
+sturgeon,
+gar, garfish, garpike, billfish, Lepisosteus osseus,
+lionfish,
+puffer, pufferfish, blowfish, globefish,
+abacus,
+abaya,
+"academic gown, academic robe, judges robe",
+accordion, piano accordion, squeeze box,
+acoustic guitar,
+aircraft carrier, carrier, flattop, attack aircraft carrier,
+airliner,
+airship, dirigible,
+altar,
+ambulance,
+amphibian, amphibious vehicle,
+analog clock,
+apiary, bee house,
+apron,
+ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin,
+assault rifle, assault gun,
+backpack, back pack, knapsack, packsack, rucksack, haversack,
+bakery, bakeshop, bakehouse,
+balance beam, beam,
+balloon,
+ballpoint, ballpoint pen, ballpen, Biro,
+Band Aid,
+banjo,
+bannister, banister, balustrade, balusters, handrail,
+barbell,
+barber chair,
+barbershop,
+barn,
+barometer,
+barrel, cask,
+barrow, garden cart, lawn cart, wheelbarrow,
+baseball,
+basketball,
+bassinet,
+bassoon,
+bathing cap, swimming cap,
+bath towel,
+bathtub, bathing tub, bath, tub,
+beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon,
+beacon, lighthouse, beacon light, pharos,
+beaker,
+bearskin, busby, shako,
+beer bottle,
+beer glass,
+bell cote, bell cot,
+bib,
+bicycle-built-for-two, tandem bicycle, tandem,
+bikini, two-piece,
+binder, ring-binder,
+binoculars, field glasses, opera glasses,
+birdhouse,
+boathouse,
+bobsled, bobsleigh, bob,
+bolo tie, bolo, bola tie, bola,
+bonnet, poke bonnet,
+bookcase,
+bookshop, bookstore, bookstall,
+bottlecap,
+bow,
+bow tie, bow-tie, bowtie,
+brass, memorial tablet, plaque,
+brassiere, bra, bandeau,
+breakwater, groin, groyne, mole, bulwark, seawall, jetty,
+breastplate, aegis, egis,
+broom,
+bucket, pail,
+buckle,
+bulletproof vest,
+bullet train, bullet,
+butcher shop, meat market,
+cab, hack, taxi, taxicab,
+caldron, cauldron,
+candle, taper, wax light,
+cannon,
+canoe,
+can opener, tin opener,
+cardigan,
+car mirror,
+carousel, carrousel, merry-go-round, roundabout, whirligig,
+"carpenters kit, tool kit",
+carton,
+car wheel,
+cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM,
+cassette,
+cassette player,
+castle,
+catamaran,
+CD player,
+cello, violoncello,
+cellular telephone, cellular phone, cellphone, cell, mobile phone,
+chain,
+chainlink fence,
+chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour,
+chain saw, chainsaw,
+chest,
+chiffonier, commode,
+chime, bell, gong,
+china cabinet, china closet,
+Christmas stocking,
+church, church building,
+cinema, movie theater, movie theatre, movie house, picture palace,
+cleaver, meat cleaver, chopper,
+cliff dwelling,
+cloak,
+clog, geta, patten, sabot,
+cocktail shaker,
+coffee mug,
+coffeepot,
+coil, spiral, volute, whorl, helix,
+combination lock,
+computer keyboard, keypad,
+confectionery, confectionary, candy store,
+container ship, containership, container vessel,
+convertible,
+corkscrew, bottle screw,
+cornet, horn, trumpet, trump,
+cowboy boot,
+cowboy hat, ten-gallon hat,
+cradle,
+crane,
+crash helmet,
+crate,
+crib, cot,
+Crock Pot,
+croquet ball,
+crutch,
+cuirass,
+dam, dike, dyke,
+desk,
+desktop computer,
+dial telephone, dial phone,
+diaper, nappy, napkin,
+digital clock,
+digital watch,
+dining table, board,
+dishrag, dishcloth,
+dishwasher, dish washer, dishwashing machine,
+disk brake, disc brake,
+dock, dockage, docking facility,
+dogsled, dog sled, dog sleigh,
+dome,
+doormat, welcome mat,
+drilling platform, offshore rig,
+drum, membranophone, tympan,
+drumstick,
+dumbbell,
+Dutch oven,
+electric fan, blower,
+electric guitar,
+electric locomotive,
+entertainment center,
+envelope,
+espresso maker,
+face powder,
+feather boa, boa,
+file, file cabinet, filing cabinet,
+fireboat,
+fire engine, fire truck,
+fire screen, fireguard,
+flagpole, flagstaff,
+flute, transverse flute,
+folding chair,
+football helmet,
+forklift,
+fountain,
+fountain pen,
+four-poster,
+freight car,
+French horn, horn,
+frying pan, frypan, skillet,
+fur coat,
+garbage truck, dustcart,
+gasmask, respirator, gas helmet,
+gas pump, gasoline pump, petrol pump, island dispenser,
+goblet,
+go-kart,
+golf ball,
+golfcart, golf cart,
+gondola,
+gong, tam-tam,
+gown,
+grand piano, grand,
+greenhouse, nursery, glasshouse,
+grille, radiator grille,
+grocery store, grocery, food market, market,
+guillotine,
+hair slide,
+hair spray,
+half track,
+hammer,
+hamper,
+hand blower, blow dryer, blow drier, hair dryer, hair drier,
+hand-held computer, hand-held microcomputer,
+handkerchief, hankie, hanky, hankey,
+hard disc, hard disk, fixed disk,
+harmonica, mouth organ, harp, mouth harp,
+harp,
+harvester, reaper,
+hatchet,
+holster,
+home theater, home theatre,
+honeycomb,
+hook, claw,
+hoopskirt, crinoline,
+horizontal bar, high bar,
+horse cart, horse-cart,
+hourglass,
+iPod,
+iron, smoothing iron,
+"jack-o-lantern",
+jean, blue jean, denim,
+jeep, landrover,
+jersey, T-shirt, tee shirt,
+jigsaw puzzle,
+jinrikisha, ricksha, rickshaw,
+joystick,
+kimono,
+knee pad,
+knot,
+lab coat, laboratory coat,
+ladle,
+lampshade, lamp shade,
+laptop, laptop computer,
+lawn mower, mower,
+lens cap, lens cover,
+letter opener, paper knife, paperknife,
+library,
+lifeboat,
+lighter, light, igniter, ignitor,
+limousine, limo,
+liner, ocean liner,
+lipstick, lip rouge,
+Loafer,
+lotion,
+loudspeaker, speaker, speaker unit, loudspeaker system, speaker system,
+"loupe, jewelers loupe",
+lumbermill, sawmill,
+magnetic compass,
+mailbag, postbag,
+mailbox, letter box,
+maillot,
+maillot, tank suit,
+manhole cover,
+maraca,
+marimba, xylophone,
+mask,
+matchstick,
+maypole,
+maze, labyrinth,
+measuring cup,
+medicine chest, medicine cabinet,
+megalith, megalithic structure,
+microphone, mike,
+microwave, microwave oven,
+military uniform,
+milk can,
+minibus,
+miniskirt, mini,
+minivan,
+missile,
+mitten,
+mixing bowl,
+mobile home, manufactured home,
+Model T,
+modem,
+monastery,
+monitor,
+moped,
+mortar,
+mortarboard,
+mosque,
+mosquito net,
+motor scooter, scooter,
+mountain bike, all-terrain bike, off-roader,
+mountain tent,
+mouse, computer mouse,
+mousetrap,
+moving van,
+muzzle,
+nail,
+neck brace,
+necklace,
+nipple,
+notebook, notebook computer,
+obelisk,
+oboe, hautboy, hautbois,
+ocarina, sweet potato,
+odometer, hodometer, mileometer, milometer,
+oil filter,
+organ, pipe organ,
+oscilloscope, scope, cathode-ray oscilloscope, CRO,
+overskirt,
+oxcart,
+oxygen mask,
+packet,
+paddle, boat paddle,
+paddlewheel, paddle wheel,
+padlock,
+paintbrush,
+"pajama, pyjama, pjs, jammies",
+palace,
+panpipe, pandean pipe, syrinx,
+paper towel,
+parachute, chute,
+parallel bars, bars,
+park bench,
+parking meter,
+passenger car, coach, carriage,
+patio, terrace,
+pay-phone, pay-station,
+pedestal, plinth, footstall,
+pencil box, pencil case,
+pencil sharpener,
+perfume, essence,
+Petri dish,
+photocopier,
+pick, plectrum, plectron,
+pickelhaube,
+picket fence, paling,
+pickup, pickup truck,
+pier,
+piggy bank, penny bank,
+pill bottle,
+pillow,
+ping-pong ball,
+pinwheel,
+pirate, pirate ship,
+pitcher, ewer,
+"plane, carpenters plane, woodworking plane",
+planetarium,
+plastic bag,
+plate rack,
+plow, plough,
+"plunger, plumbers helper",
+Polaroid camera, Polaroid Land camera,
+pole,
+police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria,
+poncho,
+pool table, billiard table, snooker table,
+pop bottle, soda bottle,
+pot, flowerpot,
+"potters wheel",
+power drill,
+prayer rug, prayer mat,
+printer,
+prison, prison house,
+projectile, missile,
+projector,
+puck, hockey puck,
+punching bag, punch bag, punching ball, punchball,
+purse,
+quill, quill pen,
+quilt, comforter, comfort, puff,
+racer, race car, racing car,
+racket, racquet,
+radiator,
+radio, wireless,
+radio telescope, radio reflector,
+rain barrel,
+recreational vehicle, RV, R.V.,
+reel,
+reflex camera,
+refrigerator, icebox,
+remote control, remote,
+restaurant, eating house, eating place, eatery,
+revolver, six-gun, six-shooter,
+rifle,
+rocking chair, rocker,
+rotisserie,
+rubber eraser, rubber, pencil eraser,
+rugby ball,
+rule, ruler,
+running shoe,
+safe,
+safety pin,
+saltshaker, salt shaker,
+sandal,
+sarong,
+sax, saxophone,
+scabbard,
+scale, weighing machine,
+school bus,
+schooner,
+scoreboard,
+screen, CRT screen,
+screw,
+screwdriver,
+seat belt, seatbelt,
+sewing machine,
+shield, buckler,
+shoe shop, shoe-shop, shoe store,
+shoji,
+shopping basket,
+shopping cart,
+shovel,
+shower cap,
+shower curtain,
+ski,
+ski mask,
+sleeping bag,
+slide rule, slipstick,
+sliding door,
+slot, one-armed bandit,
+snorkel,
+snowmobile,
+snowplow, snowplough,
+soap dispenser,
+soccer ball,
+sock,
+solar dish, solar collector, solar furnace,
+sombrero,
+soup bowl,
+space bar,
+space heater,
+space shuttle,
+spatula,
+speedboat,
+"spider web, spiders web",
+spindle,
+sports car, sport car,
+spotlight, spot,
+stage,
+steam locomotive,
+steel arch bridge,
+steel drum,
+stethoscope,
+stole,
+stone wall,
+stopwatch, stop watch,
+stove,
+strainer,
+streetcar, tram, tramcar, trolley, trolley car,
+stretcher,
+studio couch, day bed,
+stupa, tope,
+submarine, pigboat, sub, U-boat,
+suit, suit of clothes,
+sundial,
+sunglass,
+sunglasses, dark glasses, shades,
+sunscreen, sunblock, sun blocker,
+suspension bridge,
+swab, swob, mop,
+sweatshirt,
+swimming trunks, bathing trunks,
+swing,
+switch, electric switch, electrical switch,
+syringe,
+table lamp,
+tank, army tank, armored combat vehicle, armoured combat vehicle,
+tape player,
+teapot,
+teddy, teddy bear,
+television, television system,
+tennis ball,
+thatch, thatched roof,
+theater curtain, theatre curtain,
+thimble,
+thresher, thrasher, threshing machine,
+throne,
+tile roof,
+toaster,
+tobacco shop, tobacconist shop, tobacconist,
+toilet seat,
+torch,
+totem pole,
+tow truck, tow car, wrecker,
+toyshop,
+tractor,
+trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi,
+tray,
+trench coat,
+tricycle, trike, velocipede,
+trimaran,
+tripod,
+triumphal arch,
+trolleybus, trolley coach, trackless trolley,
+trombone,
+tub, vat,
+turnstile,
+typewriter keyboard,
+umbrella,
+unicycle, monocycle,
+upright, upright piano,
+vacuum, vacuum cleaner,
+vase,
+vault,
+velvet,
+vending machine,
+vestment,
+viaduct,
+violin, fiddle,
+volleyball,
+waffle iron,
+wall clock,
+wallet, billfold, notecase, pocketbook,
+wardrobe, closet, press,
+warplane, military plane,
+washbasin, handbasin, washbowl, lavabo, wash-hand basin,
+washer, automatic washer, washing machine,
+water bottle,
+water jug,
+water tower,
+whiskey jug,
+whistle,
+wig,
+window screen,
+window shade,
+Windsor tie,
+wine bottle,
+wing,
+wok,
+wooden spoon,
+wool, woolen, woollen,
+worm fence, snake fence, snake-rail fence, Virginia fence,
+wreck,
+yawl,
+yurt,
+web site, website, internet site, site,
+comic book,
+crossword puzzle, crossword,
+street sign,
+traffic light, traffic signal, stoplight,
+book jacket, dust cover, dust jacket, dust wrapper,
+menu,
+plate,
+guacamole,
+consomme,
+hot pot, hotpot,
+trifle,
+ice cream, icecream,
+ice lolly, lolly, lollipop, popsicle,
+French loaf,
+bagel, beigel,
+pretzel,
+cheeseburger,
+hotdog, hot dog, red hot,
+mashed potato,
+head cabbage,
+broccoli,
+cauliflower,
+zucchini, courgette,
+spaghetti squash,
+acorn squash,
+butternut squash,
+cucumber, cuke,
+artichoke, globe artichoke,
+bell pepper,
+cardoon,
+mushroom,
+Granny Smith,
+strawberry,
+orange,
+lemon,
+fig,
+pineapple, ananas,
+banana,
+jackfruit, jak, jack,
+custard apple,
+pomegranate,
+hay,
+carbonara,
+chocolate sauce, chocolate syrup,
+dough,
+meat loaf, meatloaf,
+pizza, pizza pie,
+potpie,
+burrito,
+red wine,
+espresso,
+cup,
+eggnog,
+alp,
+bubble,
+cliff, drop, drop-off,
+coral reef,
+geyser,
+lakeside, lakeshore,
+promontory, headland, head, foreland,
+sandbar, sand bar,
+seashore, coast, seacoast, sea-coast,
+valley, vale,
+volcano,
+ballplayer, baseball player,
+groom, bridegroom,
+scuba diver,
+rapeseed,
+daisy,
+"yellow ladys slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
+corn,
+acorn,
+hip, rose hip, rosehip,
+buckeye, horse chestnut, conker,
+coral fungus,
+agaric,
+gyromitra,
+stinkhorn, carrion fungus,
+earthstar,
+hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa,
+bolete,
+ear, spike, capitulum,
+toilet tissue, toilet paper, bathroom tissue
diff --git a/examples/Pipeline/PaddleClas/imagenet/pipeline_rpc_client.py b/examples/Pipeline/PaddleClas/imagenet/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..573ef4c8f7c17727f08aa35e3184773e0857192b
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/pipeline_rpc_client.py
@@ -0,0 +1,39 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+try:
+ from paddle_serving_server.pipeline import PipelineClient
+except ImportError:
+ from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9999'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+with open("daisy.jpg", 'rb') as file:
+ image_data = file.read()
+image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["label", "prob"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleClas/imagenet/resnet50_web_service.py b/examples/Pipeline/PaddleClas/imagenet/resnet50_web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..a4d37ed600a8eb90836b83f33f0cbe32e35d5008
--- /dev/null
+++ b/examples/Pipeline/PaddleClas/imagenet/resnet50_web_service.py
@@ -0,0 +1,68 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+from paddle_serving_app.reader import Sequential, URL2Image, Resize, CenterCrop, RGB2BGR, Transpose, Div, Normalize, Base64ToImage
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import base64, cv2
+
+
+class ImagenetOp(Op):
+ def init_op(self):
+ self.seq = Sequential([
+ Resize(256), CenterCrop(224), RGB2BGR(), Transpose((2, 0, 1)),
+ Div(255), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225],
+ True)
+ ])
+ self.label_dict = {}
+ label_idx = 0
+ with open("imagenet.label") as fin:
+ for line in fin:
+ self.label_dict[label_idx] = line.strip()
+ label_idx += 1
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ data = base64.b64decode(input_dict["image"].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ # Note: class variables(self.var) can only be used in process op mode
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ img = self.seq(im)
+ return {"image": img[np.newaxis, :].copy()}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ print(fetch_dict)
+ score_list = fetch_dict["score"]
+ result = {"label": [], "prob": []}
+ for score in score_list:
+ score = score.tolist()
+ max_score = max(score)
+ result["label"].append(self.label_dict[score.index(max_score)]
+ .strip().replace(",", ""))
+ result["prob"].append(max_score)
+ result["label"] = str(result["label"])
+ result["prob"] = str(result["prob"])
+ return result, None, ""
+
+
+class ImageService(WebService):
+ def get_pipeline_response(self, read_op):
+ image_op = ImagenetOp(name="imagenet", input_ops=[read_op])
+ return image_op
+
+
+uci_service = ImageService(name="imagenet")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/000000570688.jpg b/examples/Pipeline/PaddleDetection/faster_rcnn/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/Pipeline/PaddleDetection/faster_rcnn/000000570688.jpg differ
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/README.md b/examples/Pipeline/PaddleDetection/faster_rcnn/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..a56ecbef06d82eef59510a1242de7f19c0915d55
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/README.md
@@ -0,0 +1,18 @@
+# Faster RCNN model on Pipeline Paddle Serving
+
+### Get The Faster RCNN Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/faster_rcnn_r50_fpn_1x_coco.tar
+```
+
+### Start the service
+```
+tar xf faster_rcnn_r50_fpn_1x_coco.tar
+python3 web_service.py
+```
+
+### Perform prediction
+
+```
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark.py b/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..f8d5f2b4fd196048a139867a893b06f47d2778bb
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark.py
@@ -0,0 +1,132 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import yaml
+import requests
+import time
+import json
+import cv2
+import base64
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 30}
+ if device == "gpu":
+ config["op"]["faster_rcnn"]["local_service_conf"]["device_type"] = 1
+ config["op"]["faster_rcnn"]["local_service_conf"]["devices"] = gpu_id
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18082/faster_rcnn/prediction"
+ with open(os.path.join(".", "000000570688.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ latency_list = []
+ start = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ data = {"key": [], "value": []}
+ for j in range(batch_size):
+ data["key"].append("image_" + str(j))
+ data["value"].append(image)
+ r = requests.post(url=url, data=json.dumps(data))
+ l_end = time.time()
+ total_num += 1
+ end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ if end - start > 70:
+ #print("70s end")
+ break
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ pass
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ gpu_id = sys.argv[5]
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark.sh b/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5247891f64f7a669a20e2ad19fa1f4cb94e1fb17
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="det-FasterRCNN"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark_config.yaml b/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark_config.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..9252a27b72136eafeaf8ca833e4fad1d8587d9c5
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/benchmark_config.yaml
@@ -0,0 +1,32 @@
+
+cuda_version: "10.1"
+cudnn_version: "7.6"
+trt_version: "6.0"
+python_version: "3.7"
+gcc_version: "8.2"
+paddle_version: "2.0.2"
+
+cpu: "Xeon 6148"
+gpu: "P4"
+xpu: "None"
+api: ""
+owner: "wangjiawei04"
+
+model_name: "faster_rcnn"
+model_type: "static"
+model_source: "paddledetection"
+model_url: ""
+
+batch_size: 1
+num_of_samples: 1000
+input_shape: "3, 480, 640"
+
+runtime_device: "gpu"
+ir_optim: true
+enable_memory_optim: true
+enable_tensorrt: false
+precision: "fp32"
+enable_mkldnn: true
+cpu_math_library_num_threads: ""
+
+
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/config.yml b/examples/Pipeline/PaddleDetection/faster_rcnn/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..891b4b997c2ebb98d6694464b5dbe0532c01145c
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/config.yml
@@ -0,0 +1,29 @@
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: false
+ #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ tracer:
+ interval_s: 30
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18082
+op:
+ faster_rcnn:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 2
+ local_service_conf:
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: '2'
+ #Fetch结果列表，以bert_seq128_model中fetch_var的alias_name为准, 如果没有设置则全部返回
+ fetch_list:
+ - save_infer_model/scale_0.tmp_1
+ #模型路径
+ model_config: serving_server/
+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 9998
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+#当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/label_list.txt b/examples/Pipeline/PaddleDetection/faster_rcnn/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/pipeline_http_client.py b/examples/Pipeline/PaddleDetection/faster_rcnn/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..7037afc2f328d4a348e108a6bbeba7a2a60032af
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/pipeline_http_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+url = "http://127.0.0.1:18082/faster_rcnn/prediction"
+with open(os.path.join(".", "000000570688.jpg"), 'rb') as file:
+ image_data1 = file.read()
+image = cv2_to_base64(image_data1)
+
+for i in range(1):
+ data = {"key": ["image"], "value": [image]}
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleDetection/faster_rcnn/web_service.py b/examples/Pipeline/PaddleDetection/faster_rcnn/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..08a9122296b801689f3d5faf2c75113b293ea220
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/faster_rcnn/web_service.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import sys
+import cv2
+from paddle_serving_app.reader import *
+import base64
+
+
+class FasterRCNNOp(Op):
+ def init_op(self):
+ self.img_preprocess = Sequential([
+ BGR2RGB(), Div(255.0),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], False),
+ Resize(640, 640), Transpose((2, 0, 1))
+ ])
+ self.img_postprocess = RCNNPostprocess("label_list.txt", "output")
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ imgs = []
+ #print("keys", input_dict.keys())
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ im = self.img_preprocess(im)
+ imgs.append({
+ "image": im[np.newaxis, :],
+ "im_shape":
+ np.array(list(im.shape[1:])).reshape(-1)[np.newaxis, :],
+ "scale_factor": np.array([1.0, 1.0]).reshape(-1)[np.newaxis, :],
+ })
+ feed_dict = {
+ "image": np.concatenate(
+ [x["image"] for x in imgs], axis=0),
+ "im_shape": np.concatenate(
+ [x["im_shape"] for x in imgs], axis=0),
+ "scale_factor": np.concatenate(
+ [x["scale_factor"] for x in imgs], axis=0)
+ }
+ #for key in feed_dict.keys():
+ # print(key, feed_dict[key].shape)
+ return feed_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ #print(fetch_dict)
+ res_dict = {
+ "bbox_result":
+ str(self.img_postprocess(
+ fetch_dict, visualize=False))
+ }
+ return res_dict, None, ""
+
+
+class FasterRCNNService(WebService):
+ def get_pipeline_response(self, read_op):
+ faster_rcnn_op = FasterRCNNOp(name="faster_rcnn", input_ops=[read_op])
+ return faster_rcnn_op
+
+
+fasterrcnn_service = FasterRCNNService(name="faster_rcnn")
+fasterrcnn_service.prepare_pipeline_config("config.yml")
+fasterrcnn_service.run_service()
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/000000570688.jpg b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/000000570688.jpg differ
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/README.md b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..73087efca7abc75d9ed7d6178d962911b9a2b1cb
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/README.md
@@ -0,0 +1,19 @@
+# PPYOLO model on Pipeline Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_mbv3_large_coco.tar
+```
+
+### Start the service
+```
+tar xf ppyolo_mbv3_large_coco.tar
+python3 web_service.py
+```
+
+### Perform prediction
+```
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/README_CN.md b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..a852a9229ae6ca228814bfc8afa2485ed8c4bb5d
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/README_CN.md
@@ -0,0 +1,19 @@
+# PPYOLO model on Pipeline Paddle Serving
+
+(简体中文|[English](./README_CN.md))
+
+### 获取模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/ppyolo_mbv3_large_coco.tar
+```
+
+### 启动服务
+```
+tar xf ppyolo_mbv3_large_coco.tar
+python3 web_service.py
+```
+
+### 执行预测
+```
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark.py b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..611712b6754efd88fc7b51027e99b9bb3e82cf7d
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark.py
@@ -0,0 +1,132 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import yaml
+import requests
+import time
+import json
+import cv2
+import base64
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 30}
+ if device == "gpu":
+ config["op"]["ppyolo_mbv3"]["local_service_conf"]["device_type"] = 1
+ config["op"]["ppyolo_mbv3"]["local_service_conf"]["devices"] = gpu_id
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18082/ppyolo_mbv3/prediction"
+ with open(os.path.join(".", "000000570688.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ latency_list = []
+ start = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ data = {"key": [], "value": []}
+ for j in range(batch_size):
+ data["key"].append("image_" + str(j))
+ data["value"].append(image)
+ r = requests.post(url=url, data=json.dumps(data))
+ l_end = time.time()
+ total_num += 1
+ end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ if end - start > 70:
+ #print("70s end")
+ break
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ pass
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ gpu_id = sys.argv[5]
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark.sh b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..7237a52ab73630141785f47a143d6931f75d0c17
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="det-PPYoloMbv3"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark_config.yaml b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark_config.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..64b38c2bed55d236a7b2eb02b49b4847c80fc716
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/benchmark_config.yaml
@@ -0,0 +1,32 @@
+
+cuda_version: "10.1"
+cudnn_version: "7.6"
+trt_version: "6.0"
+python_version: "3.7"
+gcc_version: "8.2"
+paddle_version: "2.0.2"
+
+cpu: "Xeon 6148"
+gpu: "P4"
+xpu: "None"
+api: ""
+owner: "wangjiawei04"
+
+model_name: "ppyolo"
+model_type: "static"
+model_source: "paddledetection"
+model_url: ""
+
+batch_size: 1
+num_of_samples: 1000
+input_shape: "3, 480, 640"
+
+runtime_device: "gpu"
+ir_optim: true
+enable_memory_optim: true
+enable_tensorrt: false
+precision: "fp32"
+enable_mkldnn: true
+cpu_math_library_num_threads: ""
+
+
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/config.yml b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..71e93f39c7979522e73058af7fa2969575b5129c
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/config.yml
@@ -0,0 +1,30 @@
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: false
+ #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ tracer:
+ interval_s: 30
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18082
+op:
+ ppyolo_mbv3:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 10
+
+ local_service_conf:
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: '2'
+ #Fetch结果列表，以bert_seq128_model中fetch_var的alias_name为准, 如果没有设置则全部返回
+ fetch_list:
+ - save_infer_model/scale_0.tmp_1
+ #模型路径
+ model_config: serving_server/
+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 9998
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+#当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/label_list.txt b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/pipeline_http_client.py b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..4106fdf061c34a6ee6be5d5419793ee83d4f59f4
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/pipeline_http_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+url = "http://127.0.0.1:18082/ppyolo_mbv3/prediction"
+with open(os.path.join(".", "000000570688.jpg"), 'rb') as file:
+ image_data1 = file.read()
+image = cv2_to_base64(image_data1)
+
+for i in range(1):
+ data = {"key": ["image"], "value": [image]}
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleDetection/ppyolo_mbv3/web_service.py b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..1cfa0aee793d1a6fa22f109284c426b1e7676e0b
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/ppyolo_mbv3/web_service.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import sys
+import cv2
+from paddle_serving_app.reader import *
+import base64
+
+
+class PPYoloMbvOp(Op):
+ def init_op(self):
+ self.img_preprocess = Sequential([
+ BGR2RGB(), Div(255.0),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], False),
+ Resize((320, 320)), Transpose((2, 0, 1))
+ ])
+ self.img_postprocess = RCNNPostprocess("label_list.txt", "output")
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ imgs = []
+ #print("keys", input_dict.keys())
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ im = self.img_preprocess(im)
+ imgs.append({
+ "image": im[np.newaxis, :],
+ "im_shape":
+ np.array(list(im.shape[1:])).reshape(-1)[np.newaxis, :],
+ "scale_factor": np.array([1.0, 1.0]).reshape(-1)[np.newaxis, :],
+ })
+
+ feed_dict = {
+ "image": np.concatenate(
+ [x["image"] for x in imgs], axis=0),
+ "im_shape": np.concatenate(
+ [x["im_shape"] for x in imgs], axis=0),
+ "scale_factor": np.concatenate(
+ [x["scale_factor"] for x in imgs], axis=0)
+ }
+ for key in feed_dict.keys():
+ print(key, feed_dict[key].shape)
+ return feed_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ #print(fetch_dict)
+ res_dict = {
+ "bbox_result":
+ str(self.img_postprocess(
+ fetch_dict, visualize=False))
+ }
+ return res_dict, None, ""
+
+
+class PPYoloMbv(WebService):
+ def get_pipeline_response(self, read_op):
+ ppyolo_mbv3_op = PPYoloMbvOp(name="ppyolo_mbv3", input_ops=[read_op])
+ return ppyolo_mbv3_op
+
+
+ppyolo_mbv3_service = PPYoloMbv(name="ppyolo_mbv3")
+ppyolo_mbv3_service.prepare_pipeline_config("config.yml")
+ppyolo_mbv3_service.run_service()
diff --git a/examples/Pipeline/PaddleDetection/yolov3/000000570688.jpg b/examples/Pipeline/PaddleDetection/yolov3/000000570688.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..cb304bd56c4010c08611a30dcca58ea9140cea54
Binary files /dev/null and b/examples/Pipeline/PaddleDetection/yolov3/000000570688.jpg differ
diff --git a/examples/Pipeline/PaddleDetection/yolov3/README.md b/examples/Pipeline/PaddleDetection/yolov3/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..8340f1060d0be6b100575ecbcb0270db0a6227f4
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/README.md
@@ -0,0 +1,19 @@
+# YOLOv3 model on Pipeline Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+### Get Model
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/yolov3_darknet53_270e_coco.tar
+```
+
+### Start the service
+```
+tar xf yolov3_darknet53_270e_coco.tar
+python3 web_service.py
+```
+
+### Perform prediction
+```
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/PaddleDetection/yolov3/README_CN.md b/examples/Pipeline/PaddleDetection/yolov3/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..aff60651378255cb1e6c40ce6bf3e26b913029cb
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/README_CN.md
@@ -0,0 +1,19 @@
+# YOLOv3 model on Pipeline Paddle Serving
+
+(简体中文|[English](./README.md))
+
+### 获取模型
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/pddet_demo/2.0/yolov3_darknet53_270e_coco.tar
+```
+
+### 启动 WebService
+```
+tar xf yolov3_darknet53_270e_coco.tar
+python3 web_service.py
+```
+
+### 执行预测
+```
+python3 pipeline_http_client.py
+```
diff --git a/examples/Pipeline/PaddleDetection/yolov3/benchmark.py b/examples/Pipeline/PaddleDetection/yolov3/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..cb73d2f932c12d0559af307b3ecf12ecf7986390
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/benchmark.py
@@ -0,0 +1,132 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import yaml
+import requests
+import time
+import json
+import cv2
+import base64
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device, gpu_id):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 30}
+ if device == "gpu":
+ config["op"]["yolov3"]["local_service_conf"]["device_type"] = 1
+ config["op"]["yolov3"]["local_service_conf"]["devices"] = gpu_id
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18082/yolov3/prediction"
+ with open(os.path.join(".", "000000570688.jpg"), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ latency_list = []
+ start = time.time()
+ total_num = 0
+ while True:
+ l_start = time.time()
+ data = {"key": [], "value": []}
+ for j in range(batch_size):
+ data["key"].append("image_" + str(j))
+ data["value"].append(image)
+ r = requests.post(url=url, data=json.dumps(data))
+ l_end = time.time()
+ total_num += 1
+ end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ if end - start > 70:
+ #print("70s end")
+ break
+ return [[end - start], latency_list, [total_num]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ pass
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ gpu_id = sys.argv[5]
+ gen_yml(device, gpu_id)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleDetection/yolov3/benchmark.sh b/examples/Pipeline/PaddleDetection/yolov3/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..e3ac2f79398feea88a1d547047af6db0691c1581
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/benchmark.sh
@@ -0,0 +1,44 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.6"
+modelname="det-yolov3"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----${modelname} thread num: ${thread_num} batch size: ${batch_size} mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleDetection/yolov3/benchmark_config.yaml b/examples/Pipeline/PaddleDetection/yolov3/benchmark_config.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..055e96c7780fffe0f1152a124c0a7585f615bf47
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/benchmark_config.yaml
@@ -0,0 +1,32 @@
+
+cuda_version: "10.1"
+cudnn_version: "7.6"
+trt_version: "6.0"
+python_version: "3.7"
+gcc_version: "8.2"
+paddle_version: "2.0.2"
+
+cpu: "Xeon 6148"
+gpu: "P4"
+xpu: "None"
+api: ""
+owner: "wangjiawei04"
+
+model_name: "yolov3"
+model_type: "static"
+model_source: "paddledetection"
+model_url: ""
+
+batch_size: 1
+num_of_samples: 1000
+input_shape: "3, 480, 640"
+
+runtime_device: "gpu"
+ir_optim: true
+enable_memory_optim: true
+enable_tensorrt: false
+precision: "fp32"
+enable_mkldnn: true
+cpu_math_library_num_threads: ""
+
+
diff --git a/examples/Pipeline/PaddleDetection/yolov3/config.yml b/examples/Pipeline/PaddleDetection/yolov3/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..0f6d839edd3467e4dca203b9a21db850db3f4d5e
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/config.yml
@@ -0,0 +1,29 @@
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: false
+ #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ tracer:
+ interval_s: 30
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18082
+op:
+ yolov3:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 10
+ local_service_conf:
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: '2'
+ #Fetch结果列表，以bert_seq128_model中fetch_var的alias_name为准, 如果没有设置则全部返回
+ fetch_list:
+ - save_infer_model/scale_0.tmp_1
+ #模型路径
+ model_config: serving_server/
+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 9998
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+#当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
diff --git a/examples/Pipeline/PaddleDetection/yolov3/label_list.txt b/examples/Pipeline/PaddleDetection/yolov3/label_list.txt
new file mode 100644
index 0000000000000000000000000000000000000000..941cb4e1392266f6a6c09b1fdc5f79503b2e5df6
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/label_list.txt
@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorcycle
+airplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+couch
+potted plant
+bed
+dining table
+toilet
+tv
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush
diff --git a/examples/Pipeline/PaddleDetection/yolov3/pipeline_http_client.py b/examples/Pipeline/PaddleDetection/yolov3/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..5ef29ba5ad333307cacd9b9b82569dd046cf20b4
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/pipeline_http_client.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+url = "http://127.0.0.1:18082/yolov3/prediction"
+with open(os.path.join(".", "000000570688.jpg"), 'rb') as file:
+ image_data1 = file.read()
+image = cv2_to_base64(image_data1)
+
+for i in range(1):
+ data = {"key": ["image"], "value": [image]}
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleDetection/yolov3/web_service.py b/examples/Pipeline/PaddleDetection/yolov3/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..fa55f78067118184ae5b5541c1bc1fe36db617a0
--- /dev/null
+++ b/examples/Pipeline/PaddleDetection/yolov3/web_service.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import sys
+import cv2
+from paddle_serving_app.reader import *
+import base64
+
+
+class Yolov3Op(Op):
+ def init_op(self):
+ self.img_preprocess = Sequential([
+ BGR2RGB(), Div(255.0),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], False),
+ Resize((640, 640)), Transpose((2, 0, 1))
+ ])
+ self.img_postprocess = RCNNPostprocess("label_list.txt", "output")
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ imgs = []
+ #print("keys", input_dict.keys())
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ data = np.fromstring(data, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ im = self.img_preprocess(im)
+ imgs.append({
+ "image": im[np.newaxis, :],
+ "im_shape":
+ np.array(list(im.shape[1:])).reshape(-1)[np.newaxis, :],
+ "scale_factor": np.array([1.0, 1.0]).reshape(-1)[np.newaxis, :],
+ })
+ feed_dict = {
+ "image": np.concatenate(
+ [x["image"] for x in imgs], axis=0),
+ "im_shape": np.concatenate(
+ [x["im_shape"] for x in imgs], axis=0),
+ "scale_factor": np.concatenate(
+ [x["scale_factor"] for x in imgs], axis=0)
+ }
+ #for key in feed_dict.keys():
+ # print(key, feed_dict[key].shape)
+ return feed_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ #print(fetch_dict)
+ res_dict = {
+ "bbox_result":
+ str(self.img_postprocess(
+ fetch_dict, visualize=False))
+ }
+ return res_dict, None, ""
+
+
+class Yolov3Service(WebService):
+ def get_pipeline_response(self, read_op):
+ yolov3_op = Yolov3Op(name="yolov3", input_ops=[read_op])
+ return yolov3_op
+
+
+yolov3_service = Yolov3Service(name="yolov3")
+yolov3_service.prepare_pipeline_config("config.yml")
+yolov3_service.run_service()
diff --git a/examples/Pipeline/PaddleNLP/bert/README.md b/examples/Pipeline/PaddleNLP/bert/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..c396b77c9d2b9198d0474540872cb1c4dcdce5b1
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/README.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+This document will takes Imagenet service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_data.sh
+```
+
+## Start server
+
+```
+python3 web_service.py &>log.txt &
+```
+
+## RPC test
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleNLP/bert/README_CN.md b/examples/Pipeline/PaddleNLP/bert/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..841abdadf5a3848fcf1e042d8e73c051610eefaa
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/README_CN.md
@@ -0,0 +1,19 @@
+# Imagenet Pipeline WebService
+
+这里以 Imagenet 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_data.sh
+```
+
+## 启动服务
+
+```
+python3 web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_rpc_client.py
+```
diff --git a/examples/Pipeline/PaddleNLP/bert/benchmark.py b/examples/Pipeline/PaddleNLP/bert/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..ccdbbdf599943ebf757d336b96d4f19b92e1b94a
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/benchmark.py
@@ -0,0 +1,151 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+'''
+2021-03-16 10:26:01,832 ==================== TRACER ======================
+2021-03-16 10:26:01,838 Op(bert):
+2021-03-16 10:26:01,838 in[5.7833 ms]
+2021-03-16 10:26:01,838 prep[8.2001 ms]
+2021-03-16 10:26:01,838 midp[198.79853333333332 ms]
+2021-03-16 10:26:01,839 postp[0.8411 ms]
+2021-03-16 10:26:01,839 out[0.9440666666666667 ms]
+2021-03-16 10:26:01,839 idle[0.03135320683677345]
+2021-03-16 10:26:01,839 DAGExecutor:
+2021-03-16 10:26:01,839 Query count[30]
+2021-03-16 10:26:01,839 QPS[3.0 q/s]
+2021-03-16 10:26:01,839 Succ[1.0]
+2021-03-16 10:26:01,839 Error req[]
+2021-03-16 10:26:01,839 Latency:
+2021-03-16 10:26:01,839 ave[237.85519999999997 ms]
+2021-03-16 10:26:01,839 .50[179.937 ms]
+2021-03-16 10:26:01,839 .60[179.994 ms]
+2021-03-16 10:26:01,839 .70[180.515 ms]
+2021-03-16 10:26:01,840 .80[180.735 ms]
+2021-03-16 10:26:01,840 .90[182.275 ms]
+2021-03-16 10:26:01,840 .95[182.789 ms]
+2021-03-16 10:26:01,840 .99[1921.33 ms]
+2021-03-16 10:26:01,840 Channel (server worker num[1]):
+2021-03-16 10:26:01,840 chl0(In: ['@DAGExecutor'], Out: ['bert']) size[0/0]
+2021-03-16 10:26:01,841 chl1(In: ['bert'], Out: ['@DAGExecutor']) size[0/0]
+'''
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["bert"]["local_service_conf"]["device_type"] = 1
+ config["op"]["bert"]["local_service_conf"]["devices"] = "2"
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18082/bert/prediction"
+ start = time.time()
+ with open("data-c.txt", 'r') as fin:
+ start = time.time()
+ lines = fin.readlines()
+ start_idx = 0
+ while start_idx < len(lines):
+ end_idx = min(len(lines), start_idx + batch_size)
+ feed = {}
+ for i in range(start_idx, end_idx):
+ feed[str(i - start_idx)] = lines[i]
+ keys = list(feed.keys())
+ values = [feed[x] for x in keys]
+ data = {"key": keys, "value": values}
+ r = requests.post(url=url, data=json.dumps(data))
+ start_idx += batch_size
+ if start_idx > 2000:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:9998'])
+ with open("data-c.txt", 'r') as fin:
+ start = time.time()
+ lines = fin.readlines()
+ start_idx = 0
+ while start_idx < len(lines):
+ end_idx = min(len(lines), start_idx + batch_size)
+ feed = {}
+ for i in range(start_idx, end_idx):
+ feed[str(i - start_idx)] = lines[i]
+ ret = client.predict(feed_dict=feed, fetch=["res"])
+ start_idx += batch_size
+ if start_idx > 1000:
+ break
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ gen_yml(device)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleNLP/bert/benchmark.sh b/examples/Pipeline/PaddleNLP/bert/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..bff0fd8e4eb429efd37f7267d959f5e0f600d8ce
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/benchmark.sh
@@ -0,0 +1,59 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.7"
+modelname="bert"
+# HTTP
+ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+for thread_num in 1 8 16
+do
+ for batch_size in 1 10 100
+ do
+ echo "----Bert thread num: $thread_num batch size: $batch_size mode:http ----" >>profile_log_$modelname
+ rm -rf PipelineServingLogs
+ rm -rf cpu_utilization.py
+ python3 web_service.py >web.log 2>&1 &
+ sleep 3
+ nvidia-smi --id=2 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_use.log 2>&1 &
+ nvidia-smi --id=2 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ python3 benchmark.py run http $thread_num $batch_size
+ python3 cpu_utilization.py >>profile_log_$modelname
+ ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+ python3 benchmark.py dump benchmark.log benchmark.tmp
+ mv benchmark.tmp benchmark.log
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$modelname
+ cat benchmark.log >> profile_log_$modelname
+ #rm -rf gpu_use.log gpu_utilization.log
+ done
+done
+# RPC
+ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+python3 benchmark.py yaml local_predictor 1 gpu
+
+for thread_num in 1 8 16
+do
+ for batch_size in 1 10 100
+ do
+ echo "----Bert thread num: $thread_num batch size: $batch_size mode:rpc ----" >>profile_log_$modelname
+ rm -rf PipelineServingLogs
+ rm -rf cpu_utilization.py
+ python3 web_service.py >web.log 2>&1 &
+ sleep 3
+ nvidia-smi --id=2 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_use.log 2>&1 &
+ nvidia-smi --id=2 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ python3 benchmark.py run rpc $thread_num $batch_size
+ python3 cpu_utilization.py >>profile_log_$modelname
+ ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+ python3 benchmark.py dump benchmark.log benchmark.tmp
+ mv benchmark.tmp benchmark.log
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "GPU_UTILIZATION:", max}' gpu_utilization.log >> profile_log_$modelname
+ #rm -rf gpu_use.log gpu_utilization.log
+ cat benchmark.log >> profile_log_$modelname
+ done
+done
diff --git a/examples/Pipeline/PaddleNLP/bert/config.yml b/examples/Pipeline/PaddleNLP/bert/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..5f1226646bb1a14fee3460bc98e25321b6aaa27a
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/config.yml
@@ -0,0 +1,32 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
+#build_dag_each_worker, False，框架在进程内创建一条DAG；True，框架会每个进程内创建多个独立的DAG
+build_dag_each_worker: false
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: false
+ #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ tracer:
+ interval_s: 10
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18082
+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 9998
+op:
+ bert:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 2
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 1
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: '2'
+ #Fetch结果列表，以bert_seq128_model中fetch_var的alias_name为准, 如果没有设置则全部返回
+ fetch_list:
+ #bert模型路径
+ model_config: bert_seq128_model/
diff --git a/examples/Pipeline/PaddleNLP/bert/get_data.sh b/examples/Pipeline/PaddleNLP/bert/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..b10a69afdf4cc22898c8081d3a0e1e1ccd0bfdfe
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/get_data.sh
@@ -0,0 +1,6 @@
+wget https://paddle-serving.bj.bcebos.com/paddle_hub_models/text/SemanticModel/bert_chinese_L-12_H-768_A-12.tar.gz
+tar -xzf bert_chinese_L-12_H-768_A-12.tar.gz
+mv bert_chinese_L-12_H-768_A-12_model bert_seq128_model
+mv bert_chinese_L-12_H-768_A-12_client bert_seq128_client
+wget https://paddle-serving.bj.bcebos.com/bert_example/data-c.txt --no-check-certificate
+wget https://paddle-serving.bj.bcebos.com/bert_example/vocab.txt --no-check-certificate
diff --git a/examples/Pipeline/PaddleNLP/bert/pipeline_rpc_client.py b/examples/Pipeline/PaddleNLP/bert/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..1ce1998a07670a30dff197d654d2192201c12928
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/pipeline_rpc_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import yaml
+import requests
+import time
+import json
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+
+client = PipelineClient()
+client.connect(['127.0.0.1:9998'])
+batch_size = 101
+with open("data-c.txt", 'r') as fin:
+ lines = fin.readlines()
+ start_idx = 0
+ while start_idx < len(lines):
+ end_idx = min(len(lines), start_idx + batch_size)
+ feed = {}
+ for i in range(start_idx, end_idx):
+ feed[str(i - start_idx)] = lines[i]
+ ret = client.predict(feed_dict=feed, fetch=["res"])
+ print(ret)
+ start_idx += batch_size
diff --git a/examples/Pipeline/PaddleNLP/bert/web_service.py b/examples/Pipeline/PaddleNLP/bert/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..46495a850886c6bc9f33a117a1485ec0d2ea6d9a
--- /dev/null
+++ b/examples/Pipeline/PaddleNLP/bert/web_service.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import sys
+from paddle_serving_app.reader import ChineseBertReader
+_LOGGER = logging.getLogger()
+
+
+class BertOp(Op):
+ def init_op(self):
+ self.reader = ChineseBertReader({
+ "vocab_file": "vocab.txt",
+ "max_seq_len": 128
+ })
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ print("input dict", input_dict)
+ batch_size = len(input_dict.keys())
+ feed_res = []
+ for i in range(batch_size):
+ feed_dict = self.reader.process(input_dict[str(i)].encode("utf-8"))
+ for key in feed_dict.keys():
+ feed_dict[key] = np.array(feed_dict[key]).reshape(
+ (1, len(feed_dict[key]), 1))
+ feed_res.append(feed_dict)
+ feed_dict = {}
+ for key in feed_res[0].keys():
+ feed_dict[key] = np.concatenate([x[key] for x in feed_res], axis=0)
+ print(key, feed_dict[key].shape)
+ return feed_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ new_dict = {}
+ new_dict["pooled_output"] = str(fetch_dict["pooled_output"])
+ new_dict["sequence_output"] = str(fetch_dict["sequence_output"])
+ return new_dict, None, ""
+
+
+class BertService(WebService):
+ def get_pipeline_response(self, read_op):
+ bert_op = BertOp(name="bert", input_ops=[read_op])
+ return bert_op
+
+
+bert_service = BertService(name="bert")
+bert_service.prepare_pipeline_config("config.yml")
+bert_service.run_service()
diff --git a/examples/Pipeline/PaddleOCR/ocr/README.md b/examples/Pipeline/PaddleOCR/ocr/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..4c669b3cc19e6afb7a0c5178a7b251bd4e47ff31
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/README.md
@@ -0,0 +1,60 @@
+# OCR Pipeline WebService
+
+(English|[简体中文](./README_CN.md))
+
+This document will take OCR as an example to show how to use Pipeline WebService to start multi-model tandem services.
+
+This OCR example only supports Process OP.
+
+## Get Model
+```
+python3 -m paddle_serving_app.package --get_model ocr_rec
+tar -xzvf ocr_rec.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_det
+tar -xzvf ocr_det.tar.gz
+```
+
+## Get Dataset (Optional)
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/ocr/test_imgs.tar
+tar xf test_imgs.tar
+```
+
+## Run services
+
+### 1.Start a single server and client.
+```
+python3 web_service.py &>log.txt &
+```
+
+Test
+```
+python3 pipeline_http_client.py
+```
+
+
+
+
+### 2.Run benchmark
+```
+python3 web_service.py &>log.txt &
+```
+
+Test
+```
+sh benchmark.sh
+```
diff --git a/examples/Pipeline/PaddleOCR/ocr/README_CN.md b/examples/Pipeline/PaddleOCR/ocr/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..c6c2064a684c175b816887cf65e19e7a90f927fa
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/README_CN.md
@@ -0,0 +1,60 @@
+# OCR Pipeline WebService
+
+([English](./README.md)|简体中文)
+
+本文档将以 OCR 为例，介绍如何使用 Pipeline WebService 启动多模型串联的服务。
+本示例仅支持进程OP模式。
+
+## 获取模型
+```
+python3 -m paddle_serving_app.package --get_model ocr_rec
+tar -xzvf ocr_rec.tar.gz
+python3 -m paddle_serving_app.package --get_model ocr_det
+tar -xzvf ocr_det.tar.gz
+```
+
+## 获取数据集（可选）
+```
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/ocr/test_imgs.tar
+tar xf test_imgs.tar
+```
+
+## 启动 WebService
+
+### 1.启动单server、单client
+```
+python3 web_service.py &>log.txt &
+```
+
+## 测试
+```
+python3 pipeline_http_client.py
+```
+
+
+
+### 2.启动 benchmark
+```
+python3 web_service.py &>log.txt &
+```
+
+Test
+```
+sh benchmark.sh
+```
diff --git a/examples/Pipeline/PaddleOCR/ocr/benchmark.py b/examples/Pipeline/PaddleOCR/ocr/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..3c1243a1c327a5f94544c7fa56524321cad2892f
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/benchmark.py
@@ -0,0 +1,163 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import base64
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def parse_benchmark(filein, fileout):
+ with open(filein, "r") as fin:
+ res = yaml.load(fin, yaml.FullLoader)
+ del_list = []
+ for key in res["DAG"].keys():
+ if "call" in key:
+ del_list.append(key)
+ for key in del_list:
+ del res["DAG"][key]
+ with open(fileout, "w") as fout:
+ yaml.dump(res, fout, default_flow_style=False)
+
+
+def gen_yml(device):
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 10}
+ if device == "gpu":
+ config["op"]["det"]["local_service_conf"]["device_type"] = 1
+ config["op"]["det"]["local_service_conf"]["devices"] = "2"
+ config["op"]["rec"]["local_service_conf"]["device_type"] = 1
+ config["op"]["rec"]["local_service_conf"]["devices"] = "2"
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:9999/ocr/prediction"
+ start = time.time()
+ test_img_dir = "imgs/"
+ #test_img_dir = "rctw_test/images/"
+ latency_list = []
+ total_number = 0
+ for img_file in os.listdir(test_img_dir):
+ l_start = time.time()
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+ data = {"key": ["image"], "value": [image]}
+ #for i in range(100):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ end = time.time()
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_number = total_number + 1
+ return [[end - start], latency_list, [total_number]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:18090'])
+ start = time.time()
+ test_img_dir = "imgs/"
+ #test_img_dir = "rctw_test/images/"
+ latency_list = []
+ total_number = 0
+ for img_file in os.listdir(test_img_dir):
+ l_start = time.time()
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ print(ret)
+ l_end = time.time()
+ latency_list.append(l_end * 1000 - l_start * 1000)
+ total_number = total_number + 1
+ end = time.time()
+ return [[end - start], latency_list, [total_number]]
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ start = time.time()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+ end = time.time()
+ total_cost = end - start
+ avg_cost = 0
+ total_number = 0
+ for i in range(thread):
+ avg_cost += result[0][i]
+ total_number += result[2][i]
+ avg_cost = avg_cost / thread
+ print("Total cost: {}s".format(total_cost))
+ print("Each thread cost: {}s. ".format(avg_cost))
+ print("Total count: {}. ".format(total_number))
+ print("AVG QPS: {} samples/s".format(batch_size * total_number /
+ total_cost))
+ show_latency(result[1])
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ device = sys.argv[4]
+ gen_yml(device)
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
+ elif sys.argv[1] == "dump":
+ filein = sys.argv[2]
+ fileout = sys.argv[3]
+ parse_benchmark(filein, fileout)
diff --git a/examples/Pipeline/PaddleOCR/ocr/benchmark.sh b/examples/Pipeline/PaddleOCR/ocr/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..ae6027f5e552d79f5f6152dcdf9312bcf31e4576
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/benchmark.sh
@@ -0,0 +1,88 @@
+export FLAGS_profile_pipeline=1
+alias python3="python3.7"
+modelname="ocr"
+
+# HTTP
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+rm -rf profile_log_$modelname
+
+echo "Starting HTTP Clients..."
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 6 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----$modelname thread num: $thread_num batch size: $batch_size mode:http ----" >>profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-gpu=memory.used --format=csv -lms 1000 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 1000 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+ # Start http client
+ python3 benchmark.py run http $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F' ' '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo '' >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
+
+echo "Starting RPC Clients..."
+
+# RPC
+#ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+
+# Create yaml，If you already have the config.yaml, ignore it.
+#python3 benchmark.py yaml local_predictor 1 gpu
+#rm -rf profile_log_$modelname
+
+# Start a client in each thread, tesing the case of multiple threads.
+for thread_num in 1 2 4 6 8 12 16
+do
+ for batch_size in 1
+ do
+ echo "----$modelname thread num: $thread_num batch size: $batch_size mode:rpc ----" >> profile_log_$modelname
+ # Start one web service, If you start the service yourself, you can ignore it here.
+ #python3 web_service.py >web.log 2>&1 &
+ #sleep 3
+
+ # --id is the serial number of the GPU card, Must be the same as the gpu id used by the server.
+ nvidia-smi --id=3 --query-compute-apps=used_memory --format=csv -lms 100 > gpu_use.log 2>&1 &
+ nvidia-smi --id=3 --query-gpu=utilization.gpu --format=csv -lms 100 > gpu_utilization.log 2>&1 &
+ echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+
+ # Start http client
+ python3 benchmark.py run rpc $thread_num $batch_size > profile 2>&1
+
+ # Collect CPU metrics, Filter data that is zero momentarily, Record the maximum value of GPU memory and the average value of GPU utilization
+ python3 cpu_utilization.py >> profile_log_$modelname
+ grep -av '^0 %' gpu_utilization.log > gpu_utilization.log.tmp
+ awk 'BEGIN {max = 0} {if(NR>1){if ($modelname > max) max=$modelname}} END {print "MAX_GPU_MEMORY:", max}' gpu_use.log >> profile_log_$modelname
+ awk -F" " '{sum+=$1} END {print "GPU_UTILIZATION:", sum/NR, sum, NR }' gpu_utilization.log.tmp >> profile_log_$modelname
+
+ # Show profiles
+ python3 ../../../util/show_profile.py profile $thread_num >> profile_log_$modelname
+ tail -n 8 profile >> profile_log_$modelname
+ echo "" >> profile_log_$modelname
+ done
+done
+
+# Kill all nvidia-smi background task.
+pkill nvidia-smi
diff --git a/examples/Pipeline/PaddleOCR/ocr/config.yml b/examples/Pipeline/PaddleOCR/ocr/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..2767fa77ceaa975c4e20bedaaf13ffa0e2b35de3
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/config.yml
@@ -0,0 +1,90 @@
+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 18090
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 9999
+
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
+
+#build_dag_each_worker, False，框架在进程内创建一条DAG；True，框架会每个进程内创建多个独立的DAG
+build_dag_each_worker: false
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+
+ #重试次数
+ retry: 1
+
+ #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ use_profile: false
+ tracer:
+ interval_s: 10
+
+op:
+ det:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 6
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #det模型路径
+ model_config: ocr_det_model
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["concat_1.tmp_0"]
+
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 0
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: ""
+
+ #use_mkldnn
+ #use_mkldnn: True
+
+ #thread_num
+ thread_num: 2
+
+ #ir_optim
+ ir_optim: True
+ rec:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 3
+
+ #超时时间, 单位ms
+ timeout: -1
+
+ #Serving交互重试次数，默认不重试
+ retry: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #client类型，包括brpc, grpc和local_predictor。local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #rec模型路径
+ model_config: ocr_rec_model
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["ctc_greedy_decoder_0.tmp_0", "softmax_0.tmp_0"]
+ # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 0
+
+ #计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: ""
+
+ #use_mkldnn
+ #use_mkldnn: True
+
+ #thread_num
+ thread_num: 2
+
+ #ir_optim
+ ir_optim: True
diff --git a/examples/Pipeline/PaddleOCR/ocr/imgs/1.jpg b/examples/Pipeline/PaddleOCR/ocr/imgs/1.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..08010177fed2ee8c3709912c06c0b161ba546313
Binary files /dev/null and b/examples/Pipeline/PaddleOCR/ocr/imgs/1.jpg differ
diff --git a/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py b/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..9952271430b2e260be292b63f39eaf1105b06521
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/pipeline_http_client.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+url = "http://127.0.0.1:9999/ocr/prediction"
+test_img_dir = "imgs/"
+for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data1 = file.read()
+ image = cv2_to_base64(image_data1)
+
+for i in range(4):
+ data = {"key": ["image"], "value": [image]}
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
diff --git a/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py b/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..66faa0428cfcbd97a5fb0d340d4ee07be828cda2
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/pipeline_rpc_client.py
@@ -0,0 +1,41 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+try:
+ from paddle_serving_server.pipeline import PipelineClient
+except ImportError:
+ from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+import requests
+import json
+import cv2
+import base64
+import os
+
+client = PipelineClient()
+client.connect(['127.0.0.1:18090'])
+
+
+def cv2_to_base64(image):
+ return base64.b64encode(image).decode('utf8')
+
+
+test_img_dir = "imgs/"
+for img_file in os.listdir(test_img_dir):
+ with open(os.path.join(test_img_dir, img_file), 'rb') as file:
+ image_data = file.read()
+ image = cv2_to_base64(image_data)
+
+for i in range(1):
+ ret = client.predict(feed_dict={"image": image}, fetch=["res"])
+ print(ret)
diff --git a/examples/Pipeline/PaddleOCR/ocr/web_service.py b/examples/Pipeline/PaddleOCR/ocr/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..c19d481113a0563bbea92b5038968ae9d18e0ab5
--- /dev/null
+++ b/examples/Pipeline/PaddleOCR/ocr/web_service.py
@@ -0,0 +1,180 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import cv2
+import base64
+from paddle_serving_app.reader import OCRReader
+from paddle_serving_app.reader import Sequential, ResizeByFactor
+from paddle_serving_app.reader import Div, Normalize, Transpose
+from paddle_serving_app.reader import DBPostProcess, FilterBoxes, GetRotateCropImage, SortedBoxes
+
+_LOGGER = logging.getLogger()
+
+
+class DetOp(Op):
+ def init_op(self):
+ self.det_preprocess = Sequential([
+ ResizeByFactor(32, 960), Div(255),
+ Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), Transpose(
+ (2, 0, 1))
+ ])
+ self.filter_func = FilterBoxes(10, 10)
+ self.post_func = DBPostProcess({
+ "thresh": 0.3,
+ "box_thresh": 0.5,
+ "max_candidates": 1000,
+ "unclip_ratio": 1.5,
+ "min_size": 3
+ })
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ imgs = []
+ for key in input_dict.keys():
+ data = base64.b64decode(input_dict[key].encode('utf8'))
+ self.raw_im = data
+ data = np.frombuffer(data, np.uint8)
+ self.im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ self.ori_h, self.ori_w, _ = self.im.shape
+ det_img = self.det_preprocess(self.im)
+ _, self.new_h, self.new_w = det_img.shape
+ imgs.append(det_img[np.newaxis, :].copy())
+ return {"image": np.concatenate(imgs, axis=0)}, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ # print(fetch_dict)
+ det_out = fetch_dict["concat_1.tmp_0"]
+ ratio_list = [
+ float(self.new_h) / self.ori_h, float(self.new_w) / self.ori_w
+ ]
+ dt_boxes_list = self.post_func(det_out, [ratio_list])
+ dt_boxes = self.filter_func(dt_boxes_list[0], [self.ori_h, self.ori_w])
+ out_dict = {"dt_boxes": dt_boxes, "image": self.raw_im}
+ return out_dict, None, ""
+
+
+class RecOp(Op):
+ def init_op(self):
+ self.ocr_reader = OCRReader()
+ self.get_rotate_crop_image = GetRotateCropImage()
+ self.sorted_boxes = SortedBoxes()
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ raw_im = input_dict["image"]
+ data = np.frombuffer(raw_im, np.uint8)
+ im = cv2.imdecode(data, cv2.IMREAD_COLOR)
+ dt_boxes = input_dict["dt_boxes"]
+ dt_boxes = self.sorted_boxes(dt_boxes)
+ feed_list = []
+ img_list = []
+ max_wh_ratio = 0
+
+ ## One batch, the type of feed_data is dict.
+ """
+ for i, dtbox in enumerate(dt_boxes):
+ boximg = self.get_rotate_crop_image(im, dt_boxes[i])
+ img_list.append(boximg)
+ h, w = boximg.shape[0:2]
+ wh_ratio = w * 1.0 / h
+ max_wh_ratio = max(max_wh_ratio, wh_ratio)
+ _, w, h = self.ocr_reader.resize_norm_img(img_list[0],
+ max_wh_ratio).shape
+ imgs = np.zeros((len(img_list), 3, w, h)).astype('float32')
+ for id, img in enumerate(img_list):
+ norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
+ imgs[id] = norm_img
+ feed = {"image": imgs.copy()}
+
+ """
+
+ ## Many mini-batchs, the type of feed_data is list.
+ max_batch_size = len(dt_boxes)
+
+ # If max_batch_size is 0, skipping predict stage
+ if max_batch_size == 0:
+ return {}, True, None, ""
+ boxes_size = len(dt_boxes)
+ batch_size = boxes_size // max_batch_size
+ rem = boxes_size % max_batch_size
+ #_LOGGER.info("max_batch_len:{}, batch_size:{}, rem:{}, boxes_size:{}".format(max_batch_size, batch_size, rem, boxes_size))
+ for bt_idx in range(0, batch_size + 1):
+ imgs = None
+ boxes_num_in_one_batch = 0
+ if bt_idx == batch_size:
+ if rem == 0:
+ continue
+ else:
+ boxes_num_in_one_batch = rem
+ elif bt_idx < batch_size:
+ boxes_num_in_one_batch = max_batch_size
+ else:
+ _LOGGER.error("batch_size error, bt_idx={}, batch_size={}".
+ format(bt_idx, batch_size))
+ break
+
+ start = bt_idx * max_batch_size
+ end = start + boxes_num_in_one_batch
+ img_list = []
+ for box_idx in range(start, end):
+ boximg = self.get_rotate_crop_image(im, dt_boxes[box_idx])
+ img_list.append(boximg)
+ h, w = boximg.shape[0:2]
+ wh_ratio = w * 1.0 / h
+ max_wh_ratio = max(max_wh_ratio, wh_ratio)
+ _, w, h = self.ocr_reader.resize_norm_img(img_list[0],
+ max_wh_ratio).shape
+ #_LOGGER.info("---- idx:{}, w:{}, h:{}".format(bt_idx, w, h))
+
+ imgs = np.zeros((boxes_num_in_one_batch, 3, w, h)).astype('float32')
+ for id, img in enumerate(img_list):
+ norm_img = self.ocr_reader.resize_norm_img(img, max_wh_ratio)
+ imgs[id] = norm_img
+ feed = {"image": imgs.copy()}
+ feed_list.append(feed)
+ #_LOGGER.info("feed_list : {}".format(feed_list))
+
+ return feed_list, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_data, data_id, log_id):
+ res_list = []
+ if isinstance(fetch_data, dict):
+ if len(fetch_data) > 0:
+ rec_batch_res = self.ocr_reader.postprocess(
+ fetch_data, with_score=True)
+ for res in rec_batch_res:
+ res_list.append(res[0])
+ elif isinstance(fetch_data, list):
+ for one_batch in fetch_data:
+ one_batch_res = self.ocr_reader.postprocess(
+ one_batch, with_score=True)
+ for res in one_batch_res:
+ res_list.append(res[0])
+
+ res = {"res": str(res_list)}
+ return res, None, ""
+
+
+class OcrService(WebService):
+ def get_pipeline_response(self, read_op):
+ det_op = DetOp(name="det", input_ops=[read_op])
+ rec_op = RecOp(name="rec", input_ops=[det_op])
+ return rec_op
+
+
+ocr_service = OcrService(name="ocr")
+ocr_service.prepare_pipeline_config("config.yml")
+ocr_service.run_service()
diff --git a/examples/Pipeline/imdb_model_ensemble/README.md b/examples/Pipeline/imdb_model_ensemble/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..c72eab665619dd29cfe467cb36ad6b1ff31f2259
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/README.md
@@ -0,0 +1,19 @@
+# IMDB model ensemble examples
+
+## Get models
+```
+sh get_data.sh
+```
+
+## Start servers
+
+```
+python3 -m paddle_serving_server.serve --model imdb_cnn_model --port 9292 &> cnn.log &
+python3 -m paddle_serving_server.serve --model imdb_bow_model --port 9393 &> bow.log &
+python3 test_pipeline_server.py &>pipeline.log &
+```
+
+## Start clients
+```
+python3 test_pipeline_client.py
+```
diff --git a/examples/Pipeline/imdb_model_ensemble/README_CN.md b/examples/Pipeline/imdb_model_ensemble/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..79ed5c0c89bf702860c0e64cfc7bf9efdb2dd76b
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/README_CN.md
@@ -0,0 +1,19 @@
+# IMDB model ensemble 样例
+
+## 获取模型
+```
+sh get_data.sh
+```
+
+## 启动服务
+
+```
+python3 -m paddle_serving_server.serve --model imdb_cnn_model --port 9292 &> cnn.log &
+python3 -m paddle_serving_server.serve --model imdb_bow_model --port 9393 &> bow.log &
+python3 test_pipeline_server.py &>pipeline.log &
+```
+
+## 启动客户端
+```
+python3 test_pipeline_client.py
+```
diff --git a/examples/Pipeline/imdb_model_ensemble/analyse.py b/examples/Pipeline/imdb_model_ensemble/analyse.py
new file mode 100644
index 0000000000000000000000000000000000000000..61511cea28e94d7e1fa3ef379075d47c90333e05
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/analyse.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.pipeline import Analyst
+import json
+import logging
+import sys
+
+logging.basicConfig(level=logging.INFO)
+
+if __name__ == "__main__":
+ if len(sys.argv) < 3:
+ print("Usage: python analyse.py ")
+ exit(1)
+ log_filename = sys.argv[1]
+ trace_filename = sys.argv[2]
+ analyst = Analyst(log_filename)
+ analyst.save_trace(trace_filename)
+ op_analyst = analyst.get_op_analyst()
+ op_concurrency = op_analyst.concurrency_analysis("analyse.yaml")
+ print(json.dumps(op_concurrency, indent=2, separators=(',', ':')))
diff --git a/examples/Pipeline/imdb_model_ensemble/analyse.yaml b/examples/Pipeline/imdb_model_ensemble/analyse.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..9dd28f03ccea3239658cffb31aa520bb190775bf
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/analyse.yaml
@@ -0,0 +1,4 @@
+bow:
+ midp: 0
+cnn:
+ midp: 1
diff --git a/examples/Pipeline/imdb_model_ensemble/config.yml b/examples/Pipeline/imdb_model_ensemble/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..2f25fa861f3ec50d15d5d5795e5e25dbf801e861
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/config.yml
@@ -0,0 +1,100 @@
+#rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 18070
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 18071
+
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+#当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 4
+
+#build_dag_each_worker, False，框架在进程内创建一条DAG；True，框架会每个进程内创建多个独立的DAG
+build_dag_each_worker: False
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: True
+
+ #重试次数
+ retry: 1
+
+ #使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+ use_profile: False
+
+ #channel的最大长度，默认为0
+ channel_size: 0
+
+ #tracer, 跟踪框架吞吐，每个OP和channel的工作情况。无tracer时不生成数据
+ tracer:
+ #每次trace的时间间隔，单位秒/s
+ interval_s: 10
+op:
+ bow:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #client连接类型，brpc
+ client_type: brpc
+
+ #Serving交互重试次数，默认不重试
+ retry: 1
+
+ #Serving交互超时时间, 单位ms
+ timeout: 3000
+
+ #Serving IPs
+ server_endpoints: ["127.0.0.1:9393"]
+
+ #bow模型client端配置
+ client_config: "imdb_bow_client_conf/serving_client_conf.prototxt"
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
+
+ #批量查询Serving的数量, 默认1。batch_size>1要设置auto_batching_timeout，否则不足batch_size时会阻塞
+ batch_size: 1
+
+ #批量查询超时，与batch_size配合使用
+ auto_batching_timeout: 2000
+ cnn:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #client连接类型，brpc
+ client_type: brpc
+
+ #Serving交互重试次数，默认不重试
+ retry: 1
+
+ #超时时间, 单位ms
+ timeout: 3000
+
+ #Serving IPs
+ server_endpoints: ["127.0.0.1:9292"]
+
+ #cnn模型client端配置
+ client_config: "imdb_cnn_client_conf/serving_client_conf.prototxt"
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["prediction"]
+
+ #批量查询Serving的数量, 默认1。batch_size>1要设置auto_batching_timeout，否则不足batch_size时会阻塞
+ batch_size: 1
+
+ #批量查询超时，与batch_size配合使用
+ auto_batching_timeout: 2000
+ combine:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #Serving交互重试次数，默认不重试
+ retry: 1
+
+ #超时时间, 单位ms
+ timeout: 3000
+
+ #批量查询Serving的数量, 默认1。batch_size>1要设置auto_batching_timeout，否则不足batch_size时会阻塞
+ batch_size: 1
+
+ #批量查询超时，与batch_size配合使用
+ auto_batching_timeout: 2000
diff --git a/examples/Pipeline/imdb_model_ensemble/get_data.sh b/examples/Pipeline/imdb_model_ensemble/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..81d8d5d3b018f133c41e211d1501cf3cd9a3d8a4
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/get_data.sh
@@ -0,0 +1,4 @@
+wget --no-check-certificate https://fleet.bj.bcebos.com/text_classification_data.tar.gz
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/imdb-demo/imdb_model.tar.gz
+tar -zxvf text_classification_data.tar.gz
+tar -zxvf imdb_model.tar.gz
diff --git a/examples/Pipeline/imdb_model_ensemble/test_pipeline_client.py b/examples/Pipeline/imdb_model_ensemble/test_pipeline_client.py
new file mode 100644
index 0000000000000000000000000000000000000000..1737f8f782a25025547f68be6619c237975f5172
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/test_pipeline_client.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+
+client = PipelineClient()
+client.connect(['127.0.0.1:18070'])
+
+words = 'i am very sad | 0'
+
+futures = []
+for i in range(100):
+ futures.append(
+ client.predict(
+ feed_dict={"words": words,
+ "logid": 10000 + i},
+ fetch=["prediction"],
+ asyn=True,
+ profile=False))
+
+for f in futures:
+ res = f.result()
+ if res.err_no != 0:
+ print("predict failed: {}".format(res))
+ print(res)
diff --git a/examples/Pipeline/imdb_model_ensemble/test_pipeline_server.py b/examples/Pipeline/imdb_model_ensemble/test_pipeline_server.py
new file mode 100644
index 0000000000000000000000000000000000000000..30317f0ef4fbeba82c3fa6a1551284b467e0adfd
--- /dev/null
+++ b/examples/Pipeline/imdb_model_ensemble/test_pipeline_server.py
@@ -0,0 +1,111 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+import numpy as np
+from paddle_serving_app.reader.imdb_reader import IMDBDataset
+import logging
+from paddle_serving_server.web_service import WebService
+from paddle_serving_server.pipeline import Op, RequestOp, ResponseOp
+from paddle_serving_server.pipeline import PipelineServer
+from paddle_serving_server.pipeline.proto import pipeline_service_pb2
+from paddle_serving_server.pipeline.channel import ChannelDataErrcode
+
+_LOGGER = logging.getLogger()
+user_handler = logging.StreamHandler()
+user_handler.setLevel(logging.INFO)
+user_handler.setFormatter(
+ logging.Formatter(
+ "%(levelname)s %(asctime)s [%(filename)s:%(lineno)d] %(message)s"))
+_LOGGER.addHandler(user_handler)
+
+
+class ImdbRequestOp(RequestOp):
+ def init_op(self):
+ self.imdb_dataset = IMDBDataset()
+ self.imdb_dataset.load_resource('imdb.vocab')
+
+ def unpack_request_package(self, request):
+ dictdata = {}
+ for idx, key in enumerate(request.key):
+ if key != "words":
+ continue
+ words = request.value[idx]
+ word_ids, _ = self.imdb_dataset.get_words_and_label(words)
+ word_len = len(word_ids)
+ dictdata[key] = np.array(word_ids).reshape(word_len, 1)
+ dictdata["{}.lod".format(key)] = np.array([0, word_len])
+
+ log_id = None
+ if request.logid is not None:
+ log_id = request.logid
+ return dictdata, log_id, None, ""
+
+
+class CombineOp(Op):
+ def preprocess(self, input_data, data_id, log_id):
+ #_LOGGER.info("Enter CombineOp::preprocess")
+ combined_prediction = 0
+ for op_name, data in input_data.items():
+ _LOGGER.info("{}: {}".format(op_name, data["prediction"]))
+ combined_prediction += data["prediction"]
+ data = {"prediction": combined_prediction / 2}
+ return data, False, None, ""
+
+
+class ImdbResponseOp(ResponseOp):
+ # Here ImdbResponseOp is consistent with the default ResponseOp implementation
+ def pack_response_package(self, channeldata):
+ resp = pipeline_service_pb2.Response()
+ resp.err_no = channeldata.error_code
+ if resp.err_no == ChannelDataErrcode.OK.value:
+ feed = channeldata.parse()
+ # ndarray to string
+ for name, var in feed.items():
+ resp.value.append(var.__repr__())
+ resp.key.append(name)
+ else:
+ resp.err_msg = channeldata.error_info
+ return resp
+
+
+read_op = ImdbRequestOp()
+
+
+class BowOp(Op):
+ def init_op(self):
+ pass
+
+
+class CnnOp(Op):
+ def init_op(self):
+ pass
+
+
+bow_op = BowOp("bow", input_ops=[read_op])
+cnn_op = CnnOp("cnn", input_ops=[read_op])
+combine_op = CombineOp("combine", input_ops=[bow_op, cnn_op])
+
+# fetch output of bow_op
+#response_op = ImdbResponseOp(input_ops=[bow_op])
+
+# fetch output of combine_op
+response_op = ImdbResponseOp(input_ops=[combine_op])
+
+# use default ResponseOp implementation
+#response_op = ResponseOp(input_ops=[combine_op])
+
+server = PipelineServer()
+server.set_response_op(response_op)
+server.prepare_server('config.yml')
+server.run_server()
diff --git a/examples/Pipeline/simple_web_service/README.md b/examples/Pipeline/simple_web_service/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..ce2fc841b92b27e1b310353d2b8ef31ae48a2aeb
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/README.md
@@ -0,0 +1,19 @@
+# Simple Pipeline WebService
+
+This document will takes UCI service as an example to introduce how to use Pipeline WebService.
+
+## Get model
+```
+sh get_data.sh
+```
+
+## Start server
+
+```
+python3 web_service.py &>log.txt &
+```
+
+## Http test
+```
+curl -X POST -k http://localhost:18082/uci/prediction -d '{"key": ["x"], "value": ["0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332"]}'
+```
diff --git a/examples/Pipeline/simple_web_service/README_CN.md b/examples/Pipeline/simple_web_service/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..b7007d366e058af40e0383fb05f8cfcbca6e19d2
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/README_CN.md
@@ -0,0 +1,19 @@
+# Simple Pipeline WebService
+
+这里以 Uci 服务为例来介绍 Pipeline WebService 的使用。
+
+## 获取模型
+```
+sh get_data.sh
+```
+
+## 启动服务
+
+```
+python3 web_service.py &>log.txt &
+```
+
+## 测试
+```
+curl -X POST -k http://localhost:18082/uci/prediction -d '{"key": ["x"], "value": ["0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332"]}'
+```
diff --git a/examples/Pipeline/simple_web_service/benchmark.py b/examples/Pipeline/simple_web_service/benchmark.py
new file mode 100644
index 0000000000000000000000000000000000000000..88c3ea21722ad9e6420e193a69299b2cf8e443a4
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/benchmark.py
@@ -0,0 +1,85 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import yaml
+import requests
+import time
+import json
+
+from paddle_serving_server.pipeline import PipelineClient
+import numpy as np
+from paddle_serving_client.utils import MultiThreadRunner
+from paddle_serving_client.utils import benchmark_args, show_latency
+
+
+def gen_yml():
+ fin = open("config.yml", "r")
+ config = yaml.load(fin, yaml.FullLoader)
+ fin.close()
+ config["dag"]["tracer"] = {"interval_s": 5}
+ with open("config2.yml", "w") as fout:
+ yaml.dump(config, fout, default_flow_style=False)
+
+
+def run_http(idx, batch_size):
+ print("start thread ({})".format(idx))
+ url = "http://127.0.0.1:18082/uci/prediction"
+ start = time.time()
+ value = "0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332"
+ all_value = ";".join([value for i in range(batch_size)])
+ data = {"key": ["x"], "value": [all_value]}
+ for i in range(1000):
+ r = requests.post(url=url, data=json.dumps(data))
+ print(r.json())
+ end = time.time()
+ return [[end - start]]
+
+
+def multithread_http(thread, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_http, thread, batch_size)
+
+
+def run_rpc(thread, batch_size):
+ client = PipelineClient()
+ client.connect(['127.0.0.1:9998'])
+ value = "0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332"
+ all_value = ";".join([value for i in range(batch_size)])
+ data = {"key": "x", "value": all_value}
+ for i in range(1000):
+ ret = client.predict(
+ feed_dict={data["key"]: data["value"]}, fetch=["res"])
+ print(ret)
+
+
+def multithread_rpc(thraed, batch_size):
+ multi_thread_runner = MultiThreadRunner()
+ result = multi_thread_runner.run(run_rpc, thread, batch_size)
+
+
+if __name__ == "__main__":
+ if sys.argv[1] == "yaml":
+ mode = sys.argv[2] # brpc/ local predictor
+ thread = int(sys.argv[3])
+ gen_yml()
+ elif sys.argv[1] == "run":
+ mode = sys.argv[2] # http/ rpc
+ thread = int(sys.argv[3])
+ batch_size = int(sys.argv[4])
+ if mode == "http":
+ multithread_http(thread, batch_size)
+ elif mode == "rpc":
+ multithread_rpc(thread, batch_size)
diff --git a/examples/Pipeline/simple_web_service/benchmark.sh b/examples/Pipeline/simple_web_service/benchmark.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d73918f8a06a4808dde113656c60089f5a4ef874
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/benchmark.sh
@@ -0,0 +1,43 @@
+# HTTP
+ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+python3 benchmark.py yaml local_predictor 1
+
+for thread_num in 1
+do
+for batch_size in 1
+do
+rm -rf PipelineServingLogs
+rm -rf cpu_utilization.py
+python3 web_service.py >web.log 2>&1 &
+sleep 3
+echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+python3 benchmark.py run http $thread_num $batch_size
+python3 cpu_utilization.py
+echo "------------Fit a line pipeline benchmark (Thread: $thread_num) (BatchSize: $batch_size)"
+tail -n 25 PipelineServingLogs/pipeline.tracer
+ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+done
+done
+
+# RPC
+ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+sleep 3
+python3 benchmark.py yaml local_predictor 1
+
+for thread_num in 1
+do
+for batch_size in 1
+do
+rm -rf PipelineServingLogs
+rm -rf cpu_utilization.py
+python3 web_service.py >web.log 2>&1 &
+sleep 3
+echo "import psutil\ncpu_utilization=psutil.cpu_percent(1,False)\nprint('CPU_UTILIZATION:', cpu_utilization)\n" > cpu_utilization.py
+python3 benchmark.py run rpc $thread_num $batch_size
+python3 cpu_utilization.py
+echo "------------Fit a line pipeline benchmark (Thread: $thread_num) (BatchSize: $batch_size)"
+tail -n 25 PipelineServingLogs/pipeline.tracer
+ps -ef | grep web_service | awk '{print $2}' | xargs kill -9
+done
+done
diff --git a/examples/Pipeline/simple_web_service/config.yml b/examples/Pipeline/simple_web_service/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..12fae64eadbe6b6dd7e26556d0a61039b0fda47f
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/config.yml
@@ -0,0 +1,48 @@
+#worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+##当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 1
+
+#http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+rpc_port: 9998
+http_port: 18082
+
+dag:
+ #op资源类型, True, 为线程模型；False，为进程模型
+ is_thread_op: False
+
+ #tracer
+ tracer:
+ interval_s: 10
+op:
+ uci:
+ #并发数，is_thread_op=True时，为线程并发；否则为进程并发
+ concurrency: 1
+
+ #当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+ local_service_conf:
+
+ #uci模型路径
+ model_config: uci_housing_model
+
+ #计算硬件类型: 空缺时由devices决定(CPU/GPU)，0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+ device_type: 0
+
+ #计算硬件ID，优先由device_type决定硬件类型。devices为""或空缺时为CPU预测；当为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+ devices: "" # "0,1"
+
+ #client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+ client_type: local_predictor
+
+ #Fetch结果列表，以client_config中fetch_var的alias_name为准
+ fetch_list: ["price"]
+
+ #precsion, 预测精度，降低预测精度可提升预测速度
+ #GPU 支持: "fp32"(default), "fp16", "int8"；
+ #CPU 支持: "fp32"(default), "fp16", "bf16"(mkldnn); 不支持: "int8"
+ precision: "fp32"
+
+ #ir_optim开关, 默认False
+ ir_optim: True
+
+ #use_mkldnn开关, 默认False, use_mkldnn与ir_optim同时打开才有性能提升
+ use_mkldnn: True
diff --git a/examples/Pipeline/simple_web_service/get_data.sh b/examples/Pipeline/simple_web_service/get_data.sh
new file mode 100644
index 0000000000000000000000000000000000000000..84a3966a0ef323cef4b146d8e9489c70a7a8ae35
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/get_data.sh
@@ -0,0 +1,2 @@
+wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
+tar -xzf uci_housing.tar.gz
diff --git a/examples/Pipeline/simple_web_service/web_service.py b/examples/Pipeline/simple_web_service/web_service.py
new file mode 100644
index 0000000000000000000000000000000000000000..5f999f94f9da10809c0128a45c115d90f05f0f41
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/web_service.py
@@ -0,0 +1,59 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+import sys
+
+_LOGGER = logging.getLogger()
+
+
+class UciOp(Op):
+ def init_op(self):
+ self.separator = ","
+ self.batch_separator = ";"
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ (_, input_dict), = input_dicts.items()
+ _LOGGER.error("UciOp::preprocess >>> log_id:{}, input:{}".format(
+ log_id, input_dict))
+ x_value = input_dict["x"].split(self.batch_separator)
+ x_lst = []
+ for x_val in x_value:
+ x_lst.append(
+ np.array([
+ float(x.strip()) for x in x_val.split(self.separator)
+ ]).reshape(1, 13))
+ input_dict["x"] = np.concatenate(x_lst, axis=0)
+ proc_dict = {}
+ return input_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ _LOGGER.info(
+ "UciOp::postprocess >>> data_id:{}, log_id:{}, fetch_dict:{}".
+ format(data_id, log_id, fetch_dict))
+ fetch_dict["price"] = str(fetch_dict["price"])
+ return fetch_dict, None, ""
+
+
+class UciService(WebService):
+ def get_pipeline_response(self, read_op):
+ uci_op = UciOp(name="uci", input_ops=[read_op])
+ return uci_op
+
+
+uci_service = UciService(name="uci")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/Pipeline/simple_web_service/web_service_java.py b/examples/Pipeline/simple_web_service/web_service_java.py
new file mode 100644
index 0000000000000000000000000000000000000000..c4ddfb2b1b3c57b4975cac3dc048e1310aa10772
--- /dev/null
+++ b/examples/Pipeline/simple_web_service/web_service_java.py
@@ -0,0 +1,60 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from paddle_serving_server.web_service import WebService, Op
+import logging
+import numpy as np
+from numpy import array
+import sys
+import base64
+
+_LOGGER = logging.getLogger()
+np.set_printoptions(threshold=sys.maxsize)
+
+
+class UciOp(Op):
+ def init_op(self):
+ self.separator = ","
+
+ def preprocess(self, input_dicts, data_id, log_id):
+ """
+ diff with web_server.py
+ javaclient input type is INDArray, restful request input is list.
+ this function simply reshape input to the Specified shape.
+ """
+ (_, input_dict), = input_dicts.items()
+ _LOGGER.error("UciOp::preprocess >>> log_id:{}, input:{}".format(
+ log_id, input_dict))
+ proc_dict = {}
+ x_value = eval(input_dict["x"])
+ input_dict["x"] = x_value.reshape(1, 13)
+
+ return input_dict, False, None, ""
+
+ def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+ _LOGGER.info(
+ "UciOp::postprocess >>> data_id:{}, log_id:{}, fetch_dict:{}".
+ format(data_id, log_id, fetch_dict))
+ fetch_dict["price"] = str(fetch_dict["price"][0][0])
+ return fetch_dict, None, ""
+
+
+class UciService(WebService):
+ def get_pipeline_response(self, read_op):
+ uci_op = UciOp(name="uci", input_ops=[read_op])
+ return uci_op
+
+
+uci_service = UciService(name="uci")
+uci_service.prepare_pipeline_config("config.yml")
+uci_service.run_service()
diff --git a/examples/util/README.md b/examples/util/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..0f6782ab50eaf0bcbc87abb8f654b87aac3a4671
--- /dev/null
+++ b/examples/util/README.md
@@ -0,0 +1,31 @@
+## Timeline Tool Tutorial
+
+([简体中文](./README_CN.md)|English)
+
+The serving framework has a built-in function for predicting the timing of each stage of the service. The client controls whether to turn on the environment through environment variables. After opening, the information will be output to the screen.
+```
+export FLAGS_profile_client=1 #turn on the client timing tool for each stage
+export FLAGS_profile_server=1 #turn on the server timing tool for each stage
+```
+After enabling this function, the client will print the corresponding log information to standard output during the prediction process.
+
+In order to show the time consuming of each stage more intuitively, a script is provided to further analyze and process the log file.
+
+When using, first save the output of the client to a file, taking `profile` as an example.
+```
+python3 show_profile.py profile ${thread_num}
+```
+Here the `thread_num` parameter is the number of processes when the client is running, and the script will calculate the average time spent in each phase according to this parameter.
+
+The script calculates the time spent in each stage, divides by the number of threads to average, and prints to standard output.
+
+```
+python3 timeline_trace.py profile trace
+```
+The script converts the time-dot information in the log into a json format and saves it to a trace file. The trace file can be visualized through the tracing function of the Chrome browser.
+
+Specific operation: Open the chrome browser, enter `chrome://tracing/` in the address bar, jump to the tracing page, click the `load` button, and open the saved trace file to visualize the time information of each stage of the prediction service.
+
+The data visualization output is shown as follow, it uses [bert as service example](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/bert) GPU inference service. The server starts 4 GPU prediction, the client starts 4 `processes`, and the timeline of each stage when the batch size is 1. Among them, `bert_pre` represents the data preprocessing stage of the client, and `client_infer` represents the stage where the client completes sending and receiving prediction requests. `process` represents the process number of the client, and the second line of each process shows the timeline of each op of the server.
+
+
diff --git a/examples/util/README_CN.md b/examples/util/README_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..e12b4cc1a5fc69764455483b0a00dc41c31824a1
--- /dev/null
+++ b/examples/util/README_CN.md
@@ -0,0 +1,31 @@
+## Timeline工具使用
+
+(简体中文|[English](./README.md))
+
+serving框架中内置了预测服务中各阶段时间打点的功能，在client端通过环境变量来控制是否开启，开启后会将打点信息输出到屏幕。
+```
+export FLAGS_profile_client=1 #开启client端各阶段时间打点
+export FLAGS_profile_server=1 #开启server端各阶段时间打点
+```
+开启该功能后，client端在预测的过程中会将对应的日志信息打印到标准输出。
+
+为了更直观地展现各阶段的耗时，提供脚本对日志文件做进一步的分析处理。
+
+使用时先将client的输出保存到文件，以profile为例。
+```
+python3 show_profile.py profile ${thread_num}
+```
+这里thread_num参数为client运行时的进程数，脚本将按照这个参数来计算各阶段的平均耗时。
+
+脚本将计算各阶段的耗时，并除以线程数做平均，打印到标准输出。
+
+```
+python3 timeline_trace.py profile trace
+```
+脚本将日志中的时间打点信息转换成json格式保存到trace文件，trace文件可以通过chrome浏览器的tracing功能进行可视化。
+
+具体操作：打开chrome浏览器，在地址栏输入chrome://tracing/，跳转至tracing页面，点击load按钮，打开保存的trace文件，即可将预测服务的各阶段时间信息可视化。
+
+效果如下图，图中展示了使用[bert示例](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/bert)的GPU预测服务，server端开启4卡预测，client端启动4进程，batch size为1时的各阶段timeline，其中bert_pre代表client端的数据预处理阶段，client_infer代表client完成预测请求的发送和接收结果的阶段，图中的process代表的是client的进程号，每个进进程的第二行展示的是server各个op的timeline。
+
+
diff --git a/examples/util/get_acc.py b/examples/util/get_acc.py
new file mode 100644
index 0000000000000000000000000000000000000000..91796478eaabdbcee89985660521b388d43a0065
--- /dev/null
+++ b/examples/util/get_acc.py
@@ -0,0 +1,28 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import sys
+import os
+
+total = 0
+acc = 0
+for line in sys.stdin:
+ line = line.strip()
+ group = line.split()
+ if (float(group[0]) - 0.5) * (float(group[1]) - 0.5) > 0:
+ acc += 1
+ total += 1
+
+print(float(acc) / float(total))
diff --git a/examples/util/show_profile.py b/examples/util/show_profile.py
new file mode 100644
index 0000000000000000000000000000000000000000..a726e765e4c0a9b77263eb3ea67658545913de58
--- /dev/null
+++ b/examples/util/show_profile.py
@@ -0,0 +1,41 @@
+#coding=utf-8
+import sys
+import collections
+
+profile_file = sys.argv[1]
+thread_num = sys.argv[2]
+time_dict = collections.OrderedDict()
+query_count = 0
+
+
+def prase(line):
+ profile_list = line.split(" ")
+ num = len(profile_list)
+ for idx in range(int(num / 2)):
+ profile_0_list = profile_list[idx * 2].split(":")
+ profile_1_list = profile_list[idx * 2 + 1].split(":")
+ if len(profile_0_list[0].split("_")) == 2:
+ name = profile_0_list[0].split("_")[0]
+ else:
+ name = profile_0_list[0].split("_")[0] + "_" + profile_0_list[
+ 0].split("_")[1]
+ cost = int(profile_1_list[1]) - int(profile_0_list[1])
+ if name not in time_dict:
+ time_dict[name] = cost
+ else:
+ time_dict[name] += cost
+
+
+with open(profile_file) as f:
+ query_count = 0
+ for line in f.readlines():
+ line = line.strip().split("\t")
+ if line[0] == "PROFILE":
+ prase(line[2])
+ query_count += 1
+
+print("thread_num: {}".format(thread_num))
+print("query_count: {}".format(query_count))
+for name in time_dict:
+ print("{} cost: {}s in each thread ".format(name, time_dict[name] / (
+ 1000000.0 * float(thread_num))))
diff --git a/examples/util/timeline_trace.py b/examples/util/timeline_trace.py
new file mode 100644
index 0000000000000000000000000000000000000000..f1273ab616e8b685549356741d5f426899d4cb65
--- /dev/null
+++ b/examples/util/timeline_trace.py
@@ -0,0 +1,55 @@
+#coding=utf-8
+import json
+import sys
+
+profile_file = sys.argv[1]
+
+
+def prase(pid_str, time_str, counter):
+ pid = pid_str.split(":")[1]
+ event_list = time_str.split(" ")
+ trace_list = []
+ for event in event_list:
+ name, ts = event.split(":")
+ name_list = name.split("_")
+ ph = "B" if (name_list[-1] == "0") else "E"
+ if len(name_list) == 2:
+ name = name_list[0]
+ else:
+ name = "_".join(name_list[:-1])
+ name_list = name.split("#")
+ if len(name_list) > 1:
+ tid = name_list[-1]
+ name = "#".join(name_list[:-1])
+ else:
+ tid = 0
+ event_dict = {}
+ event_dict["name"] = name
+ event_dict["tid"] = tid
+ event_dict["pid"] = pid
+ event_dict["ts"] = ts
+ event_dict["ph"] = ph
+
+ trace_list.append(event_dict)
+ return trace_list
+
+
+if __name__ == "__main__":
+ profile_file = sys.argv[1]
+ trace_file = sys.argv[2]
+ all_list = []
+ counter = 0
+ with open(profile_file) as f:
+ for line in f.readlines():
+ line = line.strip().split("\t")
+ if line[0] == "PROFILE":
+ if len(line) < 2:
+ continue
+ trace_list = prase(line[1], line[2], counter)
+ counter += 1
+ for trace in trace_list:
+ all_list.append(trace)
+
+ trace = json.dumps(all_list, indent=2, separators=(',', ':'))
+ with open(trace_file, "w") as f:
+ f.write(trace)
diff --git a/python/paddle_serving_client/io/__init__.py b/python/paddle_serving_client/io/__init__.py
index 07f443196d5e460d5158112dda33bb9c186394b5..970ab144ada1d2e3e6a18b72ae9a40db289eef14 100644
--- a/python/paddle_serving_client/io/__init__.py
+++ b/python/paddle_serving_client/io/__init__.py
@@ -19,7 +19,7 @@ from paddle.fluid.framework import core
from paddle.fluid.framework import default_main_program
from paddle.fluid.framework import Program
from paddle.fluid import CPUPlace
-from paddle.fluid.io import save_inference_model
+from .paddle_io import save_inference_model, normalize_program
import paddle.fluid as fluid
from paddle.fluid.core import CipherUtils
from paddle.fluid.core import CipherFactory
@@ -191,12 +191,14 @@ def save_model(server_model_folder,
executor = Executor(place=CPUPlace())
feed_var_names = [feed_var_dict[x].name for x in feed_var_dict]
+ feed_vars = [feed_var_dict[x] for x in feed_var_dict]
target_vars = []
target_var_names = []
for key in sorted(fetch_var_dict.keys()):
target_vars.append(fetch_var_dict[key])
target_var_names.append(key)
+ main_program = normalize_program(main_program, feed_vars, target_vars)
if not encryption and not show_proto:
if not os.path.exists(server_model_folder):
os.makedirs(server_model_folder)
@@ -209,7 +211,7 @@ def save_model(server_model_folder,
new_params_path = os.path.join(server_model_folder, params_filename)
with open(new_model_path, "wb") as new_model_file:
- new_model_file.write(main_program.desc.serialize_to_string())
+ new_model_file.write(main_program._remove_training_info(False).desc.serialize_to_string())
paddle.static.save_vars(
executor=executor,
@@ -229,7 +231,7 @@ def save_model(server_model_folder,
key = CipherUtils.gen_key_to_file(128, "key")
params = fluid.io.save_persistables(
executor=executor, dirname=None, main_program=main_program)
- model = main_program.desc.serialize_to_string()
+ model = main_program._remove_training_info(False).desc.serialize_to_string()
if not os.path.exists(server_model_folder):
os.makedirs(server_model_folder)
os.chdir(server_model_folder)
diff --git a/python/paddle_serving_client/io/paddle_io.py b/python/paddle_serving_client/io/paddle_io.py
new file mode 100644
index 0000000000000000000000000000000000000000..15ec5d660b8570e9449d8e77b0d5811c529184d2
--- /dev/null
+++ b/python/paddle_serving_client/io/paddle_io.py
@@ -0,0 +1,522 @@
+# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import errno
+import inspect
+import logging
+import os
+import warnings
+import six
+import numpy as np
+
+import paddle
+from paddle.fluid import (
+ core,
+ Variable,
+ CompiledProgram,
+ default_main_program,
+ Program,
+ layers,
+ unique_name,
+ program_guard, )
+from paddle.fluid.io import prepend_feed_ops, append_fetch_ops
+from paddle.fluid.framework import static_only, Parameter
+from paddle.fluid.executor import Executor, global_scope
+from paddle.fluid.log_helper import get_logger
+
+__all__ = []
+
+_logger = get_logger(
+ __name__, logging.INFO, fmt='%(asctime)s-%(levelname)s: %(message)s')
+
+
+def _check_args(caller, args, supported_args=None, deprecated_args=None):
+ supported_args = [] if supported_args is None else supported_args
+ deprecated_args = [] if deprecated_args is None else deprecated_args
+ for arg in args:
+ if arg in deprecated_args:
+ raise ValueError(
+ "argument '{}' in function '{}' is deprecated, only {} are supported.".
+ format(arg, caller, supported_args))
+ elif arg not in supported_args:
+ raise ValueError(
+ "function '{}' doesn't support argument '{}',\n only {} are supported.".
+ format(caller, arg, supported_args))
+
+
+def _check_vars(name, var_list):
+ if not isinstance(var_list, list):
+ var_list = [var_list]
+ if not var_list or not all([isinstance(var, Variable) for var in var_list]):
+ raise ValueError(
+ "'{}' should be a Variable or a list of Variable.".format(name))
+
+
+def _normalize_path_prefix(path_prefix):
+ """
+ convert path_prefix to absolute path.
+ """
+ if not isinstance(path_prefix, six.string_types):
+ raise ValueError("'path_prefix' should be a string.")
+ if path_prefix.endswith("/"):
+ raise ValueError("'path_prefix' should not be a directory")
+ path_prefix = os.path.normpath(path_prefix)
+ path_prefix = os.path.abspath(path_prefix)
+ return path_prefix
+
+
+def _get_valid_program(program=None):
+ """
+ return default main program if program is None.
+ """
+ if program is None:
+ program = default_main_program()
+ elif isinstance(program, CompiledProgram):
+ program = program._program
+ if program is None:
+ raise TypeError(
+ "The type of input program is invalid, expected tyep is Program, but received None"
+ )
+ warnings.warn(
+ "The input is a CompiledProgram, this is not recommended.")
+ if not isinstance(program, Program):
+ raise TypeError(
+ "The type of input program is invalid, expected type is fluid.Program, but received %s"
+ % type(program))
+ return program
+
+
+def _clone_var_in_block(block, var):
+ assert isinstance(var, Variable)
+ if var.desc.type() == core.VarDesc.VarType.LOD_TENSOR:
+ return block.create_var(
+ name=var.name,
+ shape=var.shape,
+ dtype=var.dtype,
+ type=var.type,
+ lod_level=var.lod_level,
+ persistable=True)
+ else:
+ return block.create_var(
+ name=var.name,
+ shape=var.shape,
+ dtype=var.dtype,
+ type=var.type,
+ persistable=True)
+
+
+def normalize_program(program, feed_vars, fetch_vars):
+ """
+ :api_attr: Static Graph
+
+ Normalize/Optimize a program according to feed_vars and fetch_vars.
+
+ Args:
+ program(Program): Specify a program you want to optimize.
+ feed_vars(Variable | list[Variable]): Variables needed by inference.
+ fetch_vars(Variable | list[Variable]): Variables returned by inference.
+
+ Returns:
+ Program: Normalized/Optimized program.
+
+ Raises:
+ TypeError: If `program` is not a Program, an exception is thrown.
+ TypeError: If `feed_vars` is not a Variable or a list of Variable, an exception is thrown.
+ TypeError: If `fetch_vars` is not a Variable or a list of Variable, an exception is thrown.
+
+ Examples:
+ .. code-block:: python
+
+ import paddle
+
+ paddle.enable_static()
+
+ path_prefix = "./infer_model"
+
+ # User defined network, here a softmax regession example
+ image = paddle.static.data(name='img', shape=[None, 28, 28], dtype='float32')
+ label = paddle.static.data(name='label', shape=[None, 1], dtype='int64')
+ predict = paddle.static.nn.fc(image, 10, activation='softmax')
+
+ loss = paddle.nn.functional.cross_entropy(predict, label)
+
+ exe = paddle.static.Executor(paddle.CPUPlace())
+ exe.run(paddle.static.default_startup_program())
+
+ # normalize main program.
+ program = paddle.static.default_main_program()
+ normalized_program = paddle.static.normalize_program(program, [image], [predict])
+
+ """
+ if not isinstance(program, Program):
+ raise TypeError(
+ "program type must be `fluid.Program`, but received `%s`" %
+ type(program))
+ if not isinstance(feed_vars, list):
+ feed_vars = [feed_vars]
+ if not all(isinstance(v, Variable) for v in feed_vars):
+ raise TypeError(
+ "feed_vars type must be a Variable or a list of Variable.")
+ if not isinstance(fetch_vars, list):
+ fetch_vars = [fetch_vars]
+ if not all(isinstance(v, Variable) for v in fetch_vars):
+ raise TypeError(
+ "fetch_vars type must be a Variable or a list of Variable.")
+
+ # remind users to set auc_states to 0 if auc op were found.
+ for op in program.global_block().ops:
+ # clear device of Op
+ device_attr_name = core.op_proto_and_checker_maker.kOpDeviceAttrName()
+ op._set_attr(device_attr_name, "")
+ if op.type == 'auc':
+ warnings.warn("Be sure that you have set auc states to 0 "
+ "before saving inference model.")
+ break
+
+ # fix the bug that the activation op's output as target will be pruned.
+ # will affect the inference performance.
+ # TODO(Superjomn) add an IR pass to remove 1-scale op.
+ #with program_guard(program):
+ # uniq_fetch_vars = []
+ # for i, var in enumerate(fetch_vars):
+ # if var.dtype != paddle.bool:
+ # var = layers.scale(
+ # var, 1., name="save_infer_model/scale_{}".format(i))
+ # uniq_fetch_vars.append(var)
+ # fetch_vars = uniq_fetch_vars
+
+ # serialize program
+ copy_program = program.clone()
+ global_block = copy_program.global_block()
+ remove_op_idx = []
+ for i, op in enumerate(global_block.ops):
+ op.desc.set_is_target(False)
+ if op.type == "feed" or op.type == "fetch":
+ remove_op_idx.append(i)
+ for idx in remove_op_idx[::-1]:
+ global_block._remove_op(idx)
+ copy_program.desc.flush()
+
+ feed_var_names = [var.name for var in feed_vars]
+ copy_program = copy_program._prune_with_input(
+ feeded_var_names=feed_var_names, targets=fetch_vars)
+ copy_program = copy_program._inference_optimize(prune_read_op=True)
+ fetch_var_names = [var.name for var in fetch_vars]
+ prepend_feed_ops(copy_program, feed_var_names)
+ append_fetch_ops(copy_program, fetch_var_names)
+ copy_program.desc._set_version()
+ return copy_program
+
+
+def is_persistable(var):
+ """
+ Check whether the given variable is persistable.
+
+ Args:
+ var(Variable): The variable to be checked.
+
+ Returns:
+ bool: True if the given `var` is persistable
+ False if not.
+
+ Examples:
+ .. code-block:: python
+
+ import paddle
+ import paddle.fluid as fluid
+
+ paddle.enable_static()
+ param = fluid.default_main_program().global_block().var('fc.b')
+ res = fluid.io.is_persistable(param)
+ """
+ if var.desc.type() == core.VarDesc.VarType.FEED_MINIBATCH or \
+ var.desc.type() == core.VarDesc.VarType.FETCH_LIST or \
+ var.desc.type() == core.VarDesc.VarType.READER:
+ return False
+ return var.persistable
+
+
+@static_only
+def serialize_program(feed_vars, fetch_vars, **kwargs):
+ """
+ :api_attr: Static Graph
+
+ Serialize default main program according to feed_vars and fetch_vars.
+
+ Args:
+ feed_vars(Variable | list[Variable]): Variables needed by inference.
+ fetch_vars(Variable | list[Variable]): Variables returned by inference.
+ kwargs: Supported keys including 'program'.Attention please, kwargs is used for backward compatibility mainly.
+ - program(Program): specify a program if you don't want to use default main program.
+
+ Returns:
+ bytes: serialized program.
+
+ Raises:
+ ValueError: If `feed_vars` is not a Variable or a list of Variable, an exception is thrown.
+ ValueError: If `fetch_vars` is not a Variable or a list of Variable, an exception is thrown.
+
+ Examples:
+ .. code-block:: python
+
+ import paddle
+
+ paddle.enable_static()
+
+ path_prefix = "./infer_model"
+
+ # User defined network, here a softmax regession example
+ image = paddle.static.data(name='img', shape=[None, 28, 28], dtype='float32')
+ label = paddle.static.data(name='label', shape=[None, 1], dtype='int64')
+ predict = paddle.static.nn.fc(image, 10, activation='softmax')
+
+ loss = paddle.nn.functional.cross_entropy(predict, label)
+
+ exe = paddle.static.Executor(paddle.CPUPlace())
+ exe.run(paddle.static.default_startup_program())
+
+ # serialize the default main program to bytes.
+ serialized_program = paddle.static.serialize_program([image], [predict])
+
+ # deserialize bytes to program
+ deserialized_program = paddle.static.deserialize_program(serialized_program)
+
+ """
+ # verify feed_vars
+ _check_vars('feed_vars', feed_vars)
+ # verify fetch_vars
+ _check_vars('fetch_vars', fetch_vars)
+
+ program = _get_valid_program(kwargs.get('program', None))
+ program = normalize_program(program, feed_vars, fetch_vars)
+ return _serialize_program(program)
+
+
+def _serialize_program(program):
+ """
+ serialize given program to bytes.
+ """
+ return program.desc.serialize_to_string()
+
+
+@static_only
+def serialize_persistables(feed_vars, fetch_vars, executor, **kwargs):
+ """
+ :api_attr: Static Graph
+
+ Serialize parameters using given executor and default main program according to feed_vars and fetch_vars.
+
+ Args:
+ feed_vars(Variable | list[Variable]): Variables needed by inference.
+ fetch_vars(Variable | list[Variable]): Variables returned by inference.
+ kwargs: Supported keys including 'program'.Attention please, kwargs is used for backward compatibility mainly.
+ - program(Program): specify a program if you don't want to use default main program.
+
+ Returns:
+ bytes: serialized program.
+
+ Raises:
+ ValueError: If `feed_vars` is not a Variable or a list of Variable, an exception is thrown.
+ ValueError: If `fetch_vars` is not a Variable or a list of Variable, an exception is thrown.
+
+ Examples:
+ .. code-block:: python
+
+ import paddle
+
+ paddle.enable_static()
+
+ path_prefix = "./infer_model"
+
+ # User defined network, here a softmax regession example
+ image = paddle.static.data(name='img', shape=[None, 28, 28], dtype='float32')
+ label = paddle.static.data(name='label', shape=[None, 1], dtype='int64')
+ predict = paddle.static.nn.fc(image, 10, activation='softmax')
+
+ loss = paddle.nn.functional.cross_entropy(predict, label)
+
+ exe = paddle.static.Executor(paddle.CPUPlace())
+ exe.run(paddle.static.default_startup_program())
+
+ # serialize parameters to bytes.
+ serialized_params = paddle.static.serialize_persistables([image], [predict], exe)
+
+ # deserialize bytes to parameters.
+ main_program = paddle.static.default_main_program()
+ deserialized_params = paddle.static.deserialize_persistables(main_program, serialized_params, exe)
+
+ """
+ # verify feed_vars
+ _check_vars('feed_vars', feed_vars)
+ # verify fetch_vars
+ _check_vars('fetch_vars', fetch_vars)
+
+ program = _get_valid_program(kwargs.get('program', None))
+ program = normalize_program(program, feed_vars, fetch_vars)
+ return _serialize_persistables(program, executor)
+
+
+def _serialize_persistables(program, executor):
+ """
+ Serialize parameters using given program and executor.
+ """
+ vars_ = list(filter(is_persistable, program.list_vars()))
+ # warn if no variable found in model
+ if len(vars_) == 0:
+ warnings.warn("no variable in your model, please ensure there are any "
+ "variables in your model to save")
+ return None
+ # create a new program and clone persitable vars to it
+ save_program = Program()
+ save_block = save_program.global_block()
+ save_var_map = {}
+ for var in vars_:
+ if var.type != core.VarDesc.VarType.RAW:
+ var_copy = _clone_var_in_block(save_block, var)
+ save_var_map[var_copy.name] = var
+
+ # create in_vars and out_var, then append a save_combine op to save_program
+ in_vars = []
+ for name in sorted(save_var_map.keys()):
+ in_vars.append(save_var_map[name])
+
+ out_var_name = unique_name.generate("out_var")
+ out_var = save_block.create_var(
+ type=core.VarDesc.VarType.RAW, name=out_var_name)
+ out_var.desc.set_persistable(True)
+ save_block.append_op(
+ type='save_combine',
+ inputs={'X': in_vars},
+ outputs={'Y': out_var},
+ attrs={'file_path': '',
+ 'save_to_memory': True})
+ # run save_program to save vars
+ # NOTE(zhiqiu): save op will add variable kLookupTablePath to save_program.desc,
+ # which leads to diff between save_program and its desc. Call _sync_with_cpp
+ # to keep consistency.
+ save_program._sync_with_cpp()
+ executor.run(save_program)
+ # return serialized bytes in out_var
+ return global_scope().find_var(out_var_name).get_bytes()
+
+
+def save_to_file(path, content):
+ """
+ Save content to given path.
+ Args:
+ path(str): Path to write content to.
+ content(bytes): Content to write.
+ Returns:
+ None
+ """
+
+ if not isinstance(content, bytes):
+ raise ValueError("'content' type should be bytes.")
+ with open(path, "wb") as f:
+ f.write(content)
+
+
+@static_only
+def save_inference_model(path_prefix, feed_vars, fetch_vars, executor,
+ **kwargs):
+ """
+ :api_attr: Static Graph
+
+ Save current model and its parameters to given path. i.e.
+ Given path_prefix = "/path/to/modelname", after invoking
+ save_inference_model(path_prefix, feed_vars, fetch_vars, executor),
+ you will find two files named modelname.pdmodel and modelname.pdiparams
+ under "/path/to", which represent your model and parameters respectively.
+
+ Args:
+ path_prefix(str): Directory path to save model + model name without suffix.
+ feed_vars(Variable | list[Variable]): Variables needed by inference.
+ fetch_vars(Variable | list[Variable]): Variables returned by inference.
+ executor(Executor): The executor that saves the inference model. You can refer
+ to :ref:`api_guide_executor_en` for more details.
+ kwargs: Supported keys including 'program' and "clip_extra". Attention please, kwargs is used for backward compatibility mainly.
+ - program(Program): specify a program if you don't want to use default main program.
+ - clip_extra(bool): set to True if you want to clip extra information for every operator.
+ Returns:
+ None
+
+ Raises:
+ ValueError: If `feed_vars` is not a Variable or a list of Variable, an exception is thrown.
+ ValueError: If `fetch_vars` is not a Variable or a list of Variable, an exception is thrown.
+
+ Examples:
+ .. code-block:: python
+
+ import paddle
+
+ paddle.enable_static()
+
+ path_prefix = "./infer_model"
+
+ # User defined network, here a softmax regession example
+ image = paddle.static.data(name='img', shape=[None, 28, 28], dtype='float32')
+ label = paddle.static.data(name='label', shape=[None, 1], dtype='int64')
+ predict = paddle.static.nn.fc(image, 10, activation='softmax')
+
+ loss = paddle.nn.functional.cross_entropy(predict, label)
+
+ exe = paddle.static.Executor(paddle.CPUPlace())
+ exe.run(paddle.static.default_startup_program())
+
+ # Feed data and train process
+
+ # Save inference model. Note we don't save label and loss in this example
+ paddle.static.save_inference_model(path_prefix, [image], [predict], exe)
+
+ # In this example, the save_inference_mode inference will prune the default
+ # main program according to the network's input node (img) and output node(predict).
+ # The pruned inference program is going to be saved in file "./infer_model.pdmodel"
+ # and parameters are going to be saved in file "./infer_model.pdiparams".
+
+ """
+
+ # check path_prefix, set model_path and params_path
+ path_prefix = _normalize_path_prefix(path_prefix)
+ try:
+ # mkdir may conflict if pserver and trainer are running on the same machine
+ dirname = os.path.dirname(path_prefix)
+ os.makedirs(dirname)
+ except OSError as e:
+ if e.errno != errno.EEXIST:
+ raise
+ model_path = path_prefix + ".pdmodel"
+ params_path = path_prefix + ".pdiparams"
+ if os.path.isdir(model_path):
+ raise ValueError("'{}' is an existing directory.".format(model_path))
+ if os.path.isdir(params_path):
+ raise ValueError("'{}' is an existing directory.".format(params_path))
+
+ # verify feed_vars
+ _check_vars('feed_vars', feed_vars)
+ # verify fetch_vars
+ _check_vars('fetch_vars', fetch_vars)
+
+ program = _get_valid_program(kwargs.get('program', None))
+ clip_extra = kwargs.get('clip_extra', False)
+ program = normalize_program(program, feed_vars, fetch_vars)
+ # serialize and save program
+ program_bytes = _serialize_program(
+ program._remove_training_info(clip_extra=clip_extra))
+ save_to_file(model_path, program_bytes)
+ # serialize and save params
+ params_bytes = _serialize_persistables(program, executor)
+ save_to_file(params_path, params_bytes)
+
diff --git a/python/paddle_serving_server/util.py b/python/paddle_serving_server/util.py
index 85c9341c7848631394d348521131c3e0ae1455d7..8062d9a70b220d1863ab8b079a1d13259495bb64 100644
--- a/python/paddle_serving_server/util.py
+++ b/python/paddle_serving_server/util.py
@@ -16,6 +16,7 @@ import signal
import os
import time
import json
+import platform
from paddle_serving_server.env import CONF_HOME
@@ -91,7 +92,10 @@ def dump_pid_file(portList, model):
dump_pid_file([9494, 10082], 'serve')
'''
pid = os.getpid()
- gid = os.getpgid(pid)
+ if platform.system() == "Windows":
+ gid = pid
+ else:
+ gid = os.getpgid(pid)
pidInfoList = []
filepath = os.path.join(CONF_HOME, "ProcessInfo.json")
if os.path.exists(filepath):
diff --git a/python/pipeline/pipeline_client.py b/python/pipeline/pipeline_client.py
index a7aeb548008354bd1a17c4b39596575c6594e39b..f0dbd631e6a0e678746d0d564a3e1d8c5c0316ef 100644
--- a/python/pipeline/pipeline_client.py
+++ b/python/pipeline/pipeline_client.py
@@ -108,7 +108,7 @@ class PipelineClient(object):
one_tensor.name = key
if isinstance(value, str):
- one_tensor.string_data.add(value)
+ one_tensor.str_data.append(value)
one_tensor.elem_type = 12 #12 => string in proto
continue
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
- 
- 
- 
- 
+ 
+ 
+
需要注意的是：A/B Test只适用于RPC模式，不适用于WEB模式。
@@ -24,13 +24,13 @@ pip install Shapely
````
您可以直接运行下面的命令来处理数据。
-[python abtest_get_data.py](../python/examples/imdb/abtest_get_data.py)
+[python abtest_get_data.py](../../examples/C++/imdb/abtest_get_data.py)
文件中的Python代码将处理`test_data/part-0`的数据，并将处理后的数据生成并写入`processed.data`文件中。
### 启动Server端
-这里采用[Docker方式](RUN_IN_DOCKER_CN.md)启动Server端服务。
+这里采用[Docker方式](../RUN_IN_DOCKER_CN.md)启动Server端服务。
首先启动BOW Server，该服务启用`8000`端口：
@@ -62,7 +62,7 @@ exit
您可以直接使用下面的命令，进行ABTEST预测。
-[python abtest_client.py](../python/examples/imdb/abtest_client.py)
+[python abtest_client.py](../../examples/C++/imdb/abtest_client.py)
```python
from paddle_serving_client import Client
diff --git a/doc/cpp_server/ABTEST_IN_PADDLE_SERVING.md b/doc/C++_Serving/ABTest_EN.md
old mode 100644
new mode 100755
similarity index 93%
rename from doc/cpp_server/ABTEST_IN_PADDLE_SERVING.md
rename to doc/C++_Serving/ABTest_EN.md
index f250f1a176c76f8baf66411fc896a4bd9f4ce040..edbdbc09108187c442496dc81256065ac26f61eb
--- a/doc/cpp_server/ABTEST_IN_PADDLE_SERVING.md
+++ b/doc/C++_Serving/ABTest_EN.md
@@ -1,10 +1,10 @@
# ABTEST in Paddle Serving
-([简体中文](./ABTEST_IN_PADDLE_SERVING_CN.md)|English)
+([简体中文](./ABTest_CN.md)|English)
This document will use an example of text classification task based on IMDB dataset to show how to build a A/B Test framework using Paddle Serving. The structure relationship between the client and servers in the example is shown in the figure below.
-
+ 
+ 
+
+
+
+ 
+
+
+ 
-
-### 3.6 Client io used during Training
-
-PaddleServing is designed to saves the model interface that can be used during the training process, which is basically the same as the Paddle save inference model interface, feed_var_dict and fetch_var_dict
-You can alias the input and output variables. The configuration that needs to be read when the serving starts is saved in the client and server storage directories.
-
-``` python
-def save_model(server_model_folder,
- client_config_folder,
- feed_var_dict,
- fetch_var_dict,
- main_program=None)
-```
-
-## 4. Paddle Serving Underlying Framework
-
-
-
-**Model Management Framework**: Connects model files of multiple machine learning platforms and provides a unified inference interface
-**Business Scheduling Framework**: Abstracts the calculation logic of various different inference models, provides a general DAG scheduling framework, and connects different operators through DAG diagrams to complete a prediction service together. This abstract model allows users to conveniently implement their own calculation logic, and at the same time facilitates operator sharing. (Users build their own forecasting services. A large part of their work is to build DAGs and provide operators.)
-**Predict Service**: Encapsulation of the externally provided prediction service interface. Define communication fields with the client through protobuf.
-
-### 4.1 Model Management Framework
-
-The model management framework is responsible for managing the models trained by the machine learning framework. It can be abstracted into three levels: model loading, model data, and model reasoning.
-
-#### Model Loading
-
-Load model from disk to memory, support multi-version, hot-load, incremental update, etc.
-
-#### Model data
-
-Model data structure in memory, integrated fluid inference lib
-
-#### inferencer
-
-it provided united inference interface for upper layers
-
-```C++
-class FluidFamilyCore {
- virtual bool Run(const void* in_data, void* out_data);
- virtual int create(const std::string& data_path);
- virtual int clone(void* origin_core);
-};
-```
-
-### 4.2 Business Scheduling Framework
-
-#### 4.2.1 Inference Service
-
-With reference to the abstract idea of model calculation of the TensorFlow framework, the business logic is abstracted into a DAG diagram, driven by configuration, generating a workflow, and skipping C ++ code compilation. Each specific step of the service corresponds to a specific OP. The OP can configure the upstream OP that it depends on. Unified message passing between OPs is achieved by the thread-level bus and channel mechanisms. For example, the service process of a simple prediction service can be abstracted into 3 steps including reading request data-> calling the prediction interface-> writing back the prediction result, and correspondingly implemented to 3 OP: ReaderOp-> ClassifyOp-> WriteOp
-
-
-
-Regarding the dependencies between OPs, and the establishment of workflows through OPs, you can refer to [从零开始写一个预测服务](CREATING.md) (simplified Chinese Version)
-
-Server instance perspective
-
-
-
-
-#### 4.2.2 Paddle Serving Multi-Service Mechanism
-
-
-
-Paddle Serving instances can load multiple models at the same time, and each model uses a Service (and its configured workflow) to undertake services. You can refer to [service configuration file in Demo example](../tools/cpp_examples/demo-serving/conf/service.prototxt) to learn how to configure multiple services for the serving instance
-
-#### 4.2.3 Hierarchical relationship of business scheduling
-
-From the client's perspective, a Paddle Serving service can be divided into three levels: Service, Endpoint, and Variant from top to bottom.
-
-
-
-One Service corresponds to one inference model, and there is one endpoint under the model. Different versions of the model are implemented through multiple variant concepts under endpoint:
-The same model prediction service can configure multiple variants, and each variant has its own downstream IP list. The client code can configure relative weights for each variant to achieve the relationship of adjusting the traffic ratio (refer to the description of variant_weight_list in [Client Configuration](CLIENT_CONFIGURE.md) section 3.2).
-
-
-
-## 5. User Interface
-
-Under the premise of meeting certain interface specifications, the service framework does not make any restrictions on user data fields to meet different business interfaces of various forecast services. Baidu-rpc inherits the interface of Protobuf serice, and the user describes the Request and Response business interfaces according to the Protobuf syntax specification. Paddle Serving is built on the Baidu-rpc framework and supports this feature by default.
-
-No matter how the communication protocol changes, the framework only needs to ensure that the communication protocol between the client and server and the format of the business data are synchronized to ensure normal communication. This information can be broken down as follows:
-
--Protocol: Header information agreed in advance between Server and Client to ensure mutual recognition of data format. Paddle Serving uses Protobuf as the basic communication format
--Data: Used to describe the interface of Request and Response, such as the sample data to be predicted, and the score returned by the prediction. include:
- -Data fields: Field definitions included in the two data structures of Request and Return.
- -Description interface: similar to the protocol interface, it supports Protobuf by default
-
-### 5.1 Data Compression Method
-
-Baidu-rpc has built-in data compression methods such as snappy, gzip, zlib, which can be configured in the configuration file (refer to [Client Configuration](CLIENT_CONFIGURE.md) Section 3.1 for an introduction to compress_type)
-
-### 5.2 C ++ SDK API Interface
-
-```C++
-class PredictorApi {
- public:
- int create(const char* path, const char* file);
- int thrd_initialize();
- int thrd_clear();
- int thrd_finalize();
- void destroy();
-
- Predictor* fetch_predictor(std::string ep_name);
- int free_predictor(Predictor* predictor);
-};
-
-class Predictor {
- public:
- // synchronize interface
- virtual int inference(google::protobuf::Message* req,
- google::protobuf::Message* res) = 0;
-
- // asynchronize interface
- virtual int inference(google::protobuf::Message* req,
- google::protobuf::Message* res,
- DoneType done,
- brpc::CallId* cid = NULL) = 0;
-
- // synchronize interface
- virtual int debug(google::protobuf::Message* req,
- google::protobuf::Message* res,
- butil::IOBufBuilder* debug_os) = 0;
-};
-
-```
-
-### 5.3 Inferfaces related to Op
-
-```C++
-class Op {
- // ------Getters for Channel/Data/Message of dependent OP-----
-
- // Get the Channel object of dependent OP
- Channel* mutable_depend_channel(const std::string& op);
-
- // Get the Channel object of dependent OP
- const Channel* get_depend_channel(const std::string& op) const;
-
- template