+
+
+
+
+
+ +
+
+
+ 
+
-[](https://travis-ci.com/PaddlePaddle/Serving) -[](Release) -[](Issues) -[](LICENSE) -[](https://paddleserving.slack.com/archives/CU0PB4K35) +
+
+
+
+
+ +``` shell +curl -H "Content-Type:application/json" -X POST -d '{"url": "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg", "fetch": ["score"]}' http://127.0.0.1:9292/image/prediction +``` +- **返回结果示例**: +``` shell +{"label":"daisy","prob":0.9341403245925903} +``` + +
diff --git a/doc/ABTEST_IN_PADDLE_SERVING_CN.md b/doc/ABTEST_IN_PADDLE_SERVING_CN.md
index d31ddba6f72dfc23fa15defeda23468ab1785e62..e32bf783fcde20bb5dff3d2addaf764838975a81 100644
--- a/doc/ABTEST_IN_PADDLE_SERVING_CN.md
+++ b/doc/ABTEST_IN_PADDLE_SERVING_CN.md
@@ -1,5 +1,7 @@
# 如何使用Paddle Serving做ABTEST
+(简体中文|[English](./ABTEST_IN_PADDLE_SERVING.md))
+
该文档将会用一个基于IMDB数据集的文本分类任务的例子，介绍如何使用Paddle Serving搭建A/B Test框架，例中的Client端、Server端结构如下图所示。
diff --git a/doc/DESIGN.md b/doc/DESIGN.md
index 34983801759c8e2d25fb336decbc5828687e4211..5d00d02171dccf07bfdafb9cdd85222a92c20113 100644
--- a/doc/DESIGN.md
+++ b/doc/DESIGN.md
@@ -1,45 +1,47 @@
-# Paddle Serving设计方案
+# Paddle Serving Design
-## 1. 项目背景
+([简体中文](./DESIGN_CN.md)|English)
-PaddlePaddle是公司开源的机器学习框架，广泛支持各种深度学习模型的定制化开发; Paddle serving是Paddle的在线预测部分，与Paddle模型训练环节无缝衔接，提供机器学习预测云服务。本文将从模型、服务、接入等层面，自底向上描述Paddle Serving设计方案。
+## 1. Background
-1. 模型是Paddle Serving预测的核心，包括模型数据和推理计算的管理；
-2. 预测框架封装模型推理计算，对外提供RPC接口，对接不同上游；
-3. 预测服务SDK提供一套接入框架
+PaddlePaddle is the Baidu's open source machine learning framework, which supports a wide range of customized development of deep learning models; Paddle serving is the online prediction framework of Paddle, which seamlessly connects with Paddle model training, and provides cloud services for machine learning prediction. This article will describe the Paddle Serving design from the bottom up, from the model, service, and access levels.
-最终形成一套完整的serving解决方案。
+1. The model is the core of Paddle Serving prediction, including the management of model data and inference calculations;
+2. Prediction framework encapsulation model for inference calculations, providing external RPC interface to connect different upstream
+3. The prediction service SDK provides a set of access frameworks
-## 2. 名词解释
+The result is a complete serving solution.
-- baidu-rpc 百度官方开源RPC框架，支持多种常见通信协议，提供基于protobuf的自定义接口体验
-- Variant Paddle Serving架构对一个最小预测集群的抽象，其特点是内部所有实例（副本）完全同质，逻辑上对应一个model的一个固定版本
-- Endpoint 多个Variant组成一个Endpoint，逻辑上看，Endpoint代表一个model，Endpoint内部的Variant代表不同的版本
-- OP PaddlePaddle用来封装一种数值计算的算子，Paddle Serving用来表示一种基础的业务操作算子，核心接口是inference。OP通过配置其依赖的上游OP，将多个OP串联成一个workflow
-- Channel 一个OP所有请求级中间数据的抽象；OP之间通过Channel进行数据交互
-- Bus 对一个线程中所有channel的管理，以及根据DAG之间的DAG依赖图对OP和Channel两个集合间的访问关系进行调度
-- Stage Workflow按照DAG描述的拓扑图中，属于同一个环节且可并行执行的OP集合
-- Node 由某个Op算子类结合参数配置组成的Op算子实例，也是Workflow中的一个执行单元
-- Workflow 按照DAG描述的拓扑，有序执行每个OP的inference接口
-- DAG/Workflow 由若干个相互依赖的Node组成，每个Node均可通过特定接口获得Request对象，节点Op通过依赖关系获得其前置Op的输出对象，最后一个Node的输出默认就是Response对象
-- Service 对一次pv的请求封装，可配置若干条Workflow，彼此之间复用当前PV的Request对象，然后各自并行/串行执行，最后将Response写入对应的输出slot中；一个Paddle-serving进程可配置多套Service接口，上游根据ServiceName决定当前访问的Service接口。
+## 2. Terms explanation
-## 3. Python Interface设计
+- **baidu-rpc**: Baidu's official open source RPC framework, supports multiple common communication protocols, and provides a custom interface experience based on protobuf
+- **Variant**: Paddle Serving architecture is an abstraction of a minimal prediction cluster, which is characterized by all internal instances (replicas) being completely homogeneous and logically corresponding to a fixed version of a model
+- **Endpoint**: Multiple Variants form an Endpoint. Logically, Endpoint represents a model, and Variants within the Endpoint represent different versions.
+- **OP**: PaddlePaddle is used to encapsulate a numerical calculation operator, Paddle Serving is used to represent a basic business operation operator, and the core interface is inference. OP configures its dependent upstream OP to connect multiple OPs into a workflow
+- **Channel**: An abstraction of all request-level intermediate data of the OP; data exchange between OPs through Channels
+- **Bus**: manages all channels in a thread, and schedules the access relationship between the two sets of OP and Channel according to the DAG dependency graph between DAGs
+- **Stage**: Workflow according to the topology diagram described by DAG, a collection of OPs that belong to the same link and can be executed in parallel
+- **Node**: An OP operator instance composed of an OP operator class combined with parameter configuration, which is also an execution unit in Workflow
+- **Workflow**: executes the inference interface of each OP in order according to the topology described by DAG
+- **DAG/Workflow**: consists of several interdependent Nodes. Each Node can obtain the Request object through a specific interface. The node Op obtains the output object of its pre-op through the dependency relationship. The output of the last Node is the Response object by default.
+- **Service**: encapsulates a pv request, can configure several Workflows, reuse the current PV's Request object with each other, and then execute each in parallel/serial execution, and finally write the Response to the corresponding output slot; a Paddle-serving process Multiple sets of Service interfaces can be configured. The upstream determines the Service interface currently accessed based on the ServiceName.
-### 3.1 核心目标：
+## 3. Python Interface Design
-一套Paddle Serving的动态库，支持Paddle保存的通用模型的远程预估服务，通过Python Interface调用PaddleServing底层的各种功能。
+### 3.1 Core Targets:
-### 3.2 通用模型：
+A set of Paddle Serving dynamic library, support the remote estimation service of the common model saved by Paddle, and call the various underlying functions of PaddleServing through the Python Interface.
-能够使用Paddle Inference Library进行预测的模型，在训练过程中保存的模型，包含Feed Variable和Fetch Variable
+### 3.2 General Model:
-### 3.3 整体设计：
+Models that can be predicted using the Paddle Inference Library, models saved during training, including Feed Variable and Fetch Variable
-用户通过Python Client启动Client和Server，Python API有检查互联和待访问模型是否匹配的功能
-Python API背后调用的是Paddle Serving实现的client和server对应功能的pybind，互传的信息通过RPC实现
-Client Python API当前有两个简单的功能，load_inference_conf和predict，分别用来执行加载待预测的模型和预测
-Server Python API主要负责加载预估模型，以及生成Paddle Serving需要的各种配置，包括engines，workflow，resource等
+### 3.3 Overall design:
+
+- The user starts the Client and Server through the Python Client. The Python API has a function to check whether the interconnection and the models to be accessed match.
+- The Python API calls the pybind corresponding to the client and server functions implemented by Paddle Serving, and the information transmitted through RPC is implemented through RPC.
+- The Client Python API currently has two simple functions, load_inference_conf and predict, which are used to perform loading of the model to be predicted and prediction, respectively.
+- The Server Python API is mainly responsible for loading the inference model and generating various configurations required by Paddle Serving, including engines, workflow, resources, etc.
### 3.4 Server Inferface
@@ -49,10 +51,10 @@ Server Python API主要负责加载预估模型，以及生成Paddle Serving需
-### 3.6 训练过程中使用的Client io
+### 3.6 Client io used during Training
-PaddleServing设计可以在训练过程中使用的保存模型接口，与Paddle保存inference model的接口基本一致，feed_var_dict与fetch_var_dict
-可以为输入和输出变量起别名，serving启动需要读取的配置会保存在client端和server端的保存目录中。
+PaddleServing is designed to saves the model interface that can be used during the training process, which is basically the same as the Paddle save inference model interface, feed_var_dict and fetch_var_dict
+You can alias the input and output variables. The configuration that needs to be read when the serving starts is saved in the client and server storage directories.
``` python
def save_model(server_model_folder,
@@ -62,29 +64,29 @@ def save_model(server_model_folder,
main_program=None)
```
-## 4. Paddle Serving底层框架
+## 4. Paddle Serving Underlying Framework
-
+
-**模型管理框架**：对接多种机器学习平台的模型文件，向上提供统一的inference接口
-**业务调度框架**：对各种不同预测模型的计算逻辑进行抽象，提供通用的DAG调度框架，通过DAG图串联不同的算子，共同完成一次预测服务。该抽象模型使用户可以方便的实现自己的计算逻辑，同时便于算子共用。（用户搭建自己的预测服务，很大一部分工作是搭建DAG和提供算子的实现）
-**PredictService**:对外部提供的预测服务接口封装。通过protobuf定义与客户端的通信字段。
+**Model Management Framework**: Connects model files of multiple machine learning platforms and provides a unified inference interface
+**Business Scheduling Framework**: Abstracts the calculation logic of various different inference models, provides a general DAG scheduling framework, and connects different operators through DAG diagrams to complete a prediction service together. This abstract model allows users to conveniently implement their own calculation logic, and at the same time facilitates operator sharing. (Users build their own forecasting services. A large part of their work is to build DAGs and provide operators.)
+**Predict Service**: Encapsulation of the externally provided prediction service interface. Define communication fields with the client through protobuf.
-### 4.1 模型管理框架
+### 4.1 Model Management Framework
-模型管理框架负责管理机器学习框架训练出来的模型，总体可抽象成模型加载、模型数据和模型推理等3个层次。
+The model management framework is responsible for managing the models trained by the machine learning framework. It can be abstracted into three levels: model loading, model data, and model reasoning.
-#### 模型加载
+#### Model Loading
-将模型从磁盘加载到内存，支持多版本、热加载、增量更新等功能
+Load model from disk to memory, support multi-version, hot-load, incremental update, etc.
-#### 模型数据
+#### Model data
-模型在内存中的数据结构，集成fluid预测lib
+Model data structure in memory, integrated fluid inference lib
#### inferencer
-向上为预测服务提供统一的预测接口
+it provided united inference interface for upper layers
```C++
class FluidFamilyCore {
@@ -94,54 +96,54 @@ class FluidFamilyCore {
};
```
-### 4.2 业务调度框架
+### 4.2 Business Scheduling Framework
-#### 4.2.1 预测服务Service
+#### 4.2.1 Inference Service
-参考TF框架的模型计算的抽象思想，将业务逻辑抽象成DAG图，由配置驱动，生成workflow，跳过C++代码编译。业务的每个具体步骤，对应一个具体的OP，OP可配置自己依赖的上游OP。OP之间消息传递统一由线程级Bus和channel机制实现。例如，一个简单的预测服务的服务过程，可以抽象成读请求数据->调用预测接口->写回预测结果等3个步骤，相应的实现到3个OP: ReaderOp->ClassifyOp->WriteOp
+With reference to the abstract idea of model calculation of the TensorFlow framework, the business logic is abstracted into a DAG diagram, driven by configuration, generating a workflow, and skipping C ++ code compilation. Each specific step of the service corresponds to a specific OP. The OP can configure the upstream OP that it depends on. Unified message passing between OPs is achieved by the thread-level bus and channel mechanisms. For example, the service process of a simple prediction service can be abstracted into 3 steps including reading request data-> calling the prediction interface-> writing back the prediction result, and correspondingly implemented to 3 OP: ReaderOp-> ClassifyOp-> WriteOp
-
+
-关于OP之间的依赖关系，以及通过OP组建workflow，可以参考[从零开始写一个预测服务](CREATING.md)的相关章节
+Regarding the dependencies between OPs, and the establishment of workflows through OPs, you can refer to [从零开始写一个预测服务](./deprecated/CREATING.md) (simplified Chinese Version)
-服务端实例透视图
+Server instance perspective
-
+
-#### 4.2.2 Paddle Serving的多服务机制
+#### 4.2.2 Paddle Serving Multi-Service Mechanism
-
+
-Paddle Serving实例可以同时加载多个模型，每个模型用一个Service（以及其所配置的workflow）承接服务。可以参考[Demo例子中的service配置文件](../demo-serving/conf/service.prototxt)了解如何为serving实例配置多个service
+Paddle Serving instances can load multiple models at the same time, and each model uses a Service (and its configured workflow) to undertake services. You can refer to [service configuration file in Demo example](../tools/cpp_examples/demo-serving/conf/service.prototxt) to learn how to configure multiple services for the serving instance
-#### 4.2.3 业务调度层级关系
+#### 4.2.3 Hierarchical relationship of business scheduling
-从客户端看，一个Paddle Serving service从顶向下可分为Service, Endpoint, Variant等3个层级
+From the client's perspective, a Paddle Serving service can be divided into three levels: Service, Endpoint, and Variant from top to bottom.
-
+
-一个Service对应一个预测模型，模型下有1个endpoint。模型的不同版本，通过endpoint下多个variant概念实现：
-同一个模型预测服务，可以配置多个variant，每个variant有自己的下游IP列表。客户端代码可以对各个variant配置相对权重，以达到调节流量比例的关系（参考[客户端配置](CLIENT_CONFIGURE.md)第3.2节中关于variant_weight_list的说明）。
+One Service corresponds to one inference model, and there is one endpoint under the model. Different versions of the model are implemented through multiple variant concepts under endpoint:
+The same model prediction service can configure multiple variants, and each variant has its own downstream IP list. The client code can configure relative weights for each variant to achieve the relationship of adjusting the traffic ratio (refer to the description of variant_weight_list in [Client Configuration](./deprecated/CLIENT_CONFIGURE.md) section 3.2).
-
+
-## 5. 用户接口
+## 5. User Interface
-在满足一定的接口规范前提下，服务框架不对用户数据字段做任何约束，以应对各种预测服务的不同业务接口。Baidu-rpc继承了Protobuf serice的接口，用户按照Protobuf语法规范描述Request和Response业务接口。Paddle Serving基于Baidu-rpc框架搭建，默认支持该特性。
+Under the premise of meeting certain interface specifications, the service framework does not make any restrictions on user data fields to meet different business interfaces of various forecast services. Baidu-rpc inherits the interface of Protobuf serice, and the user describes the Request and Response business interfaces according to the Protobuf syntax specification. Paddle Serving is built on the Baidu-rpc framework and supports this feature by default.
-无论通信协议如何变化，框架只需确保Client和Server间通信协议和业务数据两种信息的格式同步，即可保证正常通信。这些信息又可细分如下：
+No matter how the communication protocol changes, the framework only needs to ensure that the communication protocol between the client and server and the format of the business data are synchronized to ensure normal communication. This information can be broken down as follows:
-- 协议：Server和Client之间事先约定的、确保相互识别数据格式的包头信息。Paddle Serving用Protobuf作为基础通信格式
-- 数据：用来描述Request和Response的接口，例如待预测样本数据，和预测返回的打分。包括：
- - 数据字段：请求包Request和返回包Response两种数据结构包含的字段定义
- - 描述接口：跟协议接口类似，默认支持Protobuf
+-Protocol: Header information agreed in advance between Server and Client to ensure mutual recognition of data format. Paddle Serving uses Protobuf as the basic communication format
+-Data: Used to describe the interface of Request and Response, such as the sample data to be predicted, and the score returned by the prediction. include:
+ -Data fields: Field definitions included in the two data structures of Request and Return.
+ -Description interface: similar to the protocol interface, it supports Protobuf by default
-### 5.1 数据压缩方法
+### 5.1 Data Compression Method
-Baidu-rpc内置了snappy, gzip, zlib等数据压缩方法，可在配置文件中配置（参考[客户端配置](CLIENT_CONFIGURE.md)第3.1节关于compress_type的介绍）
+Baidu-rpc has built-in data compression methods such as snappy, gzip, zlib, which can be configured in the configuration file (refer to [Client Configuration](./deprecated/CLIENT_CONFIGURE.md) Section 3.1 for an introduction to compress_type)
-### 5.2 C++ SDK API接口
+### 5.2 C ++ SDK API Interface
```C++
class PredictorApi {
@@ -176,7 +178,7 @@ class Predictor {
```
-### 5.3 OP相关接口
+### 5.3 Inferfaces related to Op
```C++
class Op {
@@ -258,7 +260,7 @@ class Op {
```
-### 5.4 框架相关接口
+### 5.4 Interfaces related to framework
Service
diff --git a/doc/DESIGN_CN.md b/doc/DESIGN_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..124e826c4591c89cb14d25153f4c9a3096ea8dfb
--- /dev/null
+++ b/doc/DESIGN_CN.md
@@ -0,0 +1,377 @@
+# Paddle Serving设计方案
+
+(简体中文|[English](./DESIGN.md))
+
+## 1. 项目背景
+
+PaddlePaddle是百度开源的机器学习框架，广泛支持各种深度学习模型的定制化开发; Paddle Serving是Paddle的在线预测部分，与Paddle模型训练环节无缝衔接，提供机器学习预测云服务。本文将从模型、服务、接入等层面，自底向上描述Paddle Serving设计方案。
+
+1. 模型是Paddle Serving预测的核心，包括模型数据和推理计算的管理；
+2. 预测框架封装模型推理计算，对外提供RPC接口，对接不同上游；
+3. 预测服务SDK提供一套接入框架
+
+最终形成一套完整的serving解决方案。
+
+## 2. 名词解释
+
+- **baidu-rpc**: 百度官方开源RPC框架，支持多种常见通信协议，提供基于protobuf的自定义接口体验
+- **Variant**: Paddle Serving架构对一个最小预测集群的抽象，其特点是内部所有实例（副本）完全同质，逻辑上对应一个model的一个固定版本
+- **Endpoint**: 多个Variant组成一个Endpoint，逻辑上看，Endpoint代表一个model，Endpoint内部的Variant代表不同的版本
+- **OP**: PaddlePaddle用来封装一种数值计算的算子，Paddle Serving用来表示一种基础的业务操作算子，核心接口是inference。OP通过配置其依赖的上游OP，将多个OP串联成一个workflow
+- **Channel**: 一个OP所有请求级中间数据的抽象；OP之间通过Channel进行数据交互
+- **Bus**: 对一个线程中所有channel的管理，以及根据DAG之间的DAG依赖图对OP和Channel两个集合间的访问关系进行调度
+- **Stage**: Workflow按照DAG描述的拓扑图中，属于同一个环节且可并行执行的OP集合
+- **Node**: 由某个OP算子类结合参数配置组成的OP算子实例，也是Workflow中的一个执行单元
+- **Workflow**: 按照DAG描述的拓扑，有序执行每个OP的inference接口
+- **DAG/Workflow**: 由若干个相互依赖的Node组成，每个Node均可通过特定接口获得Request对象，节点OP通过依赖关系获得其前置OP的输出对象，最后一个Node的输出默认就是Response对象
+- **Service**: 对一次PV的请求封装，可配置若干条Workflow，彼此之间复用当前PV的Request对象，然后各自并行/串行执行，最后将Response写入对应的输出slot中；一个Paddle-serving进程可配置多套Service接口，上游根据ServiceName决定当前访问的Service接口。
+
+## 3. Python Interface设计
+
+### 3.1 核心目标：
+
+完成一整套Paddle Serving的动态库，支持Paddle保存的通用模型的远程预估服务，通过Python Interface调用PaddleServing底层的各种功能。
+
+### 3.2 通用模型：
+
+能够使用Paddle Inference Library进行预测的模型，在训练过程中保存的模型，包含Feed Variable和Fetch Variable
+
+### 3.3 整体设计：
+
+- 用户通过Python Client启动Client和Server，Python API有检查互联和待访问模型是否匹配的功能
+- Python API背后调用的是Paddle Serving实现的client和server对应功能的pybind，互传的信息通过RPC实现
+- Client Python API当前有两个简单的功能，load_inference_conf和predict，分别用来执行加载待预测的模型和预测
+- Server Python API主要负责加载预估模型，以及生成Paddle Serving需要的各种配置，包括engines，workflow，resource等
+
+### 3.4 Server Inferface
+
+
+
+### 3.5 Client Interface
+
+
+
+### 3.6 训练过程中使用的Client io
+
+PaddleServing设计可以在训练过程中使用的保存模型接口，与Paddle保存inference model的接口基本一致，feed_var_dict与fetch_var_dict
+可以为输入和输出变量起别名，serving启动需要读取的配置会保存在client端和server端的保存目录中。
+
+``` python
+def save_model(server_model_folder,
+ client_config_folder,
+ feed_var_dict,
+ fetch_var_dict,
+ main_program=None)
+```
+
+## 4. Paddle Serving底层框架
+
+
+
+**模型管理框架**：对接多种机器学习平台的模型文件，向上提供统一的inference接口
+**业务调度框架**：对各种不同预测模型的计算逻辑进行抽象，提供通用的DAG调度框架，通过DAG图串联不同的算子，共同完成一次预测服务。该抽象模型使用户可以方便的实现自己的计算逻辑，同时便于算子共用。（用户搭建自己的预测服务，很大一部分工作是搭建DAG和提供算子的实现）
+**PredictService**:对外部提供的预测服务接口封装。通过protobuf定义与客户端的通信字段。
+
+### 4.1 模型管理框架
+
+模型管理框架负责管理机器学习框架训练出来的模型，总体可抽象成模型加载、模型数据和模型推理等3个层次。
+
+#### 模型加载
+
+将模型从磁盘加载到内存，支持多版本、热加载、增量更新等功能
+
+#### 模型数据
+
+模型在内存中的数据结构，集成fluid预测lib
+
+#### inferencer
+
+向上为预测服务提供统一的预测接口
+
+```C++
+class FluidFamilyCore {
+ virtual bool Run(const void* in_data, void* out_data);
+ virtual int create(const std::string& data_path);
+ virtual int clone(void* origin_core);
+};
+```
+
+### 4.2 业务调度框架
+
+#### 4.2.1 预测服务Service
+
+参考TF框架的模型计算的抽象思想，将业务逻辑抽象成DAG图，由配置驱动，生成workflow，跳过C++代码编译。业务的每个具体步骤，对应一个具体的OP，OP可配置自己依赖的上游OP。OP之间消息传递统一由线程级Bus和channel机制实现。例如，一个简单的预测服务的服务过程，可以抽象成读请求数据->调用预测接口->写回预测结果等3个步骤，相应的实现到3个OP: ReaderOp->ClassifyOp->WriteOp
+
+
+
+关于OP之间的依赖关系，以及通过OP组建workflow，可以参考[从零开始写一个预测服务](CREATING.md)的相关章节
+
+服务端实例透视图
+
+
+
+
+#### 4.2.2 Paddle Serving的多服务机制
+
+
+
+Paddle Serving实例可以同时加载多个模型，每个模型用一个Service（以及其所配置的workflow）承接服务。可以参考[Demo例子中的service配置文件](../tools/cpp_examples/demo-serving/conf/service.prototxt)了解如何为serving实例配置多个service
+
+#### 4.2.3 业务调度层级关系
+
+从客户端看，一个Paddle Serving service从顶向下可分为Service, Endpoint, Variant等3个层级
+
+
+
+一个Service对应一个预测模型，模型下有1个endpoint。模型的不同版本，通过endpoint下多个variant概念实现：
+同一个模型预测服务，可以配置多个variant，每个variant有自己的下游IP列表。客户端代码可以对各个variant配置相对权重，以达到调节流量比例的关系（参考[客户端配置](./deprecated/CLIENT_CONFIGURE.md)第3.2节中关于variant_weight_list的说明）。
+
+
+
+## 5. 用户接口
+
+在满足一定的接口规范前提下，服务框架不对用户数据字段做任何约束，以应对各种预测服务的不同业务接口。Baidu-rpc继承了Protobuf serice的接口，用户按照Protobuf语法规范描述Request和Response业务接口。Paddle Serving基于Baidu-rpc框架搭建，默认支持该特性。
+
+无论通信协议如何变化，框架只需确保Client和Server间通信协议和业务数据两种信息的格式同步，即可保证正常通信。这些信息又可细分如下：
+
+- 协议：Server和Client之间事先约定的、确保相互识别数据格式的包头信息。Paddle Serving用Protobuf作为基础通信格式
+- 数据：用来描述Request和Response的接口，例如待预测样本数据，和预测返回的打分。包括：
+ - 数据字段：请求包Request和返回包Response两种数据结构包含的字段定义
+ - 描述接口：跟协议接口类似，默认支持Protobuf
+
+### 5.1 数据压缩方法
+
+Baidu-rpc内置了snappy, gzip, zlib等数据压缩方法，可在配置文件中配置（参考[客户端配置](./deprecated/CLIENT_CONFIGURE.md)第3.1节关于compress_type的介绍）
+
+### 5.2 C++ SDK API接口
+
+```C++
+class PredictorApi {
+ public:
+ int create(const char* path, const char* file);
+ int thrd_initialize();
+ int thrd_clear();
+ int thrd_finalize();
+ void destroy();
+
+ Predictor* fetch_predictor(std::string ep_name);
+ int free_predictor(Predictor* predictor);
+};
+
+class Predictor {
+ public:
+ // synchronize interface
+ virtual int inference(google::protobuf::Message* req,
+ google::protobuf::Message* res) = 0;
+
+ // asynchronize interface
+ virtual int inference(google::protobuf::Message* req,
+ google::protobuf::Message* res,
+ DoneType done,
+ brpc::CallId* cid = NULL) = 0;
+
+ // synchronize interface
+ virtual int debug(google::protobuf::Message* req,
+ google::protobuf::Message* res,
+ butil::IOBufBuilder* debug_os) = 0;
+};
+
+```
+
+### 5.3 OP相关接口
+
+```C++
+class Op {
+ // ------Getters for Channel/Data/Message of dependent OP-----
+
+ // Get the Channel object of dependent OP
+ Channel* mutable_depend_channel(const std::string& op);
+
+ // Get the Channel object of dependent OP
+ const Channel* get_depend_channel(const std::string& op) const;
+
+ template
-### 2.4 微服务插件模式 -由于Paddle Serving底层采用基于C++的通信组件，并且核心框架也是基于C/C++编写，当用户想要在服务端定义复杂的前处理与后处理逻辑时，一种办法是修改Paddle Serving底层框架，重新编译源码。另一种方式可以通过在服务端嵌入轻量级的Web服务，通过在Web服务中实现更复杂的预处理逻辑，从而搭建一套逻辑完整的服务。当访问量超过了Web服务能够接受的范围，开发者有足够的理由开发一些高性能的C++预处理逻辑，并嵌入到Serving的原生服务库中。Web服务和RPC服务的关系以及他们的组合方式可以参考下文`用户类型`中的说明。 +### 2.4 Micro service plugin +The underlying communication of Paddle Serving is implemented with C++ as well as the core framework, it is hard for users who do not familiar with C++ to implement new Paddle Serving Server Operators. Another approach is to use the light-weighted Web Service in Paddle Serving Server that can be viewed as a plugin. A user can implement complex data preprocessing and postprocessing logics to build a complex AI service. If access of the AI service has a large volumn, it is worth to implement the service with high performance Paddle Serving Server operators. The relationship between Web Service and RPC Service can be referenced in `User Type`. -## 3. 工业级特性 +## 3. Industrial Features -### 3.1 分布式稀疏参数索引 +### 3.1 Distributed Sparse Parameter Indexing -分布式稀疏参数索引通常在广告推荐中出现，并与分布式训练配合形成完整的离线-在线一体化部署。下图解释了其中的流程，产品的在线服务接受用户请求后将请求发送给预估服务，同时系统会记录用户的请求以进行相应的训练日志处理和拼接。离线分布式训练系统会针对流式产出的训练日志进行模型增量训练，而增量产生的模型会配送至分布式稀疏参数索引服务，同时对应的稠密的模型参数也会配送至在线的预估服务。在线服务由两部分组成，一部分是针对用户的请求提取特征后，将需要进行模型的稀疏参数索引的特征发送请求给分布式稀疏参数索引服务，针对分布式稀疏参数索引服务返回的稀疏参数再进行后续深度学习模型的计算流程，从而完成预估。 +Distributed Sparse Parameter Indexing is commonly seen in advertising and recommendation scenarios, and is often used coupled with distributed training. The figure below explains a commonly seen architecture for online recommendation. When the recommendation service receives a request from a user, the system will automatically collects training log for the offline distributed online training. Mean while, the request is sent to Paddle Serving Server. For sparse features, distributed sparse parameter index service is called so that sparse parameters can be looked up. The dense input features together with the looked up sparse model parameters are fed into the Paddle Inference Node of the DAG in Paddle Serving Server. Then the score can be responsed through RPC to product service for item ranking.
- -为什么要使用Paddle Serving提供的分布式稀疏参数索引服务？1）在一些推荐场景中，模型的输入特征规模通常可以达到上千亿，单台机器无法支撑T级别模型在内存的保存，因此需要进行分布式存储。2）Paddle Serving提供的分布式稀疏参数索引服务，具有并发请求多个节点的能力，从而以较低的延时完成预估服务。 + +Why do we need to support distributed sparse parameter indexing in Paddle Serving? 1) In some recommendation scenarios, the number of features can be up to hundreds of billions that a single node can not hold the parameters within random access memory. 2) Paddle Serving supports distributed sparse parameter indexing that can couple with paddle inference. Users do not need to do extra work to have a low latency inference engine with hundreds of billions of parameters. -### 3.2 模型管理、在线A/B流量测试、模型热加载 +### 3.2 Model Management, online A/B test, Model Online Reloading -Paddle Serving的C++引擎支持模型管理、在线A/B流量测试、模型热加载等功能，当前在Python API还有没完全开放这部分功能的配置，敬请期待。 +Paddle Serving's C++ engine supports model management, online A/B test and model online reloading. Currently, python API is not released yet, please wait for the next release. -## 4. 用户类型 -Paddle Serving面向的用户提供RPC和HTTP两种访问协议。对于HTTP协议，我们更倾向于流量中小型的服务使用，并且对延时没有严格要求的AI服务开发者。对于RPC协议，我们面向流量较大，对延时要求更高的用户，此外RPC的客户端可能也处在一个大系统的服务中，这种情况下非常适合使用Paddle Serving提供的RPC服务。对于使用分布式稀疏参数索引服务而言，Paddle Serving的用户不需要关心底层的细节，其调用本质也是通过RPC服务再调用RPC服务。下图给出了当前设计的Paddle Serving可能会使用Serving服务的几种场景。 +## 4. User Types +Paddle Serving provides RPC and HTTP protocol for users. For HTTP service, we recommend users with median or small traffic services to use, and the latency is not a strict requirement. For RPC protocol, we recommend high traffic services and low latency required services to use. For users who use distributed sparse parameter indexing built-in service, it is not necessary to care about the underlying details of communication. The following figure gives out several scenarios that user may want to use Paddle Serving.
@@ -173,11 +180,11 @@ Paddle Serving面向的用户提供RPC和HTTP两种访问协议。对于HTTP协
-对于普通的模型而言（具体指通过Serving提供的IO保存的模型，并且没有对模型进行后处理），用户使用RPC服务不需要额外的开发即可实现服务启动，但需要开发一些Client端的代码来使用服务。对于Web服务的开发，需要用户现在Paddle Serving提供的Web Service框架中进行前后处理的开发，从而实现整个HTTP服务。 +For servable models saved from Paddle Serving IO API, users do not need to do extra coding work to startup a service, but may need some coding work on the client side. For development of Web Service plugin, a user needs to provide implementation of Web Service's preprocessing and postprocessing work if needed to get a HTTP service. -### 4.1 Web服务开发 +### 4.1 Web Service Development -Web服务有很多开源的框架，Paddle Serving当前集成了Flask框架，但这部分对用户不可见，在未来可能会提供性能更好的Web框架作为底层HTTP服务集成引擎。用户需要继承WebService，从而实现对rpc服务的输入输出进行加工的目的。 +Web Service has lots of open sourced framework. Currently Paddle Serving uses Flask as built-in service framework, and users are not aware of this. More efficient web service will be integrated in the furture if needed. ``` python from paddle_serving_server.web_service import WebService @@ -208,15 +215,15 @@ imdb_service.prepare_dict({"dict_file_path": sys.argv[4]}) imdb_service.run_server() ``` -`WebService`作为基类，提供将用户接受的HTTP请求转化为RPC输入的接口`preprocess`，同时提供对RPC请求返回的结果进行后处理的接口`postprocess`，继承`WebService`的子类，可以定义各种类型的成员函数。`WebService`的启动命令和普通RPC服务提供的启动API一致。 +`WebService` is a Base Class, providing inheritable interfaces such `preprocess` and `postprocess` for users to implement. In the inherited class of `WebService` class, users can define any functions they want and the startup function interface is the same as RPC service. -## 5. 未来计划 +## 5. Future Plan -### 5.1 有向无环图结构定义开放 -当前版本开放的python API仅支持用户定义Sequential类型的执行流，如果想要进行Server进程内复杂的计算，需要增加对应的用户API。 +### 5.1 Open DAG definition API +Current version of Paddle Serving Server supports sequential type of execution flow. DAG definition API can be more helpful to users on complex tasks. -### 5.2 云端自动部署能力 -为了方便用户更容易将Paddle的预测模型部署到线上，Paddle Serving在接下来的版本会提供Kubernetes生态下任务编排的工具。 +### 5.2 Auto Deployment on Cloud +In order to make deployment more easily on public cloud, Paddle Serving considers to provides Operators on Kubernetes in submitting a service job. -### 5.3 向量检索、树结构检索 -在推荐与广告场景的召回系统中，通常需要采用基于向量的快速检索或者基于树结构的快速检索，Paddle Serving会对这方面的检索引擎进行集成或扩展。 +### 5.3 Vector Indexing and Tree based Indexing +In recommendation and advertisement systems, it is commonly seen to use vector based index or tree based indexing service to do candidate retrievals. These retrieval tasks will be built-in services of Paddle Serving. diff --git a/doc/DESIGN_DOC_CN.md b/doc/DESIGN_DOC_CN.md new file mode 100644 index 0000000000000000000000000000000000000000..2a63d56593dc47a5ca69f9c5c324710ee6dc3fc6 --- /dev/null +++ b/doc/DESIGN_DOC_CN.md @@ -0,0 +1,224 @@ +# Paddle Serving设计文档 + +(简体中文|[English](./DESIGN_DOC.md)) + +## 1. 整体设计目标 + +- 长期使命：Paddle Serving是一个PaddlePaddle开源的在线服务框架，长期目标就是围绕着人工智能落地的最后一公里提供越来越专业、可靠、易用的服务。 + +- 工业级：为了达到工业级深度学习模型在线部署的要求， +Paddle Serving提供很多大规模场景需要的部署功能：1）分布式稀疏参数索引功能；2）高并发底层通信能力；3）模型管理、在线A/B流量测试、模型热加载。 + +- 简单易用：为了让使用Paddle的用户能够以极低的成本部署模型，PaddleServing设计了一套与Paddle训练框架无缝打通的预测部署API，普通模型可以使用一行命令进行服务部署。 + +- 功能扩展：当前，Paddle Serving支持C++、Python、Golang的客户端，未来也会面向不同类型的客户新增多种语言的客户端。在Paddle Serving的框架设计方面，尽管当前Paddle Serving以支持Paddle模型的部署为核心功能， +用户可以很容易嵌入其他的机器学习库部署在线预测。 + +## 2. 模块设计与实现 + +### 2.1 Python API接口设计 + +#### 2.1.1 训练模型的保存 +Paddle的模型预测需要重点关注的内容：1）模型的输入变量；2）模型的输出变量；3）模型结构和模型参数。Paddle Serving Python API提供用户可以在训练过程中保存模型的接口，并将Paddle Serving在部署阶段需要保存的配置打包保存，一个示例如下： +``` python +import paddle_serving_client.io as serving_io +serving_io.save_model("serving_model", "client_conf", + {"words": data}, {"prediction": prediction}, + fluid.default_main_program()) +``` +代码示例中，`{"words": data}`和`{"prediction": prediction}`分别指定了模型的输入和输出，`"words"`和`"prediction"`是输出和输出变量的别名，设计别名的目的是为了使开发者能够记忆自己训练模型的输入输出对应的字段。`data`和`prediction`则是Paddle训练过程中的`[Variable](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Variable_cn.html#variable)`，通常代表张量([Tensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Tensor_cn.html#tensor))或变长张量([LodTensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/beginners_guide/basic_concept/lod_tensor.html#lodtensor))。调用保存命令后，会按照用户指定的`"serving_model"`和`"client_conf"`生成两个目录，内容如下： +``` shell +. +├── client_conf +│ ├── serving_client_conf.prototxt +│ └── serving_client_conf.stream.prototxt +└── serving_model + ├── embedding_0.w_0 + ├── fc_0.b_0 + ├── fc_0.w_0 + ├── fc_1.b_0 + ├── fc_1.w_0 + ├── fc_2.b_0 + ├── fc_2.w_0 + ├── lstm_0.b_0 + ├── lstm_0.w_0 + ├── __model__ + ├── serving_server_conf.prototxt + └── serving_server_conf.stream.prototxt +``` +其中，`"serving_client_conf.prototxt"`和`"serving_server_conf.prototxt"`是Paddle Serving的Client和Server端需要加载的配置，`"serving_client_conf.stream.prototxt"`和`"serving_server_conf.stream.prototxt"`是配置文件的二进制形式。`"serving_model"`下保存的其他内容和Paddle保存的模型文件是一致的。我们会考虑未来在Paddle框架中直接保存可服务的配置，实现配置保存对用户无感。 + +#### 2.1.2 服务端模型加载 + +服务端的预测逻辑可以通过Paddle Serving Server端的API进行人工定义，一个例子： +``` python +import paddle_serving_server as serving +op_maker = serving.OpMaker() +read_op = op_maker.create('general_reader') +dist_kv_op = op_maker.create('general_dist_kv') +general_infer_op = op_maker.create('general_infer') +general_response_op = op_maker.create('general_response') + +op_seq_maker = serving.OpSeqMaker() +op_seq_maker.add_op(read_op) +op_seq_maker.add_op(dist_kv_op) +op_seq_maker.add_op(general_infer_op) +op_seq_maker.add_op(general_response_op) +``` + +当前Paddle Serving在Server端支持的主要Op请参考如下列表： + +
+
+
+
+
+ +### 2.4 微服务插件模式 +由于Paddle Serving底层采用基于C++的通信组件，并且核心框架也是基于C/C++编写，当用户想要在服务端定义复杂的前处理与后处理逻辑时，一种办法是修改Paddle Serving底层框架，重新编译源码。另一种方式可以通过在服务端嵌入轻量级的Web服务，通过在Web服务中实现更复杂的预处理逻辑，从而搭建一套逻辑完整的服务。当访问量超过了Web服务能够接受的范围，开发者有足够的理由开发一些高性能的C++预处理逻辑，并嵌入到Serving的原生服务库中。Web服务和RPC服务的关系以及他们的组合方式可以参考下文`用户类型`中的说明。 + +## 3. 工业级特性 + +### 3.1 分布式稀疏参数索引 + +分布式稀疏参数索引通常在广告推荐中出现，并与分布式训练配合形成完整的离线-在线一体化部署。下图解释了其中的流程，产品的在线服务接受用户请求后将请求发送给预估服务，同时系统会记录用户的请求以进行相应的训练日志处理和拼接。离线分布式训练系统会针对流式产出的训练日志进行模型增量训练，而增量产生的模型会配送至分布式稀疏参数索引服务，同时对应的稠密的模型参数也会配送至在线的预估服务。在线服务由两部分组成，一部分是针对用户的请求提取特征后，将需要进行模型的稀疏参数索引的特征发送请求给分布式稀疏参数索引服务，针对分布式稀疏参数索引服务返回的稀疏参数再进行后续深度学习模型的计算流程，从而完成预估。 + +
+
+
+
+
+ +为什么要使用Paddle Serving提供的分布式稀疏参数索引服务？1）在一些推荐场景中，模型的输入特征规模通常可以达到上千亿，单台机器无法支撑T级别模型在内存的保存，因此需要进行分布式存储。2）Paddle Serving提供的分布式稀疏参数索引服务，具有并发请求多个节点的能力，从而以较低的延时完成预估服务。 + +### 3.2 模型管理、在线A/B流量测试、模型热加载 + +Paddle Serving的C++引擎支持模型管理、在线A/B流量测试、模型热加载等功能，当前在Python API还有没完全开放这部分功能的配置，敬请期待。 + +## 4. 用户类型 +Paddle Serving面向的用户提供RPC和HTTP两种访问协议。对于HTTP协议，我们更倾向于流量中小型的服务使用，并且对延时没有严格要求的AI服务开发者。对于RPC协议，我们面向流量较大，对延时要求更高的用户，此外RPC的客户端可能也处在一个大系统的服务中，这种情况下非常适合使用Paddle Serving提供的RPC服务。对于使用分布式稀疏参数索引服务而言，Paddle Serving的用户不需要关心底层的细节，其调用本质也是通过RPC服务再调用RPC服务。下图给出了当前设计的Paddle Serving可能会使用Serving服务的几种场景。 + +
+
+
+
+
+
+对于普通的模型而言（具体指通过Serving提供的IO保存的模型，并且没有对模型进行后处理），用户使用RPC服务不需要额外的开发即可实现服务启动，但需要开发一些Client端的代码来使用服务。对于Web服务的开发，需要用户现在Paddle Serving提供的Web Service框架中进行前后处理的开发，从而实现整个HTTP服务。
+
+### 4.1 Web服务开发
+
+Web服务有很多开源的框架，Paddle Serving当前集成了Flask框架，但这部分对用户不可见，在未来可能会提供性能更好的Web框架作为底层HTTP服务集成引擎。用户需要继承WebService，从而实现对rpc服务的输入输出进行加工的目的。
+
+``` python
+from paddle_serving_server.web_service import WebService
+from imdb_reader import IMDBDataset
+import sys
+
+
+class IMDBService(WebService):
+ def prepare_dict(self, args={}):
+ if len(args) == 0:
+ exit(-1)
+ self.dataset = IMDBDataset()
+ self.dataset.load_resource(args["dict_file_path"])
+
+ def preprocess(self, feed={}, fetch=[]):
+ if "words" not in feed:
+ exit(-1)
+ res_feed = {}
+ res_feed["words"] = self.dataset.get_words_only(feed["words"])[0]
+ return res_feed, fetch
+
+
+imdb_service = IMDBService(name="imdb")
+imdb_service.load_model_config(sys.argv[1])
+imdb_service.prepare_server(
+ workdir=sys.argv[2], port=int(sys.argv[3]), device="cpu")
+imdb_service.prepare_dict({"dict_file_path": sys.argv[4]})
+imdb_service.run_server()
+```
+
+`WebService`作为基类，提供将用户接受的HTTP请求转化为RPC输入的接口`preprocess`，同时提供对RPC请求返回的结果进行后处理的接口`postprocess`，继承`WebService`的子类，可以定义各种类型的成员函数。`WebService`的启动命令和普通RPC服务提供的启动API一致。
+
+## 5. 未来计划
+
+### 5.1 有向无环图结构定义开放
+当前版本开放的python API仅支持用户定义Sequential类型的执行流，如果想要进行Server进程内复杂的计算，需要增加对应的用户API。
+
+### 5.2 云端自动部署能力
+为了方便用户更容易将Paddle的预测模型部署到线上，Paddle Serving在接下来的版本会提供Kubernetes生态下任务编排的工具。
+
+### 5.3 向量检索、树结构检索
+在推荐与广告场景的召回系统中，通常需要采用基于向量的快速检索或者基于树结构的快速检索，Paddle Serving会对这方面的检索引擎进行集成或扩展。
diff --git a/doc/README.md b/doc/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..2d51eba9e2a2902685f9385c83542f32b98e5b4f
--- /dev/null
+++ b/doc/README.md
@@ -0,0 +1,119 @@
+# Paddle Serving
+
+([简体中文](./README_CN.md)|English)
+
+Paddle Serving is PaddlePaddle's online estimation service framework, which can help developers easily implement remote prediction services that call deep learning models from mobile and server ends. At present, Paddle Serving is mainly based on models that support PaddlePaddle training. It can be used in conjunction with the Paddle training framework to quickly deploy inference services. Paddle Serving is designed around common industrial-level deep learning model deployment scenarios. Some common functions include multi-model management, model hot loading, [Baidu-rpc](https://github.com/apache/incubator-brpc)-based high-concurrency low-latency response capabilities, and online model A/B tests. The API that cooperates with the Paddle training framework can enable users to seamlessly transition between training and remote deployment, improving the landing efficiency of deep learning models.
+
+------------
+
+## Quick Start
+
+Paddle Serving's current develop version supports lightweight Python API for fast predictions, and training with Paddle can get through. We take the most classic Boston house price prediction as an example to fully explain the process of model training on a single machine and model deployment using Paddle Serving.
+
+#### Install
+
+It is highly recommended that you build Paddle Serving inside Docker, please read [How to run PaddleServing in Docker](RUN_IN_DOCKER.md)
+
+```
+pip install paddle-serving-client
+pip install paddle-serving-server
+```
+
+#### Training Script
+``` python
+import sys
+import paddle
+import paddle.fluid as fluid
+
+train_reader = paddle.batch(paddle.reader.shuffle(
+ paddle.dataset.uci_housing.train(), buf_size=500), batch_size=16)
+
+test_reader = paddle.batch(paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500), batch_size=16)
+
+x = fluid.data(name='x', shape=[None, 13], dtype='float32')
+y = fluid.data(name='y', shape=[None, 1], dtype='float32')
+
+y_predict = fluid.layers.fc(input=x, size=1, act=None)
+cost = fluid.layers.square_error_cost(input=y_predict, label=y)
+avg_loss = fluid.layers.mean(cost)
+sgd_optimizer = fluid.optimizer.SGD(learning_rate=0.01)
+sgd_optimizer.minimize(avg_loss)
+
+place = fluid.CPUPlace()
+feeder = fluid.DataFeeder(place=place, feed_list=[x, y])
+exe = fluid.Executor(place)
+exe.run(fluid.default_startup_program())
+
+import paddle_serving_client.io as serving_io
+
+for pass_id in range(30):
+ for data_train in train_reader():
+ avg_loss_value, = exe.run(
+ fluid.default_main_program(),
+ feed=feeder.feed(data_train),
+ fetch_list=[avg_loss])
+
+serving_io.save_model(
+ "serving_server_model", "serving_client_conf",
+ {"x": x}, {"y": y_predict}, fluid.default_main_program())
+```
+
+#### Server Side Code
+``` python
+import sys
+from paddle_serving.serving_server import OpMaker
+from paddle_serving.serving_server import OpSeqMaker
+from paddle_serving.serving_server import Server
+
+op_maker = OpMaker()
+read_op = op_maker.create('general_reader')
+general_infer_op = op_maker.create('general_infer')
+
+op_seq_maker = OpSeqMaker()
+op_seq_maker.add_op(read_op)
+op_seq_maker.add_op(general_infer_op)
+
+server = Server()
+server.set_op_sequence(op_seq_maker.get_op_sequence())
+server.load_model_config(sys.argv[1])
+server.prepare_server(workdir="work_dir1", port=9393, device="cpu")
+server.run_server()
+```
+
+#### Launch Server End
+``` shell
+python test_server.py serving_server_model
+```
+
+#### Client Prediction
+``` python
+from paddle_serving_client import Client
+import paddle
+import sys
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9292"])
+
+test_reader = paddle.batch(paddle.reader.shuffle(
+ paddle.dataset.uci_housing.test(), buf_size=500), batch_size=1)
+
+for data in test_reader():
+ fetch_map = client.predict(feed={"x": data[0][0]}, fetch=["y"])
+ print("{} {}".format(fetch_map["y"][0], data[0][1][0]))
+
+```
+
+### Document
+
+[Design Doc](DESIGN.md)
+
+[FAQ](./deprecated/FAQ.md)
+
+### Senior Developer Guildlines
+
+[Compile Tutorial](COMPILE.md)
+
+## Contribution
+If you want to make contributions to Paddle Serving Please refer to [CONRTIBUTE](CONTRIBUTE.md)
diff --git a/doc/README_CN.md b/doc/README_CN.md
index f8d42e6f1e72f1ac34939e5795df3e6604924bad..da5641cad333518ded9fbae4438f05ae20e30ddd 100644
--- a/doc/README_CN.md
+++ b/doc/README_CN.md
@@ -1,5 +1,7 @@
# Paddle Serving
+(简体中文|[English](./README.md))
+
Paddle Serving是PaddlePaddle的在线预估服务框架，能够帮助开发者轻松实现从移动端、服务器端调用深度学习模型的远程预测服务。当前Paddle Serving以支持PaddlePaddle训练的模型为主，可以与Paddle训练框架联合使用，快速部署预估服务。Paddle Serving围绕常见的工业级深度学习模型部署场景进行设计，一些常见的功能包括多模型管理、模型热加载、基于[Baidu-rpc](https://github.com/apache/incubator-brpc)的高并发低延迟响应能力、在线模型A/B实验等。与Paddle训练框架互相配合的API可以使用户在训练与远程部署之间无缝过度，提升深度学习模型的落地效率。
------------
@@ -10,7 +12,7 @@ Paddle Serving当前的develop版本支持轻量级Python API进行快速预测
#### 安装
-强烈建议您在Docker内构建Paddle Serving，请查看[如何在Docker中运行PaddleServing](doc/RUN_IN_DOCKER_CN.md)
+强烈建议您在Docker内构建Paddle Serving，请查看[如何在Docker中运行PaddleServing](RUN_IN_DOCKER_CN.md)
```
pip install paddle-serving-client
@@ -105,13 +107,13 @@ for data in test_reader():
### 文档
-[设计文档](doc/DESIGN.md)
+[设计文档](DESIGN_CN.md)
-[FAQ](doc/FAQ.md)
+[FAQ](./deprecated/FAQ.md)
### 资深开发者使用指南
-[编译指南](doc/INSTALL.md)
+[编译指南](COMPILE_CN.md)
## 贡献
-如果你想要给Paddle Serving做贡献，请参考[贡献指南](doc/CONTRIBUTE.md)
+如果你想要给Paddle Serving做贡献，请参考[贡献指南](CONTRIBUTE.md)
diff --git a/doc/RUN_IN_DOCKER.md b/doc/RUN_IN_DOCKER.md
index 972de2d951e602d025fb5fcb8b3229dcc300f696..708739851b8e3ec5ca8b5e204a68169ec88041b5 100644
--- a/doc/RUN_IN_DOCKER.md
+++ b/doc/RUN_IN_DOCKER.md
@@ -1,5 +1,7 @@
# How to run PaddleServing in Docker
+([简体中文](./RUN_IN_DOCKER_CN.md)|English)
+
## Requirements
Docker (GPU version requires nvidia-docker to be installed on the GPU machine)
diff --git a/doc/RUN_IN_DOCKER_CN.md b/doc/RUN_IN_DOCKER_CN.md
index 17bdd30adbcbecd971904011208fe01d1d08f5ba..9f2abba176ca89f6d03d9602c2fd1e7d4a78980b 100644
--- a/doc/RUN_IN_DOCKER_CN.md
+++ b/doc/RUN_IN_DOCKER_CN.md
@@ -1,5 +1,7 @@
# 如何在Docker中运行PaddleServing
+(简体中文|[English](RUN_IN_DOCKER.md))
+
## 环境要求
Docker（GPU版本需要在GPU机器上安装nvidia-docker）
diff --git a/doc/TRAIN_TO_SERVICE.md b/doc/TRAIN_TO_SERVICE.md
index 11e64eebed84be9889f6e833511bdade897aeb23..4219e66948a9bc3b0ae43e5cda61aad8ae35b3a0 100644
--- a/doc/TRAIN_TO_SERVICE.md
+++ b/doc/TRAIN_TO_SERVICE.md
@@ -1,36 +1,36 @@
-# 端到端完成从训练到部署全流程
+# An End-to-end Tutorial from Training to Inference Service Deployment
-Paddle Serving是Paddle的高性能在线预测服务框架，可以灵活支持大多数模型的部署。本文中将以IMDB评论情感分析任务为例通过9步展示从模型的训练到部署预测服务的全流程。
+([简体中文](./TRAIN_TO_SERVICE_CN.md)|English)
-## Step1：准备环境
+Paddle Serving is Paddle's high-performance online inference service framework, which can flexibly support the deployment of most models. In this article, the IMDB review sentiment analysis task is used as an example to show the entire process from model training to deployment of inference service through 9 steps.
-Paddle Serving可以部署在Centos和Ubuntu等Linux环境上，在其他系统上或者不希望安装serving模块的环境中仍然可以通过http服务来访问server端的预测服务。
+## Step1：Prepare for Running Environment
+Paddle Serving can be deployed on Linux environments such as Centos and Ubuntu. On other systems or in environments where you do not want to install the serving module, you can still access the server-side prediction service through the http service.
-可以根据需求和机器环境来选择安装cpu或gpu版本的server模块，在client端机器上安装client模块。当希望同http来访问server端
+You can choose to install the cpu or gpu version of the server module according to the requirements and machine environment, and install the client module on the client machine. When you want to access the server with http
```shell
-pip install paddle_serving_server #cpu版本server端
-pip install paddle_serving_server_gpu #gpu版本server端
-pip install paddle_serving_client #client端
+pip install paddle_serving_server #cpu version server side
+pip install paddle_serving_server_gpu #gpu version server side
+pip install paddle_serving_client #client version
```
-简单准备后，我们将以IMDB评论情感分析任务为例，展示从模型训练到部署预测服务的流程。示例中的所有代码都可以在Paddle Serving代码库的[IMDB示例](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb)中找到，示例中使用的数据和词典文件可以通过执行IMDB示例代码中的get_data.sh脚本得到。
+After simple preparation, we will take the IMDB review sentiment analysis task as an example to show the process from model training to deployment of prediction services. All the code in the example can be found in the [IMDB example](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb) of the Paddle Serving code base, the data and dictionary used in the example The file can be obtained by executing the get_data.sh script in the IMDB sample code.
-## Step2：确定任务和原始数据格式
+## Step2：Determine Tasks and Raw Data Format
-IMDB评论情感分析任务是对电影评论的内容进行二分类，判断该评论是属于正面评论还是负面评论。
-
-首先我们来看一下原始的数据：
+IMDB review sentiment analysis task is to classify the content of movie reviews to determine whether the review is a positive review or a negative review.
+First let's take a look at the raw data:
```
saw a trailer for this on another video, and decided to rent when it came out. boy, was i disappointed! the story is extremely boring, the acting (aside from christopher walken) is bad, and i couldn't care less about the characters, aside from really wanting to see nora's husband get thrashed. christopher walken's role is such a throw-away, what a tease! | 0
```
-这是一条英文评论样本，样本中使用|作为分隔符，分隔符之前为评论的内容，分隔符之后是样本的标签，0代表负样本，即负面评论，1代表正样本，即正面评论。
+This is a sample of English comments. The sample uses | as the separator. The content of the comment is before the separator. The label is the sample after the separator. 0 is the negative while 1 is the positive.
-## Step3：定义Reader，划分训练集、测试集
+## Step3：Define Reader, divide training set and test set
-对于原始文本我们需要将它转化为神经网络可以使用的数字id。imdb_reader.py脚本中定义了文本id化的方法，通过词典文件imdb.vocab将单词映射为整形数。
+For the original text we need to convert it to a numeric id that the neural network can use. The imdb_reader.py script defines the method of text idization, and the words are mapped to integers through the dictionary file imdb.vocab.
-映射之后的样本类似于以下的格式：
+The sample after mapping is similar to the following format:
```
257 142 52 898 7 0 12899 1083 824 122 89527 134 6 65 47 48 904 89527 13 0 87 170 8 248 9 15 4 25 1365 4360 89527 702 89527 1 89527 240 3 28 89527 19 7 0 216 219 614 89527 0 84 89527 225 3 0 15 67 2356 89527 0 498 117 2 314 282 7 38 1097 89527 1 0 174 181 38 11 71 198 44 1 3110 89527 454 89527 34 37 89527 0 15 5912 80 2 9856 7748 89527 8 421 80 9 15 14 55 2218 12 4 45 6 58 25 89527 154 119 224 41 0 151 89527 871 89527 505 89527 501 89527 29 2 773 211 89527 54 307 90 0 893 89527 9 407 4 25 2 614 15 46 89527 89527 71 8 1356 35 89527 12 0 89527 89527 89 527 577 374 3 39091 22950 1 3771 48900 95 371 156 313 89527 37 154 296 4 25 2 217 169 3 2759 7 0 15 89527 0 714 580 11 2094 559 34 0 84 539 89527 1 0 330 355 3 0 15 15607 935 80 0 5369 3 0 622 89527 2 15 36 9 2291 2 7599 6968 2449 89527 1 454 37 256 2 211 113 0 480 218 1152 700 4 1684 1253 352 10 2449 89527 39 4 1819 129 1 316 462 29 0 12957 3 6 28 89527 13 0 457 8952 7 225 89527 8 2389 0 1514 89527 1
```
-这样神经网络就可以将转化后的文本信息作为特征值进行训练。
+In this way, the neural network can train the transformed text information as feature values.
-## Step4：定义CNN网络进行训练并保存
+## Step4：Define CNN network for training and saving
-接下来我们使用[CNN模型](https://www.paddlepaddle.org.cn/documentation/docs/zh/user_guides/nlp_case/understand_sentiment/README.cn.html#cnn)来进行训练。在nets.py脚本中定义网络结构。
+Net we use [CNN Model](https://www.paddlepaddle.org.cn/documentation/docs/zh/user_guides/nlp_case/understand_sentiment/README.cn.html#cnn) for training, in nets.py we define the network structure.
imdb_reader.py
@@ -102,17 +102,17 @@ class IMDBDataset(dg.MultiSlotDataGenerator):
```
-使用训练样本进行训练，训练脚本为local_train.py。在训练结束后使用paddle_serving_client.io.save_model函数来保存部署预测服务使用的模型文件和配置文件。
+Use training dataset for training. The training script is local_train.py. After training, use the paddle_serving_client.io.save_model function to save the model files and configuration files used by the servingdeployment.
nets.py
@@ -156,7 +156,7 @@ def cnn_net(data,
-由上图可以看出模型的损失在第65轮之后开始收敛，我们在第65轮训练完成后保存模型和配置文件。保存的文件分为imdb_cnn_client_conf和imdb_cnn_model文件夹，前者包含client端的配置文件，后者包含server端的配置文件和保存的模型文件。
-save_model函数的参数列表如下：
+! [Training process](./ imdb_loss.png) As can be seen from the above figure, the loss of the model starts to converge after the 65th round. We save the model and configuration file after the 65th round of training is completed. The saved files are divided into imdb_cnn_client_conf and imdb_cnn_model folders. The former contains client-side configuration files, and the latter contains server-side configuration files and saved model files.
+The parameter list of the save_model function is as follows:
-| 参数 | 含义 |
+| Parameter | Meaning |
| -------------------- | ------------------------------------------------------------ |
-| server_model_folder | 保存server端配置文件和模型文件的目录 |
-| client_config_folder | 保存client端配置文件的目录 |
-| feed_var_dict | 用于预测的模型的输入，dict类型，key可以自定义，value为模型中的input variable，每个key对应一个variable，使用预测服务时，输入数据使用key作为输入的名称 |
-| fetch_var_dict | 用于预测的模型的输出，dict类型，key可以自定义，value为模型中的input variable，每个key对应一个variable，使用预测服务时，通过key来获取返回数据 |
-| main_program | 模型的program |
+| server_model_folder | Directory for server-side configuration files and model files |
+| client_config_folder | Directory for saving client configuration files |
+| feed_var_dict | The input of the inference model. The dict type and key can be customized. The value is the input variable in the model. Each key corresponds to a variable. When using the prediction service, the input data uses the key as the input name. |
+| fetch_var_dict | The output of the model used for prediction, dict type, key can be customized, value is the input variable in the model, and each key corresponds to a variable. When using the prediction service, use the key to get the returned data |
+| main_program | Model's program |
-## Step5：部署RPC预测服务
+## Step5: Deploy RPC Prediction Service
-Paddle Serving框架支持两种预测服务方式，一种是通过RPC进行通信，一种是通过HTTP进行通信，下面将先介绍RPC预测服务的部署和使用方法，在Step8开始介绍HTTP预测服务的部署和使用。
+The Paddle Serving framework supports two types of prediction service methods. One is to communicate through RPC and the other is to communicate through HTTP. The deployment and use of RPC prediction service will be introduced first. The deployment and use of HTTP prediction service will be introduced at Step 8. .
-```shell
-python -m paddle_serving_server.serve --model imdb_cnn_model/ --port 9292 #cpu预测服务
-python -m paddle_serving_server_gpu.serve --model imdb_cnn_model/ --port 9292 --gpu_ids 0 #gpu预测服务
-```
+`` `shell
+python -m paddle_serving_server.serve --model imdb_cnn_model / --port 9292 #cpu prediction service
+python -m paddle_serving_server_gpu.serve --model imdb_cnn_model / --port 9292 --gpu_ids 0 #gpu prediction service
+`` `
-命令中参数--model 指定在之前保存的server端的模型和配置文件目录，--port指定预测服务的端口，当使用gpu版本部署gpu预测服务时可以使用--gpu_ids指定使用的gpu 。
+The parameter --model in the command specifies the server-side model and configuration file directory previously saved, --port specifies the port of the prediction service. When deploying the gpu prediction service using the gpu version, you can use --gpu_ids to specify the gpu used.
-执行完以上命令之一，就完成了IMDB 情感分析任务的RPC预测服务部署。
+After executing one of the above commands, the RPC prediction service deployment of the IMDB sentiment analysis task is completed.
-## Step6:复用Reader，定义远程RPC客户端
-下面我们通过Python代码来访问RPC预测服务，脚本为test_client.py
+## Step6: Reuse Reader, define remote RPC client
+Below we access the RPC prediction service through Python code, the script is test_client.py
local_train.py
@@ -172,7 +172,7 @@ logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger("fluid")
logger.setLevel(logging.INFO)
-# 加载词典文件
+# load dict file
def load_vocab(filename):
vocab = {}
with open(filename) as f:
@@ -190,11 +190,11 @@ if __name__ == "__main__":
vocab = load_vocab('imdb.vocab')
dict_dim = len(vocab)
- #定义模型输入
+ #define model input
data = fluid.layers.data(
name="words", shape=[1], dtype="int64", lod_level=1)
label = fluid.layers.data(name="label", shape=[1], dtype="int64")
- #定义dataset，train_data为训练数据目录
+ #define dataset，train_data is the dataset directory
dataset = fluid.DatasetFactory().create_dataset()
filelist = ["train_data/%s" % x for x in os.listdir("train_data")]
dataset.set_use_var([data, label])
@@ -203,11 +203,11 @@ if __name__ == "__main__":
dataset.set_batch_size(4)
dataset.set_filelist(filelist)
dataset.set_thread(10)
- #定义模型
+ #define model
avg_cost, acc, prediction = cnn_net(data, label, dict_dim)
optimizer = fluid.optimizer.SGD(learning_rate=0.001)
optimizer.minimize(avg_cost)
- #执行训练
+ #execute training
exe = fluid.Executor(fluid.CPUPlace())
exe.run(fluid.default_startup_program())
epochs = 100
@@ -219,7 +219,7 @@ if __name__ == "__main__":
program=fluid.default_main_program(), dataset=dataset, debug=False)
logger.info("TRAIN --> pass: {}".format(i))
if i == 64:
- #在训练结束时使用PaddleServing中的模型保存接口保存出Serving所需的模型和配置文件
+ #At the end of training, use the model save interface in PaddleServing to save the models and configuration files required by Serving
serving_io.save_model("{}_model".format(model_name),
"{}_client_conf".format(model_name),
{"words": data}, {"prediction": prediction},
@@ -228,32 +228,32 @@ if __name__ == "__main__":
-脚本从标准输入接收数据，并打印出样本预测为1的概率与真实的label。
+The script receives data from standard input and prints out the probability that the sample whose infer result is 1 and its real label.
-## Step7：调用RPC服务，测试模型效果
+## Step7: Call the RPC service to test the model effect
-以上一步实现的客户端为例运行预测服务，使用方式如下：
-
-```shell
-cat test_data/part-0 | python test_client.py imdb_lstm_client_conf/serving_client_conf.prototxt imdb.vocab
-```
+The client implemented in the previous step runs the prediction service as an example. The usage method is as follows:
-使用test_data/part-0文件中的2084个样本进行测试测试，模型预测的准确率为88.19%。
+`` `shell
+cat test_data/part-0 | python test_client.py imdb_lstm_client_conf / serving_client_conf.prototxt imdb.vocab
+`` `
-**注意**：每次模型训练的效果可能略有不同，使用训练出的模型预测的准确率会与示例中接近但有可能不完全一致。
+Using 2084 samples in the test_data/part-0 file for test testing, the model prediction accuracy is 88.19%.
-## Step8：部署HTTP预测服务
+** Note **: The effect of each model training may be slightly different, and the accuracy of predictions using the trained model will be close to the examples but may not be exactly the same.
-使用HTTP预测服务时，client端不需要安装Paddle Serving的任何模块，仅需要能发送HTTP请求即可。当然HTTP的通信方式会相较于RPC的通信方式在通信阶段消耗更多的时间。
+## Step8: Deploy HTTP Prediction Service
-对于IMDB情感分析任务原始文本在预测之前需要进行预处理，在RPC预测服务中我们将预处理放在client的脚本中，而在HTTP预测服务中我们将预处理放在server端。Paddle Serving的HTTP预测服务框架为这种情况准备了数据预处理和后处理的接口，我们只要根据任务需要重写即可。
+When using the HTTP prediction service, the client does not need to install any modules of Paddle Serving, it only needs to be able to send HTTP requests. Of course, the HTTP method consumes more time in the communication phase than the RPC method.
-Serving提供了示例代码，通过执行[IMDB示例](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb)中的imdb_web_service_demo.sh脚本来获取。
+For the IMDB sentiment analysis task, the original text needs to be preprocessed before prediction. In the RPC prediction service, we put the preprocessing in the client's script, and in the HTTP prediction service, we put the preprocessing on the server. Paddle Serving's HTTP prediction service framework prepares data pre-processing and post-processing interfaces for this situation. We just need to rewrite it according to the needs of the task.
-下面我们来看一下启动HTTP预测服务的脚本text_classify_service.py。
+Serving provides sample code, which is obtained by executing the imdb_web_service_demo.sh script in [IMDB Example](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/imdb).
+Let's take a look at the script text_classify_service.py that starts the HTTP prediction service.
test_client.py
@@ -267,7 +267,7 @@ client = Client()
client.load_client_config(sys.argv[1])
client.connect(["127.0.0.1:9292"])
-#在这里复用了数据预处理部分的代码将原始文本转换成数字id
+#The code of the data preprocessing part is reused here to convert the original text into a numeric id
imdb_dataset = IMDBDataset()
imdb_dataset.load_resource(sys.argv[2])
@@ -281,30 +281,29 @@ for line in sys.stdin:
-启动命令
+run
```shell
python text_classify_service.py imdb_cnn_model/ workdir/ 9292 imdb.vocab
```
-以上命令中参数1为保存的server端模型和配置文件，参数2为工作目录会保存一些预测服务工作时的配置文件，该目录可以不存在但需要指定名称，预测服务会自行创建，参数3为端口号，参数4为词典文件。
+In the above command, the first parameter is the saved server-side model and configuration file. The second parameter is the working directory, which will save some configuration files for the prediction service. The directory may not exist but needs to be specified. The prediction service will be created by itself. the third parameter is Port number, the fourth parameter is the dictionary file.
-## Step9：明文数据调用预测服务
-启动完HTTP预测服务，即可通过一行命令进行预测：
+## Step9: Call the prediction service with plaintext data
+After starting the HTTP prediction service, you can make prediction with a single command:
-```
-curl -H "Content-Type:application/json" -X POST -d '{"words": "i am very sad | 0", "fetch":["prediction"]}' http://127.0.0.1:9292/imdb/prediction
-```
-预测流程正常时，会返回预测概率，示例如下。
+`` `
+curl -H "Content-Type: application / json" -X POST -d '{"words": "i am very sad | 0", "fetch": ["prediction"]}' http://127.0.0.1:9292/imdb/prediction
+`` `
+When the inference process is normal, the prediction probability is returned, as shown below.
-```
-{"prediction":[0.5592559576034546,0.44074398279190063]}
-```
+`` `
+{"prediction": [0.5592559576034546,0.44074398279190063]}
+`` `
-**注意**：每次模型训练的效果可能略有不同，使用训练出的模型预测概率数值可能与示例不一致。
+** Note **: The effect of each model training may be slightly different, and the inferred probability value using the trained model may not be consistent with the example.
diff --git a/python/examples/bert/benchmark_batch.py b/python/examples/bert/benchmark_batch.py
index 872799e64ea599554e42264c37ab5f574c0acb13..9b8e301a62eb0eee161cd701555543d329c6ae83 100644
--- a/python/examples/bert/benchmark_batch.py
+++ b/python/examples/bert/benchmark_batch.py
@@ -57,8 +57,7 @@ def single_func(idx, resource):
os.getpid(),
int(round(b_start * 1000000)),
int(round(b_end * 1000000))))
- result = client.batch_predict(
- feed_batch=feed_batch, fetch=fetch)
+ result = client.predict(feed=feed_batch, fetch=fetch)
else:
print("unsupport batch size {}".format(args.batch_size))
diff --git a/python/examples/criteo_ctr_with_cube/test_server_gpu.py b/python/examples/criteo_ctr_with_cube/test_server_gpu.py
new file mode 100755
index 0000000000000000000000000000000000000000..382be99bd37a52630d78bb84ef7e53047b018c95
--- /dev/null
+++ b/python/examples/criteo_ctr_with_cube/test_server_gpu.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# pylint: disable=doc-string-missing
+
+import os
+import sys
+from paddle_serving_server_gpu import OpMaker
+from paddle_serving_server_gpu import OpSeqMaker
+from paddle_serving_server_gpu import Server
+
+op_maker = OpMaker()
+read_op = op_maker.create('general_reader')
+general_dist_kv_infer_op = op_maker.create('general_dist_kv_infer')
+response_op = op_maker.create('general_response')
+
+op_seq_maker = OpSeqMaker()
+op_seq_maker.add_op(read_op)
+op_seq_maker.add_op(general_dist_kv_infer_op)
+op_seq_maker.add_op(response_op)
+
+server = Server()
+server.set_op_sequence(op_seq_maker.get_op_sequence())
+server.set_num_threads(4)
+server.load_model_config(sys.argv[1])
+server.prepare_server(workdir="work_dir1", port=9292, device="cpu")
+server.run_server()
diff --git a/python/paddle_serving_server_gpu/__init__.py b/python/paddle_serving_server_gpu/__init__.py
index 02b55801c35fb5d1ed7e35c249ac07e4d3eb45ab..2fd35c6d66e4bf282224a8775f1a6bf0d1c6a8c5 100644
--- a/python/paddle_serving_server_gpu/__init__.py
+++ b/python/paddle_serving_server_gpu/__init__.py
@@ -55,6 +55,7 @@ class OpMaker(object):
"general_text_reader": "GeneralTextReaderOp",
"general_text_response": "GeneralTextResponseOp",
"general_single_kv": "GeneralSingleKVOp",
+ "general_dist_kv_infer": "GeneralDistKVInferOp",
"general_dist_kv": "GeneralDistKVOp"
}
@@ -104,6 +105,7 @@ class Server(object):
self.infer_service_fn = "infer_service.prototxt"
self.model_toolkit_fn = "model_toolkit.prototxt"
self.general_model_config_fn = "general_model.prototxt"
+ self.cube_config_fn = "cube.conf"
self.workdir = ""
self.max_concurrency = 0
self.num_threads = 4
@@ -184,6 +186,11 @@ class Server(object):
"w") as fout:
fout.write(str(self.model_conf))
self.resource_conf = server_sdk.ResourceConf()
+ for workflow in self.workflow_conf.workflows:
+ for node in workflow.nodes:
+ if "dist_kv" in node.name:
+ self.resource_conf.cube_config_path = workdir
+ self.resource_conf.cube_config_file = self.cube_config_fn
self.resource_conf.model_toolkit_path = workdir
self.resource_conf.model_toolkit_file = self.model_toolkit_fn
self.resource_conf.general_model_path = workdir
diff --git a/tools/serving_build.sh b/tools/serving_build.sh
index 046218f6de004f76c71be05e428fee2d15098239..381a366c15ec826debb8a801221ed58a2925bc53 100644
--- a/tools/serving_build.sh
+++ b/tools/serving_build.sh
@@ -48,6 +48,30 @@ function rerun() {
exit 1
}
+function build_app() {
+ local TYPE=$1
+ local DIRNAME=build-app-$TYPE
+ mkdir $DIRNAME # pwd: /Serving
+ cd $DIRNAME # pwd: /Serving/build-app-$TYPE
+ pip install numpy sentencepiece
+ case $TYPE in
+ CPU|GPU)
+ cmake -DPYTHON_INCLUDE_DIR=$PYTHONROOT/include/python2.7/ \
+ -DPYTHON_LIBRARIES=$PYTHONROOT/lib/libpython2.7.so \
+ -DPYTHON_EXECUTABLE=$PYTHONROOT/bin/python \
+ -DAPP=ON ..
+ rerun "make -j2 >/dev/null" 3 # due to some network reasons, compilation may fail
+ pip install -U python/dist/paddle_serving_app* >/dev/null
+ ;;
+ *)
+ echo "error type"
+ exit 1
+ ;;
+ esac
+ echo "build app $TYPE part finished as expected."
+ cd .. # pwd: /Serving
+}
+
function build_client() {
local TYPE=$1
local DIRNAME=build-client-$TYPE
@@ -145,7 +169,7 @@ function python_test_fit_a_line() {
sleep 5 # wait for the server to start
check_cmd "python test_client.py uci_housing_client/serving_client_conf.prototxt > /dev/null"
kill_server_process
-
+
# test web
unsetproxy # maybe the proxy is used on iPipe, which makes web-test failed.
check_cmd "python -m paddle_serving_server_gpu.serve --model uci_housing_model --port 9393 --thread 2 --gpu_ids 0 --name uci > /dev/null &"
@@ -184,14 +208,14 @@ function python_run_criteo_ctr_with_cube() {
check_cmd "mv models/ctr_serving_model_kv ./"
check_cmd "mv models/data ./cube/"
check_cmd "mv models/ut_data ./"
- cp ../../../build-server-$TYPE/output/bin/cube* ./cube/
+ cp ../../../build-server-$TYPE/output/bin/cube* ./cube/
mkdir -p $PYTHONROOT/lib/python2.7/site-packages/paddle_serving_server/serving-cpu-avx-openblas-0.1.3/
yes | cp ../../../build-server-$TYPE/output/demo/serving/bin/serving $PYTHONROOT/lib/python2.7/site-packages/paddle_serving_server/serving-cpu-avx-openblas-0.1.3/
sh cube_prepare.sh &
- check_cmd "mkdir work_dir1 && cp cube/conf/cube.conf ./work_dir1/"
+ check_cmd "mkdir work_dir1 && cp cube/conf/cube.conf ./work_dir1/"
python test_server.py ctr_serving_model_kv &
check_cmd "python test_client.py ctr_client_conf/serving_client_conf.prototxt ./ut_data >score"
- tail -n 2 score
+ tail -n 2 score | awk 'NR==1'
AUC=$(tail -n 2 score | awk 'NR==1')
VAR2="0.67" #TODO: temporarily relax the threshold to 0.67
RES=$( echo "$AUC>$VAR2" | bc )
@@ -219,7 +243,7 @@ function python_run_criteo_ctr_with_cube() {
check_cmd "python test_client.py ctr_client_conf/serving_client_conf.prototxt ./ut_data >score"
tail -n 2 score | awk 'NR==1'
AUC=$(tail -n 2 score | awk 'NR==1')
- VAR2="0.67"
+ VAR2="0.67" #TODO: temporarily relax the threshold to 0.67
RES=$( echo "$AUC>$VAR2" | bc )
if [[ $RES -eq 0 ]]; then
echo "error with criteo_ctr_with_cube inference auc test, auc should > 0.70"
@@ -253,6 +277,7 @@ function main() {
init # pwd: /Serving
build_client $TYPE # pwd: /Serving
build_server $TYPE # pwd: /Serving
+ build_app $TYPE # pwd: /Serving
python_run_test $TYPE # pwd: /Serving
echo "serving $TYPE part finished as expected."
}
text_clssify_service.py
@@ -313,7 +312,7 @@ from paddle_serving_server.web_service import WebService
from imdb_reader import IMDBDataset
import sys
-#继承框架中的WebService类
+#extend class WebService
class IMDBService(WebService):
def prepare_dict(self, args={}):
if len(args) == 0:
@@ -321,7 +320,7 @@ class IMDBService(WebService):
self.dataset = IMDBDataset()
self.dataset.load_resource(args["dict_file_path"])
- #重写preprocess方法来实现数据预处理，这里也复用了训练时使用的reader脚本
+ #rewrite preprocess() to implement data preprocessing, here we reuse reader script for training
def preprocess(self, feed={}, fetch=[]):
if "words" not in feed:
exit(-1)
@@ -329,7 +328,7 @@ class IMDBService(WebService):
res_feed["words"] = self.dataset.get_words_only(feed["words"])[0]
return res_feed, fetch
-#这里需要使用name参数指定预测服务的名称，
+#Here you need to use the name parameter to specify the name of the prediction service.
imdb_service = IMDBService(name="imdb")
imdb_service.load_model_config(sys.argv[1])
imdb_service.prepare_server(
@@ -339,24 +338,24 @@ imdb_service.run_server()
```