-### 3.6 训练过程中使用的Client io
+### 3.6 Client io used during Training
-PaddleServing设计可以在训练过程中使用的保存模型接口，与Paddle保存inference model的接口基本一致，feed_var_dict与fetch_var_dict
-可以为输入和输出变量起别名，serving启动需要读取的配置会保存在client端和server端的保存目录中。
+PaddleServing is designed to saves the model interface that can be used during the training process, which is basically the same as the Paddle save inference model interface, feed_var_dict and fetch_var_dict
+You can alias the input and output variables. The configuration that needs to be read when the serving starts is saved in the client and server storage directories.
``` python
def save_model(server_model_folder,
@@ -62,29 +64,29 @@ def save_model(server_model_folder,
main_program=None)
```
-## 4. Paddle Serving底层框架
+## 4. Paddle Serving Underlying Framework
-
+
-**模型管理框架**：对接多种机器学习平台的模型文件，向上提供统一的inference接口
-**业务调度框架**：对各种不同预测模型的计算逻辑进行抽象，提供通用的DAG调度框架，通过DAG图串联不同的算子，共同完成一次预测服务。该抽象模型使用户可以方便的实现自己的计算逻辑，同时便于算子共用。（用户搭建自己的预测服务，很大一部分工作是搭建DAG和提供算子的实现）
-**PredictService**:对外部提供的预测服务接口封装。通过protobuf定义与客户端的通信字段。
+**Model Management Framework**: Connects model files of multiple machine learning platforms and provides a unified inference interface
+**Business Scheduling Framework**: Abstracts the calculation logic of various different prediction models, provides a general DAG scheduling framework, and connects different operators through DAG diagrams to complete a prediction service together. This abstract model allows users to conveniently implement their own calculation logic, and at the same time facilitates operator sharing. (Users build their own forecasting services. A large part of their work is to build DAGs and provide operators.)
+**PredictService**: Encapsulation of the externally provided prediction service interface. Define communication fields with the client through protobuf.
-### 4.1 模型管理框架
+### 4.1 Model Management Framework
-模型管理框架负责管理机器学习框架训练出来的模型，总体可抽象成模型加载、模型数据和模型推理等3个层次。
+The model management framework is responsible for managing the models trained by the machine learning framework. It can be abstracted into three levels: model loading, model data, and model reasoning.
-#### 模型加载
+#### Model Loading
-将模型从磁盘加载到内存，支持多版本、热加载、增量更新等功能
+Load model from disk to memory, support multi-version, hot-load, incremental update, etc.
-#### 模型数据
+#### Model data
-模型在内存中的数据结构，集成fluid预测lib
+Model data structure in memory, integrated fluid inference lib
#### inferencer
-向上为预测服务提供统一的预测接口
+it provided united inference interface for upper layers
```C++
class FluidFamilyCore {
@@ -94,54 +96,54 @@ class FluidFamilyCore {
};
```
-### 4.2 业务调度框架
+### 4.2 Business Scheduling Framework
-#### 4.2.1 预测服务Service
+#### 4.2.1 Inference Service
-参考TF框架的模型计算的抽象思想，将业务逻辑抽象成DAG图，由配置驱动，生成workflow，跳过C++代码编译。业务的每个具体步骤，对应一个具体的OP，OP可配置自己依赖的上游OP。OP之间消息传递统一由线程级Bus和channel机制实现。例如，一个简单的预测服务的服务过程，可以抽象成读请求数据->调用预测接口->写回预测结果等3个步骤，相应的实现到3个OP: ReaderOp->ClassifyOp->WriteOp
+With reference to the abstract idea of model calculation of the TensorFlow framework, the business logic is abstracted into a DAG diagram, driven by configuration, generating a workflow, and skipping C ++ code compilation. Each specific step of the service corresponds to a specific OP. The OP can configure the upstream OP that it depends on. Unified message passing between OPs is achieved by the thread-level bus and channel mechanisms. For example, the service process of a simple prediction service can be abstracted into 3 steps including reading request data-> calling the prediction interface-> writing back the prediction result, and correspondingly implemented to 3 OP: ReaderOp-> ClassifyOp-> WriteOp
-
+
-关于OP之间的依赖关系，以及通过OP组建workflow，可以参考[从零开始写一个预测服务](CREATING.md)的相关章节
+Regarding the dependencies between OPs, and the establishment of workflows through OPs, you can refer to [从零开始写一个预测服务](./deprecated/CREATING.md) (simplified Chinese Version)
-服务端实例透视图
+Server instance perspective
-
+
-#### 4.2.2 Paddle Serving的多服务机制
+#### 4.2.2 Paddle Serving Multi-Service Mechanism
-
+
-Paddle Serving实例可以同时加载多个模型，每个模型用一个Service（以及其所配置的workflow）承接服务。可以参考[Demo例子中的service配置文件](../demo-serving/conf/service.prototxt)了解如何为serving实例配置多个service
+Paddle Serving instances can load multiple models at the same time, and each model uses a Service (and its configured workflow) to undertake services. You can refer to [service configuration file in Demo example](../tools/cpp_examples/demo-serving/conf/service.prototxt) to learn how to configure multiple services for the serving instance
-#### 4.2.3 业务调度层级关系
+#### 4.2.3 Hierarchical relationship of business scheduling
-从客户端看，一个Paddle Serving service从顶向下可分为Service, Endpoint, Variant等3个层级
+From the client's perspective, a Paddle Serving service can be divided into three levels: Service, Endpoint, and Variant from top to bottom.
-
+
-一个Service对应一个预测模型，模型下有1个endpoint。模型的不同版本，通过endpoint下多个variant概念实现：
-同一个模型预测服务，可以配置多个variant，每个variant有自己的下游IP列表。客户端代码可以对各个variant配置相对权重，以达到调节流量比例的关系（参考[客户端配置](CLIENT_CONFIGURE.md)第3.2节中关于variant_weight_list的说明）。
+One Service corresponds to one inference model, and there is one endpoint under the model. Different versions of the model are implemented through multiple variant concepts under endpoint:
+The same model prediction service can configure multiple variants, and each variant has its own downstream IP list. The client code can configure relative weights for each variant to achieve the relationship of adjusting the traffic ratio (refer to the description of variant_weight_list in [Client Configuration](./deprecated/CLIENT_CONFIGURE.md) section 3.2).
-
+
-## 5. 用户接口
+## 5. User Interface
-在满足一定的接口规范前提下，服务框架不对用户数据字段做任何约束，以应对各种预测服务的不同业务接口。Baidu-rpc继承了Protobuf serice的接口，用户按照Protobuf语法规范描述Request和Response业务接口。Paddle Serving基于Baidu-rpc框架搭建，默认支持该特性。
+Under the premise of meeting certain interface specifications, the service framework does not make any restrictions on user data fields to meet different business interfaces of various forecast services. Baidu-rpc inherits the interface of Protobuf serice, and the user describes the Request and Response business interfaces according to the Protobuf syntax specification. Paddle Serving is built on the Baidu-rpc framework and supports this feature by default.
-无论通信协议如何变化，框架只需确保Client和Server间通信协议和业务数据两种信息的格式同步，即可保证正常通信。这些信息又可细分如下：
+No matter how the communication protocol changes, the framework only needs to ensure that the communication protocol between the client and server and the format of the business data are synchronized to ensure normal communication. This information can be broken down as follows:
-- 协议：Server和Client之间事先约定的、确保相互识别数据格式的包头信息。Paddle Serving用Protobuf作为基础通信格式
-- 数据：用来描述Request和Response的接口，例如待预测样本数据，和预测返回的打分。包括：
- - 数据字段：请求包Request和返回包Response两种数据结构包含的字段定义
- - 描述接口：跟协议接口类似，默认支持Protobuf
+-Protocol: Header information agreed in advance between Server and Client to ensure mutual recognition of data format. Paddle Serving uses Protobuf as the basic communication format
+-Data: Used to describe the interface of Request and Response, such as the sample data to be predicted, and the score returned by the prediction. include:
+ -Data fields: Field definitions included in the two data structures of Request and Return.
+ -Description interface: similar to the protocol interface, it supports Protobuf by default
-### 5.1 数据压缩方法
+### 5.1 Data Compression Method
-Baidu-rpc内置了snappy, gzip, zlib等数据压缩方法，可在配置文件中配置（参考[客户端配置](CLIENT_CONFIGURE.md)第3.1节关于compress_type的介绍）
+Baidu-rpc has built-in data compression methods such as snappy, gzip, zlib, which can be configured in the configuration file (refer to [Client Configuration](./deprecated/CLIENT_CONFIGURE.md) Section 3.1 for an introduction to compress_type)
-### 5.2 C++ SDK API接口
+### 5.2 C ++ SDK API Interface
```C++
class PredictorApi {
@@ -176,7 +178,7 @@ class Predictor {
```
-### 5.3 OP相关接口
+### 5.3 Inferfaces related to Op
```C++
class Op {
@@ -258,7 +260,7 @@ class Op {
```
-### 5.4 框架相关接口
+### 5.4 Interfaces related to framework
Service
diff --git a/doc/DESIGN_CN.md b/doc/DESIGN_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..2e10013fc46c4b121ffe5c9268e5b531fe7f9992
--- /dev/null
+++ b/doc/DESIGN_CN.md
@@ -0,0 +1,377 @@
+# Paddle Serving设计方案
+
+(简体中文|[English](./DESIGN.md))
+
+## 1. 项目背景
+
+PaddlePaddle是公司开源的机器学习框架，广泛支持各种深度学习模型的定制化开发; Paddle serving是Paddle的在线预测部分，与Paddle模型训练环节无缝衔接，提供机器学习预测云服务。本文将从模型、服务、接入等层面，自底向上描述Paddle Serving设计方案。
+
+1. 模型是Paddle Serving预测的核心，包括模型数据和推理计算的管理；
+2. 预测框架封装模型推理计算，对外提供RPC接口，对接不同上游；
+3. 预测服务SDK提供一套接入框架
+
+最终形成一套完整的serving解决方案。
+
+## 2. 名词解释
+
+- baidu-rpc 百度官方开源RPC框架，支持多种常见通信协议，提供基于protobuf的自定义接口体验
+- Variant Paddle Serving架构对一个最小预测集群的抽象，其特点是内部所有实例（副本）完全同质，逻辑上对应一个model的一个固定版本
+- Endpoint 多个Variant组成一个Endpoint，逻辑上看，Endpoint代表一个model，Endpoint内部的Variant代表不同的版本
+- OP PaddlePaddle用来封装一种数值计算的算子，Paddle Serving用来表示一种基础的业务操作算子，核心接口是inference。OP通过配置其依赖的上游OP，将多个OP串联成一个workflow
+- Channel 一个OP所有请求级中间数据的抽象；OP之间通过Channel进行数据交互
+- Bus 对一个线程中所有channel的管理，以及根据DAG之间的DAG依赖图对OP和Channel两个集合间的访问关系进行调度
+- Stage Workflow按照DAG描述的拓扑图中，属于同一个环节且可并行执行的OP集合
+- Node 由某个Op算子类结合参数配置组成的Op算子实例，也是Workflow中的一个执行单元
+- Workflow 按照DAG描述的拓扑，有序执行每个OP的inference接口
+- DAG/Workflow 由若干个相互依赖的Node组成，每个Node均可通过特定接口获得Request对象，节点Op通过依赖关系获得其前置Op的输出对象，最后一个Node的输出默认就是Response对象
+- Service 对一次pv的请求封装，可配置若干条Workflow，彼此之间复用当前PV的Request对象，然后各自并行/串行执行，最后将Response写入对应的输出slot中；一个Paddle-serving进程可配置多套Service接口，上游根据ServiceName决定当前访问的Service接口。
+
+## 3. Python Interface设计
+
+### 3.1 核心目标：
+
+一套Paddle Serving的动态库，支持Paddle保存的通用模型的远程预估服务，通过Python Interface调用PaddleServing底层的各种功能。
+
+### 3.2 通用模型：
+
+能够使用Paddle Inference Library进行预测的模型，在训练过程中保存的模型，包含Feed Variable和Fetch Variable
+
+### 3.3 整体设计：
+
+用户通过Python Client启动Client和Server，Python API有检查互联和待访问模型是否匹配的功能
+Python API背后调用的是Paddle Serving实现的client和server对应功能的pybind，互传的信息通过RPC实现
+Client Python API当前有两个简单的功能，load_inference_conf和predict，分别用来执行加载待预测的模型和预测
+Server Python API主要负责加载预估模型，以及生成Paddle Serving需要的各种配置，包括engines，workflow，resource等
+
+### 3.4 Server Inferface
+
+
+
+### 3.5 Client Interface
+
+
-### 2.4 微服务插件模式 -由于Paddle Serving底层采用基于C++的通信组件，并且核心框架也是基于C/C++编写，当用户想要在服务端定义复杂的前处理与后处理逻辑时，一种办法是修改Paddle Serving底层框架，重新编译源码。另一种方式可以通过在服务端嵌入轻量级的Web服务，通过在Web服务中实现更复杂的预处理逻辑，从而搭建一套逻辑完整的服务。当访问量超过了Web服务能够接受的范围，开发者有足够的理由开发一些高性能的C++预处理逻辑，并嵌入到Serving的原生服务库中。Web服务和RPC服务的关系以及他们的组合方式可以参考下文`用户类型`中的说明。 +### 2.4 Micro service plugin +The underlying communication of Paddle Serving is implemented with C++ as well as the core framework, it is hard for users who do not familiar with C++ to implement new Paddle Serving Server Operators. Another approach is to use the light-weighted Web Service in Paddle Serving Server that can be viewed as a plugin. A user can implement complex data preprocessing and postprocessing logics to build a complex AI service. If access of the AI service has a large volumn, it is worth to implement the service with high performance Paddle Serving Server operators. The relationship between Web Service and RPC Service can be referenced in `User Type`. -## 3. 工业级特性 +## 3. Industrial Features -### 3.1 分布式稀疏参数索引 +### 3.1 Distributed Sparse Parameter Indexing -分布式稀疏参数索引通常在广告推荐中出现，并与分布式训练配合形成完整的离线-在线一体化部署。下图解释了其中的流程，产品的在线服务接受用户请求后将请求发送给预估服务，同时系统会记录用户的请求以进行相应的训练日志处理和拼接。离线分布式训练系统会针对流式产出的训练日志进行模型增量训练，而增量产生的模型会配送至分布式稀疏参数索引服务，同时对应的稠密的模型参数也会配送至在线的预估服务。在线服务由两部分组成，一部分是针对用户的请求提取特征后，将需要进行模型的稀疏参数索引的特征发送请求给分布式稀疏参数索引服务，针对分布式稀疏参数索引服务返回的稀疏参数再进行后续深度学习模型的计算流程，从而完成预估。 +Distributed Sparse Parameter Indexing is commonly seen in advertising and recommendation scenarios, and is often used coupled with distributed training. The figure below explains a commonly seen architecture for online recommendation. When the recommendation service receives a request from a user, the system will automatically collects training log for the offline distributed online training. Mean while, the request is sent to Paddle Serving Server. For sparse features, distributed sparse parameter index service is called so that sparse parameters can be looked up. The dense input features together with the looked up sparse model parameters are fed into the Paddle Inference Node of the DAG in Paddle Serving Server. Then the score can be responsed through RPC to product service for item ranking.
- -为什么要使用Paddle Serving提供的分布式稀疏参数索引服务？1）在一些推荐场景中，模型的输入特征规模通常可以达到上千亿，单台机器无法支撑T级别模型在内存的保存，因此需要进行分布式存储。2）Paddle Serving提供的分布式稀疏参数索引服务，具有并发请求多个节点的能力，从而以较低的延时完成预估服务。 + +Why do we need to support distributed sparse parameter indexing in Paddle Serving? 1) In some recommendation scenarios, the number of features can be up to hundreds of billions that a single node can not hold the parameters within random access memory. 2) Paddle Serving supports distributed sparse parameter indexing that can couple with paddle inference. Users do not need to do extra work to have a low latency inference engine with hundreds of billions of parameters. -### 3.2 模型管理、在线A/B流量测试、模型热加载 +### 3.2 Model Management, online A/B test, Model Online Reloading -Paddle Serving的C++引擎支持模型管理、在线A/B流量测试、模型热加载等功能，当前在Python API还有没完全开放这部分功能的配置，敬请期待。 +Paddle Serving's C++ engine supports model management, online A/B test and model online reloading. Currently, python API is not released yet, please wait for the next release. -## 4. 用户类型 -Paddle Serving面向的用户提供RPC和HTTP两种访问协议。对于HTTP协议，我们更倾向于流量中小型的服务使用，并且对延时没有严格要求的AI服务开发者。对于RPC协议，我们面向流量较大，对延时要求更高的用户，此外RPC的客户端可能也处在一个大系统的服务中，这种情况下非常适合使用Paddle Serving提供的RPC服务。对于使用分布式稀疏参数索引服务而言，Paddle Serving的用户不需要关心底层的细节，其调用本质也是通过RPC服务再调用RPC服务。下图给出了当前设计的Paddle Serving可能会使用Serving服务的几种场景。 +## 4. User Types +Paddle Serving provides RPC and HTTP protocol for users. For HTTP service, we recommend users with median or small traffic services to use, and the latency is not a strict requirement. For RPC protocol, we recommend high traffic services and low latency required services to use. For users who use distributed sparse parameter indexing built-in service, it is not necessary to care about the underlying details of communication. The following figure gives out several scenarios that user may want to use Paddle Serving.
@@ -173,11 +178,11 @@ Paddle Serving面向的用户提供RPC和HTTP两种访问协议。对于HTTP协
-对于普通的模型而言（具体指通过Serving提供的IO保存的模型，并且没有对模型进行后处理），用户使用RPC服务不需要额外的开发即可实现服务启动，但需要开发一些Client端的代码来使用服务。对于Web服务的开发，需要用户现在Paddle Serving提供的Web Service框架中进行前后处理的开发，从而实现整个HTTP服务。 +For servable models saved from Paddle Serving IO API, users do not need to do extra coding work to startup a service, but may need some coding work on the client side. For development of Web Service plugin, a user needs to provide implementation of Web Service's preprocessing and postprocessing work if needed to get a HTTP service. -### 4.1 Web服务开发 +### 4.1 Web Service Development -Web服务有很多开源的框架，Paddle Serving当前集成了Flask框架，但这部分对用户不可见，在未来可能会提供性能更好的Web框架作为底层HTTP服务集成引擎。用户需要继承WebService，从而实现对rpc服务的输入输出进行加工的目的。 +Web Service has lots of open sourced framework. Currently Paddle Serving uses Flask as built-in service framework, and users are not aware of this. More efficient web service will be integrated in the furture if needed. ``` python from paddle_serving_server.web_service import WebService @@ -208,15 +213,15 @@ imdb_service.prepare_dict({"dict_file_path": sys.argv[4]}) imdb_service.run_server() ``` -`WebService`作为基类，提供将用户接受的HTTP请求转化为RPC输入的接口`preprocess`，同时提供对RPC请求返回的结果进行后处理的接口`postprocess`，继承`WebService`的子类，可以定义各种类型的成员函数。`WebService`的启动命令和普通RPC服务提供的启动API一致。 +`WebService` is a Base Class, providing inheritable interfaces such `preprocess` and `postprocess` for users to implement. In the inherited class of `WebService` class, users can define any functions they want and the startup function interface is the same as RPC service. -## 5. 未来计划 +## 5. Future Plan -### 5.1 有向无环图结构定义开放 -当前版本开放的python API仅支持用户定义Sequential类型的执行流，如果想要进行Server进程内复杂的计算，需要增加对应的用户API。 +### 5.1 Open DAG definition API +Current version of Paddle Serving Server supports sequential type of execution flow. DAG definition API can be more helpful to users on complex tasks. -### 5.2 云端自动部署能力 -为了方便用户更容易将Paddle的预测模型部署到线上，Paddle Serving在接下来的版本会提供Kubernetes生态下任务编排的工具。 +### 5.2 Auto Deployment on Cloud +In order to make deployment more easily on public cloud, Paddle Serving considers to provides Operators on Kubernetes in submitting a service job. -### 5.3 向量检索、树结构检索 -在推荐与广告场景的召回系统中，通常需要采用基于向量的快速检索或者基于树结构的快速检索，Paddle Serving会对这方面的检索引擎进行集成或扩展。 +### 5.3 Vector Indexing and Tree based Indexing +In recommendation and advertisement systems, it is commonly seen to use vector based index or tree based indexing service to do candidate retrievals. These retrieval tasks will be built-in services of Paddle Serving. diff --git a/doc/DESIGN_DOC_CN.md b/doc/DESIGN_DOC_CN.md new file mode 100644 index 0000000000000000000000000000000000000000..312379cd7543e70095e5a6d8168aab06b79a0525 --- /dev/null +++ b/doc/DESIGN_DOC_CN.md @@ -0,0 +1,222 @@ +# Paddle Serving设计文档 + +## 1. 整体设计目标 + +- 长期使命：Paddle Serving是一个PaddlePaddle开源的在线服务框架，长期目标就是围绕着人工智能落地的最后一公里提供越来越专业、可靠、易用的服务。 + +- 工业级：为了达到工业级深度学习模型在线部署的要求， +Paddle Serving提供很多大规模场景需要的部署功能：1）分布式稀疏参数索引功能；2）高并发底层通信能力；3）模型管理、在线A/B流量测试、模型热加载。 + +- 简单易用：为了让使用Paddle的用户能够以极低的成本部署模型，PaddleServing设计了一套与Paddle训练框架无缝打通的预测部署API，普通模型可以使用一行命令进行服务部署。 + +- 功能扩展：当前，Paddle Serving支持C++、Python、Golang的客户端，未来也会面向不同类型的客户新增多种语言的客户端。在Paddle Serving的框架设计方面，尽管当前Paddle Serving以支持Paddle模型的部署为核心功能， +用户可以很容易嵌入其他的机器学习库部署在线预测。 + +## 2. 模块设计与实现 + +### 2.1 Python API接口设计 + +#### 2.1.1 训练模型的保存 +Paddle的模型预测需要重点关注的内容：1）模型的输入变量；2）模型的输出变量；3）模型结构和模型参数。Paddle Serving Python API提供用户可以在训练过程中保存模型的接口，并将Paddle Serving在部署阶段需要保存的配置打包保存，一个示例如下： +``` python +import paddle_serving_client.io as serving_io +serving_io.save_model("serving_model", "client_conf", + {"words": data}, {"prediction": prediction}, + fluid.default_main_program()) +``` +代码示例中，`{"words": data}`和`{"prediction": prediction}`分别指定了模型的输入和输出，`"words"`和`"prediction"`是输出和输出变量的别名，设计别名的目的是为了使开发者能够记忆自己训练模型的输入输出对应的字段。`data`和`prediction`则是Paddle训练过程中的`[Variable](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Variable_cn.html#variable)`，通常代表张量([Tensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Tensor_cn.html#tensor))或变长张量([LodTensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/beginners_guide/basic_concept/lod_tensor.html#lodtensor))。调用保存命令后，会按照用户指定的`"serving_model"`和`"client_conf"`生成两个目录，内容如下： +``` shell +. +├── client_conf +│ ├── serving_client_conf.prototxt +│ └── serving_client_conf.stream.prototxt +└── serving_model + ├── embedding_0.w_0 + ├── fc_0.b_0 + ├── fc_0.w_0 + ├── fc_1.b_0 + ├── fc_1.w_0 + ├── fc_2.b_0 + ├── fc_2.w_0 + ├── lstm_0.b_0 + ├── lstm_0.w_0 + ├── __model__ + ├── serving_server_conf.prototxt + └── serving_server_conf.stream.prototxt +``` +其中，`"serving_client_conf.prototxt"`和`"serving_server_conf.prototxt"`是Paddle Serving的Client和Server端需要加载的配置，`"serving_client_conf.stream.prototxt"`和`"serving_server_conf.stream.prototxt"`是配置文件的二进制形式。`"serving_model"`下保存的其他内容和Paddle保存的模型文件是一致的。我们会考虑未来在Paddle框架中直接保存可服务的配置，实现配置保存对用户无感。 + +#### 2.1.2 服务端模型加载 + +服务端的预测逻辑可以通过Paddle Serving Server端的API进行人工定义，一个例子： +``` python +import paddle_serving_server as serving +op_maker = serving.OpMaker() +read_op = op_maker.create('general_reader') +dist_kv_op = op_maker.create('general_dist_kv') +general_infer_op = op_maker.create('general_infer') +general_response_op = op_maker.create('general_response') + +op_seq_maker = serving.OpSeqMaker() +op_seq_maker.add_op(read_op) +op_seq_maker.add_op(dist_kv_op) +op_seq_maker.add_op(general_infer_op) +op_seq_maker.add_op(general_response_op) +``` + +当前Paddle Serving在Server端支持的主要Op请参考如下列表： + +
+
+
+
+
+ +### 2.4 微服务插件模式 +由于Paddle Serving底层采用基于C++的通信组件，并且核心框架也是基于C/C++编写，当用户想要在服务端定义复杂的前处理与后处理逻辑时，一种办法是修改Paddle Serving底层框架，重新编译源码。另一种方式可以通过在服务端嵌入轻量级的Web服务，通过在Web服务中实现更复杂的预处理逻辑，从而搭建一套逻辑完整的服务。当访问量超过了Web服务能够接受的范围，开发者有足够的理由开发一些高性能的C++预处理逻辑，并嵌入到Serving的原生服务库中。Web服务和RPC服务的关系以及他们的组合方式可以参考下文`用户类型`中的说明。 + +## 3. 工业级特性 + +### 3.1 分布式稀疏参数索引 + +分布式稀疏参数索引通常在广告推荐中出现，并与分布式训练配合形成完整的离线-在线一体化部署。下图解释了其中的流程，产品的在线服务接受用户请求后将请求发送给预估服务，同时系统会记录用户的请求以进行相应的训练日志处理和拼接。离线分布式训练系统会针对流式产出的训练日志进行模型增量训练，而增量产生的模型会配送至分布式稀疏参数索引服务，同时对应的稠密的模型参数也会配送至在线的预估服务。在线服务由两部分组成，一部分是针对用户的请求提取特征后，将需要进行模型的稀疏参数索引的特征发送请求给分布式稀疏参数索引服务，针对分布式稀疏参数索引服务返回的稀疏参数再进行后续深度学习模型的计算流程，从而完成预估。 + +
+
+
+
+
+ +为什么要使用Paddle Serving提供的分布式稀疏参数索引服务？1）在一些推荐场景中，模型的输入特征规模通常可以达到上千亿，单台机器无法支撑T级别模型在内存的保存，因此需要进行分布式存储。2）Paddle Serving提供的分布式稀疏参数索引服务，具有并发请求多个节点的能力，从而以较低的延时完成预估服务。 + +### 3.2 模型管理、在线A/B流量测试、模型热加载 + +Paddle Serving的C++引擎支持模型管理、在线A/B流量测试、模型热加载等功能，当前在Python API还有没完全开放这部分功能的配置，敬请期待。 + +## 4. 用户类型 +Paddle Serving面向的用户提供RPC和HTTP两种访问协议。对于HTTP协议，我们更倾向于流量中小型的服务使用，并且对延时没有严格要求的AI服务开发者。对于RPC协议，我们面向流量较大，对延时要求更高的用户，此外RPC的客户端可能也处在一个大系统的服务中，这种情况下非常适合使用Paddle Serving提供的RPC服务。对于使用分布式稀疏参数索引服务而言，Paddle Serving的用户不需要关心底层的细节，其调用本质也是通过RPC服务再调用RPC服务。下图给出了当前设计的Paddle Serving可能会使用Serving服务的几种场景。 + +
+
+
+
+
+ +对于普通的模型而言（具体指通过Serving提供的IO保存的模型，并且没有对模型进行后处理），用户使用RPC服务不需要额外的开发即可实现服务启动，但需要开发一些Client端的代码来使用服务。对于Web服务的开发，需要用户现在Paddle Serving提供的Web Service框架中进行前后处理的开发，从而实现整个HTTP服务。 + +### 4.1 Web服务开发 + +Web服务有很多开源的框架，Paddle Serving当前集成了Flask框架，但这部分对用户不可见，在未来可能会提供性能更好的Web框架作为底层HTTP服务集成引擎。用户需要继承WebService，从而实现对rpc服务的输入输出进行加工的目的。 + +``` python +from paddle_serving_server.web_service import WebService +from imdb_reader import IMDBDataset +import sys + + +class IMDBService(WebService): + def prepare_dict(self, args={}): + if len(args) == 0: + exit(-1) + self.dataset = IMDBDataset() + self.dataset.load_resource(args["dict_file_path"]) + + def preprocess(self, feed={}, fetch=[]): + if "words" not in feed: + exit(-1) + res_feed = {} + res_feed["words"] = self.dataset.get_words_only(feed["words"])[0] + return res_feed, fetch + + +imdb_service = IMDBService(name="imdb") +imdb_service.load_model_config(sys.argv[1]) +imdb_service.prepare_server( + workdir=sys.argv[2], port=int(sys.argv[3]), device="cpu") +imdb_service.prepare_dict({"dict_file_path": sys.argv[4]}) +imdb_service.run_server() +``` + +`WebService`作为基类，提供将用户接受的HTTP请求转化为RPC输入的接口`preprocess`，同时提供对RPC请求返回的结果进行后处理的接口`postprocess`，继承`WebService`的子类，可以定义各种类型的成员函数。`WebService`的启动命令和普通RPC服务提供的启动API一致。 + +## 5. 未来计划 + +### 5.1 有向无环图结构定义开放 +当前版本开放的python API仅支持用户定义Sequential类型的执行流，如果想要进行Server进程内复杂的计算，需要增加对应的用户API。 + +### 5.2 云端自动部署能力 +为了方便用户更容易将Paddle的预测模型部署到线上，Paddle Serving在接下来的版本会提供Kubernetes生态下任务编排的工具。 + +### 5.3 向量检索、树结构检索 +在推荐与广告场景的召回系统中，通常需要采用基于向量的快速检索或者基于树结构的快速检索，Paddle Serving会对这方面的检索引擎进行集成或扩展。 diff --git a/doc/DESIGN_DOC_EN.md b/doc/DESIGN_DOC_EN.md deleted file mode 100644 index 2f8a36ea6686b5add2a7e4e407eabfd14167490d..0000000000000000000000000000000000000000 --- a/doc/DESIGN_DOC_EN.md +++ /dev/null @@ -1,227 +0,0 @@ -# Paddle Serving Design Doc - -## 1. Design Objectives - -- Long Term Vision: Online deployment of deep learning models will be a user-facing application in the future. Any AI developer will face the problem of deploying an online service for his or her trained model. -Paddle Serving is the official open source online deployment framework. The long term goal of Paddle Serving is to provide professional, reliable and easy-to-use online service to the last mile of AI application. - -- Easy-To-Use: For algorithmic developers to quickly deploy their models online, Paddle Serving designs APIs that can be used with Paddle's training process seamlessly, most Paddle models can be deployed as a service with one line command. - -- Industrial Oriented: To meet industrial deployment requirements, Paddle Serving supports lots of large-scale deployment functions: 1) Distributed Sparse Embedding Indexing. 2) Highly concurrent underlying communications. 3) Model Management, online A/B test, model online loading. - -- Extensibility: Paddle Serving supports C++, Python and Golang client, and will support more clients with different languages. It is very easy to extend Paddle Serving to support other machine learning inference library, although currently Paddle inference library is the only official supported inference backend. - - -## 2. Module design and implementation - -### 2.1 Python API interface design - -#### 2.1.1 save a servable model -The inference phase of Paddle model focuses on 1) input variables of the model. 2) output variables of the model. 3) model structure and model parameters. Paddle Serving Python API provides a `save_model` interface for trained model, and save necessary information for Paddle Serving to use during deployment phase. An example is as follows: - -``` python -import paddle_serving_client.io as serving_io -serving_io.save_model("serving_model", "client_conf", - {"words": data}, {"prediction": prediction}, - fluid.default_main_program()) -``` -In the example, `{"words": data}` and `{"prediction": prediction}` assign the inputs and outputs of a model. `"words"` and `"prediction"` are alias names of inputs and outputs. The design of alias name is to help developers to memorize model inputs and model outputs. `data` and `prediction` are Paddle `[Variable](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Variable_cn.html#variable)` in training phase that often represents ([Tensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Tensor_cn.html#tensor)) or ([LodTensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/beginners_guide/basic_concept/lod_tensor.html#lodtensor)). When the `save_model` API is called, two directories called `"serving_model"` and `"client_conf"` will be generated. The content of the saved model is as follows: - -``` shell -. -├── client_conf -│ ├── serving_client_conf.prototxt -│ └── serving_client_conf.stream.prototxt -└── serving_model - ├── embedding_0.w_0 - ├── fc_0.b_0 - ├── fc_0.w_0 - ├── fc_1.b_0 - ├── fc_1.w_0 - ├── fc_2.b_0 - ├── fc_2.w_0 - ├── lstm_0.b_0 - ├── lstm_0.w_0 - ├── __model__ - ├── serving_server_conf.prototxt - └── serving_server_conf.stream.prototxt -``` -`"serving_client_conf.prototxt"` and `"serving_server_conf.prototxt"` are the client side and the server side configurations of Paddle Serving, and `"serving_client_conf.stream.prototxt"` and `"serving_server_conf.stream.prototxt"` are the corresponding parts. Other contents saved in the directory are the same as Paddle saved inference model. We are considering to support `save_model` interface in Paddle training framework so that a user is not aware of the servable configurations. - -#### 2.1.2 Model loading on the server side - -Prediction logics on the server side can be defined through Paddle Serving Server API with a few lines of code, an example is as follows: -``` python -import paddle_serving_server as serving -op_maker = serving.OpMaker() -read_op = op_maker.create('general_reader') -dist_kv_op = op_maker.create('general_dist_kv') -general_infer_op = op_maker.create('general_infer') -general_response_op = op_maker.create('general_response') - -op_seq_maker = serving.OpSeqMaker() -op_seq_maker.add_op(read_op) -op_seq_maker.add_op(dist_kv_op) -op_seq_maker.add_op(general_infer_op) -op_seq_maker.add_op(general_response_op) -``` -Current Paddle Serving supports operator list on the server side as follows: - -
-
-
-
-
- -### 2.4 Micro service plugin -The underlying communication of Paddle Serving is implemented with C++ as well as the core framework, it is hard for users who do not familiar with C++ to implement new Paddle Serving Server Operators. Another approach is to use the light-weighted Web Service in Paddle Serving Server that can be viewed as a plugin. A user can implement complex data preprocessing and postprocessing logics to build a complex AI service. If access of the AI service has a large volumn, it is worth to implement the service with high performance Paddle Serving Server operators. The relationship between Web Service and RPC Service can be referenced in `User Type`. - -## 3. Industrial Features - -### 3.1 Distributed Sparse Parameter Indexing - -Distributed Sparse Parameter Indexing is commonly seen in advertising and recommendation scenarios, and is often used coupled with distributed training. The figure below explains a commonly seen architecture for online recommendation. When the recommendation service receives a request from a user, the system will automatically collects training log for the offline distributed online training. Mean while, the request is sent to Paddle Serving Server. For sparse features, distributed sparse parameter index service is called so that sparse parameters can be looked up. The dense input features together with the looked up sparse model parameters are fed into the Paddle Inference Node of the DAG in Paddle Serving Server. Then the score can be responsed through RPC to product service for item ranking. - -
-
-
-
-
- -Why do we need to support distributed sparse parameter indexing in Paddle Serving? 1) In some recommendation scenarios, the number of features can be up to hundreds of billions that a single node can not hold the parameters within random access memory. 2) Paddle Serving supports distributed sparse parameter indexing that can couple with paddle inference. Users do not need to do extra work to have a low latency inference engine with hundreds of billions of parameters. - -### 3.2 Model Management, online A/B test, Model Online Reloading - -Paddle Serving's C++ engine supports model management, online A/B test and model online reloading. Currently, python API is not released yet, please wait for the next release. - -## 4. User Types -Paddle Serving provides RPC and HTTP protocol for users. For HTTP service, we recommend users with median or small traffic services to use, and the latency is not a strict requirement. For RPC protocol, we recommend high traffic services and low latency required services to use. For users who use distributed sparse parameter indexing built-in service, it is not necessary to care about the underlying details of communication. The following figure gives out several scenarios that user may want to use Paddle Serving. - -
-
-
-
-
-
-For servable models saved from Paddle Serving IO API, users do not need to do extra coding work to startup a service, but may need some coding work on the client side. For development of Web Service plugin, a user needs to provide implementation of Web Service's preprocessing and postprocessing work if needed to get a HTTP service.
-
-### 4.1 Web Service Development
-
-Web Service has lots of open sourced framework. Currently Paddle Serving uses Flask as built-in service framework, and users are not aware of this. More efficient web service will be integrated in the furture if needed.
-
-``` python
-from paddle_serving_server.web_service import WebService
-from imdb_reader import IMDBDataset
-import sys
-
-
-class IMDBService(WebService):
- def prepare_dict(self, args={}):
- if len(args) == 0:
- exit(-1)
- self.dataset = IMDBDataset()
- self.dataset.load_resource(args["dict_file_path"])
-
- def preprocess(self, feed={}, fetch=[]):
- if "words" not in feed:
- exit(-1)
- res_feed = {}
- res_feed["words"] = self.dataset.get_words_only(feed["words"])[0]
- return res_feed, fetch
-
-
-imdb_service = IMDBService(name="imdb")
-imdb_service.load_model_config(sys.argv[1])
-imdb_service.prepare_server(
- workdir=sys.argv[2], port=int(sys.argv[3]), device="cpu")
-imdb_service.prepare_dict({"dict_file_path": sys.argv[4]})
-imdb_service.run_server()
-```
-
-`WebService` is a Base Class, providing inheritable interfaces such `preprocess` and `postprocess` for users to implement. In the inherited class of `WebService` class, users can define any functions they want and the startup function interface is the same as RPC service.
-
-## 5. Future Plan
-
-### 5.1 Open DAG definition API
-Current version of Paddle Serving Server supports sequential type of execution flow. DAG definition API can be more helpful to users on complex tasks.
-
-### 5.2 Auto Deployment on Cloud
-In order to make deployment more easily on public cloud, Paddle Serving considers to provides Operators on Kubernetes in submitting a service job.
-
-### 5.3 Vector Indexing and Tree based Indexing
-In recommendation and advertisement systems, it is commonly seen to use vector based index or tree based indexing service to do candidate retrievals. These retrieval tasks will be built-in services of Paddle Serving.
diff --git a/doc/DOCKER.md b/doc/DOCKER.md
index 325ec906c04c708d8e62ff2ae2900bc367e049b6..0e865c66e2da32a4e0ed15df9f2632b98ffbcedf 100644
--- a/doc/DOCKER.md
+++ b/doc/DOCKER.md
@@ -1,53 +1,55 @@
-# Docker编译环境准备
+# Docker compilation environment preparation
-## 环境要求
+([简体中文](./DOCKER_CN.md)|English)
-+ 开发机上已安装Docker。
-+ 编译GPU版本需要安装nvidia-docker。
+## Environmental requirements
-## Dockerfile文件
++ Docker is installed on the development machine.
++ Compiling the GPU version requires nvidia-docker.
-[CPU版本Dockerfile](../Dockerfile)
+## Dockerfile
-[GPU版本Dockerfile](../Dockerfile.gpu)
+[CPU Version Dockerfile](../tools/Dockerfile)
-## 使用方法
+[GPU Version Dockerfile](../tools/Dockerfile.gpu)
-### 构建Docker镜像
+## Instructions
-建立新目录，复制Dockerfile内容到该目录下Dockerfile文件。
+### Building Docker Image
-执行
+Create a new directory and copy the Dockerfile to this directory.
+
+Run
```bash
docker build -t serving_compile:cpu .
```
-或者
+Or
```bash
docker build -t serving_compile:cuda9 .
```
-## 进入Docker
+## Enter Docker Container
-CPU版本请执行
+CPU Version please run
```bash
docker run -it serving_compile:cpu bash
```
-GPU版本请执行
+GPU Version please run
```bash
docker run -it --runtime=nvidia -it serving_compile:cuda9 bash
```
-## Docker编译出的可执行文件支持的环境列表
+## List of supported environments compiled by Docker
-经过验证的环境列表如下：
+The list of supported environments is as follows:：
-| CPU Docker编译出的可执行文件支持的系统环境 |
+| System Environment Supported by CPU Docker Compiled Executables |
| -------------------------- |
| Centos6 |
| Centos7 |
@@ -56,7 +58,7 @@ docker run -it --runtime=nvidia -it serving_compile:cuda9 bash
-| GPU Docker编译出的可执行文件支持的系统环境 |
+| System Environment Supported by GPU Docker Compiled Executables |
| ---------------------------------- |
| Centos6_cuda9_cudnn7 |
| Centos7_cuda9_cudnn7 |
@@ -65,6 +67,6 @@ docker run -it --runtime=nvidia -it serving_compile:cuda9 bash
-**备注：**
-+ 若执行预编译版本出现找不到libcrypto.so.10、libssl.so.10的情况，可以将Docker环境中的/usr/lib64/libssl.so.10与/usr/lib64/libcrypto.so.10复制到可执行文件所在目录。
-+ CPU预编译版本仅可在CPU机器上执行，GPU预编译版本仅可在GPU机器上执行。
+**Remarks:**
++ If you cannot find libcrypto.so.10 and libssl.so.10 when you execute the pre-compiled version, you can change /usr/lib64/libssl.so.10 and /usr/lib64/libcrypto.so in the Docker environment. 10 Copy to the directory where the executable is located.
++ CPU pre-compiled version can only be executed on CPU machines, GPU pre-compiled version can only be executed on GPU machines.
diff --git a/doc/DOCKER_CN.md b/doc/DOCKER_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..92cc3ac6ea34d6399d6204ff7b9ec2d12b412601
--- /dev/null
+++ b/doc/DOCKER_CN.md
@@ -0,0 +1,72 @@
+# Docker编译环境准备
+
+(简体中文|[English](./DOCKER.md))
+
+## 环境要求
+
++ 开发机上已安装Docker。
++ 编译GPU版本需要安装nvidia-docker。
+
+## Dockerfile文件
+
+[CPU版本Dockerfile](../tools/Dockerfile)
+
+[GPU版本Dockerfile](../tools/Dockerfile.gpu)
+
+## 使用方法
+
+### 构建Docker镜像
+
+建立新目录，复制Dockerfile内容到该目录下Dockerfile文件。
+
+执行
+
+```bash
+docker build -t serving_compile:cpu .
+```
+
+或者
+
+```bash
+docker build -t serving_compile:cuda9 .
+```
+
+## 进入Docker
+
+CPU版本请执行
+
+```bash
+docker run -it serving_compile:cpu bash
+```
+
+GPU版本请执行
+
+```bash
+docker run -it --runtime=nvidia -it serving_compile:cuda9 bash
+```
+
+## Docker编译出的可执行文件支持的环境列表
+
+经过验证的环境列表如下：
+
+| CPU Docker编译出的可执行文件支持的系统环境 |
+| -------------------------- |
+| Centos6 |
+| Centos7 |
+| Ubuntu16.04 |
+| Ubuntu18.04 |
+
+
+
+| GPU Docker编译出的可执行文件支持的系统环境 |
+| ---------------------------------- |
+| Centos6_cuda9_cudnn7 |
+| Centos7_cuda9_cudnn7 |
+| Ubuntu16.04_cuda9_cudnn7 |
+| Ubuntu16.04_cuda10_cudnn7 |
+
+
+
+**备注：**
++ 若执行预编译版本出现找不到libcrypto.so.10、libssl.so.10的情况，可以将Docker环境中的/usr/lib64/libssl.so.10与/usr/lib64/libcrypto.so.10复制到可执行文件所在目录。
++ CPU预编译版本仅可在CPU机器上执行，GPU预编译版本仅可在GPU机器上执行。
diff --git a/doc/IMDB_GO_CLIENT.md b/doc/IMDB_GO_CLIENT.md
index 5b10192597f393d65f1387bfb39615e1777ec2d6..5befc0226235dd599b980d98594dba78e54bf530 100644
--- a/doc/IMDB_GO_CLIENT.md
+++ b/doc/IMDB_GO_CLIENT.md
@@ -1,5 +1,7 @@
# How to use Go Client of Paddle Serving
+([简体中文](./IMDB_GO_CLIENT_CN.md)|English)
+
This document shows how to use Go as your client language. For Go client in Paddle Serving, a simple client package is provided https://github.com/PaddlePaddle/Serving/tree/develop/go/serving_client, a user can import this package as needed. Here is a simple example of sentiment analysis task based on IMDB dataset.
### Install
@@ -15,7 +17,7 @@ pip install paddle-serving-server
### Download Text Classification Model
``` shell
-wget https://paddle-serving.bj.bcebos.com/data%2Ftext_classification%2Fimdb_serving_example.tar.gz
+wget https://paddle-serving.bj.bcebos.com/data/text_classification/imdb_serving_example.tar.gz
tar -xzf imdb_serving_example.tar.gz
```
diff --git a/doc/IMDB_GO_CLIENT_CN.md b/doc/IMDB_GO_CLIENT_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..d14abe647038846aeeeebf9484f1c02e151b4275
--- /dev/null
+++ b/doc/IMDB_GO_CLIENT_CN.md
@@ -0,0 +1,194 @@
+# 如何在Paddle Serving使用Go Client
+
+(简体中文|[English](./IMDB_GO_CLIENT.md))
+
+本文档说明了如何将Go用作客户端语言。对于Paddle Serving中的Go客户端，提供了一个简单的客户端程序包https://github.com/PaddlePaddle/Serving/tree/develop/go/serving_client, 用户可以根据需要引用该程序包。这是一个基于IMDB数据集的情感分析任务的简单示例。
+
+### 安装
+
+我们假设您已经安装了Go 1.9.2或更高版本，并且安装了python 2.7版本
+
+```shell
+go get github.com/PaddlePaddle/Serving/go/serving_client
+go get github.com/PaddlePaddle/Serving/go/proto
+pip install paddle-serving-server
+```
+### 下载文本分类模型
+
+```shell
+wget https://paddle-serving.bj.bcebos.com/data/text_classification/imdb_serving_example.tar.gz
+tar -xzf imdb_serving_example.tar.gz
+```
+
+### 服务器端代码
+
+```python
+# test_server_go.py
+import os
+import sys
+from paddle_serving_server import OpMaker
+from paddle_serving_server import OpSeqMaker
+from paddle_serving_server import Server
+
+op_maker = OpMaker ()
+read_op = op_maker.create ('general_text_reader')
+general_infer_op = op_maker.create ('general_infer')
+general_response_op = op_maker.create ('general_text_response')
+
+op_seq_maker = OpSeqMaker ()
+op_seq_maker.add_op (read_op)
+op_seq_maker.add_op (general_infer_op)
+op_seq_maker.add_op (general_response_op)
+
+server = Server ()
+server.set_op_sequence (op_seq_maker.get_op_sequence ())
+server.load_model_config (sys.argv [1])
+server.prepare_server (workdir = "work_dir1", port = 9292, device = "cpu")
+server.run_server ()
+```
+
+### 启动服务器
+
+```shell
+python test_server_go.py ./serving_server_model/ 9292
+```
+
+### 客户端代码示例
+
+```go
+// imdb_client.go
+package main
+
+import (
+ "io"
+ "fmt"
+ "strings"
+ "bufio"
+ "strconv"
+ "os"
+ serving_client "github.com/PaddlePaddle/Serving/go/serving_client"
+)
+
+func main () {
+ var config_file_path string
+ config_file_path = os.Args [1]
+ handle: = serving_client.LoadModelConfig (config_file_path)
+ handle = serving_client.Connect ("127.0.0.1", "9292", handle)
+
+ test_file_path: = os.Args [2]
+ fi, err: = os.Open (test_file_path)
+ if err! = nil {
+ fmt.Print (err)
+ }
+
+ defer fi.Close ()
+ br: = bufio.NewReader (fi)
+
+ fetch: = [] string {"cost", "acc", "prediction"}
+
+ var result map [string] [] float32
+
+ for {
+ line, err: = br.ReadString ('\ n')
+if err == io.EOF {
+break
+}
+
+line = strings.Trim (line, "\ n")
+
+var words = [] int64 {}
+
+s: = strings.Split (line, "")
+value, err: = strconv.Atoi (s [0])
+var feed_int_map map [string] [] int64
+
+for _, v: = range s [1: value + 1] {
+int_v, _: = strconv.Atoi (v)
+words = append (words, int64 (int_v))
+}
+
+label, err: = strconv.Atoi (s [len (s) -1])
+
+if err! = nil {
+panic (err)
+}
+
+feed_int_map = map [string] [] int64 {}
+feed_int_map ["words"] = words
+feed_int_map ["label"] = [] int64 {int64 (label)}
+Ranch
+result = serving_client.Predict (handle, feed_int_map, fetch)
+fmt.Println (result ["prediction"] [1], "\ t", int64 (label))
+ }
+}
+```
+
+### 基于IMDB测试集的预测
+
+```python
+go run imdb_client.go serving_client_conf / serving_client_conf.stream.prototxt test.data> result
+```
+
+### 计算精度
+
+```python
+// acc.go
+package main
+
+import (
+ "io"
+ "os"
+ "fmt"
+ "bufio"
+ "strings"
+ "strconv"
+)
+
+func main () {
+ score_file: = os.Args [1]
+ fi, err: = os.Open (score_file)
+ if err! = nil {
+ fmt.Print (err)
+ }
+
+ defer fi.Close ()
+ br: = bufio.NewReader (fi)
+
+ total: = int (0)
+ acc: = int (0)
+ for {
+ line, err: = br.ReadString ('\ n')
+ if err == io.EOF {
+ break
+ }
+
+ line = strings.Trim (line, "\ n")
+ s: = strings.Split (line, "\ t")
+ prob_str: = strings.Trim (s [0], "")
+ label_str: = strings.Trim (s [1], "")
+ prob, err: = strconv.ParseFloat (prob_str, 32)
+ if err! = nil {
+ panic (err)
+ }
+ label, err: = strconv.ParseFloat (label_str, 32)
+ if err! = nil {
+ panic (err)
+ }
+ if (prob-0.5) * (label-0.5)> 0 {
+ acc ++
+ }
+ total ++
+ }
+ fmt.Println ("total num:", total)
+ fmt.Println ("acc num:", acc)
+ fmt.Println ("acc:", float32 (acc) / float32 (total))
+
+}
+```
+
+```shell
+go run acc.go result
+total num: 25000
+acc num: 22014
+acc: 0.88056
+```
diff --git a/doc/NEW_OPERATOR.md b/doc/NEW_OPERATOR.md
index f839be94aaa2ae9993d935c0af69bcde33b9d66f..ab1ff42adea44eec26e84bd4356bc4313d420ce2 100644
--- a/doc/NEW_OPERATOR.md
+++ b/doc/NEW_OPERATOR.md
@@ -1,5 +1,7 @@
# How to write an general operator?
+([简体中文](./NEW_OPERATOR_CN.md)|English)
+
In this document, we mainly focus on how to develop a new server side operator for PaddleServing. Before we start to write a new operator, let's look at some sample code to get the basic idea of writing a new operator for server. We assume you have known the basic computation logic on server side of PaddleServing, please reference to []() if you do not know much about it. The following code can be visited at `core/general-server/op` of Serving repo.
``` c++
diff --git a/doc/NEW_OPERATOR_CN.md b/doc/NEW_OPERATOR_CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..d659b5f328cfbfc48ec7f3016037b12f34139b73
--- /dev/null
+++ b/doc/NEW_OPERATOR_CN.md
@@ -0,0 +1,149 @@
+# 如何开发一个新的General Op?
+
+(简体中文|[English](./NEW_OPERATOR.md))
+
+在本文档中，我们主要集中于如何为Paddle Serving开发新的服务器端运算符。 在开始编写新运算符之前，让我们看一些示例代码以获得为服务器编写新运算符的基本思想。 我们假设您已经知道Paddle Serving服务器端的基本计算逻辑。 下面的代码您可以在 Serving代码库下的 `core/general-server/op` 目录查阅。
+
+
+``` c++
+// Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#pragma once
+#include 
+
+imdb_reader.py
@@ -102,17 +102,17 @@ class IMDBDataset(dg.MultiSlotDataGenerator):
```
nets.py
@@ -156,7 +156,7 @@ def cnn_net(data,
local_train.py
@@ -172,7 +172,7 @@ logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger("fluid")
logger.setLevel(logging.INFO)
-# 加载词典文件
+# load dict file
def load_vocab(filename):
vocab = {}
with open(filename) as f:
@@ -190,11 +190,11 @@ if __name__ == "__main__":
vocab = load_vocab('imdb.vocab')
dict_dim = len(vocab)
- #定义模型输入
+ #define model input
data = fluid.layers.data(
name="words", shape=[1], dtype="int64", lod_level=1)
label = fluid.layers.data(name="label", shape=[1], dtype="int64")
- #定义dataset，train_data为训练数据目录
+ #define dataset，train_data is the dataset directory
dataset = fluid.DatasetFactory().create_dataset()
filelist = ["train_data/%s" % x for x in os.listdir("train_data")]
dataset.set_use_var([data, label])
@@ -203,11 +203,11 @@ if __name__ == "__main__":
dataset.set_batch_size(4)
dataset.set_filelist(filelist)
dataset.set_thread(10)
- #定义模型
+ #define model
avg_cost, acc, prediction = cnn_net(data, label, dict_dim)
optimizer = fluid.optimizer.SGD(learning_rate=0.001)
optimizer.minimize(avg_cost)
- #执行训练
+ #execute training
exe = fluid.Executor(fluid.CPUPlace())
exe.run(fluid.default_startup_program())
epochs = 100
@@ -219,7 +219,7 @@ if __name__ == "__main__":
program=fluid.default_main_program(), dataset=dataset, debug=False)
logger.info("TRAIN --> pass: {}".format(i))
if i == 64:
- #在训练结束时使用PaddleServing中的模型保存接口保存出Serving所需的模型和配置文件
+ #At the end of training, use the model save interface in PaddleServing to save the models and configuration files required by Serving
serving_io.save_model("{}_model".format(model_name),
"{}_client_conf".format(model_name),
{"words": data}, {"prediction": prediction},
@@ -228,32 +228,32 @@ if __name__ == "__main__":
test_client.py
@@ -267,7 +267,7 @@ client = Client()
client.load_client_config(sys.argv[1])
client.connect(["127.0.0.1:9292"])
-#在这里复用了数据预处理部分的代码将原始文本转换成数字id
+#The code of the data preprocessing part is reused here to convert the original text into a numeric id
imdb_dataset = IMDBDataset()
imdb_dataset.load_resource(sys.argv[2])
@@ -281,30 +281,29 @@ for line in sys.stdin:
text_clssify_service.py
@@ -313,7 +312,7 @@ from paddle_serving_server.web_service import WebService
from imdb_reader import IMDBDataset
import sys
-#继承框架中的WebService类
+#extend class WebService
class IMDBService(WebService):
def prepare_dict(self, args={}):
if len(args) == 0:
@@ -321,7 +320,7 @@ class IMDBService(WebService):
self.dataset = IMDBDataset()
self.dataset.load_resource(args["dict_file_path"])
- #重写preprocess方法来实现数据预处理，这里也复用了训练时使用的reader脚本
+ #rewrite preprocess() to implement data preprocessing, here we reuse reader script for training
def preprocess(self, feed={}, fetch=[]):
if "words" not in feed:
exit(-1)
@@ -329,7 +328,7 @@ class IMDBService(WebService):
res_feed["words"] = self.dataset.get_words_only(feed["words"])[0]
return res_feed, fetch
-#这里需要使用name参数指定预测服务的名称，
+#Here you need to use the name parameter to specify the name of the prediction service.
imdb_service = IMDBService(name="imdb")
imdb_service.load_model_config(sys.argv[1])
imdb_service.prepare_server(
@@ -339,24 +338,24 @@ imdb_service.run_server()
```
imdb_reader.py
+
+```python
+import sys
+import os
+import paddle
+import re
+import paddle.fluid.incubate.data_generator as dg
+
+
+class IMDBDataset(dg.MultiSlotDataGenerator):
+ def load_resource(self, dictfile):
+ self._vocab = {}
+ wid = 0
+ with open(dictfile) as f:
+ for line in f:
+ self._vocab[line.strip()] = wid
+ wid += 1
+ self._unk_id = len(self._vocab)
+ self._pattern = re.compile(r'(;|,|\.|\?|!|\s|\(|\))')
+ self.return_value = ("words", [1, 2, 3, 4, 5, 6]), ("label", [0])
+
+ def get_words_only(self, line):
+ sent = line.lower().replace("
", " ").strip()
+ words = [x for x in self._pattern.split(sent) if x and x != " "]
+ feas = [
+ self._vocab[x] if x in self._vocab else self._unk_id for x in words
+ ]
+ return feas
+
+ def get_words_and_label(self, line):
+ send = '|'.join(line.split('|')[:-1]).lower().replace("
",
+ " ").strip()
+ label = [int(line.split('|')[-1])]
+
+ words = [x for x in self._pattern.split(send) if x and x != " "]
+ feas = [
+ self._vocab[x] if x in self._vocab else self._unk_id for x in words
+ ]
+ return feas, label
+
+ def infer_reader(self, infer_filelist, batch, buf_size):
+ def local_iter():
+ for fname in infer_filelist:
+ with open(fname, "r") as fin:
+ for line in fin:
+ feas, label = self.get_words_and_label(line)
+ yield feas, label
+
+ import paddle
+ batch_iter = paddle.batch(
+ paddle.reader.shuffle(
+ local_iter, buf_size=buf_size),
+ batch_size=batch)
+ return batch_iter
+
+ def generate_sample(self, line):
+ def memory_iter():
+ for i in range(1000):
+ yield self.return_value
+
+ def data_iter():
+ feas, label = self.get_words_and_label(line)
+ yield ("words", feas), ("label", label)
+
+ return data_iter
+```
+nets.py
+
+```python
+import sys
+import time
+import numpy as np
+
+import paddle
+import paddle.fluid as fluid
+
+def cnn_net(data,
+ label,
+ dict_dim,
+ emb_dim=128,
+ hid_dim=128,
+ hid_dim2=96,
+ class_dim=2,
+ win_size=3):
+ """ conv net. """
+ emb = fluid.layers.embedding(
+ input=data, size=[dict_dim, emb_dim], is_sparse=True)
+
+ conv_3 = fluid.nets.sequence_conv_pool(
+ input=emb,
+ num_filters=hid_dim,
+ filter_size=win_size,
+ act="tanh",
+ pool_type="max")
+
+ fc_1 = fluid.layers.fc(input=[conv_3], size=hid_dim2)
+
+ prediction = fluid.layers.fc(input=[fc_1], size=class_dim, act="softmax")
+ cost = fluid.layers.cross_entropy(input=prediction, label=label)
+ avg_cost = fluid.layers.mean(x=cost)
+ acc = fluid.layers.accuracy(input=prediction, label=label)
+
+ return avg_cost, acc, prediction
+```
+
+local_train.py
+
+```python
+import os
+import sys
+import paddle
+import logging
+import paddle.fluid as fluid
+
+logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger("fluid")
+logger.setLevel(logging.INFO)
+
+# 加载词典文件
+def load_vocab(filename):
+ vocab = {}
+ with open(filename) as f:
+ wid = 0
+ for line in f:
+ vocab[line.strip()] = wid
+ wid += 1
+ vocab["test_client.py
+
+```python
+from paddle_serving_client import Client
+from imdb_reader import IMDBDataset
+import sys
+
+client = Client()
+client.load_client_config(sys.argv[1])
+client.connect(["127.0.0.1:9292"])
+
+#在这里复用了数据预处理部分的代码将原始文本转换成数字id
+imdb_dataset = IMDBDataset()
+imdb_dataset.load_resource(sys.argv[2])
+
+for line in sys.stdin:
+ word_ids, label = imdb_dataset.get_words_and_label(line)
+ feed = {"words": word_ids}
+ fetch = ["acc", "cost", "prediction"]
+ fetch_map = client.predict(feed=feed, fetch=fetch)
+ print("{} {}".format(fetch_map["prediction"][1], label[0]))
+```
+
+text_clssify_service.py
+
+```python
+from paddle_serving_server.web_service import WebService
+from imdb_reader import IMDBDataset
+import sys
+
+#继承框架中的WebService类
+class IMDBService(WebService):
+ def prepare_dict(self, args={}):
+ if len(args) == 0:
+ exit(-1)
+ self.dataset = IMDBDataset()
+ self.dataset.load_resource(args["dict_file_path"])
+
+ #重写preprocess方法来实现数据预处理，这里也复用了训练时使用的reader脚本
+ def preprocess(self, feed={}, fetch=[]):
+ if "words" not in feed:
+ exit(-1)
+ res_feed = {}
+ res_feed["words"] = self.dataset.get_words_only(feed["words"])[0]
+ return res_feed, fetch
+
+#这里需要使用name参数指定预测服务的名称，
+imdb_service = IMDBService(name="imdb")
+imdb_service.load_model_config(sys.argv[1])
+imdb_service.prepare_server(
+ workdir=sys.argv[2], port=int(sys.argv[3]), device="cpu")
+imdb_service.prepare_dict({"dict_file_path": sys.argv[4]})
+imdb_service.run_server()
+```
+