diff --git a/doc/HOT_LOADING_IN_SERVING.md b/doc/HOT_LOADING_IN_SERVING.md new file mode 100644 index 0000000000000000000000000000000000000000..093b703786c228558739a87be948e46ee4575045 --- /dev/null +++ b/doc/HOT_LOADING_IN_SERVING.md @@ -0,0 +1,190 @@ +# Hot Loading in Paddle Serving + +## Background + +In the industrial scenario, it is usually the remote periodic output model, and the online server needs to pull down the new model to update the old model without service interruption. + +Paddle Serving provides an automatic monitoring script. After the remote address updates the model, the new model will be pulled to update the local model. At the same time, the `fluid_time_stamp` in the local model folder will be updated to realize model hot loading. + +Currently, the following types of Monitors are supported: + +| Monitor Type | Description | Specific options | +| :----------: | :----------------------------------------------------------: | :----------------------------------------------------------: | +| General | Without authentication, you can directly access the download file by `wget` (such as FTP and BOS which do not need authentication) | `general_host` General remote host. | +| HDFS | The remote is HDFS, and relevant commands are executed through HDFS binary | `hdfs_bin` Path of HDFS binary file. | +| FTP | The remote is FTP, and relevant commands are executed through `ftplib`(Using this monitor, you need to install `ftplib` with command `pip install ftplib`) | `ftp_host` FTP remote host.
`ftp_port` FTP remote port.
`ftp_username` FTP username. Not used if anonymous access.
`ftp_password` FTP password. Not used if anonymous access. | +| AFS | The remote is AFS, and relevant commands are executed through Hadoop-client | `hadoop_bin` Path of Hadoop binary file.
`hadoop_host` AFS host. Not used if set in Hadoop-client.
`hadoop_ugi` AFS ugi, Not used if set in Hadoop-client. | + +| Monitor Shared options | Description | Default | +| :--------------------: | :----------------------------------------------------------: | :--------------------: | +| `type` | Specify the type of monitor | / | +| `remote_path` | Specify the base path for the remote | / | +| `remote_model_name` | Specify the model name to be pulled from the remote | / | +| `remote_donefile_name` | Specify the donefile name that marks the completion of the remote model update | / | +| `local_path` | Specify local work path | / | +| `local_model_name` | Specify local model name | / | +| `local_timestamp_file` | Specify the timestamp file used locally for hot loading, The file is considered to be placed in the `local_path/local_model_name` folder. | `fluid_time_file` | +| `local_tmp_path` | Specify the path of the folder where temporary files are stored locally. If it does not exist, it will be created automatically. | `_serving_monitor_tmp` | +| `interval` | Specify the polling interval in seconds. | `10` | +| `unpacked_filename` | Monitor supports the `tarfile` packaged remote model file. If the remote model is in a packaged format, you need to set this option to tell monitor the name of the extracted file. | `None` | + +The following is an example of HDFSMonitor to show the model hot loading of Paddle Serving. + +## HDFSMonitor example + +In this example, the production model is uploaded to HDFS in `product_path` folder, and the server hot loads the model in `server_path` folder: + +```shell +. +├── product_path +└── server_path +``` + +### Product model + +Run the following Python code products model in `product_path` folder. Every 60 seconds, the package file of Boston house price prediction model `uci_housing.tar.gz` will be generated and uploaded to the path of HDFS `/`. After uploading, the timestamp file `donefile` will be updated and uploaded to the path of HDFS `/`. + +```python +import os +import sys +import time +import tarfile +import paddle +import paddle.fluid as fluid +import paddle_serving_client.io as serving_io + +train_reader = paddle.batch( + paddle.reader.shuffle( + paddle.dataset.uci_housing.train(), buf_size=500), + batch_size=16) + +test_reader = paddle.batch( + paddle.reader.shuffle( + paddle.dataset.uci_housing.test(), buf_size=500), + batch_size=16) + +x = fluid.data(name='x', shape=[None, 13], dtype='float32') +y = fluid.data(name='y', shape=[None, 1], dtype='float32') + +y_predict = fluid.layers.fc(input=x, size=1, act=None) +cost = fluid.layers.square_error_cost(input=y_predict, label=y) +avg_loss = fluid.layers.mean(cost) +sgd_optimizer = fluid.optimizer.SGD(learning_rate=0.01) +sgd_optimizer.minimize(avg_loss) + +place = fluid.CPUPlace() +feeder = fluid.DataFeeder(place=place, feed_list=[x, y]) +exe = fluid.Executor(place) +exe.run(fluid.default_startup_program()) + +def push_to_hdfs(local_file_path, remote_path): + hdfs_bin = '/hadoop-3.1.2/bin/hdfs' + os.system('{} dfs -put -f {} {}'.format( + hdfs_bin, local_file_path, remote_path)) + +name = "uci_housing" +for pass_id in range(30): + for data_train in train_reader(): + avg_loss_value, = exe.run(fluid.default_main_program(), + feed=feeder.feed(data_train), + fetch_list=[avg_loss]) + # Simulate the production model every other period of time + time.sleep(60) + model_name = "{}_model".format(name) + client_name = "{}_client".format(name) + serving_io.save_model(model_name, client_name, + {"x": x}, {"price": y_predict}, + fluid.default_main_program()) + # Packing model + tar_name = "{}.tar.gz".format(name) + tar = tarfile.open(tar_name, 'w:gz') + tar.add(model_name) + tar.close() + + # Push packaged model file to hdfs + push_to_hdfs(tar_name, '/') + + # Generate donefile + donefile_name = 'donefile' + os.system('touch {}'.format(donefile_name)) + + # Push donefile to hdfs + push_to_hdfs(donefile_name, '/') +``` + +The files on HDFS are as follows: + +```bash +# hdfs dfs -ls / +Found 2 items +-rw-r--r-- 1 root supergroup 0 2020-04-02 02:54 /donefile +-rw-r--r-- 1 root supergroup 2101 2020-04-02 02:54 /uci_housing.tar.gz +``` + +### Server loading model + +Enter the `server_path` folder. + +#### Start server with the initial model + +Here, the trained Boston house price prediction model is used as the initial model: + +```shell +wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz +tar -xzf uci_housing.tar.gz +``` + +Start Server： + +```shell +python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292 +``` + +#### Execute monitor + +Use the following command to execute the HDFSMonitor: + +```shell +python -m paddle_serving_server.monitor \ + --type='hdfs' --hdfs_bin='/hadoop-3.1.2/bin/hdfs' --remote_path='/' \ + --remote_model_name='uci_housing.tar.gz' --remote_donefile_name='donefile' \ + --local_path='.' --local_model_name='uci_housing_model' \ + --local_timestamp_file='fluid_time_file' --local_tmp_path='_tmp' \ + --unpacked_filename='uci_housing_model' +``` + +The above code monitors the remote timestamp file `/donefile` of the remote HDFS address `/` every 10 seconds by polling. When the remote timestamp file changes, the remote model is considered to have been updated. Pull the remote packaging model `/uci_housing.tar.gz` to the local temporary path `./_tmp/uci_housing.tar.gz`. After unpacking to get the model file `./_tmp/uci_housing_model`, update the local model `./uci_housing_model` and the model timestamp file `./uci_housing_model/fluid_time_file` of Paddle Serving. + +#### View server logs + +View the running log of the server with the following command: + +```shell +tail -f log/serving.INFO +``` + +The log shows that the model has been hot loaded: + +```shell +I0330 09:38:40.087316 7361 server.cpp:150] Begin reload framework... +W0330 09:38:40.087399 7361 infer.h:656] Succ reload version engine: 18446744073709551615 +I0330 09:38:40.087414 7361 manager.h:131] Finish reload 1 workflow(s) +I0330 09:38:50.087535 7361 server.cpp:150] Begin reload framework... +W0330 09:38:50.087641 7361 infer.h:250] begin reload model[uci_housing_model]. +I0330 09:38:50.087972 7361 infer.h:66] InferEngineCreationParams: model_path = uci_housing_model, enable_memory_optimization = 0, static_optimization = 0, force_update_static_cache = 0 +I0330 09:38:50.088027 7361 analysis_predictor.cc:88] Profiler is deactivated, and no profiling report will be generated. +I0330 09:38:50.088393 7361 analysis_predictor.cc:841] MODEL VERSION: 1.7.1 +I0330 09:38:50.088413 7361 analysis_predictor.cc:843] PREDICTOR VERSION: 1.6.3 +I0330 09:38:50.089519 7361 graph_pattern_detector.cc:96] --- detected 1 subgraphs +I0330 09:38:50.090925 7361 analysis_predictor.cc:470] ======= optimize end ======= +W0330 09:38:50.090986 7361 infer.h:472] Succ load common model[0x7fc83c06abd0], path[uci_housing_model]. +I0330 09:38:50.091022 7361 analysis_predictor.cc:88] Profiler is deactivated, and no profiling report will be generated. +W0330 09:38:50.091050 7361 infer.h:509] td_core[0x7fc83c0ad770] clone model from pd_core[0x7fc83c06abd0] succ, cur_idx[0]. +... +W0330 09:38:50.091784 7361 infer.h:489] Succ load clone model, path[uci_housing_model] +W0330 09:38:50.091794 7361 infer.h:656] Succ reload version engine: 18446744073709551615 +I0330 09:38:50.091820 7361 manager.h:131] Finish reload 1 workflow(s) +I0330 09:39:00.091987 7361 server.cpp:150] Begin reload framework... +W0330 09:39:00.092161 7361 infer.h:656] Succ reload version engine: 18446744073709551615 +I0330 09:39:00.092177 7361 manager.h:131] Finish reload 1 workflow(s) +``` diff --git a/doc/HOT_LOADING_IN_SERVING_CN.md b/doc/HOT_LOADING_IN_SERVING_CN.md index 7aa99351a9b2879d0d74a9bf45adfb9920efadba..c210bf90b4b982ef4b777ee44e85a1684eacc9cb 100644 --- a/doc/HOT_LOADING_IN_SERVING_CN.md +++ b/doc/HOT_LOADING_IN_SERVING_CN.md @@ -4,27 +4,31 @@ 在实际的工业场景下，通常是远端定期不间断产出模型，线上服务端需要在服务不中断的情况下拉取新模型对旧模型进行更新迭代。 -Paddle Serving目前支持下面几种类型的远端监控Monitor： +## Server Monitor + +Paddle Serving提供了一个自动监控脚本，远端地址更新模型后会拉取新模型更新本地模型，同时更新本地模型文件夹中的时间戳文件`fluid_time_stamp`实现热加载。 + +目前支持下面几种类型的远端监控Monitor： | Monitor类型 | 描述 | 特殊选项 | | :---------: | :----------------------------------------------------------: | :----------------------------------------------------------: | -| General | 远端无认证，可以通过`wget`直接访问下载文件（如无需认证的FTP，OBS等） | `general_host` 通用远端host | +| General | 远端无认证，可以通过`wget`直接访问下载文件（如无需认证的FTP，BOS等） | `general_host` 通用远端host | | HDFS | 远端为HDFS，通过HDFS二进制执行相关命令 | `hdfs_bin` HDFS二进制的路径 | -| FTP | 远端为FTP，可以通过用户名、密码访问 | `ftp_host` FTP host
`ftp_port` FTP port
`ftp_username` FTP username，默认为空
`ftp_password` FTP password，默认为空 | +| FTP | 远端为FTP，通过`ftplib`进行相关访问（使用该Monitor，您可能需要执行`pip install ftplib`下载`ftplib`） | `ftp_host` FTP host
`ftp_port` FTP port
`ftp_username` FTP username，默认为空
`ftp_password` FTP password，默认为空 | | AFS | 远端为AFS，通过Hadoop-client执行相关命令 | `hadoop_bin` Hadoop二进制的路径
`hadoop_host` AFS host，默认为空
`hadoop_ugi` AFS ugi，默认为空 | -| Monitor通用选项 | 描述 | 默认值 | -| :--------------------: | :----------------------------------------------------------: | :------------------------------------------: | -| `type` | 指定Monitor类型 | 无 | -| `remote_path` | 指定远端的基础路径 | 无 | -| `remote_model_name` | 指定远端需要拉取的模型名 | 无 | -| `remote_donefile_name` | 指定远端标志模型更新完毕的donefile文件名 | 无 | -| `local_path` | 指定本地工作路径 | 无 | -| `local_model_name` | 指定本地模型名 | 无 | -| `local_timestamp_file` | 指定本地用于热加载的时间戳文件，该文件被认为在`local_path/local_model_name`下。 | `fluid_time_file` | -| `local_tmp_path` | 指定本地存放临时文件的文件夹路径。 | `_serving_monitor_tmp`（若不存在则自动创建） | -| `interval` | 指定轮询间隔时间。 | 10（秒） | -| `unpacked_filename` | Monitor支持tarfile打包的远程模型。如果远程模型是打包格式，则需要设置该选项来告知Monitor解压后的文件名。 | None | +| Monitor通用选项 | 描述 | 默认值 | +| :--------------------: | :----------------------------------------------------------: | :--------------------: | +| `type` | 指定Monitor类型 | 无 | +| `remote_path` | 指定远端的基础路径 | 无 | +| `remote_model_name` | 指定远端需要拉取的模型名 | 无 | +| `remote_donefile_name` | 指定远端标志模型更新完毕的donefile文件名 | 无 | +| `local_path` | 指定本地工作路径 | 无 | +| `local_model_name` | 指定本地模型名 | 无 | +| `local_timestamp_file` | 指定本地用于热加载的时间戳文件，该文件被认为在`local_path/local_model_name`下。 | `fluid_time_file` | +| `local_tmp_path` | 指定本地存放临时文件的文件夹路径，若不存在则自动创建。 | `_serving_monitor_tmp` | +| `interval` | 指定轮询间隔时间，单位为秒。 | `10` | +| `unpacked_filename` | Monitor支持tarfile打包的远程模型。如果远程模型是打包格式，则需要设置该选项来告知Monitor解压后的文件名。 | `None` | 下面通过HDFSMonitor示例来展示Paddle Serving的模型热加载功能。 @@ -40,7 +44,7 @@ Paddle Serving目前支持下面几种类型的远端监控Monitor： ### 生产模型 -在`product_path`下运行下面的Python代码生产模型，每隔 60 秒会产出 Boston 房价预测模型`uci_housing_model`并上传至hdfs的`/`路径下，上传完毕后更新时间戳文件`donefile`并上传至hdfs的`/`路径下。 +在`product_path`下运行下面的Python代码生产模型，每隔 60 秒会产出 Boston 房价预测模型的打包文件`uci_housing.tar.gz`并上传至hdfs的`/`路径下，上传完毕后更新时间戳文件`donefile`并上传至hdfs的`/`路径下。 ```python import os @@ -93,7 +97,7 @@ for pass_id in range(30): serving_io.save_model(model_name, client_name, {"x": x}, {"price": y_predict}, fluid.default_main_program()) - # Package model + # Packing model tar_name = "{}.tar.gz".format(name) tar = tarfile.open(tar_name, 'w:gz') tar.add(model_name) @@ -151,7 +155,7 @@ python -m paddle_serving_server.monitor \ --unpacked_filename='uci_housing_model' ``` -上面代码通过轮询方式监控远程HDFS地址`/`的时间戳文件`/donefile`，当时间戳变更则认为远程模型已经更新，将远程模型`/uci_housing_model`拉取到本地临时路径`./_tmp/uci_housing_model`下，更新本地模型`./uci_housing_model`以及Paddle Serving的时间戳文件`./uci_housing_model/fluid_time_file`。 +上面代码通过轮询方式监控远程HDFS地址`/`的时间戳文件`/donefile`，当时间戳变更则认为远程模型已经更新，将远程打包模型`/uci_housing.tar.gz`拉取到本地临时路径`./_tmp/uci_housing.tar.gz`下，解包出模型文件`./_tmp/uci_housing_model`后，更新本地模型`./uci_housing_model`以及Paddle Serving的时间戳文件`./uci_housing_model/fluid_time_file`。 #### 查看Server日志