This tutorial will introduce the detailed steps of deploying the PaddleClas classification model on the server side. The deployment of the recognition model will be supported in the near future. Please look forward to it.
---
## Contents
-[1. Prepare the environment](#1)
-[1.1 Compile OpenCV](#1.1)
-[1.2 Compile or download the Paddle Inference Library](#1.2)
-[1.2.1 Compile from the source code](#1.2.1)
-[1.2.2 Direct download and installation](#1.2.2)
-[2. Compile](#2)
-[2.1 Compile PaddleClas C++ inference demo](#2.1)
-[2.2 Compile config lib and cls lib](#2.2)
-[3. Run](#3)
-[3.1 Prepare inference model](#3.1)
-[3.2 Run demo](#3.2)
<aname="1"></a>
## 1. Prepare the environment
### Environment
- Linux, docker is recommended.
- Windows, compilation based on `Visual Studio 2019 Community` is supported. In addition, you can refer to [How to use PaddleDetection to make a complete project](https://zhuanlan.zhihu.com/p/145446681) to compile by generating the `sln solution`.
- This document mainly introduces the compilation and inference of PaddleClas using C++ in Linux environment.
- If you need to use the Inference Library in Windows environment, please refer to [The compilation tutorial in Windows](./docs/windows_vs2019_build.md) for detailed information.
<aname="1.1"></a>
### 1.1 Compile opencv
* First of all, you need to download the source code compiled package in the Linux environment from the opencv official website. Taking opencv3.4.7 as an example, the download and uncompress command are as follows.
Finally, you can see the folder of `opencv-3.4.7/` in the current directory.
* Compile opencv, the opencv source path (`root_path`) and installation path (`install_path`) should be set by yourself. Among them, `root_path` is the downloaded opencv source code path, and `install_path` is the installation path of opencv. In this case, the opencv source is `./opencv-3.4.7`.
```shell
cd ./opencv-3.4.7
export root_path=$PWD
export install_path=${root_path}/opencv3
```
* After entering the opencv source code path, you can compile it in the following way.
```shell
rm-rf build
mkdir build
cd build
cmake .. \
-DCMAKE_INSTALL_PREFIX=${install_path}\
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DWITH_IPP=OFF \
-DBUILD_IPP_IW=OFF \
-DWITH_LAPACK=OFF \
-DWITH_EIGEN=OFF \
-DCMAKE_INSTALL_LIBDIR=lib64 \
-DWITH_ZLIB=ON \
-DBUILD_ZLIB=ON \
-DWITH_JPEG=ON \
-DBUILD_JPEG=ON \
-DWITH_PNG=ON \
-DBUILD_PNG=ON \
-DWITH_TIFF=ON \
-DBUILD_TIFF=ON
make -j
make install
```
* After `make install` is completed, the opencv header file and library file will be generated in this folder for later PaddleClas source code compilation.
Take opencv3.4.7 for example, the final file structure under the opencv installation path is as follows. **NOTICE**:The following file structure may be different for different Versions of Opencv.
```
opencv3/
|-- bin
|-- include
|-- lib64
|-- share
```
<aname="1.2"></a>
### 1.2 Compile or download the Paddle Inference Library
* There are 2 ways to obtain the Paddle Inference Library, described in detail below.
<aname="1.2.1"></a>
#### 1.2.1 Compile from the source code
* If you want to get the latest Paddle Inference Library features, you can download the latest code from Paddle GitHub repository and compile the inference library from the source code.
* You can refer to [Paddle Inference Library](https://www.paddlepaddle.org.cn/documentation/docs/en/develop/guides/05_inference_deployment/inference/build_and_install_lib_en.html#build-from-source-code) to get the Paddle source code from github, and then compile To generate the latest inference library. The method of using git to access the code is as follows.
* After entering the Paddle directory, the compilation method is as follows.
```shell
rm-rf build
mkdir build
cd build
cmake .. \
-DWITH_CONTRIB=OFF \
-DWITH_MKL=ON \
-DWITH_MKLDNN=ON \
-DWITH_TESTING=OFF \
-DCMAKE_BUILD_TYPE=Release \
-DWITH_INFERENCE_API_TEST=OFF \
-DON_INFER=ON \
-DWITH_PYTHON=ON
make -j
make inference_lib_dist
```
For more compilation parameter options, please refer to the official website of the Paddle C++ inference library:[https://www.paddlepaddle.org.cn/documentation/docs/en/develop/guides/05_inference_deployment/inference/build_and_install_lib_en.html#build-from-source-code](https://www.paddlepaddle.org.cn/documentation/docs/en/develop/guides/05_inference_deployment/inference/build_and_install_lib_en.html#build-from-source-code).
* After the compilation process, you can see the following files in the folder of `build/paddle_inference_install_dir/`.
```
build/paddle_inference_install_dir/
|-- CMakeCache.txt
|-- paddle
|-- third_party
|-- version.txt
```
Among them, `paddle` is the Paddle library required for C++ prediction later, and `version.txt` contains the version information of the current inference library.
<aname="1.2.2"></a>
#### 1.2.2 Direct download and installation
* Different cuda versions of the Linux inference library (based on GCC 4.8.2) are provided on the
[Paddle Inference Library official website](https://www.paddlepaddle.org.cn/documentation/docs/en/develop/guides/05_inference_deployment/inference/build_and_install_lib_en.html). You can view and select the appropriate version of the inference library on the official website.
* Please select the `develop` version.
* After downloading, use the following method to uncompress.
```
tar -xf paddle_inference.tgz
```
Finally you can see the following files in the folder of `paddle_inference/`.
<aname="2"></a>
## 2. Compile
<aname="2.1"></a>
### 2.1 Compile PaddleClas C++ inference demo
* The compilation commands are as follows. The addresses of Paddle C++ inference library, opencv and other Dependencies need to be replaced with the actual addresses on your own machines.
```shell
sh tools/build.sh
```
Specifically, the content in `tools/build.sh` is as follows.
```shell
OPENCV_DIR=your_opencv_dir
LIB_DIR=your_paddle_inference_dir
CUDA_LIB_DIR=your_cuda_lib_dir
CUDNN_LIB_DIR=your_cudnn_lib_dir
TENSORRT_DIR=your_tensorrt_lib_dir
BUILD_DIR=build
rm-rf${BUILD_DIR}
mkdir${BUILD_DIR}
cd${BUILD_DIR}
cmake .. \
-DPADDLE_LIB=${LIB_DIR}\
-DWITH_MKL=ON \
-DDEMO_NAME=clas_system \
-DWITH_GPU=OFF \
-DWITH_STATIC_LIB=OFF \
-DWITH_TENSORRT=OFF \
-DTENSORRT_DIR=${TENSORRT_DIR}\
-DOPENCV_DIR=${OPENCV_DIR}\
-DCUDNN_LIB=${CUDNN_LIB_DIR}\
-DCUDA_LIB=${CUDA_LIB_DIR}\
make -j
```
In the above parameters of command:
*`OPENCV_DIR` is the opencv installation path;
*`LIB_DIR` is the download (`paddle_inference` folder) or the generated Paddle Inference Library path (`build/paddle_inference_install_dir` folder);
*`CUDA_LIB_DIR` is the cuda library file path, in docker; it is `/usr/local/cuda/lib64`;
*`CUDNN_LIB_DIR` is the cudnn library file path, in docker it is `/usr/lib/x86_64-linux-gnu/`.
*`TENSORRT_DIR` is the tensorrt library file path，in dokcer it is `/usr/local/TensorRT6-cuda10.0-cudnn7/`，TensorRT is just enabled for GPU.
After the compilation is completed, an executable file named `clas_system` will be generated in the `build` folder.
<aname="2.2"></a>
### 2.2 Compile config lib and cls lib
In addition to compiling the demo directly, you can also compile only config lib and cls lib by running the following command:
```shell
sh tools/build_lib.sh
```
The contents of the above command are as follows:
```shell
OpenCV_DIR=path/to/opencv
PADDLE_LIB_DIR=path/to/paddle
BUILD_DIR=./lib/build
rm-rf${BUILD_DIR}
mkdir${BUILD_DIR}
cd${BUILD_DIR}
cmake .. \
-DOpenCV_DIR=${OpenCV_DIR}\
-DPADDLE_LIB=${PADDLE_LIB_DIR}\
-DCMP_STATIC=ON \
make
```
The specific description of each compilation option is as follows:
*`DOpenCV_DIR`: The directory to the OpenCV compilation library. In this example, it is `opencv-3.4.7/opencv3/share/OpenCV`. Note that there needs to be a `OpenCVConfig.cmake` file under this directory;
*`DPADDLE_LIB`: The directory to the paddle prediction library which generally is the path of `paddle_inference` downloaded and decompressed or compiled, such as `build/paddle_inference_install_dir`. Note that there should be two subdirectories `paddle` and `third_party` in this directory;
*`DCMP_STATIC`: Whether to compile config lib and cls lib into static link library (`.a`). The default is `ON`. If you need to compile into dynamic link library (`.so`), please set it to `OFF`.
After executing the above commands, the dynamic link libraries (`libcls.so` and `libconfig.so`) or static link libraries (`cls.a` and `libconfig.a`) of config lib and cls lib will be generated in the directory. In the [2.1 Compile PaddleClas C++ inference demo](#2.1), you can specify the compilation option `DCLS_LIB` and `DCONFIG_LIB` to the path of the existing link library of `cls lib` and `config lib`, which can also be used for development.
<aname="3"></a>
## 3. Run the demo
<aname="3.1"></a>
### 3.1 Prepare the inference model
* You can refer to [Model inference](../../tools/export_model.py)，export the inference model. After the model is exported, assuming it is placed in the `inference` directory, the directory structure is as follows.
```
inference/
|--cls_infer.pdmodel
|--cls_infer.pdiparams
```
**NOTICE**: Among them, `cls_infer.pdmodel` file stores the model structure information and the `cls_infer.pdiparams` file stores the model parameter information.The paths of the two files need to correspond to the parameters of `cls_model_path` and `cls_params_path` in the configuration file `tools/config.txt`.
<aname="3.2"></a>
### 3.2 Run demo
First, please modify the `tools/config.txt` and `tools/run.sh`.
* Some key words in `tools/config.txt` is as follows.
* use_gpu: Whether to use GPU.
* gpu_id: GPU id.
* gpu_mem：GPU memory.
* cpu_math_library_num_threads：Number of thread for math library acceleration.
* use_mkldnn：Whether to use mkldnn.
* use_tensorrt: Whether to use tensorRT.
* use_fp16：Whether to use Float16 (half precision), it is just enabled when use_tensorrt is set as 1.
* cls_model_path: Model path of inference model.
* cls_params_path: Params path of inference model.
* resize_short_size：Short side length of the image after resize.
* crop_size：Image size after center crop.
* You could modify `tools/run.sh`(`./build/clas_system ./tools/config.txt ./docs/imgs/ILSVRC2012_val_00000666.JPEG`):
* ./build/clas_system: the path of executable file compiled;
* ./tools/config.txt: the path of config;
* ./docs/imgs/ILSVRC2012_val_00000666.JPEG: the path of image file to be predicted.
* Then execute the following command to complete the classification of an image.
```shell
sh tools/run.sh
```
* The prediction results will be shown on the screen, which is as follows.
* In the above results,`class id` represents the id corresponding to the category with the highest confidence, and `score` represents the probability that the image belongs to that category.
PaddleClas在Windows 平台下基于`Visual Studio 2019 Community` 进行了测试。微软从`Visual Studio 2017`开始即支持直接管理`CMake`跨平台编译项目，但是直到`2019`才提供了稳定和完全的支持，所以如果你想使用CMake管理项目编译构建，我们推荐使用`Visual Studio 2019`。如果您希望通过生成`sln解决方案`的方式进行编译，可以参考该文档：[https://zhuanlan.zhihu.com/p/145446681](https://zhuanlan.zhihu.com/p/145446681)。
## 前置条件
* Visual Studio 2019
* CUDA 9.0 / CUDA 10.0，cudnn 7.6+ （仅在使用GPU版本的预测库时需要）
* CMake 3.0+
请确保系统已经正确安装并配置好上述基本软件，其中：
* 在安装`Visual Studio 2019`时，`工作负载`需要勾选`使用C++的桌面开发`；
* CUDA需要正确安装并设置系统环境变量；
* CMake需要正确安装并将路径添加到系统环境变量中。
以下测试基于`Visual Studio 2019 Community`版本。
**下面所有示例以工作目录为 `D:\projects`演示**。
### Step1: 下载PaddlePaddle C++ 预测库 paddle_inference_install_dir
PaddlePaddle C++ 预测库针对不同的`CPU`和`CUDA`版本提供了不同的预编译版本，请根据实际情况下载: [C++预测库下载列表](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/05_inference_deployment/inference/windows_cpp_inference.html)。
PaddlePaddle supports exporting inference model for deployment. Compared with training, inference model files store network weights and network structures persistently, and PaddlePaddle supports more fast prediction engine loading inference model to deployment.
---
## Contents
-[1. 环境准备](#1)
-[2. 分类模型导出](#2)
-[3. 主体检测模型导出](#3)
-[4. 识别模型导出](#4)
-[5. 命令参数说明](#5)
<aname="1"></a>
## 1. Environmental preparation
First, refer to the [Installing PaddlePaddle](../installation/install_paddle_en.md) and the [Installing PaddleClas](../installation/install_paddleclas_en.md) to prepare environment.
<aname="2"></a>
## 2. Export classification model
Change the working directory to PaddleClas:
```shell
cd /path/to/PaddleClas
```
Taking the classification model ResNet50_vd as an example, download the pre-trained model:
The above model weights is trained by ResNet50_vd model on ImageNet1k dataset and training configuration file is `ppcls/configs/ImageNet/ResNet/ResNet50_vd.yaml`. To export the inference model, just run the following command:
The above model weights file is trained by ResNet50_vd on AliProduct dataset, and the training configuration file is `ppcls/configs/Products/ResNet50_vd_Aliproduct.yaml`. The command to export inference model is as follow:
Notice, the inference model exported by above command is truncated on embedding layer, so the output of the model is n-dimensional embedding feature.
<aname="5"></a>
## 5. Parameter description
In the above model export command, the configuration file used must be the same as the training configuration file. The following fields in the configuration file are used to configure exporting model parameters.
*`Global.image_shape`：To specify the input data size of the model, which does not contain the batch dimension;
*`Global.save_inference_dir`：To specify directory of saving inference model files exported;
*`Global.pretrained_model`：To specify the path of model weight file saved during training. This path does not need to contain the suffix `.pdparams` of model weight file;
The exporting model command will generate the following three files:
*`inference.pdmodel`：To store model network structure information;
*`inference.pdiparams`：To store model network weight information;
*`inference.pdiparams.info`：To store the parameter information of the model, which can be ignored in the classification model and recognition model;
The inference model exported is used to deployment by using prediction engine. You can refer the following docs according to different deployment modes / platforms
*[Python inference](./python_deploy.md)
*[C++ inference](./cpp_deploy.md)(Only support classification)
*[Python Whl inference](./whl_deploy.md)(Only support classification)
*[PaddleHub Serving inference](./paddle_hub_serving_deploy.md)(Only support classification)
PaddleClas supports rapid service deployment through Paddlehub. At present, it supports the deployment of image classification. Please look forward to the deployment of image recognition.
---
## Contents
-[1. Introduction](#1)
-[2. Prepare the environment](#2)
-[3. Download inference model](#3)
-[4. Install Service Module](#4)
-[5. Start service](#5)
-[5.1 Start with command line parameters](#5.1)
-[5.2 Start with configuration file](#5.2)
-[6. Send prediction requests](#6)
-[7. User defined service module modification](#7)
HubServing service pack contains 3 files, the directory is as follows:
```
hubserving/clas/
└─ __init__.py Empty file, required
└─ config.json Configuration file, optional, passed in as a parameter when using configuration to start the service
└─ module.py Main module file, required, contains the complete logic of the service
└─ params.py Parameter file, required, including parameters such as model path, pre- and post-processing parameters
Before installing the service module, you need to prepare the inference model and put it in the correct path. The default model path is:
* Model structure file: `PaddleClas/inference/inference.pdmodel`
* Model parameters file: `PaddleClas/inference/inference.pdiparams`
**Notice**:
* The model file path can be viewed and modified in `PaddleClas/deploy/hubserving/clas/params.py`.
* It should be noted that the prefix of model structure file and model parameters file must be `inference`.
* More models provided by PaddleClas can be obtained from the [model library](../../docs/en/models/models_intro_en.md). You can also use models trained by yourself.
<aname="4"></a>
## 4. Install Service Module
* On Linux platform, the examples are as follows.
```shell
cd PaddleClas/deploy
hub install hubserving/clas/
```
* On Windows platform, the examples are as follows.
```shell
cd PaddleClas\deploy
hub install hubserving\clas\
```
<aname="5"></a>
## 5. Start service
<aname="5.1"></a>
### 5.1 Start with command line parameters
This method only supports CPU. Command as follow:
```shell
$ hub serving start --modulesModule1==Version1 \
--port XXXX \
--use_multiprocess\
--workers\
```
**parameters：**
|parameters|usage|
|-|-|
|--modules/-m|PaddleHub Serving pre-installed model, listed in the form of multiple Module==Version key-value pairs<br>*`When Version is not specified, the latest version is selected by default`*|
|--port/-p|Service port, default is 8866|
|--use_multiprocess|Enable concurrent mode, the default is single-process mode, this mode is recommended for multi-core CPU machines<br>*`Windows operating system only supports single-process mode`*|
|--workers|The number of concurrent tasks specified in concurrent mode, the default is `2*cpu_count-1`, where `cpu_count` is the number of CPU cores|
For example, start service:
```shell
hub serving start -m clas_system
```
This completes the deployment of a service API, using the default port number 8866.
<aname="5.2"></a>
### 5.2 Start with configuration file
This method supports CPU and GPU. Command as follow:
```shell
hub serving start --config/-c config.json
```
Wherein, the format of `config.json` is as follows:
```json
{
"modules_info":{
"clas_system":{
"init_args":{
"version":"1.0.0",
"use_gpu":true,
"enable_mkldnn":false
},
"predict_args":{
}
}
},
"port":8866,
"use_multiprocess":false,
"workers":2
}
```
- The configurable parameters in `init_args` are consistent with the `_initialize` function interface in `module.py`. Among them,
- when `use_gpu` is `true`, it means that the GPU is used to start the service.
- when `enable_mkldnn` is `true`, it means that use MKL-DNN to accelerate.
- The configurable parameters in `predict_args` are consistent with the `predict` function interface in `module.py`.
**Note:**
- When using the configuration file to start the service, other parameters will be ignored.
- If you use GPU prediction (that is, `use_gpu` is set to `true`), you need to set the environment variable CUDA_VISIBLE_DEVICES before starting the service, such as: ```export CUDA_VISIBLE_DEVICES=0```, otherwise you do not need to set it.
-**`use_gpu` and `use_multiprocess` cannot be `true` at the same time.**
-**When both `use_gpu` and `enable_mkldnn` are set to `true` at the same time, GPU is used to run and `enable_mkldnn` will be ignored.**
For example, use GPU card No. 3 to start the 2-stage series service:
```shell
cd PaddleClas/deploy
export CUDA_VISIBLE_DEVICES=3
hub serving start -c hubserving/clas/config.json
```
<aname="6"></a>
## 6. Send prediction requests
After the service starting, you can use the following command to send a prediction request to obtain the prediction result:
-**image_path**: Test image path, can be a single image path or an image directory path
-**batch_size**: [**Optional**] batch_size. Default by `1`.
-**resize_short**: [**Optional**] In preprocessing, resize according to short size. Default by `256`。
-**crop_size**: [**Optional**] In preprocessing, centor crop size. Default by `224`。
-**normalize**: [**Optional**] In preprocessing, whether to do `normalize`. Default by `True`。
-**to_chw**: [**Optional**] In preprocessing, whether to transpose to `CHW`. Default by `True`。
**Notice**:
If you want to use `Transformer series models`, such as `DeiT_***_384`, `ViT_***_384`, etc., please pay attention to the input size of model, and need to set `--resize_short=384`, `--crop_size=384`.
The returned result is a list, including the `top_k`'s classification results, corresponding scores and the time cost of prediction, details as follows.
```
list: The returned results
└─ list: The result of first picture
└─ list: The top-k classification results, sorted in descending order of score
└─ list: The scores corresponding to the top-k classification results, sorted in descending order of score
└─ float: The time cost of predicting the picture, unit second
```
**Note：** If you need to add, delete or modify the returned fields, you can modify the corresponding module. For the details, refer to the user-defined modification service module in the next section.
<aname="7"></a>
## 7. User defined service module modification
If you need to modify the service logic, the following steps are generally required:
1. Stop service
```shell
hub serving stop --port/-p XXXX
```
2. Modify the code in the corresponding files, like `module.py` and `params.py`, according to the actual needs. You need re-install(hub install hubserving/clas/) and re-deploy after modifing `module.py`.
After modifying and installing and before deploying, you can use `python hubserving/clas/module.py` to test the installed service module.
For example, if you need to replace the model used by the deployed service, you need to modify model path parameters `cfg.model_file` and `cfg.params_file` in `params.py`. Of course, other related parameters may need to be modified at the same time. Please modify and debug according to the actual situation.
3. Uninstall old service module
```shell
hub uninstall clas_system
```
4. Install modified service module
```shell
hub install hubserving/clas/
```
5. Restart service
```shell
hub serving start -m clas_system
```
**Note**:
Common parameters can be modified in params.py:
* Directory of model files(include model structure file and model parameters file):
```python
"inference_model_dir":
```
* The number of Top-k results returned during post-processing:
```python
'topk':
```
* Mapping file corresponding to label and class ID during post-processing:
```python
'class_id_map_file':
```
In order to avoid unnecessary delay and be able to predict in batch, the preprocessing (include resize, crop and other) is completed in the client, so modify [test_hubserving.py](../../deploy/hubserving/test_hubserving.py#L35-L52) if necessary.
This tutorial will introduce how to use [Paddle-Lite](https://github.com/PaddlePaddle/Paddle-Lite) to deploy PaddleClas models on mobile phones.
Paddle-Lite is a lightweight inference engine for PaddlePaddle. It provides efficient inference capabilities for mobile phones and IoTs, and extensively integrates cross-platform hardware to provide lightweight deployment solutions for mobile-side deployment issues.
If you only want to test speed, please refer to [The tutorial of Paddle-Lite mobile-side benchmark test](../../docs/zh_CN/extension/paddle_mobile_inference.md).
---
## Contents
-[1. Preparation](#1)
-[1.1 Build Paddle-Lite library](#1.1)
-[1.2 Download inference library for Android or iOS](#1.2)
-[2. Start running](#2)
-[2.1 Inference Model Optimization](#2.1)
-[2.1.1 [RECOMMEND] Use pip to install Paddle-Lite and optimize model](#2.1.1)
-[2.1.2 Compile Paddle-Lite to generate opt tool](#2.1.2)
-[2.1.3 Demo of get the optimized model](#2.1.3)
-[2.2 Run optimized model on Phone](#2.2)
-[3. FAQ](#3)
<aname="1"></a>
## 1. Preparation
PaddeLite currently supports the following platforms:
- Computer (for compiling Paddle-Lite)
- Mobile phone (arm7 or arm8)
<aname="1.1"></a>
### 1.1 Prepare cross-compilation environment
The cross-compilation environment is used to compile the C++ demos of Paddle-Lite and PaddleClas.
For the detailed compilation directions of different development environments, please refer to the corresponding [document](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html).
<aname="1.2"></a>
## 1.2 Download inference library for Android or iOS
1. If you download the inference library from [Paddle-Lite official document](https://paddle-lite.readthedocs.io/zh/latest/quick_start/release_lib.html#android-toolchain-gcc), please choose `with_extra=ON` , `with_cv=ON` .
2. It is recommended to build inference library using [Paddle-Lite](https://github.com/PaddlePaddle/Paddle-Lite) develop branch if you want to deploy the [quantitative](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/deploy/slim/quantization/README_en.md) model to mobile phones. Please refer to the [link](https://paddle-lite.readthedocs.io/zh/latest/user_guides/Compile/Android.html#id2) for more detailed information about compiling.
The structure of the inference library is as follows:
```
inference_lite_lib.android.armv8/
|-- cxx C++ inference library and header files
| |-- include C++ header files
| | |-- paddle_api.h
| | |-- paddle_image_preprocess.h
| | |-- paddle_lite_factory_helper.h
| | |-- paddle_place.h
| | |-- paddle_use_kernels.h
| | |-- paddle_use_ops.h
| | `-- paddle_use_passes.h
| `-- lib C++ inference library
| |-- libpaddle_api_light_bundled.a C++ static library
| `-- libpaddle_light_api_shared.so C++ dynamic library
|-- java Java inference library
| |-- jar
| | `-- PaddlePredictor.jar
| |-- so
| | `-- libpaddle_lite_jni.so
| `-- src
|-- demo C++ and java demos
| |-- cxx C++ demos
| `-- java Java demos
```
<aname="2"></a>
## 2. Start running
<aname="2.1"></a>
## 2.1 Inference Model Optimization
Paddle-Lite provides a variety of strategies to automatically optimize the original training model, including quantization, sub-graph fusion, hybrid scheduling, Kernel optimization and so on. In order to make the optimization process more convenient and easy to use, Paddle-Lite provides `opt` tool to automatically complete the optimization steps and output a lightweight, optimal executable model.
**NOTE**: If you have already got the `.nb` file, you can skip this step.
<aname="2.1.1"></a>
### 2.1.1 [RECOMMEND] Use `pip` to install Paddle-Lite and optimize model
* Use pip to install Paddle-Lite. The following command uses `pip3.7` .
```shell
pip install paddlelite==2.8
```
**Note**：The version of `paddlelite`'s wheel must match that of inference lib.
* Use `paddle_lite_opt` to optimize inference model, the parameters of `paddle_lite_opt` are as follows:
| --model_dir | Path to the PaddlePaddle model (no-combined) file to be optimized. |
| --model_file | Path to the net structure file of PaddlePaddle model (combined) to be optimized. |
| --param_file | Path to the net weight files of PaddlePaddle model (combined) to be optimized. |
| --optimize_out_type | Type of output model, `protobuf` by default. Supports `protobuf` and `naive_buffer` . Compared with `protobuf`, you can use`naive_buffer` to get a more lightweight serialization/deserialization model. If you need to predict on the mobile-side, please set it to `naive_buffer`. |
| --optimize_out | Path to output model, not needed to add `.nb` suffix. |
| --valid_targets | The executable backend of the model, `arm` by default. Supports one or some of `x86` , `arm` , `opencl` , `npu` , `xpu`. If set more than one, please separate the options by space, and the `opt` tool will choose the best way automatically. If need to support Huawei NPU (DaVinci core carried by Kirin 810/990 SoC), please set it to `npu arm` . |
| --record_tailoring_info | Whether to enable `Cut the Library Files According To the Model` , `false` by default. If need to record kernel and OP infos of optimized model, please set it to `true`. |
In addition, you can run `paddle_lite_opt` to get more detailed information about how to use.
<aname="2.1.2"></a>
### 2.1.2 Compile Paddle-Lite to generate `opt` tool
Optimizing model requires Paddle-Lite's `opt` executable file, which can be obtained by compiling the Paddle-Lite. The steps are as follows:
```shell
# get the Paddle-Lite source code, if have gotten , please skip
After the compilation is complete, the `opt` file is located under `build.opt/lite/api/`.
`opt` tool is used in the same way as `paddle_lite_opt` , please refer to [4.1](#4.1).
<aname="2.1.3"></a>
### 2.1.3 Demo of get the optimized model
Taking the `MobileNetV3_large_x1_0` model of PaddleClas as an example, we will introduce how to use `paddle_lite_opt` to complete the conversion from the pre-trained model to the inference model, and then to the Paddle-Lite optimized model.
When the above code command is completed, there will be ``MobileNetV3_large_x1_0.nb` in the current directory, which is the converted model file.
<a name="2.2"></a>
## 2.2 Run optimized model on Phone
1. Prepare an Android phone with `arm8`. If the compiled inference library and `opt` file are `armv7`, you need an `arm7` phone and modify `ARM_ABI = arm7` in the Makefile.
2. Install the ADB tool on the computer.
* Install ADB for MAC
Recommend use homebrew to install.
```shell
brew cask install android-platform-tools
```
* Install ADB for Linux
```shell
sudo apt update
sudo apt install -y wget adb
```
* Install ADB for windows
If install ADB fo Windows, you need to download from Google's Android platform: [Download Link](https://developer.android.com/studio).
First, make sure the phone is connected to the computer, turn on the `USB debugging` option of the phone, and select the `file transfer` mode. Verify whether ADB is installed successfully as follows:
```shell
$ adb devices
List of devices attached
744be294 device
```
If there is `device` output like the above, it means the installation was successful.
4. Prepare optimized model, inference library files, test image and dictionary file used.
```shell
cd PaddleClas_root_path
cd deploy/lite/
# prepare.sh will put the inference library files, the test image and the dictionary files in demo/cxx/clas
sh prepare.sh /{lite inference library path}/inference_lite_lib.android.armv8
# enter the working directory of lite demo
cd /{lite inference library path}/inference_lite_lib.android.armv8/
cd demo/cxx/clas/
# copy the C++ inference dynamic library file （ie. .so) to the debug folder
The `prepare.sh` take `PaddleClas/deploy/lite/imgs/tabby_cat.jpg` as the test image, and copy it to the `demo/cxx/clas/debug/` directory.
You should put the model that optimized by `paddle_lite_opt` under the `demo/cxx/clas/debug/` directory. In this example, use `MobileNetV3_large_x1_0.nb` model file generated in [2.1.3](#4.3).
The structure of the clas demo is as follows after the above command is completed:
```
demo/cxx/clas/
|-- debug/
| |--MobileNetV3_large_x1_0.nb class model
| |--tabby_cat.jpg test image
| |--imagenet1k_label_list.txt dictionary file
| |--libpaddle_light_api_shared.so C++ .so file
| |--config.txt config file
|-- config.txt config file
|-- image_classfication.cpp source code
|-- Makefile compile file
```
**NOTE**:
* `Imagenet1k_label_list.txt` is the category mapping file of the `ImageNet1k` dataset. If use a custom category, you need to replace the category mapping file.
* `config.txt` contains the hyperparameters, as follows:
```shell
clas_model_file ./MobileNetV3_large_x1_0.nb # path of model file
label_path ./imagenet1k_label_list.txt # path of category mapping file
resize_short_size 256 # the short side length after resize
crop_size 224 # side length used for inference after cropping
visualize 0 # whether to visualize. If you set it to 1, an image file named 'clas_result.png' will be generated in the current directory.
```
5. Run Model on Phone
```shell
# run compile to get the executable file 'clas_system'
make -j
# move the compiled executable file to the debug folder
Q1：If I want to change the model, do I need to go through the all process again?
A1：If you have completed the above steps, you only need to replace the `.nb` model file after replacing the model. At the same time, you may need to modify the path of `.nb` file in the config file and change the category mapping file to be compatible the model .
Q2：How to change the test picture?
A2：Replace the test image under debug folder with the image you want to test，and then repush to the Phone again.
The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment.
The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment.
...
@@ -6,170 +6,137 @@ The model saved during the training process is the checkpoints model, which save
...
@@ -6,170 +6,137 @@ The model saved during the training process is the checkpoints model, which save
Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems.
Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems.
Next, we first introduce how to convert a trained model into an inference model, and then we will introduce mainbody detection, feature extraction based on inference model,
Please refer to the document [install paddle](../installation/install_paddle_en.md) and [install paddleclas](../installation/install_paddleclas_en.md) to prepare the environment.
then we introduce a recognition pipeline consist of mainbody detection, feature extraction and vector search. At last, we introduce classification base on inference model.
-[CONVERT TRAINING MODEL TO INFERENCE MODEL](#CONVERT)
---
-[Convert feature extraction model to inference model](#Convert_feature_extraction)
-[Convert classification model to inference model](#Convert_class)
-[MAINBODY DETECTION MODEL INFERENCE](#DETECTION_MODEL_INFERENCE)
## Contents
-[FEATURE EXTRACTION MODEL INFERENCE](#FEATURE_EXTRACTION_MODEL_INFERENCE)
-[1. Image classification inference](#1)
-[2. Mainbody detection model inference](#2)
-[3. Feature Extraction model inference](#3)
-[4. Concatenation of mainbody detection, feature extraction and vector search](#4)
-[CONCATENATION OF MAINBODY DETECTION, FEATURE EXTRACTION AND VECTOR SEARCH](#CONCATENATION)
-[CLASSIFICATION MODEL INFERENCE](#CLASSIFICATION)
<aname="1"></a>
## 1. Image classification inference
<aname="CONVERT"></a>
First, please refer to the document [export model](./export_model_en.md) to prepare the inference model files. All the command should be run under `deploy` folder of PaddleClas:
## CONVERT TRAINING MODEL TO INFERENCE MODEL
<aname="Convert_feature_extraction"></a>
### Convert feature extraction model to inference model
First please enter the root folder of PaddleClas. Download the product feature extraction model:
The above model is trained on AliProduct with ResNet50_vd as backbone. To convert the trained model into an inference model, just run the following command:
```shell
cd deploy
```
```
# -c Set the training algorithm yml configuration file
# -o Set optional parameters
# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams.
# Global.save_inference_dir Set the address where the converted model will be saved.
For classification model inference, you can execute the following commands:
```
When converting to an inference model, the configuration file used is the same as the configuration file used during training. In addition, you also need to set the `Global.pretrained_model` parameter in the configuration file.
```shell
After the conversion is successful, there are three files in the model save directory:
In the configuration file `configs/inference_cls.yaml`, the following fields are used to configure prediction parameters:
### Convert classification model to inference model
*`Global.infer_imgs`: The path of image to be predicted;
*`Global.inference_model_dir`: The directory of inference model files. There should be contain the model files (`inference.pdmodel` and `inference.pdiparams`);
*`Global.use_tensorrt`: Whether use `TensorRT`, `False` by default;
*`Global.use_gpu`: Whether use GPU, `True` by default;
*`Global.enable_mkldnn`: Whether use `MKL-DNN`, `False` by default. Valid only when `use_gpu` is `False`;
*`Global.use_fp16`: Whether use `FP16`, `False` by default;
*`PreProcess`: To config the preprocessing of image to be predicted;
*`PostProcess`: To config the postprocessing of prediction results;
*`PostProcess.Topk.class_id_map_file`: The path of file mapping label and class id. By default ImageNet1k (`./utils/imagenet1k_label_list.txt`).
Download the pretrained model:
**Notice**:
``` shell script
* If VisionTransformer series models used, such as `DeiT_***_384`, `ViT_***_384`, please notice the size of model input. And you could need to specify the `PreProcess.resize_short=384`, `PreProcess.resize=384`.
First, please refer to the document [export model](./export_model_en.md) to prepare the inference model files. All the command should be run under `deploy` folder of PaddleClas:
The following will introduce the feature extraction model inference. All the command should be run under `deploy` folder of PaddleClas:
```shell
```shell script
cd deploy
cd deploy
```
```
For feature extraction model inference, you can execute the following commands:
For feature extraction model inference, you can execute the following commands:
Paddleclas supports Python WHL package for prediction. At present, WHL package only supports image classification, but does not support subject detection, feature extraction and vector retrieval.
---
## Contents
-[1. Installation](#1)
-[2. Quick Start](#2)
-[3. Definition of Parameters](#3)
-[4. Usage](#4)
-[4.1 View help information](#4.1)
-[4.2 Prediction using inference model provide by PaddleClas](#4.2)
-[4.3 Prediction using local model files](#4.3)
-[4.4 Prediction by batch](#4.4)
-[4.5 Prediction of Internet image](#4.5)
-[4.6 Prediction of `NumPy.array` format image](#4.6)
-[4.7 Save the prediction result(s)](#4.7)
-[4.8 Specify the mapping between class id and label name](#4.8)
<aname="1"></a>
## 1. Installation
## 1. Installation
* installing from pypi
* installing from pypi
...
@@ -15,7 +35,7 @@ python3 setup.py bdist_wheel
...
@@ -15,7 +35,7 @@ python3 setup.py bdist_wheel
pip3 install dist/*
pip3 install dist/*
```
```
<aname="2"></a>
## 2. Quick Start
## 2. Quick Start
* Using the `ResNet50` model provided by PaddleClas, the following image(`'docs/images/whl/demo.jpg'`) as an example.
* Using the `ResNet50` model provided by PaddleClas, the following image(`'docs/images/whl/demo.jpg'`) as an example.
@@ -99,6 +119,7 @@ PaddleClas provides two ways to use:
...
@@ -99,6 +119,7 @@ PaddleClas provides two ways to use:
paddleclas -h
paddleclas -h
```
```
<aname="4.2"></a>
### 4.2 Prediction using inference model provide by PaddleClas
### 4.2 Prediction using inference model provide by PaddleClas
You can use the inference model provided by PaddleClas to predict, and only need to specify `model_name`. In this case, PaddleClas will automatically download files of specified model and save them in the directory `~/.paddleclas/`.
You can use the inference model provided by PaddleClas to predict, and only need to specify `model_name`. In this case, PaddleClas will automatically download files of specified model and save them in the directory `~/.paddleclas/`.
You can use the local model files trained by yourself to predict, and only need to specify `inference_model_dir`. Note that the directory must contain `inference.pdmodel` and `inference.pdiparams`.
You can use the local model files trained by yourself to predict, and only need to specify `inference_model_dir`. Note that the directory must contain `inference.pdmodel` and `inference.pdiparams`.
You can predict the Internet image, only need to specify URL of Internet image by `infer_imgs`. In this case, the image file will be downloaded and saved in the directory `~/.paddleclas/images/`.
You can predict the Internet image, only need to specify URL of Internet image by `infer_imgs`. In this case, the image file will be downloaded and saved in the directory `~/.paddleclas/images/`.
In Python code, you can predict the NumPy.array format image, only need to use the `infer_imgs` to transfer variable of image data. Note that the image data must be 3 channels.
In Python code, you can predict the NumPy.array format image, only need to use the `infer_imgs` to transfer variable of image data. Note that the image data must be 3 channels.
### 4.8 Specify the mapping between class id and label name
### 4.8 Specify the mapping between class id and label name
You can specify the mapping between class id and label name, only need to use `class_id_map_file` to specify the mapping file. PaddleClas uses ImageNet1K's mapping by default.
You can specify the mapping between class id and label name, only need to use `class_id_map_file` to specify the mapping file. PaddleClas uses ImageNet1K's mapping by default.
-[2.(Recommended) Prepare a docker environment](#2)
-[3. Install PaddlePaddle using pip](#3)
-[4. Verify installation](#4)
At present, **PaddleClas** requires **PaddlePaddle** version **>=2.0**. Docker is recomended to run Paddleclas, for more detailed information about docker and nvidia-docker, you can refer to the [tutorial](https://docs.docker.com/get-started/). If you do not want to use docker, you can skip section [2. (Recommended) Prepare a docker environment](#2), and go into section [3. Install PaddlePaddle using pip](#3).
## 1. Environment requirements
- python 3.x
- cuda >= 10.1 (necessary if paddlepaddle-gpu is used)
- cudnn >= 7.6.4 (necessary if paddlepaddle-gpu is used)
- nccl >= 2.1.2 (necessary distributed training/eval is used)
- gcc >= 8.2
**Recomends**:
* When CUDA version is 10.1, the driver version `>= 418.39`;
* When CUDA version is 10.2, the driver version `>= 440.33`;
* For more CUDA versions and specific driver versions, please refer to [link](https://docs.nvidia.com/deploy/cuda-compatibility/index.html).
<aname="2"></a>
## 2. (Recommended) Prepare a docker environment
* Switch to the working directory
```shell
cd /home/Projects
```
* Create docker container
The following commands will create a docker container named ppcls and map the current working directory to the `/paddle' directory in the container.
* The first time you use this docker image, it will be downloaded automatically. Please be patient;
* The above command will create a docker container named ppcls, and there is no need to run the command again when using the container again;
* The parameter `--shm-size=8g` will set the shared memory of the container to 8g. If conditions permit, it is recommended to set this parameter to a larger value, such as `64g`;
* You can also access [DockerHub](https://hub.Docker.com/r/paddlepaddle/paddle/tags/) to obtain the image adapted to your machine;
* Exit / Enter the docker container:
* After entering the docker container, you can exit the current container by pressing `Ctrl + P + Q` without closing the container;
* To re-enter the container, use the following command:
```shell
sudo Docker exec -it ppcls /bin/bash
```
<a name="3"></a>
## 3. Install PaddlePaddle using pip
If you want to use PaddlePaddle on GPU, you can use the following command to install PaddlePaddle.
* If you have already installed CPU version of PaddlePaddle and want to use GPU version now, you should uninstall CPU version of PaddlePaddle and then install GPU version to avoid package confusion.
* You can also compile PaddlePaddle from source code, please refer to [PaddlePaddle Installation tutorial](http://www.paddlepaddle.org.cn/install/quick) to more compilation options.
* Make sure the compiled source code is later than PaddlePaddle2.0.
* Indicate **WITH_DISTRIBUTE=ON** when compiling, Please refer to [Instruction](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/install/Tables.html#id3) for more details.
* When running in docker, in order to ensure that the container has enough shared memory for dataloader acceleration of Paddle, please set the parameter `--shm-size=8g` at creating a docker container, if conditions permit, you can set it to a larger value.
VisualDL, a visualization analysis tool of PaddlePaddle, provides a variety of charts to show the trends of parameters, and visualizes model structures, data samples, histograms of tensors, PR curves , ROC curves and high-dimensional data distributions. It enables users to understand the training process and the model structure more clearly and intuitively so as to optimize models efficiently. For more information, please refer to [VisualDL](https://github.com/PaddlePaddle/VisualDL/).
<aname='2'></a>
## 2. Use VisualDL in PaddleClas
Now PaddleClas support use VisualDL to visualize the changes of learning rate, loss, accuracy in training.
<aname='2.1'></a>
### 2.1 Set config and start training
You only need to set the field `Global.use_visualdl` to `True` in train config:
```yaml
# config.yaml
Global:
...
use_visualdl:True
...
```
PaddleClas will save the VisualDL logs to subdirectory `vdl/` under the output directory specified by `Global.output_dir`. And then you just need to start training normally:
```shell
python3 tools/train.py -c config.yaml
```
<aname='2.2'></a>
### 2.2 Start VisualDL
After starting the training program, you can start the VisualDL service in a new terminal session:
```shell
visualdl --logdir ./output/vdl/
```
In the above command, `--logdir` specify the directory of the VisualDL logs produced in training. VisualDL will traverse and iterate to find the subdirectories of the specified directory to visualize all the experimental results. You can also use the following parameters to set the IP and port number of the VisualDL service:
*`--host`：ip, default is 127.0.0.1
*`--port`：port, default is 8040
More information about the command，please refer to [VisualDL](https://github.com/PaddlePaddle/VisualDL/blob/develop/README.md#2-launch-panel).
Then you can enter the address `127.0.0.1:8840` and view the training process in the browser:
*[2.4 Model Optimization and Speed Evaluation](#2.4)
<aname='1'></a>
## 1. Introduction
[Paddle-Lite](https://github.com/PaddlePaddle/Paddle-Lite) is a set of lightweight inference engine which is fully functional, easy to use and then performs well. Lightweighting is reflected in the use of fewer bits to represent the weight and activation of the neural network, which can greatly reduce the size of the model, solve the problem of limited storage space of the mobile device, and the inference speed is better than other frameworks on the whole.
In [PaddleClas](https://github.com/PaddlePaddle/PaddleClas), we uses Paddle-Lite to [evaluate the performance on the mobile device](../models/Mobile.md), in this section we uses the `MobileNetV1` model trained on the `ImageNet1k` dataset as an example to introduce how to use `Paddle-Lite` to evaluate the model speed on the mobile terminal (evaluated on SD855)
<aname='2'></a>
## 2. Evaluation Steps
<aname='2.1'></a>
### 2.1 Export the Inference Model
* First you should transform the saved model during training to the special model which can be used to inference, the special model can be exported by `tools/export_model.py`, the specific way of transform is as follows.
Finally the `model` and `parmas` can be saved in `inference/MobileNetV1`.
<aname='2.2'></a>
### 2.2 Download Benchmark Binary File
* Use the adb (Android Debug Bridge) tool to connect the Android phone and the PC, then develop and debug. After installing adb and ensuring that the PC and the phone are successfully connected, use the following command to view the ARM version of the phone and select the pre-compiled library based on ARM version.
After the PC and mobile phone are successfully connected, use the following command to start the model evaluation.
```
sh deploy/lite/benchmark/benchmark.sh ./benchmark_bin_v8 ./inference result_armv8.txt true
```
Where `./benchmark_bin_v8` is the path of the benchmark binary file, `./inference` is the path of all the models that need to be evaluated, `result_armv8.txt` is the result file, and the final parameter `true` means that the model will be optimized before evaluation. Eventually, the evaluation result file of `result_armv8.txt` will be saved in the current folder. The specific performances are as follows.
```
PaddleLite Benchmark
Threads=1 Warmup=10 Repeats=30
MobileNetV1 min = 30.89100 max = 30.73600 average = 30.79750
Threads=2 Warmup=10 Repeats=30
MobileNetV1 min = 18.26600 max = 18.14000 average = 18.21637
Threads=4 Warmup=10 Repeats=30
MobileNetV1 min = 10.03200 max = 9.94300 average = 9.97627
```
Here is the model inference speed under different number of threads, the unit is FPS, taking model on one threads as an example, the average speed of MobileNetV1 on SD855 is `30.79750FPS`.
<aname='2.4'></a>
### 2.4 Model Optimization and Speed Evaluation
* In II.III section, we mention that the model will be optimized before evaluation, here you can first optimize the model, and then directly load the optimized model for speed evaluation
* Paddle-Lite
In Paddle-Lite, we provides multiple strategies to automatically optimize the original training model, which contain Quantify, Subgraph fusion, Hybrid scheduling, Kernel optimization and so on. In order to make the optimization more convenient and easy to use, we provide opt tools to automatically complete the optimization steps and output a lightweight, optimal and executable model in Paddle-Lite, which can be downloaded on [Paddle-Lite Model Optimization Page](https://paddle-lite.readthedocs.io/zh/latest/user_guides/model_optimize_tool.html). Here we take `MacOS` as our development environment, download[opt_mac](https://paddlelite-data.bj.bcebos.com/model_optimize_tool/opt_mac) model optimization tools and use the following commands to optimize the model.
```shell
model_file="../MobileNetV1/model"
param_file="../MobileNetV1/params"
opt_models_dir="./opt_models"
mkdir${opt_models_dir}
./opt_mac --model_file=${model_file}\
--param_file=${param_file}\
--valid_targets=arm \
--optimize_out_type=naive_buffer \
--prefer_int8_kernel=false\
--optimize_out=${opt_models_dir}/MobileNetV1
```
Where the `model_file` and `param_file` are exported model file and the file address respectively, after transforming successfully, the `MobileNetV1.nb` will be saved in `opt_models`
Use the benchmark_bin file to load the optimized model for evaluation. The commands are as follows.
[The NVIDIA Data Loading Library](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html)(DALI) is a library for data loading and pre-processing to accelerate deep learning applications. It can build Dataloader of Paddle.
Since the Deep learning relies on a large amount of data in the training stage, these data need to be loaded and preprocessed. These operations are usually executed on the CPU, which limits the further improvement of the training speed, especially when the batch_size is large, which become the bottleneck of speed. DALI can use GPU to accelerate these operations, thereby further improve the training speed.
## Installing DALI
DALI only support Linux x64 and version of CUDA is 10.2 or later.
For more information about installing DALI, please refer to [DALI](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/installation.html).
## Using DALI
Paddleclas supports training with DALI in static graph. Since DALI only supports GPU training, `CUDA_VISIBLE_DEVICES` needs to be set, and DALI needs to occupy GPU memory, so it needs to reserve GPU memory for Dali. To train with DALI, just set the fields in the training config `use_dali = True`, or start the training by the following command:
```shell
# set the GPUs that can be seen
export CUDA_VISIBLE_DEVICES="0"
# set the GPU memory used for neural network training, generally 0.8 or 0.7, and the remaining GPU memory is reserved for DALI
Transfer learning is an important part of machine learning, which is widely used in various fields such as text and images. Here we mainly introduce transfer learning in the field of image classification, which is often called domain transfer, such as migration of the ImageNet classification model to the specified image classification task, such as flower classification.
---
## Contents
*[1. Hyperparameter search](#1)
*[1.1 Grid search](#1.1)
*[1.2 Bayesian search](#1.2)
*[2. Large-scale image classification](#2)
*[3. Reference](#3)
<aname='1'></a>
## 1. Hyperparameter search
ImageNet is the widely used dataset for image classification. A series of empirical hyperparameters have been summarized. High accuracy can be got using the hyperparameters. However, when applied in the specified dataset, the hyperparameters may not be optimal. There are two commonly used hyperparameter search methods that can be used to help us obtain better model hyperparameters.
<aname='1.1'></a>
### 1.1 Grid search
For grid search, which is also called exhaustive search, the optimal value is determined by finding the best solution from all solutions in the search space. The method is simple and effective, but when the search space is large, it takes huge computing resource.
<aname='1.2'></a>
### 1.2 Bayesian search
Bayesian search, which is also called Bayesian optimization, is realized by randomly selecting a group of hyperparameters in the search space. Gaussian process is used to update the hyperparameters, compute their expected mean and variance according to the performance of the previous hyperparameters. The larger the expected mean, the greater the probability of being close to the optimal solution. The larger the expected variance, the greater the uncertainty. Usually, the hyperparameter point with large expected mean is called `exporitation`, and the hyperparameter point with large variance is called `exploration`. Acquisition function is defined to balance the expected mean and variance. The currently selected hyperparameter point is viewed as the optimal position with maximum probability.
According to the above two search schemes, we carry out some experiments based on fixed scheme and two search schemes on 8 open source datasets. As the experimental scheme in [1], we search for 4 hyperparameters, the search space and The experimental results are as follows:
a fixed set of parameter experiments and two search schemes on 8 open source data sets. With reference to the experimental scheme of [1], we search for 4 hyperparameters, the search space and the experimental results are as follows:
It takes 196 times for grid search, and takes 10 times less for Bayesian search. The baseline is trained by using ImageNet1k pretrained model based on ResNet50_vd and fixed scheme. The follow shows the experiments.
- The above experiments verify that Bayesian search only reduces the accuracy by 0% to 0.4% under the condition of reducing the number of searches by about 10 times compared to grid search.
- The search space can be expaned easily using Bayesian search.
<aname='2'></a>
## Large-scale image classification
In practical applications, due to the lack of training data, the classification model trained on the ImageNet1k data set is often used as the pretrained model for other image classification tasks. In order to further help solve practical problems, based on ResNet50_vd, Baidu open sourced a self-developed large-scale classification pretrained model, in which the training data contains 100,000 categories and 43 million pictures. The pretrained model can be downloaded as follows：[**download link**](https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ResNet50_vd_10w_pretrained.pdparams)
We conducted transfer learning experiments on 6 self-collected datasets,
using a set of fixed parameters and a grid search method, in which the number of training rounds was set to 20epochs, the ResNet50_vd model was selected, and the ImageNet pre-training accuracy was 79.12%. The comparison results of the experimental data set parameters and model accuracy are as follows:
| Dataset | Statstics | **Pretrained moel on ImageNet <br />Top-1(fixed)/Top-1(search)** | **Pretrained moel on large-scale dataset<br />Top-1(fixed)/Top-1(search)** |
- The above experiments verified that for fixed parameters, compared with the pretrained model on ImageNet, using the large-scale classification model as a pretrained model can help us improve the model performance on a new dataset in most cases. Parameter search can be further helpful to the model performance.
<aname='3'></a>
## Reference
[1] Kornblith, Simon, Jonathon Shlens, and Quoc V. Le. "Do better imagenet models transfer better?." *Proceedings of the IEEE conference on computer vision and pattern recognition*. 2019.
[2] Kolesnikov, Alexander, et al. "Large Scale Learning of General Visual Representations for Transfer." *arXiv preprint arXiv:1912.11370* (2019).
