readme_en.md 11.1 KB
Newer Older
L
LDOUBLEV 已提交
1

L
LDOUBLEV 已提交
2
# Tutorial of PaddleOCR Mobile deployment
L
LDOUBLEV 已提交
3 4 5

This tutorial will introduce how to use paddle-lite to deploy paddleOCR ultra-lightweight Chinese and English detection models on mobile phones.

蔡舒起 已提交
6
paddle-lite is a lightweight inference engine for PaddlePaddle.
L
LDOUBLEV 已提交
7 8 9 10
It provides efficient inference capabilities for mobile phones and IOTs,
and extensively integrates cross-platform hardware to provide lightweight
deployment solutions for end-side deployment issues.

L
LDOUBLEV 已提交
11
## 1. Preparation
L
LDOUBLEV 已提交
12

L
LDOUBLEV 已提交
13
- Computer (for Compiling Paddle Lite)
L
LDOUBLEV 已提交
14 15
- Mobile phone (arm7 or arm8)

M
Ming 已提交
16
## 2. Build PaddleLite library
L
LDOUBLEV 已提交
17 18 19
1. [Docker](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html#docker)
2. [Linux](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html#linux)
3. [MAC OS](https://paddle-lite.readthedocs.io/zh/latest/source_compile/compile_env.html#mac-os)
L
LDOUBLEV 已提交
20

L
LDOUBLEV 已提交
21
## 3. Download prebuild library for android and ios
L
LDOUBLEV 已提交
22 23 24

|Platform|Prebuild library Download Link|
|-|-|
25 26
|Android|[arm7](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.android.armv7.gcc.c++_shared.with_extra.with_cv.tar.gz) / [arm8](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.android.armv8.gcc.c++_shared.with_extra.with_cv.tar.gz)|
|IOS|[arm7](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.ios.armv7.with_cv.with_extra.with_log.tiny_publish.tar.gz) / [arm8](https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.6.3/inference_lite_lib.ios.armv8.with_cv.with_extra.with_log.tiny_publish.tar.gz)|
L
LDOUBLEV 已提交
27

28
note: The above pre-build inference library is compiled from the PaddleLite `release/2.6.3` branch. For more information about PaddleLite 2.6.3, please refer to [link](https://github.com/PaddlePaddle/Paddle-Lite/releases/tag/v2.6.3).
L
LDOUBLEV 已提交
29 30 31 32 33 34


The structure of the prediction library is as follows:

```
inference_lite_lib.android.armv8/
L
LDOUBLEV 已提交
35 36
|-- cxx                                        C++ prebuild library
|   |-- include                                C++
L
LDOUBLEV 已提交
37 38 39 40 41 42 43
|   |   |-- paddle_api.h
|   |   |-- paddle_image_preprocess.h
|   |   |-- paddle_lite_factory_helper.h
|   |   |-- paddle_place.h
|   |   |-- paddle_use_kernels.h
|   |   |-- paddle_use_ops.h
|   |   `-- paddle_use_passes.h
L
LDOUBLEV 已提交
44 45 46 47
|   `-- lib  
|       |-- libpaddle_api_light_bundled.a             C++ static library
|       `-- libpaddle_light_api_shared.so             C++ dynamic library
|-- java                                     Java predict library
L
LDOUBLEV 已提交
48 49 50 51 52
|   |-- jar
|   |   `-- PaddlePredictor.jar
|   |-- so
|   |   `-- libpaddle_lite_jni.so
|   `-- src
L
LDOUBLEV 已提交
53 54 55
|-- demo                                     C++ and java demo
|   |-- cxx  
|   `-- java  
L
LDOUBLEV 已提交
56 57 58 59 60 61 62
```


## 4. Inference Model Optimization

Paddle Lite provides a variety of strategies to automatically optimize the original training model, including quantization, sub-graph fusion, hybrid scheduling, Kernel optimization and so on. In order to make the optimization process more convenient and easy to use, Paddle Lite provide opt tools to automatically complete the optimization steps and output a lightweight, optimal executable model.

L
opt doc  
LDOUBLEV 已提交
63
If you have prepared the model file ending in `.nb`, you can skip this step.
L
LDOUBLEV 已提交
64

L
opt doc  
LDOUBLEV 已提交
65 66
The following table also provides a series of models that can be deployed on mobile phones to recognize Chinese.
You can directly download the optimized model.
L
LDOUBLEV 已提交
67

L
LDOUBLEV 已提交
68 69
|Version|Introduction|Model size|Detection model|Text Direction model|Recognition model|Paddle Lite branch |
|-|-|-|-|-|-|
L
LDOUBLEV 已提交
70 71 72
|V1.1|extra-lightweight chinese OCR optimized model|8.1M|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_det_opt.nb)|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_cls_opt.nb)|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_rec_opt.nb)|develop|
|[slim] V1.1|extra-lightweight chinese OCR optimized model|3.5M|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_det_prune_opt.nb)|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_cls_quant_opt.nb)|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.1_rec_quant_opt.nb)|develop|
|V1.0|lightweight Chinese OCR optimized model|8.6M|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.0_det_opt.nb)|---|[Download](https://paddleocr.bj.bcebos.com/20-09-22/mobile/lite/ch_ppocr_mobile_v1.0_rec_opt.nb)|develop|
L
LDOUBLEV 已提交
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90

If the model to be deployed is not in the above table, you need to follow the steps below to obtain the optimized model.

```
git clone https://github.com/PaddlePaddle/Paddle-Lite.git
cd Paddle-Lite
git checkout develop
./lite/tools/build.sh build_optimize_tool
```

The `opt` tool can be obtained by compiling Paddle Lite.

After the compilation is complete, the opt file is located under `build.opt/lite/api/`.

The `opt` can optimize the inference model saved by paddle.io.save_inference_model to get the model that the paddlelite API can use.

The usage of opt is as follows:
```
L
LDOUBLEV 已提交
91
# 【Recommend】V1.1 is better than V1.0. steps for convert V1.1 model to nb file are as follows
L
LDOUBLEV 已提交
92 93 94 95 96 97 98
wget  https://paddleocr.bj.bcebos.com/20-09-22/mobile-slim/det/ch_ppocr_mobile_v1.1_det_prune_infer.tar && tar xf  ch_ppocr_mobile_v1.1_det_prune_infer.tar
wget  https://paddleocr.bj.bcebos.com/20-09-22/mobile-slim/rec/ch_ppocr_mobile_v1.1_rec_quant_infer.tar && tar xf  ch_ppocr_mobile_v1.1_rec_quant_infer.tar

./opt --model_file=./ch_ppocr_mobile_v1.1_det_prune_infer/model  --param_file=./ch_ppocr_mobile_v1.1_det_prune_infer/params  --optimize_out=./ch_ppocr_mobile_v1.1_det_prune_opt --valid_targets=arm
./opt --model_file=./ch_ppocr_mobile_v1.1_rec_quant_infer/model  --param_file=./ch_ppocr_mobile_v1.1_rec_quant_infer/params  --optimize_out=./ch_ppocr_mobile_v1.1_rec_quant_opt --valid_targets=arm

# or use V1.0 model
L
LDOUBLEV 已提交
99 100 101 102 103 104 105 106
wget  https://paddleocr.bj.bcebos.com/ch_models/ch_det_mv3_db_infer.tar && tar xf ch_det_mv3_db_infer.tar
wget  https://paddleocr.bj.bcebos.com/ch_models/ch_rec_mv3_crnn_infer.tar && tar xf ch_rec_mv3_crnn_infer.tar

./opt --model_file=./ch_det_mv3_db/model --param_file=./ch_det_mv3_db/params --optimize_out_type=naive_buffer --optimize_out=./ch_det_mv3_db_opt --valid_targets=arm
./opt --model_file=./ch_rec_mv3_crnn/model --param_file=./ch_rec_mv3_crnn/params --optimize_out_type=naive_buffer --optimize_out=./ch_rec_mv3_crnn_opt --valid_targets=arm

```

L
LDOUBLEV 已提交
107
When the above code command is completed, there will be two more files `.nb` in the current directory, which is the converted model file.
L
LDOUBLEV 已提交
108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137

## 5. Run optimized model on Phone

1. Prepare an Android phone with arm8. If the compiled prediction library and opt file are armv7, you need an arm7 phone and modify ARM_ABI = arm7 in the Makefile.

2. Make sure the phone is connected to the computer, open the USB debugging option of the phone, and select the file transfer mode.

3. Install the adb tool on the computer.
    3.1 Install ADB for MAC
    ```
    brew cask install android-platform-tools
    ```
    3.2 Install ADB for Linux
    ```
    sudo apt update
    sudo apt install -y wget adb
    ```
    3.3 Install ADB for windows
    [Download Link](https://developer.android.com/studio)

    Verify whether adb is installed successfully
    ```
    $ adb devices

    List of devices attached
    744be294    device
    ```

    If there is `device` output, it means the installation was successful.

138
4. Prepare optimized models, prediction library files, test images and dictionary files used.
L
LDOUBLEV 已提交
139 140

```
141 142 143 144
 git clone https://github.com/PaddlePaddle/PaddleOCR.git
 cd PaddleOCR/deploy/lite/
 # run prepare.sh
 sh prepare.sh /{lite prediction library path}/inference_lite_lib.android.armv8
L
LDOUBLEV 已提交
145

146 147 148 149 150 151 152 153
 #
 cd /{lite prediction library path}/inference_lite_lib.android.armv8/
 cd demo/cxx/ocr/
 # copy paddle-lite C++ .so file to debug/ directory
 cp ../../../cxx/lib/libpaddle_light_api_shared.so ./debug/

 cd inference_lite_lib.android.armv8/demo/cxx/ocr/
 cp ../../../cxx/lib/libpaddle_light_api_shared.so ./debug/
L
LDOUBLEV 已提交
154 155 156

```

157 158 159
Prepare the test image, taking `PaddleOCR/doc/imgs/11.jpg` as an example, copy the image file to the `demo/cxx/ocr/debug/` folder.
Prepare the model files optimized by the lite opt tool, `ch_det_mv3_db_opt.nb, ch_rec_mv3_crnn_opt.nb`,
and place them under the `demo/cxx/ocr/debug/` folder.
L
LDOUBLEV 已提交
160 161


162 163 164 165
The structure of the OCR demo is as follows after the above command is executed:
```
demo/cxx/ocr/
|-- debug/  
L
LDOUBLEV 已提交
166 167 168
|   |--ch_ppocr_mobile_v1.1_det_prune_opt.nb           Detection model
|   |--ch_ppocr_mobile_v1.1_rec_quant_opt.nb           Recognition model
|   |--ch_ppocr_mobile_cls_quant_opt.nb                Text direction classification model
L
typo  
LDOUBLEV 已提交
169
|   |--11.jpg                           Image for OCR
170 171 172 173 174 175 176 177 178 179 180 181
|   |--ppocr_keys_v1.txt                Dictionary file
|   |--libpaddle_light_api_shared.so    C++ .so file
|   |--config.txt                       Config file
|-- config.txt  
|-- crnn_process.cc  
|-- crnn_process.h
|-- db_post_process.cc  
|-- db_post_process.h
|-- Makefile  
|-- ocr_db_crnn.cc  

```
L
LDOUBLEV 已提交
182

L
LDOUBLEV 已提交
183 184 185 186 187
#### Note:
1. ppocr_keys_v1.txt is a Chinese dictionary file.
If the nb model is used for English recognition or other language recognition, dictionary file should be replaced with a dictionary of the corresponding language.
PaddleOCR provides a variety of dictionaries under ppocr/utils/, including:
```
T
tink2123 已提交
188 189
dict/french_dict.txt     # french
dict/german_dict.txt     # german
L
LDOUBLEV 已提交
190
ic15_dict.txt       # english
T
tink2123 已提交
191 192
dict/japan_dict.txt      # japan
dict/korean_dict.txt     # korean
L
LDOUBLEV 已提交
193 194 195 196 197 198 199 200 201
ppocr_keys_v1.txt   # chinese
```

2. `config.txt`  of the detector and classifier, as shown below:
```
max_side_len  960         #  Limit the maximum image height and width to 960
det_db_thresh  0.3        # Used to filter the binarized image of DB prediction, setting 0.-0.3 has no obvious effect on the result
det_db_box_thresh  0.5    # DDB post-processing filter box threshold, if there is a missing box detected, it can be reduced as appropriate
det_db_unclip_ratio  1.6  # Indicates the compactness of the text box, the smaller the value, the closer the text box to the text
202
use_direction_classify  0  # Whether to use the direction classifier, 0 means not to use, 1 means to use
L
LDOUBLEV 已提交
203 204
```

L
LDOUBLEV 已提交
205 206 207 208 209 210 211 212 213 214 215
5. Run Model on phone

```
cd inference_lite_lib.android.armv8/demo/cxx/ocr/
make -j
mv ocr_db_crnn ./debug/
adb push debug /data/local/tmp/
adb shell
cd /data/local/tmp/debug
export LD_LIBRARY_PATH=/data/local/tmp/debug:$LD_LIBRARY_PATH
# run model
L
LDOUBLEV 已提交
216
 ./ocr_db_crnn ch_ppocr_mobile_v1.1_det_prune_opt.nb  ch_ppocr_mobile_v1.1_rec_quant_opt.nb  ch_ppocr_mobile_cls_quant_opt.nb  ./11.jpg  ppocr_keys_v1.txt
L
LDOUBLEV 已提交
217 218 219 220 221 222 223
```

The outputs are as follows:

<div align="center">
    <img src="../imgs/demo.png" width="600">
</div>
L
add faq  
LDOUBLEV 已提交
224 225 226 227 228 229 230 231 232 233 234 235

## FAQ

Q1: What if I want to change the model, do I need to run it again according to the process?
A1: If you have performed the above steps, you only need to replace the .nb model file to complete the model replacement.

Q2: How to test with another picture?
A2: Replace the .jpg test image under `./debug` with the image you want to test, and run `adb push` to push new image to the phone.

Q3: How to package it into the mobile APP?
A3: This demo aims to provide the core algorithm part that can run OCR on mobile phones.  Further,
PaddleOCR/deploy/android_demo is an example of encapsulating this demo into a mobile app for reference.