Merge branch 'dygraph' into dygraph

deploy/Jetson/readme.md

+English | [简体中文](readme_ch.md)
 
-# Jetson部署PaddleOCR模型
+# Jetson Deployment for PaddleOCR
 
-本节介绍PaddleOCR在Jetson NX、TX2、nano、AGX等系列硬件的部署。
+This section introduces the deployment of PaddleOCR on Jetson NX, TX2, nano, AGX and other series of hardware.
 
 
-## 1. 环境准备
+## 1. Prepare Environment
 
-需要准备一台Jetson开发板，如果需要TensorRT预测，需准备好TensorRT环境，建议使用7.1.3版本的TensorRT；
+You need to prepare a Jetson development hardware. If you need TensorRT, you need to prepare the TensorRT environment. It is recommended to use TensorRT version 7.1.3;
 
-1. Jetson安装PaddlePaddle
+1. Install PaddlePaddle in Jetson
 
-PaddlePaddle下载[链接](https://www.paddlepaddle.org.cn/inference/user_guides/download_lib.html#python)
-请选择适合的您Jetpack版本、cuda版本、trt版本的安装包。
+The PaddlePaddle download [link](https://www.paddlepaddle.org.cn/inference/user_guides/download_lib.html#python)
+Please select the appropriate installation package for your Jetpack version, cuda version, and trt version. Here, we download paddlepaddle_gpu-2.3.0rc0-cp36-cp36m-linux_aarch64.whl.
 
-安装命令：
+Install PaddlePaddle：
 ```shell
-# 安装paddle，以paddlepaddle_gpu-2.3.0rc0-cp36-cp36m-linux_aarch64.whl 为例
 pip3 install -U paddlepaddle_gpu-2.3.0rc0-cp36-cp36m-linux_aarch64.whl
 ```
 
 
-2. 下载PaddleOCR代码并安装依赖
+2. Download PaddleOCR code and install dependencies
 
-首先 clone PaddleOCR 代码：
+Clone the PaddleOCR code:
 ```
 git clone https://github.com/PaddlePaddle/PaddleOCR
 ```
 
-然后，安装依赖：
+and install dependencies：
 ```
 cd PaddleOCR
 pip3 install -r requirements.txt
 ```
 
-*注：jetson硬件CPU较差，依赖安装较慢，请耐心等待*
+*Note: Jetson hardware CPU is poor, dependency installation is slow, please wait patiently*
 
+## 2. Perform prediction
 
-## 2. 执行预测
+Obtain the PPOCR model from the [document](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_en/ppocr_introduction_en.md#6-model-zoo) model library. The following takes the PP-OCRv3 model as an example to introduce the use of the PPOCR model on Jetson:
 
-从[文档](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_ch/ppocr_introduction.md#6-%E6%A8%A1%E5%9E%8B%E5%BA%93) 模型库中获取PPOCR模型，下面以PP-OCRv3模型为例，介绍在PPOCR模型在jetson上的使用方式：
-
-下载并解压PP-OCRv3模型
+Download and unzip the PP-OCRv3 models.
 ```
 wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar
 wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar
@@ -48,38 +47,38 @@ tar xf ch_PP-OCRv3_det_infer.tar
 tar xf ch_PP-OCRv3_rec_infer.tar
 ```
 
-执行文本检测预测：
+The text detection inference:
 ```
 cd PaddleOCR
 python3 tools/infer/predict_det.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/  --image_dir=./doc/imgs/french_0.jpg  --use_gpu=True
 ```
 
-执行命令后在终端会打印出预测的信息，并在 `./inference_results/` 下保存可视化结果。
+After executing the command, the predicted information will be printed out in the terminal, and the visualization results will be saved in the `./inference_results/` directory.
 ![](./images/det_res_french_0.jpg)
 
 
-执行文本识别预测：
+The text recognition inference:
 ```
 python3 tools/infer/predict_det.py --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/  --image_dir=./doc/imgs_words/en/word_2.png  --use_gpu=True --rec_image_shape="3,48,320"
 ```
 
-执行命令后在终端会打印出预测的信息，输出如下：
+After executing the command, the predicted information will be printed on the terminal, and the output is as follows:
 ```
 [2022/04/28 15:41:45] root INFO: Predicts of ./doc/imgs_words/en/word_2.png:('yourself', 0.98084533)
 ```
 
-执行文本检测+文本识别串联预测：
+The text  detection and text recognition inference:
 
 ```
-python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/ --use_gpu=True --rec_image_shape="3,48,320"
+python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/00057937.jpg --use_gpu=True --rec_image_shape="3,48,320"
 ```
 
-执行命令后在终端会打印出预测的信息，并在 `./inference_results/` 下保存可视化结果。
+After executing the command, the predicted information will be printed out in the terminal, and the visualization results will be saved in the `./inference_results/` directory.
 ![](./images/00057937.jpg)
 
-开启TRT预测只需要在以上命令基础上设置`--use_tensorrt=True`即可：
+To enable TRT prediction, you only need to set `--use_tensorrt=True` on the basis of the above command:
 ```
-python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/00057937.jpg --use_gpu=True --use_tensorrt=True --rec_image_shape="3,48,320"
+python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/  --rec_image_shape="3,48,320" --use_gpu=True --use_tensorrt=True
 ```
 
-更多ppocr模型预测请参考[文档](../../doc/doc_ch/models_list.md)
+For more ppocr model predictions, please refer to[document](../../doc/doc_en/models_list_en.md)

deploy/Jetson/readme_en.md → deploy/Jetson/readme_ch.md

+[English](readme.md) | 简体中文
 
-# Jetson Deployment for PaddleOCR
+# Jetson部署PaddleOCR模型
 
-This section introduces the deployment of PaddleOCR on Jetson NX, TX2, nano, AGX and other series of hardware.
+本节介绍PaddleOCR在Jetson NX、TX2、nano、AGX等系列硬件的部署。
 
 
-## 1. Prepare Environment
+## 1. 环境准备
 
-You need to prepare a Jetson development hardware. If you need TensorRT, you need to prepare the TensorRT environment. It is recommended to use TensorRT version 7.1.3;
+需要准备一台Jetson开发板，如果需要TensorRT预测，需准备好TensorRT环境，建议使用7.1.3版本的TensorRT；
 
-1. Install PaddlePaddle in Jetson
+1. Jetson安装PaddlePaddle
 
-The PaddlePaddle download [link](https://www.paddlepaddle.org.cn/inference/user_guides/download_lib.html#python)
-Please select the appropriate installation package for your Jetpack version, cuda version, and trt version. Here, we download paddlepaddle_gpu-2.3.0rc0-cp36-cp36m-linux_aarch64.whl.
+PaddlePaddle下载[链接](https://www.paddlepaddle.org.cn/inference/user_guides/download_lib.html#python)
+请选择适合的您Jetpack版本、cuda版本、trt版本的安装包。
 
-Install PaddlePaddle：
+安装命令：
 ```shell
+# 安装paddle，以paddlepaddle_gpu-2.3.0rc0-cp36-cp36m-linux_aarch64.whl 为例
 pip3 install -U paddlepaddle_gpu-2.3.0rc0-cp36-cp36m-linux_aarch64.whl
 ```
 
 
-2. Download PaddleOCR code and install dependencies
+2. 下载PaddleOCR代码并安装依赖
 
-Clone the PaddleOCR code:
+首先 clone PaddleOCR 代码：
 ```
 git clone https://github.com/PaddlePaddle/PaddleOCR
 ```
 
-and install dependencies：
+然后，安装依赖：
 ```
 cd PaddleOCR
 pip3 install -r requirements.txt
 ```
 
-*Note: Jetson hardware CPU is poor, dependency installation is slow, please wait patiently*
+*注：jetson硬件CPU较差，依赖安装较慢，请耐心等待*
 
-## 2. Perform prediction
 
-Obtain the PPOCR model from the [document](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_en/ppocr_introduction_en.md#6-model-zoo) model library. The following takes the PP-OCRv3 model as an example to introduce the use of the PPOCR model on Jetson:
+## 2. 执行预测
 
-Download and unzip the PP-OCRv3 models.
+从[文档](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_ch/ppocr_introduction.md#6-%E6%A8%A1%E5%9E%8B%E5%BA%93) 模型库中获取PPOCR模型，下面以PP-OCRv3模型为例，介绍在PPOCR模型在jetson上的使用方式：
+
+下载并解压PP-OCRv3模型
 ```
 wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar
 wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar
@@ -46,38 +49,38 @@ tar xf ch_PP-OCRv3_det_infer.tar
 tar xf ch_PP-OCRv3_rec_infer.tar
 ```
 
-The text detection inference:
+执行文本检测预测：
 ```
 cd PaddleOCR
 python3 tools/infer/predict_det.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/  --image_dir=./doc/imgs/french_0.jpg  --use_gpu=True
 ```
 
-After executing the command, the predicted information will be printed out in the terminal, and the visualization results will be saved in the `./inference_results/` directory.
+执行命令后在终端会打印出预测的信息，并在 `./inference_results/` 下保存可视化结果。
 ![](./images/det_res_french_0.jpg)
 
 
-The text recognition inference:
+执行文本识别预测：
 ```
 python3 tools/infer/predict_det.py --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/  --image_dir=./doc/imgs_words/en/word_2.png  --use_gpu=True --rec_image_shape="3,48,320"
 ```
 
-After executing the command, the predicted information will be printed on the terminal, and the output is as follows:
+执行命令后在终端会打印出预测的信息，输出如下：
 ```
 [2022/04/28 15:41:45] root INFO: Predicts of ./doc/imgs_words/en/word_2.png:('yourself', 0.98084533)
 ```
 
-The text  detection and text recognition inference:
+执行文本检测+文本识别串联预测：
 
 ```
-python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/00057937.jpg --use_gpu=True --rec_image_shape="3,48,320"
+python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/ --use_gpu=True --rec_image_shape="3,48,320"
 ```
 
-After executing the command, the predicted information will be printed out in the terminal, and the visualization results will be saved in the `./inference_results/` directory.
+执行命令后在终端会打印出预测的信息，并在 `./inference_results/` 下保存可视化结果。
 ![](./images/00057937.jpg)
 
-To enable TRT prediction, you only need to set `--use_tensorrt=True` on the basis of the above command:
+开启TRT预测只需要在以上命令基础上设置`--use_tensorrt=True`即可：
 ```
-python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/  --rec_image_shape="3,48,320" --use_gpu=True --use_tensorrt=True
+python3 tools/infer/predict_system.py --det_model_dir=./inference/ch_PP-OCRv2_det_infer/ --rec_model_dir=./inference/ch_PP-OCRv2_rec_infer/ --image_dir=./doc/imgs/00057937.jpg --use_gpu=True --use_tensorrt=True --rec_image_shape="3,48,320"
 ```
 
-For more ppocr model predictions, please refer to[document](../../doc/doc_en/models_list_en.md)
+更多ppocr模型预测请参考[文档](../../doc/doc_ch/models_list.md)

deploy/README.md

@@ -25,7 +25,7 @@ PP-OCR has supported muti deployment schemes. Click the link to get the specific
 - [Serving (Python/C++)](./pdserving/README.md)
 - [Paddle-Lite (ARM CPU/OpenCL ARM GPU)](./lite/readme.md)
 - [Paddle.js](./paddlejs/README.md)
-- [Jetson Inference]()
+- [Jetson Inference](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/deploy/Jetson/readme.md)
 - [Paddle2ONNX](./paddle2onnx/readme.md)
 
 If you need the deployment tutorial of academic algorithm models other than PP-OCR, please directly enter the main page of corresponding algorithms, [entrance](../doc/doc_en/algorithm_overview_en.md)。

deploy/README_ch.md

@@ -25,7 +25,7 @@ PP-OCR模型已打通多种场景部署方案，点击链接获取具体的使
 - [Serving 服务化部署（Python/C++）](./pdserving/README_CN.md)
 - [Paddle-Lite 端侧部署（ARM CPU/OpenCL ARM GPU）](./lite/readme_ch.md)
 - [Paddle.js 部署](./paddlejs/README_ch.md)
-- [Jetson 推理]()
+- [Jetson 推理](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/deploy/Jetson/readme_ch.md)
 - [Paddle2ONNX 推理](./paddle2onnx/readme_ch.md)
 
-需要PP-OCR以外的学术算法模型的推理部署，请直接进入相应算法主页面，[入口](../doc/doc_ch/algorithm_overview.md)。
+需要PP-OCR以外的学术算法模型的推理部署，请直接进入相应算法主页面，[入口](../doc/doc_ch/algorithm_overview.md)。
\ No newline at end of file

deploy/hubserving/ocr_det/params.py

@@ -26,7 +26,7 @@ def read_params():
 
     #params for text detector
     cfg.det_algorithm = "DB"
-    cfg.det_model_dir = "./inference/ch_PP-OCRv2_det_infer/"
+    cfg.det_model_dir = "./inference/ch_PP-OCRv3_det_infer/"
     cfg.det_limit_side_len = 960
     cfg.det_limit_type = 'max'
 

deploy/hubserving/ocr_rec/params.py

@@ -26,9 +26,9 @@ def read_params():
 
     #params for text recognizer
     cfg.rec_algorithm = "CRNN"
-    cfg.rec_model_dir = "./inference/ch_PP-OCRv2_rec_infer/"
+    cfg.rec_model_dir = "./inference/ch_PP-OCRv3_rec_infer/"
 
-    cfg.rec_image_shape = "3, 32, 320"
+    cfg.rec_image_shape = "3, 48, 320"
     cfg.rec_batch_num = 6
     cfg.max_text_length = 25
 

deploy/hubserving/ocr_system/params.py

@@ -26,7 +26,7 @@ def read_params():
 
     #params for text detector
     cfg.det_algorithm = "DB"
-    cfg.det_model_dir = "./inference/ch_PP-OCRv2_det_infer/"
+    cfg.det_model_dir = "./inference/ch_PP-OCRv3_det_infer/"
     cfg.det_limit_side_len = 960
     cfg.det_limit_type = 'max'
 
@@ -44,9 +44,9 @@ def read_params():
 
     #params for text recognizer
     cfg.rec_algorithm = "CRNN"
-    cfg.rec_model_dir = "./inference/ch_PP-OCRv2_rec_infer/"
+    cfg.rec_model_dir = "./inference/ch_PP-OCRv3_rec_infer/"
 
-    cfg.rec_image_shape = "3, 32, 320"
+    cfg.rec_image_shape = "3, 48, 320"
     cfg.rec_batch_num = 6
     cfg.max_text_length = 25
 

deploy/hubserving/readme.md

@@ -41,6 +41,7 @@ deploy/hubserving/ocr_system/
 ```
 ## 1. 近期更新
 
+* 2022.05.05 新增PP-OCRv3检测和识别模型。
 * 2022.03.30 新增PP-Structure和表格识别两种服务。
 
 ## 2. 快速启动服务
@@ -53,10 +54,10 @@ pip3 install paddlehub==2.1.0 --upgrade -i https://mirror.baidu.com/pypi/simple
 ```
 
 ### 2.2 下载推理模型
-安装服务模块前，需要准备推理模型并放到正确路径。默认使用的是PP-OCRv2模型，默认模型路径为：
+安装服务模块前，需要准备推理模型并放到正确路径。默认使用的是PP-OCRv3模型，默认模型路径为：
 ```
-检测模型：./inference/ch_PP-OCRv2_det_infer/
-识别模型：./inference/ch_PP-OCRv2_rec_infer/
+检测模型：./inference/ch_PP-OCRv3_det_infer/
+识别模型：./inference/ch_PP-OCRv3_rec_infer/
 方向分类器：./inference/ch_ppocr_mobile_v2.0_cls_infer/
 表格结构识别模型：./inference/en_ppocr_mobile_v2.0_table_structure_infer/
 ```  
@@ -223,6 +224,7 @@ hub serving start -c deploy/hubserving/ocr_system/config.json
 
 - 2、 到相应的`module.py`和`params.py`等文件中根据实际需求修改代码。  
 例如，如果需要替换部署服务所用模型，则需要到`params.py`中修改模型路径参数`det_model_dir`和`rec_model_dir`，如果需要关闭文本方向分类器，则将参数`use_angle_cls`置为`False`，当然，同时可能还需要修改其他相关参数，请根据实际情况修改调试。 **强烈建议修改后先直接运行`module.py`调试，能正确运行预测后再启动服务测试。**
+**注意** PPOCR-v3识别模型使用的图片输入shape为`3,48,320`,因此需要修改`params.py`中的`cfg.rec_image_shape = "3, 48, 320"`，如果不使用PPOCR-v3识别模型，则无需修改该参数。
 
 - 3、 卸载旧服务包  
 ```hub uninstall ocr_system```  

deploy/hubserving/readme_en.md

@@ -41,6 +41,7 @@ deploy/hubserving/ocr_system/
 ```
 ## 1. Update
 
+* 2022.05.05 add PP-OCRv3 text detection and recognition models.
 * 2022.03.30 add PP-Structure and table recognition services。
 
 
@@ -55,10 +56,10 @@ pip3 install paddlehub==2.1.0 --upgrade -i https://pypi.tuna.tsinghua.edu.cn/sim
 ```
 
 ### 2.2 Download inference model
-Before installing the service module, you need to prepare the inference model and put it in the correct path. By default, the PP-OCRv2 models are used, and the default model path is:  
+Before installing the service module, you need to prepare the inference model and put it in the correct path. By default, the PP-OCRv3 models are used, and the default model path is:  
 ```
-text detection model: ./inference/ch_PP-OCRv2_det_infer/
-text recognition model: ./inference/ch_PP-OCRv2_rec_infer/
+text detection model: ./inference/ch_PP-OCRv3_det_infer/
+text recognition model: ./inference/ch_PP-OCRv3_rec_infer/
 text angle classifier: ./inference/ch_ppocr_mobile_v2.0_cls_infer/
 tanle recognition: ./inference/en_ppocr_mobile_v2.0_table_structure_infer/
 ```  
@@ -233,6 +234,7 @@ hub serving stop --port/-p XXXX
 ```
 - 2. Modify the code in the corresponding files, like `module.py` and `params.py`, according to the actual needs.  
 For example, if you need to replace the model used by the deployed service, you need to modify model path parameters `det_model_dir` and `rec_model_dir` in `params.py`. If you want to turn off the text direction classifier, set the parameter `use_angle_cls` to `False`. Of course, other related parameters may need to be modified at the same time. Please modify and debug according to the actual situation. It is suggested to run `module.py` directly for debugging after modification before starting the service test.  
+**Note** The image input shape used by the PPOCR-v3 recognition model is `3, 48, 320`, so you need to modify `cfg.rec_image_shape = "3, 48, 320"` in `params.py`, if you do not use the PPOCR-v3 recognition model, then there is no need to modify this parameter.
 - 3. Uninstall old service module
 ```shell
 hub uninstall ocr_system

doc/doc_ch/PP-OCRv3_introduction.md

 [English](../doc_en/PP-OCRv3_introduction_en.md) | 简体中文
 
-# PP-OCR
+# PP-OCRv3
 
 - [1. 简介](#1)
-- [2. 特性](#2)
-- [3. benchmark](#3)
+- [2. 检测优化](#2)
+- [3. 识别优化](#3)
+- [4. 端到端评估](#4)
 
 
 
 <a name="1"></a>
 ## 1. 简介
 
-PP-OCR是PaddleOCR自研的实用的超轻量OCR系统。在实现[前沿算法](algorithm.md)的基础上，考虑精度与速度的平衡，进行**模型瘦身**和**深度优化**，使其尽可能满足产业落地需求。
+PP-OCRv3在PP-OCRv2的基础上进一步升级。检测模型仍然基于DB算法，优化策略采用了带残差注意力机制的FPN结构RSEFPN、增大感受野的PAN结构LKPAN、基于DML训练的更优的教师模型；识别模型将base模型从CRNN替换成了IJCAI 2022论文[SVTR]()，并采用SVTR轻量化、带指导训练CTC、数据增广策略RecConAug、自监督训练的更好的预训练模型、无标签数据的使用进行模型加速和效果提升。更多细节请参考PP-OCRv3[技术报告](./PP-OCRv3_introduction.md)。
 
-#### PP-OCR
-
-PP-OCR是一个两阶段的OCR系统，其中文本检测算法选用[DB](algorithm_det_db.md)，文本识别算法选用[CRNN](algorithm_rec_crnn.md)，并在检测和识别模块之间添加[文本方向分类器](angle_class.md)，以应对不同方向的文本识别。
-
-PP-OCRv2系统pipeline如下：
+PP-OCRv3系统pipeline如下：
 
 <div align="center">
-    <img src="../ppocrv2_framework.jpg" width="800">
+    <img src="../ppocrv3_framework.png" width="800">
 </div>
 
-
-PP-OCR系统在持续迭代优化，目前已发布PP-OCR、PP-OCRv2、PP-OCRv3三个版本：
-
-PP-OCR从骨干网络选择和调整、预测头部的设计、数据增强、学习率变换策略、正则化参数选择、预训练模型使用以及模型自动裁剪量化8个方面，采用19个有效策略，对各个模块的模型进行效果调优和瘦身(如绿框所示)，最终得到整体大小为3.5M的超轻量中英文OCR和2.8M的英文数字OCR。更多细节请参考PP-OCR技术方案 https://arxiv.org/abs/2009.09941
-
-
-## PP-OCRv3策略简介
-
-
-### PP-OCRv3文本检测模型优化策略
+<a name="2"></a>
+## 2. 检测优化
 
 PP-OCRv3采用PP-OCRv2的[CML](https://arxiv.org/pdf/2109.03144.pdf)蒸馏策略，在蒸馏的student模型、teacher模型精度提升，CML蒸馏策略上分别做了优化。
 
@@ -78,17 +67,64 @@ LKPAN(Large Kernel PAN)是一个具有更大感受野的轻量级[PAN](https://a
 
 
 
-<a name="2"></a>
-## 2. 特性
+<a name="3"></a>
+## 3. 识别优化
 
-- 超轻量PP-OCRv2系列：检测（3.1M）+ 方向分类器（1.4M）+ 识别（8.5M）= 13.0M
-- 超轻量PP-OCR mobile移动端系列：检测（3.0M）+方向分类器（1.4M）+ 识别（5.0M）= 9.4M
-- 通用PP-OCR server系列：检测（47.1M）+方向分类器（1.4M）+ 识别（94.9M）= 143.4M
-- 支持中英文数字组合识别、竖排文本识别、长文本识别
-- 支持多语言识别：韩语、日语、德语、法语等约80种语言
+[SVTR](https://arxiv.org/abs/2205.00159) 证明了强大的单视觉模型（无需序列模型）即可高效准确完成文本识别任务，在中英文数据上均有优秀的表现。经过实验验证，SVTR_Tiny在自建的 [中文数据集上](https://arxiv.org/abs/2109.03144) ，识别精度可以提升10.7%，网络结构如下所示：
 
+<img src="../ppocr_v3/svtr_tiny.jpg" width=800>
 
-<a name="3"></a>
-## 3. benchmark
+由于 MKLDNN 加速库支持的模型结构有限，SVTR 在CPU+MKLDNN上相比PP-OCRv2慢了10倍。
+
+PP-OCRv3 期望在提升模型精度的同时，不带来额外的推理耗时。通过分析发现，SVTR_Tiny结构的主要耗时模块为Mixing Block，因此我们对 SVTR_Tiny 的结构进行了一系列优化（详细速度数据请参考下方消融实验表格）:
+
+1. 将SVTR网络前半部分替换为PP-LCNet的前三个stage，保留4个 Global Mixing Block ，精度为76%，加速69%，网络结构如下所示：
+<img src="../ppocr_v3/svtr_g4.png" width=800>
+2. 将4个 Global Attenntion Block 减小到2个，精度为72.9%，加速69%，网络结构如下所示：
+<img src="../ppocr_v3/svtr_g2.png" width=800>
+3. 实验发现 Global Attention 的预测速度与输入其特征的shape有关，因此后移Global Mixing Block的位置到池化层之后，精度下降为71.9%，速度超越 CNN-base 的PP-OCRv2 22%，网络结构如下所示：
+<img src="../ppocr_v3/ppocr_v3.png" width=800>
+
+为了提升模型精度同时不引入额外推理成本，PP-OCRv3参考GTC策略，使用Attention监督CTC训练，预测时完全去除Attention模块，在推理阶段不增加任何耗时, 精度提升3.8%，训练流程如下所示：
+<img src="../ppocr_v3/GTC.png" width=800>
+
+在训练策略方面，PP-OCRv3参考 [SSL](https://github.com/ku21fan/STR-Fewer-Labels) 设计了文本方向任务，训练了适用于文本识别的预训练模型，加速模型收敛过程，精度提升了0.6%; 使用UDML蒸馏策略，进一步提升精度1.5%，训练流程所示：
+
+<img src="../ppocr_v3/SSL.png" width="300"> <img src="../ppocr_v3/UDML.png" width="500">
+
+
+数据增强方面：
+
+1. 基于 [ConCLR](https://www.cse.cuhk.edu.hk/~byu/papers/C139-AAAI2022-ConCLR.pdf) 中的ConAug方法，设计了 RecConAug 数据增强方法，增强数据多样性，精度提升0.5%，增强可视化效果如下所示：
+<img src="../ppocr_v3/recconaug.png" width=800>
+
+2. 使用训练好的 SVTR_large 预测 120W 的 lsvt 无标注数据，取出其中得分大于0.95的数据，共得到81W识别数据加入到PP-OCRv3的训练数据中，精度提升1%。
+
+总体来讲PP-OCRv3识别从网络结构、训练策略、数据增强三个方向做了进一步优化:
+
+- 网络结构上：考虑[SVTR](https://arxiv.org/abs/2205.00159) 在中英文效果上的优越性，采用SVTR_Tiny作为base，选取Global Mixing Block和卷积组合提取特征，并将Global Mixing Block位置后移进行加速; 参考 [GTC](https://arxiv.org/pdf/2002.01276.pdf) 策略，使用注意力机制模块指导CTC训练，定位和识别字符，提升不规则文本的识别精度。
+- 训练策略上：参考 [SSL](https://github.com/ku21fan/STR-Fewer-Labels) 设计了方向分类前序任务，获取更优预训练模型，加速模型收敛过程，提升精度; 使用UDML蒸馏策略、监督attention、ctc两个分支得到更优模型。
+- 数据增强上：基于 [ConCLR](https://www.cse.cuhk.edu.hk/~byu/papers/C139-AAAI2022-ConCLR.pdf) 中的ConAug方法，改进得到 RecConAug 数据增广方法，支持随机结合任意多张图片，提升训练数据的上下文信息丰富度，增强模型鲁棒性；使用 SVTR_large 预测无标签数据，向训练集中补充81w高质量真实数据。
+
+基于上述策略，PP-OCRv3识别模型相比PP-OCRv2，在速度可比的情况下，精度进一步提升4.5%。 具体消融实验如下所示：
+
+实验细节：
+
+| id | 策略 |  模型大小 | 精度 | 速度（cpu + mkldnn)|
+|-----|-----|--------|----| --- |
+| 01 | PP-OCRv2 | 8M | 69.3% | 8.54ms |
+| 02 | SVTR_Tiny | 21M | 80.1% | 97ms |
+| 03 | LCNet_SVTR_G4 | 9.2M | 76% | 30ms |
+| 04 | LCNet_SVTR_G2 | 13M | 72.98% | 9.37ms |
+| 05 | PP-OCRv3 | 12M | 71.9% | 6.6ms |
+| 06 | + large input_shape | 12M | 73.98% | 7.6ms |
+| 06 | + GTC | 12M | 75.8% | 7.6ms |
+| 07 | + RecConAug | 12M | 76.3% | 7.6ms |
+| 08 | + SSL pretrain | 12M | 76.9% | 7.6ms |
+| 09 | + UDML | 12M | 78.4% | 7.6ms |
+| 10 | + unlabeled data | 12M | 79.4% | 7.6ms |
+
+注： 测试速度时，实验01-05输入图片尺寸均为(3,32,320)，06-10输入图片尺寸均为(3,48,320)
 
-关于PP-OCR系列模型之间的性能对比，请查看[benchmark](./benchmark.md)文档。
+<a name="4"></a>
+## 4. 端到端评估

doc/doc_ch/ppocr_introduction.md

@@ -34,11 +34,13 @@ PP-OCR从骨干网络选择和调整、预测头部的设计、数据增强、
 
 #### PP-OCRv2
 
-PP-OCRv2在PP-OCR的基础上，进一步在5个方面重点优化，检测模型采用CML协同互学习知识蒸馏策略和CopyPaste数据增广策略；识别模型采用LCNet轻量级骨干网络、UDML 改进知识蒸馏策略和[Enhanced CTC loss](./doc/doc_ch/enhanced_ctc_loss.md)损失函数改进（如上图红框所示），进一步在推理速度和预测效果上取得明显提升。更多细节请参考PP-OCRv2[技术报告](https://arxiv.org/abs/2109.03144)。
+PP-OCRv2在PP-OCR的基础上，进一步在5个方面重点优化，检测模型采用CML协同互学习知识蒸馏策略和CopyPaste数据增广策略；识别模型采用LCNet轻量级骨干网络、UDML 改进知识蒸馏策略和[Enhanced CTC loss](./enhanced_ctc_loss.md)损失函数改进（如上图红框所示），进一步在推理速度和预测效果上取得明显提升。更多细节请参考PP-OCRv2[技术报告](https://arxiv.org/abs/2109.03144)。
 
 #### PP-OCRv3
 
-PP-OCRv3在PP-OCRv2的基础上进一步升级。
+
+PP-OCRv3在PP-OCRv2的基础上进一步升级。检测模型仍然基于DB算法，优化策略采用了带残差注意力机制的FPN结构RSEFPN、增大感受野的PAN结构LKPAN、基于DML训练的更优的教师模型；识别模型将base模型从CRNN替换成了IJCAI 2022论文[SVTR](https://arxiv.org/abs/2205.00159)，并采用SVTR轻量化、带指导训练CTC、数据增广策略RecConAug、自监督训练的更好的预训练模型、无标签数据的使用进行模型加速和效果提升。更多细节请参考PP-OCRv3[技术报告](./PP-OCRv3_introduction.md)。
+
 PP-OCRv3文本检测从网络结构、蒸馏训练策略两个方向做了进一步优化:
 - 网络结构改进：提出两种改进后的FPN网络结构，RSEFPN，LKPAN，分别从channel attention、更大感受野的角度优化FPN中的特征，优化FPN提取的特征。
 - 蒸馏训练策略：首先，以resnet50作为backbone，改进后的LKPAN网络结构作为FPN，使用DML自蒸馏策略得到精度更高的teacher模型；然后，student模型FPN部分采用RSEFPN，采用PPOCRV2提出的CML蒸馏方法蒸馏，在训练过程中，动态调整CML蒸馏teacher loss的占比。
@@ -73,9 +75,16 @@ PP-OCRv3识别从网络结构、训练策略、数据增强三个方向做了进
 | 10 | + unlabeled data | 12M | 79.4% | 19ms |
 
 
+PP-OCRv3系统pipeline如下：
+
+<div align="center">
+    <img src="../ppocrv3_framework.png" width="800">
+</div>
+
 <a name="2"></a>
 ## 2. 特性
 
+- 超轻量PP-OCRv3系列：检测（3.6M）+ 方向分类器（1.4M）+ 识别（12M）= 17.0M
 - 超轻量PP-OCRv2系列：检测（3.1M）+ 方向分类器（1.4M）+ 识别（8.5M）= 13.0M
 - 超轻量PP-OCR mobile移动端系列：检测（3.0M）+方向分类器（1.4M）+ 识别（5.0M）= 9.4M
 - 通用PP-OCR server系列：检测（47.1M）+方向分类器（1.4M）+ 识别（94.9M）= 143.4M

doc/doc_en/PP-OCRv3_introduction_en.md

+English | [简体中文](../doc_ch/PP-OCRv3_introduction.md)

doc/doc_en/ppocr_introduction_en.md

@@ -32,24 +32,18 @@ PP-OCR system is in continuous optimization. At present, PP-OCR and PP-OCRv2 hav
 
 [2] On the basis of PP-OCR, PP-OCRv2 is further optimized in five aspects. The detection model adopts CML(Collaborative Mutual Learning) knowledge distillation strategy and CopyPaste data expansion strategy. The recognition model adopts LCNet lightweight backbone network, U-DML knowledge distillation strategy and enhanced CTC loss function improvement (as shown in the red box above), which further improves the inference speed and prediction effect. For more details, please refer to the technical report of PP-OCRv2 (https://arxiv.org/abs/2109.03144).
 
-[3] PP-OCRv3 is further upgraded on the basis of PP-OCRv2.
-PP-OCRv3 text detection has been further optimized from the two directions of network structure and distillation training strategy:
-- Network structure improvement: Two improved FPN network structures, RSEFPN and LKPAN, are proposed to optimize the features in the FPN from the perspective of channel attention and a larger receptive field, and optimize the features extracted by the FPN.
-- Distillation training strategy: First, use resnet50 as the backbone, the improved LKPAN network structure as the FPN, and use the DML self-distillation strategy to obtain a teacher model with higher accuracy; then, the FPN part of the student model adopts RSEFPN, and adopts the CML distillation method proposed by PPOCRV2, during the training process, dynamically adjust the proportion of CML distillation teacher loss.
+[3] PP-OCRv3 is further upgraded on the basis of PP-OCRv2. The detection model is still based on DB algorithm, and the optimization strategies include a newly proposed FPN structure with residual attention mechanism named with RSEFPN, a PAN structure with enlarged receptive field named with LKPAN, and better teacher model based on DML training; The recognition model replaces the base model from CRNN with IJCAI 2022 paper [SVTR](https://arxiv.org/abs/2205.00159), and adopts lightweight SVTR, guided training of CTC, data augmentation strategy RecConAug, better pre-trained model by self-supervised training, and the use of unlabeled data to accelerate the model and improve the effect. For more details, please refer to PP-OCRv3 [technical report](./PP-OCRv3_introduction_en.md).
 
-|Index|Method|Model SIze|Hmean|CPU inference time|
-|-|-|-|-|-|
-|0|ppocr_mobile|3M|81.3|117ms|
-|1|PPOCRV2|3M|83.3|117ms|
-|2|teacher DML|124M|86.0|-|
-|3|1 + 2 + RESFPN|3.6M|85.4|124ms|
-|4|1 + 2 + LKPAN|4.6M|86.0|156ms|
+PP-OCRv3 pipeline is as follows:
 
-*note: CPU inference time refers to the average inference time on an Intel Gold 6148CPU with mkldnn enabled.*
+<div align="center">
+    <img src="../ppocrv3_framework.png" width="800">
+</div>
 
 <a name="2"></a>
 ## 2. Features
 
+- Ultra lightweight PP-OCRv3 series models: detection (3.6M) + direction classifier (1.4M) + recognition 12M) = 17.0M
 - Ultra lightweight PP-OCRv2 series models: detection (3.1M) + direction classifier (1.4M) + recognition 8.5M) = 13.0M
 - Ultra lightweight PP-OCR mobile series models: detection (3.0M) + direction classifier (1.4M) + recognition (5.0M) = 9.4M
 - General PP-OCR server series models: detection (47.1M) + direction classifier (1.4M) + recognition (94.9M) = 143.4M