diff --git a/doc/doc_ch/algorithm_overview.md b/doc/doc_ch/algorithm_overview.md index 8cc5bab8c95fecfa6f2dfe68d1cd182c52da5d00..0fd2f5b754e87da0926e7b28d40790c2f26cc0f6 100755 --- a/doc/doc_ch/algorithm_overview.md +++ b/doc/doc_ch/algorithm_overview.md @@ -25,8 +25,8 @@ PaddleOCR开源的文本检测算法列表： 在ICDAR2015文本检测公开数据集上，算法效果如下： |模型|骨干网络|precision|recall|Hmean|下载链接| | --- | --- | --- | --- | --- | --- | -|EAST|ResNet50_vd|85.80%|86.71%|86.25%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| -|EAST|MobileNetV3|79.42%|80.64%|80.03%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| +|EAST|ResNet50_vd|88.71%|81.36%|84.88%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| +|EAST|MobileNetV3|78.2%|79.1%|78.65%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| |DB|ResNet50_vd|86.41%|78.72%|82.38%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)| |DB|MobileNetV3|77.29%|73.08%|75.12%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_db_v2.0_train.tar)| |SAST|ResNet50_vd|91.39%|83.77%|87.42%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)| diff --git a/doc/doc_ch/code_and_doc.md b/doc/doc_ch/code_and_doc.md index e5cf653eb2d73d5e6a8464670c58f9e201b241b9..5ef76914689138a197292efc3654c3e591144fa7 100644 --- a/doc/doc_ch/code_and_doc.md +++ b/doc/doc_ch/code_and_doc.md @@ -153,7 +153,7 @@ cd PaddleOCR > 多数情况下clone失败是由于网络原因，请稍后重试或配置代理 -#### 3.2.2 和 `远程仓库` 建立连接 +#### 3.2.2 通过Token方式登录与建立连接 首先查看当前 `远程仓库` 的信息。 @@ -163,7 +163,24 @@ git remote -v # origin https://github.com/{your_name}/PaddleOCR.git (push) ``` -只有clone的 `远程仓库` 的信息，也就是自己用户名下的 PaddleOCR，接下来我们创建一个原始 PaddleOCR 仓库的远程主机，命名为 upstream。 +只有clone的 `远程仓库` 的信息，也就是自己用户名下的 PaddleOCR。由于Github的登录方式变化，需要通过Token的方式重新配置 `远程仓库` 的地址。生成Token的方式如下： + +1. 找到个人访问令牌（token）：在Github页面右上角点击自己的头像，然后依次选择 Settings --> Developer settings --> Personal access tokens +2. 点击 Generate new token：在Note中填入token名称，例如’paddle‘。在Select scopes选择repo（必选）、admin:repo_hook、delete_repo等，可根据自身需要勾选。然后点击Generate token生成token。最后复制生成的token。 + +删除原始的origin配置 + +``` +git remote rm origin +``` + +将remote分支改成 `https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git`。例如：如果token值为12345，你的用户名为PPOCR，则运行下方命令 + +``` +git remote add origin https://oauth2:12345@github.com/PPOCR/PaddleOCR.git +``` + +这样我们就与自己的 `远程仓库` 建立了连接。接下来我们创建一个原始 PaddleOCR 仓库的远程主机，命名为 upstream。 ``` git remote add upstream https://github.com/PaddlePaddle/PaddleOCR.git @@ -172,8 +189,8 @@ git remote add upstream https://github.com/PaddlePaddle/PaddleOCR.git 使用 `git remote -v` 查看当前 `远程仓库` 的信息，输出如下，发现包括了origin和upstream 2个 `远程仓库` 。 ``` -origin https://github.com/{your_name}/PaddleOCR.git (fetch) -origin https://github.com/{your_name}/PaddleOCR.git (push) +origin https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git (fetch) +origin https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git (push) upstream https://github.com/PaddlePaddle/PaddleOCR.git (fetch) upstream https://github.com/PaddlePaddle/PaddleOCR.git (push) ``` @@ -182,22 +199,23 @@ upstream https://github.com/PaddlePaddle/PaddleOCR.git (push) #### 3.2.3 创建本地分支 -可以基于当前分支创建新的本地分支，命令如下。 +首先获取 upstream 的最新代码，然后基于上游仓库 (upstream)的dygraph创建new_branch分支。 ``` -git checkout -b new_branch -``` - -也可以基于远程或者上游的分支创建新的分支，命令如下。 - -``` -# 基于用户远程仓库(origin)的dygraph创建new_branch分支 -git checkout -b new_branch origin/dygraph -# 基于上游远程仓库(upstream)的dygraph创建new_branch分支 -# 如果需要从upstream创建新的分支，需要首先使用git fetch upstream获取上游代码 +git fetch upstream git checkout -b new_branch upstream/dygraph ``` +> 如果对于新Fork的PaddleOCR项目，用户远程仓库(origin)与上游(upstream)仓库的分支更新情况相同，也可以基于origin仓库的默认分支或指定分支创建新的本地分支，命令如下。 +> +> ``` +> # 基于用户远程仓库(origin)的dygraph创建new_branch分支 +> git checkout -b new_branch origin/dygraph +> +> # 基于用户远程仓库(origin)的默认分支创建new_branch分支 +> git checkout -b new_branch +> ``` + 最终会显示切换到新的分支，输出信息如下 ``` @@ -205,6 +223,8 @@ Branch new_branch set up to track remote branch develop from upstream. Switched to a new branch 'new_branch' ``` +切换分支之后即可在此分支上进行文件改动 + #### 3.2.4 使用pre-commit勾子 Paddle 开发人员使用 pre-commit 工具来管理 Git 预提交钩子。 它可以帮助我们格式化源代码（C++，Python），在提交（commit）前自动检查一些基本事宜（如每个文件只有一个 EOL，Git 中不要添加大文件等）。 @@ -234,23 +254,15 @@ pre-commit ![img](../precommit_pass.png) -使用下面的命令完成提交。 +提交修改，并写明修改内容（"your commit info"） ``` -git commit -m "your commit info" +git commit -m "your commit info" ``` -#### 3.2.6 保持本地仓库最新 +#### 3.2.6 Push到远程仓库 -获取 upstream 的最新代码并更新当前分支。这里的upstream来自于2.2节的`和远程仓库建立连接`部分。 - -``` -git fetch upstream -# 如果是希望提交到其他分支，则需要从upstream的其他分支pull代码，这里是dygraph -git pull upstream dygraph -``` - -#### 3.2.7 push到远程仓库 +使用push命令将修改的commit提交到 `远程仓库` ``` git push origin new_branch @@ -258,7 +270,7 @@ git push origin new_branch #### 3.2.7 提交Pull Request -点击new pull request，选择本地分支和目标分支，如下图所示。在PR的描述说明中，填写该PR所完成的功能。接下来等待review，如果有需要修改的地方，参照上述步骤更新 origin 中的对应分支即可。 +打开自己的远程仓库界面，选择提交的分支。点击new pull request或contribute进入PR界面。选择本地分支和目标分支，如下图所示。在PR的描述说明中，填写该PR所完成的功能。接下来等待review，如果有需要修改的地方，参照上述步骤更新 origin 中的对应分支即可。 ![banner](../pr.png) @@ -285,8 +297,8 @@ git push origin new_branch - 删除本地分支 ``` - # 切换到develop分支，否则无法删除当前分支 - git checkout develop + # 切换到dygraph分支，否则无法删除当前分支 + git checkout dygraph # 删除new_branch分支 git branch -D new_branch @@ -310,7 +322,6 @@ git push origin new_branch - 请注意每个commit的名称：应能反映当前commit的内容，不能太随意。 - 3）如果解决了某个Issue的问题，请在该Pull Request的第一个评论框中加上：fix #issue_number，这样当该Pull Request被合并后，会自动关闭对应的Issue。关键词包括：close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved，请选择合适的词汇。详细可参考[Closing issues via commit messages](https://help.github.com/articles/closing-issues-via-commit-messages)。 此外，在回复评审人意见时，请您遵守以下约定： diff --git a/doc/doc_ch/detection.md b/doc/doc_ch/detection.md index f76ae7f842fb6b7002e084be59dc7ccb31f39771..4114d9f2e6c584566dbfc6d9280074d767848ce1 100644 --- a/doc/doc_ch/detection.md +++ b/doc/doc_ch/detection.md @@ -78,11 +78,11 @@ json.dumps编码前的图像标注信息是包含多个字典的list，字典中 cd PaddleOCR/ # 根据backbone的不同选择下载对应的预训练模型 # 下载MobileNetV3的预训练模型 -wget -P ./pretrain_models/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV3_large_x0_5_pretrained.pdparams +wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams # 或，下载ResNet18_vd的预训练模型 -wget -P ./pretrain_models/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ResNet18_vd_pretrained.pdparams +wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet18_vd_pretrained.pdparams # 或，下载ResNet50_vd的预训练模型 -wget -P ./pretrain_models/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ResNet50_vd_ssld_pretrained.pdparams +wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams ``` diff --git a/doc/doc_ch/knowledge_distillation.md b/doc/doc_ch/knowledge_distillation.md index 70604762e06bca1321d54f7cae95446bb947fdae..1c33a6c1fe13671a0f95bde4db63d0b9946ca180 100644 --- a/doc/doc_ch/knowledge_distillation.md +++ b/doc/doc_ch/knowledge_distillation.md @@ -1,8 +1,27 @@ + # 知识蒸馏 - ++ [知识蒸馏](#0) + + [1. 简介](#1) + - [1.1 知识蒸馏介绍](#11) + - [1.2 PaddleOCR知识蒸馏简介](#12) + + [2. 配置文件解析](#2) + + [2.1 识别配置文件解析](#21) + - [2.1.1 模型结构](#211) + - [2.1.2 损失函数](#212) + - [2.1.3 后处理](#213) + - [2.1.4 指标计算](#214) + - [2.1.5 蒸馏模型微调](#215) + + [2.2 检测配置文件解析](#22) + - [2.2.1 模型结构](#221) + - [2.2.2 损失函数](#222) + - [2.2.3 后处理](#223) + - [2.2.4 蒸馏指标计算](#224) + - [2.2.5 检测蒸馏模型Fine-tune](#225) + + ## 1. 简介 - + ### 1.1 知识蒸馏介绍 近年来，深度神经网络在计算机视觉、自然语言处理等领域被验证是一种极其有效的解决问题的方法。通过构建合适的神经网络，加以训练，最终网络模型的性能指标基本上都会超过传统算法。 @@ -13,6 +32,7 @@ 此外，在知识蒸馏任务中，也衍生出了互学习的模型训练方法，论文[Deep Mutual Learning](https://arxiv.org/abs/1706.00384)中指出，使用两个完全相同的模型在训练的过程中互相监督，可以达到比单个模型训练更好的效果。 + ### 1.2 PaddleOCR知识蒸馏简介 无论是大模型蒸馏小模型，还是小模型之间互相学习，更新参数，他们本质上是都是不同模型之间输出或者特征图(feature map)之间的相互监督，区别仅在于 (1) 模型是否需要固定参数。(2) 模型是否需要加载预训练模型。 @@ -30,17 +50,19 @@ PaddleOCR中集成了知识蒸馏的算法，具体地，有以下几个主要 通过知识蒸馏，在中英文通用文字识别任务中，不增加任何预测耗时的情况下，可以给模型带来3%以上的精度提升，结合学习率调整策略以及模型结构微调策略，最终提升提升超过5%。 - + ## 2. 配置文件解析 在知识蒸馏训练的过程中，数据预处理、优化器、学习率、全局的一些属性没有任何变化。模型结构、损失函数、后处理、指标计算等模块的配置文件需要进行微调。 下面以识别与检测的知识蒸馏配置文件为例，对知识蒸馏的训练与配置进行解析。 + ### 2.1 识别配置文件解析 配置文件在[ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml)。 + #### 2.1.1 模型结构 知识蒸馏任务中，模型结构配置如下所示。 @@ -176,6 +198,7 @@ Architecture: } ``` + #### 2.1.2 损失函数 知识蒸馏任务中，损失函数配置如下所示。 @@ -212,7 +235,7 @@ Loss: 关于`CombinedLoss`更加具体的实现可以参考: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23)。关于`DistillationCTCLoss`等蒸馏损失函数更加具体的实现可以参考[distillation_loss.py](../../ppocr/losses/distillation_loss.py)。 - + #### 2.1.3 后处理 知识蒸馏任务中，后处理配置如下所示。 @@ -228,7 +251,7 @@ PostProcess: 关于`DistillationCTCLabelDecode`更加具体的实现可以参考: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128) - + #### 2.1.4 指标计算 知识蒸馏任务中，指标计算配置如下所示。 @@ -245,7 +268,7 @@ Metric: 关于`DistillationMetric`更加具体的实现可以参考: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24)。 - + #### 2.1.5 蒸馏模型微调 对蒸馏得到的识别蒸馏进行微调有2种方式。 @@ -279,15 +302,15 @@ paddle.save(s_params, "ch_PP-OCRv2_rec_train/student.pdparams") 转化完成之后，使用[ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml)，修改预训练模型的路径（为导出的`student.pdparams`模型路径）以及自己的数据路径，即可进行模型微调。 + ### 2.2 检测配置文件解析 - 检测模型蒸馏的配置文件在PaddleOCR/configs/det/ch_PP-OCRv2/目录下，包含三个蒸馏配置文件： - ch_PP-OCRv2_det_cml.yml，采用cml蒸馏，采用一个大模型蒸馏两个小模型，且两个小模型互相学习的方法 - ch_PP-OCRv2_det_dml.yml，采用DML的蒸馏，两个Student模型互蒸馏的方法 - ch_PP-OCRv2_det_distill.yml，采用Teacher大模型蒸馏小模型Student的方法 - + #### 2.2.1 模型结构 知识蒸馏任务中，模型结构配置如下所示： @@ -419,7 +442,8 @@ Architecture: } ``` -#### 2.1.2 损失函数 + +#### 2.2.2 损失函数 知识蒸馏任务中，检测ch_PP-OCRv2_det_distill.yml蒸馏损失函数配置如下所示。 @@ -484,8 +508,8 @@ Loss: 关于`DistillationDilaDBLoss`更加具体的实现可以参考: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185)。关于`DistillationDBLoss`等蒸馏损失函数更加具体的实现可以参考[distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148)。 - -#### 2.1.3 后处理 + +#### 2.2.3 后处理 知识蒸馏任务中，检测蒸馏后处理配置如下所示。 @@ -503,8 +527,8 @@ PostProcess: 关于`DistillationDBPostProcess`更加具体的实现可以参考: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195) - -#### 2.1.4 蒸馏指标计算 + +#### 2.2.4 蒸馏指标计算 知识蒸馏任务中，检测蒸馏指标计算配置如下所示。 @@ -518,8 +542,8 @@ Metric: 由于蒸馏需要包含多个网络，甚至多个Student网络，在计算指标的时候只需要计算一个Student网络的指标即可，`key`字段设置为`Student`则表示只计算`Student`网络的精度。 - -#### 2.1.5 检测蒸馏模型finetune + +#### 2.2.5 检测蒸馏模型finetune 检测蒸馏有三种方式： - 采用ch_PP-OCRv2_det_distill.yml，Teacher模型设置为PaddleOCR提供的模型或者您训练好的大模型 diff --git a/doc/doc_ch/recognition.md b/doc/doc_ch/recognition.md index bb7d01712a85c92a02109e41814059e6c98c7cdc..51a4b69af0a66a61dd99f95a29a909124e6283a1 100644 --- a/doc/doc_ch/recognition.md +++ b/doc/doc_ch/recognition.md @@ -63,6 +63,17 @@ train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单 | ... ``` +除上述单张图像为一行格式之外，PaddleOCR也支持对离线增广后的数据进行训练，为了防止相同样本在同一个batch中被多次采样，我们可以将相同标签对应的图片路径写在一行中，以列表的形式给出，在训练中，PaddleOCR会随机选择列表中的一张图片进行训练。对应地，标注文件的格式如下。 + +``` +["11.jpg", "12.jpg"] 简单可依赖 +["21.jpg", "22.jpg", "23.jpg"] 用科技让复杂的世界更简单 +3.jpg ocr +``` + +上述示例标注文件中，"11.jpg"和"12.jpg"的标签相同，都是`简单可依赖`，在训练的时候，对于该行标注，会随机选择其中的一张图片进行训练。 + + - 测试集 同训练集类似，测试集也需要提供一个包含所有图片的文件夹（test）和一个rec_gt_test.txt，测试集的结构如下所示： diff --git a/doc/doc_en/algorithm_overview_en.md b/doc/doc_en/algorithm_overview_en.md index 4d521c9cd35d4dbde9b8c83d8a7e5ebbd84e59c9..8d27613ba88660d760cfe1dcd3a1872ece1a51ad 100755 --- a/doc/doc_en/algorithm_overview_en.md +++ b/doc/doc_en/algorithm_overview_en.md @@ -30,8 +30,8 @@ On the ICDAR2015 dataset, the text detection result is as follows: |Model|Backbone|Precision|Recall|Hmean|Download link| | --- | --- | --- | --- | --- | --- | -|EAST|ResNet50_vd|85.80%|86.71%|86.25%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| -|EAST|MobileNetV3|79.42%|80.64%|80.03%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| +|EAST|ResNet50_vd|88.71%|81.36%|84.88%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| +|EAST|MobileNetV3|78.2%|79.1%|78.65%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| |DB|ResNet50_vd|86.41%|78.72%|82.38%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)| |DB|MobileNetV3|77.29%|73.08%|75.12%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_db_v2.0_train.tar)| |SAST|ResNet50_vd|91.39%|83.77%|87.42%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)| diff --git a/doc/doc_en/detection_en.md b/doc/doc_en/detection_en.md index a634dd4903483a819caee88cf6dd1781253e6f85..1be34b330f91b59060fc84af6f5ac44022da1a35 100644 --- a/doc/doc_en/detection_en.md +++ b/doc/doc_en/detection_en.md @@ -67,11 +67,11 @@ And the responding download link of backbone pretrain weights can be found in (h ```shell cd PaddleOCR/ # Download the pre-trained model of MobileNetV3 -wget -P ./pretrain_models/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/MobileNetV3_large_x0_5_pretrained.pdparams +wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams # or, download the pre-trained model of ResNet18_vd -wget -P ./pretrain_models/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ResNet18_vd_pretrained.pdparams +wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet18_vd_pretrained.pdparams # or, download the pre-trained model of ResNet50_vd -wget -P ./pretrain_models/ https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/ResNet50_vd_ssld_pretrained.pdparams +wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams ``` diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md new file mode 100755 index 0000000000000000000000000000000000000000..1db9faef5c97cacbba36fdb42807924e9c7a53cf --- /dev/null +++ b/doc/doc_en/knowledge_distillation_en.md @@ -0,0 +1,592 @@ + +# Knowledge Distillation + ++ [Knowledge Distillation](#0) + + [1. Introduction](#1) + - [1.1 Introduction to Knowledge Distillation](#11) + - [1.2 Introduction to PaddleOCR Knowledge Distillation](#12) + + [2. Configuration File Analysis](#2) + + [2.1 Recognition Model Configuration File Analysis](#21) + - [2.1.1 Model Structure](#211) + - [2.1.2 Loss Function ](#212) + - [2.1.3 Post-processing](#213) + - [2.1.4 Metric Calculation](#214) + - [2.1.5 Fine-tuning Distillation Model](#215) + + [2.2 Detection Model Configuration File Analysis](#22) + - [2.2.1 Model Structure](#221) + - [2.2.2 Loss Function](#222) + - [2.2.3 Post-processing](#223) + - [2.2.4 Metric Calculation](#224) + - [2.2.5 Fine-tuning Distillation Model](#225) + + + + +## 1. Introduction + +### 1.1 Introduction to Knowledge Distillation + +In recent years, deep neural networks have been proved to be an extremely effective method for solving problems in the fields of computer vision and natural language processing. +By constructing a suitable neural network and training it, the performance metrics of the final network model will basically exceed the traditional algorithm. +When the amount of data is large enough, increasing the amount of parameters by constructing a reasonable network model can significantly improve the performance of the model, +but this brings about the problem of a sharp increase in the complexity of the model. Large models are more expensive to use in actual scenarios. +Deep neural networks generally have more parameter redundancy. At present, there are several main methods to compress the model and reduce the amount of its parameters. +Such as pruning, quantification, knowledge distillation, etc., where knowledge distillation refers to the use of teacher models to guide student models to learn specific tasks, +to ensure that the small model obtains a relatively large performance improvement under the condition of unchanged parameters. +In addition, in the knowledge distillation task, a mutual learning model training method was also derived. +The paper [Deep Mutual Learning](https://arxiv.org/abs/1706.00384) pointed out that using two identical models to supervise each other during the training process can achieve better results than a single model training. + + +### 1.2 Introduction to PaddleOCR Knowledge Distillation + +Whether it is a large model distilling a small model, or a small model learning from each other and updating parameters, +they are essentially the output between different models or mutual supervision between feature maps. +The only difference is (1) whether the model requires fixed parameters. (2) Whether the model needs to be loaded with a pre-trained model. +For the case where a large model distills a small model, the large model generally needs to load the pre-trained model and fix the parameters. +For the situation where small models distill each other, the small models generally do not load the pre-trained model, and the parameters are also in a learnable state. + +In the task of knowledge distillation, it is not only the distillation between two models, but also the situation where multiple models learn from each other. +Therefore, in the knowledge distillation code framework, it is also necessary to support this type of distillation method. + +The algorithm of knowledge distillation is integrated in PaddleOCR. Specifically, it has the following main features: +- It supports mutual learning of any network, and does not require the sub-network structure to be completely consistent or to have a pre-trained model. At the same time, there is no limit to the number of sub-networks, just add it in the configuration file. +- Support arbitrarily configuring the loss function through the configuration file, not only can use a certain loss, but also a combination of multiple losses. +- Support all model-related environments such as knowledge distillation training, prediction, evaluation, and export, which is convenient for use and deployment. + +Through knowledge distillation, in the common Chinese and English text recognition task, without adding any time-consuming prediction, +the accuracy of the model can be improved by more than 3%. Combining the learning rate adjustment strategy and the model structure fine-tuning strategy, +the final improvement is more than 5%. + + +## 2. Configuration File Analysis + +In the process of knowledge distillation training, there is no change in data preprocessing, optimizer, learning rate, and some global attributes. +The configuration files of the model structure, loss function, post-processing, metric calculation and other modules need to be fine-tuned. + +The following takes the knowledge distillation configuration file for recognition and detection as an example to analyze the training and configuration of knowledge distillation. + + +### 2.1 Recognition Model Configuration File Analysis + +The configuration file is in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml). + + +#### 2.1.1 Model Structure + +In the knowledge distillation task, the model structure configuration is as follows. +```yaml +Architecture: + model_type: &model_type "rec" # Model category, recognition, detection, etc. + name: DistillationModel # Structure name, in the distillation task, it is DistillationModel + algorithm: Distillation # Algorithm name + Models: # Model, including the configuration information of the subnet + Teacher: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet + pretrained: # Does this sub-network need to load pre-training weights + freeze_params: false # Do you need fixed parameters + return_all_feats: true # Do you need to return all features, if it is False, only the final output is returned + model_type: *model_type # Model category + algorithm: CRNN # The algorithm name of the sub-network. The remaining parameters of the sub-network are consistent with the general model training configuration + Transform: + Backbone: + name: MobileNetV1Enhance + scale: 0.5 + Neck: + name: SequenceEncoder + encoder_type: rnn + hidden_size: 64 + Head: + name: CTCHead + mid_channels: 96 + fc_decay: 0.00002 + Student: # Another sub-network, here is a distillation example of DML, the two sub-networks have the same structure, and both need to learn parameters + pretrained: # The following parameters are the same as above + freeze_params: false + return_all_feats: true + model_type: *model_type + algorithm: CRNN + Transform: + Backbone: + name: MobileNetV1Enhance + scale: 0.5 + Neck: + name: SequenceEncoder + encoder_type: rnn + hidden_size: 64 + Head: + name: CTCHead + mid_channels: 96 + fc_decay: 0.00002 +``` + +If you want to add more sub-networks for training, you can also add the corresponding fields in the configuration file according to the way of adding `Student` and `Teacher`. +For example, if you want 3 models to supervise each other and train together, then `Architecture` can be written in the following format. + +```yaml +Architecture: + model_type: &model_type "rec" + name: DistillationModel + algorithm: Distillation + Models: + Teacher: + pretrained: + freeze_params: false + return_all_feats: true + model_type: *model_type + algorithm: CRNN + Transform: + Backbone: + name: MobileNetV1Enhance + scale: 0.5 + Neck: + name: SequenceEncoder + encoder_type: rnn + hidden_size: 64 + Head: + name: CTCHead + mid_channels: 96 + fc_decay: 0.00002 + Student: + pretrained: + freeze_params: false + return_all_feats: true + model_type: *model_type + algorithm: CRNN + Transform: + Backbone: + name: MobileNetV1Enhance + scale: 0.5 + Neck: + name: SequenceEncoder + encoder_type: rnn + hidden_size: 64 + Head: + name: CTCHead + mid_channels: 96 + fc_decay: 0.00002 + Student2: # The new sub-network introduced in the knowledge distillation task, the configuration is the same as above + pretrained: + freeze_params: false + return_all_feats: true + model_type: *model_type + algorithm: CRNN + Transform: + Backbone: + name: MobileNetV1Enhance + scale: 0.5 + Neck: + name: SequenceEncoder + encoder_type: rnn + hidden_size: 64 + Head: + name: CTCHead + mid_channels: 96 + fc_decay: 0.00002 +``` + +When the model is finally trained, it contains 3 sub-networks: `Teacher`, `Student`, `Student2`. + +The specific implementation code of the `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py). +The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network, +which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle). +In the recognition task, in order to add more loss functions and ensure the scalability of the distillation method, the output of each sub-network is saved as a `dict`, which contains the sub-module output. +Take the recognition model as an example. The output result of each sub-network is `dict`, the key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, `DistillationModel` The output format is as follows. + +```json +{ + "Teacher": { + "backbone_out": tensor, + "neck_out": tensor, + "head_out": tensor, + }, + "Student": { + "backbone_out": tensor, + "neck_out": tensor, + "head_out": tensor, + } +} +``` + + +#### 2.1.2 Loss Function + +In the knowledge distillation task, the loss function configuration is as follows. + +```yaml +Loss: + name: CombinedLoss # Loss function name + loss_config_list: # List of loss function configuration files, mandatory functions for CombinedLoss + - DistillationCTCLoss: # CTC loss function based on distillation, inherited from standard CTC loss + weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field + model_name_list: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the output of these two sub-networks and calculate the CTC loss with gt + key: head_out # In the sub-network output dict, take the corresponding tensor + - DistillationDMLLoss: # DML loss function, inherited from the standard DMLLoss + weight: 1.0 + act: "softmax" # Activation function, use it to process the input, can be softmax, sigmoid or None, the default is None + model_name_pairs: # The subnet name pair used to calculate DML loss. If you want to calculate the DML loss of other subnets, you can continue to add it below the list + - ["Student", "Teacher"] + key: head_out + - DistillationDistanceLoss: # Distilled distance loss function + weight: 1.0 + mode: "l2" # Support l1, l2 or smooth_l1 + model_name_pairs: # Calculate the distance loss of the subnet name pair + - ["Student", "Teacher"] + key: backbone_out +``` + +Among the above loss functions, all distillation loss functions are inherited from the standard loss function class. +The main functions are: Analyze the output of the distillation model, find the intermediate node (tensor) used to calculate the loss, +and then use the standard loss function class to calculate. + +Taking the above configuration as an example, the final distillation training loss function contains the following three parts. + +- The final output `head_out` of `Student` and `Teacher` calculates the CTC loss with gt (loss weight equals 1.0). Here, because both sub-networks need to update the parameters, both of them need to calculate the loss with gt. +- DML loss between `Student` and `Teacher`'s final output `head_out` (loss weight equals 1.0). +- L2 loss between `Student` and `Teacher`'s backbone network output `backbone_out` (loss weight equals 1.0). + +For more specific implementation of `CombinedLoss`, please refer to: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23). +For more specific implementations of distillation loss functions such as `DistillationCTCLoss`, please refer to [distillation_loss.py](../../ppocr/losses/distillation_loss.py) + + + +#### 2.1.3 Post-processing + +In the knowledge distillation task, the post-processing configuration is as follows. + +```yaml +PostProcess: + name: DistillationCTCLabelDecode # CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class + model_name: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the outputs of these two sub-networks and decode them + key: head_out # Take the corresponding tensor in the subnet output dict +``` + +Taking the above configuration as an example, the CTC decoding output of the two sub-networks `Student` and `Teahcer` will be calculated at the same time. +Among them, `key` is the name of the subnet, and `value` is the list of subnets. + +For more specific implementation of `DistillationCTCLabelDecode`, please refer to: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128) + + + +#### 2.1.4 Metric Calculation + +In the knowledge distillation task, the metric calculation configuration is as follows. + +```yaml +Metric: + name: DistillationMetric # CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class + base_metric_name: RecMetric # The base class of indicator calculation. For the output of the model, the indicator will be calculated based on this class + main_indicator: acc # The name of the indicator + key: "Student" # Select the main_indicator of this subnet as the criterion for saving the best model +``` + +Taking the above configuration as an example, the accuracy metric of the `Student` subnet will be used as the judgment metric for saving the best model. +At the same time, the accuracy metric of all subnets will be printed out in the log. + +For more specific implementation of `DistillationMetric`, please refer to: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24). + + + +#### 2.1.5 Fine-tuning Distillation Model + +There are two ways to fine-tune the recognition distillation task. + +1. Fine-tuning based on knowledge distillation: this situation is relatively simple, download the pre-trained model. Then configure the pre-training model path and your own data path in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml) to perform fine-tuning training of the model. +2. Do not use knowledge distillation in fine-tuning: In this case, you need to first extract the student model parameters from the pre-training model. The specific steps are as follows. + +- First download the pre-trained model and unzip it. +```shell +wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar +tar -xf ch_PP-OCRv2_rec_train.tar +``` + +- Then use python to extract the student model parameters + +```python +import paddle +# Load the pre-trained model +all_params = paddle.load("ch_PP-OCRv2_rec_train/best_accuracy.pdparams") +# View the keys of the weight parameter +print(all_params.keys()) +# Weight extraction of student model +s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} +# View the keys of the weight parameters of the student model +print(s_params.keys()) +# Save weight parameters +paddle.save(s_params, "ch_PP-OCRv2_rec_train/student.pdparams") +``` + +After the extraction is complete, use [ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml) to modify the path of the pre-trained model (the path of the exported `student.pdparams` model) and your own data path to fine-tune the model. + + +### 2.2 Detection Model Configuration File Analysis + +The configuration file of the detection model distillation is in the ```PaddleOCR/configs/det/ch_PP-OCRv2/``` directory, which contains three distillation configuration files: + +- ```ch_PP-OCRv2_det_cml.yml```, Use one large model to distill two small models, and the two small models learn from each other +- ```ch_PP-OCRv2_det_dml.yml```, Method of mutual distillation of two student models +- ```ch_PP-OCRv2_det_distill.yml```, The method of using large teacher model to distill small student model + + +#### 2.2.1 Model Structure + +In the knowledge distillation task, the model structure configuration is as follows: +``` +Architecture: + name: DistillationModel # Structure name, in the distillation task, it is DistillationModel + algorithm: Distillation # Algorithm name + Models: # Model, including the configuration information of the subnet + Student: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet + pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained # Does this sub-network need to load pre-training weights + freeze_params: false # Do you need fixed parameters + return_all_feats: false # Do you need to return all features, if it is False, only the final output is returned + model_type: det + algorithm: DB + Backbone: + name: MobileNetV3 + scale: 0.5 + model_name: large + disable_se: True + Neck: + name: DBFPN + out_channels: 96 + Head: + name: DBHead + k: 50 + Teacher: # Another sub-network, here is a distillation example of a large model distill a small model + pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy + freeze_params: true # The Teacher model is well-trained and does not need to participate in training + return_all_feats: false + model_type: det + algorithm: DB + Transform: + Backbone: + name: ResNet + layers: 18 + Neck: + name: DBFPN + out_channels: 256 + Head: + name: DBHead + k: 50 + +``` +If DML is used, that is, the method of two small models learning from each other, the Teacher network structure in the above configuration file needs to be set to the same configuration as the Student model. +Refer to the configuration file for details. [ch_PP-OCRv2_det_dml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.4/configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_dml.yml) + + +The following describes the configuration file parameters [ch_PP-OCRv2_det_cml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.4/configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml): + +``` +Architecture: + name: DistillationModel + algorithm: Distillation + model_type: det + Models: + Teacher: # Teacher model configuration of CML distillation + pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy + freeze_params: true # Teacher does not train + return_all_feats: false + model_type: det + algorithm: DB + Transform: + Backbone: + name: ResNet + layers: 18 + Neck: + name: DBFPN + out_channels: 256 + Head: + name: DBHead + k: 50 + Student: # Student model configuration for CML distillation + pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained + freeze_params: false + return_all_feats: false + model_type: det + algorithm: DB + Backbone: + name: MobileNetV3 + scale: 0.5 + model_name: large + disable_se: True + Neck: + name: DBFPN + out_channels: 96 + Head: + name: DBHead + k: 50 + Student2: # Student2 model configuration for CML distillation + pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained + freeze_params: false + return_all_feats: false + model_type: det + algorithm: DB + Transform: + Backbone: + name: MobileNetV3 + scale: 0.5 + model_name: large + disable_se: True + Neck: + name: DBFPN + out_channels: 96 + Head: + name: DBHead + k: 50 + +``` + +The specific implementation code of the distillation model `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py). + +The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network, +which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle). + +In the distillation task, in order to facilitate the addition of the distillation loss function, the output of each network is saved as a `dict`, which contains the sub-module output. +The key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, the output format of `DistillationModel` is as follows. + +```json +{ + "Teacher": { + "backbone_out": tensor, + "neck_out": tensor, + "head_out": tensor, + }, + "Student": { + "backbone_out": tensor, + "neck_out": tensor, + "head_out": tensor, + } +} +``` + + +#### 2.2.2 Loss Function + +In the task of detection knowledge distillation ```ch_PP-OCRv2_det_distill.yml````, the distillation loss function configuration is as follows. +```yaml +Loss: + name: CombinedLoss # Loss function name + loss_config_list: # List of loss function configuration files, mandatory functions for CombinedLoss + - DistillationDilaDBLoss: # DB loss function based on distillation, inherited from standard DBloss + weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field + model_name_pairs: # Extract the output of these two sub-networks and calculate the loss between them + - ["Student", "Teacher"] + key: maps # In the sub-network output dict, take the corresponding tensor + balance_loss: true # The following parameters are the configuration parameters of standard DBloss + main_loss_type: DiceLoss + alpha: 5 + beta: 10 + ohem_ratio: 3 + - DistillationDBLoss: # Used to calculate the loss between Student and GT + weight: 1.0 + model_name_list: ["Student"] # The model name only has Student, which means that the loss between Student and GT is calculated + name: DBLoss + balance_loss: true + main_loss_type: DiceLoss + alpha: 5 + beta: 10 + ohem_ratio: 3 +``` + +Similarly, distillation loss function configuration(`ch_PP-OCRv2_det_cml.yml`) is shown below. Compared with the loss function configuration of ch_PP-OCRv2_det_distill.yml, there are three changes: +```yaml +Loss: + name: CombinedLoss + loss_config_list: + - DistillationDilaDBLoss: + weight: 1.0 + model_name_pairs: + - ["Student", "Teacher"] + - ["Student2", "Teacher"] # 1. Calculate the loss of two Student and Teacher + key: maps + balance_loss: true + main_loss_type: DiceLoss + alpha: 5 + beta: 10 + ohem_ratio: 3 + - DistillationDMLLoss: # 2. Add to calculate the loss between two students + model_name_pairs: + - ["Student", "Student2"] + maps_name: "thrink_maps" + weight: 1.0 + # act: None + key: maps + - DistillationDBLoss: + weight: 1.0 + model_name_list: ["Student", "Student2"] # 3. Calculate the loss between two students and GT + balance_loss: true + main_loss_type: DiceLoss + alpha: 5 + beta: 10 + ohem_ratio: 3 +``` + +For more specific implementation of `DistillationDilaDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185). +For more specific implementations of distillation loss functions such as `DistillationDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148) + + +#### 2.2.3 Post-processing + +In the task of detecting knowledge distillation, the post-processing configuration of detecting distillation is as follows. + +```yaml +PostProcess: + name: DistillationDBPostProcess # The CTC decoding post-processing of the DB detection distillation task, inherited from the standard DBPostProcess class + model_name: ["Student", "Student2", "Teacher"] # Extract the output of multiple sub-networks and decode them. The network that does not require post-processing is not set in model_name + thresh: 0.3 + box_thresh: 0.6 + max_candidates: 1000 + unclip_ratio: 1.5 +``` + +Taking the above configuration as an example, the output of the three subnets `Student`, `Student2` and `Teacher` will be calculated at the same time for post-processing calculations. +Since there are multiple inputs, there are also multiple outputs returned by post-processing. +For a more specific implementation of `DistillationDBPostProcess`, please refer to: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195) + + +#### 2.2.4 Metric Calculation +In the knowledge distillation task, the metric calculation configuration is as follows. +```yaml +Metric: + name: DistillationMetric + base_metric_name: DetMetric + main_indicator: hmean + key: "Student" +``` + +Since distillation needs to include multiple networks, only one network metrics needs to be calculated when calculating the metrics. +The `key` field is set to `Student`, it means that only the metrics of the `Student` network is calculated. +Model Structure + + +#### 2.2.5 Fine-tuning Distillation Model + +There are three ways to fine-tune the detection distillation task: +- `ch_PP-OCRv2_det_distill.yml`, The teacher model is set to the model provided by PaddleOCR or the large model you have trained. +- `ch_PP-OCRv2_det_cml.yml`, Use cml distillation. Similarly, the Teacher model is set to the model provided by PaddleOCR or the large model you have trained. +- `ch_PP-OCRv2_det_dml.yml`, Distillation using DML. The method of mutual distillation of the two Student models has an accuracy improvement of about 1.7% on the data set used by PaddleOCR. + +In fine-tune, you need to set the pre-trained model to be loaded in the `pretrained` parameter of the network structure. + +In terms of accuracy improvement, `cml` > `dml` > `distill`. When the amount of data is insufficient or the accuracy of the teacher model is similar to that of the student, this conclusion may change. + +In addition, since the distillation pre-training model provided by PaddleOCR contains multiple model parameters, if you want to extract the parameters of the student model, you can refer to the following code: +```sh +# Download the parameters of the distillation training model +wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar +``` + +```python +import paddle +# Load the pre-trained model +all_params = paddle.load("ch_PP-OCRv2_det_distill_train/best_accuracy.pdparams") +# View the keys of the weight parameter +print(all_params.keys()) +# Extract the weights of the student model +s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} +# View the keys of the weight parameters of the student model +print(s_params.keys()) +# Save +paddle.save(s_params, "ch_PP-OCRv2_det_distill_train/student.pdparams") +``` + +Finally, the parameters of the student model will be saved in `ch_PP-OCRv2_det_distill_train/student.pdparams` for the fine-tune of the model. diff --git a/doc/joinus.PNG b/doc/joinus.PNG index 71b9277dc5ba025ea036cd9fce046acd2cb45795..e03e108c52385dc31fb0daece013c3d80bf08db4 100644 Binary files a/doc/joinus.PNG and b/doc/joinus.PNG differ diff --git a/paddleocr.py b/paddleocr.py index 733c83d1b4faa23212e7186148a5a9e1154ba891..f19d1defee217a8d3dc0653e6b8fd1713cb389fa 100644 --- a/paddleocr.py +++ b/paddleocr.py @@ -42,12 +42,14 @@ __all__ = [ ] SUPPORT_DET_MODEL = ['DB'] -VERSION = '2.3.0.2' +VERSION = '2.4' SUPPORT_REC_MODEL = ['CRNN'] BASE_DIR = os.path.expanduser("~/.paddleocr/") DEFAULT_OCR_MODEL_VERSION = 'PP-OCR' +SUPPORT_OCR_MODEL_VERSION = ['PP-OCR', 'PP-OCRv2'] DEFAULT_STRUCTURE_MODEL_VERSION = 'STRUCTURE' +SUPPORT_STRUCTURE_MODEL_VERSION = ['STRUCTURE'] MODEL_URLS = { 'OCR': { 'PP-OCRv2': { @@ -190,6 +192,7 @@ def parse_args(mMain=True): parser.add_argument( "--ocr_version", type=str, + choices=SUPPORT_OCR_MODEL_VERSION, default='PP-OCRv2', help='OCR Model version, the current model support list is as follows: ' '1. PP-OCRv2 Support Chinese detection and recognition model. ' @@ -198,6 +201,7 @@ def parse_args(mMain=True): parser.add_argument( "--structure_version", type=str, + choices=SUPPORT_STRUCTURE_MODEL_VERSION, default='STRUCTURE', help='Model version, the current model support list is as follows:' ' 1. STRUCTURE Support en table structure model.') @@ -257,26 +261,20 @@ def get_model_config(type, version, model_type, lang): DEFAULT_MODEL_VERSION = DEFAULT_STRUCTURE_MODEL_VERSION else: raise NotImplementedError + model_urls = MODEL_URLS[type] if version not in model_urls: - logger.warning('version {} not in {}, auto switch to version {}'.format( - version, model_urls.keys(), DEFAULT_MODEL_VERSION)) version = DEFAULT_MODEL_VERSION if model_type not in model_urls[version]: if model_type in model_urls[DEFAULT_MODEL_VERSION]: - logger.warning( - 'version {} not support {} models, auto switch to version {}'. - format(version, model_type, DEFAULT_MODEL_VERSION)) version = DEFAULT_MODEL_VERSION else: logger.error('{} models is not support, we only support {}'.format( model_type, model_urls[DEFAULT_MODEL_VERSION].keys())) sys.exit(-1) + if lang not in model_urls[version][model_type]: if lang in model_urls[DEFAULT_MODEL_VERSION][model_type]: - logger.warning( - 'lang {} is not support in {}, auto switch to version {}'. - format(lang, version, DEFAULT_MODEL_VERSION)) version = DEFAULT_MODEL_VERSION else: logger.error( @@ -296,6 +294,8 @@ class PaddleOCR(predict_system.TextSystem): """ params = parse_args(mMain=False) params.__dict__.update(**kwargs) + assert params.ocr_version in SUPPORT_OCR_MODEL_VERSION, "ocr_version must in {}, but get {}".format( + SUPPORT_OCR_MODEL_VERSION, params.ocr_version) params.use_gpu = check_gpu(params.use_gpu) if not params.show_log: @@ -398,6 +398,8 @@ class PPStructure(OCRSystem): def __init__(self, **kwargs): params = parse_args(mMain=False) params.__dict__.update(**kwargs) + assert params.structure_version in SUPPORT_STRUCTURE_MODEL_VERSION, "structure_version must in {}, but get {}".format( + SUPPORT_STRUCTURE_MODEL_VERSION, params.structure_version) params.use_gpu = check_gpu(params.use_gpu) if not params.show_log: diff --git a/ppocr/data/__init__.py b/ppocr/data/__init__.py index 0bb3d506483a331fba48feafeff9ca2d439f3782..429d5a528bda5b588d262b82671f5b3eba04939a 100644 --- a/ppocr/data/__init__.py +++ b/ppocr/data/__init__.py @@ -20,6 +20,7 @@ from __future__ import unicode_literals import os import sys import numpy as np +import skimage import paddle import signal import random diff --git a/ppocr/data/simple_dataset.py b/ppocr/data/simple_dataset.py index ee8571b8c452bbd834fc5dbcf01ce390562163d6..9f0ce352d9112f502eaf5ba607137018887683fe 100644 --- a/ppocr/data/simple_dataset.py +++ b/ppocr/data/simple_dataset.py @@ -69,6 +69,16 @@ class SimpleDataSet(Dataset): random.shuffle(self.data_lines) return + def _try_parse_filename_list(self, file_name): + # multiple images -> one gt label + if len(file_name) > 0 and file_name[0] == "[": + try: + info = json.loads(file_name) + file_name = random.choice(info) + except: + pass + return file_name + def get_ext_data(self): ext_data_num = 0 for op in self.ops: @@ -85,6 +95,7 @@ class SimpleDataSet(Dataset): data_line = data_line.decode('utf-8') substr = data_line.strip("\n").split(self.delimiter) file_name = substr[0] + file_name = self._try_parse_filename_list(file_name) label = substr[1] img_path = os.path.join(self.data_dir, file_name) data = {'img_path': img_path, 'label': label} @@ -95,7 +106,7 @@ class SimpleDataSet(Dataset): data['image'] = img data = transform(data, load_data_ops) - if data is None or data['polys'].shape[1]!=4: + if data is None or data['polys'].shape[1] != 4: continue ext_data.append(data) return ext_data @@ -107,6 +118,7 @@ class SimpleDataSet(Dataset): data_line = data_line.decode('utf-8') substr = data_line.strip("\n").split(self.delimiter) file_name = substr[0] + file_name = self._try_parse_filename_list(file_name) label = substr[1] img_path = os.path.join(self.data_dir, file_name) data = {'img_path': img_path, 'label': label} diff --git a/ppocr/utils/save_load.py b/ppocr/utils/save_load.py index f6013a406634ed110ea5af613a5f31e56ce90ead..0dd94e86c808ce4f77e27a5c819fccd59578f0c5 100644 --- a/ppocr/utils/save_load.py +++ b/ppocr/utils/save_load.py @@ -111,13 +111,16 @@ def load_pretrained_params(model, path): params = paddle.load(path + '.pdparams') state_dict = model.state_dict() new_state_dict = {} - for k1, k2 in zip(state_dict.keys(), params.keys()): - if list(state_dict[k1].shape) == list(params[k2].shape): - new_state_dict[k1] = params[k2] + for k1 in params.keys(): + if k1 not in state_dict.keys(): + logger.warning("The pretrained params {} not in model".format(k1)) else: - logger.warning( - "The shape of model params {} {} not matched with loaded params {} {} !". - format(k1, state_dict[k1].shape, k2, params[k2].shape)) + if list(state_dict[k1].shape) == list(params[k1].shape): + new_state_dict[k1] = params[k1] + else: + logger.warning( + "The shape of model params {} {} not matched with loaded params {} {} !". + format(k1, state_dict[k1].shape, k1, params[k1].shape)) model.set_state_dict(new_state_dict) logger.info("load pretrain successful from {}".format(path)) return model diff --git a/ppstructure/README.md b/ppstructure/README.md index a09a43299b11dccf99897d5a6c69704191253aaf..abcc5281aa24c56e6a3f22e377de816b0bd66712 100644 --- a/ppstructure/README.md +++ b/ppstructure/README.md @@ -1,187 +1,140 @@ English | [简体中文](README_ch.md) -# PP-Structure +- [1. Introduction](#1) +- [2. Update log](#2) +- [3. Features](#3) +- [4. Results](#4) + * [4.1 Layout analysis and table recognition](#41) + * [4.2 DOC-VQA](#42) +- [5. Quick start](#5) +- [6. PP-Structure System](#6) + * [6.1 Layout analysis and table recognition](#61) + * [6.2 DOC-VQA](#62) +- [7. Model List](#7) -PP-Structure is an OCR toolkit that can be used for complex documents analysis. The main features are as follows: -- Support the layout analysis of documents, divide the documents into 5 types of areas **text, title, table, image and list** (conjunction with Layout-Parser) -- Support to extract the texts from the text, title, picture and list areas (used in conjunction with PP-OCR) -- Support to extract excel files from the table areas -- Support python whl package and command line usage, easy to use -- Support custom training for layout analysis and table structure tasks + -## 1. Visualization - - +## 1. Introduction +PP-Structure is an OCR toolkit that can be used for document analysis and processing with complex structures, designed to help developers better complete document understanding tasks + -## 2. Installation +## 2. Update log +* 2021.12.07 add [DOC-VQA SER and RE tasks](vqa/README.md)。 -### 2.1 Install requirements + -- **（1) Install PaddlePaddle** +## 3. Features -```bash -pip3 install --upgrade pip +The main features of PP-Structure are as follows: -# GPU -python3 -m pip install paddlepaddle-gpu==2.1.1 -i https://mirror.baidu.com/pypi/simple - -# CPU - python3 -m pip install paddlepaddle==2.1.1 -i https://mirror.baidu.com/pypi/simple +- Support the layout analysis of documents, divide the documents into 5 types of areas **text, title, table, image and list** (conjunction with Layout-Parser) +- Support to extract the texts from the text, title, picture and list areas (used in conjunction with PP-OCR) +- Support to extract excel files from the table areas +- Support python whl package and command line usage, easy to use +- Support custom training for layout analysis and table structure tasks +- Support Document Visual Question Answering (DOC-VQA) tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE) -``` -For more，refer [Installation](https://www.paddlepaddle.org.cn/install/quick) . -- **(2) Install Layout-Parser** + -```bash -pip3 install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl -``` +## 4. Results -### 2.2 Install PaddleOCR（including PP-OCR and PP-Structure） + -- **（1) PIP install PaddleOCR whl package（inference only）** +### 4.1 Layout analysis and table recognition -```bash -pip install "paddleocr>=2.2" -``` + -- **（2) Clone PaddleOCR（Inference+training）** +The figure shows the pipeline of layout analysis + table recognition. The image is first divided into four areas of image, text, title and table by layout analysis, and then OCR detection and recognition is performed on the three areas of image, text and title, and the table is performed table recognition, where the image will also be stored for use. -```bash -git clone https://github.com/PaddlePaddle/PaddleOCR -``` + +### 4.2 DOC-VQA -## 3. Quick Start +* SER -### 3.1 Use by command line +![](./vqa/images/result_ser/zh_val_0_ser.jpg) | ![](./vqa/images/result_ser/zh_val_42_ser.jpg) +---|--- -```bash -paddleocr --image_dir=../doc/table/1.png --type=structure -``` +Different colored boxes in the figure represent different categories. For xfun dataset, there are three categories: query, answer and header: -### 3.2 Use by python API +* Dark purple: header +* Light purple: query +* Army green: answer -```python -import os -import cv2 -from paddleocr import PPStructure,draw_structure_result,save_structure_res +The corresponding category and OCR recognition results are also marked at the top left of the OCR detection box. -table_engine = PPStructure(show_log=True) -save_folder = './output/table' -img_path = '../doc/table/1.png' -img = cv2.imread(img_path) -result = table_engine(img) -save_structure_res(result, save_folder,os.path.basename(img_path).split('.')[0]) +* RE -for line in result: - line.pop('img') - print(line) +![](./vqa/images/result_re/zh_val_21_re.jpg) | ![](./vqa/images/result_re/zh_val_40_re.jpg) +---|--- -from PIL import Image -font_path = '../doc/fonts/simfang.ttf' -image = Image.open(img_path).convert('RGB') -im_show = draw_structure_result(image, result,font_path=font_path) -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` -### 3.3 Returned results format -The returned results of PP-Structure is a list composed of a dict, an example is as follows +In the figure, the red box represents the question, the blue box represents the answer, and the question and answer are connected by green lines. The corresponding category and OCR recognition results are also marked at the top left of the OCR detection box. -```shell -[ - { 'type': 'Text', - 'bbox': [34, 432, 345, 462], - 'res': ([[36.0, 437.0, 341.0, 437.0, 341.0, 446.0, 36.0, 447.0], [41.0, 454.0, 125.0, 453.0, 125.0, 459.0, 41.0, 460.0]], - [('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent ', 0.465441)]) - } -] -``` -The description of each field in dict is as follows -| Parameter | Description | -| --------------- | -------------| -|type|Type of image area| -|bbox|The coordinates of the image area in the original image, respectively [left upper x, left upper y, right bottom x, right bottom y]| -|res|OCR or table recognition result of image area。
Table: HTML string of the table;
OCR: A tuple containing the detection coordinates and recognition results of each single line of text| + +## 5. Quick start -### 3.4 Parameter description： +Start from [Quick Installation](./docs/quickstart.md) -| Parameter | Description | Default value | -| --------------- | ---------------------------------------- | ------------------------------------------- | -| output | The path where excel and recognition results are saved | ./output/table | -| table_max_len | The long side of the image is resized in table structure model | 488 | -| table_model_dir | inference model path of table structure model | None | -| table_char_type | dict path of table structure model | ../ppocr/utils/dict/table_structure_dict.tx | + -Most of the parameters are consistent with the paddleocr whl package, see [doc of whl](../doc/doc_en/whl_en.md) +## 6. PP-Structure System -After running, each image will have a directory with the same name under the directory specified in the output field. Each table in the picture will be stored as an excel and figure area will be cropped and saved, the excel and image file name will be the coordinates of the table in the image. + -## 4. PP-Structure Pipeline -![pipeline](../doc/table/pipeline_en.jpg) +### 6.1 Layout analysis and table recognition -In PP-Structure, the image will be analyzed by layoutparser first. In the layout analysis, the area in the image will be classified, including **text, title, image, list and table** 5 categories. For the first 4 types of areas, directly use the PP-OCR to complete the text detection and recognition. The table area will be converted to an excel file of the same table style via Table OCR. +![pipeline](../doc/table/pipeline.jpg) -### 4.1 LayoutParser +In PP-Structure, the image will be divided into 5 types of areas **text, title, image list and table**. For the first 4 types of areas, directly use PP-OCR system to complete the text detection and recognition. For the table area, after the table structuring process, the table in image is converted into an Excel file with the same table style. -Layout analysis divides the document data into regions, including the use of Python scripts for layout analysis tools, extraction of special category detection boxes, performance indicators, and custom training layout analysis models. For details, please refer to [document](layout/README_en.md). +#### 6.1.1 Layout analysis -### 4.2 Table Recognition +Layout analysis classifies image by region, including the use of Python scripts of layout analysis tools, extraction of designated category detection boxes, performance indicators, and custom training layout analysis models. For details, please refer to [document](layout/README.md). -Table Recognition converts table image into excel documents, which include the detection and recognition of table text and the prediction of table structure and cell coordinates. For detailed, please refer to [document](table/README.md) +#### 6.1.2 Table recognition -## 5. Prediction by inference engine +Table recognition converts table images into excel documents, which include the detection and recognition of table text and the prediction of table structure and cell coordinates. For detailed instructions, please refer to [document](table/README.md) -Use the following commands to complete the inference. + -```python -cd PaddleOCR/ppstructure +### 6.2 DOC-VQA -# download model -mkdir inference && cd inference -# Download the detection model of the ultra-lightweight Chinese OCR model and uncompress it -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar && tar xf ch_ppocr_mobile_v2.0_det_infer.tar -# Download the recognition model of the ultra-lightweight Chinese OCR model and uncompress it -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar && tar xf ch_ppocr_mobile_v2.0_rec_infer.tar -# Download the table structure model of the ultra-lightweight Chinese OCR model and uncompress it -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar -cd .. +Document Visual Question Answering (DOC-VQA) if a type of Visual Question Answering (VQA), which includes Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks. Based on SER task, text recognition and classification in images can be completed. Based on THE RE task, we can extract the relation of the text content in the image, such as judge the problem pair. For details, please refer to [document](vqa/README.md) -python3 predict_system.py --det_model_dir=inference/ch_ppocr_mobile_v2.0_det_infer --rec_model_dir=inference/ch_ppocr_mobile_v2.0_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --output=../output/table --vis_font_path=../doc/fonts/simfang.ttf -``` -After running, each image will have a directory with the same name under the directory specified in the output field. Each table in the picture will be stored as an excel and figure area will be cropped and saved, the excel and image file name will be the coordinates of the table in the image. -**Model List** + -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -|en_ppocr_mobile_v2.0_table_structure|Table structure prediction for English table scenarios|[table_mv3.yml](../configs/table/table_mv3.yml)|18.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) | +## 7. Model List -**Model List** +PP-Structure系列模型列表（更新中） -LayoutParser model +* Layout analysis model |model name|description|download| | --- | --- | --- | -| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis model trained on the PubLayNet data set can be divided into 5 types of areas **text, title, table, picture and list** | [PubLayNet](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) | -| ppyolov2_r50vd_dcn_365e_tableBank_word | The layout analysis model trained on the TableBank Word dataset can only detect tables | [TableBank Word](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | -| ppyolov2_r50vd_dcn_365e_tableBank_latex | The layout analysis model trained on the TableBank Latex dataset can only detect tables | [TableBank Latex](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | +| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis model trained on the PubLayNet dataset can divide image into 5 types of areas **text, title, table, picture, and list** | [PubLayNet](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) | -OCR and table recognition model + +* OCR and table recognition model |model name|description|model size|download| | --- | --- | --- | --- | |ch_ppocr_mobile_slim_v2.0_det|Slim pruned lightweight model, supporting Chinese, English, multilingual text detection|2.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/slim/ch_ppocr_mobile_v2.0_det_prune_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/slim/ch_ppocr_mobile_v2.0_det_prune_infer.tar) | |ch_ppocr_mobile_slim_v2.0_rec|Slim pruned and quantized lightweight model, supporting Chinese, English and number recognition|6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_train.tar) | -|en_ppocr_mobile_v2.0_table_det|Text detection of English table scenes trained on PubLayNet dataset|4.7M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) | -|en_ppocr_mobile_v2.0_table_rec|Text recognition of English table scene trained on PubLayNet dataset|6.9M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) | -|en_ppocr_mobile_v2.0_table_structure|Table structure prediction of English table scene trained on PubLayNet dataset|18.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) | +|en_ppocr_mobile_v2.0_table_structure|Table structure prediction of English table scene trained on PubLayNet dataset|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) | + +* DOC-VQA model +|model name|description|model size|download| +| --- | --- | --- | --- | +|PP-Layout_v1.0_ser_pretrained|SER model trained on xfun Chinese dataset based on LayoutXLM|1.4G|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar) | +|PP-Layout_v1.0_re_pretrained|RE model trained on xfun Chinese dataset based on LayoutXLM|1.4G|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_re_pretrained.tar) | -If you need to use other models, you can download the model in [model_list](../doc/doc_en/models_list_en.md) or use your own trained model to configure it to the three fields of `det_model_dir`, `rec_model_dir`, `table_model_dir` . +If you need to use other models, you can download the model in [PPOCR model_list](../doc/doc_en/models_list_en.md) and [PPStructure model_list](./docs/model_list.md) diff --git a/ppstructure/README_ch.md b/ppstructure/README_ch.md index 607efac1bf6bfaa58f0e96ceef1a0ee344189e9c..808a5c68d18df625bedeae4706da7f985d6caecd 100644 --- a/ppstructure/README_ch.md +++ b/ppstructure/README_ch.md @@ -1,14 +1,32 @@ [English](README.md) | 简体中文 -## 简介 +- [1. 简介](#1) +- [2. 近期更新](#2) +- [3. 特性](#3) +- [4. 效果展示](#4) + * [4.1 版面分析和表格识别](#41) + * [4.2 DOC-VQA](#42) +- [5. 快速体验](#5) +- [6. PP-Structure 介绍](#6) + * [6.1 版面分析+表格识别](#61) + * [6.2 DOC-VQA](#62) +- [7. 模型库](#7) + + + +## 1. 简介 PP-Structure是一个可用于复杂文档结构分析和处理的OCR工具包，旨在帮助开发者更好的完成文档理解相关任务。 -## 近期更新 -* 2021.12.07 新增VQA任务-SER和RE。 + -## 特性 +## 2. 近期更新 +* 2021.12.07 新增DOC-[VQA任务SER和RE](vqa/README.md)。 -PP-Structure是一个可用于复杂文档结构分析和处理的OCR工具包，主要特性如下： + + +## 3. 特性 + +PP-Structure的主要特性如下： - 支持对图片形式的文档进行版面分析，可以划分**文字、标题、表格、图片以及列表**5类区域（与Layout-Parser联合使用） - 支持文字、标题、图片以及列表区域提取为文字字段（与PP-OCR联合使用） - 支持表格区域进行结构化分析，最终结果输出Excel文件 @@ -17,13 +35,22 @@ PP-Structure是一个可用于复杂文档结构分析和处理的OCR工具包 - 支持文档视觉问答(Document Visual Question Answering，DOC-VQA)任务-语义实体识别(Semantic Entity Recognition，SER)和关系抽取(Relation Extraction，RE) -## 1. 效果展示 + + +## 4. 效果展示 -### 1.1 版面分析和表格识别 + + +### 4.1 版面分析和表格识别 -### 1.2 VQA +图中展示了版面分析+表格识别的整体流程，图片先有版面分析划分为图像、文本、标题和表格四种区域，然后对图像、文本和标题三种区域进行OCR的检测识别，对表格进行表格识别，其中图像还会被存储下来以便使用。 + + + + +### 4.2 DOC-VQA * SER @@ -46,36 +73,45 @@ PP-Structure是一个可用于复杂文档结构分析和处理的OCR工具包 图中红色框表示问题，蓝色框表示答案，问题和答案之间使用绿色线连接。在OCR检测框的左上方也标出了对应的类别和OCR识别结果。 -## 2. 快速体验 + + +## 5. 快速体验 + +请参考[快速安装](./docs/quickstart.md)教程。 -代码体验：从 [快速安装](./docs/quickstart.md) 开始 + -## 3. PP-Structure Pipeline介绍 +## 6. PP-Structure 介绍 -### 3.1 版面分析+表格识别 + + +### 6.1 版面分析+表格识别 ![pipeline](../doc/table/pipeline.jpg) 在PP-Structure中，图片会先经由Layout-Parser进行版面分析，在版面分析中，会对图片里的区域进行分类，包括**文字、标题、图片、列表和表格**5类。对于前4类区域，直接使用PP-OCR完成对应区域文字检测与识别。对于表格类区域，经过表格结构化处理后，表格图片转换为相同表格样式的Excel文件。 -#### 3.1.1 版面分析 +#### 6.1.1 版面分析 版面分析对文档数据进行区域分类，其中包括版面分析工具的Python脚本使用、提取指定类别检测框、性能指标以及自定义训练版面分析模型，详细内容可以参考[文档](layout/README_ch.md)。 -#### 3.1.2 表格识别 +#### 6.1.2 表格识别 + +表格识别将表格图片转换为excel文档，其中包含对于表格文本的检测和识别以及对于表格结构和单元格坐标的预测，详细说明参考[文档](table/README_ch.md)。 -表格识别将表格图片转换为excel文档，其中包含对于表格文本的检测和识别以及对于表格结构和单元格坐标的预测，详细说明参考[文档](table/README_ch.md) + +### 6.2 DOC-VQA -### 3.2 VQA +DOC-VQA指文档视觉问答，其中包括语义实体识别 (Semantic Entity Recognition, SER) 和关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务，可以完成对图像中的文本识别与分类；基于 RE 任务，可以完成对图象中的文本内容的关系提取，如判断问题对(pair)，详细说明参考[文档](vqa/README.md)。 -coming soon + -## 4. 模型库 +## 7. 模型库 PP-Structure系列模型列表（更新中） -* LayoutParser 模型 +* 版面分析模型 |模型名称|模型简介|下载地址| | --- | --- | --- | @@ -90,7 +126,7 @@ PP-Structure系列模型列表（更新中） |ch_ppocr_mobile_slim_v2.0_rec|slim裁剪量化版超轻量模型，支持中英文、数字识别|6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_train.tar) | |en_ppocr_mobile_v2.0_table_structure|PubLayNet数据集训练的英文表格场景的表格结构预测|18.6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) | -* VQA模型 +* DOC-VQA 模型 |模型名称|模型简介|模型大小|下载地址| | --- | --- | --- | --- | @@ -98,4 +134,4 @@ PP-Structure系列模型列表（更新中） |PP-Layout_v1.0_re_pretrained|基于LayoutXLM在xfun中文数据集上训练的RE模型|1.4G|[推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_re_pretrained.tar) | -更多模型下载，可以参考 [模型库](./docs/model_list.md) +更多模型下载，可以参考 [PPOCR model_list](../doc/doc_en/models_list.md) and [PPStructure model_list](./docs/model_list.md) \ No newline at end of file diff --git a/ppstructure/predict_system.py b/ppstructure/predict_system.py index e87499ccc410ae67a170f63301e5a99ef948b161..3f3dc65875a20b3f66403afecfd60f04e3d83d61 100644 --- a/ppstructure/predict_system.py +++ b/ppstructure/predict_system.py @@ -30,7 +30,6 @@ from ppocr.utils.utility import get_image_file_list, check_and_read_gif from ppocr.utils.logging import get_logger from tools.infer.predict_system import TextSystem from ppstructure.table.predict_table import TableSystem, to_excel -from ppstructure.vqa.infer_ser_e2e import SerPredictor, draw_ser_results from ppstructure.utility import parse_args, draw_structure_result logger = get_logger() @@ -66,6 +65,7 @@ class OCRSystem(object): self.use_angle_cls = args.use_angle_cls self.drop_score = args.drop_score elif self.mode == 'vqa': + from ppstructure.vqa.infer_ser_e2e import SerPredictor, draw_ser_results self.vqa_engine = SerPredictor(args) def __call__(self, img): diff --git a/ppstructure/vqa/README.md b/ppstructure/vqa/README.md index 529690cc83d282f765fe89ac0c10f348803ac92f..36f2e6aad068007624c1863ab1e7c4afca680189 100644 --- a/ppstructure/vqa/README.md +++ b/ppstructure/vqa/README.md @@ -62,7 +62,7 @@ PP-Structure 里的 DOC-VQA算法基于PaddleNLP自然语言处理算法库进 - **（1) 安装PaddlePaddle** ```bash -pip3 install --upgrade pip +python3 -m pip install --upgrade pip # GPU安装 python3 -m pip install paddlepaddle-gpu==2.2 -i https://mirror.baidu.com/pypi/simple @@ -79,7 +79,7 @@ python3 -m pip install paddlepaddle==2.2 -i https://mirror.baidu.com/pypi/simple - **（1）pip快速安装PaddleOCR whl包（仅预测）** ```bash -pip install paddleocr +python3 -m pip install paddleocr ``` - **（2）下载VQA源码（预测+训练）** @@ -93,18 +93,11 @@ git clone https://gitee.com/paddlepaddle/PaddleOCR # 注：码云托管代码可能无法实时同步本github项目更新，存在3~5天延时，请优先使用推荐方式。 ``` -- **（3）安装PaddleNLP** - -```bash -pip3 install "paddlenlp>=2.2.1" -``` - - - **（4）安装VQA的`requirements`** ```bash cd ppstructure/vqa -pip install -r requirements.txt +python3 -m pip install -r requirements.txt ``` ## 4. 使用 @@ -131,7 +124,7 @@ wget https://paddleocr.bj.bcebos.com/dataset/XFUND.tar * 启动训练 ```shell -python3.7 train_ser.py \ +python3 train_ser.py \ --model_name_or_path "layoutxlm-base-uncased" \ --ser_model_type "LayoutXLM" \ --train_data_dir "XFUND/zh_train/image" \ @@ -152,7 +145,7 @@ python3.7 train_ser.py \ * 恢复训练 ```shell -python3.7 train_ser.py \ +python3 train_ser.py \ --model_name_or_path "model_path" \ --ser_model_type "LayoutXLM" \ --train_data_dir "XFUND/zh_train/image" \ @@ -189,7 +182,7 @@ python3 eval_ser.py \ ```shell export CUDA_VISIBLE_DEVICES=0 -python3.7 infer_ser.py \ +python3 infer_ser.py \ --model_name_or_path "PP-Layout_v1.0_ser_pretrained/" \ --ser_model_type "LayoutXLM" \ --output_dir "output/ser/" \ @@ -203,7 +196,7 @@ python3.7 infer_ser.py \ ```shell export CUDA_VISIBLE_DEVICES=0 -python3.7 infer_ser_e2e.py \ +python3 infer_ser_e2e.py \ --model_name_or_path "PP-Layout_v1.0_ser_pretrained/" \ --ser_model_type "LayoutXLM" \ --max_seq_length 512 \ @@ -215,7 +208,7 @@ python3.7 infer_ser_e2e.py \ ```shell export CUDA_VISIBLE_DEVICES=0 -python3.7 helper/eval_with_label_end2end.py --gt_json_path XFUND/zh_val/xfun_normalize_val.json --pred_json_path output_res/infer_results.txt +python3 helper/eval_with_label_end2end.py --gt_json_path XFUND/zh_val/xfun_normalize_val.json --pred_json_path output_res/infer_results.txt ``` @@ -310,7 +303,7 @@ python3 infer_re.py \ ```shell export CUDA_VISIBLE_DEVICES=0 -python3.7 infer_ser_re_e2e.py \ +python3 infer_ser_re_e2e.py \ --model_name_or_path "PP-Layout_v1.0_ser_pretrained/" \ --re_model_name_or_path "PP-Layout_v1.0_re_pretrained/" \ --ser_model_type "LayoutXLM" \ diff --git a/ppstructure/vqa/requirements.txt b/ppstructure/vqa/requirements.txt index 9c935ae619024c9f47ced820eae35a3a1c976953..0042ec0baedcc3e7bbecb922d10b93c95219219d 100644 --- a/ppstructure/vqa/requirements.txt +++ b/ppstructure/vqa/requirements.txt @@ -1,3 +1,4 @@ sentencepiece yacs -seqeval \ No newline at end of file +seqeval +paddlenlp>=2.2.1 \ No newline at end of file diff --git a/test_tipc/configs/ch_PP-OCRv2_det_KL/model_linux_gpu_normal_normal_infer_python_linux_gpu_cpu.txt b/test_tipc/configs/ch_PP-OCRv2_det_KL/model_linux_gpu_normal_normal_infer_python_linux_gpu_cpu.txt index 3aa1e71306069222b17b642ecb6539730f2d1337..1aad65b687992155133ed11533a14f642510361d 100644 --- a/test_tipc/configs/ch_PP-OCRv2_det_KL/model_linux_gpu_normal_normal_infer_python_linux_gpu_cpu.txt +++ b/test_tipc/configs/ch_PP-OCRv2_det_KL/model_linux_gpu_normal_normal_infer_python_linux_gpu_cpu.txt @@ -3,7 +3,7 @@ model_name:PPOCRv2_ocr_det_kl python:python3.7 Global.pretrained_model:null Global.save_inference_dir:null -infer_model:./inference/ch_PP-OCRv2_det_infer +infer_model:./inference/ch_PP-OCRv2_det_infer/ infer_export:deploy/slim/quantization/quant_kl.py -c configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml -o infer_quant:True inference:tools/infer/predict_det.py diff --git a/test_tipc/prepare.sh b/test_tipc/prepare.sh index 4f703832aada798d57f397f81139dcf49da07e71..570c6832e7a6682f634d7ab7538a228256446372 100644 --- a/test_tipc/prepare.sh +++ b/test_tipc/prepare.sh @@ -248,6 +248,8 @@ if [ ${MODE} = "klquant_whole_infer" ]; then if [ ${model_name} = "PPOCRv2_ocr_rec_kl" ]; then wget -nc -P ./inference https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar --no-check-certificate wget -nc -P ./inference/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/test/rec_inference.tar --no-check-certificate + wget -nc -P ./train_data/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/test/ic15_data.tar --no-check-certificate + cd ./train_data/ && tar xf ic15_data.tar && cd ../ cd ./inference && tar xf rec_inference.tar && tar xf ch_PP-OCRv2_rec_infer.tar && cd ../ fi if [ ${model_name} = "PPOCRv2_ocr_det_kl" ]; then