Merge branch 'dygraph' of https://github.com/PaddlePaddle/PaddleOCR into tipc

applications/发票关键信息抽取.md

+
+# 基于VI-LayoutXLM的发票关键信息抽取
+
+- [1. 项目背景及意义](#1-项目背景及意义)
+- [2. 项目内容](#2-项目内容)
+- [3. 安装环境](#3-安装环境)
+- [4. 关键信息抽取](#4-关键信息抽取)
+  - [4.1 文本检测](#41-文本检测)
+  - [4.2 文本识别](#42-文本识别)
+  - [4.3 语义实体识别](#43-语义实体识别)
+  - [4.4 关系抽取](#44-关系抽取)
+
+
+
+## 1. 项目背景及意义
+
+关键信息抽取在文档场景中被广泛使用，如身份证中的姓名、住址信息抽取，快递单中的姓名、联系方式等关键字段内容的抽取。传统基于模板匹配的方案需要针对不同的场景制定模板并进行适配，较为繁琐，不够鲁棒。基于该问题，我们借助飞桨提供的PaddleOCR套件中的关键信息抽取方案，实现对增值税发票场景的关键信息抽取。
+
+## 2. 项目内容
+
+本项目基于PaddleOCR开源套件，以VI-LayoutXLM多模态关键信息抽取模型为基础，针对增值税发票场景进行适配，提取该场景的关键信息。
+
+## 3. 安装环境
+
+```bash
+# 首先git官方的PaddleOCR项目，安装需要的依赖
+# 第一次运行打开该注释
+git clone https://gitee.com/PaddlePaddle/PaddleOCR.git
+cd PaddleOCR
+# 安装PaddleOCR的依赖
+pip install -r requirements.txt
+# 安装关键信息抽取任务的依赖
+pip install -r ./ppstructure/vqa/requirements.txt
+```
+
+## 4. 关键信息抽取
+
+基于文档图像的关键信息抽取包含3个部分：（1）文本检测（2）文本识别（3）关键信息抽取方法，包括语义实体识别或者关系抽取，下面分别进行介绍。
+
+### 4.1 文本检测
+
+
+本文重点关注发票的关键信息抽取模型训练与预测过程，因此在关键信息抽取过程中，直接使用标注的文本检测与识别标注信息进行测试，如果你希望自定义该场景的文本检测模型，完成端到端的关键信息抽取部分，请参考[文本检测模型训练教程](../doc/doc_ch/detection.md)，按照训练数据格式准备数据，并完成该场景下垂类文本检测模型的微调过程。
+
+
+### 4.2 文本识别
+
+本文重点关注发票的关键信息抽取模型训练与预测过程，因此在关键信息抽取过程中，直接使用提供的文本检测与识别标注信息进行测试，如果你希望自定义该场景的文本检测模型，完成端到端的关键信息抽取部分，请参考[文本识别模型训练教程](../doc/doc_ch/recognition.md)，按照训练数据格式准备数据，并完成该场景下垂类文本识别模型的微调过程。
+
+### 4.3 语义实体识别 （Semantic Entity Recognition）
+
+语义实体识别指的是给定一段文本行，确定其类别（如`姓名`、`住址`等类别）。PaddleOCR中提供了基于VI-LayoutXLM的多模态语义实体识别方法，融合文本、位置与版面信息，相比LayoutXLM多模态模型，去除了其中的视觉骨干网络特征提取部分，引入符合阅读顺序的文本行排序方法，同时使用UDML联合互蒸馏方法进行训练，最终在精度与速度方面均超越LayoutXLM。更多关于VI-LayoutXLM的算法介绍与精度指标，请参考：[VI-LayoutXLM算法介绍](../doc/doc_ch/algorithm_kie_vi_layoutxlm.md)。
+
+#### 4.3.1 准备数据
+
+发票场景为例，我们首先需要标注出其中的关键字段，我们将其标注为`问题-答案`的key-value pair，如下，编号No为12270830，则`No`字段标注为question，`12270830`字段标注为answer。如下图所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185381131-76b6e260-04fe-46d9-baca-6bdd7fe0d0ce.jpg" width="800">
+</div>
+
+**注意：**
+
+* 如果文本检测模型数据标注过程中，没有标注 **非关键信息内容** 的检测框，那么在标注关键信息抽取任务的时候，也不需要标注该部分，如上图所示；如果标注的过程，如果同时标注了**非关键信息内容** 的检测框，那么我们需要将该部分的label记为other。
+* 标注过程中，需要以文本行为单位进行标注，无需标注单个字符的位置信息。
+
+
+已经处理好的增值税发票数据集从这里下载：[增值税发票数据集下载链接](https://aistudio.baidu.com/aistudio/datasetdetail/165561)。
+
+下载好发票数据集，并解压在train_data目录下，目录结构如下所示。
+
+```
+train_data
+  |--zzsfp
+       |---class_list.txt
+       |---imgs/
+       |---train.json
+       |---val.json
+```
+
+其中`class_list.txt`是包含`other`, `question`, `answer`，3个种类的的类别列表（不区分大小写），`imgs`目录底下，`train.json`与`val.json`分别表示训练与评估集合的标注文件。训练集中包含30张图片，验证集中包含8张图片。部分标注如下所示。
+
+```py
+b33.jpg [{"transcription": "No", "label": "question", "points": [[2882, 472], [3026, 472], [3026, 588], [2882, 588]], }, {"transcription": "12269563", "label": "answer", "points": [[3066, 448], [3598, 448], [3598, 576], [3066, 576]], ]}]
+```
+
+相比于OCR检测的标注，仅多了`label`字段。
+
+
+#### 4.3.2 开始训练
+
+
+VI-LayoutXLM的配置为[ser_vi_layoutxlm_xfund_zh_udml.yml](../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)，需要修改数据、类别数目以及配置文件。
+
+```yml
+Architecture:
+  model_type: &model_type "vqa"
+  name: DistillationModel
+  algorithm: Distillation
+  Models:
+    Teacher:
+      pretrained:
+      freeze_params: false
+      return_all_feats: true
+      model_type: *model_type
+      algorithm: &algorithm "LayoutXLM"
+      Transform:
+      Backbone:
+        name: LayoutXLMForSer
+        pretrained: True
+        # one of base or vi
+        mode: vi
+        checkpoints:
+        # 定义类别数目
+        num_classes: &num_classes 5
+   ...
+
+PostProcess:
+  name: DistillationSerPostProcess
+  model_name: ["Student", "Teacher"]
+  key: backbone_out
+  # 定义类别文件
+  class_path: &class_path train_data/zzsfp/class_list.txt
+
+Train:
+  dataset:
+    name: SimpleDataSet
+    # 定义训练数据目录与标注文件
+    data_dir: train_data/zzsfp/imgs
+    label_file_list:
+      - train_data/zzsfp/train.json
+  ...
+
+Eval:
+  dataset:
+    # 定义评估数据目录与标注文件
+    name: SimpleDataSet
+    data_dir: train_data/zzsfp/imgs
+    label_file_list:
+      - train_data/zzsfp/val.json
+  ...
+```
+
+LayoutXLM与VI-LayoutXLM针对该场景的训练结果如下所示。
+
+| 模型 | 迭代轮数 | Hmean |
+| :---: | :---: | :---: |
+| LayoutXLM | 50 | 100% |
+| VI-LayoutXLM | 50 | 100% |
+
+可以看出，由于当前数据量较少，场景比较简单，因此2个模型的Hmean均达到了100%。
+
+
+#### 4.3.3 模型评估
+
+模型训练过程中，使用的是知识蒸馏的策略，最终保留了学生模型的参数，在评估时，我们需要针对学生模型的配置文件进行修改: [ser_vi_layoutxlm_xfund_zh.yml](../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml)，修改内容与训练配置相同，包括**类别数、类别映射文件、数据目录**。
+
+修改完成后，执行下面的命令完成评估过程。
+
+```bash
+# 注意：需要根据你的配置文件地址与保存的模型地址，对评估命令进行修改
+python3 tools/eval.py -c ./fapiao/ser_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy
+```
+
+输出结果如下所示。
+
+```
+[2022/08/18 08:49:58] ppocr INFO: metric eval ***************
+[2022/08/18 08:49:58] ppocr INFO: precision:1.0
+[2022/08/18 08:49:58] ppocr INFO: recall:1.0
+[2022/08/18 08:49:58] ppocr INFO: hmean:1.0
+[2022/08/18 08:49:58] ppocr INFO: fps:1.9740402401574881
+```
+
+#### 4.3.4 模型预测
+
+使用下面的命令进行预测。
+
+```bash
+python3 tools/infer_vqa_token_ser.py -c fapiao/ser_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False
+```
+
+预测结果会保存在配置文件中的`Global.save_res_path`目录中。
+
+部分预测结果如下所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185310636-6ce02f7c-790d-479f-b163-ea97a5a04808.jpg" width="800">
+</div>
+
+
+* 注意：在预测时，使用的文本检测与识别结果为标注的结果，直接从json文件里面进行读取。
+
+如果希望使用OCR引擎结果得到的结果进行推理，则可以使用下面的命令进行推理。
+
+
+```bash
+python3 tools/infer_vqa_token_ser.py -c fapiao/ser_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy Global.infer_img=./train_data/zzsfp/imgs/b25.jpg Global.infer_mode=True
+```
+
+结果如下所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185384321-61153faa-e407-45c4-8e7c-a39540248189.jpg" width="800">
+</div>
+
+它会使用PP-OCRv3的文本检测与识别模型进行获取文本位置与内容信息。
+
+可以看出，由于训练的过程中，没有标注额外的字段为other类别，所以大多数检测出来的字段被预测为question或者answer。
+
+如果希望构建基于你在垂类场景训练得到的OCR检测与识别模型，可以使用下面的方法传入检测与识别的inference 模型路径，即可完成OCR文本检测与识别以及SER的串联过程。
+
+```bash
+python3 tools/infer_vqa_token_ser.py -c fapiao/ser_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy Global.infer_img=./train_data/zzsfp/imgs/b25.jpg Global.infer_mode=True Global.kie_rec_model_dir="your_rec_model" Global.kie_det_model_dir="your_det_model"
+```
+
+### 4.4 关系抽取（Relation Extraction）
+
+使用SER模型，可以获取图像中所有的question与answer的字段，继续这些字段的类别，我们需要进一步获取question与answer之间的连接，因此需要进一步训练关系抽取模型，解决该问题。本文也基于VI-LayoutXLM多模态预训练模型，进行下游RE任务的模型训练。
+
+#### 4.4.1 准备数据
+
+以发票场景为例，相比于SER任务，RE中还需要标记每个文本行的id信息以及链接关系linking，如下所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185387870-dc9125a0-9ceb-4036-abf3-184b6e65dc7d.jpg" width="800">
+</div>
+
+
+标注文件的部分内容如下所示。
+
+```py
+b33.jpg [{"transcription": "No", "label": "question", "points": [[2882, 472], [3026, 472], [3026, 588], [2882, 588]], "id": 0, "linking": [[0, 1]]}, {"transcription": "12269563", "label": "answer", "points": [[3066, 448], [3598, 448], [3598, 576], [3066, 576]], "id": 1, "linking": [[0, 1]]}]
+```
+
+相比与SER的标注，多了`id`与`linking`的信息，分别表示唯一标识以及连接关系。
+
+已经处理好的增值税发票数据集从这里下载：[增值税发票数据集下载链接](https://aistudio.baidu.com/aistudio/datasetdetail/165561)。
+
+#### 4.4.2 开始训练
+
+基于VI-LayoutXLM的RE任务配置为[re_vi_layoutxlm_xfund_zh_udml.yml](../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)，需要修改**数据路径、类别列表文件**。
+
+```yml
+Train:
+  dataset:
+    name: SimpleDataSet
+    # 定义训练数据目录与标注文件
+    data_dir: train_data/zzsfp/imgs
+    label_file_list:
+      - train_data/zzsfp/train.json
+    transforms:
+      - DecodeImage: # load image
+          img_mode: RGB
+          channel_first: False
+      - VQATokenLabelEncode: # Class handling label
+          contains_re: True
+          algorithm: *algorithm
+          class_path: &class_path train_data/zzsfp/class_list.txt
+  ...
+
+Eval:
+  dataset:
+    # 定义评估数据目录与标注文件
+    name: SimpleDataSet
+    data_dir: train_data/zzsfp/imgs
+    label_file_list:
+      - train_data/zzsfp/val.json
+  ...
+
+```
+
+LayoutXLM与VI-LayoutXLM针对该场景的训练结果如下所示。
+
+| 模型 | 迭代轮数 | Hmean |
+| :---: | :---: | :---: |
+| LayoutXLM | 50 | 98.0% |
+| VI-LayoutXLM | 50 | 99.3% |
+
+可以看出，对于VI-LayoutXLM相比LayoutXLM的Hmean高了1.3%。
+
+
+#### 4.4.3 模型评估
+
+模型训练过程中，使用的是知识蒸馏的策略，最终保留了学生模型的参数，在评估时，我们需要针对学生模型的配置文件进行修改: [re_vi_layoutxlm_xfund_zh.yml](../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml)，修改内容与训练配置相同，包括**类别映射文件、数据目录**。
+
+修改完成后，执行下面的命令完成评估过程。
+
+```bash
+# 注意：需要根据你的配置文件地址与保存的模型地址，对评估命令进行修改
+python3 tools/eval.py -c ./fapiao/re_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/re_vi_layoutxlm_fapiao_udml/best_accuracy
+```
+
+输出结果如下所示。
+
+```py
+[2022/08/18 12:17:14] ppocr INFO: metric eval ***************
+[2022/08/18 12:17:14] ppocr INFO: precision:1.0
+[2022/08/18 12:17:14] ppocr INFO: recall:0.9873417721518988
+[2022/08/18 12:17:14] ppocr INFO: hmean:0.9936305732484078
+[2022/08/18 12:17:14] ppocr INFO: fps:2.765963539771157
+```
+
+#### 4.4.4 模型预测
+
+使用下面的命令进行预测。
+
+```bash
+# -c 后面的是RE任务的配置文件
+# -o 后面的字段是RE任务的配置
+# -c_ser 后面的是SER任务的配置文件
+# -c_ser 后面的字段是SER任务的配置
+python3 tools/infer_vqa_token_ser_re.py -c fapiao/re_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/re_vi_layoutxlm_fapiao_udml/best_accuracy Global.infer_img=./train_data/zzsfp/val.json Global.infer_mode=False -c_ser fapiao/ser_vi_layoutxlm.yml -o_ser Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy
+```
+
+预测结果会保存在配置文件中的`Global.save_res_path`目录中。
+
+部分预测结果如下所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185393805-c67ff571-cf7e-4217-a4b0-8b396c4f22bb.jpg" width="800">
+</div>
+
+
+* 注意：在预测时，使用的文本检测与识别结果为标注的结果，直接从json文件里面进行读取。
+
+如果希望使用OCR引擎结果得到的结果进行推理，则可以使用下面的命令进行推理。
+
+```bash
+python3 tools/infer_vqa_token_ser_re.py -c fapiao/re_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/re_vi_layoutxlm_fapiao_udml/best_accuracy Global.infer_img=./train_data/zzsfp/val.json Global.infer_mode=True -c_ser fapiao/ser_vi_layoutxlm.yml -o_ser Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy
+```
+
+如果希望构建基于你在垂类场景训练得到的OCR检测与识别模型，可以使用下面的方法传入，即可完成SER + RE的串联过程。
+
+```bash
+python3 tools/infer_vqa_token_ser_re.py -c fapiao/re_vi_layoutxlm.yml -o Architecture.Backbone.checkpoints=fapiao/models/re_vi_layoutxlm_fapiao_udml/best_accuracy Global.infer_img=./train_data/zzsfp/val.json Global.infer_mode=True -c_ser fapiao/ser_vi_layoutxlm.yml -o_ser Architecture.Backbone.checkpoints=fapiao/models/ser_vi_layoutxlm_fapiao_udml/best_accuracy Global.kie_rec_model_dir="your_rec_model" Global.kie_det_model_dir="your_det_model"
+```

configs/kie/layoutlm_series/re_layoutlmv2_xfund_zh.yml

@@ -11,11 +11,11 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_21.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_21.jpg
   save_res_path: ./output/re_layoutlmv2_xfund_zh/res/
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutLMv2"
   Transform:
   Backbone:

configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml

@@ -11,11 +11,11 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_21.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_21.jpg
   save_res_path: ./output/re_layoutxlm_xfund_zh/res/
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutXLM"
   Transform:
   Backbone:

configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml

@@ -11,11 +11,11 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_42.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_42.jpg
   save_res_path: ./output/re_layoutlm_xfund_zh/res
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutLM"
   Transform:
   Backbone:

configs/kie/layoutlm_series/ser_layoutlmv2_xfund_zh.yml

@@ -11,11 +11,11 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_42.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_42.jpg
   save_res_path: ./output/ser_layoutlmv2_xfund_zh/res/
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutLMv2"
   Transform:
   Backbone:

configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml

@@ -11,11 +11,11 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_42.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_42.jpg
   save_res_path: ./output/ser_layoutxlm_xfund_zh/res
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutXLM"
   Transform:
   Backbone:

configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml

@@ -11,11 +11,13 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_21.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_21.jpg
   save_res_path: ./output/re/xfund_zh/with_gt
+  kie_rec_model_dir: 
+  kie_det_model_dir:
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutXLM"
   Transform:
   Backbone:

configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml

@@ -11,11 +11,11 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_21.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_21.jpg
   save_res_path: ./output/re/xfund_zh/with_gt
 
 Architecture:
-  model_type: &model_type "vqa"
+  model_type: &model_type "kie"
   name: DistillationModel
   algorithm: Distillation
   Models:

configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml

@@ -11,16 +11,18 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_42.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_42.jpg
   # if you want to predict using the groundtruth ocr info,
   # you can use the following config
   # infer_img: train_data/XFUND/zh_val/val.json
   # infer_mode: False
 
   save_res_path: ./output/ser/xfund_zh/res
+  kie_rec_model_dir: 
+  kie_det_model_dir:
 
 Architecture:
-  model_type: vqa
+  model_type: kie
   algorithm: &algorithm "LayoutXLM"
   Transform:
   Backbone:

configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml

@@ -11,12 +11,12 @@ Global:
   save_inference_dir:
   use_visualdl: False
   seed: 2022
-  infer_img: ppstructure/docs/vqa/input/zh_val_42.jpg
+  infer_img: ppstructure/docs/kie/input/zh_val_42.jpg
   save_res_path: ./output/ser_layoutxlm_xfund_zh/res
 
 
 Architecture:
-  model_type: &model_type "vqa"
+  model_type: &model_type "kie"
   name: DistillationModel
   algorithm: Distillation
   Models:

configs/table/SLANet_ch.yml

+Global:
+  use_gpu: True
+  epoch_num: 400
+  log_smooth_window: 20
+  print_batch_step: 20
+  save_model_dir: ./output/SLANet_ch
+  save_epoch_step: 400
+  # evaluation is run every 331 iterations after the 0th iteration
+  eval_batch_step: [0, 331]
+  cal_metric_during_train: True
+  pretrained_model: 
+  checkpoints: 
+  save_inference_dir: ./output/SLANet_ch/infer
+  use_visualdl: False
+  infer_img: doc/table/table.jpg
+  # for data or label process
+  character_dict_path: ppocr/utils/dict/table_structure_dict_ch.txt
+  character_type: en
+  max_text_length: &max_text_length 500
+  box_format: &box_format xyxyxyxy # 'xywh', 'xyxy', 'xyxyxyxy'
+  infer_mode: False
+  use_sync_bn: True
+  save_res_path: output/infer
+
+Optimizer:
+  name: Adam
+  beta1: 0.9
+  beta2: 0.999
+  clip_norm: 5.0
+  lr:
+    learning_rate: 0.001
+  regularizer:
+    name: 'L2'
+    factor: 0.00000
+
+Architecture:
+  model_type: table
+  algorithm: SLANet
+  Backbone:
+    name: PPLCNet
+    scale: 1.0
+    pretrained: True
+    use_ssld: True
+  Neck:
+    name: CSPPAN
+    out_channels: 96
+  Head:
+    name: SLAHead
+    hidden_size: 256
+    max_text_length: *max_text_length
+    loc_reg_num: &loc_reg_num 8
+
+Loss:
+  name: SLALoss
+  structure_weight: 1.0
+  loc_weight: 2.0
+  loc_loss: smooth_l1
+
+PostProcess:
+  name: TableLabelDecode
+  merge_no_span_structure: &merge_no_span_structure True
+
+Metric:
+  name: TableMetric
+  main_indicator: acc
+  compute_bbox_metric: False
+  loc_reg_num: *loc_reg_num
+  box_format: *box_format
+  del_thead_tbody: True
+
+Train:
+  dataset:
+    name: PubTabDataSet
+    data_dir: train_data/table/train/
+    label_file_list: [train_data/table/train.txt]
+    transforms:
+      - DecodeImage:
+          img_mode: BGR
+          channel_first: False
+      - TableLabelEncode:
+          learn_empty_box: False
+          merge_no_span_structure: *merge_no_span_structure
+          replace_empty_cell_token: False
+          loc_reg_num: *loc_reg_num
+          max_text_length: *max_text_length
+      - TableBoxEncode:
+          in_box_format: *box_format
+          out_box_format: *box_format
+      - ResizeTableImage:
+          max_len: 488
+      - NormalizeImage:
+          scale: 1./255.
+          mean: [0.485, 0.456, 0.406]
+          std: [0.229, 0.224, 0.225]
+          order: 'hwc'
+      - PaddingTableImage:
+          size: [488, 488]
+      - ToCHWImage:
+      - KeepKeys:
+          keep_keys: [ 'image', 'structure', 'bboxes', 'bbox_masks', 'shape' ]
+  loader:
+    shuffle: True
+    batch_size_per_card: 48
+    drop_last: True
+    num_workers: 1
+
+Eval:
+  dataset:
+    name: PubTabDataSet
+     data_dir: train_data/table/val/
+    label_file_list: [train_data/table/val.txt]
+    transforms:
+      - DecodeImage:
+          img_mode: BGR
+          channel_first: False
+      - TableLabelEncode:
+          learn_empty_box: False
+          merge_no_span_structure: *merge_no_span_structure
+          replace_empty_cell_token: False
+          loc_reg_num: *loc_reg_num
+          max_text_length: *max_text_length
+      - TableBoxEncode:
+          in_box_format: *box_format
+          out_box_format: *box_format
+      - ResizeTableImage:
+          max_len: 488
+      - NormalizeImage:
+          scale: 1./255.
+          mean: [0.485, 0.456, 0.406]
+          std: [0.229, 0.224, 0.225]
+          order: 'hwc'
+      - PaddingTableImage:
+          size: [488, 488]
+      - ToCHWImage:
+      - KeepKeys:
+          keep_keys: [ 'image', 'structure', 'bboxes', 'bbox_masks', 'shape' ]
+  loader:
+    shuffle: False
+    drop_last: False
+    batch_size_per_card: 48
+    num_workers: 1

configs/table/table_mv3.yml

@@ -17,7 +17,7 @@ Global:
   # for data or label process
   character_dict_path: ppocr/utils/dict/table_structure_dict.txt
   character_type: en
-  max_text_length: &max_text_length 800
+  max_text_length: &max_text_length 500
   box_format: &box_format 'xyxy' # 'xywh', 'xyxy', 'xyxyxyxy'
   infer_mode: False
 
@@ -38,7 +38,8 @@ Architecture:
   Backbone:
     name: MobileNetV3
     scale: 1.0
-    model_name: large
+    model_name: small
+    disable_se: true
   Head:
     name: TableAttentionHead
     hidden_size: 256
@@ -89,7 +90,7 @@ Train:
           keep_keys: [ 'image', 'structure', 'bboxes', 'bbox_masks', 'shape' ]
   loader:
     shuffle: True
-    batch_size_per_card: 32
+    batch_size_per_card: 48
     drop_last: True
     num_workers: 1
 
@@ -124,5 +125,5 @@ Eval:
   loader:
     shuffle: False
     drop_last: False
-    batch_size_per_card: 16
+    batch_size_per_card: 48
     num_workers: 1

deploy/android_demo/app/src/main/cpp/native.cpp

@@ -47,7 +47,7 @@ str_to_cpu_mode(const std::string &cpu_mode) {
   std::string upper_key;
   std::transform(cpu_mode.cbegin(), cpu_mode.cend(), upper_key.begin(),
                  ::toupper);
-  auto index = cpu_mode_map.find(upper_key);
+  auto index = cpu_mode_map.find(upper_key.c_str());
   if (index == cpu_mode_map.end()) {
     LOGE("cpu_mode not found %s", upper_key.c_str());
     return paddle::lite_api::LITE_POWER_HIGH;
@@ -116,4 +116,4 @@ Java_com_baidu_paddle_lite_demo_ocr_OCRPredictorNative_release(
   ppredictor::OCR_PPredictor *ppredictor =
       (ppredictor::OCR_PPredictor *)java_pointer;
   delete ppredictor;
-}
\ No newline at end of file
+}

deploy/android_demo/app/src/main/java/com/baidu/paddle/lite/demo/ocr/OCRPredictorNative.java

@@ -54,7 +54,7 @@ public class OCRPredictorNative {
     }
 
     public void destory() {
-        if (nativePointer > 0) {
+        if (nativePointer != 0) {
             release(nativePointer);
             nativePointer = 0;
         }

deploy/cpp_infer/docs/windows_vs2019_build.md

@@ -109,8 +109,10 @@ CUDA_LIB、CUDNN_LIB、TENSORRT_DIR、WITH_GPU、WITH_TENSORRT
 
 运行之前，将下面文件拷贝到`build/Release/`文件夹下
 1. `paddle_inference/paddle/lib/paddle_inference.dll`
-2. `opencv/build/x64/vc15/bin/opencv_world455.dll`
-3. 如果使用openblas版本的预测库还需要拷贝 `paddle_inference/third_party/install/openblas/lib/openblas.dll`
+2. `paddle_inference/third_party/install/onnxruntime/lib/onnxruntime.dll`
+3. `paddle_inference/third_party/install/paddle2onnx/lib/paddle2onnx.dll`
+4. `opencv/build/x64/vc15/bin/opencv_world455.dll`
+5. 如果使用openblas版本的预测库还需要拷贝 `paddle_inference/third_party/install/openblas/lib/openblas.dll`
 
 ### Step4: 预测
 

deploy/slim/quantization/README_en.md

@@ -73,4 +73,4 @@ python deploy/slim/quantization/export_model.py -c configs/det/ch_ppocr_v2.0/ch_
 The numerical range of the quantized model parameters derived from the above steps is still FP32, but the numerical range of the parameters is int8.
 The derived model can be converted through the `opt tool` of PaddleLite.
 
-For quantitative model deployment, please refer to [Mobile terminal model deployment](../../lite/readme_en.md)
+For quantitative model deployment, please refer to [Mobile terminal model deployment](../../lite/readme.md)

doc/doc_ch/algorithm_kie_layoutxlm.md

@@ -66,10 +66,10 @@ LayoutXLM模型基于SER任务进行推理，可以执行如下命令：
 
 ```bash
 cd ppstructure
-python3 vqa/predict_vqa_token_ser.py \
-  --vqa_algorithm=LayoutXLM \
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
   --ser_model_dir=../inference/ser_layoutxlm_infer \
-  --image_dir=./docs/vqa/input/zh_val_42.jpg \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
   --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
   --vis_font_path=../doc/fonts/simfang.ttf
 ```
@@ -77,7 +77,7 @@ python3 vqa/predict_vqa_token_ser.py \
 SER可视化结果默认保存到`./output`文件夹里面，结果示例如下：
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_ser/zh_val_42_ser.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
 </div>
 
 

doc/doc_ch/algorithm_kie_vi_layoutxlm.md

@@ -59,10 +59,10 @@ VI-LayoutXLM模型基于SER任务进行推理，可以执行如下命令：
 
 ```bash
 cd ppstructure
-python3 vqa/predict_vqa_token_ser.py \
-  --vqa_algorithm=LayoutXLM \
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
   --ser_model_dir=../inference/ser_vi_layoutxlm_infer \
-  --image_dir=./docs/vqa/input/zh_val_42.jpg \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
   --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
   --vis_font_path=../doc/fonts/simfang.ttf \
   --ocr_order_method="tb-yx"
@@ -71,7 +71,7 @@ python3 vqa/predict_vqa_token_ser.py \
 SER可视化结果默认保存到`./output`文件夹里面，结果示例如下：
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_ser/zh_val_42_ser.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
 </div>
 
 

doc/doc_ch/dataset/kie_datasets.md

-# 信息抽取数据集
+# 关键信息抽取数据集
 
 这里整理了常见的DocVQA数据集，持续更新中，欢迎各位小伙伴贡献数据集～
+
 - [FUNSD数据集](#funsd)
 - [XFUND数据集](#xfund)
 - [wildreceipt数据集](#wildreceipt)

doc/doc_ch/kie.md

@@ -64,7 +64,7 @@ zh_train_1.jpg   [{"transcription": "中国人体器官捐献", "label": "other"
 
 验证集构建方式与训练集相同。
 
-* 字典文件
+**（3）字典文件**
 
 训练集与验证集中的文本行包含标签信息，所有标签的列表存在字典文件中（如`class_list.txt`），字典文件中的每一行表示为一个类别名称。
 
@@ -103,7 +103,7 @@ HEADER
 
 ## 1.3. 数据下载
 
-如果你没有本地数据集，可以从[XFUND](https://github.com/doc-analysis/XFUND)或者[FUNSD](https://guillaumejaume.github.io/FUNSD/)官网下载数据，然后使用XFUND与FUNSD的处理脚本([XFUND](../../ppstructure/vqa/tools/trans_xfun_data.py), [FUNSD](../../ppstructure/vqa/tools/trans_funsd_label.py))，生成用于PaddleOCR训练的数据格式，并使用公开数据集快速体验关键信息抽取的流程。
+如果你没有本地数据集，可以从[XFUND](https://github.com/doc-analysis/XFUND)或者[FUNSD](https://guillaumejaume.github.io/FUNSD/)官网下载数据，然后使用XFUND与FUNSD的处理脚本([XFUND](../../ppstructure/kie/tools/trans_xfun_data.py), [FUNSD](../../ppstructure/kie/tools/trans_funsd_label.py))，生成用于PaddleOCR训练的数据格式，并使用公开数据集快速体验关键信息抽取的流程。
 
 更多关于公开数据集的介绍，请参考[关键信息抽取数据集说明文档](./dataset/kie_datasets.md)。
 
@@ -209,7 +209,7 @@ Architecture:
     num_classes: &num_classes 7
 
 PostProcess:
-  name: VQASerTokenLayoutLMPostProcess
+  name: kieSerTokenLayoutLMPostProcess
   # 修改字典文件的路径为你自定义的数据集的字典路径
   class_path: &class_path train_data/XFUND/class_list_xfun.txt
 
@@ -347,25 +347,25 @@ output/ser_vi_layoutxlm_xfund_zh/
 
 
 ```bash
-python3 tools/infer_vqa_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./ppstructure/docs/vqa/input/zh_val_42.jpg
+python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
 ```
 
 预测图片如下所示，图片会存储在`Global.save_res_path`路径中。
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_ser/zh_val_42_ser.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
 </div>
 
 预测过程中，默认会加载PP-OCRv3的检测识别模型，用于OCR的信息抽取，如果希望加载预先获取的OCR结果，可以使用下面的方式进行预测，指定`Global.infer_img`为标注文件，其中包含图片路径以及OCR信息，同时指定`Global.infer_mode`为False，表示此时不使用OCR预测引擎。
 
 ```bash
-python3 tools/infer_vqa_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False
+python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False
 ```
 
 对于上述图片，如果使用标注的OCR结果进行信息抽取，预测结果如下。
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_ser_with_gt_ocr/zh_val_42_ser.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_ser_with_gt_ocr/zh_val_42_ser.jpg" width="800">
 </div>
 
 可以看出，部分检测框信息更加准确，但是整体信息抽取识别结果基本一致。
@@ -375,20 +375,26 @@ python3 tools/infer_vqa_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxl
 
 
 ```bash
-python3 ./tools/infer_vqa_token_ser_re.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/re_layoutxlm_xfund_zh_v4_udml/best_accuracy/ Global.infer_img=./train_data/XFUND/zh_val/image/ -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
+python3 ./tools/infer_kie_token_ser_re.py \
+  -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \
+  Global.infer_img=./train_data/XFUND/zh_val/image/ \
+  -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o_ser Architecture.Backbone.checkpoints=pretrain_models/ \
+  ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
 ```
 
 预测结果如下所示。
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_re/zh_val_42_re.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_re/zh_val_42_re.jpg" width="800">
 </div>
 
 
 如果希望使用标注或者预先获取的OCR信息进行关键信息抽取，同上，可以指定`Global.infer_mode`为False，指定`Global.infer_img`为标注文件。
 
 ```bash
-python3 ./tools/infer_vqa_token_ser_re.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/re_layoutxlm_xfund_zh_v4_udml/best_accuracy/ Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
+python3 ./tools/infer_kie_token_ser_re.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/re_layoutxlm_xfund_zh_v4_udml/best_accuracy/ Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
 ```
 
 其中`c_ser`表示SER的配置文件，`o_ser` 后面需要加上待修改的SER模型与配置文件，如预训练权重等。
@@ -397,7 +403,7 @@ python3 ./tools/infer_vqa_token_ser_re.py -c configs/kie/vi_layoutxlm/re_vi_layo
 预测结果如下所示。
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_re_with_gt_ocr/zh_val_42_re.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_re_with_gt_ocr/zh_val_42_re.jpg" width="800">
 </div>
 
 可以看出，直接使用标注的OCR结果的RE预测结果要更加准确一些。
@@ -417,8 +423,8 @@ inference 模型（`paddle.jit.save`保存的模型）
 ```bash
 # -c 后面设置训练算法的yml配置文件
 # -o 配置可选参数
-# Global.pretrained_model 参数设置待转换的训练模型地址。
-# Global.save_inference_dir参数设置转换的模型将保存的地址。
+# Architecture.Backbone.checkpoints 参数设置待转换的训练模型地址
+# Global.save_inference_dir 参数设置转换的模型将保存的地址
 
 python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm
 ```
@@ -440,10 +446,10 @@ VI-LayoutXLM模型基于SER任务进行推理，可以执行如下命令：
 
 ```bash
 cd ppstructure
-python3 vqa/predict_vqa_token_ser.py \
-  --vqa_algorithm=LayoutXLM \
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
   --ser_model_dir=../inference/ser_vi_layoutxlm \
-  --image_dir=./docs/vqa/input/zh_val_42.jpg \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
   --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
   --vis_font_path=../doc/fonts/simfang.ttf \
   --ocr_order_method="tb-yx"
@@ -452,7 +458,7 @@ python3 vqa/predict_vqa_token_ser.py \
 可视化SER结果结果默认保存到`./output`文件夹里面。结果示例如下：
 
 <div align="center">
-    <img src="../../ppstructure/docs/vqa/result_ser/zh_val_42_ser.jpg" width="800">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
 </div>
 
 

doc/doc_en/algorithm_kie_layoutxlm_en.md

+# KIE Algorithm - LayoutXLM
+
+
+- [1. Introduction](#1-introduction)
+- [2. Environment](#2-environment)
+- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction)
+- [4. Inference and Deployment](#4-inference-and-deployment)
+  - [4.1 Python Inference](#41-python-inference)
+  - [4.2 C++ Inference](#42-c-inference)
+  - [4.3 Serving](#43-serving)
+  - [4.4 More](#44-more)
+- [5. FAQ](#5-faq)
+- [Citation](#Citation)
+
+
+## 1. Introduction
+
+Paper:
+
+> [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836)
+>
+> Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
+>
+> 2021
+
+On XFUND_zh dataset, the algorithm reproduction Hmean is as follows.
+
+|Model|Backbone|Task |Cnnfig|Hmean|Download link|
+| --- | --- |--|--- | --- | --- |
+|LayoutXLM|LayoutXLM-base|SER |[ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)/[inference model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh_infer.tar)|
+|LayoutXLM|LayoutXLM-base|RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)/[inference model(coming soon)]()|
+
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code.
+
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [KIE tutorial](./kie_en.md)。PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different models.
+
+
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+**Note:** Currently, the RE model inference process is still in the process of adaptation. We take SER model as an example to introduce the KIE process based on LayoutXLM model.
+
+First, we need to export the trained model into inference model. Take LayoutXLM model trained on XFUND_zh as an example ([trained model download link](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)). Use the following command to export.
+
+
+``` bash
+wget https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar
+tar -xf ser_LayoutXLM_xfun_zh.tar
+python3 tools/export_model.py -c configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./ser_LayoutXLM_xfun_zh/best_accuracy Global.save_inference_dir=./inference/ser_layoutxlm
+```
+
+Use the following command to infer using LayoutXLM SER model.
+
+```bash
+cd ppstructure
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
+  --ser_model_dir=../inference/ser_layoutxlm_infer \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
+  --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
+  --vis_font_path=../doc/fonts/simfang.ttf
+```
+
+The SER visualization results are saved in the `./output` directory by default. The results are as follows.
+
+
+<div align="center">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
+</div>
+
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{DBLP:journals/corr/abs-2104-08836,
+  author    = {Yiheng Xu and
+               Tengchao Lv and
+               Lei Cui and
+               Guoxin Wang and
+               Yijuan Lu and
+               Dinei Flor{\^{e}}ncio and
+               Cha Zhang and
+               Furu Wei},
+  title     = {LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich
+               Document Understanding},
+  journal   = {CoRR},
+  volume    = {abs/2104.08836},
+  year      = {2021},
+  url       = {https://arxiv.org/abs/2104.08836},
+  eprinttype = {arXiv},
+  eprint    = {2104.08836},
+  timestamp = {Thu, 14 Oct 2021 09:17:23 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2104-08836.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+
+@article{DBLP:journals/corr/abs-1912-13318,
+  author    = {Yiheng Xu and
+               Minghao Li and
+               Lei Cui and
+               Shaohan Huang and
+               Furu Wei and
+               Ming Zhou},
+  title     = {LayoutLM: Pre-training of Text and Layout for Document Image Understanding},
+  journal   = {CoRR},
+  volume    = {abs/1912.13318},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1912.13318},
+  eprinttype = {arXiv},
+  eprint    = {1912.13318},
+  timestamp = {Mon, 01 Jun 2020 16:20:46 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1912-13318.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+
+@article{DBLP:journals/corr/abs-2012-14740,
+  author    = {Yang Xu and
+               Yiheng Xu and
+               Tengchao Lv and
+               Lei Cui and
+               Furu Wei and
+               Guoxin Wang and
+               Yijuan Lu and
+               Dinei A. F. Flor{\^{e}}ncio and
+               Cha Zhang and
+               Wanxiang Che and
+               Min Zhang and
+               Lidong Zhou},
+  title     = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding},
+  journal   = {CoRR},
+  volume    = {abs/2012.14740},
+  year      = {2020},
+  url       = {https://arxiv.org/abs/2012.14740},
+  eprinttype = {arXiv},
+  eprint    = {2012.14740},
+  timestamp = {Tue, 27 Jul 2021 09:53:52 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2012-14740.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```

doc/doc_en/algorithm_kie_sdmgr_en.md

+
+# KIE Algorithm - SDMGR
+
+- [1. Introduction](#1-introduction)
+- [2. Environment](#2-environment)
+- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction)
+- [4. Inference and Deployment](#4-inference-and-deployment)
+  - [4.1 Python Inference](#41-python-inference)
+  - [4.2 C++ Inference](#42-c-inference)
+  - [4.3 Serving](#43-serving)
+  - [4.4 More](#44-more)
+- [5. FAQ](#5-faq)
+- [Citation](#Citation)
+
+## 1. Introduction
+
+Paper:
+
+> [Spatial Dual-Modality Graph Reasoning for Key Information Extraction](https://arxiv.org/abs/2103.14470)
+>
+> Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang
+>
+> 2021
+
+On wildreceipt dataset, the algorithm reproduction Hmean is as follows.
+
+|Model|Backbone |Cnnfig|Hmean|Download link|
+| --- | --- | --- | --- | --- |
+|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.7%|[trained model]( https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)/[inference model(coming soon)]()|
+
+
+
+## 2. 环境配置
+
+Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code.
+
+
+
+## 3. Model Training / Evaluation / Prediction
+
+SDMGR is a key information extraction algorithm that classifies each detected textline into predefined categories, such as order ID, invoice number, amount, etc.
+
+The training and test data are collected in the wildreceipt dataset, use following command to downloaded the dataset.
+
+
+```bash
+wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar && tar xf wildreceipt.tar
+```
+
+Create dataset soft link to `PaddleOCR/train_data` directory.
+
+```bash
+cd PaddleOCR/ && mkdir train_data && cd train_data
+ln -s ../../wildreceipt ./
+```
+
+
+### 3.1 Model training
+
+The config file is `configs/kie/sdmgr/kie_unet_sdmgr.yml`， the default dataset path is `train_data/wildreceipt`.
+
+Use the following command to train the model.
+
+```bash
+python3 tools/train.py -c configs/kie/sdmgr/kie_unet_sdmgr.yml -o Global.save_model_dir=./output/kie/
+```
+
+### 3.2 Model evaluation
+
+Use the following command to evaluate the model.
+
+```bash
+python3 tools/eval.py -c configs/kie/sdmgr/kie_unet_sdmgr.yml -o Global.checkpoints=./output/kie/best_accuracy
+```
+
+An example of output information is shown below.
+
+```py
+[2022/08/10 05:22:23] ppocr INFO: metric eval ***************
+[2022/08/10 05:22:23] ppocr INFO: hmean:0.8670120239257812
+[2022/08/10 05:22:23] ppocr INFO: fps:10.18816520530961
+```
+
+### 3.3 Model prediction
+
+Use the following command to load the model and predict. During the prediction, the text file storing the image path and OCR information needs to be loaded in advance. Use `Global.infer_img` to assign.
+
+```bash
+python3 tools/infer_kie.py -c configs/kie/kie_unet_sdmgr.yml -o Global.checkpoints=kie_vgg16/best_accuracy  Global.infer_img=./train_data/wildreceipt/1.txt
+```
+
+The visualization results and texts are saved in the `./output/sdmgr_kie/` directory by default. The results are as follows.
+
+
+<div align="center">
+    <img src="../../ppstructure/docs/imgs/sdmgr_result.png" width="800">
+</div>
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+Not supported
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@misc{sun2021spatial,
+      title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction},
+      author={Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang},
+      year={2021},
+      eprint={2103.14470},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```

doc/doc_en/algorithm_kie_vi_layoutxlm_en.md

+# KIE Algorithm - VI-LayoutXLM
+
+
+- [1. Introduction](#1-introduction)
+- [2. Environment](#2-environment)
+- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction)
+- [4. Inference and Deployment](#4-inference-and-deployment)
+  - [4.1 Python Inference](#41-python-inference)
+  - [4.2 C++ Inference](#42-c-inference)
+  - [4.3 Serving](#43-serving)
+  - [4.4 More](#44-more)
+- [5. FAQ](#5-faq)
+- [Citation](#Citation)
+
+
+## 1. Introduction
+
+VI-LayoutXLM is improved based on LayoutXLM. In the process of downstream finetuning, the visual backbone network module is removed, and the model infernce speed is further improved on the basis of almost lossless accuracy.
+
+On XFUND_zh dataset, the algorithm reproduction Hmean is as follows.
+
+|Model|Backbone|Task |Cnnfig|Hmean|Download link|
+| --- | --- |---| --- | --- | --- |
+|VI-LayoutXLM |VI-LayoutXLM-base | SER |[ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|93.19%|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)/[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar)|
+|VI-LayoutXLM |VI-LayoutXLM-base |RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|83.92%|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)/[inference model(coming soon)]()|
+
+
+Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code.
+
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [KIE tutorial](./kie_en.md)。PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different models.
+
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+**Note:** Currently, the RE model inference process is still in the process of adaptation. We take SER model as an example to introduce the KIE process based on VI-LayoutXLM model.
+
+First, we need to export the trained model into inference model. Take VI-LayoutXLM model trained on XFUND_zh as an example ([trained model download link](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)). Use the following command to export.
+
+
+``` bash
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar
+tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
+python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./ser_vi_layoutxlm_xfund_pretrained/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm_infer
+```
+
+Use the following command to infer using VI-LayoutXLM SER model.
+
+
+```bash
+cd ppstructure
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
+  --ser_model_dir=../inference/ser_vi_layoutxlm_infer \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
+  --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
+  --vis_font_path=../doc/fonts/simfang.ttf \
+  --ocr_order_method="tb-yx"
+```
+
+The SER visualization results are saved in the `./output` folder by default. The results are as follows.
+
+
+<div align="center">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
+</div>
+
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+
+```bibtex
+@article{DBLP:journals/corr/abs-2104-08836,
+  author    = {Yiheng Xu and
+               Tengchao Lv and
+               Lei Cui and
+               Guoxin Wang and
+               Yijuan Lu and
+               Dinei Flor{\^{e}}ncio and
+               Cha Zhang and
+               Furu Wei},
+  title     = {LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich
+               Document Understanding},
+  journal   = {CoRR},
+  volume    = {abs/2104.08836},
+  year      = {2021},
+  url       = {https://arxiv.org/abs/2104.08836},
+  eprinttype = {arXiv},
+  eprint    = {2104.08836},
+  timestamp = {Thu, 14 Oct 2021 09:17:23 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2104-08836.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+
+@article{DBLP:journals/corr/abs-1912-13318,
+  author    = {Yiheng Xu and
+               Minghao Li and
+               Lei Cui and
+               Shaohan Huang and
+               Furu Wei and
+               Ming Zhou},
+  title     = {LayoutLM: Pre-training of Text and Layout for Document Image Understanding},
+  journal   = {CoRR},
+  volume    = {abs/1912.13318},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1912.13318},
+  eprinttype = {arXiv},
+  eprint    = {1912.13318},
+  timestamp = {Mon, 01 Jun 2020 16:20:46 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1912-13318.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+
+@article{DBLP:journals/corr/abs-2012-14740,
+  author    = {Yang Xu and
+               Yiheng Xu and
+               Tengchao Lv and
+               Lei Cui and
+               Furu Wei and
+               Guoxin Wang and
+               Yijuan Lu and
+               Dinei A. F. Flor{\^{e}}ncio and
+               Cha Zhang and
+               Wanxiang Che and
+               Min Zhang and
+               Lidong Zhou},
+  title     = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding},
+  journal   = {CoRR},
+  volume    = {abs/2012.14740},
+  year      = {2020},
+  url       = {https://arxiv.org/abs/2012.14740},
+  eprinttype = {arXiv},
+  eprint    = {2012.14740},
+  timestamp = {Tue, 27 Jul 2021 09:53:52 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2012-14740.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```

doc/doc_en/algorithm_overview_en.md

-# OCR Algorithms
+# Algorithms
 
-- [1. Two-stage Algorithms](#1)
+- [1. Two-stage OCR Algorithms](#1)
   - [1.1 Text Detection Algorithms](#11)
   - [1.2 Text Recognition Algorithms](#12)
-- [2. End-to-end Algorithms](#2)
+- [2. End-to-end OCR Algorithms](#2)
 - [3. Table Recognition Algorithms](#3)
-
+- [4. Key Information Extraction Algorithms](#4)
 
 This tutorial lists the OCR algorithms supported by PaddleOCR, as well as the models and metrics of each algorithm on **English public datasets**. It is mainly used for algorithm introduction and algorithm performance comparison. For more models on other datasets including Chinese, please refer to [PP-OCR v2.0 models list](./models_list_en.md).
 
 <a name="1"></a>
 
-## 1. Two-stage Algorithms
+## 1. Two-stage OCR Algorithms
 
 <a name="11"></a>
 
@@ -98,11 +98,12 @@ Refer to [DTRB](https://arxiv.org/abs/1904.01906), the training and evaluation r
 
 <a name="2"></a>
 
-## 2. End-to-end Algorithms
+## 2. End-to-end OCR Algorithms
 
 Supported end-to-end algorithms (Click the link to get the tutorial):
 - [x]  [PGNet](./algorithm_e2e_pgnet_en.md)
 
+
 <a name="3"></a>
 ## 3. Table Recognition Algorithms
 
@@ -114,3 +115,34 @@ On the PubTabNet dataset, the algorithm result is as follows:
 |Model|Backbone|Config|Acc|Download link|
 |---|---|---|---|---|
 |TableMaster|TableResNetExtra|[configs/table/table_master.yml](../../configs/table/table_master.yml)|77.47%|[trained](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_train.tar) / [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_infer.tar)|
+
+
+<a name="4"></a>
+
+## 4. Key Information Extraction Algorithms
+
+Supported KIE algorithms (Click the link to get the tutorial):
+
+- [x]  [VI-LayoutXLM](./algorithm_kie_vi_laoutxlm_en.md)
+- [x]  [LayoutLM](./algorithm_kie_laoutxlm_en.md)
+- [x]  [LayoutLMv2](./algorithm_kie_laoutxlm_en.md)
+- [x]  [LayoutXLM](./algorithm_kie_laoutxlm_en.md)
+- [x]  [SDMGR](./algorithm_kie_sdmgr_en.md)
+
+On wildreceipt dataset, the algorithm result is as follows:
+
+|Model|Backbone|Config|Hmean|Download link|
+| --- | --- | --- | --- | --- |
+|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.7%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)|
+
+On XFUND_zh dataset, the algorithm result is as follows:
+
+|Model|Backbone|Task|Config|Hmean|Download link|
+| --- | --- |  --- | --- | --- | --- |
+|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)|
+|LayoutLM| LayoutLM-base | SER | [ser_layoutlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml)|77.31%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar)|
+|LayoutLMv2| LayoutLMv2-base | SER | [ser_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlmv2_xfund_zh.yml)|85.44%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar)|
+|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)|
+|LayoutLMv2| LayoutLMv2-base | RE | [re_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutlmv2_xfund_zh.yml)|67.77%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar)|

doc/doc_en/dataset/docvqa_datasets_en.md → doc/doc_en/dataset/kie_datasets_en.md

-## DocVQA dataset
-Here are the common DocVQA datasets, which are being updated continuously. Welcome to contribute datasets~
+## Key Imnformation Extraction dataset
+
+Here are the common DocVQA datasets, which are being updated continuously. Welcome to contribute datasets.
 - [FUNSD dataset](#funsd)
 - [XFUND dataset](#xfund)
+- [wildreceipt dataset](#wildreceipt数据集)
 
 <a name="funsd"></a>
 #### 1. FUNSD dataset
@@ -25,3 +27,21 @@ Here are the common DocVQA datasets, which are being updated continuously. Welco
 </div>
 
 - **Download address**: https://github.com/doc-analysis/XFUND/releases/tag/v1.0
+
+<a name="wildreceipt"></a>
+
+## 3. wildreceipt dataset
+
+- **Data source**: https://arxiv.org/abs/2103.14470
+- **Data introduction**: XFUND is an English receipt dataset, which contains 26 different categories. There are 1267 training images and 472 evaluation images, in which 50,000 textlines and boxes are annotated. Part of the image and the annotation box visualization are shown below.
+
+<div align="center">
+    <img src="../../datasets/wildreceipt_demo/2769.jpeg" width="500">
+    <img src="../../datasets/wildreceipt_demo/1bbe854b8817dedb8585e0732089fd1f752d2cec.jpeg" width="500">
+</div>
+
+**Note：** Boxes with category `Ignore` or `Others` are not visualized here.
+
+- **Download address**：
+    - Offical dataset: [link](https://download.openmmlab.com/mmocr/data/wildreceipt.tar)
+    - Dataset converted for PaddleOCR training process: [link](https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar)

paddleocr.py

@@ -35,7 +35,7 @@ from tools.infer import predict_system
 from ppocr.utils.logging import get_logger
 
 logger = get_logger()
-from ppocr.utils.utility import check_and_read_gif, get_image_file_list
+from ppocr.utils.utility import check_and_read, get_image_file_list
 from ppocr.utils.network import maybe_download, download_with_progressbar, is_link, confirm_model_dir_url
 from tools.infer.utility import draw_ocr, str2bool, check_gpu
 from ppstructure.utility import init_args, draw_structure_result
@@ -289,7 +289,8 @@ MODEL_URLS = {
                 'ch': {
                     'url':
                     'https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout_infer.tar',
-                    'dict_path': 'ppocr/utils/dict/layout_publaynet_dict.txt'
+                    'dict_path':
+                    'ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt'
                 }
             }
         }
@@ -490,7 +491,7 @@ class PaddleOCR(predict_system.TextSystem):
                 download_with_progressbar(img, 'tmp.jpg')
                 img = 'tmp.jpg'
             image_file = img
-            img, flag = check_and_read_gif(image_file)
+            img, flag, _ = check_and_read(image_file)
             if not flag:
                 with open(image_file, 'rb') as f:
                     np_arr = np.frombuffer(f.read(), dtype=np.uint8)
@@ -584,7 +585,7 @@ class PPStructure(StructureSystem):
                 download_with_progressbar(img, 'tmp.jpg')
                 img = 'tmp.jpg'
             image_file = img
-            img, flag = check_and_read_gif(image_file)
+            img, flag, _ = check_and_read(image_file)
             if not flag:
                 with open(image_file, 'rb') as f:
                     np_arr = np.frombuffer(f.read(), dtype=np.uint8)
@@ -635,4 +636,6 @@ def main():
 
             for item in result:
                 item.pop('img')
+                item.pop('res')
                 logger.info(item)
+            logger.info('result save to {}'.format(args.output))

ppocr/data/imaug/copy_paste.py

@@ -35,10 +35,12 @@ class CopyPaste(object):
         point_num = data['polys'].shape[1]
         src_img = data['image']
         src_polys = data['polys'].tolist()
+        src_texts = data['texts']
         src_ignores = data['ignore_tags'].tolist()
         ext_data = data['ext_data'][0]
         ext_image = ext_data['image']
         ext_polys = ext_data['polys']
+        ext_texts = ext_data['texts']
         ext_ignores = ext_data['ignore_tags']
 
         indexs = [i for i in range(len(ext_ignores)) if not ext_ignores[i]]
@@ -53,7 +55,7 @@ class CopyPaste(object):
         src_img = cv2.cvtColor(src_img, cv2.COLOR_BGR2RGB)
         ext_image = cv2.cvtColor(ext_image, cv2.COLOR_BGR2RGB)
         src_img = Image.fromarray(src_img).convert('RGBA')
-        for poly, tag in zip(select_polys, select_ignores):
+        for idx, poly, tag in zip(select_idxs, select_polys, select_ignores):
             box_img = get_rotate_crop_image(ext_image, poly)
 
             src_img, box = self.paste_img(src_img, box_img, src_polys)
@@ -62,6 +64,7 @@ class CopyPaste(object):
                 for _ in range(len(box), point_num):
                     box.append(box[-1])
                 src_polys.append(box)
+                src_texts.append(ext_texts[idx])
                 src_ignores.append(tag)
         src_img = cv2.cvtColor(np.array(src_img), cv2.COLOR_RGB2BGR)
         h, w = src_img.shape[:2]
@@ -70,6 +73,7 @@ class CopyPaste(object):
         src_polys[:, :, 1] = np.clip(src_polys[:, :, 1], 0, h)
         data['image'] = src_img
         data['polys'] = src_polys
+        data['texts'] = src_texts
         data['ignore_tags'] = np.array(src_ignores)
         return data
 

ppocr/metrics/rec_metric.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import Levenshtein
+from rapidfuzz.distance import Levenshtein
 import string
 
 
@@ -46,8 +46,7 @@ class RecMetric(object):
             if self.is_filter:
                 pred = self._normalize_text(pred)
                 target = self._normalize_text(target)
-            norm_edit_dis += Levenshtein.distance(pred, target) / max(
-                len(pred), len(target), 1)
+            norm_edit_dis += Levenshtein.normalized_distance(pred, target)
             if pred == target:
                 correct_num += 1
             all_num += 1

ppocr/modeling/backbones/__init__.py

@@ -52,17 +52,15 @@ def build_backbone(config, model_type):
         support_dict = ['ResNet']
     elif model_type == 'kie':
         from .kie_unet_sdmgr import Kie_backbone
-        support_dict = ['Kie_backbone']
+        from .vqa_layoutlm import LayoutLMForSer, LayoutLMv2ForSer, LayoutLMv2ForRe, LayoutXLMForSer, LayoutXLMForRe
+        support_dict = [
+            'Kie_backbone', 'LayoutLMForSer', 'LayoutLMv2ForSer',
+            'LayoutLMv2ForRe', 'LayoutXLMForSer', 'LayoutXLMForRe'
+        ]
     elif model_type == 'table':
         from .table_resnet_vd import ResNet
         from .table_mobilenet_v3 import MobileNetV3
         support_dict = ['ResNet', 'MobileNetV3']
-    elif model_type == 'vqa':
-        from .vqa_layoutlm import LayoutLMForSer, LayoutLMv2ForSer, LayoutLMv2ForRe, LayoutXLMForSer, LayoutXLMForRe
-        support_dict = [
-            'LayoutLMForSer', 'LayoutLMv2ForSer', 'LayoutLMv2ForRe',
-            'LayoutXLMForSer', 'LayoutXLMForRe'
-        ]
     else:
         raise NotImplementedError
 

ppocr/utils/dict/kie_dict/xfund_class_list.txt

+OTHER
+QUESTION
+ANSWER
+HEADER

ppocr/utils/save_load.py

@@ -54,13 +54,15 @@ def load_model(config, model, optimizer=None, model_type='det'):
     pretrained_model = global_config.get('pretrained_model')
     best_model_dict = {}
     is_float16 = False
+    is_nlp_model = model_type == 'kie' and config["Architecture"][
+        "algorithm"] not in ["SDMGR"]
 
-    if model_type == 'vqa':
-        # NOTE: for vqa model dsitillation, resume training is not supported now
+    if is_nlp_model is True:
+        # NOTE: for kie model dsitillation, resume training is not supported now
         if config["Architecture"]["algorithm"] in ["Distillation"]:
             return best_model_dict
         checkpoints = config['Architecture']['Backbone']['checkpoints']
-        # load vqa method metric
+        # load kie method metric
         if checkpoints:
             if os.path.exists(os.path.join(checkpoints, 'metric.states')):
                 with open(os.path.join(checkpoints, 'metric.states'),
@@ -154,7 +156,7 @@ def load_pretrained_params(model, path):
 
     new_state_dict = {}
     is_float16 = False
-    
+
     for k1 in params.keys():
 
         if k1 not in state_dict.keys():
@@ -194,10 +196,13 @@ def save_model(model,
     _mkdir_if_not_exist(model_path, logger)
     model_prefix = os.path.join(model_path, prefix)
     paddle.save(optimizer.state_dict(), model_prefix + '.pdopt')
-    if config['Architecture']["model_type"] != 'vqa':
+
+    is_nlp_model = config['Architecture']["model_type"] == 'kie' and config[
+        "Architecture"]["algorithm"] not in ["SDMGR"]
+    if is_nlp_model is not True:
         paddle.save(model.state_dict(), model_prefix + '.pdparams')
         metric_prefix = model_prefix
-    else:  # for vqa system, we follow the save/load rules in NLP
+    else:  # for kie system, we follow the save/load rules in NLP
         if config['Global']['distributed']:
             arch = model._layers
         else:

ppocr/utils/utility.py

@@ -50,7 +50,7 @@ def get_check_global_params(mode):
 
 
 def _check_image_file(path):
-    img_end = {'jpg', 'bmp', 'png', 'jpeg', 'rgb', 'tif', 'tiff', 'gif'}
+    img_end = {'jpg', 'bmp', 'png', 'jpeg', 'rgb', 'tif', 'tiff', 'gif', 'pdf'}
     return any([path.lower().endswith(e) for e in img_end])
 
 
@@ -59,7 +59,7 @@ def get_image_file_list(img_file):
     if img_file is None or not os.path.exists(img_file):
         raise Exception("not found any img file in {}".format(img_file))
 
-    img_end = {'jpg', 'bmp', 'png', 'jpeg', 'rgb', 'tif', 'tiff', 'gif'}
+    img_end = {'jpg', 'bmp', 'png', 'jpeg', 'rgb', 'tif', 'tiff', 'gif', 'pdf'}
     if os.path.isfile(img_file) and _check_image_file(img_file):
         imgs_lists.append(img_file)
     elif os.path.isdir(img_file):
@@ -73,7 +73,7 @@ def get_image_file_list(img_file):
     return imgs_lists
 
 
-def check_and_read_gif(img_path):
+def check_and_read(img_path):
     if os.path.basename(img_path)[-3:] in ['gif', 'GIF']:
         gif = cv2.VideoCapture(img_path)
         ret, frame = gif.read()
@@ -84,8 +84,26 @@ def check_and_read_gif(img_path):
         if len(frame.shape) == 2 or frame.shape[-1] == 1:
             frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2RGB)
         imgvalue = frame[:, :, ::-1]
-        return imgvalue, True
-    return None, False
+        return imgvalue, True, False
+    elif os.path.basename(img_path)[-3:] in ['pdf']:
+        import fitz
+        from PIL import Image
+        imgs = []
+        with fitz.open(img_path) as pdf:
+            for pg in range(0, pdf.pageCount):
+                page = pdf[pg]
+                mat = fitz.Matrix(2, 2)
+                pm = page.getPixmap(matrix=mat, alpha=False)
+
+                # if width or height > 2000 pixels, don't enlarge the image
+                if pm.width > 2000 or pm.height > 2000:
+                    pm = page.getPixmap(matrix=fitz.Matrix(1, 1), alpha=False)
+
+                img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples)
+                img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
+                imgs.append(img)
+            return imgs, False, True
+    return None, False, False
 
 
 def load_vqa_bio_label_maps(label_map_path):

ppstructure/README.md

@@ -5,25 +5,25 @@ English | [简体中文](README_ch.md)
 - [3. Features](#3-features)
 - [4. Results](#4-results)
   - [4.1 Layout analysis and table recognition](#41-layout-analysis-and-table-recognition)
-  - [4.2 DOC-VQA](#42-doc-vqa)
+  - [4.2 KIE](#42-kie)
 - [5. Quick start](#5-quick-start)
 - [6. PP-Structure System](#6-pp-structure-system)
   - [6.1 Layout analysis and table recognition](#61-layout-analysis-and-table-recognition)
     - [6.1.1 Layout analysis](#611-layout-analysis)
     - [6.1.2 Table recognition](#612-table-recognition)
-  - [6.2 DOC-VQA](#62-doc-vqa)
+  - [6.2 KIE](#62-kie)
 - [7. Model List](#7-model-list)
   - [7.1 Layout analysis model](#71-layout-analysis-model)
   - [7.2 OCR and table recognition model](#72-ocr-and-table-recognition-model)
-  - [7.3 DOC-VQA model](#73-doc-vqa-model)
+  - [7.3 KIE model](#73-kie-model)
 
 ## 1. Introduction
 
 PP-Structure is an OCR toolkit that can be used for document analysis and processing with complex structures, designed to help developers better complete document understanding tasks
 
 ## 2. Update log
-* 2022.02.12 DOC-VQA add LayoutLMv2 model。
-* 2021.12.07 add [DOC-VQA SER and RE tasks](vqa/README.md)。
+* 2022.02.12 KIE add LayoutLMv2 model。
+* 2021.12.07 add [KIE SER and RE tasks](kie/README.md)。
 
 ## 3. Features
 
@@ -34,7 +34,7 @@ The main features of PP-Structure are as follows:
 - Support to extract excel files from the table areas
 - Support python whl package and command line usage, easy to use
 - Support custom training for layout analysis and table structure tasks
-- Support Document Visual Question Answering (DOC-VQA) tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE)
+- Support Document Key Information Extraction (KIE) tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE)
 
 ## 4. Results
 
@@ -44,11 +44,11 @@ The main features of PP-Structure are as follows:
 
 The figure shows the pipeline of layout analysis + table recognition. The image is first divided into four areas of image, text, title and table by layout analysis, and then OCR detection and recognition is performed on the three areas of image, text and title, and the table is performed table recognition, where the image will also be stored for use.
 
-### 4.2 DOC-VQA
+### 4.2 KIE
 
 * SER
 *
-![](docs/vqa/result_ser/zh_val_0_ser.jpg) | ![](docs/vqa/result_ser/zh_val_42_ser.jpg)
+![](docs/kie/result_ser/zh_val_0_ser.jpg) | ![](docs/kie/result_ser/zh_val_42_ser.jpg)
 ---|---
 
 Different colored boxes in the figure represent different categories. For xfun dataset, there are three categories: query, answer and header:
@@ -62,7 +62,7 @@ The corresponding category and OCR recognition results are also marked at the to
 
 * RE
 
-![](docs/vqa/result_re/zh_val_21_re.jpg) | ![](docs/vqa/result_re/zh_val_40_re.jpg)
+![](docs/kie/result_re/zh_val_21_re.jpg) | ![](docs/kie/result_re/zh_val_40_re.jpg)
 ---|---
 
 
@@ -88,9 +88,9 @@ Layout analysis classifies image by region, including the use of Python scripts
 
 Table recognition converts table images into excel documents, which include the detection and recognition of table text and the prediction of table structure and cell coordinates. For detailed instructions, please refer to [document](table/README.md)
 
-### 6.2 DOC-VQA
+### 6.2 KIE
 
-Document Visual Question Answering (DOC-VQA) if a type of Visual Question Answering (VQA), which includes Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks. Based on SER task, text recognition and classification in images can be completed. Based on THE RE task, we can extract the relation of the text content in the image, such as judge the problem pair. For details, please refer to [document](vqa/README.md)
+Multi-modal based Key Information Extraction (KIE) methods include Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks. Based on SER task, text recognition and classification in images can be completed. Based on THE RE task, we can extract the relation of the text content in the image, such as judge the problem pair. For details, please refer to [document](kie/README.md)
 
 ## 7. Model List
 
@@ -106,11 +106,11 @@ PP-Structure Series Model List (Updating)
 
 |model name|description|model size|download|
 | --- | --- | --- | --- |
-|ch_PP-OCRv3_det_slim|[New] slim quantization with distillation lightweight model, supporting Chinese, English, multilingual text detection| 1.1M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_distill_train.tar)|
-|ch_PP-OCRv3_rec_slim |[New] Slim qunatization with distillation lightweight model, supporting Chinese, English text recognition| 4.9M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_train.tar) |
-|ch_ppstructure_mobile_v2.0_SLANet|Chinese table recognition model trained on PubTabNet dataset based on SLANet|9.3M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
+|ch_PP-OCRv3_det| [New] Lightweight model, supporting Chinese, English, multilingual text detection | 3.8M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar)|
+|ch_PP-OCRv3_rec| [New] Lightweight model, supporting Chinese, English, multilingual text recognition | 12.4M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) |
+|ch_ppstructure_mobile_v2.0_SLANet|Chinese table recognition model based on SLANet|9.3M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
 
-### 7.3 DOC-VQA model
+### 7.3 KIE model
 
 |model name|description|model size|download|
 | --- | --- | --- | --- |

ppstructure/README_ch.md

@@ -7,17 +7,17 @@
 - [3. 特性](#3)
 - [4. 效果展示](#4)
   - [4.1 版面分析和表格识别](#41)
-  - [4.2 DocVQA](#42)
+  - [4.2 关键信息抽取](#42)
 - [5. 快速体验](#5)
 - [6. PP-Structure 介绍](#6)
   - [6.1 版面分析+表格识别](#61)
     - [6.1.1 版面分析](#611)
     - [6.1.2 表格识别](#612)
-  - [6.2 DocVQA](#62)
+  - [6.2 关键信息抽取](#62)
 - [7. 模型库](#7)
   - [7.1 版面分析模型](#71)
   - [7.2 OCR和表格识别模型](#72)
-  - [7.3 DocVQA 模型](#73)
+  - [7.3 关键信息抽取模型](#73)
 
 <a name="1"></a>
 ## 1. 简介
@@ -25,8 +25,8 @@ PP-Structure是一个可用于复杂文档结构分析和处理的OCR工具包
 
 <a name="2"></a>
 ## 2. 近期更新
-* 2022.02.12 DocVQA增加LayoutLMv2模型。
-* 2021.12.07 新增[DOC-VQA任务SER和RE](vqa/README.md)。
+* 2022.02.12 KIE增加LayoutLMv2模型。
+* 2021.12.07 新增[关键信息抽取任务SER和RE](kie/README.md)。
 
 <a name="3"></a>
 ## 3. 特性
@@ -37,7 +37,7 @@ PP-Structure的主要特性如下：
 - 支持表格区域进行结构化分析，最终结果输出Excel文件
 - 支持python whl包和命令行两种方式，简单易用
 - 支持版面分析和表格结构化两类任务自定义训练
-- 支持文档视觉问答(Document Visual Question Answering，DocVQA)任务-语义实体识别(Semantic Entity Recognition，SER)和关系抽取(Relation Extraction，RE)
+- 支持基于多模态的关键信息抽取(Key Information Extraction，KIE)任务-语义实体识别(Semantic Entity Recognition，SER)和关系抽取(Relation Extraction，RE)
 
 <a name="4"></a>
 ## 4. 效果展示
@@ -50,11 +50,11 @@ PP-Structure的主要特性如下：
 图中展示了版面分析+表格识别的整体流程，图片先有版面分析划分为图像、文本、标题和表格四种区域，然后对图像、文本和标题三种区域进行OCR的检测识别，对表格进行表格识别，其中图像还会被存储下来以便使用。
 
 <a name="42"></a>
-### 4.2 DOC-VQA
+### 4.2 关键信息抽取
 
 * SER
 
-![](./docs/vqa/result_ser/zh_val_0_ser.jpg) | ![](./docs/vqa/result_ser/zh_val_42_ser.jpg)
+![](./docs/kie/result_ser/zh_val_0_ser.jpg) | ![](./docs/kie/result_ser/zh_val_42_ser.jpg)
 ---|---
 
 图中不同颜色的框表示不同的类别，对于XFUN数据集，有`QUESTION`, `ANSWER`, `HEADER` 3种类别
@@ -67,7 +67,7 @@ PP-Structure的主要特性如下：
 
 * RE
 
-![](./docs/vqa/result_re/zh_val_21_re.jpg) | ![](./docs/vqa/result_re/zh_val_40_re.jpg)
+![](./docs/kie/result_re/zh_val_21_re.jpg) | ![](./docs/kie/result_re/zh_val_40_re.jpg)
 ---|---
 
 
@@ -99,9 +99,9 @@ PP-Structure的主要特性如下：
 表格识别将表格图片转换为excel文档，其中包含对于表格文本的检测和识别以及对于表格结构和单元格坐标的预测，详细说明参考[文档](table/README_ch.md)。
 
 <a name="62"></a>
-### 6.2 DocVQA
+### 6.2 关键信息抽取
 
-DocVQA指文档视觉问答，其中包括语义实体识别 (Semantic Entity Recognition, SER) 和关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务，可以完成对图像中的文本识别与分类；基于 RE 任务，可以完成对图象中的文本内容的关系提取，如判断问题对(pair)，详细说明参考[文档](vqa/README.md)。
+关键信息抽取包括语义实体识别 (Semantic Entity Recognition, SER) 和关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务，可以完成对图像中的文本识别与分类；基于 RE 任务，可以完成对图象中的文本内容的关系提取，如判断问题对(pair)，详细说明参考[文档](kie/README.md)。
 
 <a name="7"></a>
 ## 7. 模型库
@@ -120,13 +120,13 @@ PP-Structure系列模型列表（更新中）
 
 |模型名称|模型简介|模型大小|下载地址|
 | --- | --- | --- | --- |
-|ch_PP-OCRv3_det_slim|【最新】slim量化+蒸馏版超轻量模型，支持中英文、多语种文本检测| 1.1M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_distill_train.tar)|
-|ch_PP-OCRv3_rec_slim |【最新】slim量化版超轻量模型，支持中英文、数字识别| 4.9M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_train.tar) |
-|ch_ppstructure_mobile_v2.0_SLANet|基于SLANet在PubTabNet数据集上训练的中文表格识别模型|9.3M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
+|ch_PP-OCRv3_det| 【最新】超轻量模型，支持中英文、多语种文本检测 | 3.8M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar)|
+|ch_PP-OCRv3_rec|【最新】超轻量模型，支持中英文、数字识别|12.4M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) |
+|ch_ppstructure_mobile_v2.0_SLANet|基于SLANet的中文表格识别模型|9.3M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
 
 
 <a name="73"></a>
-### 7.3 DocVQA 模型
+### 7.3 KIE 模型
 
 |模型名称|模型简介|模型大小|下载地址|
 | --- | --- | --- | --- |

ppstructure/docs/inference.md

@@ -4,7 +4,7 @@
   - [1.1 版面分析+表格识别](#1.1)
   - [1.2 版面分析](#1.2)
   - [1.3 表格识别](#1.3)
-- [2. DocVQA](#2)
+- [2. 关键信息抽取](#2)
 
 <a name="1"></a>
 ## 1. Structure
@@ -16,23 +16,26 @@ cd ppstructure
 下载模型
 ```bash
 mkdir inference && cd inference
-# 下载PP-OCRv2文本检测模型并解压
-wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_slim_quant_infer.tar && tar xf ch_PP-OCRv2_det_slim_quant_infer.tar
-# 下载PP-OCRv2文本识别模型并解压
-wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_quant_infer.tar && tar xf ch_PP-OCRv2_rec_slim_quant_infer.tar
-# 下载超轻量级英文表格预测模型并解压
-wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
+# 下载PP-Structurev2版面分析模型并解压
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout_infer.tar && tar xf picodet_lcnet_x1_0_layout_infer.tar
+# 下载PP-OCRv3文本检测模型并解压
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar && tar xf ch_PP-OCRv3_det_infer.tar
+# 下载PP-OCRv3文本识别模型并解压
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar && tar xf ch_PP-OCRv3_rec_infer.tar
+# 下载PP-Structurev2表格识别模型并解压
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
 cd ..
 ```
 <a name="1.1"></a>
 ### 1.1 版面分析+表格识别
 ```bash
-python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_infer \
-                          --rec_model_dir=inference/ch_PP-OCRv2_rec_slim_quant_infer \
-                          --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer \
+python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv3_det_infer \
+                          --rec_model_dir=inference/ch_PP-OCRv3_rec_infer \
+                          --table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
+                          --layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
                           --image_dir=./docs/table/1.png \
                           --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
-                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
                           --output=../output \
                           --vis_font_path=../doc/fonts/simfang.ttf
 ```
@@ -41,19 +44,23 @@ python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_i
 <a name="1.2"></a>
 ### 1.2 版面分析
 ```bash
-python3 predict_system.py --image_dir=./docs/table/1.png --table=false --ocr=false --output=../output/
+python3 predict_system.py --layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
+                          --image_dir=./docs/table/1.png \
+                          --output=../output \
+                          --table=false \
+                          --ocr=false
 ```
 运行完成后，每张图片会在`output`字段指定的目录下的`structure`目录下有一个同名目录，图片区域会被裁剪之后保存下来，图片名为表格在图片里的坐标。版面分析结果会存储在`res.txt`文件中。
 
 <a name="1.3"></a>
 ### 1.3 表格识别
 ```bash
-python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_infer \
-                          --rec_model_dir=inference/ch_PP-OCRv2_rec_slim_quant_infer \
-                          --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer \
+python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv3_det_infer \
+                          --rec_model_dir=inference/ch_PP-OCRv3_rec_infer \
+                          --table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
                           --image_dir=./docs/table/table.jpg \
                           --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
-                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
                           --output=../output \
                           --vis_font_path=../doc/fonts/simfang.ttf \
                           --layout=false
@@ -61,20 +68,22 @@ python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_i
 运行完成后，每张图片会在`output`字段指定的目录下的`structure`目录下有一个同名目录，表格会存储为一个excel，excel文件名为`[0,0,img_h,img_w]`。
 
 <a name="2"></a>
-## 2. DocVQA
+## 2. 关键信息抽取
 
 ```bash
 cd ppstructure
 
-# 下载模型
 mkdir inference && cd inference
-# 下载SER xfun 模型并解压
-wget https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar && tar xf PP-Layout_v1.0_ser_pretrained.tar
+# 下载SER XFUND 模型并解压
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
 cd ..
-
-python3 predict_system.py --model_name_or_path=vqa/PP-Layout_v1.0_ser_pretrained/ \
-                          --mode=vqa \
-                          --image_dir=vqa/images/input/zh_val_0.jpg  \
-                          --vis_font_path=../doc/fonts/simfang.ttf
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
+  --ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
+  --ser_dict_path=../ppocr/utils/dict/kie_dict/xfund_class_list.txt \
+  --vis_font_path=../doc/fonts/simfang.ttf \
+  --ocr_order_method="tb-yx"
 ```
-运行完成后，每张图片会在`output`字段指定的目录下的`vqa`目录下存放可视化之后的图片，图片名和输入图片名一致。
+
+运行完成后，每张图片会在`output`字段指定的目录下的`kie`目录下存放可视化之后的图片，图片名和输入图片名一致。

ppstructure/docs/inference_en.md

@@ -4,7 +4,7 @@
   - [1.1 layout analysis + table recognition](#1.1)
   - [1.2 layout analysis](#1.2)
   - [1.3 table recognition](#1.3)
-- [2. DocVQA](#2)
+- [2. KIE](#2)
 
 <a name="1"></a>
 ## 1. Structure
@@ -18,23 +18,26 @@ download model
 
 ```bash
 mkdir inference && cd inference
-# Download the PP-OCRv2 text detection model and unzip it
-wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_slim_quant_infer.tar && tar xf ch_PP-OCRv2_det_slim_quant_infer.tar
-# Download the PP-OCRv2 text recognition model and unzip it
-wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_quant_infer.tar && tar xf ch_PP-OCRv2_rec_slim_quant_infer.tar
-# Download the ultra-lightweight English table structure model and unzip it
-wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
+# Download the PP-Structurev2 layout analysis model and unzip it
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout_infer.tar && tar xf picodet_lcnet_x1_0_layout_infer.tar
+# Download the PP-OCRv3 text detection model and unzip it
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar && tar xf ch_PP-OCRv3_det_infer.tar
+# Download the PP-OCRv3 text recognition model and unzip it
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar && tar xf ch_PP-OCRv3_rec_infer.tar
+# Download the PP-Structurev2 form recognition model and unzip it
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
 cd ..
 ```
 <a name="1.1"></a>
 ### 1.1 layout analysis + table recognition
 ```bash
-python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_infer \
-                          --rec_model_dir=inference/ch_PP-OCRv2_rec_slim_quant_infer \
-                          --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer \
+python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv3_det_infer \
+                          --rec_model_dir=inference/ch_PP-OCRv3_rec_infer \
+                          --table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
+                          --layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
                           --image_dir=./docs/table/1.png \
                           --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
-                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
                           --output=../output \
                           --vis_font_path=../doc/fonts/simfang.ttf
 ```
@@ -43,19 +46,23 @@ After the operation is completed, each image will have a directory with the same
 <a name="1.2"></a>
 ### 1.2 layout analysis
 ```bash
-python3 predict_system.py --image_dir=./docs/table/1.png --table=false --ocr=false --output=../output/
+python3 predict_system.py --layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
+                          --image_dir=./docs/table/1.png \
+                          --output=../output \
+                          --table=false \
+                          --ocr=false
 ```
 After the operation is completed, each image will have a directory with the same name in the `structure` directory under the directory specified by the `output` field. Each picture in image will be cropped and saved. The filename of picture area is their coordinates in the image. Layout analysis results will be stored in the `res.txt` file
 
 <a name="1.3"></a>
 ### 1.3 table recognition
 ```bash
-python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_infer \
-                          --rec_model_dir=inference/ch_PP-OCRv2_rec_slim_quant_infer \
-                          --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer \
+python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv3_det_infer \
+                          --rec_model_dir=inference/ch_PP-OCRv3_rec_infer \
+                          --table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
                           --image_dir=./docs/table/table.jpg \
                           --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
-                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
                           --output=../output \
                           --vis_font_path=../doc/fonts/simfang.ttf \
                           --layout=false
@@ -63,19 +70,22 @@ python3 predict_system.py --det_model_dir=inference/ch_PP-OCRv2_det_slim_quant_i
 After the operation is completed, each image will have a directory with the same name in the `structure` directory under the directory specified by the `output` field. Each table in the image will be stored as an excel. The filename of excel is their coordinates in the image.
 
 <a name="2"></a>
-## 2. DocVQA
+## 2. KIE
 
 ```bash
 cd ppstructure
 
-# download model
 mkdir inference && cd inference
-wget https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar && tar xf PP-Layout_v1.0_ser_pretrained.tar
+# download model
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
 cd ..
-
-python3 predict_system.py --model_name_or_path=vqa/PP-Layout_v1.0_ser_pretrained/ \
-                          --mode=vqa \
-                          --image_dir=vqa/images/input/zh_val_0.jpg  \
-                          --vis_font_path=../doc/fonts/simfang.ttf
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
+  --ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
+  --ser_dict_path=../ppocr/utils/dict/kie_dict/xfund_class_list.txt \
+  --vis_font_path=../doc/fonts/simfang.ttf \
+  --ocr_order_method="tb-yx"
 ```
-After the operation is completed, each image will store the visualized image in the `vqa` directory under the directory specified by the `output` field, and the image name is the same as the input image name.
+
+After the operation is completed, each image will store the visualized image in the `kie` directory under the directory specified by the `output` field, and the image name is the same as the input image name.

ppstructure/docs/installation.md

 - [快速安装](#快速安装)
   - [1. PaddlePaddle 和 PaddleOCR](#1-paddlepaddle-和-paddleocr)
   - [2. 安装其他依赖](#2-安装其他依赖)
-    - [2.1  VQA所需依赖](#21--vqa所需依赖)
+    - [2.1 KIE所需依赖](#21-kie所需依赖)
 
 # 快速安装
 
@@ -11,16 +11,11 @@
 
 ## 2. 安装其他依赖
 
-### 2.1  VQA所需依赖
-* paddleocr
+### 2.1  KIE所需依赖
 
-```bash
-pip3 install paddleocr
-```
+* paddleocr
 
-* PaddleNLP
 ```bash
-git clone https://github.com/PaddlePaddle/PaddleNLP -b develop
-cd PaddleNLP
-pip3 install -e .
+pip install paddleocr -U
+pip install -r ./kie/requirements.txt
 ```

ppstructure/docs/installation_en.md

@@ -2,7 +2,7 @@
 
 - [1. PaddlePaddle 和 PaddleOCR](#1)
 - [2. Install other dependencies](#2)
-  - [2.1  VQA](#21)
+  - [2.1 KIE](#21)
 
 
 <a name="1"></a>
@@ -14,17 +14,11 @@ Please refer to [PaddleOCR installation documentation](../../doc/doc_en/installa
 ## 2. Install other dependencies
 
 <a name="21"></a>
-### 2.1  VQA
+### 2.1  KIE
 
 * paddleocr
 
 ```bash
-pip3 install paddleocr
-```
-
-* PaddleNLP
-```bash
-git clone https://github.com/PaddlePaddle/PaddleNLP -b develop
-cd PaddleNLP
-pip3 install -e .
+pip install paddleocr -U
+pip install -r ./kie/requirements.txt
 ```

ppstructure/docs/models_list.md

@@ -10,13 +10,17 @@
 <a name="1"></a>
 ## 1. 版面分析模型
 
-|模型名称|模型简介|下载地址|label_map|
-| --- | --- | --- | --- |
-| ppyolov2_r50vd_dcn_365e_publaynet | PubLayNet 数据集训练的版面分析模型，可以划分**文字、标题、表格、图片以及列表**5类区域 | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [训练模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) |{0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}|
-| ppyolov2_r50vd_dcn_365e_tableBank_word | TableBank Word 数据集训练的版面分析模型，只能检测表格 | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | {0:"Table"}|
-| ppyolov2_r50vd_dcn_365e_tableBank_latex | TableBank Latex 数据集训练的版面分析模型，只能检测表格 | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | {0:"Table"}|
+|模型名称|模型简介|推理模型大小|下载地址|dict path|
+| --- | --- | --- | --- | --- |
+| picodet_lcnet_x1_0_fgd_layout | 基于PicoDet LCNet_x1_0和FGD蒸馏在PubLayNet 数据集训练的英文版面分析模型，可以划分**文字、标题、表格、图片以及列表**5类区域 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams) | [PubLayNet dict](../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt) |
+| ppyolov2_r50vd_dcn_365e_publaynet | 基于PP-YOLOv2在PubLayNet数据集上训练的英文版面分析模型 | 221M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [训练模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) | 同上 |
+| picodet_lcnet_x1_0_fgd_layout_cdla | CDLA数据集训练的中文版面分析模型，可以划分为**表格、图片、图片标题、表格、表格标题、页眉、脚本、引用、公式**10类区域 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla.pdparams) | [CDLA dict](../../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt) |
+| picodet_lcnet_x1_0_fgd_layout_table | 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table.pdparams) | [Table dict](../../ppocr/utils/dict/layout_dict/layout_table_dict.txt) |
+| ppyolov2_r50vd_dcn_365e_tableBank_word | 基于PP-YOLOv2在TableBank Word 数据集训练的版面分析模型，支持英文文档表格区域的检测 | 221M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | 同上 |
+| ppyolov2_r50vd_dcn_365e_tableBank_latex | 基于PP-YOLOv2在TableBank Latex数据集训练的版面分析模型，支持英文文档表格区域的检测 | 221M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | 同上 |
 
 <a name="2"></a>
+
 ## 2. OCR和表格识别模型
 
 <a name="21"></a>
@@ -24,8 +28,8 @@
 
 |模型名称|模型简介|推理模型大小|下载地址|
 | --- | --- | --- | --- |
-|en_ppocr_mobile_v2.0_table_det|PubLayNet数据集训练的英文表格场景的文字检测|4.7M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
-|en_ppocr_mobile_v2.0_table_rec|PubLayNet数据集训练的英文表格场景的文字识别|6.9M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
+|en_ppocr_mobile_v2.0_table_det|PubTabNet数据集训练的英文表格场景的文字检测|4.7M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
+|en_ppocr_mobile_v2.0_table_rec|PubTabNet数据集训练的英文表格场景的文字识别|6.9M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
 
 如需要使用其他OCR模型，可以在 [PP-OCR model_list](../../doc/doc_ch/models_list.md) 下载模型或者使用自己训练好的模型配置到 `det_model_dir`, `rec_model_dir`两个字段即可。
 
@@ -34,9 +38,9 @@
 
 |模型名称|模型简介|推理模型大小|下载地址|
 | --- | --- | --- | --- |
-|en_ppocr_mobile_v2.0_table_structure|基于TableRec-RARE在PubTabNet数据集上训练的英文表格识别模型|18.6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
-|en_ppstructure_mobile_v2.0_SLANet|基于SLANet在PubTabNet数据集上训练的英文表格识别模型|9M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
-|ch_ppstructure_mobile_v2.0_SLANet|基于SLANet在PubTabNet数据集上训练的中文表格识别模型|9.3M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
+|en_ppocr_mobile_v2.0_table_structure|基于TableRec-RARE在PubTabNet数据集上训练的英文表格识别模型|6.8M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
+|en_ppstructure_mobile_v2.0_SLANet|基于SLANet在PubTabNet数据集上训练的英文表格识别模型|9.2M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
+|ch_ppstructure_mobile_v2.0_SLANet|基于SLANet的中文表格识别模型|9.3M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
 
 <a name="3"></a>
 

ppstructure/docs/models_list_en.md

@@ -4,18 +4,20 @@
 - [2. OCR and Table Recognition](#2-ocr-and-table-recognition)
   - [2.1 OCR](#21-ocr)
   - [2.2 Table Recognition](#22-table-recognition)
-- [3. VQA](#3-vqa)
-- [4. KIE](#4-kie)
-
+- [3. KIE](#3-kie)
 
 <a name="1"></a>
+
 ## 1. Layout Analysis
 
-|model name| description                                                                                                                                             |download|label_map|
-| --- |---------------------------------------------------------------------------------------------------------------------------------------------------------| --- | --- |
-| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis model trained on the PubLayNet dataset, the model can recognition 5 types of areas such as **text, title, table, picture and list** | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [trained model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) |{0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}|
-| ppyolov2_r50vd_dcn_365e_tableBank_word | The layout analysis model trained on the TableBank Word dataset, the model can only detect tables                                                       | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | {0:"Table"}|
-| ppyolov2_r50vd_dcn_365e_tableBank_latex | The layout analysis model trained on the TableBank Latex dataset, the model can only detect tables                                                      | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | {0:"Table"}|
+|model name| description                                                                                                                                             | inference model size                                                                                                                         |download|dict path|
+| --- |---------------------------------------------------------------------------------------------------------------------------------------------------------| --- | --- | --- |
+| picodet_lcnet_x1_0_fgd_layout | The layout analysis English model trained on the PubLayNet dataset based on PicoDet LCNet_x1_0 and FGD . the model can recognition 5 types of areas such as **Text, Title, Table, Picture and List** | 9.7M | [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams) | [PubLayNet dict](../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt) |
+| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis English model trained on the PubLayNet dataset based on PP-YOLOv2 | 221M | [inference_moel]](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [trained model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) | sme as above |
+| picodet_lcnet_x1_0_fgd_layout_cdla | The layout analysis Chinese model trained on the CDLA dataset, the model can recognition 10 types of areas such as **Table、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation** | 9.7M | [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla.pdparams) | [CDLA dict](../../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt) |
+| picodet_lcnet_x1_0_fgd_layout_table | The layout analysis model trained on the table dataset, the model can detect tables in Chinese and English documents                     | 9.7M                                                  | [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table.pdparams) | [Table dict](../../ppocr/utils/dict/layout_dict/layout_table_dict.txt) |
+| ppyolov2_r50vd_dcn_365e_tableBank_word | The layout analysis model trained on the TableBank Word dataset based on PP-YOLOv2, the model can detect  tables  in English documents | 221M | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | same as above |
+| ppyolov2_r50vd_dcn_365e_tableBank_latex | The layout analysis model trained on the TableBank Latex dataset based on PP-YOLOv2, the model can detect  tables  in English documents | 221M                 | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | same as above |
 
 <a name="2"></a>
 ## 2. OCR and Table Recognition
@@ -35,24 +37,30 @@ If you need to use other OCR models, you can download the model in [PP-OCR model
 
 |model| description                                                                 |inference model size|download|
 | --- |-----------------------------------------------------------------------------| --- | --- |
-|en_ppocr_mobile_v2.0_table_structure| English table recognition model trained on PubTabNet dataset based on TableRec-RARE |18.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
-|en_ppstructure_mobile_v2.0_SLANet|English table recognition model trained on PubTabNet dataset based on SLANet|9M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
-|ch_ppstructure_mobile_v2.0_SLANet|Chinese table recognition model trained on PubTabNet dataset based on SLANet|9.3M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
+|en_ppocr_mobile_v2.0_table_structure| English table recognition model trained on PubTabNet dataset based on TableRec-RARE |6.8M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
+|en_ppstructure_mobile_v2.0_SLANet|English table recognition model trained on PubTabNet dataset based on SLANet|9.2M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
+|ch_ppstructure_mobile_v2.0_SLANet|Chinese table recognition model based on SLANet|9.3M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
 
 <a name="3"></a>
-## 3. VQA
-
-|model| description                                                    |inference model size|download|
-| --- |----------------------------------------------------------------| --- | --- |
-|ser_LayoutXLM_xfun_zh| SER model trained on xfun Chinese dataset based on LayoutXLM   |1.4G|[inference model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar) |
-|re_LayoutXLM_xfun_zh| Re model trained on xfun Chinese dataset based on LayoutXLM    |1.4G|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar) |
-|ser_LayoutLMv2_xfun_zh| SER model trained on xfun Chinese dataset based on LayoutXLMv2 |778M|[inference model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar) |
-|re_LayoutLMv2_xfun_zh| Re model trained on xfun Chinese dataset based on LayoutXLMv2  |765M|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar) |
-|ser_LayoutLM_xfun_zh| SER model trained on xfun Chinese dataset based on LayoutLM    |430M|[inference model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar) |
-
-<a name="4"></a>
-## 4. KIE
-
-|model|description|model size|download|
-| --- | --- | --- | --- |
-|SDMGR|Key Information Extraction Model|78M|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)|
+## 3. KIE
+
+On XFUND_zh dataset, Accuracy and time cost of different models on V100 GPU are as follows.
+
+|Model|Backbone|Task|Config|Hmean|Time cost(ms)|Download link|
+| --- | --- |  --- | --- | --- | --- |--- |
+|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**| 15.49| [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%| 19.49 |[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)|
+|LayoutLM| LayoutLM-base | SER | [ser_layoutlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml)|77.31%|-|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar)|
+|LayoutLMv2| LayoutLMv2-base | SER | [ser_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlmv2_xfund_zh.yml)|85.44%|31.46|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar)|
+|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**|15.49|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|19.49|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)|
+|LayoutLMv2| LayoutLMv2-base | RE | [re_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutlmv2_xfund_zh.yml)|67.77%|31.46|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar)|
+
+* Note: The above time cost information just considers inference time without preprocess or postprocess, test environment: `V100 GPU + CUDA 10.2 + CUDNN 8.1.1 + TRT 7.2.3.4`
+
+
+On wildreceipt dataset, the algorithm result is as follows:
+
+|Model|Backbone|Config|Hmean|Download link|
+| --- | --- | --- | --- | --- |
+|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.7%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)|

ppstructure/docs/quickstart.md

@@ -7,16 +7,22 @@
     - [2.1.2 版面分析+表格识别](#212-版面分析表格识别)
     - [2.1.3 版面分析](#213-版面分析)
     - [2.1.4 表格识别](#214-表格识别)
-    - [2.1.5 DocVQA](#215-docvqa)
+    - [2.1.5 关键信息抽取](#215-关键信息抽取)
+    - [2.1.6 版面恢复](#216-版面恢复)
   - [2.2 代码使用](#22-代码使用)
-    - [2.2.1 图像方向分类版面分析表格识别](#221-图像方向分类版面分析表格识别)
+
+    - [2.2.1 图像方向+分类版面分析+表格识别](#221-图像方向分类版面分析表格识别)
     - [2.2.2 版面分析+表格识别](#222-版面分析表格识别)
     - [2.2.3 版面分析](#223-版面分析)
     - [2.2.4 表格识别](#224-表格识别)
-    - [2.2.5 DocVQA](#225-docvqa)
+
+    - [2.2.5 关键信息抽取](#225-关键信息抽取)
+    - [2.2.6 版面恢复](#226-版面恢复)
+
   - [2.3 返回结果说明](#23-返回结果说明)
     - [2.3.1 版面分析+表格识别](#231-版面分析表格识别)
-    - [2.3.2 DocVQA](#232-docvqa)
+    - [2.3.2 关键信息抽取](#232-关键信息抽取)
+
   - [2.4 参数说明](#24-参数说明)
 
 
@@ -24,11 +30,12 @@
 ## 1. 安装依赖包
 
 ```bash
-# 安装 paddleocr，推荐使用2.5+版本
-pip3 install "paddleocr>=2.5"
-# 安装 DocVQA依赖包paddlenlp（如不需要DocVQA功能，可跳过）
-pip install paddlenlp
-
+# 安装 paddleocr，推荐使用2.6版本
+pip3 install "paddleocr>=2.6"
+# 安装 关键信息抽取 依赖包（如不需要KIE功能，可跳过）
+pip install -r kie/requirements.txt
+# 安装 图像方向分类依赖包paddleclas（如不需要图像方向分类功能，可跳过）
+pip3 install paddleclas
 ```
 
 <a name="2"></a>
@@ -62,15 +69,24 @@ paddleocr --image_dir=PaddleOCR/ppstructure/docs/table/table.jpg --type=structur
 ```
 
 <a name="215"></a>
-#### 2.1.5 DocVQA
 
-请参考：[文档视觉问答](../vqa/README.md)。
+#### 2.1.5 关键信息抽取
+请参考：[关键信息抽取教程](../kie/README_ch.md)。
+
+<a name="216"></a>
+
+#### 2.1.6 版面恢复
+
+```bash
+paddleocr --image_dir=PaddleOCR/ppstructure/docs/table/1.png --type=structure --recovery=true
+```
 
 <a name="22"></a>
+
 ### 2.2 代码使用
 
 <a name="221"></a>
-#### 2.2.1 图像方向分类版面分析表格识别
+#### 2.2.1 图像方向分类+版面分析+表格识别
 
 ```python
 import os
@@ -149,6 +165,7 @@ for line in result:
 ```
 
 <a name="224"></a>
+
 #### 2.2.4 表格识别
 
 ```python
@@ -170,9 +187,36 @@ for line in result:
 ```
 
 <a name="225"></a>
-#### 2.2.5 DocVQA
+#### 2.2.5 关键信息抽取
+
+请参考：[关键信息抽取教程](../kie/README_ch.md)。
 
-请参考：[文档视觉问答](../vqa/README.md)。
+<a name="226"></a>
+
+#### 2.2.6 版面恢复
+
+```python
+import os
+import cv2
+from paddleocr import PPStructure,save_structure_res
+from paddelocr.ppstructure.recovery.recovery_to_doc import sorted_layout_boxes, convert_info_docx
+
+table_engine = PPStructure(layout=False, show_log=True)
+
+save_folder = './output'
+img_path = 'PaddleOCR/ppstructure/docs/table/1.png'
+img = cv2.imread(img_path)
+result = table_engine(img)
+save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
+
+for line in result:
+    line.pop('img')
+    print(line)
+
+h, w, _ = img.shape
+res = sorted_layout_boxes(res, w)
+convert_info_docx(img, result, save_folder, os.path.basename(img_path).split('.')[0])
+```
 
 <a name="23"></a>
 ### 2.3 返回结果说明
@@ -208,9 +252,9 @@ dict 里各个字段说明如下
   ```
 
 <a name="232"></a>
-#### 2.3.2 DocVQA
+#### 2.3.2 关键信息抽取
 
-请参考：[文档视觉问答](../vqa/README.md)。
+请参考：[关键信息抽取教程](../kie/README_ch.md)。
 
 <a name="24"></a>
 ### 2.4 参数说明
@@ -226,15 +270,16 @@ dict 里各个字段说明如下
 | layout_dict_path  | 版面分析模型字典| ../ppocr/utils/dict/layout_publaynet_dict.txt |
 | layout_score_threshold  | 版面分析模型检测框阈值| 0.5|
 | layout_nms_threshold  | 版面分析模型nms阈值| 0.5|
-| vqa_algorithm  | vqa模型算法| LayoutXLM|
+| kie_algorithm  | kie模型算法| LayoutXLM|
 | ser_model_dir  | ser模型  inference 模型地址| None|
 | ser_dict_path  | ser模型字典| ../train_data/XFUND/class_list_xfun.txt|
-| mode | structure or vqa  | structure   |
+| mode | structure or kie  | structure   |
 | image_orientation | 前向中是否执行图像方向分类  | False   |
 | layout | 前向中是否执行版面分析  | True   |
 | table  | 前向中是否执行表格识别  | True   |
 | ocr    | 对于版面分析中的非表格区域，是否执行ocr。当layout为False时会被自动设置为False| True |
 | recovery    | 前向中是否执行版面恢复| False |
+| save_pdf | 版面恢复导出docx文件的同时，是否导出pdf文件 | False |
 | structure_version |  模型版本，可选 PP-structure和PP-structurev2  | PP-structure |
 
 大部分参数和PaddleOCR whl包保持一致，见 [whl包文档](../../doc/doc_ch/whl.md)

ppstructure/docs/quickstart_en.md

@@ -7,16 +7,19 @@
     - [2.1.2 layout analysis + table recognition](#212-layout-analysis--table-recognition)
     - [2.1.3 layout analysis](#213-layout-analysis)
     - [2.1.4 table recognition](#214-table-recognition)
-    - [2.1.5 DocVQA](#215-docvqa)
+    - [2.1.5 Key Information Extraction](#215-Key-Information-Extraction)
+    - [2.1.6 layout recovery](#216-layout-recovery)
   - [2.2 Use by code](#22-use-by-code)
     - [2.2.1 image orientation + layout analysis + table recognition](#221-image-orientation--layout-analysis--table-recognition)
     - [2.2.2 layout analysis + table recognition](#222-layout-analysis--table-recognition)
     - [2.2.3 layout analysis](#223-layout-analysis)
     - [2.2.4 table recognition](#224-table-recognition)
-    - [2.2.5 DocVQA](#225-docvqa)
+    - [2.2.5 DocVQA](#225-dockie)
+    - [2.2.5 Key Information Extraction](#225-Key-Information-Extraction)
+    - [2.2.6 layout recovery](#226-layout-recovery)  
   - [2.3 Result description](#23-result-description)
     - [2.3.1 layout analysis + table recognition](#231-layout-analysis--table-recognition)
-    - [2.3.2 DocVQA](#232-docvqa)
+    - [2.3.2 Key Information Extraction](#232-Key-Information-Extraction)
   - [2.4 Parameter Description](#24-parameter-description)
 
 
@@ -24,14 +27,16 @@
 ## 1. Install package
 
 ```bash
-# Install paddleocr, version 2.5+ is recommended
-pip3 install "paddleocr>=2.5"
-# Install the DocVQA dependency package paddlenlp (if you do not use the DocVQA, you can skip it)
-pip install paddlenlp
-
+# Install paddleocr, version 2.6 is recommended
+pip3 install "paddleocr>=2.6"
+# Install the KIE dependency packages (if you do not use the KIE, you can skip it)
+pip install -r kie/requirements.txt
+# Install the image direction classification dependency package paddleclas (if you do not use the image direction classification, you can skip it)
+pip3 install paddleclas
 ```
 
 <a name="2"></a>
+
 ## 2. Use
 
 <a name="21"></a>
@@ -62,9 +67,15 @@ paddleocr --image_dir=PaddleOCR/ppstructure/docs/table/table.jpg --type=structur
 ```
 
 <a name="215"></a>
-#### 2.1.5 DocVQA
+#### 2.1.5 Key Information Extraction
 
-Please refer to: [Documentation Visual Q&A](../vqa/README.md) .
+Please refer to: [Key Information Extraction](../kie/README.md) .
+
+<a name="216"></a>
+#### 2.1.6 layout recovery
+```bash
+paddleocr --image_dir=PaddleOCR/ppstructure/docs/table/1.png --type=structure --recovery=true
+```
 
 <a name="22"></a>
 ### 2.2 Use by code
@@ -120,7 +131,7 @@ for line in result:
 
 from PIL import Image
 
-font_path = 'PaddleOCR/doc/fonts/simfang.ttf' # PaddleOCR下提供字体包
+font_path = 'PaddleOCR/doc/fonts/simfang.ttf' # font provieded in PaddleOCR
 image = Image.open(img_path).convert('RGB')
 im_show = draw_structure_result(image, result,font_path=font_path)
 im_show = Image.fromarray(im_show)
@@ -170,9 +181,35 @@ for line in result:
 ```
 
 <a name="225"></a>
-#### 2.2.5 DocVQA
+#### 2.2.5 Key Information Extraction
+
+Please refer to: [Key Information Extraction](../kie/README.md) .
+
+<a name="226"></a>
+#### 2.2.6 layout recovery
+
+```python
+import os
+import cv2
+from paddleocr import PPStructure,save_structure_res
+from paddelocr.ppstructure.recovery.recovery_to_doc import sorted_layout_boxes, convert_info_docx
 
-Please refer to: [Documentation Visual Q&A](../vqa/README.md) .
+table_engine = PPStructure(layout=False, show_log=True)
+
+save_folder = './output'
+img_path = 'PaddleOCR/ppstructure/docs/table/1.png'
+img = cv2.imread(img_path)
+result = table_engine(img)
+save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
+
+for line in result:
+    line.pop('img')
+    print(line)
+
+h, w, _ = img.shape
+res = sorted_layout_boxes(res, w)
+convert_info_docx(img, result, save_folder, os.path.basename(img_path).split('.')[0])
+```
 
 <a name="23"></a>
 ### 2.3 Result description
@@ -208,9 +245,9 @@ After the recognition is completed, each image will have a directory with the sa
   ```
 
 <a name="232"></a>
-#### 2.3.2 DocVQA
+#### 2.3.2 Key Information Extraction
 
-Please refer to: [Documentation Visual Q&A](../vqa/README.md) .
+Please refer to: [Key Information Extraction](../kie/README.md) .
 
 <a name="24"></a>
 ### 2.4 Parameter Description
@@ -226,15 +263,16 @@ Please refer to: [Documentation Visual Q&A](../vqa/README.md) .
 | layout_dict_path  | The dictionary path of layout analysis model| ../ppocr/utils/dict/layout_publaynet_dict.txt |
 | layout_score_threshold  | The box threshold path of layout analysis model| 0.5|
 | layout_nms_threshold  | The nms threshold path of layout analysis model| 0.5|
-| vqa_algorithm  | vqa model algorithm| LayoutXLM|
+| kie_algorithm  | kie model algorithm| LayoutXLM|
 | ser_model_dir  | Ser model inference model path| None|
 | ser_dict_path  | The dictionary path of Ser model| ../train_data/XFUND/class_list_xfun.txt|
-| mode | structure or vqa  | structure   |
+| mode | structure or kie  | structure   |
 | image_orientation | Whether to perform image orientation classification in forward  | False   |
 | layout | Whether to perform layout analysis in forward  | True   |
 | table  | Whether to perform table recognition in forward  | True   |
 | ocr    | Whether to perform ocr for non-table areas in layout analysis. When layout is False, it will be automatically set to False| True |
 | recovery    | Whether to perform layout recovery in forward| False |
+| save_pdf    | Whether to convert docx to pdf when recovery| False |
 | structure_version |  Structure version, optional PP-structure and PP-structurev2  | PP-structure |
 
 Most of the parameters are consistent with the PaddleOCR whl package, see [whl package documentation](../../doc/doc_en/whl.md)

ppstructure/kie/README.md

+English | [简体中文](README_ch.md)
+
+- [1. Introduction](#1-introduction)
+
+- [2. Accuracy and performance](#2-Accuracy-and-performance)
+- [3. Visualization](#3-Visualization)
+  - [3.1 SER](#31-ser)
+  - [3.2 RE](#32-re)
+- [4. Usage](#4-usage)
+  - [4.1 Prepare for the environment](#41-Prepare-for-the-environment)
+  - [4.2 Quick start](#42-Quick-start)
+  - [4.3 More](#43-More)
+- [5. Reference](#5-Reference)
+- [6. License](#6-License)
+
+
+## 1. Introduction
+
+Key information extraction (KIE) refers to extracting key information from text or images. As downstream task of OCR, the key information extraction task of document image has many practical application scenarios, such as form recognition, ticket information extraction, ID card information extraction, etc.
+
+PP-Structure conducts research based on the LayoutXLM multi-modal, and proposes the VI-LayoutXLM, which gets rid of visual features when finetuning the downstream tasks. An textline sorting method is also utilized to fit in reading order. What's more, UDML knowledge distillation is used for higher accuracy. Finally, the accuracy and inference speed of VI-LayoutXLM surpass those of LayoutXLM.
+
+The main features of the key information extraction module in PP-Structure are as follows.
+
+
+- Integrate multi-modal methods such as [LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf), VI-LayoutXLM, and PP-OCR inference engine.
+- Supports Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks based on multimodal methods. Based on the SER task, the text recognition and classification in the image can be completed; based on the RE task, the relationship extraction of the text content in the image can be completed, such as judging the problem pair (pair).
+- Supports custom training for SER tasks and RE tasks.
+- Supports end-to-end system prediction and evaluation of OCR+SER.
+- Supports end-to-end system prediction of OCR+SER+RE.
+- Support SER model export and inference using PaddleInference.
+
+
+## 2. Accuracy and performance
+
+We evaluate the methods on the Chinese dataset of [XFUND](https://github.com/doc-analysis/XFUND), and the performance is as follows
+
+|Model | Backbone | Task | Config file | Hmean | Inference time (ms) | Download link|
+| --- | --- |  --- | --- | --- | --- | --- |
+|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**| 15.49|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%| 19.49 | [trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)|
+|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**| 15.49|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%| 19.49|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)|
+
+
+* Note：Inference environment：V100 GPU + cuda10.2 + cudnn8.1.1 + TensorRT 7.2.3.4，tested using fp16.
+
+For more KIE models in PaddleOCR, please refer to [KIE model zoo](../../doc/doc_en/algorithm_overview_en.md).
+
+
+## 3. Visualization
+
+There are two main solutions to the key information extraction task based on VI-LayoutXLM series model.
+
+(1) Text detection + text recognition + semantic entity recognition (SER)
+
+(2) Text detection + text recognition + semantic entity recognition (SER) + relationship extraction (RE)
+
+
+The following images are demo results of the SER and RE models. For more detailed introduction to the above solutions, please refer to [KIE Guide](./how_to_do_kie.md).
+
+### 3.1 SER
+
+Demo results for SER task are as follows.
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185539141-68e71c75-5cf7-4529-b2ca-219d29fa5f68.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185310636-6ce02f7c-790d-479f-b163-ea97a5a04808.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185539517-ccf2372a-f026-4a7c-ad28-c741c770f60a.png" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185539735-37b5c2ef-629d-43fe-9abb-44bb717ef7ee.jpg" width="600">
+</div>
+
+
+
+**Note:** test pictures are from [xfund dataset](https://github.com/doc-analysis/XFUND), [invoice dataset](https://aistudio.baidu.com/aistudio/datasetdetail/165561) and a composite ID card dataset.
+
+
+Boxes of different colors in the image represent different categories.
+
+The invoice and application form images have three categories: `request`, `answer` and `header`. The `question` and 'answer' can be used to extract the relationship.
+
+For the ID card image, the mdoel can be directly identify the key information such as `name`, `gender`, `nationality`, so that the subsequent relationship extraction process is not required, and the key information extraction task can be completed using only on model.
+
+### 3.2 RE
+
+Demo results for RE task are as follows.
+
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185393805-c67ff571-cf7e-4217-a4b0-8b396c4f22bb.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185540080-0431e006-9235-4b6d-b63d-0b3c6e1de48f.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185540291-f64e5daf-6d42-4e7c-bbbb-471e3fac4fcc.png" width="600">
+</div>
+
+Red boxes are questions, blue boxes are answers. The green lines means the two conected objects are a pair.
+
+
+## 4. Usage
+
+### 4.1 Prepare for the environment
+
+
+Use the following command to install KIE dependencies.
+
+
+```bash
+git clone https://github.com/PaddlePaddle/PaddleOCR.git
+cd PaddleOCR
+pip install -r requirements.txt
+pip install -r ppstructure/kie/requirements.txt
+# 安装PaddleOCR引擎用于预测
+pip install paddleocr -U
+```
+
+The visualized results of SER are saved in the `./output` folder by default. Examples of results are as follows.
+
+
+<div align="center">
+    <img src="../../ppstructure/docs/kie/result_ser/zh_val_42_ser.jpg" width="800">
+</div>
+
+
+### 4.2 Quick start
+
+Here we use XFUND dataset to quickly experience the SER model and RE model.
+
+
+#### 4.2.1 Prepare for the dataset
+
+```bash
+mkdir train_data
+cd train_data
+# download and uncompress the dataset
+wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar
+cd ..
+```
+
+#### 4.2.2 Predict images using the trained model
+
+Use the following command to download the models.
+
+```bash
+mkdir pretrained_model
+cd pretrained_model
+# download and uncompress the SER trained model
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar && tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
+
+# download and uncompress the RE trained model
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar && tar -xf re_vi_layoutxlm_xfund_pretrained.tar
+```
+
+
+If you want to use OCR engine to obtain end-to-end prediction results, you can use the following command to predict.
+
+```bash
+# just predict using SER trained model
+python3 tools/infer_kie_token_ser.py \
+  -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
+
+# predict using SER and RE trained model at the same time
+python3 ./tools/infer_kie_token_ser_re.py \
+  -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./train_data/XFUND/zh_val/image/zh_val_42.jpg \
+  -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o_ser Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
+```
+
+The visual result images and the predicted text file will be saved in the `Global.save_res_path` directory.
+
+
+If you want to load the text detection and recognition results collected before, you can use the following command to predict.
+
+```bash
+# just predict using SER trained model
+python3 tools/infer_kie_token_ser.py \
+  -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./train_data/XFUND/zh_val/val.json \
+  Global.infer_mode=False
+
+# predict using SER and RE trained model at the same time
+python3 ./tools/infer_kie_token_ser_re.py \
+  -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./train_data/XFUND/zh_val/val.json \
+  Global.infer_mode=False \
+  -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o_ser Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
+```
+
+#### 4.2.3 Inference using PaddleInference
+
+At present, only SER model supports inference using PaddleInference.
+
+Firstly, download the inference SER inference model.
+
+
+```bash
+mkdir inference
+cd inference
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
+```
+
+Use the following command for inference.
+
+
+```bash
+cd ppstructure
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
+  --ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
+  --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
+  --vis_font_path=../doc/fonts/simfang.ttf \
+  --ocr_order_method="tb-yx"
+```
+
+The visual results and text file will be saved in directory `output`.
+
+
+### 4.3 More
+
+For training, evaluation and inference tutorial for KIE models, please refer to [KIE doc](../../doc/doc_en/kie_en.md).
+
+For training, evaluation and inference tutorial for text detection models, please refer to [text detection doc](../../doc/doc_en/detection_en.md).
+
+For training, evaluation and inference tutorial for text recognition models, please refer to [text recognition doc](../../doc/doc_en/recognition.md).
+
+If you want to finish the KIE tasks in your scene, and don't know what to prepare,  please refer to [End cdoc](../../doc/doc_en/recognition.md).
+
+To complete the key information extraction task in your own scenario from data preparation to model selection, please refer to: [Guide to End-to-end KIE](./how_to_do_kie_en.md)。
+
+
+## 5. Reference
+
+- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding, https://arxiv.org/pdf/2104.08836.pdf
+- microsoft/unilm/layoutxlm, https://github.com/microsoft/unilm/tree/master/layoutxlm
+- XFUND dataset, https://github.com/doc-analysis/XFUND
+
+## 6. License
+
+The content of this project itself is licensed under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)

ppstructure/kie/README_ch.md

+[English](README.md) | 简体中文
+
+# 关键信息抽取
+
+- [1. 简介](#1-简介)
+- [2. 精度与性能](#2-精度与性能)
+- [3. 效果演示](#3-效果演示)
+  - [3.1 SER](#31-ser)
+  - [3.2 RE](#32-re)
+- [4. 使用](#4-使用)
+  - [4.1 准备环境](#41-准备环境)
+  - [4.2 快速开始](#42-快速开始)
+  - [4.3 更多](#43-更多)
+- [5. 参考链接](#5-参考链接)
+- [6. License](#6-License)
+
+
+## 1. 简介
+
+关键信息抽取 (Key Information Extraction, KIE)指的是是从文本或者图像中，抽取出关键的信息。针对文档图像的关键信息抽取任务作为OCR的下游任务，存在非常多的实际应用场景，如表单识别、车票信息抽取、身份证信息抽取等。
+
+PP-Structure 基于 LayoutXLM 文档多模态系列方法进行研究与优化，设计了视觉特征无关的多模态模型结构VI-LayoutXLM，同时引入符合阅读顺序的文本行排序方法以及UDML联合互学习蒸馏方法，最终在精度与速度均超越LayoutXLM。
+
+PP-Structure中关键信息抽取模块的主要特性如下：
+
+- 集成[LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf)、VI-LayoutXLM等多模态模型以及PP-OCR预测引擎。
+- 支持基于多模态方法的语义实体识别 (Semantic Entity Recognition, SER) 以及关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务，可以完成对图像中的文本识别与分类；基于 RE 任务，可以完成对图象中的文本内容的关系提取，如判断问题对(pair)。
+- 支持SER任务和RE任务的自定义训练。
+- 支持OCR+SER的端到端系统预测与评估。
+- 支持OCR+SER+RE的端到端系统预测。
+- 支持SER模型的动转静导出与基于PaddleInfernece的模型推理。
+
+
+## 2. 精度与性能
+
+
+我们在 [XFUND](https://github.com/doc-analysis/XFUND) 的中文数据集上对算法进行了评估，SER与RE上的任务性能如下
+
+|模型|骨干网络|任务|配置文件|hmean|预测耗时(ms)|下载链接|
+| --- | --- |  --- | --- | --- | --- | --- |
+|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**| 15.49|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%| 19.49 | [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)|
+|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**| 15.49|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)|
+|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%| 19.49|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)|
+
+
+* 注：预测耗时测试条件：V100 GPU + cuda10.2 + cudnn8.1.1 + TensorRT 7.2.3.4，使用FP16进行测试。
+
+更多关于PaddleOCR中关键信息抽取模型的介绍，请参考[关键信息抽取模型库](../../doc/doc_ch/algorithm_overview.md)。
+
+
+## 3. 效果演示
+
+基于多模态模型的关键信息抽取任务有2种主要的解决方案。
+
+（1）文本检测 + 文本识别 + 语义实体识别(SER)
+（2）文本检测 + 文本识别 + 语义实体识别(SER) + 关系抽取(RE)
+
+下面给出SER与RE任务的示例效果，关于上述解决方案的详细介绍，请参考[关键信息抽取全流程指南](./how_to_do_kie.md)。
+
+### 3.1 SER
+
+对于SER任务，效果如下所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185539141-68e71c75-5cf7-4529-b2ca-219d29fa5f68.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185310636-6ce02f7c-790d-479f-b163-ea97a5a04808.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185539517-ccf2372a-f026-4a7c-ad28-c741c770f60a.png" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185539735-37b5c2ef-629d-43fe-9abb-44bb717ef7ee.jpg" width="600">
+</div>
+
+**注意：** 测试图片来源于[XFUND数据集](https://github.com/doc-analysis/XFUND)、[发票数据集](https://aistudio.baidu.com/aistudio/datasetdetail/165561)以及合成的身份证数据集。
+
+
+图中不同颜色的框表示不同的类别。
+
+图中的发票以及申请表图像，有`QUESTION`, `ANSWER`, `HEADER` 3种类别，识别的`QUESTION`, `ANSWER`可以用于后续的问题与答案的关系抽取。
+
+图中的身份证图像，则直接识别出其中的`姓名`、`性别`、`民族`等关键信息，这样就无需后续的关系抽取过程，一个模型即可完成关键信息抽取。
+
+
+### 3.2 RE
+
+对于RE任务，效果如下所示。
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185393805-c67ff571-cf7e-4217-a4b0-8b396c4f22bb.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185540080-0431e006-9235-4b6d-b63d-0b3c6e1de48f.jpg" width="600">
+</div>
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185540291-f64e5daf-6d42-4e7c-bbbb-471e3fac4fcc.png" width="600">
+</div>
+
+
+红色框是问题，蓝色框是答案。绿色线条表示连接的两端为一个key-value的pair。
+
+## 4. 使用
+
+### 4.1 准备环境
+
+使用下面的命令安装运行SER与RE关键信息抽取的依赖。
+
+```bash
+git clone https://github.com/PaddlePaddle/PaddleOCR.git
+cd PaddleOCR
+pip install -r requirements.txt
+pip install -r ppstructure/kie/requirements.txt
+# 安装PaddleOCR引擎用于预测
+pip install paddleocr -U
+```
+
+### 4.2 快速开始
+
+下面XFUND数据集，快速体验SER模型与RE模型。
+
+#### 4.2.1 准备数据
+
+```bash
+mkdir train_data
+cd train_data
+# 下载与解压数据
+wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar
+cd ..
+```
+
+#### 4.2.2 基于动态图的预测
+
+首先下载模型。
+
+```bash
+mkdir pretrained_model
+cd pretrained_model
+# 下载并解压SER预训练模型
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar && tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
+
+# 下载并解压RE预训练模型
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar && tar -xf re_vi_layoutxlm_xfund_pretrained.tar
+```
+
+如果希望使用OCR引擎，获取端到端的预测结果，可以使用下面的命令进行预测。
+
+```bash
+# 仅预测SER模型
+python3 tools/infer_kie_token_ser.py \
+  -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
+
+# SER + RE模型串联
+python3 ./tools/infer_kie_token_ser_re.py \
+  -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./train_data/XFUND/zh_val/image/zh_val_42.jpg \
+  -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o_ser Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
+```
+
+`Global.save_res_path`目录中会保存可视化的结果图像以及预测的文本文件。
+
+
+如果希望加载标注好的文本检测与识别结果，仅预测可以使用下面的命令进行预测。
+
+```bash
+# 仅预测SER模型
+python3 tools/infer_kie_token_ser.py \
+  -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./train_data/XFUND/zh_val/val.json \
+  Global.infer_mode=False
+
+# SER + RE模型串联
+python3 ./tools/infer_kie_token_ser_re.py \
+  -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
+  -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
+  Global.infer_img=./train_data/XFUND/zh_val/val.json \
+  Global.infer_mode=False \
+  -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
+  -o_ser Architecture.Backbone.checkpoints=./pretrain_models/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
+```
+
+#### 4.2.3 基于PaddleInference的预测
+
+目前仅SER模型支持PaddleInference推理。
+
+首先下载SER的推理模型。
+
+
+```bash
+mkdir inference
+cd inference
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
+```
+
+执行下面的命令进行预测。
+
+```bash
+cd ppstructure
+python3 kie/predict_kie_token_ser.py \
+  --kie_algorithm=LayoutXLM \
+  --ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
+  --image_dir=./docs/kie/input/zh_val_42.jpg \
+  --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
+  --vis_font_path=../doc/fonts/simfang.ttf \
+  --ocr_order_method="tb-yx"
+```
+
+可视化结果保存在`output`目录下。
+
+### 4.3 更多
+
+关于KIE模型的训练评估与推理，请参考：[关键信息抽取教程](../../doc/doc_ch/kie.md)。
+
+关于文本检测模型的训练评估与推理，请参考：[文本检测教程](../../doc/doc_ch/detection.md)。
+
+关于文本识别模型的训练评估与推理，请参考：[文本识别教程](../../doc/doc_ch/recognition.md)。
+
+关于怎样在自己的场景中完成关键信息抽取任务，请参考：[关键信息抽取全流程指南](./how_to_do_kie.md)。
+
+
+## 5. 参考链接
+
+- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding, https://arxiv.org/pdf/2104.08836.pdf
+- microsoft/unilm/layoutxlm, https://github.com/microsoft/unilm/tree/master/layoutxlm
+- XFUND dataset, https://github.com/doc-analysis/XFUND
+
+## 6. License
+
+The content of this project itself is licensed under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)

ppstructure/kie/how_to_do_kie_en.md

+
+# Key Information Extraction Pipeline
+
+- [1. Introduction](#1-Introduction)
+  - [1.1 Background](#11-Background)
+  - [1.2 Mainstream Deep-learning Solutions](#12-Mainstream-Deep-learning-Solutions)
+- [2. KIE Pipeline](#2-KIE-Pipeline)
+  - [2.1 Train OCR Models](#21-Train-OCR-Models)
+  - [2.2 Train KIE Models](#22-Train-KIE-Models)
+- [3. Reference](#3-Reference)
+
+
+## 1. Introduction
+
+### 1.1 Background
+
+Key information extraction (KIE) refers to extracting key information from text or images. As the downstream task of OCR, KIE of document image has many practical application scenarios, such as form recognition, ticket information extraction, ID card information extraction, etc. However, it is time-consuming and laborious to extract  key information from these document images by manpower. It's challengable but also valuable to combine multi-modal features (visual, layout, text, etc) together and complete KIE tasks.
+
+For the document images in a specific scene, the position and layout of the key information are relatively fixed. Therefore, in the early stage of the research, there are many methods based on template matching to extract the key information. This method is still widely used in many simple scenarios at present. However, it takes long time to adjut the template for different scenarios.
+
+
+The KIE in the document image generally contains 2 subtasks, which is as shown follows.
+
+* (1) SER: semantic entity recognition, which classifies each detected textline, such as dividing it into name and ID card. As shown in the red boxes in the following figure.
+
+* (2) RE: relationship extraction, which matches the question and answer based on SER results. As shown in the figure below, the yellow arrows match the question and answer.
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185726510-faba470d-2c79-4784-b8da-6c1aa5af9572.png" width="800">
+</div>
+
+
+
+### 1.2 Mainstream Deep-learning Solutions
+
+General KIE methods are based on Named Entity Recognition (NER), but such methods only use text information and ignore location and visual feature information, which leads to limited accuracy. In recent years, most scholars have started to combine mutil-modal features to improve the accuracy of KIE model. The main methods are as follows:
+
+* (1) Grid based methods. These methods mainly focus on the fusion of multi-modal information at the image level. Most texts are of character granularity. The text and structure information embedding method is simple, such as the algorithm of chargrid [1].
+
+* (2) Token based methods. These methods refer to the NLP methods such as Bert, which encode the position, vision and other feature information into the multi-modal model, and conduct pre-training on large-scale datasets, so that in downstream tasks, only a small amount of annotation data is required to obtain excellent results. The representative algorithms are layoutlm [2], layoutlmv2 [3], layoutxlm [4], structext [5], etc.
+
+* (3) GCN based methods. These methods try to learn the structural information between images and characters, so as to solve the problem of extracting open set information (templates not seen in the training set), such as GCN [6], SDMGR [7] and other algorithms.
+
+* (4) End to end based methods: these methods put the existing OCR character recognition and KIE information extraction tasks into a unified network for common learning, and strengthen each other in the learning process. Such as TRIE [8].
+
+
+For more detailed introduction of the algorithms, please refer to Chapter 6 of [Diving into OCR](https://aistudio.baidu.com/aistudio/education/group/info/25207).
+
+## 2. KIE Pipeline
+
+Token based methods such as LayoutXLM are implemented in PaddleOCR. What's more, in PP-Structurev2, we simplify the LayoutXLM model and proposed VI-LayoutXLM, in which the visual feature extraction module is removed for speed-up. The textline sorting strategy conforming to the human reading order and UDML knowledge distillation strategy are utilized for higher model accuracy.
+
+
+In the non end-to-end KIE method, KIE needs at least ** 2 steps**. Firstly, the OCR model is used to extract the text and its position. Secondly, the KIE model is used to extract the key information according to the image, text position and text content.
+
+
+### 2.1 Train OCR Models
+
+#### 2.1.1 Text Detection
+
+**(1) Data**
+
+Most of the models provided in PaddleOCR are general models. In the process of text detection, the detection of adjacent text lines is generally based on the distance of the position. As shown in the figure above, when using PP-OCRv3 general English detection model for text detection, it is easy to detect the two fields representing different propoerties as one. Therefore, it is suggested to finetune a detection model according to your scenario firstly during the KIE task.
+
+
+During data annotation, the different key information needs to be separated. Otherwise, it will increase the difficulty of subsequent KIE tasks.
+
+For downstream tasks, generally speaking, `200~300` training images can guarantee the basic training effect. If there is not too much prior knowledge, **`200~300`** images can be labeled firstly for subsequent text detection model training.
+
+**(2) Model**
+
+In terms of model selection, PP-OCRv3 detection model is recommended. For more information about the training methods of the detection model, please refer to: [Text detection tutorial](../../doc/doc_en/detection_en.md) and [PP-OCRv3 detection model tutorial](../../doc/doc_ch/PPOCRv3_det_train.md).
+
+#### 2.1.2 Text recognition
+
+
+Compared with the natural scene, the text recognition in the document image is generally relatively easier (the background is not too complex), so **it is suggested to** try the PP-OCRv3 general text recognition model provided in PaddleOCR ([PP-OCRv3 model list](../../doc/doc_en/models_list_en.md))
+
+
+**(1) Data**
+
+However, there are also some challenges in some document scenarios, such as rare words in ID card scenarios and special fonts in invoice and other scenarios. These problems will increase the difficulty of text recognition. At this time, if you want to ensure or further improve the model accuracy, it is recommended to load PP-OCRv3 model based on the text recognition dataset of specific document scenarios for finetuning.
+
+In the process of model finetuning, it is recommended to prepare at least `5000` vertical scene text recognition images to ensure the basic model fine-tuning effect. If you want to improve the accuracy and generalization ability of the model, you can synthesize more text recognition images similar to the scene, collect general real text recognition data from the public data set, and add them to the text recognition training process. In the training process, it is suggested that the ratio of real data, synthetic data and general data of each epoch should be around `1:1:1`, which can be controlled by setting the sampling ratio of different data sources. If there are 3 training text files, including 10k, 20k and 50k pieces of data respectively, the data can be set in the configuration file as follows:
+
+```yml
+Train:
+  dataset:
+    name: SimpleDataSet
+    data_dir: ./train_data/
+    label_file_list:
+    - ./train_data/train_list_10k.txt
+    - ./train_data/train_list_10k.txt
+    - ./train_data/train_list_50k.txt
+    ratio_list: [1.0, 0.5, 0.2]
+    ...
+```
+
+**(2) Model**
+
+In terms of model selection, PP-OCRv3 recognition model is recommended. For more information about the training methods of the recognition model, please refer to: [Text recognition tutorial](../../doc/doc_en/recognition_en.md) and [PP-OCRv3 model list](../../doc/doc_en/models_list_en.md).
+
+
+### 2.2 Train KIE Models
+
+There are two main methods to extract the key information from the recognized texts.
+
+(1) Directly use SER model to obtain the key information category. For example, in the ID card scenario, we mark "name" and "Geoff Sample" as "name_key" and "name_value", respectively. The **text field** corresponding to the category "name_value" finally identified is the key information we need.
+
+(2) Joint use SER and RE models. For this case, we firstly use SER model to obtain all questions (keys) and questions (values) for the image text, and then use RE model to match all keys and values to find the relationship, so as to complete the extraction of key information.
+
+#### 2.2.1 SER
+
+Take the ID card scenario as an example. The key information generally includes `name`, `DOB`, etc. We can directly mark the corresponding fields as specific categories, as shown in the following figure.
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185728456-dc396f47-0880-4279-9c7c-c99601bf16a7.png" width="500">
+</div>
+
+**Note:**
+
+- In the labeling process, text content without key information about KIE shall be labeled as`other`, which is equivalent to background information. For example, in the ID card scenario, if we do not pay attention to `DOB` information, we can mark the categories of `DOB` and `Area manager` as `other`.
+- In the annotation process of, it is required to annotate the **textline** position rather than the character.
+
+
+In terms of data, generally speaking, for relatively fixed scenes, **50** training images can achieve acceptable effects. You can refer to [PPOCRLabel](../../PPOCRLabel/README.md) for finish the labeling process.
+
+In terms of model, it is recommended to use the VI-layoutXLM model proposed in PP-Structurev2. It is improved based on the LayoutXLM model, removing the visual feature extraction module, and further improving the model inference speed without the significant reduction on model accuracy. For more tutorials, please refer to [VI-LayoutXLM introduction](../../doc/doc_en/algorithm_kie_vi_layoutxlm_en.md) and [KIE tutorial](../../doc/doc_en/kie_en.md).
+
+
+#### 2.2.2 SER + RE
+
+The SER model is mainly used to identify all keys and values in the document image, and the RE model is mainly used to match all keys and values.
+
+Taking the ID card scenario as an example, the key information generally includes key information such as `name`, `DOB`, etc. in the SER stage, we need to identify all questions (keys) and answers (values). The demo annotation is as follows. All keys can be annotated as `question`, and all keys can be annotated as `answer`.
+
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185728881-b6055e01-034c-4584-aaa6-97c9c25fb61b.png" width="500">
+</div>
+
+
+In the RE stage, the ID and connection information of each field need to be marked, as shown in the following figure.
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/14270174/185728948-a4208013-5038-4025-9a93-0c6d51447488.png" width="500">
+</div>
+
+For each textline, you need to add 'ID' and 'linking' field information. The 'ID' records the unique identifier of the textline. Different text contents in the same images cannot be repeated. The 'linking' is a list that records the connection information between different texts. If the ID of the field "name" is 0 and the ID of the field "Geoff Sample" is 1, then they all have [[0, 1]] 'linking' marks, indicating that the fields with `id=0` and `id=1` form a key value relationship (the fields such as DOB and Expires are similar, and will not be repeated here).
+
+
+**Note:**
+
+-During annotation, if value is multiple textines, a key value pair can be added in linking, such as `[[0, 1], [0, 2]]`.
+
+In terms of data, generally speaking, for relatively fixed scenes, about **50** training images can achieve acceptable effects.
+
+In terms of model, it is recommended to use the VI-layoutXLM model proposed in PP-Structurev2. It is improved based on the LayoutXLM model, removing the visual feature extraction module, and further improving the model inference speed without the significant reduction on model accuracy. For more tutorials, please refer to [VI-LayoutXLM introduction](../../doc/doc_en/algorithm_kie_vi_layoutxlm_en.md) and [KIE tutorial](../../doc/doc_en/kie_en.md).
+
+
+
+## 3. Reference
+
+
+[1] Katti A R, Reisswig C, Guder C, et al. Chargrid: Towards understanding 2d documents[J]. arXiv preprint arXiv:1809.08799, 2018.
+
+[2] Xu Y, Li M, Cui L, et al. Layoutlm: Pre-training of text and layout for document image understanding[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020: 1192-1200.
+
+[3] Xu Y, Xu Y, Lv T, et al. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding[J]. arXiv preprint arXiv:2012.14740, 2020.
+
+[4]: Xu Y, Lv T, Cui L, et al. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding[J]. arXiv preprint arXiv:2104.08836, 2021.
+
+[5] Li Y, Qian Y, Yu Y, et al. StrucTexT: Structured Text Understanding with Multi-Modal Transformers[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 1912-1920.
+
+[6] Liu X, Gao F, Zhang Q, et al. Graph convolution for multimodal information extraction from visually rich documents[J]. arXiv preprint arXiv:1903.11279, 2019.
+
+[7] Sun H, Kuang Z, Yue X, et al. Spatial Dual-Modality Graph Reasoning for Key Information Extraction[J]. arXiv preprint arXiv:2103.14470, 2021.
+
+[8] Zhang P, Xu Y, Cheng Z, et al. Trie: End-to-end text reading and information extraction for document understanding[C]//Proceedings of the 28th ACM International Conference on Multimedia. 2020: 1413-1422.

ppstructure/vqa/predict_vqa_token_ser.py → ppstructure/kie/predict_kie_token_ser.py

@@ -30,7 +30,7 @@ from ppocr.data import create_operators, transform
 from ppocr.postprocess import build_post_process
 from ppocr.utils.logging import get_logger
 from ppocr.utils.visual import draw_ser_results
-from ppocr.utils.utility import get_image_file_list, check_and_read_gif
+from ppocr.utils.utility import get_image_file_list, check_and_read
 from ppstructure.utility import parse_args
 
 from paddleocr import PaddleOCR
@@ -49,7 +49,7 @@ class SerPredictor(object):
 
         pre_process_list = [{
             'VQATokenLabelEncode': {
-                'algorithm': args.vqa_algorithm,
+                'algorithm': args.kie_algorithm,
                 'class_path': args.ser_dict_path,
                 'contains_re': False,
                 'ocr_engine': self.ocr_engine,
@@ -138,7 +138,7 @@ def main(args):
             os.path.join(args.output, 'infer.txt'), mode='w',
             encoding='utf-8') as f_w:
         for image_file in image_file_list:
-            img, flag = check_and_read_gif(image_file)
+            img, flag, _ = check_and_read(image_file)
             if not flag:
                 img = cv2.imread(image_file)
                 img = img[:, :, ::-1]

ppstructure/vqa/requirements.txt → ppstructure/kie/requirements.txt

 sentencepiece
 yacs
 seqeval
-paddlenlp>=2.2.1
+git+https://github.com/PaddlePaddle/PaddleNLP
 pypandoc
 attrdict
-python_docx
\ No newline at end of file
+python_docx

ppstructure/vqa/tools/eval_with_label_end2end.py → ppstructure/kie/tools/eval_with_label_end2end.py

@@ -20,7 +20,7 @@ from shapely.geometry import Polygon
 import numpy as np
 from collections import defaultdict
 import operator
-import Levenshtein
+from rapidfuzz.distance import Levenshtein
 import argparse
 import json
 import copy

ppstructure/layout/README.md

@@ -63,7 +63,7 @@ python3 -m pip install "paddlepaddle>=2.2" -i https://mirror.baidu.com/pypi/simp
 git clone https://github.com/PaddlePaddle/PaddleDetection.git
 ```
 
-- **（2）安装其他依赖 **
+- **（2）安装其他依赖**
 
 ```bash
 cd PaddleDetection
@@ -138,7 +138,7 @@ json文件包含所有图像的标注，数据以字典嵌套的方式存放，
 
   ```
   {
-  
+
       'segmentation':             # 物体的分割标注
       'area': 60518.099043117836, # 物体的区域面积
       'iscrowd': 0,               # iscrowd
@@ -166,15 +166,17 @@ json文件包含所有图像的标注，数据以字典嵌套的方式存放，
 
 提供了训练脚本、评估脚本和预测脚本，本节将以PubLayNet预训练模型为例进行讲解。
 
-如果不希望训练，直接体验后面的模型评估、预测、动转静、推理的流程，可以下载提供的预训练模型，并跳过本部分。
+如果不希望训练，直接体验后面的模型评估、预测、动转静、推理的流程，可以下载提供的预训练模型(PubLayNet数据集)，并跳过本部分。
 
 ```
 mkdir pretrained_model
 cd pretrained_model
-# 下载并解压PubLayNet预训练模型
+# 下载PubLayNet预训练模型
 wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout.pdparams
 ```
 
+下载更多[版面分析模型](../docs/models_list.md)（中文CDLA数据集预训练模型、表格预训练模型）
+
 ### 4.1. 启动训练
 
 开始训练:
@@ -184,7 +186,7 @@ wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_
 如果你希望训练自己的数据集，需要修改配置文件中的数据配置、类别数。
 
 
-以`configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 为例，修改的内容如下所示。
+以`configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml` 为例，修改的内容如下所示。
 
 ```yaml
 metric: COCO
@@ -223,16 +225,20 @@ TestDataset:
 # 训练日志会自动保存到 log 目录中
 
 # 单卡训练
+export CUDA_VISIBLE_DEVICES=0
 python3 tools/train.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
 	--eval
 
 # 多卡训练，通过--gpus参数指定卡号
+export CUDA_VISIBLE_DEVICES=0,1,2,3
 python3 -m paddle.distributed.launch --gpus '0,1,2,3'  tools/train.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
 	--eval
 ```
 
+**注意：**如果训练时显存out memory，将TrainReader中batch_size调小，同时LearningRate中base_lr等比例减小。发布的config均由8卡训练得到，如果改变GPU卡数为1，那么base_lr需要减小8倍。
+
 正常启动训练后，会看到以下log输出：
 
 ```
@@ -254,9 +260,11 @@ PaddleDetection支持了基于FGD([Focal and Global Knowledge Distillation for D
 更换数据集，修改【TODO】配置中的数据配置、类别数，具体可以参考4.1。启动训练：
 
 ```bash
-python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
+# 单卡训练
+export CUDA_VISIBLE_DEVICES=0
+python3 tools/train.py \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
+	--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
 	--eval
 ```
 
@@ -267,13 +275,13 @@ python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
 
 ### 5.1. 指标评估
 
-训练中模型参数默认保存在`output/picodet_lcnet_x1_0_layout`目录下。在评估指标时，需要设置`weights`指向保存的参数文件。评估数据集可以通过 `configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml`  修改`EvalDataset`中的 `image_dir`、`anno_path`和`dataset_dir` 设置。
+训练中模型参数默认保存在`output/picodet_lcnet_x1_0_layout`目录下。在评估指标时，需要设置`weights`指向保存的参数文件。评估数据集可以通过 `configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml`  修改`EvalDataset`中的 `image_dir`、`anno_path`和`dataset_dir` 设置。
 
 ```bash
 # GPU 评估， weights 为待测权重
 python3 tools/eval.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	-o weigths=./output/picodet_lcnet_x1_0_layout/best_model
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
+	-o weights=./output/picodet_lcnet_x1_0_layout/best_model
 ```
 
 会输出以下信息，打印出mAP、AP0.5等信息。
@@ -299,8 +307,8 @@ python3 tools/eval.py \
 
 ```
 python3 tools/eval.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
+	--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
 	-o weights=output/picodet_lcnet_x2_5_layout/best_model
 ```
 
@@ -311,18 +319,17 @@ python3 tools/eval.py \
 ### 5.2. 测试版面分析结果
 
 
-预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 完成了模型的训练过程。
-
-使用 PaddleDetection 训练好的模型，您可以使用如下命令进行中文模型预测。
+预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml` 完成了模型的训练过程。
 
+使用 PaddleDetection 训练好的模型，您可以使用如下命令进行模型预测。
 
 ```bash
 python3 tools/infer.py \
-    -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+    -c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
     -o weights='output/picodet_lcnet_x1_0_layout/best_model.pdparams' \
     --infer_img='docs/images/layout.jpg' \
     --output_dir=output_dir/ \
-    --draw_threshold=0.4
+    --draw_threshold=0.5
 ```
 
 - `--infer_img`: 推理单张图片，也可以通过`--infer_dir`推理文件中的所有图片。
@@ -335,16 +342,15 @@ python3 tools/infer.py \
 
 ```
 python3 tools/infer.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
+	--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
 	-o weights='output/picodet_lcnet_x2_5_layout/best_model.pdparams' \
 	--infer_img='docs/images/layout.jpg' \
 	--output_dir=output_dir/ \
-	--draw_threshold=0.4
+	--draw_threshold=0.5
 ```
 
 
-
 ## 6. 模型导出与预测
 
 
@@ -356,7 +362,7 @@ inference 模型（`paddle.jit.save`保存的模型） 一般是模型训练，
 
 ```bash
 python3 tools/export_model.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
 	-o weights=output/picodet_lcnet_x1_0_layout/best_model \
 	--output_dir=output_inference/
 ```
@@ -377,8 +383,8 @@ FGD蒸馏模型转inference模型步骤如下：
 
 ```bash
 python3 tools/export_model.py \
-	-c configs/picodet/legacy_model/application/publayernet_lcnet_x1_5/picodet_student.yml \
-	--slim_config configs/picodet/legacy_model/application/publayernet_lcnet_x1_5/picodet_teacher.yml \
+	-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
+	--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
 	-o weights=./output/picodet_lcnet_x2_5_layout/best_model \
 	--output_dir=output_inference/
 ```
@@ -404,7 +410,7 @@ python3 deploy/python/infer.py \
 ------------------------------------------
 -----------  Model Configuration -----------
 Model Arch: PicoDet
-Transform Order: 
+Transform Order:
 --transform op: Resize
 --transform op: NormalizeImage
 --transform op: Permute
@@ -466,4 +472,3 @@ preprocess_time(ms): 2172.50, inference_time(ms): 11.90, postprocess_time(ms): 1
   year={2022}
 }
 ```
-

ppstructure/layout/__init__.py

+# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

ppstructure/layout/layout_in_ocr.md

-- [1. 简介](#1-简介)
-
-- [2. 安装](#2-安装)
-
-  - [2.1 安装PaddlePaddle](#21-安装paddlepaddle)
-  - [2.2 安装PaddleDetection](#22-安装paddledetection)
-
-- [3. 数据准备](#3-数据准备)
-
-  - [3.1 英文数据集](#31-英文数据集)
-  - [3.2 更多数据集](#32-更多数据集)
-
-- [4. 开始训练](#4-开始训练)
-
-  - [4.1 启动训练](#41-启动训练)
-  - [4.2 FGD蒸馏训练](#42-FGD蒸馏训练)
-
-- [5. 模型评估与预测](#5-模型评估与预测)
-
-  - [5.1 指标评估](#51-指标评估)
-  - [5.2 测试版面分析结果](#52-测试版面分析结果)
-
-- [6 模型导出与预测](#6-模型导出与预测)
-
-  - [6.1 模型导出](#61-模型导出)
-
-  - [6.2 模型推理](#62-模型推理)
-
-# 版面分析
-
-## 1. 简介
-
-版面分析指的是对图片形式的文档进行区域划分，定位其中的关键区域，如文字、标题、表格、图片等。版面分析算法基于[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)的轻量模型PP-PicoDet进行开发。
-
-<div align="center">
-    <img src="../docs/layout/layout.png" width="800">
-</div>
-
-
-
-## 2. 安装依赖
-
-### 2.1. 安装PaddlePaddle
-
-- **（1) 安装PaddlePaddle**
-
-```bash
-python3 -m pip install --upgrade pip
-
-# GPU安装
-python3 -m pip install "paddlepaddle-gpu>=2.2" -i https://mirror.baidu.com/pypi/simple
-
-# CPU安装
-python3 -m pip install "paddlepaddle>=2.2" -i https://mirror.baidu.com/pypi/simple
-```
-更多需求，请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
-
-### 2.2. 安装PaddleDetection
-
-- **（1）下载PaddleDetection源码**
-
-```bash
-git clone https://github.com/PaddlePaddle/PaddleDetection.git
-```
-
-- **（2）安装其他依赖 **
-
-```bash
-cd PaddleDetection
-python3 -m pip install -r requirements.txt
-```
-
-## 3. 数据准备
-
-如果希望直接体验预测过程，可以跳过数据准备，下载我们提供的预训练模型。
-
-### 3.1. 英文数据集
-
-下载文档分析数据集[PubLayNet](https://developer.ibm.com/exchanges/data/all/publaynet/)（数据集96G），包含5个类：`{0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}`
-
-```
-# 下载数据
-wget https://dax-cdn.cdn.appdomain.cloud/dax-publaynet/1.0.0/publaynet.tar.gz
-# 解压数据
-tar -xvf publaynet.tar.gz
-```
-
-解压之后的**目录结构：**
-
-```
-|-publaynet
-  |- test
-     |- PMC1277013_00004.jpg
-     |- PMC1291385_00002.jpg
-     | ...
-  |- train.json
-  |- train
-     |- PMC1291385_00002.jpg
-     |- PMC1277013_00004.jpg
-     | ...
-  |- val.json
-  |- val
-     |- PMC538274_00004.jpg
-     |- PMC539300_00004.jpg
-     | ...
-```
-
-**数据分布：**
-
-| File or Folder | Description    | num     |
-| :------------- | :------------- | ------- |
-| `train/`       | 训练集图片     | 335,703 |
-| `val/`         | 验证集图片     | 11,245  |
-| `test/`        | 测试集图片     | 11,405  |
-| `train.json`   | 训练集标注文件 | -       |
-| `val.json`     | 验证集标注文件 | -       |
-
-**标注格式：**
-
-json文件包含所有图像的标注，数据以字典嵌套的方式存放，包含以下key：
-
-- info，表示标注文件info。
-
-- licenses，表示标注文件licenses。
-
-- images，表示标注文件中图像信息列表，每个元素是一张图像的信息。如下为其中一张图像的信息：
-
-  ```
-  {
-      'file_name': 'PMC4055390_00006.jpg',    # file_name
-      'height': 601,                      # image height
-      'width': 792,                       # image width
-      'id': 341427                        # image id
-  }
-  ```
-
-- annotations，表示标注文件中目标物体的标注信息列表，每个元素是一个目标物体的标注信息。如下为其中一个目标物体的标注信息：
-
-  ```
-  {
-  
-      'segmentation':             # 物体的分割标注
-      'area': 60518.099043117836, # 物体的区域面积
-      'iscrowd': 0,               # iscrowd
-      'image_id': 341427,         # image id
-      'bbox': [50.58, 490.86, 240.15, 252.16], # bbox [x1,y1,w,h]
-      'category_id': 1,           # category_id
-      'id': 3322348               # image id
-  }
-  ```
-
-### 3.2. 更多数据集
-
-我们提供了CDLA(中文版面分析)、TableBank(表格版面分析)等数据集的下连接，处理为上述标注文件json格式，即可以按相同方式进行训练。
-
-| dataset                                                      | 简介                                                         |
-| ------------------------------------------------------------ | ------------------------------------------------------------ |
-| [cTDaR2019_cTDaR](https://cndplab-founder.github.io/cTDaR2019/) | 用于表格检测(TRACKA)和表格识别(TRACKB)。图片类型包含历史数据集(以cTDaR_t0开头，如cTDaR_t00872.jpg)和现代数据集(以cTDaR_t1开头，cTDaR_t10482.jpg)。 |
-| [IIIT-AR-13K](http://cvit.iiit.ac.in/usodi/iiitar13k.php)    | 手动注释公开的年度报告中的图形或页面而构建的数据集，包含5类：table, figure, natural image, logo, and signature |
-| [CDLA](https://github.com/buptlihang/CDLA)                   | 中文文档版面分析数据集，面向中文文献类（论文）场景，包含10类：Table、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation |
-| [TableBank](https://github.com/doc-analysis/TableBank)       | 用于表格检测和识别大型数据集，包含Word和Latex2种文档格式     |
-| [DocBank](https://github.com/doc-analysis/DocBank)           | 使用弱监督方法构建的大规模数据集(500K文档页面)，用于文档布局分析，包含12类：Author、Caption、Date、Equation、Figure、Footer、List、Paragraph、Reference、Section、Table、Title |
-
-
-## 4. 开始训练
-
-提供了训练脚本、评估脚本和预测脚本，本节将以PubLayNet预训练模型为例进行讲解。
-
-如果不希望训练，直接体验后面的模型评估、预测、动转静、推理的流程，可以下载提供的预训练模型，并跳过本部分。
-
-```
-mkdir pretrained_model
-cd pretrained_model
-# 下载并解压PubLayNet预训练模型
-wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout.pdparams
-```
-
-### 4.1. 启动训练
-
-开始训练:
-
-* 修改配置文件
-
-如果你希望训练自己的数据集，需要修改配置文件中的数据配置、类别数。
-
-
-以`configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 为例，修改的内容如下所示。
-
-```yaml
-metric: COCO
-# 类别数
-num_classes: 5
-
-TrainDataset:
-  !COCODataSet
-    # 修改为你自己的训练数据目录
-    image_dir: train
-    # 修改为你自己的训练数据标签文件
-    anno_path: train.json
-    # 修改为你自己的训练数据根目录
-    dataset_dir: /root/publaynet/
-    data_fields: ['image', 'gt_bbox', 'gt_class', 'is_crowd']
-
-EvalDataset:
-  !COCODataSet
-    # 修改为你自己的验证数据目录
-    image_dir: val
-    # 修改为你自己的验证数据标签文件
-    anno_path: val.json
-    # 修改为你自己的验证数据根目录
-    dataset_dir: /root/publaynet/
-
-TestDataset:
-  !ImageFolder
-    # 修改为你自己的测试数据标签文件
-    anno_path: /root/publaynet/val.json
-```
-
-* 开始训练，在训练时，会默认下载PP-PicoDet预训练模型，这里无需预先下载。
-
-```bash
-# GPU训练 支持单卡，多卡训练
-# 训练日志会自动保存到 log 目录中
-
-# 单卡训练
-python3 tools/train.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--eval
-
-# 多卡训练，通过--gpus参数指定卡号
-python3 -m paddle.distributed.launch --gpus '0,1,2,3'  tools/train.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--eval
-```
-
-正常启动训练后，会看到以下log输出：
-
-```
-[08/15 04:02:30] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/LCNet_x1_0_pretrained.pdparams
-[08/15 04:02:46] ppdet.engine INFO: Epoch: [0] [   0/1929] learning_rate: 0.040000 loss_vfl: 1.216707 loss_bbox: 1.142163 loss_dfl: 0.544196 loss: 2.903065 eta: 17 days, 13:50:26 batch_cost: 15.7452 data_cost: 2.9112 ips: 1.5243 images/s
-[08/15 04:03:19] ppdet.engine INFO: Epoch: [0] [  20/1929] learning_rate: 0.064000 loss_vfl: 1.180627 loss_bbox: 0.939552 loss_dfl: 0.442436 loss: 2.628206 eta: 2 days, 12:18:53 batch_cost: 1.5770 data_cost: 0.0008 ips: 15.2184 images/s
-[08/15 04:03:47] ppdet.engine INFO: Epoch: [0] [  40/1929] learning_rate: 0.088000 loss_vfl: 0.543321 loss_bbox: 1.071401 loss_dfl: 0.457817 loss: 2.057003 eta: 2 days, 0:07:03 batch_cost: 1.3190 data_cost: 0.0007 ips: 18.1954 images/s
-[08/15 04:04:12] ppdet.engine INFO: Epoch: [0] [  60/1929] learning_rate: 0.112000 loss_vfl: 0.630989 loss_bbox: 0.859183 loss_dfl: 0.384702 loss: 1.883143 eta: 1 day, 19:01:29 batch_cost: 1.2177 data_cost: 0.0006 ips: 19.7087 images/s
-```
-
-- `--eval`表示训练的同时，进行评估， 评估过程中默认将最佳模型，保存为 `output/picodet_lcnet_x1_0_layout/best_accuracy` 。
-
-**注意，预测/评估时的配置文件请务必与训练一致。**
-
-### 4.2. FGD蒸馏训练
-
-PaddleDetection支持了基于FGD([Focal and Global Knowledge Distillation for Detectors](https://arxiv.org/abs/2111.11837v1))蒸馏的目标检测模型训练过程，FGD蒸馏分为两个部分`Focal`和`Global`。`Focal`蒸馏分离图像的前景和背景，让学生模型分别关注教师模型的前景和背景部分特征的关键像素；`Global`蒸馏部分重建不同像素之间的关系并将其从教师转移到学生，以补偿`Focal`蒸馏中丢失的全局信息。
-
-更换数据集，修改【TODO】配置中的数据配置、类别数，具体可以参考4.1。启动训练：
-
-```bash
-python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
-	--eval
-```
-
-- `-c`: 指定模型配置文件。
-- `--slim_config`: 指定压缩策略配置文件。
-
-## 5. 模型评估与预测
-
-### 5.1. 指标评估
-
-训练中模型参数默认保存在`output/picodet_lcnet_x1_0_layout`目录下。在评估指标时，需要设置`weights`指向保存的参数文件。评估数据集可以通过 `configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml`  修改`EvalDataset`中的 `image_dir`、`anno_path`和`dataset_dir` 设置。
-
-```bash
-# GPU 评估， weights 为待测权重
-python3 tools/eval.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	-o weigths=./output/picodet_lcnet_x1_0_layout/best_model
-```
-
-会输出以下信息，打印出mAP、AP0.5等信息。
-
-```py
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.935
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.979
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.956
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.404
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.782
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.969
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.539
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.938
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.949
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.495
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.818
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.978
-[08/15 07:07:09] ppdet.engine INFO: Total sample number: 11245, averge FPS: 24.405059207157436
-[08/15 07:07:09] ppdet.engine INFO: Best test bbox ap is 0.935.
-```
-
-使用FGD蒸馏模型进行评估：
-
-```
-python3 tools/eval.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
-	-o weights=output/picodet_lcnet_x2_5_layout/best_model
-```
-
-- `-c`: 指定模型配置文件。
-- `--slim_config`: 指定蒸馏策略配置文件。
-- `-o weights`: 指定蒸馏算法训好的模型路径。
-
-### 5.2. 测试版面分析结果
-
-
-预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 完成了模型的训练过程。
-
-使用 PaddleDetection 训练好的模型，您可以使用如下命令进行中文模型预测。
-
-
-```bash
-python3 tools/infer.py \
-    -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-    -o weights='output/picodet_lcnet_x1_0_layout/best_model.pdparams' \
-    --infer_img='docs/images/layout.jpg' \
-    --output_dir=output_dir/ \
-    --draw_threshold=0.4
-```
-
-- `--infer_img`: 推理单张图片，也可以通过`--infer_dir`推理文件中的所有图片。
-- `--output_dir`: 指定可视化结果保存路径。
-- `--draw_threshold`:指定绘制结果框的NMS阈值。
-
-预测图片如下所示，图片会存储在`output_dir`路径中。
-
-使用FGD蒸馏模型进行测试：
-
-```
-python3 tools/infer.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	--slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
-	-o weights='output/picodet_lcnet_x2_5_layout/best_model.pdparams' \
-	--infer_img='docs/images/layout.jpg' \
-	--output_dir=output_dir/ \
-	--draw_threshold=0.4
-```
-
-
-
-## 6. 模型导出与预测
-
-
-### 6.1 模型导出
-
-inference 模型（`paddle.jit.save`保存的模型） 一般是模型训练，把模型结构和模型参数保存在文件中的固化模型，多用于预测部署场景。 训练过程中保存的模型是checkpoints模型，保存的只有模型的参数，多用于恢复训练等。 与checkpoints模型相比，inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
-
-版面分析模型转inference模型步骤如下：
-
-```bash
-python3 tools/export_model.py \
-	-c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
-	-o weights=output/picodet_lcnet_x1_0_layout/best_model \
-	--output_dir=output_inference/
-```
-
-* 如无需导出后处理，请指定：`-o export.benchmark=True`（如果-o已出现过，此处删掉-o）
-* 如无需导出NMS，请指定：`-o export.nms=False`
-
-转换成功后，在目录下有三个文件：
-
-```
-output_inference/picodet_lcnet_x1_0_layout/
-    ├── model.pdiparams         # inference模型的参数文件
-    ├── model.pdiparams.info    # inference模型的参数信息，可忽略
-    └── model.pdmodel           # inference模型的模型结构文件
-```
-
-FGD蒸馏模型转inference模型步骤如下：
-
-```bash
-python3 tools/export_model.py \
-	-c configs/picodet/legacy_model/application/publayernet_lcnet_x1_5/picodet_student.yml \
-	--slim_config configs/picodet/legacy_model/application/publayernet_lcnet_x1_5/picodet_teacher.yml \
-	-o weights=./output/picodet_lcnet_x2_5_layout/best_model \
-	--output_dir=output_inference/
-```
-
-
-
-### 6.2 模型推理
-
-版面恢复任务进行推理，可以执行如下命令：
-
-```bash
-python3 deploy/python/infer.py \
-	--model_dir=output_inference/picodet_lcnet_x1_0_layout/ \
-	--image_file=docs/images/layout.jpg \
-	--device=CPU
-```
-
-- --device：指定GPU、CPU设备
-
-模型推理完成，会看到以下log输出
-
-```
-------------------------------------------
------------  Model Configuration -----------
-Model Arch: PicoDet
-Transform Order: 
---transform op: Resize
---transform op: NormalizeImage
---transform op: Permute
---transform op: PadStride
---------------------------------------------
-class_id:0, confidence:0.9921, left_top:[20.18,35.66],right_bottom:[341.58,600.99]
-class_id:0, confidence:0.9914, left_top:[19.77,611.42],right_bottom:[341.48,901.82]
-class_id:0, confidence:0.9904, left_top:[369.36,375.10],right_bottom:[691.29,600.59]
-class_id:0, confidence:0.9835, left_top:[369.60,608.60],right_bottom:[691.38,736.72]
-class_id:0, confidence:0.9830, left_top:[369.58,805.38],right_bottom:[690.97,901.80]
-class_id:0, confidence:0.9716, left_top:[383.68,271.44],right_bottom:[688.93,335.39]
-class_id:0, confidence:0.9452, left_top:[370.82,34.48],right_bottom:[688.10,63.54]
-class_id:1, confidence:0.8712, left_top:[370.84,771.03],right_bottom:[519.30,789.13]
-class_id:3, confidence:0.9856, left_top:[371.28,67.85],right_bottom:[685.73,267.72]
-save result to: output/layout.jpg
-Test iter 0
------------------- Inference Time Info ----------------------
-total_time(ms): 2196.0, img_num: 1
-average latency time(ms): 2196.00, QPS: 0.455373
-preprocess_time(ms): 2172.50, inference_time(ms): 11.90, postprocess_time(ms): 11.60
-```
-
-- Model：模型结构
-- Transform Order：预处理操作
-- class_id、confidence、left_top、right_bottom：分别表示类别id、置信度、左上角坐标、右下角坐标
-- save result to：可视化版面分析结果保存路径，默认保存到`./output`文件夹
-- Inference Time Info：推理时间，其中preprocess_time表示预处理耗时，inference_time表示模型预测耗时，postprocess_time表示后处理耗时
-
-可视化版面结果如下图所示
-
-<div align="center">
-    <img src="../docs/layout/layout_res.jpg" width="800">
-</div>
-
-
-
-## Citations
-
-```
-@inproceedings{zhong2019publaynet,
-  title={PubLayNet: largest dataset ever for document layout analysis},
-  author={Zhong, Xu and Tang, Jianbin and Yepes, Antonio Jimeno},
-  booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)},
-  year={2019},
-  volume={},
-  number={},
-  pages={1015-1022},
-  doi={10.1109/ICDAR.2019.00166},
-  ISSN={1520-5363},
-  month={Sep.},
-  organization={IEEE}
-}
-
-@inproceedings{yang2022focal,
-  title={Focal and global knowledge distillation for detectors},
-  author={Yang, Zhendong and Li, Zhe and Jiang, Xiaohu and Gong, Yuan and Yuan, Zehuan and Zhao, Danpei and Yuan, Chun},
-  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
-  pages={4643--4652},
-  year={2022}
-}
-```
-

ppstructure/layout/predict_layout.py

@@ -28,12 +28,13 @@ import tools.infer.utility as utility
 from ppocr.data import create_operators, transform
 from ppocr.postprocess import build_post_process
 from ppocr.utils.logging import get_logger
-from ppocr.utils.utility import get_image_file_list, check_and_read_gif
+from ppocr.utils.utility import get_image_file_list, check_and_read
 from ppstructure.utility import parse_args
 from picodet_postprocess import PicoDetPostProcess
 
 logger = get_logger()
 
+
 class LayoutPredictor(object):
     def __init__(self, args):
         pre_process_list = [{
@@ -109,7 +110,7 @@ def main(args):
 
     repeats = 50
     for image_file in image_file_list:
-        img, flag = check_and_read_gif(image_file)
+        img, flag, _ = check_and_read(image_file)
         if not flag:
             img = cv2.imread(image_file)
         if img is None:

ppstructure/predict_system.py

@@ -28,13 +28,12 @@ import time
 import logging
 from copy import deepcopy
 
-from ppocr.utils.utility import get_image_file_list, check_and_read_gif
+from ppocr.utils.utility import get_image_file_list, check_and_read
 from ppocr.utils.logging import get_logger
 from tools.infer.predict_system import TextSystem
 from ppstructure.layout.predict_layout import LayoutPredictor
 from ppstructure.table.predict_table import TableSystem, to_excel
 from ppstructure.utility import parse_args, draw_structure_result
-from ppstructure.recovery.recovery_to_doc import convert_info_docx
 
 logger = get_logger()
 
@@ -75,10 +74,10 @@ class StructureSystem(object):
                 else:
                     self.table_system = TableSystem(args)
 
-        elif self.mode == 'vqa':
+        elif self.mode == 'kie':
             raise NotImplementedError
 
-    def __call__(self, img, return_ocr_result_in_table=False):
+    def __call__(self, img, img_idx=0, return_ocr_result_in_table=False):
         time_dict = {
             'image_orientation': 0,
             'layout': 0,
@@ -86,7 +85,7 @@ class StructureSystem(object):
             'table_match': 0,
             'det': 0,
             'rec': 0,
-            'vqa': 0,
+            'kie': 0,
             'all': 0
         }
         start = time.time()
@@ -143,8 +142,8 @@ class StructureSystem(object):
                         time_dict['det'] += ocr_time_dict['det']
                         time_dict['rec'] += ocr_time_dict['rec']
 
-                        # remove style char, 
-                        # when using the recognition model trained on the PubtabNet dataset, 
+                        # remove style char,
+                        # when using the recognition model trained on the PubtabNet dataset,
                         # it will recognize the text format in the table, such as <b>
                         style_token = [
                             '<strike>', '<strike>', '<sup>', '</sub>', '<b>',
@@ -169,36 +168,40 @@ class StructureSystem(object):
                     'type': region['label'].lower(),
                     'bbox': [x1, y1, x2, y2],
                     'img': roi_img,
-                    'res': res
+                    'res': res,
+                    'img_idx': img_idx
                 })
             end = time.time()
             time_dict['all'] = end - start
             return res_list, time_dict
-        elif self.mode == 'vqa':
+        elif self.mode == 'kie':
             raise NotImplementedError
         return None, None
 
 
-def save_structure_res(res, save_folder, img_name):
+def save_structure_res(res, save_folder, img_name, img_idx=0):
     excel_save_folder = os.path.join(save_folder, img_name)
     os.makedirs(excel_save_folder, exist_ok=True)
     res_cp = deepcopy(res)
     # save res
     with open(
-            os.path.join(excel_save_folder, 'res.txt'), 'w',
+            os.path.join(excel_save_folder, 'res_{}.txt'.format(img_idx)),
+            'w',
             encoding='utf8') as f:
         for region in res_cp:
             roi_img = region.pop('img')
             f.write('{}\n'.format(json.dumps(region)))
 
-            if region['type'] == 'table' and len(region[
+            if region['type'].lower() == 'table' and len(region[
                     'res']) > 0 and 'html' in region['res']:
-                excel_path = os.path.join(excel_save_folder,
-                                          '{}.xlsx'.format(region['bbox']))
+                excel_path = os.path.join(
+                    excel_save_folder,
+                    '{}_{}.xlsx'.format(region['bbox'], img_idx))
                 to_excel(region['res']['html'], excel_path)
-            elif region['type'] == 'figure':
-                img_path = os.path.join(excel_save_folder,
-                                        '{}.jpg'.format(region['bbox']))
+            elif region['type'].lower() == 'figure':
+                img_path = os.path.join(
+                    excel_save_folder,
+                    '{}_{}.jpg'.format(region['bbox'], img_idx))
                 cv2.imwrite(img_path, roi_img)
 
 
@@ -214,28 +217,75 @@ def main(args):
 
     for i, image_file in enumerate(image_file_list):
         logger.info("[{}/{}] {}".format(i, img_num, image_file))
-        img, flag = check_and_read_gif(image_file)
+        img, flag_gif, flag_pdf = check_and_read(image_file)
         img_name = os.path.basename(image_file).split('.')[0]
 
-        if not flag:
+        if not flag_gif and not flag_pdf:
             img = cv2.imread(image_file)
-        if img is None:
-            logger.error("error in loading image:{}".format(image_file))
-            continue
-        res, time_dict = structure_sys(img)
 
-        if structure_sys.mode == 'structure':
-            save_structure_res(res, save_folder, img_name)
-            draw_img = draw_structure_result(img, res, args.vis_font_path)
-            img_save_path = os.path.join(save_folder, img_name, 'show.jpg')
-        elif structure_sys.mode == 'vqa':
-            raise NotImplementedError
-            # draw_img = draw_ser_results(img, res, args.vis_font_path)
-            # img_save_path = os.path.join(save_folder, img_name + '.jpg')
-        cv2.imwrite(img_save_path, draw_img)
-        logger.info('result save to {}'.format(img_save_path))
-        if args.recovery:
-            convert_info_docx(img, res, save_folder, img_name)
+        if not flag_pdf:
+            if img is None:
+                logger.error("error in loading image:{}".format(image_file))
+                continue
+            res, time_dict = structure_sys(img)
+
+            if structure_sys.mode == 'structure':
+                save_structure_res(res, save_folder, img_name)
+                draw_img = draw_structure_result(img, res, args.vis_font_path)
+                img_save_path = os.path.join(save_folder, img_name, 'show.jpg')
+            elif structure_sys.mode == 'kie':
+                raise NotImplementedError
+                # draw_img = draw_ser_results(img, res, args.vis_font_path)
+                # img_save_path = os.path.join(save_folder, img_name + '.jpg')
+            cv2.imwrite(img_save_path, draw_img)
+            logger.info('result save to {}'.format(img_save_path))
+            if args.recovery:
+                try:
+                    from ppstructure.recovery.recovery_to_doc import sorted_layout_boxes, convert_info_docx
+                    h, w, _ = img.shape
+                    res = sorted_layout_boxes(res, w)
+                    convert_info_docx(img, res, save_folder, img_name,
+                                      args.save_pdf)
+                except Exception as ex:
+                    logger.error(
+                        "error in layout recovery image:{}, err msg: {}".format(
+                            image_file, ex))
+                    continue
+        else:
+            pdf_imgs = img
+            all_res = []
+            for index, img in enumerate(pdf_imgs):
+
+                res, time_dict = structure_sys(img, index)
+                if structure_sys.mode == 'structure' and res != []:
+                    save_structure_res(res, save_folder, img_name, index)
+                    draw_img = draw_structure_result(img, res,
+                                                     args.vis_font_path)
+                    img_save_path = os.path.join(save_folder, img_name,
+                                                 'show_{}.jpg'.format(index))
+                elif structure_sys.mode == 'kie':
+                    raise NotImplementedError
+                    # draw_img = draw_ser_results(img, res, args.vis_font_path)
+                    # img_save_path = os.path.join(save_folder, img_name + '.jpg')
+                if res != []:
+                    cv2.imwrite(img_save_path, draw_img)
+                    logger.info('result save to {}'.format(img_save_path))
+                if args.recovery and res != []:
+                    from ppstructure.recovery.recovery_to_doc import sorted_layout_boxes, convert_info_docx
+                    h, w, _ = img.shape
+                    res = sorted_layout_boxes(res, w)
+                    all_res += res
+
+            if args.recovery and all_res != []:
+                try:
+                    convert_info_docx(img, all_res, save_folder, img_name,
+                                      args.save_pdf)
+                except Exception as ex:
+                    logger.error(
+                        "error in layout recovery image:{}, err msg: {}".format(
+                            image_file, ex))
+                    continue
+
         logger.info("Predict time : {:.3f}s".format(time_dict['all']))
 
 

ppstructure/recovery/README.md

@@ -6,10 +6,12 @@ English | [简体中文](README_ch.md)
     - [2.1 Installation dependencies](#2.1)
     - [2.2 Install PaddleOCR](#2.2)
   - [3. Quick Start](#3)
+    - [3.1 Download models](#3.1)
+    - [3.2 Layout recovery](#3.2)
 
 <a name="1"></a>
 
-## 1.  Introduction
+## 1. Introduction
 
 Layout recovery means that after OCR recognition, the content is still arranged like the original document pictures, and the paragraphs are output to word document in the same order.
 
@@ -17,8 +19,9 @@ Layout recovery combines [layout analysis](../layout/README.md)、[table recogni
 The following figure shows the result：
 
 <div align="center">
-<img src="../docs/table/recovery.jpg"  width = "700" />
+<img src="../docs/recovery/recovery.jpg"  width = "700" />
 </div>
+
 <a name="2"></a>
 
 ## 2. Install
@@ -33,14 +36,14 @@ The following figure shows the result：
 python3 -m pip install --upgrade pip
 
 # GPU installation
-python3 -m pip install "paddlepaddle-gpu>=2.2" -i https://mirror.baidu.com/pypi/simple
+python3 -m pip install "paddlepaddle-gpu" -i https://mirror.baidu.com/pypi/simple
 
 # CPU installation
-python3 -m pip install "paddlepaddle>=2.2" -i https://mirror.baidu.com/pypi/simple
+python3 -m pip install "paddlepaddle" -i https://mirror.baidu.com/pypi/simple
 
 ````
 
-For more requirements, please refer to the instructions in [Installation Documentation](https://www.paddlepaddle.org.cn/install/quick).
+For more requirements, please refer to the instructions in [Installation Documentation](https://www.paddlepaddle.org.cn/en/install/quick?docurl=/documentation/docs/en/install/pip/macos-pip_en.html).
 
 <a name="2.2"></a>
 
@@ -67,20 +70,61 @@ python3 -m pip install -r ppstructure/recovery/requirements.txt
 
 ## 3. Quick Start
 
-```python
+<a name="3.1"></a>
+### 3.1 Download models
+
+If input is English document, download English models:
+
+```bash
 cd PaddleOCR/ppstructure
 
 # download model
 mkdir inference && cd inference
 # Download the detection model of the ultra-lightweight English PP-OCRv3 model and unzip it
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar && tar xf ch_PP-OCRv3_det_infer.tar
+https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar && tar xf en_PP-OCRv3_det_infer.tar
 # Download the recognition model of the ultra-lightweight English PP-OCRv3 model and unzip it
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar && tar xf  ch_PP-OCRv3_rec_infer.tar
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar && tar xf en_PP-OCRv3_rec_infer.tar
 # Download the ultra-lightweight English table inch model and unzip it
-wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
+# Download the layout model of publaynet dataset and unzip it
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar && tar xf picodet_lcnet_x1_0_fgd_layout_infer.tar
 cd ..
-# run
-python3 predict_system.py --det_model_dir=inference/en_PP-OCRv3_det_infer --rec_model_dir=inference/en_PP-OCRv3_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --rec_char_dict_path=../ppocr/utils/en_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --output ./output/table --rec_image_shape=3,48,320 --vis_font_path=../doc/fonts/simfang.ttf --recovery=True --image_dir=./docs/table/1.png
+```
+If input is Chinese document，download Chinese models:
+[Chinese and English ultra-lightweight PP-OCRv3 model](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/README.md#pp-ocr-series-model-listupdate-on-september-8th)、[表格识别模型](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppstructure/docs/models_list.md#22-表格识别模型)、[版面分析模型](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppstructure/docs/models_list.md#1-版面分析模型)
+
+<a name="3.2"></a>
+### 3.2 Layout recovery
+
+
+```bash
+python3 predict_system.py \
+    --image_dir=./docs/table/1.png \
+    --det_model_dir=inference/en_PP-OCRv3_det_infer \
+    --rec_model_dir=inference/en_PP-OCRv3_rec_infer \
+    --rec_char_dict_path=../ppocr/utils/en_dict.txt \
+    --table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
+    --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+    --layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_infer \
+    --layout_dict_path=../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt \
+    --vis_font_path=../doc/fonts/simfang.ttf \
+    --recovery=True \
+    --save_pdf=False \
+    --output=../output/
 ```
 
-After running, the docx  of each picture will be saved in the directory specified by the output field
\ No newline at end of file
+After running, the docx of each picture will be saved in the directory specified by the output field
+
+Field：
+
+- image_dir：test file测试文件， can be picture, picture directory, pdf file, pdf file directory
+- det_model_dir：OCR detection model path
+- rec_model_dir：OCR recognition model path
+- rec_char_dict_path：OCR recognition dict path. If the Chinese model is used, change to "../ppocr/utils/ppocr_keys_v1.txt". And if you trained the model on your own dataset, change to the trained dictionary
+- table_model_dir：tabel recognition model path
+- table_char_dict_path：tabel recognition dict path. If the Chinese model is used, no need to change
+- layout_model_dir：layout analysis model path
+- layout_dict_path：layout analysis dict path. If the Chinese model is used, change to "../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt"
+- recovery：whether to enable layout of recovery, default False
+- save_pdf：when recovery file, whether to save pdf file, default False
+- output：save the recovery result path

ppstructure/recovery/README_ch.md

@@ -8,19 +8,22 @@
   - [2.2 安装PaddleOCR](#2.2)
 
 - [3. 使用](#3)
+  - [3.1 下载模型](#3.1)
+  - [3.2 版面恢复](#3.2)
 
 
 <a name="1"></a>
 
-## 1.  简介
+## 1. 简介
 
 版面恢复就是在OCR识别后，内容仍然像原文档图片那样排列着，段落不变、顺序不变的输出到word文档中等。
 
-版面恢复结合了[版面分析](../layout/README_ch.md)、[表格识别](../table/README_ch.md)技术，从而更好地恢复图片、表格、标题等内容，下图展示了版面恢复的结果：
+版面恢复结合了[版面分析](../layout/README_ch.md)、[表格识别](../table/README_ch.md)技术，从而更好地恢复图片、表格、标题等内容，支持pdf文档、文档图片格式的输入文件，下图展示了版面恢复的结果：
 
 <div align="center">
-<img src="../docs/table/recovery.jpg"  width = "700" />
+<img src="../docs/recovery/recovery.jpg"  width = "700" />
 </div>
+
 <a name="2"></a>
 
 ## 2. 安装
@@ -35,21 +38,15 @@
 python3 -m pip install --upgrade pip
 
 # GPU安装
-python3 -m pip install "paddlepaddle-gpu>=2.2" -i https://mirror.baidu.com/pypi/simple
+python3 -m pip install "paddlepaddle-gpu" -i https://mirror.baidu.com/pypi/simple
 
 # CPU安装
-python3 -m pip install "paddlepaddle>=2.2" -i https://mirror.baidu.com/pypi/simple
+python3 -m pip install "paddlepaddle" -i https://mirror.baidu.com/pypi/simple
 
 ```
 
 更多需求，请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
 
-* **(2)安装依赖**
-
-```bash
-python3 -m pip install -r ppstructure/recovery/requirements.txt
-```
-
 <a name="2.2"></a>
 
 ### 2.2 安装PaddleOCR
@@ -75,23 +72,66 @@ python3 -m pip install -r ppstructure/recovery/requirements.txt
 
 ## 3. 使用
 
-恢复给定文档的版面：
+<a name="3.1"></a>
+
+### 3.1 下载模型
+
+如果输入为英文文档类型，下载英文模型
 
-```python
+```bash
 cd PaddleOCR/ppstructure
 
 # 下载模型
 mkdir inference && cd inference
-# 下载超英文轻量级PP-OCRv3模型的检测模型并解压
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar && tar xf ch_PP-OCRv3_det_infer.tar
-# 下载英文轻量级PP-OCRv3模型的识别模型并解压
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar && tar xf  ch_PP-OCRv3_rec_infer.tar
-# 下载超轻量级英文表格英寸模型并解压
-wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
+# 下载英文超轻量PP-OCRv3检测模型并解压
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar && tar xf en_PP-OCRv3_det_infer.tar
+# 下载英文超轻量PP-OCRv3识别模型并解压
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar && tar xf en_PP-OCRv3_rec_infer.tar
+# 下载英文表格识别模型并解压
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
+# 下载英文版面分析模型
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar && tar xf picodet_lcnet_x1_0_fgd_layout_infer.tar
 cd ..
-# 执行预测
-python3 predict_system.py --det_model_dir=inference/en_PP-OCRv3_det_infer --rec_model_dir=inference/en_PP-OCRv3_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --rec_char_dict_path=../ppocr/utils/en_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --output ./output/table --rec_image_shape=3,48,320 --vis_font_path=../doc/fonts/simfang.ttf --recovery=True --image_dir=./docs/table/1.png
 ```
 
-运行完成后，每张图片的docx文档会保存到output字段指定的目录下
+如果输入为中文文档类型，在下述链接中下载中文模型即可：
+
+[PP-OCRv3中英文超轻量文本检测和识别模型](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/README_ch.md#pp-ocr%E7%B3%BB%E5%88%97%E6%A8%A1%E5%9E%8B%E5%88%97%E8%A1%A8%E6%9B%B4%E6%96%B0%E4%B8%AD)、[表格识别模型](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppstructure/docs/models_list.md#22-表格识别模型)、[版面分析模型](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppstructure/docs/models_list.md#1-版面分析模型)
+
+<a name="3.2"></a>
+
+### 3.2 版面恢复
+
+使用下载的模型恢复给定文档的版面，以英文模型为例，执行如下命令：
+
+```bash
+python3 predict_system.py \
+    --image_dir=./docs/table/1.png \
+    --det_model_dir=inference/en_PP-OCRv3_det_infer \
+    --rec_model_dir=inference/en_PP-OCRv3_rec_infer \
+    --rec_char_dict_path=../ppocr/utils/en_dict.txt \
+    --table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
+    --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+    --layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_infer \
+    --layout_dict_path=../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt \
+    --vis_font_path=../doc/fonts/simfang.ttf \
+    --recovery=True \
+    --save_pdf=False \
+    --output=../output/
+```
 
+运行完成后，恢复版面的docx文档会保存到`output`字段指定的目录下
+
+字段含义：
+
+- image_dir：测试文件，可以是图片、图片目录、pdf文件、pdf文件目录
+- det_model_dir：OCR检测模型路径
+- rec_model_dir：OCR识别模型路径
+- rec_char_dict_path：OCR识别字典，如果更换为中文模型，需要更改为"../ppocr/utils/ppocr_keys_v1.txt"，如果您在自己的数据集上训练的模型，则更改为训练的字典的文件
+- table_model_dir：表格识别模型路径
+- table_char_dict_path：表格识别字典，如果更换为中文模型，不需要更换字典
+- layout_model_dir：版面分析模型路径
+- layout_dict_path：版面分析字典，如果更换为中文模型，需要更改为"../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt"
+- recovery：是否进行版面恢复，默认False
+- save_pdf：进行版面恢复导出docx文档的同时，是否保存为pdf文件，默认为False
+- output：版面恢复结果保存路径

ppstructure/recovery/__init__.py

+# copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

ppstructure/recovery/recovery_to_doc.py

@@ -22,21 +22,23 @@ from docx import shared
 from docx.enum.text import WD_ALIGN_PARAGRAPH
 from docx.enum.section import WD_SECTION
 from docx.oxml.ns import qn
+from docx.enum.table import WD_TABLE_ALIGNMENT
+
+from ppstructure.recovery.table_process import HtmlToDocx
 
 from ppocr.utils.logging import get_logger
 logger = get_logger()
 
 
-def convert_info_docx(img, res, save_folder, img_name):
+def convert_info_docx(img, res, save_folder, img_name, save_pdf):
     doc = Document()
     doc.styles['Normal'].font.name = 'Times New Roman'
     doc.styles['Normal']._element.rPr.rFonts.set(qn('w:eastAsia'), u'宋体')
     doc.styles['Normal'].font.size = shared.Pt(6.5)
-    h, w, _ = img.shape
 
-    res = sorted_layout_boxes(res, w)
     flag = 1
     for i, region in enumerate(res):
+        img_idx = region['img_idx']
         if flag == 2 and region['layout'] == 'single':
             section = doc.add_section(WD_SECTION.CONTINUOUS)
             section._sectPr.xpath('./w:cols')[0].set(qn('w:num'), '1')
@@ -46,10 +48,10 @@ def convert_info_docx(img, res, save_folder, img_name):
             section._sectPr.xpath('./w:cols')[0].set(qn('w:num'), '2')
             flag = 2
 
-        if region['type'] == 'Figure':
+        if region['type'].lower() == 'figure':
             excel_save_folder = os.path.join(save_folder, img_name)
             img_path = os.path.join(excel_save_folder,
-                                    '{}.jpg'.format(region['bbox']))
+                                    '{}_{}.jpg'.format(region['bbox'], img_idx))
             paragraph_pic = doc.add_paragraph()
             paragraph_pic.alignment = WD_ALIGN_PARAGRAPH.CENTER
             run = paragraph_pic.add_run("")
@@ -57,40 +59,38 @@ def convert_info_docx(img, res, save_folder, img_name):
                 run.add_picture(img_path, width=shared.Inches(5))
             elif flag == 2:
                 run.add_picture(img_path, width=shared.Inches(2))
-        elif region['type'] == 'Title':
+        elif region['type'].lower() == 'title':
             doc.add_heading(region['res'][0]['text'])
-        elif region['type'] == 'Text':
+        elif region['type'].lower() == 'table':
+            paragraph = doc.add_paragraph()
+            new_parser = HtmlToDocx()
+            new_parser.table_style = 'TableGrid'
+            table = new_parser.handle_table(html=region['res']['html'])
+            new_table = deepcopy(table)
+            new_table.alignment = WD_TABLE_ALIGNMENT.CENTER
+            paragraph.add_run().element.addnext(new_table._tbl)
+
+        else:
             paragraph = doc.add_paragraph()
             paragraph_format = paragraph.paragraph_format
             for i, line in enumerate(region['res']):
                 if i == 0:
                     paragraph_format.first_line_indent = shared.Inches(0.25)
                 text_run = paragraph.add_run(line['text'] + ' ')
-                text_run.font.size = shared.Pt(9)
-        elif region['type'] == 'Table':
-            pypandoc.convert(
-                source=region['res']['html'],
-                format='html',
-                to='docx',
-                outputfile='tmp.docx')
-            tmp_doc = Document('tmp.docx')
-            paragraph = doc.add_paragraph()
-
-            table = tmp_doc.tables[0]
-            new_table = deepcopy(table)
-            new_table.style = doc.styles['Table Grid']
-            from docx.enum.table import WD_TABLE_ALIGNMENT
-            new_table.alignment = WD_TABLE_ALIGNMENT.CENTER
-            paragraph.add_run().element.addnext(new_table._tbl)
-            os.remove('tmp.docx')
-        else:
-            continue
+                text_run.font.size = shared.Pt(10)
 
     # save to docx
     docx_path = os.path.join(save_folder, '{}.docx'.format(img_name))
     doc.save(docx_path)
     logger.info('docx save to {}'.format(docx_path))
 
+    # save to pdf
+    if save_pdf:
+        pdf_path = os.path.join(save_folder, '{}.pdf'.format(img_name))
+        from docx2pdf import convert
+        convert(docx_path, pdf_path)
+        logger.info('pdf save to {}'.format(pdf_path))
+
 
 def sorted_layout_boxes(res, w):
     """
@@ -112,7 +112,7 @@ def sorted_layout_boxes(res, w):
     res_left = []
     res_right = []
     i = 0
-    
+
     while True:
         if i >= num_boxes:
             break
@@ -137,7 +137,7 @@ def sorted_layout_boxes(res, w):
             res_left = []
             res_right = []
             break
-        elif _boxes[i]['bbox'][0] < w / 4 and _boxes[i]['bbox'][2] < 3*w / 4:
+        elif _boxes[i]['bbox'][0] < w / 4 and _boxes[i]['bbox'][2] < 3 * w / 4:
             _boxes[i]['layout'] = 'double'
             res_left.append(_boxes[i])
             i += 1
@@ -157,4 +157,4 @@ def sorted_layout_boxes(res, w):
         new_res += res_left
     if res_right:
         new_res += res_right
-    return new_res
\ No newline at end of file
+    return new_res

ppstructure/recovery/requirements.txt

-opencv-contrib-python==4.4.0.46
 pypandoc
-python-docx
\ No newline at end of file
+python-docx
+docx2pdf
+fitz
+PyMuPDF
\ No newline at end of file

ppstructure/table/README.md

@@ -33,8 +33,8 @@ We evaluated the algorithm on the PubTabNet<sup>[1]</sup> eval dataset, and the
 |Method|Acc|[TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src)|Speed|
 | --- | --- | --- | ---|
 | EDD<sup>[2]</sup> |x| 88.3 |x|
-| TableRec-RARE(ours) |73.8%| 95.3% |1550ms|
-| SLANet(ours) | 76.2%|    95.85% |766ms|
+| TableRec-RARE(ours) | 71.73%| 93.88% |779ms|
+| SLANet(ours) | 76.31%|    95.89%|766ms|
 
 The performance indicators are explained as follows:
 - Acc: The accuracy of the table structure in each image, a wrong token is considered an error.
@@ -59,16 +59,16 @@ cd PaddleOCR/ppstructure
 # download model
 mkdir inference && cd inference
 # Download the PP-OCRv3 text detection model and unzip it
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.tar && tar xf ch_PP-OCRv3_det_slim_infer.tar
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar && tar xf ch_PP-OCRv3_det_infer.tar
 # Download the PP-OCRv3 text recognition model and unzip it
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.tar && tar xf ch_PP-OCRv3_rec_slim_infer.tar
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar && tar xf ch_PP-OCRv3_rec_infer.tar
 # Download the PP-Structurev2 form recognition model and unzip it
 wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
 cd ..
 # run
 python3.7 table/predict_table.py \
-    --det_model_dir=inference/ch_PP-OCRv3_det_slim_infer \
-    --rec_model_dir=inference/ch_PP-OCRv3_rec_slim_infer  \
+    --det_model_dir=inference/ch_PP-OCRv3_det_infer \
+    --rec_model_dir=inference/ch_PP-OCRv3_rec_infer  \
     --table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
     --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
     --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \

ppstructure/table/README_ch.md

@@ -39,8 +39,8 @@
 |算法|Acc|[TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src)|Speed|
 | --- | --- | --- | ---|
 | EDD<sup>[2]</sup> |x| 88.3% |x|
-| TableRec-RARE(ours) |73.8%| 95.3% |1550ms|
-| SLANet(ours) | 76.2%|	95.85% |766ms|
+| TableRec-RARE(ours) | 71.73%| 93.88% |779ms|
+| SLANet(ours) |76.31%|	95.89%|766ms|
 
 性能指标解释如下：
 - Acc: 模型对每张图像里表格结构的识别准确率，错一个token就算错误。
@@ -64,16 +64,16 @@ cd PaddleOCR/ppstructure
 # 下载模型
 mkdir inference && cd inference
 # 下载PP-OCRv3文本检测模型并解压
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.tar && tar xf ch_PP-OCRv3_det_slim_infer.tar
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar && tar xf ch_PP-OCRv3_det_infer.tar
 # 下载PP-OCRv3文本识别模型并解压
-wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.tar && tar xf ch_PP-OCRv3_rec_slim_infer.tar
+wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar && tar xf ch_PP-OCRv3_rec_infer.tar
 # 下载PP-Structurev2表格识别模型并解压
 wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
 cd ..
 # 执行表格识别
 python table/predict_table.py \
-    --det_model_dir=inference/ch_PP-OCRv3_det_slim_infer \
-    --rec_model_dir=inference/ch_PP-OCRv3_rec_slim_infer  \
+    --det_model_dir=inference/ch_PP-OCRv3_det_infer \
+    --rec_model_dir=inference/ch_PP-OCRv3_rec_infer  \
     --table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
     --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
     --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \

ppstructure/table/predict_structure.py

@@ -29,7 +29,7 @@ import tools.infer.utility as utility
 from ppocr.data import create_operators, transform
 from ppocr.postprocess import build_post_process
 from ppocr.utils.logging import get_logger
-from ppocr.utils.utility import get_image_file_list, check_and_read_gif
+from ppocr.utils.utility import get_image_file_list, check_and_read
 from ppocr.utils.visual import draw_rectangle
 from ppstructure.utility import parse_args
 
@@ -133,7 +133,7 @@ def main(args):
             os.path.join(args.output, 'infer.txt'), mode='w',
             encoding='utf-8') as f_w:
         for image_file in image_file_list:
-            img, flag = check_and_read_gif(image_file)
+            img, flag, _ = check_and_read(image_file)
             if not flag:
                 img = cv2.imread(image_file)
             if img is None:

ppstructure/table/predict_table.py

@@ -31,7 +31,7 @@ import tools.infer.predict_rec as predict_rec
 import tools.infer.predict_det as predict_det
 import tools.infer.utility as utility
 from tools.infer.predict_system import sorted_boxes
-from ppocr.utils.utility import get_image_file_list, check_and_read_gif
+from ppocr.utils.utility import get_image_file_list, check_and_read
 from ppocr.utils.logging import get_logger
 from ppstructure.table.matcher import TableMatch
 from ppstructure.table.table_master_match import TableMasterMatcher
@@ -194,7 +194,7 @@ def main(args):
 
     for i, image_file in enumerate(image_file_list):
         logger.info("[{}/{}] {}".format(i, img_num, image_file))
-        img, flag = check_and_read_gif(image_file)
+        img, flag, _ = check_and_read(image_file)
         excel_path = os.path.join(
             args.output, os.path.basename(image_file).split('.')[0] + '.xlsx')
         if not flag:

requirements.txt

@@ -6,7 +6,7 @@ lmdb
 tqdm
 numpy
 visualdl
-python-Levenshtein
+rapidfuzz
 opencv-contrib-python==4.4.0.46
 cython
 lxml