-[1.1 Introduction to Knowledge Distillation](#11)
-[1.2 Introduction to PaddleOCR Knowledge Distillation](#12)
+[2. Configuration File Analysis](#2)
+[2.1 Recognition Model Configuration File Analysis](#21)
-[2.1.1 Model Structure](#211)
-[2.1.2 Loss Function ](#212)
-[2.1.3 Post-processing](#213)
-[2.1.4 Metric Calculation](#214)
-[2.1.5 Fine-tuning Distillation Model](#215)
+[2.2 Detection Model Configuration File Analysis](#22)
-[2.2.1 Model Structure](#221)
-[2.2.2 Loss Function](#222)
-[2.2.3 Post-processing](#223)
-[2.2.4 Metric Calculation](#224)
-[2.2.5 Fine-tuning Distillation Model](#225)
<aname="1"></a>
## 1. Introduction
<aname="11"></a>
### 1.1 Introduction to Knowledge Distillation
In recent years, deep neural networks have been proved to be an extremely effective method for solving problems in the fields of computer vision and natural language processing.
By constructing a suitable neural network and training it, the performance metrics of the final network model will basically exceed the traditional algorithm.
When the amount of data is large enough, increasing the amount of parameters by constructing a reasonable network model can significantly improve the performance of the model,
but this brings about the problem of a sharp increase in the complexity of the model. Large models are more expensive to use in actual scenarios.
Deep neural networks generally have more parameter redundancy. At present, there are several main methods to compress the model and reduce the amount of its parameters.
Such as pruning, quantification, knowledge distillation, etc., where knowledge distillation refers to the use of teacher models to guide student models to learn specific tasks,
to ensure that the small model obtains a relatively large performance improvement under the condition of unchanged parameters.
In addition, in the knowledge distillation task, a mutual learning model training method was also derived.
The paper [Deep Mutual Learning](https://arxiv.org/abs/1706.00384) pointed out that using two identical models to supervise each other during the training process can achieve better results than a single model training.
<aname="12"></a>
### 1.2 Introduction to PaddleOCR Knowledge Distillation
Whether it is a large model distilling a small model, or a small model learning from each other and updating parameters,
they are essentially the output between different models or mutual supervision between feature maps.
The only difference is (1) whether the model requires fixed parameters. (2) Whether the model needs to be loaded with a pre-trained model.
For the case where a large model distills a small model, the large model generally needs to load the pre-trained model and fix the parameters.
For the situation where small models distill each other, the small models generally do not load the pre-trained model, and the parameters are also in a learnable state.
In the task of knowledge distillation, it is not only the distillation between two models, but also the situation where multiple models learn from each other.
Therefore, in the knowledge distillation code framework, it is also necessary to support this type of distillation method.
The algorithm of knowledge distillation is integrated in PaddleOCR. Specifically, it has the following main features:
- It supports mutual learning of any network, and does not require the sub-network structure to be completely consistent or to have a pre-trained model. At the same time, there is no limit to the number of sub-networks, just add it in the configuration file.
- Support arbitrarily configuring the loss function through the configuration file, not only can use a certain loss, but also a combination of multiple losses.
- Support all model-related environments such as knowledge distillation training, prediction, evaluation, and export, which is convenient for use and deployment.
Through knowledge distillation, in the common Chinese and English text recognition task, without adding any time-consuming prediction,
the accuracy of the model can be improved by more than 3%. Combining the learning rate adjustment strategy and the model structure fine-tuning strategy,
the final improvement is more than 5%.
<aname="2"></a>
## 2. Configuration File Analysis
In the process of knowledge distillation training, there is no change in data preprocessing, optimizer, learning rate, and some global attributes.
The configuration files of the model structure, loss function, post-processing, metric calculation and other modules need to be fine-tuned.
The following takes the knowledge distillation configuration file for recognition and detection as an example to analyze the training and configuration of knowledge distillation.
<aname="21"></a>
### 2.1 Recognition Model Configuration File Analysis
The configuration file is in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml).
<aname="211"></a>
#### 2.1.1 Model Structure
In the knowledge distillation task, the model structure configuration is as follows.
```yaml
Architecture:
model_type:&model_type"rec"# Model category, recognition, detection, etc.
name:DistillationModel# Structure name, in the distillation task, it is DistillationModel
algorithm:Distillation# Algorithm name
Models:# Model, including the configuration information of the subnet
Teacher:# The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet
pretrained:# Does this sub-network need to load pre-training weights
freeze_params:false# Do you need fixed parameters
return_all_feats:true# Do you need to return all features, if it is False, only the final output is returned
model_type:*model_type# Model category
algorithm:CRNN# The algorithm name of the sub-network. The remaining parameters of the sub-network are consistent with the general model training configuration
Transform:
Backbone:
name:MobileNetV1Enhance
scale:0.5
Neck:
name:SequenceEncoder
encoder_type:rnn
hidden_size:64
Head:
name:CTCHead
mid_channels:96
fc_decay:0.00002
Student:# Another sub-network, here is a distillation example of DML, the two sub-networks have the same structure, and both need to learn parameters
pretrained:# The following parameters are the same as above
freeze_params:false
return_all_feats:true
model_type:*model_type
algorithm:CRNN
Transform:
Backbone:
name:MobileNetV1Enhance
scale:0.5
Neck:
name:SequenceEncoder
encoder_type:rnn
hidden_size:64
Head:
name:CTCHead
mid_channels:96
fc_decay:0.00002
```
If you want to add more sub-networks for training, you can also add the corresponding fields in the configuration file according to the way of adding `Student` and `Teacher`.
For example, if you want 3 models to supervise each other and train together, then `Architecture` can be written in the following format.
```yaml
Architecture:
model_type:&model_type"rec"
name:DistillationModel
algorithm:Distillation
Models:
Teacher:
pretrained:
freeze_params:false
return_all_feats:true
model_type:*model_type
algorithm:CRNN
Transform:
Backbone:
name:MobileNetV1Enhance
scale:0.5
Neck:
name:SequenceEncoder
encoder_type:rnn
hidden_size:64
Head:
name:CTCHead
mid_channels:96
fc_decay:0.00002
Student:
pretrained:
freeze_params:false
return_all_feats:true
model_type:*model_type
algorithm:CRNN
Transform:
Backbone:
name:MobileNetV1Enhance
scale:0.5
Neck:
name:SequenceEncoder
encoder_type:rnn
hidden_size:64
Head:
name:CTCHead
mid_channels:96
fc_decay:0.00002
Student2:# The new sub-network introduced in the knowledge distillation task, the configuration is the same as above
pretrained:
freeze_params:false
return_all_feats:true
model_type:*model_type
algorithm:CRNN
Transform:
Backbone:
name:MobileNetV1Enhance
scale:0.5
Neck:
name:SequenceEncoder
encoder_type:rnn
hidden_size:64
Head:
name:CTCHead
mid_channels:96
fc_decay:0.00002
```
When the model is finally trained, it contains 3 sub-networks: `Teacher`, `Student`, `Student2`.
The specific implementation code of the `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py).
The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network,
which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle).
In the recognition task, in order to add more loss functions and ensure the scalability of the distillation method, the output of each sub-network is saved as a `dict`, which contains the sub-module output.
Take the recognition model as an example. The output result of each sub-network is `dict`, the key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, `DistillationModel` The output format is as follows.
```json
{
"Teacher":{
"backbone_out":tensor,
"neck_out":tensor,
"head_out":tensor,
},
"Student":{
"backbone_out":tensor,
"neck_out":tensor,
"head_out":tensor,
}
}
```
<aname="212"></a>
#### 2.1.2 Loss Function
In the knowledge distillation task, the loss function configuration is as follows.
```yaml
Loss:
name:CombinedLoss# Loss function name
loss_config_list:# List of loss function configuration files, mandatory functions for CombinedLoss
-DistillationCTCLoss:# CTC loss function based on distillation, inherited from standard CTC loss
weight:1.0# The weight of the loss function. In loss_config_list, each loss function must include this field
model_name_list:["Student","Teacher"]# For the prediction results of the distillation model, extract the output of these two sub-networks and calculate the CTC loss with gt
key:head_out# In the sub-network output dict, take the corresponding tensor
-DistillationDMLLoss:# DML loss function, inherited from the standard DMLLoss
weight:1.0
act:"softmax"# Activation function, use it to process the input, can be softmax, sigmoid or None, the default is None
model_name_pairs:# The subnet name pair used to calculate DML loss. If you want to calculate the DML loss of other subnets, you can continue to add it below the list
-["Student","Teacher"]
key:head_out
-DistillationDistanceLoss:# Distilled distance loss function
weight:1.0
mode:"l2"# Support l1, l2 or smooth_l1
model_name_pairs:# Calculate the distance loss of the subnet name pair
-["Student","Teacher"]
key:backbone_out
```
Among the above loss functions, all distillation loss functions are inherited from the standard loss function class.
The main functions are: Analyze the output of the distillation model, find the intermediate node (tensor) used to calculate the loss,
and then use the standard loss function class to calculate.
Taking the above configuration as an example, the final distillation training loss function contains the following three parts.
- The final output `head_out` of `Student` and `Teacher` calculates the CTC loss with gt (loss weight equals 1.0). Here, because both sub-networks need to update the parameters, both of them need to calculate the loss with gt.
- DML loss between `Student` and `Teacher`'s final output `head_out` (loss weight equals 1.0).
- L2 loss between `Student` and `Teacher`'s backbone network output `backbone_out` (loss weight equals 1.0).
For more specific implementation of `CombinedLoss`, please refer to: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23).
For more specific implementations of distillation loss functions such as `DistillationCTCLoss`, please refer to [distillation_loss.py](../../ppocr/losses/distillation_loss.py)
<aname="213"></a>
#### 2.1.3 Post-processing
In the knowledge distillation task, the post-processing configuration is as follows.
```yaml
PostProcess:
name:DistillationCTCLabelDecode# CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class
model_name:["Student","Teacher"]# For the prediction results of the distillation model, extract the outputs of these two sub-networks and decode them
key:head_out# Take the corresponding tensor in the subnet output dict
```
Taking the above configuration as an example, the CTC decoding output of the two sub-networks `Student` and `Teahcer` will be calculated at the same time.
Among them, `key` is the name of the subnet, and `value` is the list of subnets.
For more specific implementation of `DistillationCTCLabelDecode`, please refer to: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128)
<aname="214"></a>
#### 2.1.4 Metric Calculation
In the knowledge distillation task, the metric calculation configuration is as follows.
```yaml
Metric:
name:DistillationMetric# CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class
base_metric_name:RecMetric# The base class of indicator calculation. For the output of the model, the indicator will be calculated based on this class
main_indicator:acc# The name of the indicator
key:"Student"# Select the main_indicator of this subnet as the criterion for saving the best model
```
Taking the above configuration as an example, the accuracy metric of the `Student` subnet will be used as the judgment metric for saving the best model.
At the same time, the accuracy metric of all subnets will be printed out in the log.
For more specific implementation of `DistillationMetric`, please refer to: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24).
<aname="215"></a>
#### 2.1.5 Fine-tuning Distillation Model
There are two ways to fine-tune the recognition distillation task.
1. Fine-tuning based on knowledge distillation: this situation is relatively simple, download the pre-trained model. Then configure the pre-training model path and your own data path in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml) to perform fine-tuning training of the model.
2. Do not use knowledge distillation in fine-tuning: In this case, you need to first extract the student model parameters from the pre-training model. The specific steps are as follows.
- First download the pre-trained model and unzip it.
After the extraction is complete, use [ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml) to modify the path of the pre-trained model (the path of the exported `student.pdparams` model) and your own data path to fine-tune the model.
<aname="22"></a>
### 2.2 Detection Model Configuration File Analysis
The configuration file of the detection model distillation is in the ```PaddleOCR/configs/det/ch_PP-OCRv2/``` directory, which contains three distillation configuration files:
-```ch_PP-OCRv2_det_cml.yml```, Use one large model to distill two small models, and the two small models learn from each other
-```ch_PP-OCRv2_det_dml.yml```, Method of mutual distillation of two student models
-```ch_PP-OCRv2_det_distill.yml```, The method of using large teacher model to distill small student model
<aname="221"></a>
#### 2.2.1 Model Structure
In the knowledge distillation task, the model structure configuration is as follows:
```
Architecture:
name: DistillationModel # Structure name, in the distillation task, it is DistillationModel
algorithm: Distillation # Algorithm name
Models: # Model, including the configuration information of the subnet
Student: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet
pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained # Does this sub-network need to load pre-training weights
freeze_params: false # Do you need fixed parameters
return_all_feats: false # Do you need to return all features, if it is False, only the final output is returned
model_type: det
algorithm: DB
Backbone:
name: MobileNetV3
scale: 0.5
model_name: large
disable_se: True
Neck:
name: DBFPN
out_channels: 96
Head:
name: DBHead
k: 50
Teacher: # Another sub-network, here is a distillation example of a large model distill a small model
freeze_params: true # The Teacher model is well-trained and does not need to participate in training
return_all_feats: false
model_type: det
algorithm: DB
Transform:
Backbone:
name: ResNet
layers: 18
Neck:
name: DBFPN
out_channels: 256
Head:
name: DBHead
k: 50
```
If DML is used, that is, the method of two small models learning from each other, the Teacher network structure in the above configuration file needs to be set to the same configuration as the Student model.
Refer to the configuration file for details. [ch_PP-OCRv2_det_dml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.4/configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_dml.yml)
The following describes the configuration file parameters [ch_PP-OCRv2_det_cml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.4/configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml):
```
Architecture:
name: DistillationModel
algorithm: Distillation
model_type: det
Models:
Teacher: # Teacher model configuration of CML distillation
The specific implementation code of the distillation model `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py).
The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network,
which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle).
In the distillation task, in order to facilitate the addition of the distillation loss function, the output of each network is saved as a `dict`, which contains the sub-module output.
The key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, the output format of `DistillationModel` is as follows.
```json
{
"Teacher":{
"backbone_out":tensor,
"neck_out":tensor,
"head_out":tensor,
},
"Student":{
"backbone_out":tensor,
"neck_out":tensor,
"head_out":tensor,
}
}
```
<aname="222"></a>
#### 2.2.2 Loss Function
In the task of detection knowledge distillation ```ch_PP-OCRv2_det_distill.yml````, the distillation loss function configuration is as follows.
```yaml
Loss:
name: CombinedLoss # Loss function name
loss_config_list: # List of loss function configuration files, mandatory functions for CombinedLoss
- DistillationDilaDBLoss: # DB loss function based on distillation, inherited from standard DBloss
weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field
model_name_pairs: # Extract the output of these two sub-networks and calculate the loss between them
- ["Student", "Teacher"]
key: maps # In the sub-network output dict, take the corresponding tensor
balance_loss: true # The following parameters are the configuration parameters of standard DBloss
main_loss_type: DiceLoss
alpha: 5
beta: 10
ohem_ratio: 3
- DistillationDBLoss: # Used to calculate the loss between Student and GT
weight: 1.0
model_name_list: ["Student"] # The model name only has Student, which means that the loss between Student and GT is calculated
name: DBLoss
balance_loss: true
main_loss_type: DiceLoss
alpha: 5
beta: 10
ohem_ratio: 3
```
Similarly, distillation loss function configuration(`ch_PP-OCRv2_det_cml.yml`) is shown below. Compared with the loss function configuration of ch_PP-OCRv2_det_distill.yml, there are three changes:
```yaml
Loss:
name: CombinedLoss
loss_config_list:
- DistillationDilaDBLoss:
weight: 1.0
model_name_pairs:
- ["Student", "Teacher"]
- ["Student2", "Teacher"] # 1. Calculate the loss of two Student and Teacher
key: maps
balance_loss: true
main_loss_type: DiceLoss
alpha: 5
beta: 10
ohem_ratio: 3
- DistillationDMLLoss: # 2. Add to calculate the loss between two students
model_name_pairs:
- ["Student", "Student2"]
maps_name: "thrink_maps"
weight: 1.0
# act: None
key: maps
- DistillationDBLoss:
weight: 1.0
model_name_list: ["Student", "Student2"] # 3. Calculate the loss between two students and GT
balance_loss: true
main_loss_type: DiceLoss
alpha: 5
beta: 10
ohem_ratio: 3
```
For more specific implementation of `DistillationDilaDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185).
For more specific implementations of distillation loss functions such as `DistillationDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148)
<a name="223"></a>
#### 2.2.3 Post-processing
In the task of detecting knowledge distillation, the post-processing configuration of detecting distillation is as follows.
```yaml
PostProcess:
name: DistillationDBPostProcess # The CTC decoding post-processing of the DB detection distillation task, inherited from the standard DBPostProcess class
model_name: ["Student", "Student2", "Teacher"] # Extract the output of multiple sub-networks and decode them. The network that does not require post-processing is not set in model_name
thresh: 0.3
box_thresh: 0.6
max_candidates: 1000
unclip_ratio: 1.5
```
Taking the above configuration as an example, the output of the three subnets `Student`, `Student2` and `Teacher` will be calculated at the same time for post-processing calculations.
Since there are multiple inputs, there are also multiple outputs returned by post-processing.
For a more specific implementation of `DistillationDBPostProcess`, please refer to: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195)
<a name="224"></a>
#### 2.2.4 Metric Calculation
In the knowledge distillation task, the metric calculation configuration is as follows.
```yaml
Metric:
name: DistillationMetric
base_metric_name: DetMetric
main_indicator: hmean
key: "Student"
```
Since distillation needs to include multiple networks, only one network metrics needs to be calculated when calculating the metrics.
The `key` field is set to `Student`, it means that only the metrics of the `Student` network is calculated.
Model Structure
<a name="225"></a>
#### 2.2.5 Fine-tuning Distillation Model
There are three ways to fine-tune the detection distillation task:
- `ch_PP-OCRv2_det_distill.yml`, The teacher model is set to the model provided by PaddleOCR or the large model you have trained.
- `ch_PP-OCRv2_det_cml.yml`, Use cml distillation. Similarly, the Teacher model is set to the model provided by PaddleOCR or the large model you have trained.
- `ch_PP-OCRv2_det_dml.yml`, Distillation using DML. The method of mutual distillation of the two Student models has an accuracy improvement of about 1.7% on the data set used by PaddleOCR.
In fine-tune, you need to set the pre-trained model to be loaded in the `pretrained` parameter of the network structure.
In terms of accuracy improvement, `cml` > `dml` > `distill`. When the amount of data is insufficient or the accuracy of the teacher model is similar to that of the student, this conclusion may change.
In addition, since the distillation pre-training model provided by PaddleOCR contains multiple model parameters, if you want to extract the parameters of the student model, you can refer to the following code:
```sh
# Download the parameters of the distillation training model
PP-Structure is an OCR toolkit that can be used for complex documents analysis. The main features are as follows:
- Support the layout analysis of documents, divide the documents into 5 types of areas **text, title, table, image and list** (conjunction with Layout-Parser)
- Support to extract the texts from the text, title, picture and list areas (used in conjunction with PP-OCR)
- Support to extract excel files from the table areas
- Support python whl package and command line usage, easy to use
- Support custom training for layout analysis and table structure tasks
PP-Structure is an OCR toolkit that can be used for document analysis and processing with complex structures, designed to help developers better complete document understanding tasks
<aname="2"></a>
## 2. Installation
## 2. Update log
* 2021.12.07 add [DOC-VQA SER and RE tasks](vqa/README.md)。
- Support the layout analysis of documents, divide the documents into 5 types of areas **text, title, table, image and list** (conjunction with Layout-Parser)
- Support to extract the texts from the text, title, picture and list areas (used in conjunction with PP-OCR)
- Support to extract excel files from the table areas
- Support python whl package and command line usage, easy to use
- Support custom training for layout analysis and table structure tasks
- Support Document Visual Question Answering (DOC-VQA) tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE)
```
For more，refer [Installation](https://www.paddlepaddle.org.cn/install/quick) .
The figure shows the pipeline of layout analysis + table recognition. The image is first divided into four areas of image, text, title and table by layout analysis, and then OCR detection and recognition is performed on the three areas of image, text and title, and the table is performed table recognition, where the image will also be stored for use.
The returned results of PP-Structure is a list composed of a dict, an example is as follows
In the figure, the red box represents the question, the blue box represents the answer, and the question and answer are connected by green lines. The corresponding category and OCR recognition results are also marked at the top left of the OCR detection box.
[('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent ', 0.465441)])
}
]
```
The description of each field in dict is as follows
| Parameter | Description |
| --------------- | -------------|
|type|Type of image area|
|bbox|The coordinates of the image area in the original image, respectively [left upper x, left upper y, right bottom x, right bottom y]|
|res|OCR or table recognition result of image area。<br> Table: HTML string of the table; <br> OCR: A tuple containing the detection coordinates and recognition results of each single line of text|
<aname="5"></a>
## 5. Quick start
### 3.4 Parameter description：
Start from [Quick Installation](./docs/quickstart.md)
| output | The path where excel and recognition results are saved | ./output/table |
| table_max_len | The long side of the image is resized in table structure model | 488 |
| table_model_dir | inference model path of table structure model | None |
| table_char_type | dict path of table structure model | ../ppocr/utils/dict/table_structure_dict.tx |
<aname="6"></a>
Most of the parameters are consistent with the paddleocr whl package, see [doc of whl](../doc/doc_en/whl_en.md)
## 6. PP-Structure System
After running, each image will have a directory with the same name under the directory specified in the output field. Each table in the picture will be stored as an excel and figure area will be cropped and saved, the excel and image file name will be the coordinates of the table in the image.
<aname="61"></a>
## 4. PP-Structure Pipeline
### 6.1 Layout analysis and table recognition
In PP-Structure, the image will be analyzed by layoutparser first. In the layout analysis, the area in the image will be classified, including **text, title, image, list and table** 5 categories. For the first 4 types of areas, directly use the PP-OCR to complete the text detection and recognition. The table area will be converted to an excel file of the same table style via Table OCR.
### 4.1 LayoutParser
In PP-Structure, the image will be divided into 5 types of areas **text, title, image list and table**. For the first 4 types of areas, directly use PP-OCR system to complete the text detection and recognition. For the table area, after the table structuring process, the table in image is converted into an Excel file with the same table style.
Layout analysis divides the document data into regions, including the use of Python scripts for layout analysis tools, extraction of special category detection boxes, performance indicators, and custom training layout analysis models. For details, please refer to [document](layout/README_en.md).
#### 6.1.1 Layout analysis
### 4.2 Table Recognition
Layout analysis classifies image by region, including the use of Python scripts of layout analysis tools, extraction of designated category detection boxes, performance indicators, and custom training layout analysis models. For details, please refer to [document](layout/README.md).
Table Recognition converts table image into excel documents, which include the detection and recognition of table text and the prediction of table structure and cell coordinates. For detailed, please refer to [document](table/README.md)
#### 6.1.2 Table recognition
## 5. Prediction by inference engine
Table recognition converts table images into excel documents, which include the detection and recognition of table text and the prediction of table structure and cell coordinates. For detailed instructions, please refer to [document](table/README.md)
Use the following commands to complete the inference.
<aname="62"></a>
```python
cdPaddleOCR/ppstructure
### 6.2 DOC-VQA
# download model
mkdirinference&&cdinference
# Download the detection model of the ultra-lightweight Chinese OCR model and uncompress it
Document Visual Question Answering (DOC-VQA) if a type of Visual Question Answering (VQA), which includes Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks. Based on SER task, text recognition and classification in images can be completed. Based on THE RE task, we can extract the relation of the text content in the image, such as judge the problem pair. For details, please refer to [document](vqa/README.md)
After running, each image will have a directory with the same name under the directory specified in the output field. Each table in the picture will be stored as an excel and figure area will be cropped and saved, the excel and image file name will be the coordinates of the table in the image.
|en_ppocr_mobile_v2.0_table_structure|Table structure prediction for English table scenarios|[table_mv3.yml](../configs/table/table_mv3.yml)|18.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) |
## 7. Model List
**Model List**
PP-Structure系列模型列表（更新中）
LayoutParser model
* Layout analysis model
|model name|description|download|
| --- | --- | --- |
| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis model trained on the PubLayNet data set can be divided into 5 types of areas **text, title, table, picture and list** | [PubLayNet](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) |
| ppyolov2_r50vd_dcn_365e_tableBank_word | The layout analysis model trained on the TableBank Word dataset can only detect tables | [TableBank Word](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) |
| ppyolov2_r50vd_dcn_365e_tableBank_latex | The layout analysis model trained on the TableBank Latex dataset can only detect tables | [TableBank Latex](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) |
| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis model trained on the PubLayNet dataset can divide image into 5 types of areas **text, title, table, picture, and list** | [PubLayNet](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) |
|ch_ppocr_mobile_slim_v2.0_rec|Slim pruned and quantized lightweight model, supporting Chinese, English and number recognition|6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_train.tar) |
|en_ppocr_mobile_v2.0_table_det|Text detection of English table scenes trained on PubLayNet dataset|4.7M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
|en_ppocr_mobile_v2.0_table_rec|Text recognition of English table scene trained on PubLayNet dataset|6.9M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar)[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
|en_ppocr_mobile_v2.0_table_structure|Table structure prediction of English table scene trained on PubLayNet dataset|18.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
|en_ppocr_mobile_v2.0_table_structure|Table structure prediction of English table scene trained on PubLayNet dataset|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
* DOC-VQA model
|model name|description|model size|download|
| --- | --- | --- | --- |
|PP-Layout_v1.0_ser_pretrained|SER model trained on xfun Chinese dataset based on LayoutXLM|1.4G|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar) |
|PP-Layout_v1.0_re_pretrained|RE model trained on xfun Chinese dataset based on LayoutXLM|1.4G|[inference model coming soon]() / [trained model](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_re_pretrained.tar) |
If you need to use other models, you can download the model in [model_list](../doc/doc_en/models_list_en.md) or use your own trained model to configure it to the three fields of `det_model_dir`, `rec_model_dir`, `table_model_dir` .
If you need to use other models, you can download the model in [PPOCR model_list](../doc/doc_en/models_list_en.md) and [PPStructure model_list](./docs/model_list.md)
