+ 
+
+
+
+
+## 2. 安装依赖
+
+### 2.1. 安装PaddlePaddle
+
+- **（1) 安装PaddlePaddle**
-## 1. Install whl package
```bash
-wget https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
-pip install -U layoutparser-0.0.0-py3-none-any.whl
+python3 -m pip install --upgrade pip
+
+# GPU安装
+python3 -m pip install "paddlepaddle-gpu>=2.2" -i https://mirror.baidu.com/pypi/simple
+
+# CPU安装
+python3 -m pip install "paddlepaddle>=2.2" -i https://mirror.baidu.com/pypi/simple
```
+更多需求，请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
-## 2. Quick Start
+### 2.2. 安装PaddleDetection
-Use LayoutParser to identify the layout of a document:
+- **（1）下载PaddleDetection源码**
-```python
-import cv2
-import layoutparser as lp
-image = cv2.imread("doc/table/layout.jpg")
-image = image[..., ::-1]
+```bash
+git clone https://github.com/PaddlePaddle/PaddleDetection.git
+```
-# load model
-model = lp.PaddleDetectionLayoutModel(config_path="lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config",
- threshold=0.5,
- label_map={0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"},
- enforce_cpu=False,
- enable_mkldnn=True)
-# detect
-layout = model.detect(image)
+- **（2）安装其他依赖 **
-# show result
-show_img = lp.draw_box(image, layout, box_width=3, show_element_type=True)
-show_img.show()
+```bash
+cd PaddleDetection
+python3 -m pip install -r requirements.txt
```
-The following figure shows the result, with different colored detection boxes representing different categories and displaying specific categories in the upper left corner of the box with `show_element_type`
+## 3. 数据准备
-
+
-
-
-`PaddleDetectionLayoutModel`parameters are described as follows:
+如果希望直接体验预测过程，可以跳过数据准备，下载我们提供的预训练模型。
+
+### 3.1. 英文数据集
+
+下载文档分析数据集[PubLayNet](https://developer.ibm.com/exchanges/data/all/publaynet/)（数据集96G），包含5个类：`{0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}`
-| parameter | description | default | remark |
-| :------------: | :------------------------------------------------------: | :---------: | :----------------------------------------------------------: |
-| config_path | model config path | None | Specify config_ path will automatically download the model (only for the first time,the model will exist and will not be downloaded again) |
-| model_path | model path | None | local model path, config_ path and model_ path must be set to one, cannot be none at the same time |
-| threshold | threshold of prediction score | 0.5 | \ |
-| input_shape | picture size of reshape | [3,640,640] | \ |
-| batch_size | testing batch size | 1 | \ |
-| label_map | category mapping table | None | Setting config_ path, it can be none, and the label is automatically obtained according to the dataset name_ map, You need to specify it manually when setting model_path |
-| enforce_cpu | whether to use CPU | False | False to use GPU, and True to force the use of CPU |
-| enforce_mkldnn | whether mkldnn acceleration is enabled in CPU prediction | True | \ |
-| thread_num | the number of CPU threads | 10 | \ |
+```
+# 下载数据
+wget https://dax-cdn.cdn.appdomain.cloud/dax-publaynet/1.0.0/publaynet.tar.gz
+# 解压数据
+tar -xvf publaynet.tar.gz
+```
-The following model configurations and label maps are currently supported, which you can use by modifying '--config_path' and '--label_map' to detect different types of content:
+解压之后的**目录结构：**
-| dataset | config_path | label_map |
-| ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------- |
-| [TableBank](https://doc-analysis.github.io/tablebank-page/index.html) word | lp://TableBank/ppyolov2_r50vd_dcn_365e_tableBank_word/config | {0:"Table"} |
-| TableBank latex | lp://TableBank/ppyolov2_r50vd_dcn_365e_tableBank_latex/config | {0:"Table"} |
-| [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) | lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config | {0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"} |
+```
+|-publaynet
+ |- test
+ |- PMC1277013_00004.jpg
+ |- PMC1291385_00002.jpg
+ | ...
+ |- train.json
+ |- train
+ |- PMC1291385_00002.jpg
+ |- PMC1277013_00004.jpg
+ | ...
+ |- val.json
+ |- val
+ |- PMC538274_00004.jpg
+ |- PMC539300_00004.jpg
+ | ...
+```
-* TableBank word and TableBank latex are trained on datasets of word documents and latex documents respectively;
-* Download TableBank dataset contains both word and latex。
+**数据分布：**
-## 3. PostProcess
+| File or Folder | Description | num |
+| :------------- | :------------- | ------- |
+| `train/` | 训练集图片 | 335,703 |
+| `val/` | 验证集图片 | 11,245 |
+| `test/` | 测试集图片 | 11,405 |
+| `train.json` | 训练集标注文件 | - |
+| `val.json` | 验证集标注文件 | - |
-Layout parser contains multiple categories, if you only want to get the detection box for a specific category (such as the "Text" category), you can use the following code:
+**标注格式：**
-```python
-# follow the above code
-# filter areas for a specific text type
-text_blocks = lp.Layout([b for b in layout if b.type=='Text'])
-figure_blocks = lp.Layout([b for b in layout if b.type=='Figure'])
+json文件包含所有图像的标注，数据以字典嵌套的方式存放，包含以下key：
-# text areas may be detected within the image area, delete these areas
-text_blocks = lp.Layout([b for b in text_blocks \
- if not any(b.is_in(b_fig) for b_fig in figure_blocks)])
+- info，表示标注文件info。
-# sort text areas and assign ID
-h, w = image.shape[:2]
+- licenses，表示标注文件licenses。
-left_interval = lp.Interval(0, w/2*1.05, axis='x').put_on_canvas(image)
+- images，表示标注文件中图像信息列表，每个元素是一张图像的信息。如下为其中一张图像的信息：
-left_blocks = text_blocks.filter_by(left_interval, center=True)
-left_blocks.sort(key = lambda b:b.coordinates[1])
+ ```
+ {
+ 'file_name': 'PMC4055390_00006.jpg', # file_name
+ 'height': 601, # image height
+ 'width': 792, # image width
+ 'id': 341427 # image id
+ }
+ ```
-right_blocks = [b for b in text_blocks if b not in left_blocks]
-right_blocks.sort(key = lambda b:b.coordinates[1])
+- annotations，表示标注文件中目标物体的标注信息列表，每个元素是一个目标物体的标注信息。如下为其中一个目标物体的标注信息：
-# the two lists are merged and the indexes are added in order
-text_blocks = lp.Layout([b.set(id = idx) for idx, b in enumerate(left_blocks + right_blocks)])
+ ```
+ {
+
+ 'segmentation': # 物体的分割标注
+ 'area': 60518.099043117836, # 物体的区域面积
+ 'iscrowd': 0, # iscrowd
+ 'image_id': 341427, # image id
+ 'bbox': [50.58, 490.86, 240.15, 252.16], # bbox [x1,y1,w,h]
+ 'category_id': 1, # category_id
+ 'id': 3322348 # image id
+ }
+ ```
+
+### 3.2. 更多数据集
+
+我们提供了CDLA(中文版面分析)、TableBank(表格版面分析)等数据集的下连接，处理为上述标注文件json格式，即可以按相同方式进行训练。
+
+| dataset | 简介 |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| [cTDaR2019_cTDaR](https://cndplab-founder.github.io/cTDaR2019/) | 用于表格检测(TRACKA)和表格识别(TRACKB)。图片类型包含历史数据集(以cTDaR_t0开头，如cTDaR_t00872.jpg)和现代数据集(以cTDaR_t1开头，cTDaR_t10482.jpg)。 |
+| [IIIT-AR-13K](http://cvit.iiit.ac.in/usodi/iiitar13k.php) | 手动注释公开的年度报告中的图形或页面而构建的数据集，包含5类：table, figure, natural image, logo, and signature |
+| [CDLA](https://github.com/buptlihang/CDLA) | 中文文档版面分析数据集，面向中文文献类（论文）场景，包含10类：Table、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation |
+| [TableBank](https://github.com/doc-analysis/TableBank) | 用于表格检测和识别大型数据集，包含Word和Latex2种文档格式 |
+| [DocBank](https://github.com/doc-analysis/DocBank) | 使用弱监督方法构建的大规模数据集(500K文档页面)，用于文档布局分析，包含12类：Author、Caption、Date、Equation、Figure、Footer、List、Paragraph、Reference、Section、Table、Title |
+
+
+## 4. 开始训练
+
+提供了训练脚本、评估脚本和预测脚本，本节将以PubLayNet预训练模型为例进行讲解。
+
+如果不希望训练，直接体验后面的模型评估、预测、动转静、推理的流程，可以下载提供的预训练模型，并跳过本部分。
-# display result
-show_img = lp.draw_box(image, text_blocks,
- box_width=3,
- show_element_id=True)
-show_img.show()
+```
+mkdir pretrained_model
+cd pretrained_model
+# 下载并解压PubLayNet预训练模型
+wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout.pdparams
```
-Displays results with only the "Text" category：
+### 4.1. 启动训练
+
+开始训练:
+
+* 修改配置文件
+
+如果你希望训练自己的数据集，需要修改配置文件中的数据配置、类别数。
-
-
-
-
-## 4. Results
+以`configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 为例，修改的内容如下所示。
-| Dataset | mAP | CPU time cost | GPU time cost |
-| --------- | ---- | ------------- | ------------- |
-| PubLayNet | 93.6 | 1713.7ms | 66.6ms |
-| TableBank | 96.2 | 1968.4ms | 65.1ms |
+```yaml
+metric: COCO
+# 类别数
+num_classes: 5
+
+TrainDataset:
+ !COCODataSet
+ # 修改为你自己的训练数据目录
+ image_dir: train
+ # 修改为你自己的训练数据标签文件
+ anno_path: train.json
+ # 修改为你自己的训练数据根目录
+ dataset_dir: /root/publaynet/
+ data_fields: ['image', 'gt_bbox', 'gt_class', 'is_crowd']
+
+EvalDataset:
+ !COCODataSet
+ # 修改为你自己的验证数据目录
+ image_dir: val
+ # 修改为你自己的验证数据标签文件
+ anno_path: val.json
+ # 修改为你自己的验证数据根目录
+ dataset_dir: /root/publaynet/
+
+TestDataset:
+ !ImageFolder
+ # 修改为你自己的测试数据标签文件
+ anno_path: /root/publaynet/val.json
+```
+
+* 开始训练，在训练时，会默认下载PP-PicoDet预训练模型，这里无需预先下载。
+
+```bash
+# GPU训练 支持单卡，多卡训练
+# 训练日志会自动保存到 log 目录中
+
+# 单卡训练
+python3 tools/train.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ --eval
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ --eval
+```
+
+正常启动训练后，会看到以下log输出：
+
+```
+[08/15 04:02:30] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/LCNet_x1_0_pretrained.pdparams
+[08/15 04:02:46] ppdet.engine INFO: Epoch: [0] [ 0/1929] learning_rate: 0.040000 loss_vfl: 1.216707 loss_bbox: 1.142163 loss_dfl: 0.544196 loss: 2.903065 eta: 17 days, 13:50:26 batch_cost: 15.7452 data_cost: 2.9112 ips: 1.5243 images/s
+[08/15 04:03:19] ppdet.engine INFO: Epoch: [0] [ 20/1929] learning_rate: 0.064000 loss_vfl: 1.180627 loss_bbox: 0.939552 loss_dfl: 0.442436 loss: 2.628206 eta: 2 days, 12:18:53 batch_cost: 1.5770 data_cost: 0.0008 ips: 15.2184 images/s
+[08/15 04:03:47] ppdet.engine INFO: Epoch: [0] [ 40/1929] learning_rate: 0.088000 loss_vfl: 0.543321 loss_bbox: 1.071401 loss_dfl: 0.457817 loss: 2.057003 eta: 2 days, 0:07:03 batch_cost: 1.3190 data_cost: 0.0007 ips: 18.1954 images/s
+[08/15 04:04:12] ppdet.engine INFO: Epoch: [0] [ 60/1929] learning_rate: 0.112000 loss_vfl: 0.630989 loss_bbox: 0.859183 loss_dfl: 0.384702 loss: 1.883143 eta: 1 day, 19:01:29 batch_cost: 1.2177 data_cost: 0.0006 ips: 19.7087 images/s
+```
+
+- `--eval`表示训练的同时，进行评估， 评估过程中默认将最佳模型，保存为 `output/picodet_lcnet_x1_0_layout/best_accuracy` 。
+
+**注意，预测/评估时的配置文件请务必与训练一致。**
+
+### 4.2. FGD蒸馏训练
+
+PaddleDetection支持了基于FGD([Focal and Global Knowledge Distillation for Detectors](https://arxiv.org/abs/2111.11837v1))蒸馏的目标检测模型训练过程，FGD蒸馏分为两个部分`Focal`和`Global`。`Focal`蒸馏分离图像的前景和背景，让学生模型分别关注教师模型的前景和背景部分特征的关键像素；`Global`蒸馏部分重建不同像素之间的关系并将其从教师转移到学生，以补偿`Focal`蒸馏中丢失的全局信息。
+
+更换数据集，修改【TODO】配置中的数据配置、类别数，具体可以参考4.1。启动训练：
+
+```bash
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ --slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
+ --eval
+```
-**Envrionment：**
+- `-c`: 指定模型配置文件。
+- `--slim_config`: 指定压缩策略配置文件。
- **CPU：** Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz，24core
+## 5. 模型评估与预测
- **GPU：** a single NVIDIA Tesla P40
+### 5.1. 指标评估
-## 5. Training
+训练中模型参数默认保存在`output/picodet_lcnet_x1_0_layout`目录下。在评估指标时，需要设置`weights`指向保存的参数文件。评估数据集可以通过 `configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 修改`EvalDataset`中的 `image_dir`、`anno_path`和`dataset_dir` 设置。
+
+```bash
+# GPU 评估， weights 为待测权重
+python3 tools/eval.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ -o weigths=./output/picodet_lcnet_x1_0_layout/best_model
+```
+
+会输出以下信息，打印出mAP、AP0.5等信息。
+
+```py
+ Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.935
+ Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.979
+ Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.956
+ Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.404
+ Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.782
+ Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.969
+ Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.539
+ Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.938
+ Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.949
+ Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.495
+ Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.818
+ Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.978
+[08/15 07:07:09] ppdet.engine INFO: Total sample number: 11245, averge FPS: 24.405059207157436
+[08/15 07:07:09] ppdet.engine INFO: Best test bbox ap is 0.935.
+```
+
+使用FGD蒸馏模型进行评估：
+
+```
+python3 tools/eval.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ --slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
+ -o weights=output/picodet_lcnet_x2_5_layout/best_model
+```
+
+- `-c`: 指定模型配置文件。
+- `--slim_config`: 指定蒸馏策略配置文件。
+- `-o weights`: 指定蒸馏算法训好的模型路径。
+
+### 5.2. 测试版面分析结果
+
+
+预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml` 完成了模型的训练过程。
+
+使用 PaddleDetection 训练好的模型，您可以使用如下命令进行中文模型预测。
+
+
+```bash
+python3 tools/infer.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ -o weights='output/picodet_lcnet_x1_0_layout/best_model.pdparams' \
+ --infer_img='docs/images/layout.jpg' \
+ --output_dir=output_dir/ \
+ --draw_threshold=0.4
+```
+
+- `--infer_img`: 推理单张图片，也可以通过`--infer_dir`推理文件中的所有图片。
+- `--output_dir`: 指定可视化结果保存路径。
+- `--draw_threshold`:指定绘制结果框的NMS阈值。
+
+预测图片如下所示，图片会存储在`output_dir`路径中。
+
+使用FGD蒸馏模型进行测试：
+
+```
+python3 tools/infer.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ --slim_config configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x2_5_layout.yml \
+ -o weights='output/picodet_lcnet_x2_5_layout/best_model.pdparams' \
+ --infer_img='docs/images/layout.jpg' \
+ --output_dir=output_dir/ \
+ --draw_threshold=0.4
+```
+
+
+
+## 6. 模型导出与预测
+
+
+### 6.1 模型导出
+
+inference 模型（`paddle.jit.save`保存的模型） 一般是模型训练，把模型结构和模型参数保存在文件中的固化模型，多用于预测部署场景。 训练过程中保存的模型是checkpoints模型，保存的只有模型的参数，多用于恢复训练等。 与checkpoints模型相比，inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
+
+版面分析模型转inference模型步骤如下：
+
+```bash
+python3 tools/export_model.py \
+ -c configs/picodet/legacy_model/application/layout_detection/picodet_lcnet_x1_0_layout.yml \
+ -o weights=output/picodet_lcnet_x1_0_layout/best_model \
+ --output_dir=output_inference/
+```
+
+* 如无需导出后处理，请指定：`-o export.benchmark=True`（如果-o已出现过，此处删掉-o）
+* 如无需导出NMS，请指定：`-o export.nms=False`
+
+转换成功后，在目录下有三个文件：
+
+```
+output_inference/picodet_lcnet_x1_0_layout/
+ ├── model.pdiparams # inference模型的参数文件
+ ├── model.pdiparams.info # inference模型的参数信息，可忽略
+ └── model.pdmodel # inference模型的模型结构文件
+```
+
+FGD蒸馏模型转inference模型步骤如下：
+
+```bash
+python3 tools/export_model.py \
+ -c configs/picodet/legacy_model/application/publayernet_lcnet_x1_5/picodet_student.yml \
+ --slim_config configs/picodet/legacy_model/application/publayernet_lcnet_x1_5/picodet_teacher.yml \
+ -o weights=./output/picodet_lcnet_x2_5_layout/best_model \
+ --output_dir=output_inference/
+```
+
+
+
+### 6.2 模型推理
+
+版面恢复任务进行推理，可以执行如下命令：
+
+```bash
+python3 deploy/python/infer.py \
+ --model_dir=output_inference/picodet_lcnet_x1_0_layout/ \
+ --image_file=docs/images/layout.jpg \
+ --device=CPU
+```
+
+- --device：指定GPU、CPU设备
+
+模型推理完成，会看到以下log输出
+
+```
+------------------------------------------
+----------- Model Configuration -----------
+Model Arch: PicoDet
+Transform Order:
+--transform op: Resize
+--transform op: NormalizeImage
+--transform op: Permute
+--transform op: PadStride
+--------------------------------------------
+class_id:0, confidence:0.9921, left_top:[20.18,35.66],right_bottom:[341.58,600.99]
+class_id:0, confidence:0.9914, left_top:[19.77,611.42],right_bottom:[341.48,901.82]
+class_id:0, confidence:0.9904, left_top:[369.36,375.10],right_bottom:[691.29,600.59]
+class_id:0, confidence:0.9835, left_top:[369.60,608.60],right_bottom:[691.38,736.72]
+class_id:0, confidence:0.9830, left_top:[369.58,805.38],right_bottom:[690.97,901.80]
+class_id:0, confidence:0.9716, left_top:[383.68,271.44],right_bottom:[688.93,335.39]
+class_id:0, confidence:0.9452, left_top:[370.82,34.48],right_bottom:[688.10,63.54]
+class_id:1, confidence:0.8712, left_top:[370.84,771.03],right_bottom:[519.30,789.13]
+class_id:3, confidence:0.9856, left_top:[371.28,67.85],right_bottom:[685.73,267.72]
+save result to: output/layout.jpg
+Test iter 0
+------------------ Inference Time Info ----------------------
+total_time(ms): 2196.0, img_num: 1
+average latency time(ms): 2196.00, QPS: 0.455373
+preprocess_time(ms): 2172.50, inference_time(ms): 11.90, postprocess_time(ms): 11.60
+```
+
+- Model：模型结构
+- Transform Order：预处理操作
+- class_id、confidence、left_top、right_bottom：分别表示类别id、置信度、左上角坐标、右下角坐标
+- save result to：可视化版面分析结果保存路径，默认保存到`./output`文件夹
+- Inference Time Info：推理时间，其中preprocess_time表示预处理耗时，inference_time表示模型预测耗时，postprocess_time表示后处理耗时
+
+可视化版面结果如下图所示
+
+
-
+ 
+
+
+
+
+## Citations
+
+```
+@inproceedings{zhong2019publaynet,
+ title={PubLayNet: largest dataset ever for document layout analysis},
+ author={Zhong, Xu and Tang, Jianbin and Yepes, Antonio Jimeno},
+ booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)},
+ year={2019},
+ volume={},
+ number={},
+ pages={1015-1022},
+ doi={10.1109/ICDAR.2019.00166},
+ ISSN={1520-5363},
+ month={Sep.},
+ organization={IEEE}
+}
+
+@inproceedings{yang2022focal,
+ title={Focal and global knowledge distillation for detectors},
+ author={Yang, Zhendong and Li, Zhe and Jiang, Xiaohu and Gong, Yuan and Yuan, Zehuan and Zhao, Danpei and Yuan, Chun},
+ booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+ pages={4643--4652},
+ year={2022}
+}
+```
-The above model is based on [PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection). If you want to train your own layout parser model，please refer to：[train_layoutparser_model](train_layoutparser_model.md)
diff --git a/ppstructure/layout/README_ch.md b/ppstructure/layout/README_ch.md
deleted file mode 100644
index 69419ad1eee3523d498b0d845a72133b619b3787..0000000000000000000000000000000000000000
--- a/ppstructure/layout/README_ch.md
+++ /dev/null
@@ -1,133 +0,0 @@
-[English](README.md) | 简体中文
-
-# 版面分析使用说明
-
-- [1. 安装whl包](#1)
-- [2. 使用](#2)
-- [3. 后处理](#3)
-- [4. 指标](#4)
-- [5. 训练版面分析模型](#5)
-
-
-
-## 1. 安装whl包
-```bash
-pip install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
-```
-
-
-## 2. 使用
-
-使用layoutparser识别给定文档的布局：
-
-```python
-import cv2
-import layoutparser as lp
-image = cv2.imread("ppstructure/docs/table/layout.jpg")
-image = image[..., ::-1]
-
-# 加载模型
-model = lp.PaddleDetectionLayoutModel(config_path="lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config",
- threshold=0.5,
- label_map={0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"},
- enforce_cpu=False,
- enable_mkldnn=True)
-# 检测
-layout = model.detect(image)
-
-# 显示结果
-show_img = lp.draw_box(image, layout, box_width=3, show_element_type=True)
-show_img.show()
-```
-
-下图展示了结果，不同颜色的检测框表示不同的类别，并通过`show_element_type`在框的左上角显示具体类别：
-
-
+
-
-
-
-`PaddleDetectionLayoutModel`函数参数说明如下:
-
-| 参数 | 含义 | 默认值 | 备注 |
-| :------------: | :-------------------------: | :---------: | :----------------------------------------------------------: |
-| config_path | 模型配置路径 | None | 指定config_path会自动下载模型(仅第一次，之后模型存在，不会再下载) |
-| model_path | 模型路径 | None | 本地模型路径，config_path和model_path必须设置一个，不能同时为None |
-| threshold | 预测得分的阈值 | 0.5 | \ |
-| input_shape | reshape之后图片尺寸 | [3,640,640] | \ |
-| batch_size | 测试batch size | 1 | \ |
-| label_map | 类别映射表 | None | 设置config_path时，可以为None，根据数据集名称自动获取label_map,设置model_path时需要手动指定 |
-| enforce_cpu | 代码是否使用CPU运行 | False | 设置为False表示使用GPU，True表示强制使用CPU |
-| enforce_mkldnn | CPU预测中是否开启MKLDNN加速 | True | \ |
-| thread_num | 设置CPU线程数 | 10 | \ |
-
-目前支持以下几种模型配置和label map，您可以通过修改 `--config_path`和 `--label_map`使用这些模型，从而检测不同类型的内容：
-
-| dataset | config_path | label_map |
-| ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------- |
-| [TableBank](https://doc-analysis.github.io/tablebank-page/index.html) word | lp://TableBank/ppyolov2_r50vd_dcn_365e_tableBank_word/config | {0:"Table"} |
-| TableBank latex | lp://TableBank/ppyolov2_r50vd_dcn_365e_tableBank_latex/config | {0:"Table"} |
-| [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) | lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config | {0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"} |
-
-* TableBank word和TableBank latex分别在word文档、latex文档数据集训练；
-* 下载的TableBank数据集里同时包含word和latex。
-
-
-## 3. 后处理
-
-版面分析检测包含多个类别，如果只想获取指定类别(如"Text"类别)的检测框、可以使用下述代码：
-
-```python
-# 接上面代码
-# 首先过滤特定文本类型的区域
-text_blocks = lp.Layout([b for b in layout if b.type=='Text'])
-figure_blocks = lp.Layout([b for b in layout if b.type=='Figure'])
-
-# 因为在图像区域内可能检测到文本区域，所以只需要删除它们
-text_blocks = lp.Layout([b for b in text_blocks \
- if not any(b.is_in(b_fig) for b_fig in figure_blocks)])
-
-# 对文本区域排序并分配id
-h, w = image.shape[:2]
-
-left_interval = lp.Interval(0, w/2*1.05, axis='x').put_on_canvas(image)
-
-left_blocks = text_blocks.filter_by(left_interval, center=True)
-left_blocks.sort(key = lambda b:b.coordinates[1])
-
-right_blocks = [b for b in text_blocks if b not in left_blocks]
-right_blocks.sort(key = lambda b:b.coordinates[1])
-
-# 最终合并两个列表，并按顺序添加索引
-text_blocks = lp.Layout([b.set(id = idx) for idx, b in enumerate(left_blocks + right_blocks)])
-
-# 显示结果
-show_img = lp.draw_box(image, text_blocks,
- box_width=3,
- show_element_id=True)
-show_img.show()
-```
-
-显示只有"Text"类别的结果：
-
-
-
-
-
-
-
-## 4. 指标
-
-| Dataset | mAP | CPU time cost | GPU time cost |
-| --------- | ---- | ------------- | ------------- |
-| PubLayNet | 93.6 | 1713.7ms | 66.6ms |
-| TableBank | 96.2 | 1968.4ms | 65.1ms |
-
-**Envrionment：**
-
- **CPU：** Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz，24core
-
- **GPU：** a single NVIDIA Tesla P40
-
-
-## 5. 训练版面分析模型
-
-上述模型基于[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection) 训练，如果您想训练自己的版面分析模型，请参考：[train_layoutparser_model](train_layoutparser_model_ch.md)
diff --git a/ppstructure/layout/train_layoutparser_model.md b/ppstructure/layout/train_layoutparser_model.md
deleted file mode 100644
index e877c9c0c901e8be8299101daa5ce6248de0a1dc..0000000000000000000000000000000000000000
--- a/ppstructure/layout/train_layoutparser_model.md
+++ /dev/null
@@ -1,174 +0,0 @@
-English | [简体中文](train_layoutparser_model_ch.md)
-- [Training layout-parse](#training-layout-parse)
- - [1. Installation](#1--installation)
- - [1.1 Requirements](#11-requirements)
- - [1.2 Install PaddleDetection](#12-install-paddledetection)
- - [2. Data preparation](#2-data-preparation)
- - [3. Configuration](#3-configuration)
- - [4. Training](#4-training)
- - [5. Prediction](#5-prediction)
- - [6. Deployment](#6-deployment)
- - [6.1 Export model](#61-export-model)
- - [6.2 Inference](#62-inference)
-
-# Training layout-parse
-
-## 1. Installation
-
-### 1.1 Requirements
-
-- PaddlePaddle 2.1
-- OS 64 bit
-- Python 3(3.5.1+/3.6/3.7/3.8/3.9)，64 bit
-- pip/pip3(9.0.1+), 64 bit
-- CUDA >= 10.1
-- cuDNN >= 7.6
-
-### 1.2 Install PaddleDetection
-
-```bash
-# Clone PaddleDetection repository
-cd
-混合精度 | - | - | | SAR |rec_r31_sar | 识别 | 支持 | 多机多卡
混合精度 | - | - | | SPIN |rec_r32_gaspin_bilstm_att | 识别 | 支持 | 多机多卡
混合精度 | - | - | +| RobustScanner |rec_r31_robustscanner | 识别 | 支持 | 多机多卡
混合精度 | - | - | | PGNet |rec_r34_vd_none_none_ctc_v2.0 | 端到端| 支持 | 多机多卡
混合精度 | - | - | | TableMaster |table_structure_tablemaster_train | 表格识别| 支持 | 多机多卡
混合精度 | - | - | diff --git a/test_tipc/test_inference_cpp.sh b/test_tipc/test_inference_cpp.sh index c0c7c18a38a46b00c839757e303049135a508691..aadaa8b0773632885138806861fc851ede503f3d 100644 --- a/test_tipc/test_inference_cpp.sh +++ b/test_tipc/test_inference_cpp.sh @@ -84,7 +84,7 @@ function func_cpp_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done done @@ -117,7 +117,7 @@ function func_cpp_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done diff --git a/test_tipc/test_inference_python.sh b/test_tipc/test_inference_python.sh index 2a31a468f0d54d1979e82c8f0da98cac6f4edcec..e9908df1f6049f9d38524dc6598499ddd2b58af8 100644 --- a/test_tipc/test_inference_python.sh +++ b/test_tipc/test_inference_python.sh @@ -88,7 +88,7 @@ function func_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done done @@ -119,7 +119,7 @@ function func_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done @@ -146,14 +146,15 @@ if [ ${MODE} = "whole_infer" ]; then for infer_model in ${infer_model_dir_list[*]}; do # run export if [ ${infer_run_exports[Count]} != "null" ];then + _save_log_path="${_log_path}/python_infer_gpu_usetrt_${use_trt}_precision_${precision}_batchsize_${batch_size}_infermodel_${infer_model}.log" save_infer_dir=$(dirname $infer_model) set_export_weight=$(func_set_params "${export_weight}" "${infer_model}") set_save_infer_key=$(func_set_params "${save_infer_key}" "${save_infer_dir}") - export_cmd="${python} ${infer_run_exports[Count]} ${set_export_weight} ${set_save_infer_key}" + export_cmd="${python} ${infer_run_exports[Count]} ${set_export_weight} ${set_save_infer_key} > ${_save_log_path} 2>&1 " echo ${infer_run_exports[Count]} eval $export_cmd status_export=$? - status_check $status_export "${export_cmd}" "${status_log}" "${model_name}" + status_check $status_export "${export_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" else save_infer_dir=${infer_model} fi diff --git a/test_tipc/test_paddle2onnx.sh b/test_tipc/test_paddle2onnx.sh index 78d79d0b8eaac782f98c1e883d091a001443f41a..bace6b2d4684e0ad40ffbd76b37a78ddf1e70722 100644 --- a/test_tipc/test_paddle2onnx.sh +++ b/test_tipc/test_paddle2onnx.sh @@ -66,7 +66,7 @@ function func_paddle2onnx(){ trans_model_cmd="${padlle2onnx_cmd} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_save_model} ${set_opset_version} ${set_enable_onnx_checker} > ${trans_det_log} 2>&1 " eval $trans_model_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" "${trans_det_log}" # trans rec set_dirname=$(func_set_params "--model_dir" "${rec_infer_model_dir_value}") set_model_filename=$(func_set_params "${model_filename_key}" "${model_filename_value}") @@ -78,7 +78,7 @@ function func_paddle2onnx(){ trans_model_cmd="${padlle2onnx_cmd} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_save_model} ${set_opset_version} ${set_enable_onnx_checker} > ${trans_rec_log} 2>&1 " eval $trans_model_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" "${trans_rec_log}" elif [[ ${model_name} =~ "det" ]]; then # trans det set_dirname=$(func_set_params "--model_dir" "${det_infer_model_dir_value}") @@ -91,7 +91,7 @@ function func_paddle2onnx(){ trans_model_cmd="${padlle2onnx_cmd} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_save_model} ${set_opset_version} ${set_enable_onnx_checker} > ${trans_det_log} 2>&1 " eval $trans_model_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" "${trans_det_log}" elif [[ ${model_name} =~ "rec" ]]; then # trans rec set_dirname=$(func_set_params "--model_dir" "${rec_infer_model_dir_value}") @@ -104,7 +104,7 @@ function func_paddle2onnx(){ trans_model_cmd="${padlle2onnx_cmd} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_save_model} ${set_opset_version} ${set_enable_onnx_checker} > ${trans_rec_log} 2>&1 " eval $trans_model_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" "${trans_rec_log}" fi # python inference @@ -127,7 +127,7 @@ function func_paddle2onnx(){ eval $infer_model_cmd last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${infer_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${infer_model_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" elif [ ${use_gpu} = "True" ] || [ ${use_gpu} = "gpu" ]; then _save_log_path="${LOG_PATH}/paddle2onnx_infer_gpu.log" set_gpu=$(func_set_params "${use_gpu_key}" "${use_gpu}") @@ -146,7 +146,7 @@ function func_paddle2onnx(){ eval $infer_model_cmd last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${infer_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${infer_model_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" else echo "Does not support hardware other than CPU and GPU Currently!" fi @@ -158,4 +158,4 @@ echo "################### run test ###################" export Count=0 IFS="|" -func_paddle2onnx \ No newline at end of file +func_paddle2onnx diff --git a/test_tipc/test_ptq_inference_python.sh b/test_tipc/test_ptq_inference_python.sh index e2939fd5e638ad0f6b4c44422a6fec6459903d1c..caf3d506029ee066aa5abebc25b739439b6e9d75 100644 --- a/test_tipc/test_ptq_inference_python.sh +++ b/test_tipc/test_ptq_inference_python.sh @@ -84,7 +84,7 @@ function func_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done done @@ -109,7 +109,7 @@ function func_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done @@ -145,7 +145,7 @@ if [ ${MODE} = "whole_infer" ]; then echo $export_cmd eval $export_cmd status_export=$? - status_check $status_export "${export_cmd}" "${status_log}" "${model_name}" + status_check $status_export "${export_cmd}" "${status_log}" "${model_name}" "${export_log_path}" else save_infer_dir=${infer_model} fi diff --git a/test_tipc/test_serving_infer_cpp.sh b/test_tipc/test_serving_infer_cpp.sh index 0be6a45adf3105f088a96336dddfbe9ac612f19b..10ddecf3fa26805fef7bc6ae10d78ee5e741cd27 100644 --- a/test_tipc/test_serving_infer_cpp.sh +++ b/test_tipc/test_serving_infer_cpp.sh @@ -83,7 +83,7 @@ function func_serving(){ trans_model_cmd="${python_list[0]} ${trans_model_py} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_serving_server} ${set_serving_client} > ${trans_rec_log} 2>&1 " eval $trans_model_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${trans_model_cmd}" "${status_log}" "${model_name}" "${trans_rec_log}" set_image_dir=$(func_set_params "${image_dir_key}" "${image_dir_value}") python_list=(${python_list}) cd ${serving_dir_value} @@ -95,14 +95,14 @@ function func_serving(){ web_service_cpp_cmd="nohup ${python_list[0]} ${web_service_py} --model ${det_server_value} ${rec_server_value} ${op_key} ${op_value} ${port_key} ${port_value} > ${server_log_path} 2>&1 &" eval $web_service_cpp_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cpp_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cpp_cmd}" "${status_log}" "${model_name}" "${server_log_path}" sleep 5s _save_log_path="${LOG_PATH}/cpp_client_cpu.log" cpp_client_cmd="${python_list[0]} ${cpp_client_py} ${det_client_value} ${rec_client_value} > ${_save_log_path} 2>&1" eval $cpp_client_cmd last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${cpp_client_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${cpp_client_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" ps ux | grep -i ${port_value} | awk '{print $2}' | xargs kill -s 9 else server_log_path="${LOG_PATH}/cpp_server_gpu.log" @@ -114,7 +114,7 @@ function func_serving(){ eval $cpp_client_cmd last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${cpp_client_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${cpp_client_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" ps ux | grep -i ${port_value} | awk '{print $2}' | xargs kill -s 9 fi done diff --git a/test_tipc/test_serving_infer_python.sh b/test_tipc/test_serving_infer_python.sh index 4b7dfcf785a3c8459cce95d55744dbcd4f97027a..c7d305d5d2dcd2ea1bf5a7c3254eea4231d59879 100644 --- a/test_tipc/test_serving_infer_python.sh +++ b/test_tipc/test_serving_infer_python.sh @@ -126,19 +126,19 @@ function func_serving(){ web_service_cmd="nohup ${python} ${web_service_py} ${web_use_gpu_key}="" ${web_use_mkldnn_key}=${use_mkldnn} ${set_cpu_threads} ${set_det_model_config} ${set_rec_model_config} > ${server_log_path} 2>&1 &" eval $web_service_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" "${server_log_path}" elif [[ ${model_name} =~ "det" ]]; then set_det_model_config=$(func_set_params "${det_server_key}" "${det_server_value}") web_service_cmd="nohup ${python} ${web_service_py} ${web_use_gpu_key}="" ${web_use_mkldnn_key}=${use_mkldnn} ${set_cpu_threads} ${set_det_model_config} > ${server_log_path} 2>&1 &" eval $web_service_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" "${server_log_path}" elif [[ ${model_name} =~ "rec" ]]; then set_rec_model_config=$(func_set_params "${rec_server_key}" "${rec_server_value}") web_service_cmd="nohup ${python} ${web_service_py} ${web_use_gpu_key}="" ${web_use_mkldnn_key}=${use_mkldnn} ${set_cpu_threads} ${set_rec_model_config} > ${server_log_path} 2>&1 &" eval $web_service_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" "${server_log_path}" fi sleep 2s for pipeline in ${pipeline_py[*]}; do @@ -147,7 +147,7 @@ function func_serving(){ eval $pipeline_cmd last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${pipeline_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${pipeline_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" sleep 2s done ps ux | grep -E 'web_service' | awk '{print $2}' | xargs kill -s 9 @@ -177,19 +177,19 @@ function func_serving(){ web_service_cmd="nohup ${python} ${web_service_py} ${set_tensorrt} ${set_precision} ${set_det_model_config} ${set_rec_model_config} > ${server_log_path} 2>&1 &" eval $web_service_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" "${server_log_path}" elif [[ ${model_name} =~ "det" ]]; then set_det_model_config=$(func_set_params "${det_server_key}" "${det_server_value}") web_service_cmd="nohup ${python} ${web_service_py} ${set_tensorrt} ${set_precision} ${set_det_model_config} > ${server_log_path} 2>&1 &" eval $web_service_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" "${server_log_path}" elif [[ ${model_name} =~ "rec" ]]; then set_rec_model_config=$(func_set_params "${rec_server_key}" "${rec_server_value}") web_service_cmd="nohup ${python} ${web_service_py} ${set_tensorrt} ${set_precision} ${set_rec_model_config} > ${server_log_path} 2>&1 &" eval $web_service_cmd last_status=${PIPESTATUS[0]} - status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${web_service_cmd}" "${status_log}" "${model_name}" "${server_log_path}" fi sleep 2s for pipeline in ${pipeline_py[*]}; do @@ -198,7 +198,7 @@ function func_serving(){ eval $pipeline_cmd last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${pipeline_cmd}" "${status_log}" "${model_name}" + status_check $last_status "${pipeline_cmd}" "${status_log}" "${model_name}" "${_save_log_path}" sleep 2s done ps ux | grep -E 'web_service' | awk '{print $2}' | xargs kill -s 9 diff --git a/test_tipc/test_train_inference_python.sh b/test_tipc/test_train_inference_python.sh index 545cdbba2051c8123ef7f70f2aeb4b4b5a57b7c5..e182fa57f060c81af012a5da89b892bde02b4a2b 100644 --- a/test_tipc/test_train_inference_python.sh +++ b/test_tipc/test_train_inference_python.sh @@ -133,7 +133,7 @@ function func_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done done @@ -164,7 +164,7 @@ function func_inference(){ eval $command last_status=${PIPESTATUS[0]} eval "cat ${_save_log_path}" - status_check $last_status "${command}" "${status_log}" "${model_name}" + status_check $last_status "${command}" "${status_log}" "${model_name}" "${_save_log_path}" done done @@ -201,7 +201,7 @@ if [ ${MODE} = "whole_infer" ]; then echo $export_cmd eval $export_cmd status_export=$? - status_check $status_export "${export_cmd}" "${status_log}" "${model_name}" + status_check $status_export "${export_cmd}" "${status_log}" "${model_name}" "${export_log_path}" else save_infer_dir=${infer_model} fi @@ -298,7 +298,7 @@ else # run train eval $cmd eval "cat ${save_log}/train.log >> ${save_log}.log" - status_check $? "${cmd}" "${status_log}" "${model_name}" + status_check $? "${cmd}" "${status_log}" "${model_name}" "${save_log}.log" set_eval_pretrain=$(func_set_params "${pretrain_model_key}" "${save_log}/${train_model_name}") @@ -309,7 +309,7 @@ else eval_log_path="${LOG_PATH}/${trainer}_gpus_${gpu}_autocast_${autocast}_nodes_${nodes}_eval.log" eval_cmd="${python} ${eval_py} ${set_eval_pretrain} ${set_use_gpu} ${set_eval_params1} > ${eval_log_path} 2>&1 " eval $eval_cmd - status_check $? "${eval_cmd}" "${status_log}" "${model_name}" + status_check $? "${eval_cmd}" "${status_log}" "${model_name}" "${eval_log_path}" fi # run export model if [ ${run_export} != "null" ]; then @@ -320,7 +320,7 @@ else set_save_infer_key=$(func_set_params "${save_infer_key}" "${save_infer_path}") export_cmd="${python} ${run_export} ${set_export_weight} ${set_save_infer_key} > ${export_log_path} 2>&1 " eval $export_cmd - status_check $? "${export_cmd}" "${status_log}" "${model_name}" + status_check $? "${export_cmd}" "${status_log}" "${model_name}" "${export_log_path}" #run inference eval $env diff --git a/tools/eval.py b/tools/eval.py index 2fc53488efa2c4c475d31af47f69b3560e6cc69a..38d72d178db45a4787ddc09c865afba9222f385a 100755 --- a/tools/eval.py +++ b/tools/eval.py @@ -73,7 +73,7 @@ def main(): config['Architecture']["Head"]['out_channels'] = char_num model = build_model(config['Architecture']) - extra_input_models = ["SRN", "NRTR", "SAR", "SEED", "SVTR", "VisionLAN"] + extra_input_models = ["SRN", "NRTR", "SAR", "SEED", "SVTR", "VisionLAN", "RobustScanner"] extra_input = False if config['Architecture']['algorithm'] == 'Distillation': for key in config['Architecture']["Models"]: diff --git a/tools/export_model.py b/tools/export_model.py index c6763374a634dfca125f64b63d7c85716f68f142..193988cc1b62a6c4536a8d2ec640e3e5fc81a79c 100755 --- a/tools/export_model.py +++ b/tools/export_model.py @@ -58,6 +58,8 @@ def export_single_model(model, other_shape = [ paddle.static.InputSpec( shape=[None, 3, 48, 160], dtype="float32"), + [paddle.static.InputSpec( + shape=[None], dtype="float32")] ] model = to_static(model, input_spec=other_shape) elif arch_config["algorithm"] == "SVTR": @@ -109,6 +111,22 @@ def export_single_model(model, shape=[None, 3, 64, 256], dtype="float32"), ] model = to_static(model, input_spec=other_shape) + elif arch_config["algorithm"] == "RobustScanner": + max_text_length = arch_config["Head"]["max_text_length"] + other_shape = [ + paddle.static.InputSpec( + shape=[None, 3, 48, 160], dtype="float32"), + + [ + paddle.static.InputSpec( + shape=[None, ], + dtype="float32"), + paddle.static.InputSpec( + shape=[None, max_text_length], + dtype="int64") + ] + ] + model = to_static(model, input_spec=other_shape) elif arch_config["algorithm"] in ["LayoutLM", "LayoutLMv2", "LayoutXLM"]: input_spec = [ paddle.static.InputSpec( @@ -128,7 +146,7 @@ def export_single_model(model, else: infer_shape = [3, -1, -1] if arch_config["model_type"] == "rec": - infer_shape = [3, 48, -1] # for rec model, H must be 32 + infer_shape = [3, 32, -1] # for rec model, H must be 32 if "Transform" in arch_config and arch_config[ "Transform"] is not None and arch_config["Transform"][ "name"] == "TPS": @@ -234,4 +252,4 @@ def main(): if __name__ == "__main__": - main() \ No newline at end of file + main() diff --git a/tools/infer/predict_rec.py b/tools/infer/predict_rec.py index 449f69ed6a22cb12af8a6a2ef8f2eedc1aca087c..53dab6f26d8b84a224360f2fa6fe5f411eea751f 100755 --- a/tools/infer/predict_rec.py +++ b/tools/infer/predict_rec.py @@ -68,7 +68,7 @@ class TextRecognizer(object): 'name': 'SARLabelDecode', "character_dict_path": args.rec_char_dict_path, "use_space_char": args.use_space_char - } + } elif self.rec_algorithm == "VisionLAN": postprocess_params = { 'name': 'VLLabelDecode', @@ -93,6 +93,13 @@ class TextRecognizer(object): "character_dict_path": args.rec_char_dict_path, "use_space_char": args.use_space_char } + elif self.rec_algorithm == "RobustScanner": + postprocess_params = { + 'name': 'SARLabelDecode', + "character_dict_path": args.rec_char_dict_path, + "use_space_char": args.use_space_char, + "rm_symbol": True + } self.postprocess_op = build_post_process(postprocess_params) self.predictor, self.input_tensor, self.output_tensors, self.config = \ utility.create_predictor(args, 'rec', logger) @@ -390,6 +397,18 @@ class TextRecognizer(object): img_list[indices[ino]], self.rec_image_shape) norm_img = norm_img[np.newaxis, :] norm_img_batch.append(norm_img) + elif self.rec_algorithm == "RobustScanner": + norm_img, _, _, valid_ratio = self.resize_norm_img_sar( + img_list[indices[ino]], self.rec_image_shape, width_downsample_ratio=0.25) + norm_img = norm_img[np.newaxis, :] + valid_ratio = np.expand_dims(valid_ratio, axis=0) + valid_ratios = [] + valid_ratios.append(valid_ratio) + norm_img_batch.append(norm_img) + word_positions_list = [] + word_positions = np.array(range(0, 40)).astype('int64') + word_positions = np.expand_dims(word_positions, axis=0) + word_positions_list.append(word_positions) else: norm_img = self.resize_norm_img(img_list[indices[ino]], max_wh_ratio) @@ -437,10 +456,40 @@ class TextRecognizer(object): preds = {"predict": outputs[2]} elif self.rec_algorithm == "SAR": valid_ratios = np.concatenate(valid_ratios) + inputs = [ + norm_img_batch, + np.array( + [valid_ratios], dtype=np.float32), + ] + if self.use_onnx: + input_dict = {} + input_dict[self.input_tensor.name] = norm_img_batch + outputs = self.predictor.run(self.output_tensors, + input_dict) + preds = outputs[0] + else: + input_names = self.predictor.get_input_names() + for i in range(len(input_names)): + input_tensor = self.predictor.get_input_handle( + input_names[i]) + input_tensor.copy_from_cpu(inputs[i]) + self.predictor.run() + outputs = [] + for output_tensor in self.output_tensors: + output = output_tensor.copy_to_cpu() + outputs.append(output) + if self.benchmark: + self.autolog.times.stamp() + preds = outputs[0] + elif self.rec_algorithm == "RobustScanner": + valid_ratios = np.concatenate(valid_ratios) + word_positions_list = np.concatenate(word_positions_list) inputs = [ norm_img_batch, valid_ratios, + word_positions_list ] + if self.use_onnx: input_dict = {} input_dict[self.input_tensor.name] = norm_img_batch diff --git a/tools/infer/utility.py b/tools/infer/utility.py index 81d0196ccd6b86741e73524d9321618f3f5cc34b..1eebc73f31e6b48a473c20d907ca401ad919fe0b 100644 --- a/tools/infer/utility.py +++ b/tools/infer/utility.py @@ -231,89 +231,10 @@ def create_predictor(args, mode, logger): ) config.enable_tuned_tensorrt_dynamic_shape( args.shape_info_filename, True) - - use_dynamic_shape = True - if mode == "det": - min_input_shape = { - "x": [1, 3, 50, 50], - "conv2d_92.tmp_0": [1, 120, 20, 20], - "conv2d_91.tmp_0": [1, 24, 10, 10], - "conv2d_59.tmp_0": [1, 96, 20, 20], - "nearest_interp_v2_1.tmp_0": [1, 256, 10, 10], - "nearest_interp_v2_2.tmp_0": [1, 256, 20, 20], - "conv2d_124.tmp_0": [1, 256, 20, 20], - "nearest_interp_v2_3.tmp_0": [1, 64, 20, 20], - "nearest_interp_v2_4.tmp_0": [1, 64, 20, 20], - "nearest_interp_v2_5.tmp_0": [1, 64, 20, 20], - "elementwise_add_7": [1, 56, 2, 2], - "nearest_interp_v2_0.tmp_0": [1, 256, 2, 2] - } - max_input_shape = { - "x": [1, 3, 1536, 1536], - "conv2d_92.tmp_0": [1, 120, 400, 400], - "conv2d_91.tmp_0": [1, 24, 200, 200], - "conv2d_59.tmp_0": [1, 96, 400, 400], - "nearest_interp_v2_1.tmp_0": [1, 256, 200, 200], - "conv2d_124.tmp_0": [1, 256, 400, 400], - "nearest_interp_v2_2.tmp_0": [1, 256, 400, 400], - "nearest_interp_v2_3.tmp_0": [1, 64, 400, 400], - "nearest_interp_v2_4.tmp_0": [1, 64, 400, 400], - "nearest_interp_v2_5.tmp_0": [1, 64, 400, 400], - "elementwise_add_7": [1, 56, 400, 400], - "nearest_interp_v2_0.tmp_0": [1, 256, 400, 400] - } - opt_input_shape = { - "x": [1, 3, 640, 640], - "conv2d_92.tmp_0": [1, 120, 160, 160], - "conv2d_91.tmp_0": [1, 24, 80, 80], - "conv2d_59.tmp_0": [1, 96, 160, 160], - "nearest_interp_v2_1.tmp_0": [1, 256, 80, 80], - "nearest_interp_v2_2.tmp_0": [1, 256, 160, 160], - "conv2d_124.tmp_0": [1, 256, 160, 160], - "nearest_interp_v2_3.tmp_0": [1, 64, 160, 160], - "nearest_interp_v2_4.tmp_0": [1, 64, 160, 160], - "nearest_interp_v2_5.tmp_0": [1, 64, 160, 160], - "elementwise_add_7": [1, 56, 40, 40], - "nearest_interp_v2_0.tmp_0": [1, 256, 40, 40] - } - min_pact_shape = { - "nearest_interp_v2_26.tmp_0": [1, 256, 20, 20], - "nearest_interp_v2_27.tmp_0": [1, 64, 20, 20], - "nearest_interp_v2_28.tmp_0": [1, 64, 20, 20], - "nearest_interp_v2_29.tmp_0": [1, 64, 20, 20] - } - max_pact_shape = { - "nearest_interp_v2_26.tmp_0": [1, 256, 400, 400], - "nearest_interp_v2_27.tmp_0": [1, 64, 400, 400], - "nearest_interp_v2_28.tmp_0": [1, 64, 400, 400], - "nearest_interp_v2_29.tmp_0": [1, 64, 400, 400] - } - opt_pact_shape = { - "nearest_interp_v2_26.tmp_0": [1, 256, 160, 160], - "nearest_interp_v2_27.tmp_0": [1, 64, 160, 160], - "nearest_interp_v2_28.tmp_0": [1, 64, 160, 160], - "nearest_interp_v2_29.tmp_0": [1, 64, 160, 160] - } - min_input_shape.update(min_pact_shape) - max_input_shape.update(max_pact_shape) - opt_input_shape.update(opt_pact_shape) - elif mode == "rec": - if args.rec_algorithm not in ["CRNN", "SVTR_LCNet"]: - use_dynamic_shape = False - imgH = int(args.rec_image_shape.split(',')[-2]) - min_input_shape = {"x": [1, 3, imgH, 10]} - max_input_shape = {"x": [args.rec_batch_num, 3, imgH, 2304]} - opt_input_shape = {"x": [args.rec_batch_num, 3, imgH, 320]} - config.exp_disable_tensorrt_ops(["transpose2"]) - elif mode == "cls": - min_input_shape = {"x": [1, 3, 48, 10]} - max_input_shape = {"x": [args.rec_batch_num, 3, 48, 1024]} - opt_input_shape = {"x": [args.rec_batch_num, 3, 48, 320]} else: - use_dynamic_shape = False - if use_dynamic_shape: - config.set_trt_dynamic_shape_info( - min_input_shape, max_input_shape, opt_input_shape) + logger.info( + f"when using tensorrt, dynamic shape is a suggested option, you can use '--shape_info_filename=shape.txt' for offline dygnamic shape tuning" + ) elif args.use_xpu: config.enable_xpu(10 * 1024 * 1024) diff --git a/tools/infer_rec.py b/tools/infer_rec.py index 182694e6cda12ead0e263bb94a7d6483a6f7f212..14b14544eb11e9fb0a0c2cdf92aff9d7cb4b5ba7 100755 --- a/tools/infer_rec.py +++ b/tools/infer_rec.py @@ -96,6 +96,8 @@ def main(): ] elif config['Architecture']['algorithm'] == "SAR": op[op_name]['keep_keys'] = ['image', 'valid_ratio'] + elif config['Architecture']['algorithm'] == "RobustScanner": + op[op_name]['keep_keys'] = ['image', 'valid_ratio', 'word_positons'] else: op[op_name]['keep_keys'] = ['image'] transforms.append(op) @@ -131,12 +133,20 @@ def main(): if config['Architecture']['algorithm'] == "SAR": valid_ratio = np.expand_dims(batch[-1], axis=0) img_metas = [paddle.to_tensor(valid_ratio)] + if config['Architecture']['algorithm'] == "RobustScanner": + valid_ratio = np.expand_dims(batch[1], axis=0) + word_positons = np.expand_dims(batch[2], axis=0) + img_metas = [paddle.to_tensor(valid_ratio), + paddle.to_tensor(word_positons), + ] images = np.expand_dims(batch[0], axis=0) images = paddle.to_tensor(images) if config['Architecture']['algorithm'] == "SRN": preds = model(images, others) elif config['Architecture']['algorithm'] == "SAR": preds = model(images, img_metas) + elif config['Architecture']['algorithm'] == "RobustScanner": + preds = model(images, img_metas) else: preds = model(images) post_result = post_process_class(preds) diff --git a/tools/infer_sr.py b/tools/infer_sr.py index 0bc2f6aaa7c4400676268ec64d37e721af0f99c2..df4334f3427e57b9062dd819aa16c110fd771e8c 100755 --- a/tools/infer_sr.py +++ b/tools/infer_sr.py @@ -63,14 +63,14 @@ def main(): elif op_name in ['SRResize']: op[op_name]['infer_mode'] = True elif op_name == 'KeepKeys': - op[op_name]['keep_keys'] = ['imge_lr'] + op[op_name]['keep_keys'] = ['img_lr'] transforms.append(op) global_config['infer_mode'] = True ops = create_operators(transforms, global_config) - save_res_path = config['Global'].get('save_res_path', "./infer_result") - if not os.path.exists(os.path.dirname(save_res_path)): - os.makedirs(os.path.dirname(save_res_path)) + save_visual_path = config['Global'].get('save_visual', "infer_result/") + if not os.path.exists(os.path.dirname(save_visual_path)): + os.makedirs(os.path.dirname(save_visual_path)) model.eval() for file in get_image_file_list(config['Global']['infer_img']): @@ -87,7 +87,7 @@ def main(): fm_sr = (sr_img.numpy() * 255).transpose(1, 2, 0).astype(np.uint8) fm_lr = (lr_img.numpy() * 255).transpose(1, 2, 0).astype(np.uint8) img_name_pure = os.path.split(file)[-1] - cv2.imwrite("infer_result/sr_{}".format(img_name_pure), + cv2.imwrite("{}/sr_{}".format(save_visual_path, img_name_pure), fm_sr[:, :, ::-1]) logger.info("The visualized image saved in infer_result/sr_{}".format( img_name_pure)) diff --git a/tools/program.py b/tools/program.py index b450bc5a3abf0be500b42712d72c81c190412f34..5a4d3ea4d2ec6832e6735d15096d46fbb62f86dd 100755 --- a/tools/program.py +++ b/tools/program.py @@ -162,18 +162,18 @@ def to_float32(preds): for k in preds: if isinstance(preds[k], dict) or isinstance(preds[k], list): preds[k] = to_float32(preds[k]) - else: - preds[k] = paddle.to_tensor(preds[k], dtype='float32') + elif isinstance(preds[k], paddle.Tensor): + preds[k] = preds[k].astype(paddle.float32) elif isinstance(preds, list): for k in range(len(preds)): if isinstance(preds[k], dict): preds[k] = to_float32(preds[k]) elif isinstance(preds[k], list): preds[k] = to_float32(preds[k]) - else: - preds[k] = paddle.to_tensor(preds[k], dtype='float32') - else: - preds = paddle.to_tensor(preds, dtype='float32') + elif isinstance(preds[k], paddle.Tensor): + preds[k] = preds[k].astype(paddle.float32) + elif isinstance(preds, paddle.Tensor): + preds = preds.astype(paddle.float32) return preds @@ -190,7 +190,8 @@ def train(config, pre_best_model_dict, logger, log_writer=None, - scaler=None): + scaler=None, + amp_level='O2'): cal_metric_during_train = config['Global'].get('cal_metric_during_train', False) calc_epoch_interval = config['Global'].get('calc_epoch_interval', 1) @@ -230,7 +231,8 @@ def train(config, use_srn = config['Architecture']['algorithm'] == "SRN" extra_input_models = [ - "SRN", "NRTR", "SAR", "SEED", "SVTR", "SPIN", "VisionLAN" + "SRN", "NRTR", "SAR", "SEED", "SVTR", "SPIN", "VisionLAN", + "RobustScanner" ] extra_input = False if config['Architecture']['algorithm'] == 'Distillation': @@ -276,7 +278,8 @@ def train(config, model_average = True # use amp if scaler: - with paddle.amp.auto_cast(level='O2'): + custom_black_list = config['Global'].get('amp_custom_black_list',[]) + with paddle.amp.auto_cast(level=amp_level, custom_black_list=custom_black_list): if model_type == 'table' or extra_input: preds = model(images, data=batch[1:]) elif model_type in ["kie", 'vqa']: @@ -502,18 +505,9 @@ def eval(model, preds = model(batch) sr_img = preds["sr_img"] lr_img = preds["lr_img"] - - for i in (range(sr_img.shape[0])): - fm_sr = (sr_img[i].numpy() * 255).transpose( - 1, 2, 0).astype(np.uint8) - fm_lr = (lr_img[i].numpy() * 255).transpose( - 1, 2, 0).astype(np.uint8) - cv2.imwrite("output/images/{}_{}_sr.jpg".format( - sum_images, i), fm_sr) - cv2.imwrite("output/images/{}_{}_lr.jpg".format( - sum_images, i), fm_lr) else: preds = model(images) + preds = to_float32(preds) else: if model_type == 'table' or extra_input: preds = model(images, data=batch[1:]) @@ -523,16 +517,6 @@ def eval(model, preds = model(batch) sr_img = preds["sr_img"] lr_img = preds["lr_img"] - - for i in (range(sr_img.shape[0])): - fm_sr = (sr_img[i].numpy() * 255).transpose( - 1, 2, 0).astype(np.uint8) - fm_lr = (lr_img[i].numpy() * 255).transpose( - 1, 2, 0).astype(np.uint8) - cv2.imwrite("output/images/{}_{}_sr.jpg".format( - sum_images, i), fm_sr) - cv2.imwrite("output/images/{}_{}_lr.jpg".format( - sum_images, i), fm_lr) else: preds = model(images) @@ -653,7 +637,7 @@ def preprocess(is_train=False): 'CLS', 'PGNet', 'Distillation', 'NRTR', 'TableAttn', 'SAR', 'PSE', 'SEED', 'SDMGR', 'LayoutXLM', 'LayoutLM', 'LayoutLMv2', 'PREN', 'FCE', 'SVTR', 'ViTSTR', 'ABINet', 'DB++', 'TableMaster', 'SPIN', 'VisionLAN', - 'Gestalt', 'SLANet' + 'Gestalt', 'SLANet', 'RobustScanner' ] if use_xpu: diff --git a/tools/train.py b/tools/train.py index 0c881ecae8daf78860829b1419178358c2209f25..5f310938f3ae3488281b47ccdb436697595b5578 100755 --- a/tools/train.py +++ b/tools/train.py @@ -147,6 +147,7 @@ def main(config, device, logger, vdl_writer): len(valid_dataloader))) use_amp = config["Global"].get("use_amp", False) + amp_level = config["Global"].get("amp_level", 'O2') if use_amp: AMP_RELATED_FLAGS_SETTING = { 'FLAGS_cudnn_batchnorm_spatial_persistent': 1, @@ -159,8 +160,9 @@ def main(config, device, logger, vdl_writer): scaler = paddle.amp.GradScaler( init_loss_scaling=scale_loss, use_dynamic_loss_scaling=use_dynamic_loss_scaling) - model, optimizer = paddle.amp.decorate( - models=model, optimizers=optimizer, level='O2', master_weight=True) + if amp_level == "O2": + model, optimizer = paddle.amp.decorate( + models=model, optimizers=optimizer, level=amp_level, master_weight=True) else: scaler = None @@ -169,7 +171,7 @@ def main(config, device, logger, vdl_writer): # start train program.train(config, train_dataloader, valid_dataloader, device, model, loss_class, optimizer, lr_scheduler, post_process_class, - eval_class, pre_best_model_dict, logger, vdl_writer, scaler) + eval_class, pre_best_model_dict, logger, vdl_writer, scaler,amp_level) def test_reader(config, device, logger):