diff --git a/README_en.md b/README_en.md index 705fdb66e28c057c805587b90c08b14c5bc7673e..ea7edb1ed602ebd69344b76e7114f806b658ca47 100644 --- a/README_en.md +++ b/README_en.md @@ -62,6 +62,14 @@ Quick experience of **P**ractical **U**ltra **L**ight-weight image **C**lassific - [PULC Quick Start](docs/en/PULC/PULC_quickstart_en.md) - [PULC Model Zoo](docs/en/PULC/PULC_model_list_en.md) - [PULC Classification Model of Someone or Nobody](docs/en/PULC/PULC_person_exists_en.md) + - [PULC Classification Model of Person Attribute](docs/en/PULC/PULC_person_attribute_en.md) + - [PULC Classification Model of Wearing or Unwearing Safety Helmet](docs/en/PULC/PULC_safety_helmet_en.md) + - [PULC Classification Model of Traffic Sing](docs/en/PULC/PULC_traffic_sign_en.md) + - [PULC Classification Model of Vehicle Attribute](docs/en/PULC/PULC_vehicle_attribute_en.md) + - [PULC Classification Model of Containing or Uncontaining Car](docs/en/PULC/PULC_car_exists_en.md) + - [PULC Classification Model of Text Image Orientation](docs/en/PULC/PULC_text_image_orientation_en.md) + - [PULC Classification Model of Textline Orientation](docs/en/PULC/PULC_textline_orientation_en.md) + - [PULC Classification Model of Language](docs/en/PULC/PULC_language_classification_en.md) - [Quick Start of Recognition](./docs/en/tutorials/quick_start_recognition_en.md) - [Quick Start of Recognition](./docs/en/quick_start/quick_start_recognition_en.md) - [Introduction to Image Recognition Systems](#Introduction_to_Image_Recognition_Systems) diff --git a/docs/en/PULC/PULC_car_exists_en.md b/docs/en/PULC/PULC_car_exists_en.md new file mode 100644 index 0000000000000000000000000000000000000000..7284fbe3414b43ee22ca166d3348851d01126dd3 --- /dev/null +++ b/docs/en/PULC/PULC_car_exists_en.md @@ -0,0 +1,457 @@ +# PULC Classification Model of Containing or Uncontaining Car + +------ + +## Catalogue + +- [1. Introduction](#1) +- [2. Quick Start](#2) + - [2.1 PaddlePaddle Installation](#2.1) + - [2.2 PaddleClas Installation](#2.2) + - [2.3 Prediction](#2.3) +- [3. Training, Evaluation and Inference](#3) + - [3.1 Installation](#3.1) + - [3.2 Dataset](#3.2) + - [3.2.1 Dataset Introduction](#3.2.1) + - [3.2.2 Getting Dataset](#3.2.2) + - [3.3 Training](#3.3) + - [3.4 Evaluation](#3.4) + - [3.5 Inference](#3.5) +- [4. Model Compression](#4) + - [4.1 SKL-UGI Knowledge Distillation](#4.1) + - [4.1.1 Teacher Model Training](#4.1.1) + - [4.1.2 Knowledge Distillation Training](#4.1.2) +- [5. SHAS](#5) +- [6. Inference Deployment](#6) + - [6.1 Getting Paddle Inference Model](#6.1) + - [6.1.1 Exporting Paddle Inference Model](#6.1.1) + - [6.1.2 Downloading Inference Model](#6.1.2) + - [6.2 Prediction with Python](#6.2) + - [6.2.1 Image Prediction](#6.2.1) + - [6.2.2 Images Prediction](#6.2.2) + - [6.3 Deployment with C++](#6.3) + - [6.4 Deployment as Service](#6.4) + - [6.5 Deployment on Mobile](#6.5) + - [6.6 Converting To ONNX and Deployment](#6.6) + + + +## 1. Introduction + +This case provides a way for users to quickly build a lightweight, high-precision and practical classification model of car exists using PaddleClas PULC (Practical Ultra Lightweight Classification). The model can be widely used in monitoring scenarios, massive data filtering scenarios, etc. + +The following table lists the relevant indicators of the model. The first two lines means that using SwinTransformer_tiny and MobileNetV3_small_x0_35 as the backbone to training. The third to sixth lines means that the backbone is replaced by PPLCNet, additional use of EDA strategy and additional use of EDA strategy and SKL-UGI knowledge distillation strategy. + +| Backbone | Tpr(%) | Latency(ms) | Size(M)| Training Strategy | +|-------|----------------|----------|---------------|---------------| +| SwinTranformer_tiny | 97.71 | 95.30 | 107 | using ImageNet pretrained model | +| MobileNetV3_small_x0_35 | 81.23 | 2.85 | 1.6 | using ImageNet pretrained model | +| PPLCNet_x1_0 | 94.72 | 2.12 | 6.5 | using ImageNet pretrained model | +| PPLCNet_x1_0 | 95.48 | 2.12 | 6.5 | using SSLD pretrained model | +| PPLCNet_x1_0 | 95.48 | 2.12 | 6.5 | using SSLD pretrained model + EDA strategy | +| PPLCNet_x1_0 | 95.92 | 2.12 | 6.5 | using SSLD pretrained model + EDA strategy + SKL-UGI knowledge distillation strategy| + +It can be seen that high Tpr can be getted when backbone is SwinTranformer_tiny, but the speed is slow. Replacing backbone with the lightweight model MobileNetV3_small_x0_35, the speed can be greatly improved, but the Tpr will be greatly reduced. Replacing backbone with faster backbone PPLCNet_x1_0, the Tpr is higher more 13 percentage points than MobileNetv3_small_x0_35. At the same time, the speed can be more than 20% faster. After additional using the SSLD pretrained model, the Tpr can be improved by about 0.7 percentage points without affecting the inference speed. Finally, after additional using the SKL-UGI knowledge distillation, the Tpr can be further improved by 0.44 percentage points. At this point, the Tpr close to that of SwinTranformer_tiny, but the speed is more than 40 times faster. The training method and deployment instructions of PULC will be introduced in detail below. + +**Note**: + +* About `Tpr` metric, please refer to [3.2 section](#3.2) for more information . +* The Latency is tested on Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz. The MKLDNN is enabled and the number of threads is 10. +* About PP-LCNet, please refer to [PP-LCNet Introduction](../models/PP-LCNet_en.md) and [PP-LCNet Paper](https://arxiv.org/abs/2109.15099). + + + +## 2. Quick Start + + + +### 2.1 PaddlePaddle Installation + +- Run the following command to install if CUDA9 or CUDA10 is available. + +```bash +python3 -m pip install paddlepaddle-gpu -i https://mirror.baidu.com/pypi/simple +``` + +- Run the following command to install if GPU device is unavailable. + +```bash +python3 -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple +``` + +Please refer to [PaddlePaddle Installation](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/en/install/pip/linux-pip_en.html) for more information about installation, for examples other versions. + + + +### 2.2 PaddleClas wheel Installation + +The command of PaddleClas installation as bellow: + +```bash +pip3 install paddleclas +``` + + + +### 2.3 Prediction + +First, please click [here](https://paddleclas.bj.bcebos.com/data/PULC/pulc_demo_imgs.zip) to download and unzip to get the test demo images. + +* Prediction with CLI + +```bash +paddleclas --model_name=car_exists --infer_imgs=pulc_demo_imgs/car_exists/objects365_00001507.jpeg +``` + +Results: + +``` +>>> result +class_ids: [1], scores: [0.9871138], label_names: ['contains_car'], filename: pulc_demo_imgs/car_exists/objects365_00001507.jpeg +Predict complete! +``` + +**Note**: If you want to test other images, only need to specify the `--infer_imgs` argument, and the directory containing images is also supported. + +* Prediction in Python + +```python +import paddleclas +model = paddleclas.PaddleClas(model_name="car_exists") +result = model.predict(input_data="pulc_demo_imgs/car_exists/objects365_00001507.jpeg") +print(next(result)) +``` + +**Note**: The `result` returned by `model.predict()` is a generator, so you need to use the `next()` function to call it or `for` loop to loop it. And it will predict with `batch_size` size batch and return the prediction results when called. The default `batch_size` is 1, and you also specify the `batch_size` when instantiating, such as `model = paddleclas.PaddleClas(model_name="car_exists", batch_size=2)`. The result of demo above: + +``` +>>> result +[{'class_ids': [1], 'scores': [0.9871138], 'label_names': ['contains_car'], 'filename': 'pulc_demo_imgs/car_exists/objects365_00001507.jpeg'}] +``` + + + +## 3. Training, Evaluation and Inference + + + +### 3.1 Installation + +Please refer to [Installation](../installation/install_paddleclas_en.md) to get the description about installation. + + + +### 3.2 Dataset + + + +#### 3.2.1 Dataset Introduction + +All datasets used in this case are open source data. Train and validation data are the subset of [Object365](https://www.objects365.org/overview.html) data. ImageNet_val is [ImageNet-1k](https://www.image-net.org/) validation data. + + + +#### 3.2.2 Getting Dataset + +The data used in this case can be getted by processing the open source data. The detailed processes are as follows: + +- Training data. This case deals with the annotation file of Objects365 data training data. If a certain image contains the label of "car" and the area of this box is greater than 10% in the whole image, it is considered that the image contains car. If there is no label of any vehicle in a certain image, such as car, bus and so on, it is considered that the image does not contain car. After processing, 108629 images were obtained, including 27422 images containing car and 81207 images uncontaining car. +- Validation data: Same as Training data, but checked manually to remove some labeled wrong images. + +**Note**: the labels of objects365 are not completely mutually exclusive. For example, F1 racing cars may be "F1 formula" or "car". In order to reduce the interference, we only keep the "car" label as containing car, and the figure without any vehicle as uncontaining car. + +Some image of the processed dataset is as follows: + +![](../../images/PULC/docs/car_exists_data_demo.jpeg) + +And you can also download the data processed directly. + +``` +cd path_to_PaddleClas +``` + +Enter the `dataset/` directory, download and unzip the dataset. + +```shell +cd dataset +wget https://paddleclas.bj.bcebos.com/data/PULC/car_exists.tar +tar -xf car_exists.tar +cd ../ +``` + +The datas under `car_exists` directory: + +``` + +├── objects365_car +│   ├── objects365_00000039.jpg +│   ├── objects365_00000099.jpg +├── ImageNet_val +│   ├── ILSVRC2012_val_00000001.JPEG +│   ├── ILSVRC2012_val_00000002.JPEG +... +├── train_list.txt +├── train_list.txt.debug +├── train_list_for_distill.txt +├── val_list.txt +└── val_list.txt.debug +``` + +Where `train/` and `val/` are training set and validation set respectively. The `train_list.txt` and `val_list.txt` are label files of training data and validation data respectively. The file `train_list.txt.debug` and `val_list.txt.debug` are subset of `train_list.txt` and `val_list.txt` respectively. `ImageNet_val/` is the validation data of ImageNet-1k, which will be used for SKL-UGI knowledge distillation, and its label file is `train_list_for_distill.txt`. + +**Note**: + +* About the contents format of `train_list.txt` and `val_list.txt`, please refer to [Description about Classification Dataset in PaddleClas](../data_preparation/classification_dataset_en.md). +* About the `train_list_for_distill.txt`, please refer to [Knowledge Distillation Label](../advanced_tutorials/distillation/distillation_en.md). + + + +### 3.3 Training + +The details of training config in `ppcls/configs/PULC/car_exists/PPLCNet_x1_0.yaml`. The command about training as follows: + +```shell +export CUDA_VISIBLE_DEVICES=0,1,2,3 +python3 -m paddle.distributed.launch \ + --gpus="0,1,2,3" \ + tools/train.py \ + -c ./ppcls/configs/PULC/car_exists/PPLCNet_x1_0.yaml +``` + +The best metric of validation data is between `0.95` and `0.96`. There would be fluctuations because the data size is small. + +**Note**: + +* The metric Tpr, that describe the True Positive Rate when False Positive Rate is less than a certain threshold(1/100 used in this case), is one of the commonly used metric for binary classification. About the details of Fpr and Tpr, please refer [here](https://en.wikipedia.org/wiki/Receiver_operating_characteristic). +* When evaluation, the best metric TprAtFpr will be printed that include `Fpr`, `Tpr` and the current `threshold`. The `Tpr` means the Recall rate under the current `Fpr`. The `Tpr` higher, the model better. The `threshold` would be used in deployment, which means the classification threshold under best `Fpr` metric. + + + +### 3.4 Evaluation + +After training, you can use the following commands to evaluate the model. + +```bash +python3 tools/eval.py \ + -c ./ppcls/configs/PULC/car_exists/PPLCNet_x1_0.yaml \ + -o Global.pretrained_model="output/PPLCNet_x1_0/best_model" +``` + +Among the above command, the argument `-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"` specify the path of the best model weight file. You can specify other path if needed. + + + +### 3.5 Inference + +After training, you can use the model that trained to infer. Command is as follow: + +```python +python3 tools/infer.py \ + -c ./ppcls/configs/PULC/car_exists/PPLCNet_x1_0.yaml \ + -o Global.pretrained_model=output/PPLCNet_x1_0/best_model +``` + +The results: + +``` +[{'class_ids': [1], 'scores': [0.9871138], 'label_names': ['contains_car'], 'filename': 'deploy/images/PULC/car_exists/objects365_00001507.jpeg'}] +``` + +**Note**: + +* Among the above command, argument `-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"` specify the path of the best model weight file. You can specify other path if needed. +* The default test image is `deploy/images/PULC/car_exists/objects365_00001507.jpeg`. And you can test other image, only need to specify the argument `-o Infer.infer_imgs=path_to_test_image`. +* The default threshold is `0.5`. If needed, you can specify the argument `Infer.PostProcess.threshold`, such as: `-o Infer.PostProcess.threshold=0.9794`. And the argument `threshold` is needed to be specified according by specific case. The `0.9794` is the best threshold when `Fpr` is less than `1/100` in this valuation dataset. + + + +## 4. Model Compression + + + +### 4.1 SKL-UGI Knowledge Distillation + +SKL-UGI is a simple but effective knowledge distillation algrithem proposed by PaddleClas. + + + + + + +#### 4.1.1 Teacher Model Training + +Training the teacher model with hyperparameters specified in `ppcls/configs/PULC/car_exists/PPLCNet/PPLCNet_x1_0.yaml`. The command is as follow: + +```shell +export CUDA_VISIBLE_DEVICES=0,1,2,3 +python3 -m paddle.distributed.launch \ + --gpus="0,1,2,3" \ + tools/train.py \ + -c ./ppcls/configs/PULC/car_exists/PPLCNet_x1_0.yaml \ + -o Arch.name=ResNet101_vd +``` + +The best metric of validation data is between `0.96` and `0.98`. The best teacher model weight would be saved in file `output/ResNet101_vd/best_model.pdparams`. + + + +#### 4.1.2 Knowledge Distillation Training + +The training strategy, specified in training config file `ppcls/configs/PULC/car_exists/PPLCNet_x1_0_distillation.yaml`, the teacher model is `ResNet101_vd`, the student model is `PPLCNet_x1_0` and the additional unlabeled training data is validation data of ImageNet1k. The command is as follow: + +```shell +export CUDA_VISIBLE_DEVICES=0,1,2,3 +python3 -m paddle.distributed.launch \ + --gpus="0,1,2,3" \ + tools/train.py \ + -c ./ppcls/configs/PULC/car_exists/PPLCNet_x1_0_distillation.yaml \ + -o Arch.models.0.Teacher.pretrained=output/ResNet101_vd/best_model +``` + +The best metric is between `0.95` and `0.97`. The best student model weight would be saved in file `output/DistillationModel/best_model_student.pdparams`. + + + +## 5. Hyperparameters Searching + +The hyperparameters used by [3.2 section](#3.2) and [4.1 section](#4.1) are according by `Hyperparameters Searching` in PaddleClas. If you want to get better results on your own dataset, you can refer to [Hyperparameters Searching](PULC_train_en.md#4) to get better hyperparameters. + +**Note**: This section is optional. Because the search process will take a long time, you can selectively run according to your specific. If not replace the dataset, you can ignore this section. + + + +## 6. Inference Deployment + + + +### 6.1 Getting Paddle Inference Model + +Paddle Inference is the original Inference Library of the PaddlePaddle, provides high-performance inference for server deployment. And compared with directly based on the pretrained model, Paddle Inference can use tools to accelerate prediction, so as to achieve better inference performance. Please refer to [Paddle Inference](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/infer/inference/inference_cn.html) for more information. + +Paddle Inference need Paddle Inference Model to predict. Two process provided to get Paddle Inference Model. If want to use the provided by PaddleClas, you can download directly, click [Downloading Inference Model](#6.1.2). + + + +### 6.1.1 Exporting Paddle Inference Model + +The command about exporting Paddle Inference Model is as follow: + +```bash +python3 tools/export_model.py \ + -c ./ppcls/configs/PULC/car_exists/PPLCNet_x1_0.yaml \ + -o Global.pretrained_model=output/DistillationModel/best_model_student \ + -o Global.save_inference_dir=deploy/models/PPLCNet_x1_0_car_exists_infer +``` + +After running above command, the inference model files would be saved in `deploy/models/PPLCNet_x1_0_car_exists_infer`, as shown below: + +``` +├── PPLCNet_x1_0_car_exists_infer +│ ├── inference.pdiparams +│ ├── inference.pdiparams.info +│ └── inference.pdmodel +``` + +**Note**: The best model is from knowledge distillation training. If knowledge distillation training is not used, the best model would be saved in `output/PPLCNet_x1_0/best_model.pdparams`. + + + +### 6.1.2 Downloading Inference Model + +You can also download directly. + +``` +cd deploy/models +# download the inference model and decompression +wget https://paddleclas.bj.bcebos.com/models/PULC/car_exists_infer.tar && tar -xf car_exists_infer.tar +``` + +After decompression, the directory `models` should be shown below. + +``` +├── car_exists_infer +│ ├── inference.pdiparams +│ ├── inference.pdiparams.info +│ └── inference.pdmodel +``` + + + +### 6.2 Prediction with Python + + + +#### 6.2.1 Image Prediction + +Return the directory `deploy`: + +``` +cd ../ +``` + +Run the following command to classify whether there are cars in the image `./images/PULC/car_exists/objects365_00001507.jpeg`. + +```shell +# Use the following command to predict with GPU. +python3.7 python/predict_cls.py -c configs/PULC/car_exists/inference_car_exists.yaml +# Use the following command to predict with CPU. +python3.7 python/predict_cls.py -c configs/PULC/car_exists/inference_car_exists.yaml -o Global.use_gpu=False +``` + +The prediction results: + +``` +objects365_00001507.jpeg: class id(s): [1], score(s): [0.99], label_name(s): ['contains_car'] +``` + +**Note**: The default threshold is `0.5`. If needed, you can specify the argument `Infer.PostProcess.threshold`, such as: `-o Infer.PostProcess.threshold=0.9794`. And the argument `threshold` is needed to be specified according by specific case. The `0.9794` is the best threshold when `Fpr` is less than `1/100` in this valuation dataset. Please refer to [3.3 section](#3.3) for details. + + + +#### 6.2.2 Images Prediction + +If you want to predict images in directory, please specify the argument `Global.infer_imgs` as directory path by `-o Global.infer_imgs`. The command is as follow. + +```shell +# Use the following command to predict with GPU. If want to replace with CPU, you can add argument -o Global.use_gpu=False +python3.7 python/predict_cls.py -c configs/PULC/car_exists/inference_car_exists.yaml -o Global.infer_imgs="./images/PULC/car_exists/" +``` + +All prediction results will be printed, as shown below. + +``` +objects365_00001507.jpeg: class id(s): [1], score(s): [0.99], label_name(s): ['contains_car'] +objects365_00001521.jpeg: class id(s): [0], score(s): [0.99], label_name(s): ['no_car'] +``` + +Among the prediction results above, `contains_car` means that there is a car in the image, `no_car` means that there is no car in the image. + + + +### 6.3 Deployment with C++ + +PaddleClas provides an example about how to deploy with C++. Please refer to [Deployment with C++](../inference_deployment/cpp_deploy_en.md). + + + +### 6.4 Deployment as Service + +Paddle Serving is a flexible, high-performance carrier for machine learning models, and supports different protocol, such as RESTful, gRPC, bRPC and so on, which provides different deployment solutions for a variety of heterogeneous hardware and operating system environments. Please refer [Paddle Serving](https://github.com/PaddlePaddle/Serving) for more information. + +PaddleClas provides an example about how to deploy as service by Paddle Serving. Please refer to [Paddle Serving Deployment](../inference_deployment/paddle_serving_deploy_en.md). + + + +### 6.5 Deployment on Mobile + +Paddle-Lite is an open source deep learning framework that designed to make easy to perform inference on mobile, embeded, and IoT devices. Please refer to [Paddle-Lite](https://github.com/PaddlePaddle/Paddle-Lite) for more information. + +PaddleClas provides an example of how to deploy on mobile by Paddle-Lite. Please refer to [Paddle-Lite deployment](../inference_deployment/paddle_lite_deploy_en.md). + + + +### 6.6 Converting To ONNX and Deployment + +Paddle2ONNX support convert Paddle Inference model to ONNX model. And you can deploy with ONNX model on different inference engine, such as TensorRT, OpenVINO, MNN/TNN, NCNN and so on. About Paddle2ONNX details, please refer to [Paddle2ONNX](https://github.com/PaddlePaddle/Paddle2ONNX). + +PaddleClas provides an example of how to convert Paddle Inference model to ONNX model by paddle2onnx toolkit and predict by ONNX model. You can refer to [paddle2onnx](../../../deploy/paddle2onnx/readme_en.md) for deployment details. diff --git a/docs/en/PULC/PULC_person_exists_en.md b/docs/en/PULC/PULC_person_exists_en.md index f3d6b9aa5f189ccdb8ded838841630c9895d0418..93f583392b1890aa8a6c7d26affe33e4c4a7935a 100644 --- a/docs/en/PULC/PULC_person_exists_en.md +++ b/docs/en/PULC/PULC_person_exists_en.md @@ -390,7 +390,7 @@ Return the directory `deploy`: cd ../ ``` -Run the following command to classify whether there are human in the image `./images/PULC/person_exists/objects365_02035329.jpg`. +Run the following command to classify whether there are humans in the image `./images/PULC/person_exists/objects365_02035329.jpg`. ```shell # Use the following command to predict with GPU. diff --git a/docs/en/PULC/PULC_text_image_orientation_en.md b/docs/en/PULC/PULC_text_image_orientation_en.md new file mode 100644 index 0000000000000000000000000000000000000000..8933bdbc25a259e67284faed20351c91b186db7c --- /dev/null +++ b/docs/en/PULC/PULC_text_image_orientation_en.md @@ -0,0 +1,469 @@ +# PULC Classification Model of Text Image Orientation + +## Catalogue + +- [1. Introduction](#1) +- [2. Quick Start](#2) + - [2.1 PaddlePaddle Installation](#2.1) + - [2.2 PaddleClas Installation](#2.2) + - [2.3 Prediction](#2.3) +- [3. Training, Evaluation and Inference](#3) + - [3.1 Installation](#3.1) + - [3.2 Dataset](#3.2) + - [3.2.1 Dataset Introduction](#3.2.1) + - [3.2.2 Getting Dataset](#3.2.2) + - [3.3 Training](#3.3) + - [3.4 Evaluation](#3.4) + - [3.5 Inference](#3.5) +- [4. Model Compression](#4) + - [4.1 SKL-UGI Knowledge Distillation](#4.1) + - [4.1.1 Teacher Model Training](#4.1.1) + - [4.1.2 Knowledge Distillation Training](#4.1.2) +- [5. SHAS](#5) +- [6. Inference Deployment](#6) + - [6.1 Getting Paddle Inference Model](#6.1) + - [6.1.1 Exporting Paddle Inference Model](#6.1.1) + - [6.1.2 Downloading Inference Model](#6.1.2) + - [6.2 Prediction with Python](#6.2) + - [6.2.1 Image Prediction](#6.2.1) + - [6.2.2 Images Prediction](#6.2.2) + - [6.3 Deployment with C++](#6.3) + - [6.4 Deployment as Service](#6.4) + - [6.5 Deployment on Mobile](#6.5) + - [6.6 Converting To ONNX and Deployment](#6.6) + + + +## 1. Introduction + +In the process of document scanning, license shooting and so on, sometimes in order to shoot more clearly, the camera device will be rotated, resulting in photo in different directions. At this time, the standard OCR process cannot cope with these issues well. Using the text image orientation classification technology, the direction of the text image can be predicted and adjusted, so as to improve the accuracy of OCR processing. This case provides a way for users to use PaddleClas PULC (Practical Ultra Lightweight Classification) to quickly build a lightweight, high-precision, practical classification model of text image orientation. This model can be widely used in OCR processing scenarios of rotating pictures in financial, government and other industries. + +The following table lists the relevant indicators of the model. The first two lines means that using SwinTransformer_tiny and MobileNetV3_small_x0_35 as the backbone to training. The third to fifth lines means that the backbone is replaced by PPLCNet, additional use of SSLD pretrained model and additional use of EDA strategy and SKL-UGI knowledge distillation strategy. + +下表列出了判断含文字图像方向分类模型的相关指标，前两行展现了使用 SwinTranformer_tiny 和 MobileNetV3_small_x0_35 作为 backbone 训练得到的模型的相关指标，第三行至第五行依次展现了替换 backbone 为 PPLCNet_x1_0、使用 SSLD 预训练模型、使用 超参数搜索策略训练得到的模型的相关指标。 + +| 模型 | 精度（%） | 延时（ms） | 存储（M） | 策略 | +| ----------------------- | --------- | ---------- | --------- | ------------------------------------- | +| SwinTranformer_tiny | 99.12 | 89.65 | 107 | 使用ImageNet预训练模型 | +| MobileNetV3_small_x0_35 | 83.61 | 2.95 | 17 | 使用ImageNet预训练模型 | +| PPLCNet_x1_0 | 97.85 | 2.16 | 6.5 | 使用ImageNet预训练模型 | +| PPLCNet_x1_0 | 98.02 | 2.16 | 6.5 | 使用SSLD预训练模型 | +| **PPLCNet_x1_0** | **99.06** | **2.16** | **6.5** | 使用SSLD预训练模型+SHAS超参数搜索策略 | + +从表中可以看出，backbone 为 SwinTranformer_tiny 时精度比较高，但是推理速度较慢。将 backbone 替换为轻量级模型 MobileNetV3_small_x0_35 后，速度提升明显，但精度有了大幅下降。将 backbone 替换为 PPLCNet_x1_0 时，速度略为提升，同时精度较 MobileNetV3_small_x0_35 高了 14.24 个百分点。在此基础上，使用 SSLD 预训练模型后，在不改变推理速度的前提下，精度可以提升 0.17 个百分点，进一步地，当使用SHAS超参数搜索策略搜索最优超参数后，精度可以再提升 1.04 个百分点。此时，PPLCNet_x1_0 与 SwinTranformer_tiny 的精度差别不大，但是速度明显变快。关于 PULC 的训练方法和推理部署方法将在下面详细介绍。 + +**Note**: + +* The Latency is tested on Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz. The MKLDNN is enabled and the number of threads is 10. +* About PP-LCNet, please refer to [PP-LCNet Introduction](../models/PP-LCNet_en.md) and [PP-LCNet Paper](https://arxiv.org/abs/2109.15099). + + + +## 2. Quick Start + + + +### 2.1 PaddlePaddle Installation + +- Run the following command to install if CUDA9 or CUDA10 is available. + +```bash +python3 -m pip install paddlepaddle-gpu -i https://mirror.baidu.com/pypi/simple +``` + +- Run the following command to install if GPU device is unavailable. + +```bash +python3 -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple +``` + +Please refer to [PaddlePaddle Installation](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/en/install/pip/linux-pip_en.html) for more information about installation, for examples other versions. + + + +### 2.2 PaddleClas wheel Installation + +The command of PaddleClas installation as bellow: + +```bash +pip3 install paddleclas +``` + + + +### 2.3 Prediction + +First, please click [here](https://paddleclas.bj.bcebos.com/data/PULC/pulc_demo_imgs.zip) to download and unzip to get the test demo images. + +* Prediction with CLI + +```bash +paddleclas --model_name=text_image_orientation --infer_imgs=pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg +``` + +Results: + +``` +>>> result +class_ids: [0, 2], scores: [0.85615, 0.05046], label_names: ['0', '180'], filename: pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg +Predict complete! +``` + +**Note**: If you want to test other images, only need to specify the `--infer_imgs` argument, and the directory containing images is also supported. + +* Prediction in Python + +```python +import paddleclas +model = paddleclas.PaddleClas(model_name="text_image_orientation") +result = model.predict(input_data="pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg") +print(next(result)) +``` + +**Note**: The `result` returned by `model.predict()` is a generator, so you need to use the `next()` function to call it or `for` loop to loop it. And it will predict with `batch_size` size batch and return the prediction results when called. The default `batch_size` is 1, and you also specify the `batch_size` when instantiating, such as `model = paddleclas.PaddleClas(model_name="text_image_orientation", batch_size=2)`. The result of demo above: + +``` +>>> result +[{'class_ids': [0, 2], 'scores': [0.85615, 0.05046], 'label_names': ['0', '180'], 'filename': 'pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg'}] +``` + + + +## 3. Training, Evaluation and Inference + + + +### 3.1 Installation + +Please refer to [Installation](../installation/install_paddleclas_en.md) to get the description about installation. + + + +### 3.2 Dataset + + + +#### 3.2.1 Dataset Introduction + +The model provided in [1 section](#1) is trained using internal data, which has not been open source. So we provide a dataset with [ICDAR2019-ArT](https://ai.baidu.com/broad/introduction?dataset=art), [XFUND](https://github.com/doc-analysis/XFUND) and [ICDAR2015](https://rrc.cvc.uab.es/?ch=4&com=introduction) to experience. + +![](../../images/PULC/docs/text_image_orientation_original_data.png) + + + +#### 3.2.2 Getting Dataset + +The data used in this case can be getted by processing the open source data. The detailed processes are as follows: + +Considering the resolution of original image is too high to need long training time, all the data are scaled in advance. Keeping image aspect ratio, the short edge is scaled to 384. Then rotate the data clockwise to generate composite data of 90 degrees, 180 degrees and 270 degrees respectively. Among them, 41460 images generated by ICDAR2019-ArT and XFUND are randomly divided into training set and verification set according to the ratio of 9:1. 6000 images generated by ICDAR2015 are used as supplementary data in the experiment of `SKL-UGI knowledge distillation`. + +Some image of the processed dataset is as follows: + +![](../../images/PULC/docs/text_image_orientation_data_demo.png) + +And you can also download the data processed directly. + + +``` +cd path_to_PaddleClas +``` + +Enter the `dataset/` directory, download and unzip the dataset. + +```shell +cd dataset +wget https://paddleclas.bj.bcebos.com/data/PULC/text_image_orientation.tar +tar -xf text_image_orientation.tar +cd ../ +``` + +The datas under `text_image_orientation` directory: + +``` +├── img_0 +│ ├── img_rot0_0.jpg +│ ├── img_rot0_1.png +... +├── img_90 +│ ├── img_rot90_0.jpg +│ ├── img_rot90_1.png +... +├── img_180 +│ ├── img_rot180_0.jpg +│ ├── img_rot180_1.png +... +├── img_270 +│ ├── img_rot270_0.jpg +│ ├── img_rot270_1.png +... +├── distill_data +│ ├── gt_7060_0.jpg +│ ├── gt_7060_90.jpg +... +├── train_list.txt +├── train_list.txt.debug +├── train_list_for_distill.txt +├── test_list.txt +├── test_list.txt.debug +└── label_list.txt +``` + +Where `img_0/`, `img_90/`, `img_180/` and `img_270/` are data of 4 angles respectively. The `train_list.txt` and `val_list.txt` are label files of training data and validation data respectively. The file `train_list.txt.debug` and `val_list.txt.debug` are subset of `train_list.txt` and `val_list.txt` respectively. `distill_data/` is the supplementary data, which will be used for SKL-UGI knowledge distillation, and its label file is `train_list_for_distill.txt`. + +**Note**: + +* About the contents format of `train_list.txt` and `val_list.txt`, please refer to [Description about Classification Dataset in PaddleClas](../data_preparation/classification_dataset_en.md). +* About the `train_list_for_distill.txt`, please refer to [Knowledge Distillation Label](../advanced_tutorials/distillation/distillation_en.md). + + + +### 3.3 Training + +The details of training config in `ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml`. The command about training as follows: + +```shell +export CUDA_VISIBLE_DEVICES=0,1,2,3 +python3 -m paddle.distributed.launch \ + --gpus="0,1,2,3" \ + tools/train.py \ + -c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml +``` + +The best metric of validation data is about `0.99`. + + +**Note**: +* The metric mentioned in this document are training on large-scale internal dataset. When using demo data to train, this metric cannot be achieved because the dataset is small and the distribution is different from large-scale internal data. You can further expand your own data and use the optimization method described in this case to achieve higher accuracy. + + + +### 3.4 Evaluation + +After training, you can use the following commands to evaluate the model. + +```bash +python3 tools/eval.py \ + -c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \ + -o Global.pretrained_model="output/PPLCNet_x1_0/best_model" +``` + +Among the above command, the argument `-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"` specify the path of the best model weight file. You can specify other path if needed. + + + +### 3.5 Inference + +After training, you can use the model that trained to infer. Command is as follow: + +```bash +python3 tools/infer.py \ + -c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \ + -o Global.pretrained_model="output/PPLCNet_x1_0/best_model" +``` + +The results: + +``` +[{'class_ids': [0, 2], 'scores': [0.85615, 0.05046], 'file_name': 'deploy/images/PULC/text_image_orientation/img_rot0_demo.jpg', 'label_names': ['0', '180']}] +``` + +**Note**: + +* Among the above command, argument `-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"` specify the path of the best model weight file. You can specify other path if needed. +* The default test image is `deploy/images/PULC/text_image_orientation/img_rot0_demo.jpg`. And you can test other image, only need to specify the argument `-o Infer.infer_imgs=path_to_test_image`. +* The Top2 result would be printed. `0` means that the text direction of the drawing is 0 degrees, `90` means that 90 degrees clockwise, `180` means that 180 degrees clockwise, `270` means that 270 degrees clockwise. + + + +## 4. Model Compression + + + +### 4.1 SKL-UGI Knowledge Distillation + +SKL-UGI is a simple but effective knowledge distillation algrithem proposed by PaddleClas. + + + + + + +#### 4.1.1 Teacher Model Training + +Training the teacher model with hyperparameters specified in `ppcls/configs/PULC/text_image_orientation/PPLCNet/PPLCNet_x1_0.yaml`. The command is as follow: + +```shell +export CUDA_VISIBLE_DEVICES=0,1,2,3 +python3 -m paddle.distributed.launch \ + --gpus="0,1,2,3" \ + tools/train.py \ + -c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \ + -o Arch.name=ResNet101_vd +``` + +The best metric of validation data is about `0.996`. The best teacher model weight would be saved in file `output/ResNet101_vd/best_model.pdparams`. + +**Note**: Training ResNet101_vd need more GPU memory. So you can reduce `batch_size` and `learning rate` at the same time, such as: `-o DataLoader.Train.sampler.batch_size=64`, `Optimizer.lr.learning_rate=0.1`. + + + +#### 4.1.2 Knowledge Distillation Training + +The training strategy, specified in training config file `ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0_distillation.yaml`, the teacher model is `ResNet101_vd`, the student model is `PPLCNet_x1_0` and the additional unlabeled training data is validation data of ImageNet1k. + +The command is as follow: + +```shell +export CUDA_VISIBLE_DEVICES=0,1,2,3 +python3 -m paddle.distributed.launch \ + --gpus="0,1,2,3" \ + tools/train.py \ + -c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0_distillation.yaml \ + -o Arch.models.0.Teacher.pretrained=output/ResNet101_vd/best_model +``` + +The best metric is about `0.99`. The best student model weight would be saved in file `output/DistillationModel/best_model_student.pdparams`. + + + +## 5. Hyperparameters Searching + +The hyperparameters used by [3.2 section](#3.2) and [4.1 section](#4.1) are according by `Hyperparameters Searching` in PaddleClas. If you want to get better results on your own dataset, you can refer to [Hyperparameters Searching](PULC_train_en.md#4) to get better hyperparameters. + +**Note**: This section is optional. Because the search process will take a long time, you can selectively run according to your specific. If not replace the dataset, you can ignore this section. + + + +## 6. Inference Deployment + + + +### 6.1 Getting Paddle Inference Model + +Paddle Inference is the original Inference Library of the PaddlePaddle, provides high-performance inference for server deployment. And compared with directly based on the pretrained model, Paddle Inference can use tools to accelerate prediction, so as to achieve better inference performance. Please refer to [Paddle Inference](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/infer/inference/inference_cn.html) for more information. + +Paddle Inference need Paddle Inference Model to predict. Two process provided to get Paddle Inference Model. If want to use the provided by PaddleClas, you can download directly, click [Downloading Inference Model](#6.1.2). + + + +### 6.1.1 Exporting Paddle Inference Model + +The command about exporting Paddle Inference Model is as follow: + +```bash +python3 tools/export_model.py \ + -c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \ + -o Global.pretrained_model=output/DistillationModel/best_model_student \ + -o Global.save_inference_dir=deploy/models/PPLCNet_x1_0_text_image_orientation_infer +``` + +After running above command, the inference model files would be saved in `deploy/models/PPLCNet_x1_0_text_image_orientation_infer`, as shown below: + +``` +├── PPLCNet_x1_0_text_image_orientation_infer +│ ├── inference.pdiparams +│ ├── inference.pdiparams.info +│ └── inference.pdmodel +``` + +**Note**: The best model is from knowledge distillation training. If knowledge distillation training is not used, the best model would be saved in `output/PPLCNet_x1_0/best_model.pdparams`. + + + +### 6.1.2 Downloading Inference Model + +You can also download directly. + +``` +cd deploy/models +# download the inference model and decompression +wget https://paddleclas.bj.bcebos.com/models/PULC/text_image_orientation_infer.tar && tar -xf text_image_orientation_infer.tar +``` + +After decompression, the directory `models` should be shown below. + +``` +├── text_image_orientation_infer +│ ├── inference.pdiparams +│ ├── inference.pdiparams.info +│ └── inference.pdmodel +``` + + + +### 6.2 Prediction with Python + + + +#### 6.2.1 Image Prediction + +Return the directory `deploy`: + +``` +cd ../ +``` + +Run the following command to classify text image orientation about image `./images/PULC/text_image_orientation/img_rot0_demo.png`. + +```shell +# Use the following command to predict with GPU. +python3.7 python/predict_cls.py -c configs/PULC/text_image_orientation/inference_text_image_orientation.yaml +# Use the following command to predict with CPU. +python3.7 python/predict_cls.py -c configs/PULC/text_image_orientation/inference_text_image_orientation.yaml -o Global.use_gpu=False +``` + +The prediction results: + +``` +img_rot0_demo.jpg: class id(s): [0, 2], score(s): [0.86, 0.05], label_name(s): ['0', '180'] +``` + +Among the results, `0` means that the text direction of the drawing is 0 degrees, `90` means that 90 degrees clockwise, `180` means that 180 degrees clockwise, `270` means that 270 degrees clockwise. + + + +#### 6.2.2 Images Prediction + +If you want to predict images in directory, please specify the argument `Global.infer_imgs` as directory path by `-o Global.infer_imgs`. The command is as follow. + +```shell +# Use the following command to predict with GPU. If want to replace with CPU, you can add argument -o Global.use_gpu=False +python3.7 python/predict_cls.py -c configs/PULC/text_image_orientation/inference_text_image_orientation.yaml -o Global.infer_imgs="./images/PULC/text_image_orientation/" +``` + +All prediction results will be printed, as shown below. + +``` +img_rot0_demo.jpg: class id(s): [0, 2], score(s): [0.86, 0.05], label_name(s): ['0', '180'] +img_rot180_demo.jpg: class id(s): [2, 1], score(s): [0.88, 0.04], label_name(s): ['180', '90'] +``` + + + +### 6.3 Deployment with C++ + +PaddleClas provides an example about how to deploy with C++. Please refer to [Deployment with C++](../inference_deployment/cpp_deploy_en.md). + + + +### 6.4 Deployment as Service + +Paddle Serving is a flexible, high-performance carrier for machine learning models, and supports different protocol, such as RESTful, gRPC, bRPC and so on, which provides different deployment solutions for a variety of heterogeneous hardware and operating system environments. Please refer [Paddle Serving](https://github.com/PaddlePaddle/Serving) for more information. + +PaddleClas provides an example about how to deploy as service by Paddle Serving. Please refer to [Paddle Serving Deployment](../inference_deployment/paddle_serving_deploy_en.md). + + + +### 6.5 Deployment on Mobile + +Paddle-Lite is an open source deep learning framework that designed to make easy to perform inference on mobile, embeded, and IoT devices. Please refer to [Paddle-Lite](https://github.com/PaddlePaddle/Paddle-Lite) for more information. + +PaddleClas provides an example of how to deploy on mobile by Paddle-Lite. Please refer to [Paddle-Lite deployment](../inference_deployment/paddle_lite_deploy_en.md). + + + +### 6.6 Converting To ONNX and Deployment + +Paddle2ONNX support convert Paddle Inference model to ONNX model. And you can deploy with ONNX model on different inference engine, such as TensorRT, OpenVINO, MNN/TNN, NCNN and so on. About Paddle2ONNX details, please refer to [Paddle2ONNX](https://github.com/PaddlePaddle/Paddle2ONNX). + +PaddleClas provides an example of how to convert Paddle Inference model to ONNX model by paddle2onnx toolkit and predict by ONNX model. You can refer to [paddle2onnx](../../../deploy/paddle2onnx/readme_en.md) for deployment details. diff --git a/docs/en/PULC/PULC_traffic_sign_en.md b/docs/en/PULC/PULC_traffic_sign_en.md index 9bd8ec3a778897eb314bd9d5547588eaadb7d2e8..21abd45438b906755662497c2527c88b3009c326 100644 --- a/docs/en/PULC/PULC_traffic_sign_en.md +++ b/docs/en/PULC/PULC_traffic_sign_en.md @@ -182,7 +182,6 @@ Some image of the processed dataset is as follows: You can also download the data processed directly. And the process script file `deal.py` is also included. - ``` cd path_to_PaddleClas ``` diff --git a/docs/zh_CN/PULC/PULC_car_exists.md b/docs/zh_CN/PULC/PULC_car_exists.md index 0ccb0c2d45a94968f326146d8533b9c551bfd00f..86cc565c800ed29f1d126e258cb2c5ccf5d3c397 100644 --- a/docs/zh_CN/PULC/PULC_car_exists.md +++ b/docs/zh_CN/PULC/PULC_car_exists.md @@ -150,7 +150,7 @@ print(next(result)) #### 3.2.1 数据集来源 -本案例中所使用的所有数据集均为开源数据，`train`和`val` 集合均为[Objects365 数据](https://www.objects365.org/overview.html)的训练集的子集，`ImageNet_val` 为[ImageNet-1k 数据](https://www.image-net.org/)的验证集。 +本案例中所使用的所有数据集均为开源数据，`train`和`val` 集合均为[Objects365 数据](https://www.objects365.org/overview.html)的子集，`ImageNet_val` 为[ImageNet-1k 数据](https://www.image-net.org/)的验证集。