-[3. Model evaluation and inference deployment](#3-model-evaluation-and-inference-deployment)
-[3.1.2 Accuracy metrics](#312-accuracy-metrics)
-[3.1 Model Evaluation](#31-model-evaluation)
-[3.1.3 Data Preparation](#313-data-preparation)
-[3.2 Model Inference and Deployment](#32-model-inference-and-deployment)
-[3.1.4 Model training](#314-model-training)
-[3.2.1 Inference model preparation](#321-inference-model-preparation)
-[4. Model evaluation and inference deployment](#4-model-evaluation-and-inference-deployment)
-[3.2.2 Inference based on Python prediction engine](#322-inference-based-on-python-prediction-engine)
-[4.1 Model Evaluation](#41-model-evaluation)
-[3.2.3 Inference based on C++ prediction engine](#323-inference-based-on-c-prediction-engine)
-[4.2 Model Inference](#42-model-inference)
-[3.3 Service Deployment](#33-service-deployment)
-[4.2.1 Inference model preparation](#421-inference-model-preparation)
-[3.4 Lite deployment](#34-lite-deployment)
-[4.2.2 Inference based on Python prediction engine](#422-inference-based-on-python-prediction-engine)
-[3.5 Paddle2ONNX model conversion and prediction](#35-paddle2onnx-model-conversion-and-prediction)
-[4.2.3 Inference based on C++ prediction engine](#423-inference-based-on-c-prediction-engine)
-[4. Summary](#4-summary)
-[4.3 Service deployment](#43-service-deployment)
-[4.1 Method summary and comparison](#41-method-summary-and-comparison)
-[4.4 Lite deployment](#44-lite-deployment)
-[4.2 Usage advice/FAQ](#42-usage-advicefaq)
-[4.5 Paddle2ONNX Model Conversion and Prediction](#45-paddle2onnx-model-conversion-and-prediction)
-[4. References](#4-references)
-[5. Summary](#5-summary)
-[5.1 Method summary and comparison](#51-method-summary-and-comparison)
-[5.2 Usage advice/FAQ](#52-usage-advicefaq)
-[6. References](#6-references)
### 1. Introduction to algorithms/application scenarios
### 1. Introduction to algorithms/application scenarios
Pedestrian re-identification, also known as pedestrian re-identification, is a technology that uses computer vision technology to determine whether there is a specific pedestrian in an image or video sequence. Widely regarded as a sub-problem of image retrieval. Given a surveillance pedestrian image, retrieve the pedestrian image across devices. It is designed to make up for the visual limitations of fixed cameras, and can be combined with pedestrian detection/pedestrian tracking technology, which can be widely used in intelligent video surveillance, intelligent security and other fields.
Person re-identification (Re-ID), also known as person re-identification, has been widely studied as a cross-shot pedestrian retrieval problem. Given a pedestrian image captured by a certain camera, the goal is to determine whether the pedestrian has appeared in images captured by different cameras or in different time periods. The given pedestrian data can be a picture, a video frame, or even a text description. In recent years, the application demand of this technology in the field of public safety has been increasing, and the influence of pedestrian re-identification in intelligent monitoring technology is also increasing.
At present, pedestrian re-identification is still a challenging task, especially the problems of different viewpoints, resolutions, illumination changes, occlusions, multi-modalities, as well as complex camera environment and background, labeling data noise, etc. There is great uncertainty. In addition, when the actual landing, the shooting camera may change, the large-scale retrieval database, the distribution shift of the data set, the unknown scene, the incremental update of the model, and the change of the clothing of the retrieval person, which also increases a lot of difficulties.
Early work on person re-identification mainly focused on hand-designed feature extraction operators, including adding human pose features, or learning distance metric functions. With the development of deep learning technology, pedestrian recognition has also made great progress. In general, the whole process of pedestrian re-identification includes 5 steps: 1) data collection, 2) pedestrian location box annotation, 3) pedestrian category annotation, 4) model training, and 5) pedestrian retrieval (model testing).
The common person re-identification method extracts the local/global, single-granularity/multi-granularity features of the input image through the feature extraction module, and then obtains a high-dimensional feature vector through the fusion module. Use the classification head to convert the feature vector into the probability of each category during training to optimize the feature extraction model in the way of classification tasks; directly use the high-dimensional feature vector as the image description vector in the retrieval vector library during testing or inference search to get the search results. The ReID strong-baseline algorithm proposes several methods to effectively optimize training and retrieval to improve the overall model performance.
Among them, $N$ is the number of query samples, and $result_i$ is the label set of the retrieval results of each query sample. According to the formula, the CMC curve can be understood as an array composed of Top1-Acc, Top2-Acc, ..., TopK-Acc , which is obviously a monotonic curve. Among them, the common Rank-1 and Top1-Acc metric refer to CMC(1)
2. mAP
Assuming that a query sample is used and a set of query results is returned, then according to the following formula, consider the first K query results one by one, and for each K, calculate the precision rate $Precision$ and recall rate $Recall$.
The obtained multiple groups (Precision, Recall) are converted into a curve graph, and the area enclosed by the curve and the coordinate axis is called Average Precision (AP),
For each sample, calculate its AP value, and then take the average to get the mAP.
### 3. ReID algorithm
#### 2.1 ReID strong-baseline
#### 3.1 ReID strong-baseline
Paper source: [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
Paper source: [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
Based on the commonly used person re-identification model based on ResNet50, the author explores and summarizes the following effective and applicable optimization methods, which greatly improves the indicators on multiple person re-identification datasets.
Based on the commonly used person re-identification model based on ResNet50, the author explores and summarizes the following effective and applicable optimization methods, which greatly improves the indicators on multiple person re-identification datasets.
...
@@ -52,40 +91,40 @@ Based on the commonly used person re-identification model based on ResNet50, the
...
@@ -52,40 +91,40 @@ Based on the commonly used person re-identification model based on ResNet50, the
6. Center loss: Give each category a learnable cluster center, and make the intra-class features close to the cluster center during training to reduce intra-class differences and increase inter-class differences.
6. Center loss: Give each category a learnable cluster center, and make the intra-class features close to the cluster center during training to reduce intra-class differences and increase inter-class differences.
7. Reranking: Consider the neighbor candidates of the query image during retrieval, optimize the distance matrix according to whether the neighbor images of the candidate object also contain the query image, and finally improve the retrieval accuracy.
7. Reranking: Consider the neighbor candidates of the query image during retrieval, optimize the distance matrix according to whether the neighbor images of the candidate object also contain the query image, and finally improve the retrieval accuracy.
##### 2.1.2 Accuracy Index
##### 3.1.2 Accuracy metrics
The following table summarizes the accuracy metrics of the 3 configurations of the recurring ReID strong-baseline on the Market1501 dataset,
The following table summarizes the accuracy metrics of the 3 configurations of the recurring ReID strong-baseline on the Market1501 dataset,
| Configuration file | recall@1(\%) | mAP(\%) | Refer to recall@1(\%) | Refer to mAP(\%) | Pre-trained model download | Inference Model Download Address |
Note: The above reference indicators are obtained by using the author's open source code to train on our equipment for many times. Due to different system environments, torch versions, and CUDA versions, there may be slight differences with the indicators provided by the author.
Note: The above reference indicators are obtained by using the author's open source code to train on our equipment for many times. Due to different system environment, torch version, CUDA version and other reasons, there may be slight differences with the indicators provided by the author.
Next, we mainly take the `softmax_triplet_with_center.yaml` configuration and trained model file as an example to show the process of training, testing, and inference on the Market1501 dataset.
Next, we mainly take the `softmax_triplet_with_center.yaml` configuration and trained model file as an example to show the process of training, testing, and inference on the Market1501 dataset.
##### 2.1.3 Data Preparation
##### 3.1.3 Data Preparation
Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1654142245770) dataset, extract it to `PaddleClas/dataset/`, and organize it into the following file structure :
Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1654142245770) dataset, extract it to `PaddleClas/dataset/`, and organize it into the following file structure :
```shell
```shell
PaddleClas/dataset/market1501
PaddleClas/dataset/market1501
└── Market-1501-v15.09.15/
└── Market-1501-v15.09.15/
├── bounding_box_test/ # gallery set pictures
├── bounding_box_test/ # gallery set pictures
├── bounding_box_train/ # training set image
├── bounding_box_train/ # training set image
├── gt_bbox/
├── gt_bbox/
├── gt_query/
├── gt_query/
├── query/ # query set image
├── query/ # query set image
├── generate_anno.py
├── generate_anno.py
├── bounding_box_test.txt # gallery set path
├── bounding_box_test.txt # gallery set path
├── bounding_box_train.txt # training set path
├── bounding_box_train.txt # training set path
├── query.txt # query set path
├── query.txt # query set path
└── readme.txt
└── readme.txt
```
```
##### 2.1.4 Model training
##### 3.1.4 Model training
1. Execute the following command to start training
1. Execute the following command to start training
...
@@ -96,7 +135,7 @@ Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1
...
@@ -96,7 +135,7 @@ Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1
Doka training:
Doka training:
For Doka training, you need to modify the sampler field of the training configuration as follows:
For multi-card training, you need to modify the sampler field of the training configuration to adapt to distributed training, as follows:
```yaml
```yaml
sampler:
sampler:
name: PKSampler
name: PKSampler
...
@@ -106,7 +145,7 @@ Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1
...
@@ -106,7 +145,7 @@ Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1
sample_method: id_avg_prob
sample_method: id_avg_prob
shuffle: True
shuffle: True
```
```
Then execute the following command(example of 4 gpus below):
@@ -118,9 +157,9 @@ Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1
...
@@ -118,9 +157,9 @@ Download the [Market-1501-v15.09.15.zip](https://pan.baidu.com/s/1ntIi2Op?_at_=1
During the training process, indicator information such as loss will be printed on the screen in real time, and the log file `train.log`, model parameter file `*.pdparams`, optimizer parameter file `*.pdopt` and other contents will be saved to `Global.output_dir` `Under the specified folder, the default is under the `PaddleClas/output/RecModel/` folder.
During the training process, indicator information such as loss will be printed on the screen in real time, and the log file `train.log`, model parameter file `*.pdparams`, optimizer parameter file `*.pdopt` and other contents will be saved to `Global.output_dir` `Under the specified folder, the default is under the `PaddleClas/output/RecModel/` folder.
### 3. Model evaluation and inference deployment
### 4. Model evaluation and inference deployment
#### 3.1 Model Evaluation
#### 4.1 Model Evaluation
Prepare the `*.pdparams` model parameter file for evaluation. You can use the trained model or the model saved in [2.1.4 Model training] (#214-model training).
Prepare the `*.pdparams` model parameter file for evaluation. You can use the trained model or the model saved in [2.1.4 Model training] (#214-model training).
...
@@ -132,7 +171,7 @@ Prepare the `*.pdparams` model parameter file for evaluation. You can use the tr
...
@@ -132,7 +171,7 @@ Prepare the `*.pdparams` model parameter file for evaluation. You can use the tr
-to train wellTake the model as an example, download [softmax_triplet_with_center_pretrained.pdparams](https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/reid/pretrain/softmax_triplet_with_center_pretrained.pdparams) to `PaddleClas/ In the pretrained_models` folder, execute the following command to evaluate.
-Take the trained model as an example, download [softmax_triplet_with_center_pretrained.pdparams](https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/reid/pretrain/softmax_triplet_with_center_pretrained.pdparams) to `PaddleClas/ In the pretrained_models` folder, execute the following command to evaluate.
```shell
```shell
# download model
# download model
...
@@ -168,11 +207,11 @@ Prepare the `*.pdparams` model parameter file for evaluation. You can use the tr
...
@@ -168,11 +207,11 @@ Prepare the `*.pdparams` model parameter file for evaluation. You can use the tr
The default evaluation log is saved in `PaddleClas/output/RecModel/eval.log`. You can see that the evaluation metrics of the `softmax_triplet_with_center_pretrained.pdparams` model we provided on the Market1501 dataset are recall@1=0.94507, recall@5 =0.98248, mAP=0.85827
The default evaluation log is saved in `PaddleClas/output/RecModel/eval.log`. You can see that the evaluation indicators of the `softmax_triplet_with_center_pretrained.pdparams` model provided by us on the Market1501 dataset are recall@1=0.94507, recall@5=0.98248 , mAP=0.85827
#### 3.2 Model Inference and Deployment
#### 4.2 Model Inference
##### 3.2.1 Inference model preparation
##### 4.2.1 Inference model preparation
You can convert the model file saved during training into an inference model and inference, or use the converted inference model we provide for direct inference
You can convert the model file saved during training into an inference model and inference, or use the converted inference model we provide for direct inference
- Convert the model file saved during the training process to an inference model, also take `latest.pdparams` as an example, execute the following command to convert
- Convert the model file saved during the training process to an inference model, also take `latest.pdparams` as an example, execute the following command to convert
...
@@ -191,9 +230,11 @@ You can convert the model file saved during training into an inference model and
...
@@ -191,9 +230,11 @@ You can convert the model file saved during training into an inference model and
cd ../
cd ../
```
```
##### 3.2.2 Inference based on Python prediction engine
##### 4.2.2 Inference based on Python prediction engine
1. Modify `PaddleClas/deploy/configs/inference_rec.yaml`. Change the field after `infer_imgs:` to any image path under the query folder in Market1501 (the configuration below uses the path of the `0294_c1s1_066631_00.jpg` image); change the field after `rec_inference_model_dir:` to extract it softmax_triplet_with_center_infer folder path; change the preprocessing configuration under the `transform_ops` field to the preprocessing configuration under `Eval.Query.dataset` in `softmax_triplet_with_center.yaml`. As follows
1. Modify `PaddleClas/deploy/configs/inference_rec.yaml`- Change the path segment after `infer_imgs:` to any image path under the query folder in Market1501 (the configuration below uses the path of the `0294_c1s1_066631_00.jpg` image)
- Change the field after `rec_inference_model_dir:` to the decompressed softmax_triplet_with_center_infer folder path
- Change the preprocessing configuration under the `transform_ops:` field to the preprocessing configuration under `Eval.Query.dataset` in `softmax_triplet_with_center.yaml`
```yaml
```yaml
Global:
Global:
...
@@ -203,7 +244,7 @@ You can convert the model file saved during training into an inference model and
...
@@ -203,7 +244,7 @@ You can convert the model file saved during training into an inference model and
use_gpu: False
use_gpu: False
enable_mkldnn: True
enable_mkldnn: True
cpu_num_threads: 10
cpu_num_threads: 10
enable_benchmark: True
enable_benchmark: False
use_fp16: False
use_fp16: False
ir_optim: True
ir_optim: True
use_tensorrt: False
use_tensorrt: False
...
@@ -240,43 +281,44 @@ You can convert the model file saved during training into an inference model and
...
@@ -240,43 +281,44 @@ You can convert the model file saved during training into an inference model and
```
```
The output vector for inference is stored in the `result_dict` variable in [predict_rec.py](../../../deploy/python/predict_rec.py#L134-L135).
The output vector for inference is stored in the `result_dict` variable in [predict_rec.py](../../../deploy/python/predict_rec.py#L134-L135).
4. For batch prediction, change the path after `infer_imgs:` in the configuration file to a folder, such as `../dataset/market1501/Market-1501-v15.09.15/query`, it will predict and output The feature vector of all images under query.
4. For batch prediction, change the path after `infer_imgs:` in the configuration file to a folder, such as `../dataset/market1501/Market-1501-v15.09.15/query`, it will predict and output queries one by one The feature vectors of all the images below.
##### 3.2.3 Inference based on C++ prediction engine
##### 4.2.3 Inference based on C++ prediction engine
PaddleClas provides an example of inference based on C++ prediction engine, you can refer to [C++ prediction](../inference_deployment/cpp_deploy_en.md) to complete the corresponding inference deployment. If you are using the Windows platform, you can refer to the Visual Studio 2019 Community CMake Compilation Guide to complete the corresponding prediction library compilation and model prediction work.
PaddleClas provides an example of inference based on the C++ prediction engine, you can refer to [Server-side C++ prediction](../inference_deployment/cpp_deploy.md) to complete the corresponding inference deployment. If you are using the Windows platform, you can refer to the Visual Studio 2019 Community CMake Compilation Guide to complete the corresponding prediction library compilation and model prediction work.
#### 3.3 Service Deployment
#### 4.3 Service deployment
Paddle Serving provides high-performance, flexible and easy-to-use industrial-grade online inference services. Paddle Serving supports RESTful, gRPC, bRPC and other protocols, and provides inference solutions in a variety of heterogeneous hardware and operating system environments. For more introduction to Paddle Serving, please refer to the Paddle Serving code repository.
Paddle Serving provides high-performance, flexible and easy-to-use industrial-grade online inference services. Paddle Serving supports RESTful, gRPC, bRPC and other protocols, and provides inference solutions in a variety of heterogeneous hardware and operating system environments. For more introduction to Paddle Serving, please refer to the Paddle Serving code repository.
PaddleClas provides an example of model serving deployment based on Paddle Serving. You can refer to [Model serving deployment](../inference_deployment/paddle_serving_deploy_en.md) to complete the corresponding deployment work.
PaddleClas provides an example of model serving deployment based on Paddle Serving. You can refer to [Model serving deployment](../inference_deployment/paddle_serving_deploy.md) to complete the corresponding deployment.
#### 3.4 Lite deployment
#### 4.4 Lite deployment
Paddle Lite is a high-performance, lightweight, flexible and easily extensible deep learning inference framework, positioned to support mobileMultiple hardware platforms including client, embedded and server. For more introduction to Paddle Lite, please refer to the Paddle Lite code repository.
Paddle Lite is a high-performance, lightweight, flexible and easily extensible deep learning inference framework, positioned to support multiple hardware platforms including mobile, embedded and server. For more introduction to Paddle Lite, please refer to the Paddle Lite code repository.
PaddleClas provides an example of deploying models based on Paddle Lite. You can refer to [Deployment](../inference_deployment/paddle_lite_deploy_en.md) to complete the corresponding deployment.
PaddleClas provides an example of deploying models based on Paddle Lite. You can refer to [Deployment](../inference_deployment/paddle_lite_deploy.md) to complete the corresponding deployment.
#### 3.5 Paddle2ONNX model conversion and prediction
#### 4.5 Paddle2ONNX Model Conversion and Prediction
Paddle2ONNX supports converting PaddlePaddle model format to ONNX model format. The deployment of Paddle models to various inference engines can be completed through ONNX, including TensorRT/OpenVINO/MNN/TNN/NCNN, as well as other inference engines or hardware that support the ONNX open source format. For more information about Paddle2ONNX, please refer to the Paddle2ONNX code repository.
Paddle2ONNX supports converting PaddlePaddle model format to ONNX model format. The deployment of Paddle models to various inference engines can be completed through ONNX, including TensorRT/OpenVINO/MNN/TNN/NCNN, and other inference engines or hardware that support the ONNX open source format. For more information about Paddle2ONNX, please refer to the Paddle2ONNX code repository.
PaddleClas provides an example of converting an inference model to an ONNX model and making inference prediction based on Paddle2ONNX. You can refer to [Paddle2ONNX model conversion and prediction](../../../deploy/paddle2onnx/readme.md) to complete the corresponding deployment work.
PaddleClas provides an example of converting an inference model to an ONNX model and making inference prediction based on Paddle2ONNX. You can refer to [Paddle2ONNX model conversion and prediction](../../../deploy/paddle2onnx/readme.md) to complete the corresponding deployment work.
### 4. Summary
### 5. Summary
#### 4.1 Method summary and comparison
#### 5.1 Method summary and comparison
The above algorithm can be quickly migrated to most ReID models, which can further improve the performance of ReID models.
The above algorithm can be quickly migrated to most ReID models, which can further improve the performance of ReID models.
#### 4.2 Usage advice/FAQ
#### 5.2 Usage advice/FAQ
The Market1501 dataset is relatively small, so you can try to train multiple times to get the highest accuracy.
The Market1501 dataset is relatively small, so you can try to train multiple times to get the highest accuracy.
### 4. References
### 6. References
1.[Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
1.[Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
论文出处：[Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
论文出处：[Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
PaddleClas 提供了基于 C++ 预测引擎推理的示例，您可以参考[服务器端 C++ 预测](../inference_deployment/cpp_deploy.md)来完成相应的推理部署。如果您使用的是 Windows 平台，可以参考基于 Visual Studio 2019 Community CMake 编译指南完成相应的预测库编译和模型预测工作。
PaddleClas 提供了基于 C++ 预测引擎推理的示例，您可以参考[服务器端 C++ 预测](../inference_deployment/cpp_deploy.md)来完成相应的推理部署。如果您使用的是 Windows 平台，可以参考基于 Visual Studio 2019 Community CMake 编译指南完成相应的预测库编译和模型预测工作。
1.[Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
1.[Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf)
