introduction_en.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. PP-Vehicle introduction\n",
    "PaddleDetection has provide out-of-the-box tools in pedestrian and vehicle analysis, and it support multiple input format such as images/videos/multi-videos/online video streams. This make it popular in smart-city\\smart transportation and so on. It can be deployed easily with GPU server and TensorRT, which achieves real-time performace\n",
    "\n",
    "PP-Vehicle has four major toolbox for vehicle analysis: The license plate recognition、vechile attributes、in-out counting、illegal_parking recognition.\n",
    "\n",
    "PP-Vehicle is officially produced by PaddlePaddle，is a vehicle analysis pipeline based on PaddleDetection.For more information about PaddleDetection, click https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/deploy/pipeline 。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Model effects and application scenarios\n",
    "### 2.1 PP-Vehicle Model effects:\n",
    "\n",
    "| Task                                   | End-to-End Speed（ms） | Model                                                                                                                                                                                                                                                                                                                           | Size                                                                                                   |\n",
    "|:--------------------------------------:|:--------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------:|\n",
    "| Vehicle detection (high precision)  | 25.7ms               | [object detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip)                                                                                                                                                                                                                      | 182M                                                                                                   |\n",
    "| Vehicle detection (lightweight)     | 13.2ms               | [object detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_ppvehicle.zip)                                                                                                                                                                                                                      | 27M                                                                                                    |\n",
    "| Vehicle tracking (high precision)   | 40ms               | [multi-object tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip)                                                                                                                                                                                                                      | 182M                                                                                                   |\n",
    "| Vehicle tracking (lightweight)      | 25ms               | [multi-object tracking](https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip)                                                                                                                                                                                                                      | 27M                                                                                                    |\n",
    "| Plate Recognition                   | 4.68ms     | [plate detection](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_det_infer.tar.gz)<br>[plate recognition](https://bj.bcebos.com/v1/paddledet/models/pipeline/ch_PP-OCRv3_rec_infer.tar.gz)                                                                                         | Plate detection：3.9M<br>Plate recognition：12M                                    |\n",
    "| Vehicle attribute      | 7.31ms               | [attribute recognition](https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip)                                                                                                                                                                                                                      | 7.2M                                                                                                    |\n",
    "\n",
    "### 2.2 application scenarios：\n",
    "| Feature       |Advantages                                                                                    |Example                                                                                                                                         |\n",
    "| ---------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |\n",
    "| **License Plate-Recognition**   | Both support for traditional plate and new green plate <br/><br/> Sample the frame in a time windows to recognice the plate license, and vots the license in many samples, which lead less compute cost and better accuracy, and the result is much more stable. <br/><br/>  hmean of text detector: 0.979;  accuracy of recognition model: 0.773   <br/><br/>                                  | <img title=\"\" src=\"https://user-images.githubusercontent.com/48054808/185027987-6144cafd-0286-4c32-8425-7ab9515d1ec3.png\" alt=\"\" width=\"191\"> |\n",
    "| **Vehicle Attributes** | Identify 10 vehicle colors and 9 models <br/><br/> More powerfull backbone: PP-HGNet/PP-LCNet, with higher accuracy and faster speed <br/><br/> accuracy of model: 90.81 <br/><br/>  | <img title=\"\" src=\"https://user-images.githubusercontent.com/48054808/185044490-00edd930-1885-4e79-b3d4-3a39a77dea93.gif\" alt=\"\" width=\"207\"> |\n",
    "| **Illegal Parking**   | Easy to use with one line command, and define the illegal area by yourself <br/><br/> Get the license of illegal car <br/><br/>  | <img title=\"\" src=\"https://user-images.githubusercontent.com/48054808/185028419-58ae0af8-a035-42e7-9583-25f5e4ce0169.png\" alt=\"\" width=\"209\"> |\n",
    "| **in-out counting**  | Easy to use with one line command, and define the in-out line by yourself <br/><br/> Target route visualize with high tracking performance        | <img title=\"\" src=\"https://user-images.githubusercontent.com/48054808/185028798-9e07379f-7486-4266-9d27-3aec943593e0.gif\" alt=\"\" width=\"200\"> |\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. How to use the model\n",
    "\n",
    "（You need to add \"!\" when running on Jupyter Notebook, Add \"%\" if cd command）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# clone PaddleDetection\n",
    "%cd ~/work\n",
    "!git clone https://github.com/PaddlePaddle/PaddleDetection.git\n",
    "\n",
    "# Other Dependencies\n",
    "%cd PaddleDetection\n",
    "!pip install -r requirements.txt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### 3.1 Configuration\n",
    "\n",
    "PP-Vehicle related configuration locates in ``deploy/pipeline/config/infer_cfg_ppvehicle.yml``. Developers need to set specific task types to use different features.\n",
    "\n",
    "The features and corresponding task types are as follows.\n",
    "\n",
    "| Input               | Feature               | Task                                        | Config           |\n",
    "| ------------------- | --------------------- | ------------------------------------------- | ---------------- |\n",
    "| Image               | Attribute Recognition | Object Detection Attribute Recognition      | DET ATTR         |\n",
    "| Single-camera video | Attribute Recognition | Multi-Object Tracking Attribute Recognition | MOT ATTR         |\n",
    "| Single-camera video | License-plate Recognition | Multi-Object Tracking License-plate Recognition | MOT VEHICLEPLATE |\n",
    "\n",
    "Take attribute recognition based on video input as an example: Its task type includes multi-object tracking and attributes recognition. The specific configuration is as follows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "```\n",
    "crop_thresh: 0.5\n",
    "visual: True\n",
    "warmup_frame: 50\n",
    "\n",
    "MOT:\n",
    "  model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_l_36e_ppvehicle.zip\n",
    "  tracker_config: deploy/pipeline/config/tracker_config.yml\n",
    "  batch_size: 1\n",
    "  enable: True\n",
    "\n",
    "VEHICLE_ATTR:\n",
    "  model_dir: https://bj.bcebos.com/v1/paddledet/models/pipeline/vehicle_attribute_model.zip\n",
    "  batch_size: 8\n",
    "  color_threshold: 0.5\n",
    "  type_threshold: 0.5\n",
    "  enable: True\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notes:**\n",
    "\n",
    "- If the developer needs to carry out different tasks, set the corresponding enables option to be True in the configuration file.\n",
    "- If the developer only needs to modify the model file path, run the command line with `-o MOT.model_dir=ppyoloe/` after --config, or manually modify the corresponding model path in the configuration file. For more details, please refer to the following parameter descriptions\n",
    "\n",
    "### 3.2 Inference Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# 1. Use the default configuration directly or the configuration file in examples, or modify the configuration in `infer_cfg_ppvehicle.yml`\n",
    "# Example：In vehicle detection，specify configuration file path and test image\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml --image_file=test_image.jpg --device=gpu\n",
    "\n",
    "# Example：In license plate recognition，directly configure the examples\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_plate.yml --video_file=test_video.mp4 --device=gpu\n",
    "\n",
    "\n",
    "#2.Use the command line to enable functions or change the model path.\n",
    "# Example：In vehicle tracking，specify configuration file path and test video, Turn on the MOT model and modify the model path on the command line, the model path specified on the command line has higher priority than the configuration file\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/infer_cfg_ppvehicle.yml -o MOT.enable=True MOT.model_dir=ppyoloe_infer/ --video_file=test_video.mp4 --device=gpu\n",
    "\n",
    "# Example：In vehicle illegal action analysis，specify configuration file path and test video,Setting of designated violation area and judgment of violation time in the command line\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_illegal_parking.yml \\\n",
    "                                                   --video_file=../car_test.mov \\\n",
    "                                                   --device=gpu \\\n",
    "                                                   --draw_center_traj \\\n",
    "                                                   --illegal_parking_time=3 \\\n",
    "                                                   --region_type=custom \\\n",
    "                                                   --region_polygon 600 300 1300 300 1300 800 600 800\n",
    "\n",
    "#3. rtsp push/pull stream\n",
    "#For rtsp pull stream, use --rtsp RTSP [RTSP ...] parameter to specify one or more rtsp streams. Separate the multiple addresses with a space, or replace the video address directly after the video_file with the rtsp stream address), examples as follows\n",
    "\n",
    "# Example: Single video stream for pedestrian attribute recognition\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE]  --device=gpu\n",
    "\n",
    "# Example: Multiple-video stream for pedestrian attribute recognition\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_attr.yml -o visual=False --rtsp rtsp://[YOUR_RTSP_SITE1]  rtsp://[YOUR_RTSP_SITE2] --device=gpu\n",
    "\n",
    "\n",
    "#rtsp push stream\n",
    "#For rtsp push stream, use --pushurl rtsp:[IP] parameter to push stream to a IP set, and you can visualize the output video by [VLC Player](https://vlc.onl/) with the `open network` funciton. the whole url path is `rtsp:[IP]/videoname`, the videoname here is the basename of the video file to infer, and the default of videoname is `output` when the video come from local camera and there is no video name. \n",
    "#Example：license plate recognition，in this example the whole url path is: rtsp://[YOUR_SERVER_IP]:8554/test_video\n",
    "!python deploy/pipeline/pipeline.py --config deploy/pipeline/config/examples/infer_cfg_vehicle_attr.yml -o visual=False --video_file=test_video.mp4  --device=gpu --pushurl rtsp://[YOUR_SERVER_IP]:8554\n",
    "\n",
    "#Note: \n",
    "#1. rtsp push stream is based on [rtsp-simple-server](https://github.com/aler9/rtsp-simple-server), please enable this serving first.\n",
    "#2. the output visualize will be frozen frequently if the model cost too much time, we suggest to use faster model like ppyoloe_s in tracking, this is simply replace mot_ppyoloe_l_36e_pipeline.zip with mot_ppyoloe_s_36e_pipeline.zip in model config yaml file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 3.3 Jetson Deployment\n",
    "\n",
    "Due to the large gap in computing power of the Jetson platform compared to the server, we suggest:\n",
    "\n",
    "1. choose a lightweight model, especially for tracking model, `ppyoloe_s: https://bj.bcebos.com/v1/paddledet/models/pipeline/mot_ppyoloe_s_36e_pipeline.zip` is recommended\n",
    "2. For frame skipping of tracking; we recommend 2 or 3: `skip_frame_num: 3`\n",
    "\n",
    "With this recommended configuration, it is possible to achieve higher speeds on the TX2 platform. It has been tested with attribute case, with speeds up to 20fps. The configuration file can be modified directly (recommended) or from the command line (not recommended due to its long fields).\n",
    "\n",
    "### Parameters\n",
    "\n",
    "| Parameters             | Necessity | Implications                                                                                                                                                                                                                                      |\n",
    "| ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n",
    "| --config               | Yes       | Path to configuration file                                                                                                                                                                                                                        |\n",
    "| -o                     | Option    | Overwrite the corresponding configuration in the configuration file                                                                                                                                                                               |\n",
    "| --image_file           | Option    | Images to be predicted                                                                                                                                                                                                                            |\n",
    "| --image_dir            | Option    | Path to the images folder to be predicted                                                                                                                                                                                                         |\n",
    "| --video_file           | Option    | Video to be predicted, or rtsp stream address (rtsp parameter recommended)                                                                                                                                                                        |\n",
    "| --rtsp                 | Option    | rtsp video stream address, supports one or more simultaneous streams input                                                                                                                                                                        |\n",
    "| --camera_id            | Option    | The camera ID for prediction, default is -1 ( for no camera prediction, can be set to 0 - (number of cameras - 1) ), press `q` in the visualization interface during the prediction process to output the prediction result to: output/output.mp4 |\n",
    "| --device               | Option    | Running device, options include `CPU/GPU/XPU`, and the default is `CPU`.                                                                                                                                                                          |\n",
    "| --pushurl              | Option    | push the output video to rtsp stream, normaly start with `rtsp://`; this has higher priority than local video save, while this is set, pipeline will not save local visualize video, the default is \"\", means this will not work now.\n",
    "                |\n",
    "| --output_dir           | Option    | The root directory for the visualization results, and the default is output/                                                                                                                                                                      |\n",
    "| --run_mode             | Option    | For GPU, the default is paddle, with (paddle/trt_fp32/trt_fp16/trt_int8) as optional                                                                                                                                                              |\n",
    "| --enable_mkldnn        | Option    | Whether to enable MKLDNN acceleration in CPU prediction, the default is False                                                                                                                                                                     |\n",
    "| --cpu_threads          | Option    | Set the number of cpu threads, and the default is 1                                                                                                                                                                                               |\n",
    "| --trt_calib_mode       | Option    | Whether TensorRT uses the calibration function, and the default is False; set to True when using TensorRT's int8 function and False when using the PaddleSlim quantized model                                                                     |\n",
    "| --do_entrance_counting | Option    | Whether to count entrance/exit traffic flows, the default is False                                                                                                                                                                                |\n",
    "| --draw_center_traj     | Option    | Whether to draw center trajectory, the default is False                                                                                                                                                                                           |\n",
    "| --region_type          | Option    | 'horizontal' (default), 'vertical': traffic count direction; 'custom': set illegal parking area                                                                                                                                                   |\n",
    "| --region_polygon       | Option    | Set the coordinates of the polygon multipoint in the illegal parking area. No default.                                                                                                                                                            |\n",
    "| --illegal_parking_time | Option    | Set the time threshold for illegal parking in seconds (s), -1 (default) indicates no check                                                                                                                                                        |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Solutions\n",
    "The overall solution for PP-Vehicle is shown in the graph below:\n",
    "\n",
    "<div width=\"1000\" align=\"center\">\n",
    "  <img src=\"ppvehicle.png\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "### Vehicle detection\n",
    "- Take PP-YOLOE L as the object detection model\n",
    "- - For detailed documentation, please refer to(https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/configs/ppyoloe) and [Multiple-Object-Tracking](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/deploy/pipeline/docs/tutorials/ppvehicle_mot_en.md)\n",
    "\n",
    "### Vehicle tracking\n",
    "- Vehicle tracking by SDE solution\n",
    "- Adopt PP-YOLOE L (high precision) and S (lightweight) for detection models\n",
    "- Adopt the OC-SORT solution for racking module\n",
    "- efer to[OC-SORT](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/configs/mot/ocsort) and [Multiple-Object-Tracking](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/deploy/pipeline/docs/tutorials/ppvehicle_mot_en.md)\n",
    "\n",
    "### Attribute Recognition\n",
    "- Use PP-LCNet provided by PaddleClas to recognize vehicle colours and model attributes.\n",
    "- For details, please refer to [Attribute Recognition](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/deploy/pipeline/docs/tutorials/ppvehicle_attribute_en.md)\n",
    "\n",
    "### License plate recognition\n",
    "- Use ch_PP-OCRv3_det+ch_PP-OCRv3_rec model to recognize license plate number\n",
    "- For details, please refer to [Plate Recognition](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/deploy/pipeline/docs/tutorials/ppvehicle_plate_en.md)\n",
    "\n",
    "### Illegal Parking Detection\n",
    "- Use vehicle tracking model (high precision) PP-YOLOE L to determine whether the parking is illegal based on the vehicle's trajectory and the designated illegal parking area. If it is illegal parking, display the illegal parking plate number.\n",
    "\n",
    "- For details, please refer to [Illegal Parking Detection](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/deploy/pipeline/docs/tutorials/ppvehicle_illegal_parking_en.md)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}