Mixed image data augmentation is similar to Mosaic and MixUp, in which the annotation information of multiple images needs to be fused during the runtime. In the OpenMMLab data augmentation pipeline, the other indexes of the dataset are generally not available. In order to achieve the above function, in the [MultiImageMixDataset](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/datasets/dataset_wrappers.py#L338) the concept of dataset wrapper is proposed in YOLOX, which is reproduced in MMDet.
`MultiImageMixDataset` dataset wrapper will include some data augmentation method such as `Mosaic` and `RandAffine`, while `CocoDataset` will also include the `pipeline` to achieve the img and annotation loading function. through this way we can achieve mix data augmentation quickly. The configuration method is as follows：
# use MultiImageMixDataset wrapper to support mosaic and mixup
type='MultiImageMixDataset',
dataset=dict(
type='CocoDataset',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations',with_bbox=True)
]),
pipeline=train_pipeline)
```
But above method will cause a problem: to the users who are not familiar with MMDet ,they will forget to match data augmentation methods like Mosaic together with `MultiImageMixDataset`, which could extremely increase the Complexity, and it could be hard to understand.
To solve this problem we make a simplification in MMYOLO, which directly make `pipeline` catch the `dataset` ，and make the data augmentation methods like `Mosaic` be achieved and used as random flip, without data wrapper anymore. The new configuration method is as follows:
```python
pre_transform=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations',with_bbox=True)
]
train_pipeline=[
*pre_transform,
dict(
type='Mosaic',
img_scale=img_scale,
pad_val=114.0,
pre_transform=pre_transform),
dict(
type='mmdet.RandomAffine',
scaling_ratio_range=(0.1,2),
border=(-img_scale[0]//2,-img_scale[1]//2)),
dict(
type='YOLOXMixUp',
img_scale=img_scale,
ratio_range=(0.8,1.6),
pad_val=114.0,
pre_transform=pre_transform),
...
]
```
A more complex YOLOv5-m configuration include MixUp is shown as follows:
It is very easy to be achieved, just pass the object of Dataset to the pipeline.
```python
defprepare_data(self,idx)->Any:
"""Pass the dataset to the pipeline during training to support mixed
data augmentation, such as Mosaic and MixUp."""
ifself.test_modeisFalse:
data_info=self.get_data_info(idx)
data_info['dataset']=self
returnself.pipeline(data_info)
else:
returnsuper().prepare_data(idx)
```
# Mixed image data augmentation update
Mixed image data augmentation is similar to Mosaic and MixUp, in which the annotation information of multiple images needs to be fused during the runtime. In the OpenMMLab data augmentation pipeline, the other indexes of the dataset are generally not available. In order to achieve the above function, in the [MultiImageMixDataset](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/datasets/dataset_wrappers.py#L338) the concept of dataset wrapper is proposed in YOLOX, which is reproduced in MMDet.
`MultiImageMixDataset` dataset wrapper will include some data augmentation method such as `Mosaic` and `RandAffine`, while `CocoDataset` will also include the `pipeline` to achieve the img and annotation loading function. through this way we can achieve mix data augmentation quickly. The configuration method is as follows：
# use MultiImageMixDataset wrapper to support mosaic and mixup
type='MultiImageMixDataset',
dataset=dict(
type='CocoDataset',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations',with_bbox=True)
]),
pipeline=train_pipeline)
```
But above method will cause a problem: to the users who are not familiar with MMDet ,they will forget to match data augmentation methods like Mosaic together with `MultiImageMixDataset`, which could extremely increase the Complexity, and it could be hard to understand.
To solve this problem we make a simplification in MMYOLO, which directly make `pipeline` catch the `dataset` ，and make the data augmentation methods like `Mosaic` be achieved and used as random flip, without data wrapper anymore. The new configuration method is as follows:
```python
pre_transform=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations',with_bbox=True)
]
train_pipeline=[
*pre_transform,
dict(
type='Mosaic',
img_scale=img_scale,
pad_val=114.0,
pre_transform=pre_transform),
dict(
type='mmdet.RandomAffine',
scaling_ratio_range=(0.1,2),
border=(-img_scale[0]//2,-img_scale[1]//2)),
dict(
type='YOLOXMixUp',
img_scale=img_scale,
ratio_range=(0.8,1.6),
pad_val=114.0,
pre_transform=pre_transform),
...
]
```
A more complex YOLOv5-m configuration include MixUp is shown as follows:
Most of the YOLO series algorithms adopt a unified algorithm building structure, typically as Darknet + PAFPN. In order to let users quickly understand the YOLO series algorithm architecture, we deliberately designed the BaseBackbone + BaseYOLONeck structure as shown in the above graph.
The benefit of abstract BaseBackbone includes:
1. Subclasses do not need to concern about the forward process, just build the model as the builder pattern.
2. It can be configured to achieve custom plug-in functions, the users can easily insert some similar attention module.
3. All subclasses automatically support frozen certain stage and frozen bn functions.
BaseYOLONeck has the same benefit as BaseBackbone.
### BaseBackbone
We can see in the above graph，as for P5，BaseBackbone include 1 stem layer and 4 stage layers which are similar to the basic structural of ResNet. Different backbone network algorithms inheritance the BaseBackbone, users can achieve construction of every layer of the network by using self-custom basic module through `build_xx` method.
### BaseYOLONeck
We reproduce the YOLO series Neck component by the similar method of the BaseBackbone, we can mainly divide them into Reduce layer, UpSample layer,TopDown layer,DownSample layer，BottomUP layer and output convolution layer, every layer can self-custom its inside construction by inheritance and rewrite `build_xx` method.
### BaseDenseHead
The YOLO series uses the BaseDenseHead designed in MMDet as the base class of the Head structure.[HeadModule](#headmodule)base class's forward function replace original forward method. Different algorithms only need to inheritance and rewrite [HeadModule](#headmodule)，`loss_by_feat` and `predict_by_feat` method.
Methods implementation in the [MMDetection](https://github.com/open-mmlab/mmdetection) is shown in the above graph ，The solid line is the implementation in [MMDet-YOLO](<>), which has the following advantages over the original implementation:
1. MMDet in the bbox_head split into assigner + box coder + sampler three large components, but for the generality of passing through the 3 components , the model need to encapsulate additional objects to handle, and after the unification, the user needn't separate them. The benefits of not deliberately forcing the division of the three components are: no longer need to data encapsulation of internal data, simplifying the code logic, reducing the difficulty of use and the difficulty of algorithm implementation.
2. MMYOLO is Faster, the user can customize the implementation of the algorithm when the original framework does not depend on the deep optimization of part of the code.
In general, in the MMYOLO, they only need to implement the decouple of the model + loss_by_feat parts, and users can achieve any model with any `loss_by_feat` calculation process through modify the configuration. For example, applying the YOLOX `loss_by_feat` to the YOLOv5 model, etc.
Taking the YOLOX configuration in MMDet as an example, the Head module configuration is written as follows:
Most of the YOLO series algorithms adopt a unified algorithm building structure, typically as Darknet + PAFPN. In order to let users quickly understand the YOLO series algorithm architecture, we deliberately designed the BaseBackbone + BaseYOLONeck structure as shown in the above graph.
The benefit of abstract BaseBackbone includes:
1. Subclasses do not need to concern about the forward process, just build the model as the builder pattern.
2. It can be configured to achieve custom plug-in functions, the users can easily insert some similar attention module.
3. All subclasses automatically support frozen certain stage and frozen bn functions.
BaseYOLONeck has the same benefit as BaseBackbone.
### BaseBackbone
We can see in the above graph，as for P5，BaseBackbone include 1 stem layer and 4 stage layers which are similar to the basic structural of ResNet. Different backbone network algorithms inheritance the BaseBackbone, users can achieve construction of every layer of the network by using self-custom basic module through `build_xx` method.
### BaseYOLONeck
We reproduce the YOLO series Neck component by the similar method of the BaseBackbone, we can mainly divide them into Reduce layer, UpSample layer,TopDown layer,DownSample layer，BottomUP layer and output convolution layer, every layer can self-custom its inside construction by inheritance and rewrite `build_xx` method.
### BaseDenseHead
The YOLO series uses the BaseDenseHead designed in MMDet as the base class of the Head structure.[HeadModule](#headmodule)base class's forward function replace original forward method. Different algorithms only need to inheritance and rewrite [HeadModule](#headmodule)，`loss_by_feat` and `predict_by_feat` method.
Methods implementation in the [MMDetection](https://github.com/open-mmlab/mmdetection) is shown in the above graph ，The solid line is the implementation in [MMDet-YOLO](<>), which has the following advantages over the original implementation:
1. MMDet in the bbox_head split into assigner + box coder + sampler three large components, but for the generality of passing through the 3 components , the model need to encapsulate additional objects to handle, and after the unification, the user needn't separate them. The benefits of not deliberately forcing the division of the three components are: no longer need to data encapsulation of internal data, simplifying the code logic, reducing the difficulty of use and the difficulty of algorithm implementation.
2. MMYOLO is Faster, the user can customize the implementation of the algorithm when the original framework does not depend on the deep optimization of part of the code.
In general, in the MMYOLO, they only need to implement the decouple of the model + loss_by_feat parts, and users can achieve any model with any `loss_by_feat` calculation process through modify the configuration. For example, applying the YOLOX `loss_by_feat` to the YOLOv5 model, etc.
Taking the YOLOX configuration in MMDet as an example, the Head module configuration is written as follows:
MMYOLO is a YOLO series algorithm toolbox, which currently implements only the target detection task and will subsequently support various tasks such as instance segmentation, panoramic segmentation and key point detection. It includes a rich set of target detection algorithms and related components and modules, and the following is its overall framework.
MMYOLO file structure is identical to the MMDetection. To allow full reuse of the MMDetection code, MMYOLO includes only custom content, which consists of 3 main parts:datasets、models、engine.
-**datasets** supports a variety of data sets for target detection
-**transforms** includes various data enhancement transforms
-**models** is the most important part of the detector, which include different component of it.
-**detectors** define all detection model classes
-**data_preprocessors** is used to preprocess the dataset of the model
-**backbones** include various backbone networks
-**necks** include various neck components
-**dense_heads** include various dense head of different tasks
-**losses** include various loss functions
-**task_modules** provide component for testing tasks,such as assigners、samplers、box coders and prior generators。
-**layers** provide some basic network layers
-**engine** is a component of running
-**optimizers** provide optimizers and packages for optimizers
-**hooks** provide hooks for runner
## How to use this tutorial
The detailed instruction of MMYOLO is as following
1. Look up install instruction through [start your first step](get_started.md)
2. Basic method of how to use MMYOLO can be found here:
-[Training and testing](https://mmyolo.readthedocs.io/en/latest/user_guides/index.html#train-test)
-[From getting started to deployment tutorial](https://mmyolo.readthedocs.io/en/latest/user_guides/index.html#from-getting-started-to-deployment-tutorial)
-[Full explanation of the model and implementation](https://mmyolo.readthedocs.io/en/latest/algorithm_descriptions/index.html#algorithm-principles-and-implementation)
4. Refer to the following tutorials for an in-depth look:
MMYOLO is a YOLO series algorithm toolbox, which currently implements only the target detection task and will subsequently support various tasks such as instance segmentation, panoramic segmentation and key point detection. It includes a rich set of target detection algorithms and related components and modules, and the following is its overall framework.
MMYOLO file structure is identical to the MMDetection. To allow full reuse of the MMDetection code, MMYOLO includes only custom content, which consists of 3 main parts:datasets、models、engine.
-**datasets** supports a variety of data sets for target detection
-**transforms** includes various data enhancement transforms
-**models** is the most important part of the detector, which include different component of it.
-**detectors** define all detection model classes
-**data_preprocessors** is used to preprocess the dataset of the model
-**backbones** include various backbone networks
-**necks** include various neck components
-**dense_heads** include various dense head of different tasks
-**losses** include various loss functions
-**task_modules** provide component for testing tasks,such as assigners、samplers、box coders and prior generators。
-**layers** provide some basic network layers
-**engine** is a component of running
-**optimizers** provide optimizers and packages for optimizers
-**hooks** provide hooks for runner
## How to use this tutorial
The detailed instruction of MMYOLO is as following
1. Look up install instruction through [start your first step](get_started.md)
2. Basic method of how to use MMYOLO can be found here:
-[Training and testing](https://mmyolo.readthedocs.io/en/latest/user_guides/index.html#train-test)
-[From getting started to deployment tutorial](https://mmyolo.readthedocs.io/en/latest/user_guides/index.html#from-getting-started-to-deployment-tutorial)
-[Full explanation of the model and implementation](https://mmyolo.readthedocs.io/en/latest/algorithm_descriptions/index.html#algorithm-principles-and-implementation)
4. Refer to the following tutorials for an in-depth look:
MMYOLO and other OpenMMLab repositories use [MMEngine's config system](https://mmengine.readthedocs.io/en/latest/tutorials/config.html). It has a modular and inheritance design, which is convenient to conduct various experiments.
...
...
@@ -46,7 +46,7 @@ model = dict(
norm_cfg=dict(type='BN',momentum=0.03,eps=0.001),# The config of normalization layers.
act_cfg=dict(type='SiLU',inplace=True)),# The config of activation function
neck=dict(
type='YOLOv5PAFPN',# The neck of detector is YOLOv5FPN，We also support 'YOLOv6RepPAFPN', 'YOLOXPAFPN'.
type='YOLOv5PAFPN',# The neck of detector is YOLOv5FPN，We also support 'YOLOv6RepPAFPN', 'YOLOXPAFPN'.
deepen_factor=deepen_factor,# The scaling factor that controls the depth of the network structure
widen_factor=widen_factor,# The scaling factor that controls the width of the network structure
in_channels=[256,512,1024],# The input channels, this is consistent with the output channels of backbone
...
...
@@ -71,7 +71,7 @@ model = dict(
test_cfg=dict(
multi_label=True,# The config of multi-label for multi-clas prediction. THe default setting is True.
nms_pre=30000,# The number of boxes before NMS
score_thr=0.001,# Threshold to filter out boxes.
score_thr=0.001,# Threshold to filter out boxes.
nms=dict(type='nms',# Type of NMS
iou_threshold=0.65),# NMS threshold
max_per_img=300))# Max number of detections of each image
In the testing phase of YOLOv5, the letterbox resize method resizes all the test images to the same scale, which preserves the aspect ratio of all testing images. Therefore, the validation and testing phases share the same data pipeline.
In the testing phase of YOLOv5, the letterbox resize method resizes all the test images to the same scale, which preserves the aspect ratio of all testing images. Therefore, the validation and testing phases share the same data pipeline.
test_mode=True,# # Turn on test mode of the dataset to avoid filtering annotations or images
data_prefix=dict(img='val2017/'),# Prefix of image path
ann_file='annotations/instances_val2017.json',# Path of annotation file
pipeline=test_pipeline,
pipeline=test_pipeline,
batch_shapes_cfg=dict(# Config of batch shapes
type='BatchShapePolicy',# Policy that makes paddings with least pixels during batch inference process, which does not require the image scales of all batches to be the same throughout validation.
batch_size=val_batch_size_pre_gpu,# Batch size for batch shapes strategy, equals to validation batch size on single GPU
...
...
@@ -253,7 +253,7 @@ val_cfg = dict(type='ValLoop') # The validation loop type
test_cfg=dict(type='TestLoop')# The testing loop type
```
MMEngine also support dynamic intervals for evaluation. For example, you can run validation every 10 epochs on the first 280 epochs, and run validation every epoch on the final 20 epochs. The configurations are as follows.
MMEngine also support dynamic intervals for evaluation. For example, you can run validation every 10 epochs on the first 280 epochs, and run validation every epoch on the final 20 epochs. The configurations are as follows.
```python
max_epochs=300# Maximum training epochs: 300 epochs
...
...
@@ -352,7 +352,7 @@ resume = False # Whether to resume from the checkpoint defined in `load_from`.
## Config file inheritance
`config/_base_` contains default runtime. The configs that are composed by components from `_base_` are called _primitive_.
`config/_base_` contains default runtime. The configs that are composed by components from `_base_` are called _primitive_.
For all configs under the same folder, it is recommended to have only **one** _primitive_ config. All other configs should inherit from the _primitive_ config. In this way, the maximum of inheritance level is 3.
...
...
@@ -421,7 +421,7 @@ model = dict(
### Use intermediate variables in configs
Some intermediate variables are used in the configs files, like `train_pipeline`/`test_pipeline` in datasets. It's worth noting that when modifying intermediate variables in the children configs, user need to pass the intermediate variables into corresponding fields again.
Some intermediate variables are used in the configs files, like `train_pipeline`/`test_pipeline` in datasets. It's worth noting that when modifying intermediate variables in the children configs, user need to pass the intermediate variables into corresponding fields again.
For example, we would like to change the `image_scale` during training and add `YOLOv5MixUp` data augmentation, `img_scale/train_pipeline/test_pipeline` are intermediate variable we would like modify.
**Notice**：`YOLOv5MixUp` requires adding the `pre_transform` and `mosaic_affine_pipeline` to its own `pre_transform` field. Please refer to [The description of YOLOv5 algorithm and its implementation](../algorithm_descriptions/yolov5_description.md) for detailed process and diagrams.
...
...
@@ -430,7 +430,7 @@ For example, we would like to change the `image_scale` during training and add `
_base_='./yolov5_s-v61_syncbn_8xb16-300e_coco.py'
img_scale=(1280,1280)# image height, image width
affine_scale=0.9
affine_scale=0.9
mosaic_affine_pipeline=[
dict(
...
...
@@ -511,7 +511,6 @@ model = dict(
If the users want to reuse the variables in the base file, they can get a copy of the corresponding variable by using `{{_base_.xxx}}`. The latest version of MMEngine also support reusing variables without `{{}}` usage. E.g:
```python
_base_='../_base_/default_runtime.py'
...
...
@@ -550,7 +549,7 @@ The file name is divided to 8 name fields, which has 4 required parts and 4 opti
-`{algorithm name}`: The name of the algorithm. It can be a detector name such as `yolov5`, `yolov6`, `yolox` etc.
-`{component names}`: Names of the components used in the algorithm such as backbone, neck, etc. For example, `yolov5_s` means its `deepen_factor` is `0.33` and its `widen_factor` is `0.5`.
-`[version_id]` (optional): Since the evolution of YOLO series is much faster than traditional object detection algorithms, `version id` is used to distinguish the differences between different sub-versions. E.g, YOLOv5-3.0 uses the `Focus` layer as the stem layer, and YOLOv5-6.0 uses the `Conv` layer as the stem layer.
-`[version_id]` (optional): Since the evolution of YOLO series is much faster than traditional object detection algorithms, `version id` is used to distinguish the differences between different sub-versions. E.g, YOLOv5-3.0 uses the `Focus` layer as the stem layer, and YOLOv5-6.0 uses the `Conv` layer as the stem layer.
-`[data preprocessor type]` (optional): `fast` incorporates `YOLOv5DetDataPreprocessor` and `yolov5_collate` to preprocess data. The training speed is faster than default `mmdet.DetDataPreprocessor`, while results in extending tge overall pipeline to multi-task learning.
-`{training settings}`: Information of training settings such as batch size, augmentations, loss trick, scheduler, and epochs/iterations. For example: `8xb16-300e_coco` means using 8-gpus x 16-images-per-gpu, and train 300 epochs.
