As a submodule of PaddlePaddle framework, PaddleSlim is an open-source library for deep model compression and architecture search. PaddleSlim supports current popular deep compression techniques such as pruning, quantization, and knowledge distillation. Further, it also automates the search of hyperparameters and the design of lightweight deep architectures. In the future, we will develop more practically useful compression techniques for industrial-level applications and transfer these techniques to models in NLP.
## Outline
- Key Features
- Architecture of PaddleSlim
- Methods
- Experimental Results
## Key Features
The main key features of PaddleSlim are:
### Simple APIs
- It provides simple APIs for building and deploying lightweight and energy-efficient deep models on different platforms. Experimental hyperparameters can be set up by a simple configuration file.
- It requires just a little coding work for a model compression.
### Outstanding Performance
- For MobileNetV1 with limited redundancy, channel-based pruning can ensure lossless compression.
- Knowledge distillation can promote the performance of baseline models with a clear margin.
- Quantization after knowledge distillation can reduce model size and increase accuracy of models.
### Flexible APIs
- We automate the pruning process.
- Pruning strategy can be applied onto various deep architectures.
- We can distill multiple kinds of knowledge from teacher models to student models and self-defined losses for the corresponding knowledge distillation are supported.
- We support the deployment of multiple compression strategies.
## Architecture of PaddleSlim
To make the usage of PaddleSlim clearer and easier, we briefly introduce the background of how to implement the library.
The architecture of PaddleSlim is demonstrated in **Figure 1**. The high-level APIs often depend on several low-level APIs. We can see, knowledge distillation, quantization, and pruning are indirectly based on the Paddle framework. Currently, as a part of PaddlePaddle, user can use PaddleSlim for model compression and search after kindly download and install Paddle framework.
As shown in **Figure 1**, the top-level module, marked as yellow, is the API exposed to users. When we deploy compression methods in Python, we only need to construct an instance of Compressor.
PaddleSlim is a toolkit for model compression. It contains a collection of compression strategies, such as pruning, fixed point quantization, knowledge distillation, hyperparameter searching and neural architecture search.
We encapsulate each compression and search method to a compression strategy class. When we train the deep model to be compressed, the strategy class will be instantiated by using the configuration information registered by users, as shown in **Figure 2**. The logic of training process is encapsulated in our compression method. The jobs that users should do by themself is to define the structure of deep models, to prepare the training data, and to choose optimization strategy. This would surely help users save much effort.
PaddleSlim provides solutions of compression on computer vision models, such as image classification, object detection and semantic segmentation. Meanwhile, PaddleSlim Keeps exploring advanced compression strategies for language model. Furthermore, benckmark of compression strategies on some open tasks is available for your reference.
PaddleSlim also provides auxiliary and primitive API for developer and researcher to survey, implement and apply the method in latest papers. PaddleSlim will support developer in ability of framework and technology consulting.
## Methods
## Features
### Pruning
- Here, PaddleSlim supports uniform prunning, sensitivity-based prunning, and automated model pruning methods.
- PaddleSlim supports pruning of various deep architectures such as VGG, ResNet, and MobileNet.
- PaddleSlim supports self-defined range of pruning, i.e., layers to be pruned.
- Uniform pruning of convolution
- Sensitivity-based prunning
- Automated pruning based evolution search strategy
- Support pruning of various deep architectures such as VGG, ResNet, and MobileNet.
- Support self-defined range of pruning, i.e., layers to be pruned.
### Quantization
### Fixed Point Quantization
- PaddleSlim supports training-aware quantization with static and dynamic estimation of quantization hyperparameters such as scale.
- Dynamic strategy: During inference, we quantize models with hyperparameters dynamically estimated from small batches of samples.
- Static strategy: During inference, we quantize models with the same hyperparameters estimated from training data.
- PaddleSlim supports layer-wise and channel-wise quantization.
- PaddleSlim provides models compatible with Paddle Mobile for final inference.
-**Training aware**
- Dynamic strategy: During inference, we quantize models with hyperparameters dynamically estimated from small batches of samples.
- Static strategy: During inference, we quantize models with the same hyperparameters estimated from training data.
- Support layer-wise and channel-wise quantization.
-**Post training**
### Knowledge Distillation
-**Naive knowledge distillation**: transfers dark knowledge by merging the teacher and student model into the same Program, and supports the following losses added on any paired layers between teacher and student models:
- Flow of the solution procedure (FSP) loss;
- L2 loss;
- Softmax with cross-entropy loss.
-**Paddle large-scale scalable knowledge distillation framework [Pantheon](paddleslim/pantheon)**: a universal solution for knowledge distillation, more flexible than the naive knowledge distillation, and easier to scale to the large-scale applications.
- Decouple the teacher and student models --- they run in different processes in the same or different nodes, and transfer knowledge via TCP/IP ports or local files;
- Friendly to assemble multiple teacher models and each of them can work in either online or offline mode independently;
- Merge knowledge from different teachers and make batch data for the student model automatically;
- Support the large-scale knowledge prediction of teacher models on multiple devices.
- PaddleSlim supports FLOPs and latency constrained search.
- PaddleSlim supports the latency estimation on different hardware and platforms.
## Experimental Results
In this section, we will show some experimental results conducted on PaddleSlim.
### Quantization
-**Naive knowledge distillation:** transfers dark knowledge by merging the teacher and student model into the same Program
-**Paddle large-scale scalable knowledge distillation framework Pantheon:** a universal solution for knowledge distillation, more flexible than the naive knowledge distillation, and easier to scale to the large-scale applications.
We evaluate the quantized models on ImageNet2012. The top-5/top-1 accuracies are compared,
Note: abs_max refers to dynamic strategy; moving_average_abs_max refers to static strategy; channel_wise_abs_max refers channel-wise quantization for weights in convolutional layers.
### Pruning
- Decouple the teacher and student models --- they run in different processes in the same or different nodes, and transfer knowledge via TCP/IP ports or local files;
- Friendly to assemble multiple teacher models and each of them can work in either online or offline mode independently;
- Merge knowledge from different teachers and make batch data for the student model automatically;
- Support the large-scale knowledge prediction of teacher models on multiple devices.
Data: ImageNet2012
Baseline model: MobileNetV1
Model size: 17M
Top-5/top-1 accuracies: 89.54% / 70.91%
### Neural Architecture Search
#### Uniform pruning
- Neural architecture search based on evolution strategy.
- Support distributed search.
- One-Shot neural architecture search.
- Support FLOPs and latency constrained search.
- Support the latency estimation on different hardware and platforms.
