The table about hardware latency is used to evaluate the inference time in special environment and inference engine. The following text used to introduce the format that PaddleSlim support.
## Introduce
The table about hardware latency saved all possible operations, one operation in the table including type and parameters, such as: type can be `conv2d`, and corresponding parameters can be the size of feature map, number of kernel, and the size of kernel.
The latency of every operation depends on hardware and inference engine.
## Overview format
The table about hardware latency saved in the way of file or multi-line string.
The first line of the table about hardware latency saved the information about version, every line in the following represents a operation and its latency.
## Version
The information about version split by comma in the english format, and the detail is hardware, inference engine and timestamp.
- ** hardware: ** Used to mark the environment of hardware, including type of architecture, version and so on.
- ** inference engine: ** Used to mark inference engine, including the name of inference engine, version, optimize options and so on.
- ** timestamp: ** Used to mark the time of this table created.
## Operation
The information about operation split by comma in the english format, the information about operation and latency split by tabs.
PaddleSlim is a toolkit for model compression. It contains a collection of compression strategies, such as pruning, fixed point quantization, knowledge distillation, hyperparameter searching and neural architecture search.
# Introduction
PaddleSlim provides solutions of compression on computer vision models, such as image classification, object detection and semantic segmentation. Meanwhile, PaddleSlim Keeps exploring advanced compression strategies for language model. Furthermore, benckmark of compression strategies on some open tasks is available for your reference.
PaddleSlim also provides auxiliary and primitive API for developer and researcher to survey, implement and apply the method in latest papers. PaddleSlim will support developer in ability of framework and technology consulting.
## Features
### Pruning
- Uniform pruning of convolution
- Sensitivity-based prunning
- Automated pruning based evolution search strategy
- Support pruning of various deep architectures such as VGG, ResNet, and MobileNet.
- Support self-defined range of pruning, i.e., layers to be pruned.
### Fixed Point Quantization
- **Training aware**
- Dynamic strategy: During inference, we quantize models with hyperparameters dynamically estimated from small batches of samples.
- Static strategy: During inference, we quantize models with the same hyperparameters estimated from training data.
- Support layer-wise and channel-wise quantization.
- **Post training**
### Knowledge Distillation
- **Naive knowledge distillation:** transfers dark knowledge by merging the teacher and student model into the same Program
- **Paddle large-scale scalable knowledge distillation framework Pantheon:** a universal solution for knowledge distillation, more flexible than the naive knowledge distillation, and easier to scale to the large-scale applications.
- Decouple the teacher and student models --- they run in different processes in the same or different nodes, and transfer knowledge via TCP/IP ports or local files;
- Friendly to assemble multiple teacher models and each of them can work in either online or offline mode independently;
- Merge knowledge from different teachers and make batch data for the student model automatically;
- Support the large-scale knowledge prediction of teacher models on multiple devices.
### Neural Architecture Search
- Neural architecture search based on evolution strategy.
- Support distributed search.
- One-Shot neural architecture search.
- Support FLOPs and latency constrained search.
- Support the latency estimation on different hardware and platforms.
As a submodule of PaddlePaddle framework, PaddleSlim is an open-source library for deep model compression and architecture search. PaddleSlim supports current popular deep compression techniques such as pruning, quantization, and knowledge distillation. Further, it also automates the search of hyperparameters and the design of lightweight deep architectures. In the future, we will develop more practically useful compression techniques for industrial-level applications and transfer these techniques to models in NLP.
| Method | mAP(baseline: 76.2%) | Model Size(baseline: 94MB) |
Note: The `_vd` suffix indicates that the pre-trained model uses Mixup. Please refer to the detailed introduction: [mixup: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.09412)
Note: latency is based on latency_855.txt, the file is test on 855 by PaddleLite。The config of BlazeFace-NASV2 is in [there](https://github.com/PaddlePaddle/PaddleDetection/blob/master/configs/face_detection/blazeface_nas_v2.yml).
# Pruning of image classification model - sensitivity
In this tutorial, you will learn how to use [sensitivity API of PaddleSlim](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity) by a demo of MobileNetV1 model on MNIST dataset。
This tutorial following workflow:
1. Import dependency
2. Build model
3. Define data reader
4. Define function for test
5. Training model
6. Get names of parameter
7. Compute sensitivities
8. Pruning model
## 1. Import dependency
PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below:
```python
import paddle
import paddle.fluid as fluid
import paddleslim as slim
```
## 2. Build model
This section will build a classsification model based `MobileNetV1` for MNIST task. The shape of the input is `[1, 28, 28]` and the output number is 10.
To make the code simple, we define a function in package `paddleslim.models` to build classification model.
>Note：The functions in paddleslim.models is just used in tutorials or demos.
## 3 Define data reader
MNIST dataset is used for making the demo can be executed quickly. It defines some functions for downloading and reading MNIST dataset in package `paddle.dataset.mnist`.
Show as below：
```python
import paddle.dataset.mnist as reader
train_reader = paddle.batch(
reader.train(), batch_size=128, drop_last=True)
test_reader = paddle.batch(
reader.test(), batch_size=128, drop_last=True)
data_feeder = fluid.DataFeeder(inputs, place)
```
## 4. Define test function
To get the performance of model on test dataset after pruning a convolution layer, we define a test function as below:
```python
import numpy as np
def test(program):
acc_top1_ns = []
acc_top5_ns = []
for data in test_reader():
acc_top1_n, acc_top5_n, _ = exe.run(
program,
feed=data_feeder.feed(data),
fetch_list=outputs)
acc_top1_ns.append(np.mean(acc_top1_n))
acc_top5_ns.append(np.mean(acc_top5_n))
print("Final eva - acc_top1: {}; acc_top5: {}".format(
Sensitivity analysis is dependent on pretrained model. So we should train the model defined in section 2 for some epochs. One epoch training is enough for this simple demo while more epochs may be necessary for other model. Or you can load pretrained model from filesystem.
Training model as below:
```python
for data in train_reader():
acc1, acc5, loss = exe.run(train_program, feed=data_feeder.feed(data), fetch_list=outputs)
Get the performance using the test function defined in section 4:
```python
test(val_program)
```
## 6. Get names of parameters
```python
params = []
for param in train_program.global_block().all_parameters():
if "_sep_weights" in param.name:
params.append(param.name)
print(params)
params = params[:5]
```
## 7. Compute sensitivities
### 7.1 Compute in single process
Apply sensitivity analysis on pretrained model by calling [sensitivity API](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity).
The sensitivities will be appended into the file given by option `sensitivities_file` during computing.
The information in this file won`t be computed repeatedly.
Remove the file `sensitivities_0.data` in current directory:
```python
!rm -rf sensitivities_0.data
```
Apart from the parameters to be analyzed, it also support for setting the ratios that each convolutoin will be pruned.
If one model losses 90% accuracy on test dataset when its single convolution layer is pruned by 40%, then we can set `pruned_ratios` to `[0.1, 0.2, 0.3, 0.4]`.
The granularity of `pruned_ratios` should be small to get more reasonable sensitivities. But small granularity of `pruned_ratios` will slow down the computing.
```python
sens_0 = slim.prune.sensitivity(
val_program,
place,
params,
test,
sensitivities_file="sensitivities_0.data",
pruned_ratios=[0.1, 0.2])
print(sens_0)
```
### 7.2 Expand sensitivities
We can expand `pruned_ratios` to `[0.1, 0.2, 0.3]` based the sensitivities generated in section 7.1.
```python
sens_0 = slim.prune.sensitivity(
val_program,
place,
params,
test,
sensitivities_file="sensitivities_0.data",
pruned_ratios=[0.3])
print(sens_0)
```
### 7.3 Computing sensitivity in multi-process
The time cost of computing sensitivities is dependent on the count of parameters and the speed of model evaluation on test dataset. We can speed up computing by multi-process.
Split `pruned_ratios` into multi-process, and merge the sensitivities from multi-process.
#### 7.3.1 Computing in each process
We have compute the sensitivities when `pruned_ratios=[0.1, 0.2, 0.3]` and saved the sensitivities into file named `sensitivities_0.data`.
在另一个进程中，The we start a task by setting `pruned_ratios=[0.4]` in another process and save result into file named `sensitivities_1.data`. Show as below：
```python
sens_1 = slim.prune.sensitivity(
val_program,
place,
params,
test,
sensitivities_file="sensitivities_1.data",
pruned_ratios=[0.4])
print(sens_1)
```
#### 7.3.2 Load sensitivity file generated in multi-process
式中，$x$是待被量化的浮点值，$M$是待量化浮点数中的绝对值最大值。$left lfloor right rceil$表示将结果四舍五入到最近的整数。对于8bit量化，PaddleSlim采用<codeclass="docutils literal"><spanclass="pre">int8_t</span></code>，即$n=2^7=128$。$q$是量化得到的整数。
第二种是基于FSP的蒸馏方法（参考论文：<aclass="reference external"href="http://openaccess.thecvf.com/content_cvpr_2017/papers/Yim_A_Gift_From_CVPR_2017_paper.pdf">A Gift from Knowledge Distillation:
<dd>第一种是传统的蒸馏方法（参考论文：<aclass="reference external"href="https://arxiv.org/pdf/1503.02531.pdf">Distilling the Knowledge in a Neural Network</a>）</dd>
Fast Optimization, Network Minimization and Transfer Learning</a>）
第二种是基于FSP的蒸馏方法（参考论文：<ahref="#id23"><spanclass="problematic"id="id24">`</span></a>A Gift from Knowledge Distillation:</p>
</div></blockquote>
<dlclass="docutils">
<dt>Fast Optimization, Network Minimization and Transfer Learning <<aclass="reference external"href="http://openaccess.thecvf.com/content_cvpr_2017/papers/Yim_A_Gift_From_CVPR_2017_paper.pdf">http://openaccess.thecvf.com/content_cvpr_2017/papers/Yim_A_Gift_From_CVPR_2017_paper.pdf</a>>`_）</dt>
<li><aclass="reference external"href="https://media.nips.cc/Conferences/2015/tutorialslides/Dally-NIPS-Tutorial-2015.pdf">High-Performance Hardware for Machine Learning</a></li>
<li><aclass="reference external"href="https://media.nips.cc/Conferences/2015/tutorialslides/Dally-NIPS-Tutorial-2015.pdf">High-Performance Hardware for Machine Learning</a></li>
<li><aclass="reference external"href="https://arxiv.org/pdf/1806.08342.pdf">Quantizing deep convolutional networks for efficient inference: A whitepaper</a></li>
<li><aclass="reference external"href="https://arxiv.org/pdf/1806.08342.pdf">Quantizing deep convolutional networks for efficient inference: A whitepaper</a></li>
<li><aclass="reference external"href="https://arxiv.org/pdf/1608.08710.pdf">Pruning Filters for Efficient ConvNets</a></li>
<li><aclass="reference external"href="https://arxiv.org/pdf/1608.08710.pdf">Pruning Filters for Efficient ConvNets</a></li>
<p>: Start the knowledge service to generate and transfer knowledge data. In GPU mode, the devices to execute knowledge prediction will be determined by the
<dlclass="docutils">
environment variable <strong>FLAGS_selected_gpus</strong>, or by <strong>CUDA_VISIBLE_DEVICES</strong> if it is not set, and by <strong>CPU_NUM</strong> (default 1) in CPU mode. Only supported in static graph.</p>
<dt>: Start the knowledge service to generate and transfer knowledge data. In GPU mode, the devices to execute knowledge prediction will be determined by the</dt>
<p><strong>Args:</strong></p>
<dd><blockquoteclass="first">
<div>environment variable <strong>FLAGS_selected_gpus</strong>, or by <strong>CUDA_VISIBLE_DEVICES</strong> if it is not set, and by <strong>CPU_NUM</strong> (default 1) in CPU mode. Only supported in static graph.</div></blockquote>
<pclass="last"><strong>Args:</strong></p>
</dd>
</dl>
<ul>
<ul>
<li><pclass="first"><strong>feed_list (list):</strong> - A list of feed Variables or their names for the
<li><pclass="first"><strong>feed_list (list):</strong> - A list of feed Variables or their names for the
input teacher Program.</p>
.. code-block:: guess</p>
<blockquote>
<div><p>input teacher Program.</p>
</div></blockquote>
</li>
</li>
<li><pclass="first"><strong>schema (dict):</strong> - A dict to specify keys and fetched Variablesto generate knowledge.</p>
<li><pclass="first"><strong>schema (dict):</strong> - A dict to specify keys and fetched Variables
.. code-block:: guess</p>
<blockquote>
<div><p>to generate knowledge.</p>
</div></blockquote>
</li>
</li>
<li><pclass="first"><strong>program (fluid.Program):</strong> - Inference Program of the teacher model.</p>
<li><pclass="first"><strong>program (fluid.Program):</strong> - Inference Program of the teacher model.</p>
</li>
</li>
<li><pclass="first"><strong>reader_config (dict):</strong> - The config for data reader. Support all the three types of generators used by <aclass="reference external"href="https://www.paddlepaddle.org.cn/documentation/docs/en/api/io/PyReader.html">fluid.io.PyReader</a> and <aclass="reference external"href="https://www.paddlepaddle.org.cn/documentation/docs/en/api/io/DataLoader.html#dataloader">fluid.io.DataLoader</a>, and their configs contain the key-value pair of the generator type and a generator object, plus other necessary argument pairs. See the following:</p>
<li><pclass="first"><strong>reader_config (dict):</strong> - The config for data reader. Support all the three types of generators used by <aclass="reference external"href="https://www.paddlepaddle.org.cn/documentation/docs/en/api/io/PyReader.html">fluid.io.PyReader</a> and <aclass="reference external"href="https://www.paddlepaddle.org.cn/documentation/docs/en/api/io/DataLoader.html#dataloader">fluid.io.DataLoader</a>, and their configs contain the key-value pair of the generator type and a generator object, plus other necessary argument pairs. See the following:</p>
<p>: The class defined for the student model. Receive knowledge data from
<dlclass="docutils">
teacher model and carry out knowledge merging.</p>
<dt>: The class defined for the student model. Receive knowledge data from</dt>
<p><strong>Args:</strong></p>
<dd><blockquoteclass="first">
<div>teacher model and carry out knowledge merging.</div></blockquote>
<pclass="last"><strong>Args:</strong></p>
</dd>
</dl>
<ulclass="simple">
<ulclass="simple">
<li><strong>merge_strategy (dict|None):</strong> - A dict whose keys are the common schemas shared by different teachers, and each corresponding value specifies the merging strategy for different schemas respectively, supporting <strong>sum</strong> and <strong>mean</strong> now.</li>
<li><strong>merge_strategy (dict|None):</strong> - A dict whose keys are the common schemas shared by different teachers, and each corresponding value specifies the merging strategy for different schemas respectively, supporting <strong>sum</strong> and <strong>mean</strong> now.</li>
</ul>
</ul>
...
@@ -289,11 +316,11 @@ teacher model and carry out knowledge merging.</p>
...
@@ -289,11 +316,11 @@ teacher model and carry out knowledge merging.</p>
<p><strong>Args:</strong></p>
<p><strong>Args:</strong></p>
<ulclass="simple">
<ulclass="simple">
<li><strong>in_path (str|None):</strong> The input file path. Default None.</li>
<li><strong>in_path (str|None):</strong> The input file path. Default None.</li>
<li><strong>in_address (str|None):</strong> The input IP address, in the format “<IP_address>:<IP_port>“ (e.g. “127.0.0.1:8080“). Default None.</li>
<li><strong>in_address (str|None):</strong> The input IP address, in the format “&lt;IP_address&gt;:&lt;IP_port&gt;” (e.g. “127.0.0.1:8080”). Default None.</li>
<linkrel="prev"title="Pruning of image classification model - sensitivity"href="../tutorials/sensitivity_tutorial_en.html"/>
<linkrel="prev"title="Pruning of image classification model - sensitivity"href="../tutorials/image_classification_sensitivity_analysis_tutorial_en.html"/>
<ahref="../tutorials/sensitivity_tutorial_en.html"class="btn btn-neutral"title="Pruning of image classification model - sensitivity"accesskey="p"><spanclass="fa fa-arrow-circle-left"></span> Previous</a>
<ahref="../tutorials/image_classification_sensitivity_analysis_tutorial_en.html"class="btn btn-neutral"title="Pruning of image classification model - sensitivity"accesskey="p"><spanclass="fa fa-arrow-circle-left"></span> Previous</a>
<h1>Table about hardware lantency<aclass="headerlink"href="#table-about-hardware-lantency"title="Permalink to this headline">¶</a></h1>
<p>The table about hardware latency is used to evaluate the inference time in special environment and inference engine. The following text used to introduce the format that PaddleSlim support.</p>
<divclass="section"id="introduce">
<h2>Introduce<aclass="headerlink"href="#introduce"title="Permalink to this headline">¶</a></h2>
<p>The table about hardware latency saved all possible operations, one operation in the table including type and parameters, such as: type can be <codeclass="docutils literal"><spanclass="pre">conv2d</span></code>, and corresponding parameters can be the size of feature map, number of kernel, and the size of kernel.
The latency of every operation depends on hardware and inference engine.</p>
</div>
<divclass="section"id="overview-format">
<h2>Overview format<aclass="headerlink"href="#overview-format"title="Permalink to this headline">¶</a></h2>
<p>The table about hardware latency saved in the way of file or multi-line string.
The first line of the table about hardware latency saved the information about version, every line in the following represents a operation and its latency.</p>
</div>
<divclass="section"id="version">
<h2>Version<aclass="headerlink"href="#version"title="Permalink to this headline">¶</a></h2>
<p>The information about version split by comma in the english format, and the detail is hardware, inference engine and timestamp.</p>
<ulclass="simple">
<li>** hardware: ** Used to mark the environment of hardware, including type of architecture, version and so on.</li>
<li>** inference engine: ** Used to mark inference engine, including the name of inference engine, version, optimize options and so on.</li>
<li>** timestamp: ** Used to mark the time of this table created.</li>
</ul>
</div>
<divclass="section"id="operation">
<h2>Operation<aclass="headerlink"href="#operation"title="Permalink to this headline">¶</a></h2>
<p>The information about operation split by comma in the english format, the information about operation and latency split by tabs.</p>
<divclass="section"id="conv2d">
<h3>conv2d<aclass="headerlink"href="#conv2d"title="Permalink to this headline">¶</a></h3>
<li><strong>op_type(str)</strong> - The type of this op.</li>
<li><strong>flag_global_pooling (int)</strong> - Whether is global pooling or not(0: is not global, 1: is global pooling).</li>
<li><strong>n_in (int)</strong> - The batch size of input.</li>
<li><strong>c_in (int)</strong> - The number of channel about input.</li>
<li><strong>h_in (int)</strong> - The height of input feature map.</li>
<li><strong>w_in (int)</strong> - The width of input feature map.</li>
<li><strong>kernel (int)</strong> - The size of kernel.</li>
<li><strong>padding (int)</strong> - The size of padding.</li>
<li><strong>stride (int)</strong> - The size of stride.</li>
<li><strong>ceil_mode (int)</strong> - Whether to compute height and width by using ceil function(0: use floor function, 1: use ceil function).</li>
<li><strong>pool_type (int)</strong> - The type of pooling(1: max pooling 2: average pooling including padding 3: average pooling excluding padding).</li>
<li><strong>latency (float)</strong> - The latency of this op.</li>
</ul>
</div>
<divclass="section"id="softmax">
<h3>softmax<aclass="headerlink"href="#softmax"title="Permalink to this headline">¶</a></h3>
<li><strong>op_type(str)</strong> - The type of this op.</li>
<li><strong>axis (int)</strong> - The index to compute softmax, index in the range of [-1, rank-1], <codeclass="docutils literal"><spanclass="pre">rank</span></code> is the rank of input.</li>
<li><strong>n_in (int)</strong> - The batch size of input.</li>
<li><strong>c_in (int)</strong> - The number of channel about input.</li>
<li><strong>h_in (int)</strong> - The height of input feature map.</li>
<li><strong>w_in (int)</strong> - The width of input feature map.</li>
<li><strong>latency (float)</strong> - The latency of this op.</li>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
<h1>Introduction<aclass="headerlink"href="#introduction"title="Permalink to this headline">¶</a></h1>
<p>PaddleSlim is a toolkit for model compression. It contains a collection of compression strategies, such as pruning, fixed point quantization, knowledge distillation, hyperparameter searching and neural architecture search.</p>
<p>PaddleSlim provides solutions of compression on computer vision models, such as image classification, object detection and semantic segmentation. Meanwhile, PaddleSlim Keeps exploring advanced compression strategies for language model. Furthermore, benckmark of compression strategies on some open tasks is available for your reference.</p>
<p>PaddleSlim also provides auxiliary and primitive API for developer and researcher to survey, implement and apply the method in latest papers. PaddleSlim will support developer in ability of framework and technology consulting.</p>
<divclass="section"id="features">
<h2>Features<aclass="headerlink"href="#features"title="Permalink to this headline">¶</a></h2>
<divclass="section"id="pruning">
<h3>Pruning<aclass="headerlink"href="#pruning"title="Permalink to this headline">¶</a></h3>
<ulclass="simple">
<li>Uniform pruning of convolution</li>
<li>Sensitivity-based prunning</li>
<li>Automated pruning based evolution search strategy</li>
<li>Support pruning of various deep architectures such as VGG, ResNet, and MobileNet.</li>
<li>Support self-defined range of pruning, i.e., layers to be pruned.</li>
</ul>
</div>
<divclass="section"id="fixed-point-quantization">
<h3>Fixed Point Quantization<aclass="headerlink"href="#fixed-point-quantization"title="Permalink to this headline">¶</a></h3>
<ulclass="simple">
<li><strong>Training aware</strong><ul>
<li>Dynamic strategy: During inference, we quantize models with hyperparameters dynamically estimated from small batches of samples.</li>
<li>Static strategy: During inference, we quantize models with the same hyperparameters estimated from training data.</li>
<li>Support layer-wise and channel-wise quantization.</li>
</ul>
</li>
<li><strong>Post training</strong></li>
</ul>
</div>
<divclass="section"id="knowledge-distillation">
<h3>Knowledge Distillation<aclass="headerlink"href="#knowledge-distillation"title="Permalink to this headline">¶</a></h3>
<ulclass="simple">
<li><strong>Naive knowledge distillation:</strong> transfers dark knowledge by merging the teacher and student model into the same Program</li>
<li><strong>Paddle large-scale scalable knowledge distillation framework Pantheon:</strong> a universal solution for knowledge distillation, more flexible than the naive knowledge distillation, and easier to scale to the large-scale applications.<ul>
<li>Decouple the teacher and student models — they run in different processes in the same or different nodes, and transfer knowledge via TCP/IP ports or local files;</li>
<li>Friendly to assemble multiple teacher models and each of them can work in either online or offline mode independently;</li>
<li>Merge knowledge from different teachers and make batch data for the student model automatically;</li>
<li>Support the large-scale knowledge prediction of teacher models on multiple devices.</li>
<h4>Dataset: Pascal VOC; Model: MobileNet-V1-YOLOv3<aclass="headerlink"href="#dataset-pascal-voc-model-mobilenet-v1-yolov3"title="Permalink to this headline">¶</a></h4>
<h4>Dataset: COCO; Model: MobileNet-V1-YOLOv3<aclass="headerlink"href="#dataset-coco-model-mobilenet-v1-yolov3"title="Permalink to this headline">¶</a></h4>
<p>Please ensure you have installed PaddlePaddle1.7+. <aclass="reference external"href="https://www.paddlepaddle.org.cn/install/quick">How to install PaddlePaddle</a>。</p>
<h1>Introduction<aclass="headerlink"href="#introduction"title="Permalink to this headline">¶</a></h1>
<h1>Introduction<aclass="headerlink"href="#introduction"title="Permalink to this headline">¶</a></h1>
<p>As a submodule of PaddlePaddle framework, PaddleSlim is an open-source library for deep model compression and architecture search. PaddleSlim supports current popular deep compression techniques such as pruning, quantization, and knowledge distillation. Further, it also automates the search of hyperparameters and the design of lightweight deep architectures. In the future, we will develop more practically useful compression techniques for industrial-level applications and transfer these techniques to models in NLP.</p>
<p>PaddleSlim is a toolkit for model compression. It contains a collection of compression strategies, such as pruning, fixed point quantization, knowledge distillation, hyperparameter searching and neural architecture search.</p>
<divclass="section"id="methods">
<p>PaddleSlim provides solutions of compression on computer vision models, such as image classification, object detection and semantic segmentation. Meanwhile, PaddleSlim Keeps exploring advanced compression strategies for language model. Furthermore, benckmark of compression strategies on some open tasks is available for your reference.</p>
<h2>Methods<aclass="headerlink"href="#methods"title="Permalink to this headline">¶</a></h2>
<p>PaddleSlim also provides auxiliary and primitive API for developer and researcher to survey, implement and apply the method in latest papers. PaddleSlim will support developer in ability of framework and technology consulting.</p>
<divclass="section"id="features">
<h2>Features<aclass="headerlink"href="#features"title="Permalink to this headline">¶</a></h2>
<divclass="section"id="pruning">
<h3>Pruning<aclass="headerlink"href="#pruning"title="Permalink to this headline">¶</a></h3>
<ulclass="simple">
<ulclass="simple">
<li>Pruning<ul>
<li>Uniform pruning of convolution</li>
<li>Uniform pruning</li>
<li>Sensitivity-based prunning</li>
<li>Sensitivity-based pruning</li>
<li>Automated pruning based evolution search strategy</li>
<li>Automated model pruning</li>
<li>Support pruning of various deep architectures such as VGG, ResNet, and MobileNet.</li>
<li>Support self-defined range of pruning, i.e., layers to be pruned.</li>
</ul>
</ul>
</li>
</div>
<li>Quantization<ul>
<divclass="section"id="fixed-point-quantization">
<li>Training-aware quantization: Quantize models with hyperparameters dynamically estimated from small batches of samples.</li>
<h3>Fixed Point Quantization<aclass="headerlink"href="#fixed-point-quantization"title="Permalink to this headline">¶</a></h3>
<li>Training-aware quantization: Quantize models with the same hyperparameters estimated from training data.</li>
<ulclass="simple">
<li>Support global quantization of weights and Channel-Wise quantization</li>
<li><strong>Training aware</strong><ul>
<li>Dynamic strategy: During inference, we quantize models with hyperparameters dynamically estimated from small batches of samples.</li>
<li>Static strategy: During inference, we quantize models with the same hyperparameters estimated from training data.</li>
<li>Support layer-wise and channel-wise quantization.</li>
<li>PaddleSlim supports FLOPs and latency constrained search.</li>
<li><strong>Naive knowledge distillation:</strong> transfers dark knowledge by merging the teacher and student model into the same Program</li>
<li>PaddleSlim supports the latency estimation on different hardware and platforms.</li>
<li><strong>Paddle large-scale scalable knowledge distillation framework Pantheon:</strong> a universal solution for knowledge distillation, more flexible than the naive knowledge distillation, and easier to scale to the large-scale applications.<ul>
<li>Decouple the teacher and student models — they run in different processes in the same or different nodes, and transfer knowledge via TCP/IP ports or local files;</li>
<li>Friendly to assemble multiple teacher models and each of them can work in either online or offline mode independently;</li>
<li>Merge knowledge from different teachers and make batch data for the student model automatically;</li>
<li>Support the large-scale knowledge prediction of teacher models on multiple devices.</li>
<h4>Dataset: Pascal VOC; Model: MobileNet-V1-YOLOv3<aclass="headerlink"href="#dataset-pascal-voc-model-mobilenet-v1-yolov3"title="Permalink to this headline">¶</a></h4>
<h4>Dataset: COCO; Model: MobileNet-V1-YOLOv3<aclass="headerlink"href="#dataset-coco-model-mobilenet-v1-yolov3"title="Permalink to this headline">¶</a></h4>
</table><p>Note: The <codeclass="docutils literal"><spanclass="pre">_vd</span></code> suffix indicates that the pre-trained model uses Mixup. Please refer to the detailed introduction: <aclass="reference external"href="https://arxiv.org/abs/1710.09412">mixup: Beyond Empirical Risk Minimization</a></p>
</table><p>Note: latency is based on latency_855.txt, the file is test on 855 by PaddleLite。The config of BlazeFace-NASV2 is in <aclass="reference external"href="https://github.com/PaddlePaddle/PaddleDetection/blob/master/configs/face_detection/blazeface_nas_v2.yml">there</a>.</p>
</div>
</div>
<divclass="section"id="id9">
</div>
<h2>3. 图像分割<aclass="headerlink"href="#id9"title="Permalink to this headline">¶</a></h2>
<divclass="section"id="image-segmentation">
<p>数据集：Cityscapes</p>
<h2>3. Image Segmentation<aclass="headerlink"href="#image-segmentation"title="Permalink to this headline">¶</a></h2>
<divclass="section"id="id10">
<p>Dataset：Cityscapes</p>
<h3>3.1 量化<aclass="headerlink"href="#id10"title="Permalink to this headline">¶</a></h3>
<divclass="section"id="id5">
<h3>3.1 Quantization<aclass="headerlink"href="#id5"title="Permalink to this headline">¶</a></h3>
<h1>Knowledge Distillation for Image Classification<aclass="headerlink"href="#knowledge-distillation-for-image-classification"title="Permalink to this headline">¶</a></h1>
<h1>Knowledge Distillation for Image Classification<aclass="headerlink"href="#knowledge-distillation-for-image-classification"title="Permalink to this headline">¶</a></h1>
<p>In this tutorial, you will learn how to use knowledge distillation API of PaddleSlim
<p>In this tutorial, you will learn how to use knowledge distillation API of PaddleSlim
by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflow:</p>
by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflow:</p>
<olclass="simple">
<olclass="arabic simple">
<li>Import dependency</li>
<li>Import dependency</li>
<li>Define student_program and teacher_program</li>
<li>Define student_program and teacher_program</li>
<li>Select feature maps</li>
<li>Select feature maps</li>
...
@@ -144,16 +144,16 @@ by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflo
...
@@ -144,16 +144,16 @@ by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflo
<divclass="section"id="import-dependency">
<divclass="section"id="import-dependency">
<h2>1. Import dependency<aclass="headerlink"href="#import-dependency"title="Permalink to this headline">¶</a></h2>
<h2>1. Import dependency<aclass="headerlink"href="#import-dependency"title="Permalink to this headline">¶</a></h2>
<p>PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below:</p>
<p>PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below:</p>
<h2>2. Define student_program and teacher_program<aclass="headerlink"href="#define-student-program-and-teacher-program"title="Permalink to this headline">¶</a></h2>
<h2>2. Define student_program and teacher_program<aclass="headerlink"href="#define-student-program-and-teacher-program"title="Permalink to this headline">¶</a></h2>
<p>This tutorial trains and verifies distillation model on the MNIST dataset. The input image shape is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code>and the number of output categories is 10.
<p>This tutorial trains and verifies distillation model on the MNIST dataset. The input image shape is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code>and the number of output categories is 10.
Select <codeclass="docutils literal"><spanclass="pre">ResNet50</span></code> as the teacher to perform distillation training on the students of the<codeclass="docutils literal"><spanclass="pre">MobileNet</span></code> architecture.</p>
Select <codeclass="docutils literal"><spanclass="pre">ResNet50</span></code> as the teacher to perform distillation training on the students of the<codeclass="docutils literal"><spanclass="pre">MobileNet</span></code> architecture.</p>
@@ -183,7 +183,7 @@ Select <code class="docutils literal"><span class="pre">ResNet50</span></code> a
...
@@ -183,7 +183,7 @@ Select <code class="docutils literal"><span class="pre">ResNet50</span></code> a
</div>
</div>
<divclass="section"id="select-feature-maps">
<divclass="section"id="select-feature-maps">
<h2>3. Select feature maps<aclass="headerlink"href="#select-feature-maps"title="Permalink to this headline">¶</a></h2>
<h2>3. Select feature maps<aclass="headerlink"href="#select-feature-maps"title="Permalink to this headline">¶</a></h2>
<p>We can use the student_program‘s list_vars method to observe all the Variables, and select one or more variables from it to fit the corresponding variables of the teacher.</p>
<p>We can use the student_program’s list_vars method to observe all the Variables, and select one or more variables from it to fit the corresponding variables of the teacher.</p>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="c1"># get all student variables</span>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="c1"># get all student variables</span>
@@ -201,12 +201,12 @@ Select <code class="docutils literal"><span class="pre">ResNet50</span></code> a
...
@@ -201,12 +201,12 @@ Select <code class="docutils literal"><span class="pre">ResNet50</span></code> a
<spanclass="c1">#print(teacher_vars)</span>
<spanclass="c1">#print(teacher_vars)</span>
</pre></div>
</pre></div>
</div>
</div>
<p>we can see that the shape of ‘bn5c_branch2b.output.1.tmp_3‘ in the teacher_program and the ‘depthwise_conv2d_11.tmp_0‘ of the student are the same and can form the distillation loss function.</p>
<p>we can see that the shape of ‘bn5c_branch2b.output.1.tmp_3’ in the teacher_program and the ‘depthwise_conv2d_11.tmp_0’ of the student are the same and can form the distillation loss function.</p>
<h2>4. Merge program and add distillation loss<aclass="headerlink"href="#merge-program-and-add-distillation-loss"title="Permalink to this headline">¶</a></h2>
<h2>4. Merge program and add distillation loss<aclass="headerlink"href="#merge-program-and-add-distillation-loss"title="Permalink to this headline">¶</a></h2>
<p>The merge operation adds all Variables and Ops in teacher_program to student_Program. At the same time, in order to avoid naming conflicts caused by variables with the same name in two programs, merge will also add a unified naming prefix <strong>name_prefix</strong> for Variables in teacher_program, which The default value is ‘teacher_‘_.</p>
<p>The merge operation adds all Variables and Ops in teacher_program to student_Program. At the same time, in order to avoid naming conflicts caused by variables with the same name in two programs, merge will also add a unified naming prefix <strong>name_prefix</strong> for Variables in teacher<em>program, which The default value is ‘teacher</em>‘_.</p>
<p>In order to ensure that the data of the teacher network and the student network are the same, the merge operation also merges the input data layers of the two programs, so you need to specify a data layer name mapping <em><strong>data_name_map</strong></em>, where key is the input data name of the teacher, and value Is student‘s.</p>
<p>In order to ensure that the data of the teacher network and the student network are the same, the merge operation also merges the input data layers of the two programs, so you need to specify a data layer name mapping <ahref="#id1"><spanclass="problematic"id="id2">**</span></a><em>data_name_map</em><ahref="#id3"><spanclass="problematic"id="id4">**</span></a>, where key is the input data name of the teacher, and value Is student’s.</p>
<h1>Nerual Architecture Search for Image Classification<aclass="headerlink"href="#nerual-architecture-search-for-image-classification"title="Permalink to this headline">¶</a></h1>
<h1>Nerual Architecture Search for Image Classification<aclass="headerlink"href="#nerual-architecture-search-for-image-classification"title="Permalink to this headline">¶</a></h1>
<p>This tutorial shows how to use <aclass="reference external"href="../api/nas_api">API</a> about SANAS in PaddleSlim. We start experiment based on MobileNetV2 as example. The tutorial contains follow section.</p>
<p>This tutorial shows how to use <aclass="reference external"href="../api/nas_api.md">API</a> about SANAS in PaddleSlim. We start experiment based on MobileNetV2 as example. The tutorial contains follow section.</p>
<olclass="simple">
<olclass="arabic simple">
<li>necessary imports</li>
<li>necessary imports</li>
<li>initial SANAS instance</li>
<li>initial SANAS instance</li>
<li>define function about building program</li>
<li>define function about building program</li>
...
@@ -303,7 +303,7 @@
...
@@ -303,7 +303,7 @@
</div>
</div>
<divclass="section"id="reture-score">
<divclass="section"id="reture-score">
<h3>7.6 reture score<aclass="headerlink"href="#reture-score"title="Permalink to this headline">¶</a></h3>
<h3>7.6 reture score<aclass="headerlink"href="#reture-score"title="Permalink to this headline">¶</a></h3>
<h1>Channel Pruning for Image Classification<aclass="headerlink"href="#channel-pruning-for-image-classification"title="Permalink to this headline">¶</a></h1>
<h1>Channel Pruning for Image Classification<aclass="headerlink"href="#channel-pruning-for-image-classification"title="Permalink to this headline">¶</a></h1>
<p>In this tutorial, you will learn how to use channel pruning API of PaddleSlim
<p>In this tutorial, you will learn how to use channel pruning API of PaddleSlim
by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflow:</p>
by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflow:</p>
<olclass="simple">
<olclass="arabic simple">
<li>Import dependency</li>
<li>Import dependency</li>
<li>Build model</li>
<li>Build model</li>
<li>Prune model</li>
<li>Prune model</li>
...
@@ -169,9 +169,9 @@ by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflo
...
@@ -169,9 +169,9 @@ by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflo
<divclass="section"id="import-dependency">
<divclass="section"id="import-dependency">
<h2>1. Import dependency<aclass="headerlink"href="#import-dependency"title="Permalink to this headline">¶</a></h2>
<h2>1. Import dependency<aclass="headerlink"href="#import-dependency"title="Permalink to this headline">¶</a></h2>
<p>PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below:</p>
<p>PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below:</p>
@@ -180,8 +180,8 @@ by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflo
...
@@ -180,8 +180,8 @@ by a demo of MobileNetV1 model on MNIST dataset. This tutorial following workflo
<p>This section will build a classsification model based <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> for MNIST task. The shape of the input is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and the output number is 10.</p>
<p>This section will build a classsification model based <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> for MNIST task. The shape of the input is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and the output number is 10.</p>
<p>To make the code simple, we define a function in package <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> to build classification model.
<p>To make the code simple, we define a function in package <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> to build classification model.
<h3>3.2 Pruning<aclass="headerlink"href="#pruning"title="Permalink to this headline">¶</a></h3>
<h3>3.2 Pruning<aclass="headerlink"href="#pruning"title="Permalink to this headline">¶</a></h3>
<p>The section will prune the parameters named <codeclass="docutils literal"><spanclass="pre">conv2_1_sep_weights</span></code> and <codeclass="docutils literal"><spanclass="pre">conv2_2_sep_weights</span></code> by 20% and 30%.</p>
<p>The section will prune the parameters named <codeclass="docutils literal"><spanclass="pre">conv2_1_sep_weights</span></code> and <codeclass="docutils literal"><spanclass="pre">conv2_2_sep_weights</span></code> by 20% and 30%.</p>
<p>It will change the shapes of parameters defined in <codeclass="docutils literal"><spanclass="pre">train_program</span></code>. And the parameters<codeclass="docutils literal"><spanclass="pre">values</span><spanclass="pre">stored</span><spanclass="pre">in</span></code>fluid.global_scope()` will be pruned.</p>
<p>It will change the shapes of parameters defined in <codeclass="docutils literal"><spanclass="pre">train_program</span></code>. And the parameters<codeclass="docutils literal"><spanclass="pre">values</span><spanclass="pre">stored</span><spanclass="pre">in</span></code>fluid.global_scope()` will be pruned.</p>
@@ -224,9 +224,9 @@ Excute following code to build a model,</p>
...
@@ -224,9 +224,9 @@ Excute following code to build a model,</p>
<h3>4.1 Define dataset<aclass="headerlink"href="#define-dataset"title="Permalink to this headline">¶</a></h3>
<h3>4.1 Define dataset<aclass="headerlink"href="#define-dataset"title="Permalink to this headline">¶</a></h3>
<p>To make you easily run this demo, it will training on MNIST dataset. The package <codeclass="docutils literal"><spanclass="pre">paddle.dataset.mnist</span></code> of Paddle defines the downloading and reading of MNIST dataset.
<p>To make you easily run this demo, it will training on MNIST dataset. The package <codeclass="docutils literal"><spanclass="pre">paddle.dataset.mnist</span></code> of Paddle defines the downloading and reading of MNIST dataset.
Define training data reader and test data reader as below：</p>
Define training data reader and test data reader as below：</p>
<h1>Training-aware Quantization of image classification model - quick start<aclass="headerlink"href="#training-aware-quantization-of-image-classification-model-quick-start"title="Permalink to this headline">¶</a></h1>
<h1>Training-aware Quantization of image classification model - quick start<aclass="headerlink"href="#training-aware-quantization-of-image-classification-model-quick-start"title="Permalink to this headline">¶</a></h1>
<p>This tutorial shows how to do training-aware quantization using <aclass="reference external"href="https://github.com/PaddlePaddle/PaddleSlim/blob/develop/docs/docs/api/quantization_api">API</a> in PaddleSlim. We use MobileNetV1 to train image classification model as example. The tutorial contains follow sections:</p>
<p>This tutorial shows how to do training-aware quantization using <aclass="reference external"href="https://github.com/PaddlePaddle/PaddleSlim/blob/develop/docs/docs/api/quantization_api.md">API</a> in PaddleSlim. We use MobileNetV1 to train image classification model as example. The tutorial contains follow sections:</p>
<olclass="simple">
<olclass="arabic simple">
<li>Necessary imports</li>
<li>Necessary imports</li>
<li>Model architecture</li>
<li>Model architecture</li>
<li>Train normal model</li>
<li>Train normal model</li>
...
@@ -176,7 +176,7 @@
...
@@ -176,7 +176,7 @@
</div>
</div>
<divclass="section"id="model-architecture">
<divclass="section"id="model-architecture">
<h2>2. Model architecture<aclass="headerlink"href="#model-architecture"title="Permalink to this headline">¶</a></h2>
<h2>2. Model architecture<aclass="headerlink"href="#model-architecture"title="Permalink to this headline">¶</a></h2>
<p>The section constructs a classification model, which use <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> and MNIST dataset. The model‘s input size is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and output size is 10. In order to show tutorial conveniently, we pre-defined a method to get image classification model in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code>.</p>
<p>The section constructs a classification model, which use <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> and MNIST dataset. The model’s input size is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and output size is 10. In order to show tutorial conveniently, we pre-defined a method to get image classification model in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code>.</p>
<blockquote>
<blockquote>
<div>note: The APIs in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> are not formal inferface in PaddleSlim. They are defined to simplify the tutorial such as the definition of model structure and the construction of Program.</div></blockquote>
<div>note: The APIs in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> are not formal inferface in PaddleSlim. They are defined to simplify the tutorial such as the definition of model structure and the construction of Program.</div></blockquote>
<h2>3. Train normal model<aclass="headerlink"href="#train-normal-model"title="Permalink to this headline">¶</a></h2>
<h2>3. Train normal model<aclass="headerlink"href="#train-normal-model"title="Permalink to this headline">¶</a></h2>
<p>The section shows how to define model inputs, train and test model. The reason for training the normal image classification model first is that the quantization model‘s training process is performed on the well-trained model. We add quantization and dequantization operators in well-trained model and finetune using smaller learning rate.</p>
<p>The section shows how to define model inputs, train and test model. The reason for training the normal image classification model first is that the quantization model’s training process is performed on the well-trained model. We add quantization and dequantization operators in well-trained model and finetune using smaller learning rate.</p>
<divclass="section"id="input-data-definition">
<divclass="section"id="input-data-definition">
<h3>3.1 input data definition<aclass="headerlink"href="#input-data-definition"title="Permalink to this headline">¶</a></h3>
<h3>3.1 input data definition<aclass="headerlink"href="#input-data-definition"title="Permalink to this headline">¶</a></h3>
<p>To speed up training process, we select MNIST dataset to train image classification model. The API <codeclass="docutils literal"><spanclass="pre">paddle.dataset.mnist</span></code> in Paddle framework contains downloading and reading the images in dataset.</p>
<p>To speed up training process, we select MNIST dataset to train image classification model. The API <codeclass="docutils literal"><spanclass="pre">paddle.dataset.mnist</span></code> in Paddle framework contains downloading and reading the images in dataset.</p>
<h2>6. Save model after quantization<aclass="headerlink"href="#save-model-after-quantization"title="Permalink to this headline">¶</a></h2>
<h2>6. Save model after quantization<aclass="headerlink"href="#save-model-after-quantization"title="Permalink to this headline">¶</a></h2>
<p>The model in <codeclass="docutils literal"><spanclass="pre">4.</span><spanclass="pre">Quantization</span></code> after calling <codeclass="docutils literal"><spanclass="pre">slim.quant.quant_aware</span></code> API is only suitable to train. To get the inference model, we should use <aclass="reference external"href="https://paddlepaddle.github.io/PaddleSlim/api/quantization_api/#convert">slim.quant.convert</a> API to change model architecture and use <aclass="reference external"href="https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api_cn/io_cn/save_inference_model_cn.html#save-inference-model">fluid.io.save_inference_model</a> to save model. <codeclass="docutils literal"><spanclass="pre">float_prog</span></code>‘s parameters are float32 dtype but in int8‘s range which can be used in <codeclass="docutils literal"><spanclass="pre">fluid</span></code> or <codeclass="docutils literal"><spanclass="pre">paddle-lite</span></code>. <codeclass="docutils literal"><spanclass="pre">paddle-lite</span></code> will change the parameters‘ dtype from float32 to int8 first when loading the inference model. <codeclass="docutils literal"><spanclass="pre">int8_prog</span></code>‘s parameters are int8 dtype and we can get model size after quantization by saving it. <codeclass="docutils literal"><spanclass="pre">int8_prog</span></code> cannot be used in <codeclass="docutils literal"><spanclass="pre">fluid</span></code> or <codeclass="docutils literal"><spanclass="pre">paddle-lite</span></code>.</p>
<p>The model in <codeclass="docutils literal"><spanclass="pre">4.</span><spanclass="pre">Quantization</span></code> after calling <codeclass="docutils literal"><spanclass="pre">slim.quant.quant_aware</span></code> API is only suitable to train. To get the inference model, we should use <aclass="reference external"href="https://paddlepaddle.github.io/PaddleSlim/api/quantization_api/#convert">slim.quant.convert</a> API to change model architecture and use <aclass="reference external"href="https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api_cn/io_cn/save_inference_model_cn.html#save-inference-model">fluid.io.save_inference_model</a> to save model. <codeclass="docutils literal"><spanclass="pre">float_prog</span></code>‘s parameters are float32 dtype but in int8’s range which can be used in <codeclass="docutils literal"><spanclass="pre">fluid</span></code> or <codeclass="docutils literal"><spanclass="pre">paddle-lite</span></code>. <codeclass="docutils literal"><spanclass="pre">paddle-lite</span></code> will change the parameters’ dtype from float32 to int8 first when loading the inference model. <codeclass="docutils literal"><spanclass="pre">int8_prog</span></code>‘s parameters are int8 dtype and we can get model size after quantization by saving it. <codeclass="docutils literal"><spanclass="pre">int8_prog</span></code> cannot be used in <codeclass="docutils literal"><spanclass="pre">fluid</span></code> or <codeclass="docutils literal"><spanclass="pre">paddle-lite</span></code>.</p>
<h1>Post-training Quantization of image classification model - quick start<aclass="headerlink"href="#post-training-quantization-of-image-classification-model-quick-start"title="Permalink to this headline">¶</a></h1>
<h1>Post-training Quantization of image classification model - quick start<aclass="headerlink"href="#post-training-quantization-of-image-classification-model-quick-start"title="Permalink to this headline">¶</a></h1>
<p>This tutorial shows how to do post training quantization using <aclass="reference external"href="https://github.com/PaddlePaddle/PaddleSlim/blob/develop/docs/docs/api/quantization_api">API</a> in PaddleSlim. We use MobileNetV1 to train image classification model as example. The tutorial contains follow sections:</p>
<p>This tutorial shows how to do post training quantization using <aclass="reference external"href="https://github.com/PaddlePaddle/PaddleSlim/blob/develop/docs/docs/api/quantization_api.md">API</a> in PaddleSlim. We use MobileNetV1 to train image classification model as example. The tutorial contains follow sections:</p>
<olclass="simple">
<olclass="arabic simple">
<li>Necessary imports</li>
<li>Necessary imports</li>
<li>Model architecture</li>
<li>Model architecture</li>
<li>Train normal model</li>
<li>Train normal model</li>
...
@@ -172,7 +172,7 @@
...
@@ -172,7 +172,7 @@
</div>
</div>
<divclass="section"id="model-architecture">
<divclass="section"id="model-architecture">
<h2>2. Model architecture<aclass="headerlink"href="#model-architecture"title="Permalink to this headline">¶</a></h2>
<h2>2. Model architecture<aclass="headerlink"href="#model-architecture"title="Permalink to this headline">¶</a></h2>
<p>The section constructs a classification model, which use <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> and MNIST dataset. The model‘s input size is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and output size is 10. In order to show tutorial conveniently, we pre-defined a method to get image classification model in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code>.</p>
<p>The section constructs a classification model, which use <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> and MNIST dataset. The model’s input size is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and output size is 10. In order to show tutorial conveniently, we pre-defined a method to get image classification model in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code>.</p>
<blockquote>
<blockquote>
<div>note: The APIs in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> are not formal inferface in PaddleSlim. They are defined to simplify the tutorial such as the definition of model structure and the construction of Program.</div></blockquote>
<div>note: The APIs in <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> are not formal inferface in PaddleSlim. They are defined to simplify the tutorial such as the definition of model structure and the construction of Program.</div></blockquote>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
<h1>Pruning of image classification model - sensitivity<aclass="headerlink"href="#pruning-of-image-classification-model-sensitivity"title="Permalink to this headline">¶</a></h1>
<p>In this tutorial, you will learn how to use <aclass="reference external"href="https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity">sensitivity API of PaddleSlim</a> by a demo of MobileNetV1 model on MNIST dataset。
This tutorial following workflow:</p>
<olclass="arabic simple">
<li>Import dependency</li>
<li>Build model</li>
<li>Define data reader</li>
<li>Define function for test</li>
<li>Training model</li>
<li>Get names of parameter</li>
<li>Compute sensitivities</li>
<li>Pruning model</li>
</ol>
<divclass="section"id="import-dependency">
<h2>1. Import dependency<aclass="headerlink"href="#import-dependency"title="Permalink to this headline">¶</a></h2>
<p>PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below:</p>
<h2>2. Build model<aclass="headerlink"href="#build-model"title="Permalink to this headline">¶</a></h2>
<p>This section will build a classsification model based <codeclass="docutils literal"><spanclass="pre">MobileNetV1</span></code> for MNIST task. The shape of the input is <codeclass="docutils literal"><spanclass="pre">[1,</span><spanclass="pre">28,</span><spanclass="pre">28]</span></code> and the output number is 10.</p>
<p>To make the code simple, we define a function in package <codeclass="docutils literal"><spanclass="pre">paddleslim.models</span></code> to build classification model.
<div>Note：The functions in paddleslim.models is just used in tutorials or demos.</div></blockquote>
</div>
<divclass="section"id="define-data-reader">
<h2>3 Define data reader<aclass="headerlink"href="#define-data-reader"title="Permalink to this headline">¶</a></h2>
<p>MNIST dataset is used for making the demo can be executed quickly. It defines some functions for downloading and reading MNIST dataset in package <codeclass="docutils literal"><spanclass="pre">paddle.dataset.mnist</span></code>.
<h2>5. Training model<aclass="headerlink"href="#training-model"title="Permalink to this headline">¶</a></h2>
<p>Sensitivity analysis is dependent on pretrained model. So we should train the model defined in section 2 for some epochs. One epoch training is enough for this simple demo while more epochs may be necessary for other model. Or you can load pretrained model from filesystem.</p>
<h3>7.1 Compute in single process<aclass="headerlink"href="#compute-in-single-process"title="Permalink to this headline">¶</a></h3>
<p>Apply sensitivity analysis on pretrained model by calling <aclass="reference external"href="https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity">sensitivity API</a>.</p>
<p>The sensitivities will be appended into the file given by option <codeclass="docutils literal"><spanclass="pre">sensitivities_file</span></code> during computing.
The information in this file won`t be computed repeatedly.</p>
<p>Remove the file <codeclass="docutils literal"><spanclass="pre">sensitivities_0.data</span></code> in current directory:</p>
<p>Apart from the parameters to be analyzed, it also support for setting the ratios that each convolutoin will be pruned.</p>
<p>If one model losses 90% accuracy on test dataset when its single convolution layer is pruned by 40%, then we can set <codeclass="docutils literal"><spanclass="pre">pruned_ratios</span></code> to <codeclass="docutils literal"><spanclass="pre">[0.1,</span><spanclass="pre">0.2,</span><spanclass="pre">0.3,</span><spanclass="pre">0.4]</span></code>.</p>
<p>The granularity of <codeclass="docutils literal"><spanclass="pre">pruned_ratios</span></code> should be small to get more reasonable sensitivities. But small granularity of <codeclass="docutils literal"><spanclass="pre">pruned_ratios</span></code> will slow down the computing.</p>
<h3>7.2 Expand sensitivities<aclass="headerlink"href="#expand-sensitivities"title="Permalink to this headline">¶</a></h3>
<p>We can expand <codeclass="docutils literal"><spanclass="pre">pruned_ratios</span></code> to <codeclass="docutils literal"><spanclass="pre">[0.1,</span><spanclass="pre">0.2,</span><spanclass="pre">0.3]</span></code> based the sensitivities generated in section 7.1.</p>
<h3>7.3 Computing sensitivity in multi-process<aclass="headerlink"href="#computing-sensitivity-in-multi-process"title="Permalink to this headline">¶</a></h3>
<p>The time cost of computing sensitivities is dependent on the count of parameters and the speed of model evaluation on test dataset. We can speed up computing by multi-process.</p>
<p>Split <codeclass="docutils literal"><spanclass="pre">pruned_ratios</span></code> into multi-process, and merge the sensitivities from multi-process.</p>
<h4>7.3.1 Computing in each process<aclass="headerlink"href="#computing-in-each-process"title="Permalink to this headline">¶</a></h4>
<p>We have compute the sensitivities when <codeclass="docutils literal"><spanclass="pre">pruned_ratios=[0.1,</span><spanclass="pre">0.2,</span><spanclass="pre">0.3]</span></code> and saved the sensitivities into file named <codeclass="docutils literal"><spanclass="pre">sensitivities_0.data</span></code>.</p>
<p>在另一个进程中，The we start a task by setting <codeclass="docutils literal"><spanclass="pre">pruned_ratios=[0.4]</span></code> in another process and save result into file named <codeclass="docutils literal"><spanclass="pre">sensitivities_1.data</span></code>. Show as below：</p>
<h4>7.3.2 Load sensitivity file generated in multi-process<aclass="headerlink"href="#load-sensitivity-file-generated-in-multi-process"title="Permalink to this headline">¶</a></h4>
<h2>8. Pruning model<aclass="headerlink"href="#pruning-model"title="Permalink to this headline">¶</a></h2>
<p>Pruning model according to the sensitivities generated in section 7.3.3.</p>
<divclass="section"id="get-pruning-ratios">
<h3>8.1 Get pruning ratios<aclass="headerlink"href="#get-pruning-ratios"title="Permalink to this headline">¶</a></h3>
<p>Get a group of ratios by calling <aclass="reference external"href="https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#get_ratios_by_loss">get_ratios_by_loss</a> fuction：</p>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="s2">"FLOPs before pruning: </span><spanclass="si">{}</span><spanclass="s2">"</span><spanclass="o">.</span><spanclass="n">format</span><spanclass="p">(</span><spanclass="n">slim</span><spanclass="o">.</span><spanclass="n">analysis</span><spanclass="o">.</span><spanclass="n">flops</span><spanclass="p">(</span><spanclass="n">train_program</span><spanclass="p">)))</span>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="s2">"FLOPs after pruning: </span><spanclass="si">{}</span><spanclass="s2">"</span><spanclass="o">.</span><spanclass="n">format</span><spanclass="p">(</span><spanclass="n">slim</span><spanclass="o">.</span><spanclass="n">analysis</span><spanclass="o">.</span><spanclass="n">flops</span><spanclass="p">(</span><spanclass="n">pruned_program</span><spanclass="p">)))</span>
</pre></div>
</div>
</div>
<divclass="section"id="pruning-test-network">
<h3>8.3 Pruning test network<aclass="headerlink"href="#pruning-test-network"title="Permalink to this headline">¶</a></h3>
<p>Note：The <codeclass="docutils literal"><spanclass="pre">only_graph</span></code> should be set to True while pruning test network. <aclass="reference external"href="https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#pruner">Pruner API</a></p>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="s2">"FLOPs before pruning: </span><spanclass="si">{}</span><spanclass="s2">"</span><spanclass="o">.</span><spanclass="n">format</span><spanclass="p">(</span><spanclass="n">slim</span><spanclass="o">.</span><spanclass="n">analysis</span><spanclass="o">.</span><spanclass="n">flops</span><spanclass="p">(</span><spanclass="n">val_program</span><spanclass="p">)))</span>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="s2">"FLOPs after pruning: </span><spanclass="si">{}</span><spanclass="s2">"</span><spanclass="o">.</span><spanclass="n">format</span><spanclass="p">(</span><spanclass="n">slim</span><spanclass="o">.</span><spanclass="n">analysis</span><spanclass="o">.</span><spanclass="n">flops</span><spanclass="p">(</span><spanclass="n">pruned_val_program</span><spanclass="p">)))</span>
</pre></div>
</div>
<p>Get accuracy of pruned model on test dataset:</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
<liclass="toctree-l2"><aclass="reference internal"href="sensitivity_tutorial_en.html">Pruning of image classification model - sensitivity</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="image_classification_sensitivity_analysis_tutorial_en.html">Pruning of image classification model - sensitivity</a></li>
<h1>Aadvanced Tutorials<aclass="headerlink"href="#aadvanced-tutorials"title="Permalink to this headline">¶</a></h1>
<h1>Aadvanced Tutorials<aclass="headerlink"href="#aadvanced-tutorials"title="Permalink to this headline">¶</a></h1>
<divclass="toctree-wrapper compound">
<divclass="toctree-wrapper compound">
<ul>
<ul>
<liclass="toctree-l1"><aclass="reference internal"href="sensitivity_tutorial_en.html">Pruning of image classification model - sensitivity</a></li>
<liclass="toctree-l1"><aclass="reference internal"href="image_classification_sensitivity_analysis_tutorial_en.html">Pruning of image classification model - sensitivity</a></li>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
