From 98fe49ae9917014685774b36573d4a32af11bb6e Mon Sep 17 00:00:00 2001 From: whs Date: Mon, 24 Feb 2020 14:51:28 +0800 Subject: [PATCH] [cherry-pick/release1.0.0]Translate tutorials of sensitivity. (#141) --- ...cation_sensitivity_analysis_tutorial_en.md | 263 ++++++++++++++++++ docs/en/tutorials/index_en.rst | 2 +- docs/en/tutorials/sensitivity_tutorial_en.md | 25 -- 3 files changed, 264 insertions(+), 26 deletions(-) create mode 100644 docs/en/tutorials/image_classification_sensitivity_analysis_tutorial_en.md delete mode 100644 docs/en/tutorials/sensitivity_tutorial_en.md diff --git a/docs/en/tutorials/image_classification_sensitivity_analysis_tutorial_en.md b/docs/en/tutorials/image_classification_sensitivity_analysis_tutorial_en.md new file mode 100644 index 00000000..2b084074 --- /dev/null +++ b/docs/en/tutorials/image_classification_sensitivity_analysis_tutorial_en.md @@ -0,0 +1,263 @@ +# Pruning of image classification model - sensitivity + +In this tutorial, you will learn how to use [sensitivity API of PaddleSlim](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity) by a demo of MobileNetV1 model on MNIST dataset。 +This tutorial following workflow: + +1. Import dependency +2. Build model +3. Define data reader +4. Define function for test +5. Training model +6. Get names of parameter +7. Compute sensitivities +8. Pruning model + + +## 1. Import dependency + +PaddleSlim dependents on Paddle1.7. Please ensure that you have installed paddle correctly. Import Paddle and PaddleSlim as below: + +```python +import paddle +import paddle.fluid as fluid +import paddleslim as slim +``` + +## 2. Build model + +This section will build a classsification model based `MobileNetV1` for MNIST task. The shape of the input is `[1, 28, 28]` and the output number is 10. + +To make the code simple, we define a function in package `paddleslim.models` to build classification model. +Excute following code to build a model, + + +```python +exe, train_program, val_program, inputs, outputs = slim.models.image_classification("MobileNet", [1, 28, 28], 10, use_gpu=True) +place = fluid.CUDAPlace(0) +``` + +>Note：The functions in paddleslim.models is just used in tutorials or demos. + +## 3 Define data reader + +MNIST dataset is used for making the demo can be executed quickly. It defines some functions for downloading and reading MNIST dataset in package `paddle.dataset.mnist`. +Show as below： + +```python +import paddle.dataset.mnist as reader +train_reader = paddle.batch( + reader.train(), batch_size=128, drop_last=True) +test_reader = paddle.batch( + reader.test(), batch_size=128, drop_last=True) +data_feeder = fluid.DataFeeder(inputs, place) +``` + +## 4. Define test function + +To get the performance of model on test dataset after pruning a convolution layer, we define a test function as below: + +```python +import numpy as np +def test(program): + acc_top1_ns = [] + acc_top5_ns = [] + for data in test_reader(): + acc_top1_n, acc_top5_n, _ = exe.run( + program, + feed=data_feeder.feed(data), + fetch_list=outputs) + acc_top1_ns.append(np.mean(acc_top1_n)) + acc_top5_ns.append(np.mean(acc_top5_n)) + print("Final eva - acc_top1: {}; acc_top5: {}".format( + np.mean(np.array(acc_top1_ns)), np.mean(np.array(acc_top5_ns)))) + return np.mean(np.array(acc_top1_ns)) +``` + +## 5. Training model + +Sensitivity analysis is dependent on pretrained model. So we should train the model defined in section 2 for some epochs. One epoch training is enough for this simple demo while more epochs may be necessary for other model. Or you can load pretrained model from filesystem. + +Training model as below: + + +```python +for data in train_reader(): + acc1, acc5, loss = exe.run(train_program, feed=data_feeder.feed(data), fetch_list=outputs) +print(np.mean(acc1), np.mean(acc5), np.mean(loss)) +``` + +Get the performance using the test function defined in section 4: + +```python +test(val_program) +``` + +## 6. Get names of parameters + +```python +params = [] +for param in train_program.global_block().all_parameters(): + if "_sep_weights" in param.name: + params.append(param.name) +print(params) +params = params[:5] +``` + +## 7. Compute sensitivities + +### 7.1 Compute in single process + +Apply sensitivity analysis on pretrained model by calling [sensitivity API](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity). + +The sensitivities will be appended into the file given by option `sensitivities_file` during computing. +The information in this file won`t be computed repeatedly. + +Remove the file `sensitivities_0.data` in current directory: + +```python +!rm -rf sensitivities_0.data +``` + +Apart from the parameters to be analyzed, it also support for setting the ratios that each convolutoin will be pruned. + +If one model losses 90% accuracy on test dataset when its single convolution layer is pruned by 40%, then we can set `pruned_ratios` to `[0.1, 0.2, 0.3, 0.4]`. + +The granularity of `pruned_ratios` should be small to get more reasonable sensitivities. But small granularity of `pruned_ratios` will slow down the computing. + +```python +sens_0 = slim.prune.sensitivity( + val_program, + place, + params, + test, + sensitivities_file="sensitivities_0.data", + pruned_ratios=[0.1, 0.2]) +print(sens_0) +``` + +### 7.2 Expand sensitivities + +We can expand `pruned_ratios` to `[0.1, 0.2, 0.3]` based the sensitivities generated in section 7.1. + +```python +sens_0 = slim.prune.sensitivity( + val_program, + place, + params, + test, + sensitivities_file="sensitivities_0.data", + pruned_ratios=[0.3]) +print(sens_0) +``` + +### 7.3 Computing sensitivity in multi-process + +The time cost of computing sensitivities is dependent on the count of parameters and the speed of model evaluation on test dataset. We can speed up computing by multi-process. + +Split `pruned_ratios` into multi-process, and merge the sensitivities from multi-process. + +#### 7.3.1 Computing in each process + +We have compute the sensitivities when `pruned_ratios=[0.1, 0.2, 0.3]` and saved the sensitivities into file named `sensitivities_0.data`. + +在另一个进程中，The we start a task by setting `pruned_ratios=[0.4]` in another process and save result into file named `sensitivities_1.data`. Show as below： + + +```python +sens_1 = slim.prune.sensitivity( + val_program, + place, + params, + test, + sensitivities_file="sensitivities_1.data", + pruned_ratios=[0.4]) +print(sens_1) +``` + +#### 7.3.2 Load sensitivity file generated in multi-process + +```python +s_0 = slim.prune.load_sensitivities("sensitivities_0.data") +s_1 = slim.prune.load_sensitivities("sensitivities_1.data") +print(s_0) +print(s_1) +``` + +#### 7.3.3 Merge sensitivies + + +```python +s = slim.prune.merge_sensitive([s_0, s_1]) +print(s) +``` + +## 8. Pruning model + +Pruning model according to the sensitivities generated in section 7.3.3. + +### 8.1 Get pruning ratios + +Get a group of ratios by calling [get_ratios_by_loss](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#get_ratios_by_loss) fuction： + + +```python +loss = 0.01 +ratios = slim.prune.get_ratios_by_loss(s_0, loss) +print(ratios) +``` + +### 8.2 Pruning training network + + +```python +pruner = slim.prune.Pruner() +print("FLOPs before pruning: {}".format(slim.analysis.flops(train_program))) +pruned_program, _, _ = pruner.prune( + train_program, + fluid.global_scope(), + params=ratios.keys(), + ratios=ratios.values(), + place=place) +print("FLOPs after pruning: {}".format(slim.analysis.flops(pruned_program))) +``` + +### 8.3 Pruning test network + +Note：The `only_graph` should be set to True while pruning test network. [Pruner API](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#pruner) + + +```python +pruner = slim.prune.Pruner() +print("FLOPs before pruning: {}".format(slim.analysis.flops(val_program))) +pruned_val_program, _, _ = pruner.prune( + val_program, + fluid.global_scope(), + params=ratios.keys(), + ratios=ratios.values(), + place=place, + only_graph=True) +print("FLOPs after pruning: {}".format(slim.analysis.flops(pruned_val_program))) +``` + +Get accuracy of pruned model on test dataset: + +```python +test(pruned_val_program) +``` + +### 8.4 Training pruned model + +Training pruned model: + + +```python +for data in train_reader(): + acc1, acc5, loss = exe.run(pruned_program, feed=data_feeder.feed(data), fetch_list=outputs) +print(np.mean(acc1), np.mean(acc5), np.mean(loss)) +``` + +Get accuracy of model after training: + +```python +test(pruned_val_program) +``` diff --git a/docs/en/tutorials/index_en.rst b/docs/en/tutorials/index_en.rst index 44241358..10bdbd38 100644 --- a/docs/en/tutorials/index_en.rst +++ b/docs/en/tutorials/index_en.rst @@ -4,5 +4,5 @@ Aadvanced Tutorials .. toctree:: :maxdepth: 1 - sensitivity_tutorial_en.md + image_classification_sensitivity_analysis_tutorial_en.md diff --git a/docs/en/tutorials/sensitivity_tutorial_en.md b/docs/en/tutorials/sensitivity_tutorial_en.md deleted file mode 100644 index c51ac202..00000000 --- a/docs/en/tutorials/sensitivity_tutorial_en.md +++ /dev/null @@ -1,25 +0,0 @@ -# Pruning of image classification model - sensitivity - -该教程以图像分类模型MobileNetV1为例，说明如何快速使用[PaddleSlim的敏感度分析接口](https://paddlepaddle.github.io/PaddleSlim/api/prune_api/#sensitivity)。 -该示例包含以下步骤： - -1. 导入依赖 -2. 构建模型 -3. 定义输入数据 -4. 定义模型评估方法 -5. 训练模型 -6. 获取待分析卷积参数名称 -7. 分析敏感度 -8. 剪裁模型 - -以下章节依次次介绍每个步骤的内容。 - -## 1. 导入依赖 - -PaddleSlim依赖Paddle1.7版本，请确认已正确安装Paddle，然后按以下方式导入Paddle和PaddleSlim: - -``` -import paddle -import paddle.fluid as fluid -import paddleslim as slim -``` -- GitLab