Support pruning dygraph (#506)

.gitmodules

-[submodule "demo/ocr/PaddleOCR"]
-	path = demo/ocr/PaddleOCR
-	url = https://github.com/PaddlePaddle/PaddleOCR

demo/dygraph/pruning/filter_pruning.md

+# Paddle动态图卷积模型Filter剪裁教程
+
+该教程以MobileNetV1模型和Cifar10分类任务为例，介绍如何使用PaddleSlim对动态图卷积模型进行filter剪裁。
+
+## 1. 模型定义
+
+PaddlePaddle提供的`vision`模块提供了一些构建好的分类模型结构，并提供在`ImageNet`数据集上的预训练模型。为了简化教程，我们不再重新定义网络结构，而是直接从`vision`模块导入模型结构。代码如下所示，我们导入`MobileNetV1`模型，并查看模型的结构信息。
+
+```python
+import paddle
+from paddle.vision.models import mobilenet_v1
+net = mobilenet_v1(pretrained=False)
+paddle.summary(net, (1, 3, 32, 32))
+```
+
+## 2. 准备数据
+
+我们直接使用`vision`模块提供的`Cifar10`数据集，并通过飞桨高层API `paddle.vision.transforms`对数据进行预处理。在声明`paddle.vision.datasets.Cifar10`对象时，会自动下载数据并缓存到本地文件系统。代码如下所示：
+
+```python
+import paddle.vision.transforms as T
+transform = T.Compose([
+                    T.Transpose(),
+                    T.Normalize([127.5], [127.5])
+                ])
+train_dataset = paddle.vision.datasets.Cifar10(mode="train", backend="cv2",transform=transform)
+val_dataset = paddle.vision.datasets.Cifar10(mode="test", backend="cv2",transform=transform)
+```
+
+我们可以通过以下代码查看训练集和测试集的样本数量，并尝试取出训练集中的第一个样本，观察其图片的`shape`和对应的`label`。
+
+```python
+from __future__ import print_function
+print(f'train samples count: {len(train_dataset)}')
+print(f'val samples count: {len(val_dataset)}')
+for data in train_dataset:
+    print(f'image shape: {data[0].shape}; label: {data[1]}')
+    break
+```
+
+## 3. 模型训练准备工作
+
+在对卷积网络进行剪裁之前，我们需要在测试集上评估网络中各层的重要性。在剪裁之后，我们需要对得到的小模型进行重训练。在本示例中，我们将会使用Paddle高层API `paddle.Model`进行训练和评估工作。以下代码声明了`paddle.Model`实例，并指定了训练相关的一些设置，包括：
+- 输入的shape
+- 优化器
+- 损失函数
+- 模型评估指标
+
+
+```python
+from paddle.static import InputSpec as Input
+optimizer = paddle.optimizer.Momentum(
+        learning_rate=0.1,
+        parameters=net.parameters())
+
+inputs = [Input([None, 3, 224, 224], 'float32', name='image')]
+labels = [Input([None, 1], 'int64', name='label')]
+
+model = paddle.Model(net, inputs, labels)
+
+model.prepare(
+        optimizer,
+        paddle.nn.CrossEntropyLoss(),
+        paddle.metric.Accuracy(topk=(1, 5)))
+```
+
+以上代码声明了用于训练的`model`对象，接下来可以调用`model`的`fit`接口和`evaluate`接口分别进行训练和评估：
+
+```python
+model.fit(train_dataset, epochs=2, batch_size=128, verbose=1)
+result = model.evaluate(val_dataset,batch_size=128, log_freq=10)
+print(result)
+```
+
+## 4. 剪裁
+
+本节内容分为两部分：卷积层重要性分析和`Filters`剪裁，其中『卷积层重要性分析』也可以被称作『卷积层敏感度分析』，我们定义越重要的卷积层越敏感。
+PaddleSlim提供了工具类`Pruner`来进行重要性分析和剪裁操作，不同的`Pruner`的子类对应不同的分析和剪裁策略，本示例以`L1NormFilterPruner`为例说明。首先我们声明一个`L1NormFilterPruner`对象，如下所示：
+
+```python
+from paddleslim.dygraph import L1NormFilterPruner
+pruner = L1NormFilterPruner(net, [1, 3, 224, 224])
+```
+
+如果本地文件系统已有一个存储敏感度信息（见4.1节）的文件，声明`L1NormFilterPruner`对象时，可以通过指定`sen_file`选项加载计算好的敏感度信息，如下：
+
+```python
+#pruner = L1NormFilterPruner(net, [1, 3, 224, 224]), sen_file="./sen.pickle")
+```
+
+### 4.1 卷积重要性分析
+
+在对卷积网络中的filters进行剪裁时，我们需要判断哪些`filters`不重要，然后优先剪掉不重要的`filters`。
+在一个卷积内，我们使用`filter`的`L1 Norm`来代表重要性，`L1 Norm`越大的`filters`越重要。在多个卷积间，我们通过敏感度代表卷积的重要性，越敏感的卷积越重要，重要的卷积会被剪掉相对较少的`filters`。
+单个卷积内的filters重要性计算会在剪裁时进行，无需用户关注。本小节，我们只介绍多个卷积间如何分析重要性，即『敏感度分析』。
+
+#### 敏感度定义
+
+如图4-1所示，某个卷积网络包含K个卷积层，每个卷积层有4个`filters`，原始网络精度为90。
+
+第一步：从『卷积1』中剪裁掉25%的filters，也就是『卷积1』中第2个Filters，然后直接在测试集上评估精度结果为85，得到左边坐标图的第二个红点。恢复模型到初始状态。
+第二步：从『卷积1』中裁掉2个卷积，然后在测试集上评估精度为70，得到坐标图的第3个红点。恢复模型到初始状态。
+第三步：同理得到第4个红点。把『卷积1』对应的4个红点链接成曲线，即为『卷积1』的敏感度曲线。
+第四步：同理得到『卷积K』的敏感度曲线。
+
+<div align="center">
+    <img src="filter_pruning/4-1.png" width="600" height="300">
+</div>
+<div align="center">
+图4-1 敏感度计算过程示意图
+</div>
+
+如图4-2所示，为VGG-16在CIFAR10分类任务上的敏感度曲线示意图：
+
+<div align="center">
+    <img src="filter_pruning/4-2.png" width="600" height="500">
+</div>
+<div align="center">
+图4-2 VGG-16敏感度示例
+</div>
+
+
+考虑到不同的模型在不同的任务上的精度数值差别较大，甚至不在同一个量级，所以，PaddleSlim在计算和存储卷积层敏感度时，使用的是精度的损失比例。如图4-3所示，为PaddleSlim计算出的MobileNetV1-YOLOv3在VOC检测任务上的敏感度示意图，其中，纵轴为精度损失：
+
+<div align="center">
+    <img src="filter_pruning/4-3.png" width="600" height="600">
+</div>
+<div align="center">
+图4-3 用精度损失表示的敏感度
+</div>
+
+
+#### 敏感度计算
+
+调用`pruner`对象的`sensitive`方法进行敏感度分析，在调用`sensitive`之前，我们简单对`model.evaluate`进行包装，使其符合`sensitive`接口的规范。执行如下代码，会进行敏感度计算，并将计算结果存入本地文件系统：
+
+```python
+def eval_fn():
+        result = model.evaluate(
+            val_dataset,
+            batch_size=128)
+        return result['acc_top1']
+pruner.sensitive(eval_func=eval_fn, sen_file="./sen.pickle")
+```
+
+上述代码执行完毕后，敏感度信息会存放在pruner对象中，可以通过以下方式查看敏感度信息内容：
+
+```python
+print(pruner.sensitive())
+```
+
+
+### 4.2 剪裁
+
+
+`pruner`对象提供了`sensitive_prune`方法根据敏感度信息对模型进行剪裁，用户只需要传入期望的FLOPs减少比例。首先，我们记录下剪裁之前的模型的FLOPs数值，如下：
+
+```python
+from paddleslim.analysis import dygraph_flops
+flops = dygraph_flops(net, [1, 3, 32, 32])
+print(f"FLOPs before pruning: {flops}")
+```
+
+执行剪裁操作，期望跳过最后一层卷积层并剪掉40%的FLOPs：
+
+```python
+plan = pruner.sensitive_prune(0.4, skip_vars=["conv2d_26.w_0"])
+flops = dygraph_flops(net, [1, 3, 32, 32])
+print(f"FLOPs after pruning: {flops}")
+print(f"Pruned FLOPs: {round(plan.pruned_flops*100, 2)}%")
+```
+
+剪裁之后，在测试集上重新评估精度，会发现精度大幅下降，如下所示：
+
+```python
+result = model.evaluate(val_dataset,batch_size=128, log_freq=10)
+print(f"before fine-tuning: {result}")
+```
+
+对剪裁后的模型重新训练, 并再测试集上测试精度，如下：
+
+```python
+optimizer = paddle.optimizer.Momentum(
+        learning_rate=0.1,
+        parameters=net.parameters())
+model.prepare(
+        optimizer,
+        paddle.nn.CrossEntropyLoss(),
+        paddle.metric.Accuracy(topk=(1, 5)))
+model.fit(train_dataset, epochs=2, batch_size=128, verbose=1)
+result = model.evaluate(val_dataset,batch_size=128, log_freq=10)
+print(f"after fine-tuning: {result}")
+```
+
+经过重新训练，精度有所提升，最后看下剪裁后模型的结构信息，如下：
+
+```python
+paddle.summary(net, (1, 3, 32, 32))
+```

demo/dygraph/pruning/self_defined_filter_pruning.md

+# Paddle动态图自定义剪裁策略教程
+
+## 1. 概述
+
+该教程介绍如果在PaddleSlim提供的接口基础上快速自定义`Filters`剪裁策略。
+在PaddleSlim中，所有剪裁`Filters`的`Pruner`继承自基类`FilterPruner`。`FilterPruner`中自定义了一系列通用方法，用户只需要重载实现`FilterPruner`的`cal_mask`接口，`cal_mask`接口定义如下：
+
+```python
+def cal_mask(self, var_name, pruned_ratio, group):
+    raise NotImplemented()
+```
+
+`cal_mask`接口接受的参数说明如下：
+- **var_name:** 要剪裁的目标变量，一般为卷积层的权重参数的名称。在Paddle中，卷积层的权重参数格式为`[output_channel, input_channel, kernel_size, kernel_size]`，其中，`output_channel`为当前卷积层的输出通道数，`input_channel`为当前卷积层的输入通道数，`kernel_size`为卷积核大小。
+- **pruned_ratio:** 对名称为`var_name`的变量的剪裁率。
+- **group:** 与待裁目标变量相关的所有变量的信息。
+
+### 1.1 Group概念介绍
+
+<div align="center">
+    <img src="self_define_filter_pruning/1-1.png" width="600" height="230">
+</div>
+<div align="center">
+<strong>图1-1 卷积层关联关系示意图</strong>
+</div>
+
+如图1-1所示，在给定模型中有两个卷积层，第一个卷积层有3个`filters`，第二个卷积层有2个`filters`。如果删除第一个卷积绿色的`filter`，第一个卷积的输出特征图的通道数也会减1，同时需要删掉第二个卷积层绿色的`kernels`。如上所述的两个卷积共同组成一个group，表示如下：
+
+```python
+group = {
+            "conv_1.weight":{
+                "pruned_dims": [0],
+                "layer": conv_layer_1,
+                "var": var_instance_1,
+                "value": var_value_1,
+            },
+            "conv_2.weight":{
+                "pruned_dims": [1],
+                "layer": conv_layer_2,
+                "var": var_instance_2,
+                "value": var_value_2,
+            }
+        }
+```
+
+在上述表示`group`的数据结构示例中，`conv_1.weight`为第一个卷积权重参数的名称，其对应的value也是一个dict实例，存放了当前参数的一些信息，包括：
+- **pruned_dims:** 类型为`list<int>`，表示当前参数在哪些维度上被裁。
+- **layer:** 类型为[paddle.nn.Layer](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/dygraph_cn/Layer_cn.html#layer), 表示当前参数所在`Layer`。
+- **var:** 类型为[paddle.Tensor](https://www.paddlepaddle.org.cn/documentation/docs/zh/api_cn/fluid_cn/Variable_cn.html#variable), 表示当前参数对应的实例。
+- **value:** 类型为numpy.array类型，待裁参数所存的具体数值，方便开发者使用。
+
+图1-2为更复杂的情况，其中，`Add`操作的所有输入的通道数需要保持一致，`Concat`操作的输出通道数的调整可能会影响到所有输入的通道数，因此`group`中可能包含多个卷积的参数或变量，可以是：卷积权重、卷积bias、`batch norm`相关参数等。
+
+
+<div align="center">
+    <img src="self_define_filter_pruning/1-2.png" width="388" height="350">
+</div>
+<div align="center">
+<strong>图1-2 复杂网络示例</strong>
+</div>
+
+
+## 2. 定义模型
+
+```python
+import paddle
+from paddle.vision.models import mobilenet_v1
+net = mobilenet_v1(pretrained=False)
+paddle.summary(net, (1, 3, 32, 32))
+```
+
+## 3. L2NormFilterPruner
+
+该小节参考`L1NormFilterPruner`实现`L2NormFilterPruner`，方式为集成`FIlterPruner`并重载`cal_mask`接口。代码如下所示：
+
+```python
+import numpy as np
+from paddleslim.dygraph import FilterPruner
+
+class L2NormFilterPruner(FilterPruner):
+
+    def __init__(self, model, input_shape, sen_file=None):
+        super(L2NormFilterPruner, self).__init__(
+            model, input_shape, sen_file=sen_file)
+
+    def cal_mask(self, var_name, pruned_ratio, group):
+        value = group[var_name]['value']
+        pruned_dims = group[var_name]['pruned_dims']
+        reduce_dims = [
+                    i for i in range(len(value.shape)) if i not in pruned_dims
+                ]
+
+        # scores = np.mean(np.abs(value), axis=tuple(reduce_dims))
+        scores = np.sqrt(np.sum(np.square(value), axis=tuple(reduce_dims)))
+        sorted_idx = scores.argsort()
+        pruned_num = int(round(len(sorted_idx) * pruned_ratio))
+        pruned_idx = sorted_idx[:pruned_num]
+        mask_shape = [value.shape[i] for i in pruned_dims]
+        mask = np.ones(mask_shape, dtype="int32")
+        mask[pruned_idx] = 0
+        return mask
+```
+
+如上述代码所示，我们重载了`FilterPruner`基类的`cal_mask`方法，并在`L1NormFilterPruner`代码基础上，修改了计算通道重要性的语句，将其修改为了计算L2Norm的逻辑：
+
+```python
+scores = np.sqrt(np.sum(np.square(value), axis=tuple(reduce_dims)))
+```
+
+接下来定义一个`L2NormFilterPruner`对象，并调用`prune_var`方法对单个卷积层进行剪裁，`prune_var`方法继承自`FilterPruner`，开发者不用再重载实现。
+按以下代码调用`prune_var`方法后，参数名称为`conv2d_0.w_0`的卷积层会被裁掉50%的`filters`，与之相关关联的后续卷积和`BatchNorm`相关的参数也会被剪裁。`prune_var`不仅会对待裁模型进行`inplace`的裁剪，还会返回保存裁剪详细信息的`PruningPlan`对象，用户可以直接打印`PruningPlan`对象内容。
+最后，可以通过调用`Pruner`的`restore`方法，将已被裁剪的模型恢复到初始状态。
+
+```python
+pruner = L2NormFilterPruner(net, [1, 3, 32, 32])
+plan = pruner.prune_var("conv2d_0.w_0", 0, 0.5)
+print(plan)
+pruner.restore()
+```
+
+## 4. FPGMFilterPruner
+
+参考：[Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://arxiv.org/abs/1811.00250)
+
+### 4.1 原理介绍
+
+如图4-1所示，传统基于Norm统计方法的filter重要性评估方式的有效性取决于卷积层权重数值的分布，比较理想的分布式要满足两个条件：
+
+- 偏差(deviation)要大
+- 最小值要小(图4-1中v1)
+
+满足上述条件后，我们才能裁掉更多Norm统计值较小的参数，如图4-1中红色部分所示。
+
+
+<div align="center">
+    <img src="self_define_filter_pruning/4-1.png" width="600" height="170">
+</div>
+<div align="center">
+<strong>图4-1</strong>
+</div>
+
+而现实中的模型的权重分布如图4-2中绿色分布所示，总是有较小的偏差或较大的最小值。
+
+<div align="center">
+    <img src="self_define_filter_pruning/4-2.png" width="600" height="224">
+</div>
+<div align="center">
+<strong>图4-2</strong>
+</div>
+
+考虑到上述传统方法的缺点，FPGM则用filter之间的几何距离来表示重要性，其遵循的原则就是：几何距离比较近的filters，作用也相近。
+如图4-3所示，有3个filters，将各个filter展开为向量，并两两计算几何距离。其中，绿色filter的重要性得分就是它到其它两个filter的距离和，即0.7071+0.5831=1.2902。同理算出另外两个filters的得分，绿色filter得分最高，其重要性最高。
+
+<div align="center">
+    <img src="self_define_filter_pruning/4-3.png" width="400" height="560">
+</div>
+<div align="center">
+<strong>图4-3</strong>
+</div>
+
+### 4.2 实现
+
+以下代码通过继承`FilterPruner`并重载`cal_mask`实现了`FPGMFilterPruner`，其中，`get_distance_sum`用于计算第`out_idx`个filter的重要性。
+
+```python
+import numpy as np
+from paddleslim.dygraph import FilterPruner
+
+class FPGMFilterPruner(FilterPruner):
+
+    def __init__(self, model, input_shape, sen_file=None):
+        super(FPGMFilterPruner, self).__init__(
+            model, input_shape, sen_file=sen_file)
+
+    def cal_mask(self, var_name, pruned_ratio, group):
+        value = group[var_name]['value']
+        pruned_dims = group[var_name]['pruned_dims']
+        assert(pruned_dims == [0])
+
+        dist_sum_list = []
+        for out_i in range(value.shape[0]):
+            dist_sum = self.get_distance_sum(value, out_i)
+            dist_sum_list.append(dist_sum)
+        scores = np.array(dist_sum_list)
+
+        sorted_idx = scores.argsort()
+        pruned_num = int(round(len(sorted_idx) * pruned_ratio))
+        pruned_idx = sorted_idx[:pruned_num]
+        mask_shape = [value.shape[i] for i in pruned_dims]
+        mask = np.ones(mask_shape, dtype="int32")
+        mask[pruned_idx] = 0
+        return mask
+
+    def get_distance_sum(self, value, out_idx):
+        w = value.view()
+        w.shape = value.shape[0], np.product(value.shape[1:])
+        selected_filter = np.tile(w[out_idx], (w.shape[0], 1))
+        x = w - selected_filter
+        x = np.sqrt(np.sum(x * x, -1))
+        return x.sum()
+```
+
+接下来声明一个FPGMFilterPruner对象进行验证:
+
+```python
+pruner = FPGMFilterPruner(net, [1, 3, 32, 32])
+plan = pruner.prune_var("conv2d_0.w_0", 0, 0.5)
+print(plan)
+pruner.restore()
+```
+
+## 5. 敏感度剪裁
+
+在第3节和第4节，开发者自定义实现的`L2NormFilterPruner`和`FPGMFilterPruner`也继承了`FilterPruner`的敏感度计算方法`sensitive`和剪裁方法`sensitive_prune`。
+
+### 5.1 预训练
+
+```python
+import paddle.vision.transforms as T
+transform = T.Compose([
+                    T.Transpose(),
+                    T.Normalize([127.5], [127.5])
+                ])
+train_dataset = paddle.vision.datasets.Cifar10(mode="train", backend="cv2",transform=transform)
+val_dataset = paddle.vision.datasets.Cifar10(mode="test", backend="cv2",transform=transform)
+from paddle.static import InputSpec as Input
+optimizer = paddle.optimizer.Momentum(
+        learning_rate=0.1,
+        parameters=net.parameters())
+
+inputs = [Input([None, 3, 32, 32], 'float32', name='image')]
+labels = [Input([None, 1], 'int64', name='label')]
+model = paddle.Model(net, inputs, labels)
+model.prepare(
+        optimizer,
+        paddle.nn.CrossEntropyLoss(),
+        paddle.metric.Accuracy(topk=(1, 5)))
+model.fit(train_dataset, epochs=2, batch_size=128, verbose=1)
+result = model.evaluate(val_dataset,batch_size=128, log_freq=10)
+print(result)
+```
+
+### 5.2 计算敏感度
+
+```python
+pruner = FPGMFilterPruner(net, [1, 3, 32, 32])
+def eval_fn():
+        result = model.evaluate(
+            val_dataset,
+            batch_size=128)
+        return result['acc_top1']
+sen = pruner.sensitive(eval_func=eval_fn, sen_file="./fpgm_sen.pickle")
+print(sen)
+```
+
+### 5.3 剪裁
+
+```python
+from paddleslim.analysis import dygraph_flops
+flops = dygraph_flops(net, [1, 3, 32, 32])
+print(f"FLOPs before pruning: {flops}")
+plan = pruner.sensitive_prune(0.4, skip_vars=["conv2d_26.w_0"])
+flops = dygraph_flops(net, [1, 3, 32, 32])
+print(f"FLOPs after pruning: {flops}")
+print(f"Pruned FLOPs: {round(plan.pruned_flops*100, 2)}%")
+result = model.evaluate(val_dataset,batch_size=128, log_freq=10)
+print(f"before fine-tuning: {result}")
+```
+
+### 5.4 重训练
+
+```python
+optimizer = paddle.optimizer.Momentum(
+        learning_rate=0.1,
+        parameters=net.parameters())
+model.prepare(
+        optimizer,
+        paddle.nn.CrossEntropyLoss(),
+        paddle.metric.Accuracy(topk=(1, 5)))
+model.fit(train_dataset, epochs=2, batch_size=128, verbose=1)
+result = model.evaluate(val_dataset,batch_size=128, log_freq=10)
+print(f"after fine-tuning: {result}")
+```

PaddleOCR @ 2bdaea56

-Subproject commit 2bdaea56566257cf73bd1cbbf834a16f4f7ac4cf

paddleslim/analysis/__init__.py

@@ -11,8 +11,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from .flops import flops
+from .flops import flops, dygraph_flops
 from .model_size import model_size
 from .latency import LatencyEvaluator, TableLatencyEvaluator
 
-__all__ = ['flops', 'model_size', 'LatencyEvaluator', 'TableLatencyEvaluator']
+__all__ = [
+    'flops', 'dygraph_flops', 'model_size', 'LatencyEvaluator',
+    'TableLatencyEvaluator'
+]

paddleslim/analysis/flops.py

@@ -11,11 +11,12 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
+import paddle
 import numpy as np
+import paddle.jit as jit
 from ..core import GraphWrapper
 
-__all__ = ["flops"]
+__all__ = ["flops", "dygraph_flops"]
 
 
 def flops(program, only_conv=True, detail=False):
@@ -80,3 +81,12 @@ def _graph_flops(graph, only_conv=True, detail=False):
         return flops, params2flops
     else:
         return flops
+
+
+def dygraph_flops(model, input_shape, only_conv=False, detail=False):
+    data = np.ones(tuple(input_shape)).astype("float32")
+    in_var = paddle.to_tensor(data)
+    _, traced = paddle.jit.TracedLayer.trace(model, [in_var])
+    program = traced.program
+    graph = GraphWrapper(program)
+    return _graph_flops(graph, only_conv=only_conv, detail=detail)

paddleslim/core/graph_wrapper.py

@@ -160,13 +160,21 @@ class OpWrapper(object):
         """
         Get all the varibales by the input name.
         """
-        return [self._graph.var(var_name) for var_name in self._op.input(name)]
+        if name in self._op.input_names:
+            return [
+                self._graph.var(var_name) for var_name in self._op.input(name)
+            ]
+        return []
 
     def outputs(self, name):
         """
         Get all the varibales by the output name.
         """
-        return [self._graph.var(var_name) for var_name in self._op.output(name)]
+        if name in self._op.output_names:
+            return [
+                self._graph.var(var_name) for var_name in self._op.output(name)
+            ]
+        return []
 
     def set_attr(self, key, value):
         """

paddleslim/dygraph/__init__.py

+from . import var_group
+from .var_group import *
+from . import l1norm_pruner
+from .l1norm_pruner import *
+from . import pruner
+from .pruner import *
+from . import filter_pruner
+from .filter_pruner import *
+from . import l2norm_pruner
+from .l2norm_pruner import *
+from . import fpgm_pruner
+from .fpgm_pruner import *
+
+__all__ = []
+
+__all__ += var_group.__all__
+__all__ += l1norm_pruner.__all__
+__all__ += l2norm_pruner.__all__
+__all__ += fpgm_pruner.__all__
+__all__ += pruner.__all__
+__all__ += filter_pruner.__all__

paddleslim/dygraph/filter_pruner.py

+import os
+import logging
+import numpy as np
+import pickle
+import paddle
+from ..common import get_logger
+from .var_group import *
+from .pruning_plan import *
+from .pruner import Pruner
+from ..analysis import dygraph_flops as flops
+from .var_group import VarGroup
+
+__all__ = ['Status', 'FilterPruner']
+
+_logger = get_logger(__name__, logging.INFO)
+
+CONV_OP_TYPE = paddle.nn.Conv2D
+FILTER_DIM = [0]
+CONV_WEIGHT_NAME = "weight"
+
+
+class Status():
+    def __init__(self, src=None):
+        self.sensitivies = {}
+        self.accumulates = {}
+        self.is_ckp = True
+        if src is not None:
+            self.load(src)
+
+    def save(self, dst):
+        with open(dst, 'wb') as f:
+            pickle.dump(self, f)
+            _logger.info("Save status into {}".format(dst))
+
+    def load(self, src):
+        with open(src, 'rb') as f:
+            data = pickle.load(f)
+            self.sensitivies = data.sensitivies
+            self.accumulates = data.accumulates
+            self.is_ckp = data.is_ckp
+            _logger.info("Load status from {}".format(src))
+
+
+class FilterPruner(Pruner):
+    """
+    Pruner used to prune filter structure in convolution layer.
+
+    Args:
+        model(paddle.nn.Layer): The target model to be pruned.
+        input_shape(list<int>): The input shape of model. It is used to trace the graph of the model.
+        sen_file(str, optional): The absolute path of file that stores computed sensitivities. If it is
+                              set rightly, 'FilterPruner::sensitive' function can not be called anymore
+                              in next step. Default: None.
+    
+    """
+
+    def __init__(self, model, input_shape, sen_file=None):
+        super(FilterPruner, self).__init__(model, input_shape)
+        self._status = Status(sen_file)
+        # sensitive and var_group are just used in filter pruning
+        self.var_group = VarGroup(model, input_shape)
+
+    def sensitive(self,
+                  eval_func=None,
+                  sen_file=None,
+                  target_vars=None,
+                  skip_vars=None):
+        """
+        Compute or get sensitivities of model in current pruner. It will return a cached sensitivities when all the arguments are "None".
+
+        This function return a dict storing sensitivities as below:
+    
+        .. code-block:: python
+    
+               {"weight_0":
+                   {0.1: 0.22,
+                    0.2: 0.33
+                   },
+                 "weight_1":
+                   {0.1: 0.21,
+                    0.2: 0.4
+                   }
+               }
+    
+        ``weight_0`` is parameter name of convolution. ``sensitivities['weight_0']`` is a dict in which key is pruned ratio and value is the percent of losses.
+
+        Args:
+          eval_func(function, optional): The function to evaluate the model in current pruner. This function should have an empy arguments list and return a score with type "float32". Default: None.
+          sen_file(str, optional): The absolute path of file to save sensitivities into local filesystem. Default: None.
+          target_vars(list, optional): The names of tensors whose sensitivity will be computed. "None" means all weights in convolution layer will be computed. Default: None.
+          skip_vars(list, optional): The names of tensors whose sensitivity won't be computed. "None" means skip nothing. Default: None.
+    
+        Returns:
+           dict: A dict storing sensitivities.       
+
+        """
+        if eval_func is None and sen_file is None:
+            return self._status.sensitivies
+        if sen_file is not None and os.path.isfile(sen_file):
+            self._status.load(sen_file)
+
+        if not self._status.is_ckp:
+            return self._status
+
+        self._cal_sensitive(
+            self.model,
+            eval_func,
+            status_file=sen_file,
+            target_vars=target_vars,
+            skip_vars=skip_vars)
+
+        self._status.is_ckp = False
+        return self._status.sensitivies
+
+    def _get_ratios_by_loss(self, sensitivities, loss, skip_vars=[]):
+        """
+        Get the max ratio of each parameter. The loss of accuracy must be less than given `loss`
+        when the single parameter was pruned by the max ratio. 
+        
+        Args:
+          
+          sensitivities(dict): The sensitivities used to generate a group of pruning ratios. The key of dict
+                               is name of parameters to be pruned. The value of dict is a list of tuple with
+                               format `(pruned_ratio, accuracy_loss)`.
+          loss(float): The threshold of accuracy loss.
+          skip_vars(list, optional): The names of tensors whose sensitivity won't be computed. "None" means skip nothing. Default: None.
+    
+        Returns:
+    
+          dict: A group of ratios. The key of dict is name of parameters while the value is the ratio to be pruned.
+        """
+        ratios = {}
+        for param, losses in sensitivities.items():
+            if param in skip_vars:
+                continue
+            losses = losses.items()
+            losses = list(losses)
+            losses.sort()
+            for i in range(len(losses))[::-1]:
+                if losses[i][1] <= loss:
+                    if i == (len(losses) - 1):
+                        ratios[param] = float(losses[i][0])
+                    else:
+                        r0, l0 = losses[i]
+                        r1, l1 = losses[i + 1]
+                        r0 = float(r0)
+                        r1 = float(r1)
+                        d0 = loss - l0
+                        d1 = l1 - loss
+
+                        ratio = r0 + (loss - l0) * (r1 - r0) / (l1 - l0)
+                        ratios[param] = ratio
+                        if ratio > 1:
+                            _logger.info(losses, ratio, (r1 - r0) / (l1 - l0),
+                                         i)
+
+                    break
+        return ratios
+
+    def _round_to(self, ratios, dims=[0], factor=8):
+        ret = {}
+        for name in ratios:
+            ratio = ratios[name]
+            dim = self._var_shapes[name][dims[0]]
+            remained = round((1 - ratio) * dim / factor) * factor
+            if remained == 0:
+                remained = factor
+            ratio = float(dim - remained) / dim
+            ratio = ratio if ratio > 0 else 0.
+            ret[name] = ratio
+        return ret
+
+    def get_ratios_by_sensitivity(self,
+                                  pruned_flops,
+                                  align=None,
+                                  dims=[0],
+                                  skip_vars=[]):
+        """
+         Get a group of ratios by sensitivities.
+         Args:
+             pruned_flops(float): The excepted rate of FLOPs to be pruned. It should be in range (0, 1).
+             align(int, optional): Round the size of each pruned dimension to multiple of 'align' if 'align' is not None. Default: None.
+             dims(list, optional): The dims to be pruned on. [0] means pruning channels of output for convolution. Default: [0].
+             skip_vars(list, optional): The names of tensors whose sensitivity won't be computed. "None" means skip nothing. Default: None.
+
+        Returns:
+            tuple: A tuple with format ``(ratios, pruned_flops)`` . "ratios" is a dict whose key is name of tensor and value is ratio to be pruned. "pruned_flops" is the ratio of total pruned FLOPs in the model.
+        """
+        base_flops = flops(self.model, self.input_shape)
+
+        _logger.info("Base FLOPs: {}".format(base_flops))
+        low = 0.
+        up = 1.0
+        history = set()
+        while low < up:
+            loss = (low + up) / 2
+            ratios = self._get_ratios_by_loss(
+                self._status.sensitivies, loss, skip_vars=skip_vars)
+            _logger.debug("pruning ratios: {}".format(ratios))
+            if align is not None:
+                ratios = self._round_to(ratios, dims=dims, factor=align)
+            plan = self.prune_vars(ratios, axis=dims)
+            _logger.debug("pruning plan: {}".format(plan))
+            c_flops = flops(self.model, self.input_shape)
+            _logger.debug("FLOPs after pruning: {}".format(c_flops))
+            c_pruned_flops = (base_flops - c_flops) / base_flops
+            plan.restore(self.model)
+            _logger.debug("Seaching ratios, pruned FLOPs: {}".format(
+                c_pruned_flops))
+            key = str(round(c_pruned_flops, 4))
+            if key in history:
+                return ratios, c_pruned_flops
+            history.add(key)
+            if c_pruned_flops < pruned_flops:
+                low = loss
+            elif c_pruned_flops > pruned_flops:
+                up = loss
+            else:
+                return ratios, c_pruned_flops
+        return ratios, c_pruned_flops
+
+    def _cal_sensitive(self,
+                       model,
+                       eval_func,
+                       status_file=None,
+                       target_vars=None,
+                       skip_vars=None):
+        sensitivities = self._status.sensitivies
+        baseline = eval_func()
+        ratios = np.arange(0.1, 1, step=0.1)
+        for group in self.var_group.groups:
+            var_name = group[0][0]
+            dims = group[0][1]
+
+            if target_vars is not None and var_name not in target_vars:
+                continue
+            if skip_vars is not None and var_name in skip_vars:
+                continue
+
+            if var_name not in sensitivities:
+                sensitivities[var_name] = {}
+            for ratio in ratios:
+                ratio = round(ratio, 2)
+                if ratio in sensitivities[var_name]:
+                    _logger.debug("{}, {} has computed.".format(var_name,
+                                                                ratio))
+                    continue
+                plan = self.prune_var(var_name, dims, ratio, apply="lazy")
+                pruned_metric = eval_func()
+                loss = (baseline - pruned_metric) / baseline
+                _logger.info("pruned param: {}; {}; loss={}".format(
+                    var_name, ratio, loss))
+                sensitivities[var_name][ratio] = loss
+                self._status.save(status_file)
+                plan.restore(model)
+
+        return sensitivities
+
+    def sensitive_prune(self, pruned_flops, skip_vars=[], align=None):
+
+        # skip depthwise convolutions
+        for layer in self.model.sublayers():
+            if isinstance(layer,
+                          paddle.nn.layer.conv.Conv2D) and layer._groups > 1:
+                for param in layer.parameters(include_sublayers=False):
+                    skip_vars.append(param.name)
+        _logger.debug("skip vars: {}".format(skip_vars))
+        self.restore()
+        ratios, pruned_flops = self.get_ratios_by_sensitivity(
+            pruned_flops, align=align, dims=FILTER_DIM, skip_vars=skip_vars)
+        _logger.debug("ratios: {}".format(ratios))
+        self.plan = self.prune_vars(ratios, FILTER_DIM)
+        self.plan._pruned_flops = pruned_flops
+        return self.plan
+
+    def restore(self):
+        if self.plan is not None:
+            self.plan.restore(self.model)
+
+    def cal_mask(self, var_name, pruned_ratio, group):
+        """
+        
+        {
+          var_name: {
+                        'layer': sub_layer,
+                        'var': variable,
+                        'value': np.array([]),
+                        'pruned_dims': [1],
+                      }
+        }
+        """
+        raise NotImplemented("cal_mask is not implemented")
+
+    def prune_var(self, var_name, pruned_dims, pruned_ratio, apply="impretive"):
+        """
+        Pruning a variable.
+        Parameters:
+            var_name(str): The name of variable.
+            pruned_dims(list<int>): The axies to be pruned. For convolution with format [out_c, in_c, k, k],
+                             'axis=[0]' means pruning filters and 'axis=[0, 1]' means pruning kernels.
+            pruned_ratio(float): The ratio of pruned values in one variable.
+
+        Returns:
+            plan: An instance of PruningPlan that can be applied on model by calling 'plan.apply(model)'.
+
+        """
+
+        if isinstance(pruned_dims, int):
+            pruned_dims = [pruned_dims]
+        group = self.var_group.find_group(var_name, pruned_dims)
+        _logger.debug("found group with {}: {}".format(var_name, group))
+        plan = PruningPlan(self.model.full_name)
+
+        for sub_layer in self.model.sublayers():
+            for param in sub_layer.parameters(include_sublayers=False):
+                if param.name in group:
+                    group[param.name]['layer'] = sub_layer
+                    group[param.name]['var'] = param
+                    group[param.name]['value'] = np.array(param.value()
+                                                          .get_tensor())
+                    _logger.debug(f"set value of {param.name} into group")
+
+        mask = self.cal_mask(var_name, pruned_ratio, group)
+        for _name in group:
+            dims = group[_name]['pruned_dims']
+            if isinstance(dims, int):
+                dims = [dims]
+            plan.add(_name, PruningMask(dims, mask, pruned_ratio))
+        if apply == "lazy":
+            plan.apply(self.model, lazy=True)
+        elif apply == "impretive":
+            plan.apply(self.model, lazy=False)
+        return plan

paddleslim/dygraph/fpgm_pruner.py

+import logging
+import numpy as np
+import paddle
+from ..common import get_logger
+from .var_group import *
+from .pruning_plan import *
+from .filter_pruner import FilterPruner
+
+__all__ = ['FPGMFilterPruner']
+
+_logger = get_logger(__name__, logging.INFO)
+
+
+class FPGMFilterPruner(FilterPruner):
+    def __init__(self, model, input_shape, sen_file=None):
+        super(FPGMFilterPruner, self).__init__(
+            model, input_shape, sen_file=sen_file)
+
+    def cal_mask(self, var_name, pruned_ratio, group):
+        value = group[var_name]['value']
+        pruned_dims = group[var_name]['pruned_dims']
+        assert (pruned_dims == [0])
+
+        dist_sum_list = []
+        for out_i in range(value.shape[0]):
+            dist_sum = self.get_distance_sum(value, out_i)
+            dist_sum_list.append(dist_sum)
+        scores = np.array(dist_sum_list)
+
+        sorted_idx = scores.argsort()
+        pruned_num = int(round(len(sorted_idx) * pruned_ratio))
+        pruned_idx = sorted_idx[:pruned_num]
+        mask_shape = [value.shape[i] for i in pruned_dims]
+        mask = np.ones(mask_shape, dtype="int32")
+        mask[pruned_idx] = 0
+        return mask
+
+    def get_distance_sum(self, value, out_idx):
+        w = value.view()
+        w.shape = value.shape[0], np.product(value.shape[1:])
+        selected_filter = np.tile(w[out_idx], (w.shape[0], 1))
+        x = w - selected_filter
+        x = np.sqrt(np.sum(x * x, -1))
+        return x.sum()

paddleslim/dygraph/l1norm_pruner.py

+import logging
+import numpy as np
+import paddle
+from ..common import get_logger
+from .var_group import *
+from .pruning_plan import *
+from .filter_pruner import FilterPruner
+
+__all__ = ['L1NormFilterPruner']
+
+_logger = get_logger(__name__, logging.INFO)
+
+
+class L1NormFilterPruner(FilterPruner):
+    def __init__(self, model, input_shape, sen_file=None):
+        super(L1NormFilterPruner, self).__init__(
+            model, input_shape, sen_file=sen_file)
+
+    def cal_mask(self, var_name, pruned_ratio, group):
+        value = group[var_name]['value']
+        pruned_dims = group[var_name]['pruned_dims']
+        reduce_dims = [
+            i for i in range(len(value.shape)) if i not in pruned_dims
+        ]
+        l1norm = np.mean(np.abs(value), axis=tuple(reduce_dims))
+        sorted_idx = l1norm.argsort()
+        pruned_num = int(round(len(sorted_idx) * pruned_ratio))
+        pruned_idx = sorted_idx[:pruned_num]
+        mask_shape = [value.shape[i] for i in pruned_dims]
+        mask = np.ones(mask_shape, dtype="int32")
+        mask[pruned_idx] = 0
+        return mask

paddleslim/dygraph/l2norm_pruner.py

+import logging
+import numpy as np
+import paddle
+from ..common import get_logger
+from .var_group import *
+from .pruning_plan import *
+from .filter_pruner import FilterPruner
+
+__all__ = ['L2NormFilterPruner']
+
+_logger = get_logger(__name__, logging.INFO)
+
+
+class L2NormFilterPruner(FilterPruner):
+    def __init__(self, model, input_shape, sen_file=None):
+        super(L2NormFilterPruner, self).__init__(
+            model, input_shape, sen_file=sen_file)
+
+    def cal_mask(self, var_name, pruned_ratio, group):
+        value = group[var_name]['value']
+        pruned_dims = group[var_name]['pruned_dims']
+        reduce_dims = [
+            i for i in range(len(value.shape)) if i not in pruned_dims
+        ]
+
+        # scores = np.mean(np.abs(value), axis=tuple(reduce_dims))
+        scores = np.sqrt(np.sum(np.square(value), axis=tuple(reduce_dims)))
+        sorted_idx = scores.argsort()
+        pruned_num = int(round(len(sorted_idx) * pruned_ratio))
+        pruned_idx = sorted_idx[:pruned_num]
+        mask_shape = [value.shape[i] for i in pruned_dims]
+        mask = np.ones(mask_shape, dtype="int32")
+        mask[pruned_idx] = 0
+        return mask

paddleslim/dygraph/pruner.py

+import os
+import pickle
+import numpy as np
+import logging
+from .pruning_plan import PruningPlan
+from ..common import get_logger
+
+__all__ = ["Pruner"]
+
+_logger = get_logger(__name__, level=logging.INFO)
+
+
+class Pruner(object):
+    """
+    Pruner used to resize or mask dimensions of variables.
+    Args:
+        model(paddle.nn.Layer): The target model to be pruned.
+        input_shape(list<int>): The input shape of model. It is used to trace the graph of the model.
+        
+    """
+
+    def __init__(self, model, input_shape):
+        self.model = model
+        self.input_shape = input_shape
+        self._var_shapes = {}
+        for var in model.parameters():
+            self._var_shapes[var.name] = var.shape
+        self.plan = None
+
+    def status(self, data=None, eval_func=None, status_file=None):
+        raise NotImplemented("status is not implemented")
+
+    def prune_var(self, var_name, axis, pruned_ratio, apply="impretive"):
+        raise NotImplemented("prune_var is not implemented")
+
+    def prune_vars(self, ratios, axis, apply="impretive"):
+        """
+        Pruning variables by given ratios.
+        Args:
+            ratios(dict<str, float>): The key is the name of variable to be pruned and the
+                                      value is the pruned ratio.
+            axis(list): The dimensions to be pruned on.
+
+        Returns:
+            plan(PruningPlan): The pruning plan.
+        """
+        global_plan = PruningPlan(self.model.full_name)
+        for var, ratio in ratios.items():
+            if not global_plan.contains(var, axis):
+                plan = self.prune_var(var, axis, ratio, apply=None)
+                global_plan.extend(plan)
+        if apply == "lazy":
+            global_plan.apply(self.model, lazy=True)
+        elif apply == "impretive":
+            global_plan.apply(self.model, lazy=False)
+
+        return global_plan

paddleslim/dygraph/pruning_plan.py

+import paddle
+import collections
+import numpy as np
+import logging
+from ..common import get_logger
+from paddle.fluid import core
+_logger = get_logger(__name__, level=logging.INFO)
+
+__all__ = ['PruningPlan', 'PruningMask']
+
+
+class PruningMask():
+    def __init__(self, dims, mask, ratio):
+        self._dims = dims
+        self._mask = mask
+        self._pruned_ratio = ratio
+
+    @property
+    def dims(self):
+        return self._dims
+
+    @dims.setter
+    def dims(self, value):
+        if not isinstance(value, collections.Iterator):
+            raise ValueError(
+                "The dims of PruningMask must be instance of collections.Iterator."
+            )
+        if self._mask is not None:
+            assert (
+                len(self._mask.shape) == len(value),
+                "The length of value must be same with shape of mask in current PruningMask instance."
+            )
+        self._dims = list(value)
+
+    @property
+    def mask(self):
+        return self._mask
+
+    @mask.setter
+    def mask(self, value):
+        assert (isinstance(value, PruningMask))
+        if self._dims is not None:
+            assert (
+                len(self._mask.shape) == len(value),
+                "The length of value must be same with shape of mask in current PruningMask instance."
+            )
+        self._mask = value
+
+    def __str__(self):
+        return "{}\t{}".format(self._pruned_ratio, self._dims)
+
+
+class PruningPlan():
+    def __init__(self, model_name=None):
+        # {"conv_weight": (axies, mask)}
+
+        self._model_name = model_name
+        self._plan_id = model_name
+        self._masks = {}  #{param_name: pruning_mask}
+        self._dims = {}
+        self._pruned_size = None
+        self._total_size = None
+        self._pruned_flops = None
+        self._pruned_size = None
+        self._model_size = None
+
+    @property
+    def pruned_flops(self):
+        return self._pruned_flops
+
+    @pruned_flops.setter
+    def pruned_flops(self, value):
+        self._pruned_flops = value
+
+    def add(self, var_name, pruning_mask):
+        assert (isinstance(pruning_mask, PruningMask))
+        if var_name not in self._masks:
+            self._masks[var_name] = []
+        self._masks[var_name].append(pruning_mask)
+        if var_name not in self._dims:
+            self._dims[var_name] = []
+        self._dims[var_name].append(pruning_mask.dims)
+
+    @property
+    def masks(self):
+        return self._masks
+
+    def extend(self, plan):
+        assert (isinstance(plan, PruningPlan))
+        for var_name in plan.masks:
+            for mask in plan.masks[var_name]:
+                if not self.contains(var_name, mask.dims):
+                    self.add(var_name, mask)
+
+    def contains(self, var_name, dims=None):
+        return (var_name in self._dims) and (dims is None or
+                                             dims in self._dims[var_name])
+
+    def __str__(self):
+        details = "\npruned FLOPs: {}".format(self._pruned_flops)
+        head = "variable name\tpruned ratio\tpruned dims\n"
+        return head + "\n".join([
+            "{}:\t{}".format(name, ",".join([str(m) for m in mask]))
+            for name, mask in self._masks.items()
+        ]) + details
+
+    def apply(self, model, lazy=False):
+        if lazy:
+            self.lazy_apply(model)
+        else:
+            self.imperative_apply(model)
+
+    def lazy_apply(self, model):
+        for name, sub_layer in model.named_sublayers():
+            for param in sub_layer.parameters(include_sublayers=False):
+                if param.name in self._masks:
+                    for _mask in self._masks[param.name]:
+                        dims = _mask.dims
+                        mask = _mask.mask
+                        t_value = param.value().get_tensor()
+                        value = np.array(t_value).astype("float32")
+                        # The name of buffer can not contains "."
+                        backup_name = param.name.replace(".", "_") + "_backup"
+                        if backup_name not in sub_layer._buffers:
+                            sub_layer.register_buffer(backup_name,
+                                                      paddle.to_tensor(value))
+                            _logger.debug("Backup values of {} into buffers.".
+                                          format(param.name))
+                        expand_mask_shape = [1] * len(value.shape)
+                        for i in dims:
+                            expand_mask_shape[i] = value.shape[i]
+                        _logger.debug("Expanded mask shape: {}".format(
+                            expand_mask_shape))
+                        expand_mask = mask.reshape(expand_mask_shape).astype(
+                            "float32")
+
+                        p = t_value._place()
+                        if p.is_cpu_place():
+                            place = paddle.CPUPlace()
+                        elif p.is_cuda_pinned_place():
+                            place = paddle.CUDAPinnedPlace()
+                        else:
+                            p = core.Place()
+                            p.set_place(t_value._place())
+                            place = paddle.CUDAPlace(p.gpu_device_id())
+
+                        t_value.set(value * expand_mask, place)
+
+    def imperative_apply(self, model):
+        """
+        Pruning values of variable imperatively. It is valid when pruning
+        on one dimension.
+        """
+
+        for name, sub_layer in model.named_sublayers():
+            for param in sub_layer.parameters(include_sublayers=False):
+                if param.name in self._masks:
+                    for _mask in self._masks[param.name]:
+                        dims = _mask.dims
+                        mask = _mask.mask
+                        assert (
+                            len(dims) == 1,
+                            "Imperative mode only support for pruning"
+                            "on one dimension, but get dims {} when pruning parameter {}".
+                            format(dims, param.name))
+                        t_value = param.value().get_tensor()
+                        value = np.array(t_value).astype("float32")
+                        # The name of buffer can not contains "."
+                        backup_name = param.name.replace(".", "_") + "_backup"
+                        if backup_name not in sub_layer._buffers:
+                            sub_layer.register_buffer(backup_name,
+                                                      paddle.to_tensor(value))
+                            _logger.debug("Backup values of {} into buffers.".
+                                          format(param.name))
+                        bool_mask = mask.astype(bool)
+                        pruned_value = np.apply_along_axis(
+                            lambda data: data[bool_mask], dims[0], value)
+
+                        p = t_value._place()
+                        if p.is_cpu_place():
+                            place = paddle.CPUPlace()
+                        elif p.is_cuda_pinned_place():
+                            place = paddle.CUDAPinnedPlace()
+                        else:
+                            p = core.Place()
+                            p.set_place(t_value._place())
+                            place = paddle.CUDAPlace(p.gpu_device_id())
+
+                        t_value.set(pruned_value, place)
+                        if isinstance(sub_layer, paddle.nn.layer.conv.Conv2D):
+                            if sub_layer._groups > 1:
+                                _logger.debug(
+                                    "Update groups of conv form {} to {}".
+                                    format(sub_layer._groups,
+                                           pruned_value.shape[0]))
+                                sub_layer._groups = pruned_value.shape[0]
+                    # for training
+                    if param.trainable:
+                        param.clear_gradient()
+
+    def restore(self, model):
+        for name, sub_layer in model.named_sublayers():
+            for param in sub_layer.parameters(include_sublayers=False):
+                backup_name = "_".join([param.name.replace(".", "_"), "backup"])
+                if backup_name in sub_layer._buffers:
+                    _logger.debug("Restore values of variable: {}".format(
+                        param.name))
+                    t_value = param.value().get_tensor()
+                    t_backup = sub_layer._buffers[backup_name].value(
+                    ).get_tensor()
+
+                    p = t_value._place()
+                    if p.is_cpu_place():
+                        place = paddle.CPUPlace()
+                    elif p.is_cuda_pinned_place():
+                        place = paddle.CUDAPinnedPlace()
+                    else:
+                        p = core.Place()
+                        p.set_place(t_value._place())
+                        place = paddle.CUDAPlace(p.gpu_device_id())
+
+                    t_value.set(np.array(t_backup).astype("float32"), place)
+
+                    if isinstance(sub_layer, paddle.nn.layer.conv.Conv2D):
+                        if sub_layer._groups > 1:
+                            _logger.debug(
+                                "Update groups of conv form {} to {}".format(
+                                    sub_layer._groups, t_value.shape()[0]))
+                            sub_layer._groups = t_value.shape()[0]
+                    del sub_layer._buffers[backup_name]

paddleslim/dygraph/var_group.py

+import numpy as np
+import logging
+import paddle
+from paddle.fluid.dygraph import TracedLayer
+from ..core import GraphWrapper
+from ..prune import collect_convs
+from ..common import get_logger
+
+__all__ = ["VarGroup"]
+
+_logger = get_logger(__name__, level=logging.INFO)
+
+
+class VarGroup():
+    def __init__(self, model, input_shape):
+        self.groups = []
+        self._parse_model(model, input_shape)
+
+    def _to_dict(self, group):
+        ret = {}
+        for _name, _axis, _idx in group:
+            if isinstance(_axis, int):
+                _axis = [_axis]  # TODO: fix
+            ret[_name] = {'pruned_dims': _axis, 'pruned_idx': _idx}
+        return ret
+
+    def find_group(self, var_name, axis):
+        for group in self.groups:
+            for _name, _axis, _ in group:
+                if isinstance(_axis, int):
+                    _axis = [_axis]  # TODO: fix
+                if _name == var_name and _axis == axis:
+                    return self._to_dict(group)
+
+    def _parse_model(self, model, input_shape):
+        _logger.debug("Parsing model with input: {}".format(input_shape))
+        data = np.ones(tuple(input_shape)).astype("float32")
+        in_var = paddle.to_tensor(data)
+        out_dygraph, static_layer = TracedLayer.trace(model, inputs=[in_var])
+        graph = GraphWrapper(static_layer.program)
+
+        visited = {}
+        for name, param in model.named_parameters():
+            group = collect_convs([param.name], graph,
+                                  visited)[0]  # [(name, axis, pruned_idx)]
+            if len(group) > 0:
+                self.groups.append(group)
+        _logger.debug("Found {} groups.".format(len(self.groups)))
+
+    def __str__(self):
+        return "\n".join([str(group) for group in self.groups])

paddleslim/models/__init__.py

@@ -16,7 +16,11 @@ from __future__ import absolute_import
 from .util import image_classification
 from .slimfacenet import SlimFaceNet_A_x0_60, SlimFaceNet_B_x0_75, SlimFaceNet_C_x0_75
 from .slim_mobilenet import SlimMobileNet_v1, SlimMobileNet_v2, SlimMobileNet_v3, SlimMobileNet_v4, SlimMobileNet_v5
-from .mobilenet import MobileNet
-from .resnet import ResNet50
+from ..models import mobilenet
+from .mobilenet import *
+from ..models import resnet
+from .resnet import *
 
-__all__ = ["image_classification", "MobileNet", "ResNet50"]
+__all__ = ["image_classification"]
+__all__ += mobilenet.__all__
+__all__ += resnet.__all__

paddleslim/prune/group_param.py

@@ -13,11 +13,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import logging
 from ..core import GraphWrapper
+from ..common import get_logger
 from .prune_walker import PRUNE_WORKER
 
 __all__ = ["collect_convs"]
 
+_logger = get_logger(__name__, level=logging.INFO)
+
 
 def collect_convs(params, graph, visited={}):
     """Collect convolution layers of graph into groups. The layers in the same group is relative on pruning operation.
@@ -66,6 +70,11 @@ def collect_convs(params, graph, visited={}):
                     break
         else:
             cls = PRUNE_WORKER.get(target_op.type())
+            if cls is None:
+                _logger.info("No walker for operator: {}".format(target_op.type(
+                )))
+                groups.append(pruned_params)
+                continue
             walker = cls(target_op,
                          pruned_params=pruned_params,
                          visited=visited)

paddleslim/prune/prune_walker.py

@@ -94,7 +94,29 @@ class conv2d(PruneWorker):
     def __init__(self, op, pruned_params, visited={}):
         super(conv2d, self).__init__(op, pruned_params, visited)
 
+    def _is_depthwise_conv(self, op):
+        data_format = self.op.attr("data_format")
+        channel_axis = 1
+        if data_format == "NHWC":
+            channel_axis = 3
+
+        filter_shape = self.op.inputs("Filter")[0].shape()
+        input_shape = self.op.inputs("Input")[0].shape()
+        num_channels = input_shape[channel_axis]
+        groups = self.op.attr("groups")
+        num_filters = filter_shape[0]
+        return (num_channels == groups and num_channels != 1 and
+                num_filters % num_channels == 0)
+
     def _prune(self, var, pruned_axis, pruned_idx):
+
+        if self._is_depthwise_conv(self.op):
+            _logger.debug(f"Meet conv2d who is depthwise conv2d actually.")
+            walker = depthwise_conv2d(
+                self.op, self.pruned_params, visited=self.visited)
+            walker._prune(var, pruned_axis, pruned_idx)
+            return
+
         data_format = self.op.attr("data_format")
         channel_axis = 1
         if data_format == "NHWC":
@@ -234,8 +256,11 @@ class elementwise_op(PruneWorker):
 
     def _prune(self, var, pruned_axis, pruned_idx):
         axis = self.op.attr("axis")
-        if axis == -1:  # TODO
-            axis = 0
+        if axis == -1:
+            x = self.op.inputs("X")[0]
+            y = self.op.inputs("Y")[0]
+            axis = len(x.shape()) - len(y.shape())
+
         if var in self.op.outputs("Out"):
             for name in ["X", "Y"]:
                 actual_axis = pruned_axis
@@ -251,19 +276,27 @@ class elementwise_op(PruneWorker):
         else:
             if var in self.op.inputs("X"):
                 in_var = self.op.inputs("Y")[0]
-                if not (len(in_var.shape()) == 1 and in_var.shape()[0] == 1):
-                    if in_var.is_parameter():
-                        self.pruned_params.append(
-                            (in_var, pruned_axis - axis, pruned_idx))
+                y_pruned_axis = pruned_axis
+                if len(in_var.shape()) != len(var.shape()):
+                    assert (len(var.shape()) > len(in_var.shape()))
+                    if axis == -1:
+                        axis = len(var.shape()) - len(in_var.shape())
+                    y_pruned_axis = pruned_axis - axis
+
+                if y_pruned_axis >= 0 and not (len(in_var.shape()) == 1 and
+                                               in_var.shape()[0] == 1):
+                    self.pruned_params.append(
+                        (in_var, y_pruned_axis, pruned_idx))
                     pre_ops = in_var.inputs()
                     for op in pre_ops:
-                        self._prune_op(op, in_var, pruned_axis - axis,
-                                       pruned_idx)
+                        self._prune_op(op, in_var, y_pruned_axis, pruned_idx)
             elif var in self.op.inputs("Y"):
                 in_var = self.op.inputs("X")[0]
-                if not (len(in_var.shape()) == 1 and in_var.shape()[0] == 1):
-                    pre_ops = in_var.inputs()
+                if len(in_var.shape()) != len(var.shape()):
+                    assert (len(var.shape()) < len(in_var.shape()))
                     pruned_axis = pruned_axis + axis
+                if pruned_axis <= len(in_var.shape()):
+                    pre_ops = in_var.inputs()
                     for op in pre_ops:
                         self._prune_op(op, in_var, pruned_axis, pruned_idx)
 

paddleslim/prune/pruner.py

@@ -111,8 +111,10 @@ class Pruner():
                         "The weights have been pruned once.")
                 group_values = []
                 for name, axis, pruned_idx in group:
-                    values = np.array(scope.find_var(name).get_tensor())
-                    group_values.append((name, values, axis, pruned_idx))
+                    var = scope.find_var(name)
+                    if var is not None:
+                        values = np.array(var.get_tensor())
+                        group_values.append((name, values, axis, pruned_idx))
 
                 scores = self.criterion(
                     group_values, graph)  # [(name, axis, score, pruned_idx)]

tests/dygraph/test_filter_pruner.py

+# Copyright (c) 2019  PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+sys.path.append("../../")
+import unittest
+import time
+import numpy as np
+import paddle
+import paddle.fluid as fluid
+import paddle.vision.transforms as T
+from paddle.static import InputSpec as Input
+from paddleslim.dygraph import L1NormFilterPruner, L2NormFilterPruner, FPGMFilterPruner
+from paddleslim.dygraph import Status
+
+
+class TestStatus(unittest.TestCase):
+    def runTest(self):
+        status = Status()
+        status.sensitivies = {
+            "conv2d_1.weights": {
+                0.1: 0.11,
+                0.2: 0.22,
+                0.3: 0.33
+            }
+        }
+        local_file = "./sen_{}.pickle".format(time.time())
+        status.save(local_file)
+        status1 = Status(local_file)
+        for _name in status.sensitivies:
+            for _ratio, _loss in status.sensitivies[_name].items():
+                self.assertTrue(status1.sensitivies[_name][_ratio], _loss)
+
+
+class TestFilterPruner(unittest.TestCase):
+    def __init__(self, methodName='runTest', param_names=[]):
+        super(TestFilterPruner, self).__init__(methodName)
+        self._param_names = param_names
+        transform = T.Compose([T.Transpose(), T.Normalize([127.5], [127.5])])
+        self.train_dataset = paddle.vision.datasets.MNIST(
+            mode="train", backend="cv2", transform=transform)
+        self.val_dataset = paddle.vision.datasets.MNIST(
+            mode="test", backend="cv2", transform=transform)
+
+        def _reader():
+            for data in self.val_dataset:
+                yield data
+
+        self.val_reader = _reader
+
+    def runTest(self):
+        with fluid.unique_name.guard():
+            net = paddle.vision.models.LeNet()
+            optimizer = paddle.optimizer.Adam(
+                learning_rate=0.001, parameters=net.parameters())
+            inputs = [Input([None, 1, 28, 28], 'float32', name='image')]
+            labels = [Input([None, 1], 'int64', name='label')]
+            model = paddle.Model(net, inputs, labels)
+            model.prepare(
+                optimizer,
+                paddle.nn.CrossEntropyLoss(),
+                paddle.metric.Accuracy(topk=(1, 5)))
+            model.fit(self.train_dataset, epochs=1, batch_size=128, verbose=1)
+            pruners = []
+            pruner = L1NormFilterPruner(net, [1, 1, 28, 28])
+            pruners.append(pruner)
+            pruner = FPGMFilterPruner(net, [1, 1, 28, 28])
+            pruners.append(pruner)
+            pruner = L2NormFilterPruner(net, [1, 1, 28, 28])
+            pruners.append(pruner)
+
+            def eval_fn():
+                result = model.evaluate(
+                    self.val_dataset, batch_size=128, verbose=1)
+                return result['acc_top1']
+
+            sen_file = "_".join(["./dygraph_sen_", str(time.time())])
+            for pruner in pruners:
+                sen = pruner.sensitive(
+                    eval_func=eval_fn,
+                    sen_file=sen_file,
+                    target_vars=self._param_names)
+                base_acc = eval_fn()
+                plan = pruner.sensitive_prune(0.01)
+                pruner.restore()
+                restore_acc = eval_fn()
+                self.assertTrue(restore_acc == base_acc)
+
+                plan = pruner.sensitive_prune(0.01, align=4)
+                for param in net.parameters():
+                    if param.name in self._param_names:
+                        self.assertTrue(param.shape[0] % 4 == 0)
+                pruner.restore()
+
+
+def add_cases(suite):
+    suite.addTest(TestStatus())
+    suite.addTest(TestFilterPruner(param_names=["conv2d_0.w_0"]))
+
+
+def load_tests(loader, standard_tests, pattern):
+    suite = unittest.TestSuite()
+    add_cases(suite)
+    return suite
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/dygraph/test_flops.py

+import sys
+sys.path.append("../../")
+import unittest
+from paddleslim.analysis import dygraph_flops as flops
+from paddle.vision.models import mobilenet_v1, resnet50
+
+
+class TestFlops(unittest.TestCase):
+    def __init__(self, methodName='runTest', net=None, gt=None):
+        super(TestFlops, self).__init__(methodName)
+        self._net = net
+        self._gt = gt
+
+    def runTest(self):
+        net = self._net(pretrained=False)
+        FLOPs = flops(net, (1, 3, 32, 32))
+        self.assertTrue(FLOPs == self._gt)
+
+
+def add_cases(suite):
+    suite.addTest(TestFlops(net=mobilenet_v1, gt=11792896.0))
+    suite.addTest(TestFlops(net=resnet50, gt=83872768.0))
+
+
+def load_tests(loader, standard_tests, pattern):
+    suite = unittest.TestSuite()
+    add_cases(suite)
+    return suite
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/dygraph/test_prune.py

+# Copyright (c) 2019  PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+sys.path.append("../../")
+import unittest
+import paddle
+import paddle.fluid as fluid
+from paddleslim.dygraph import L1NormFilterPruner
+from paddle.vision.models import mobilenet_v1, resnet50
+from paddleslim.prune import Pruner
+
+
+class TestPrune(unittest.TestCase):
+    def __init__(self, methodName='runTest', ratios=None, net=None):
+        super(TestPrune, self).__init__(methodName)
+        self._net = net
+        self._ratios = ratios
+
+    def runTest(self):
+        static_shapes = self.static_prune(self._net, self._ratios)
+        dygraph_shapes = self.dygraph_prune(self._net, self._ratios)
+        all_right = True
+        for _name, _shape in static_shapes.items():
+            if dygraph_shapes[_name] != list(_shape):
+                print(
+                    f"name: {_name}; static shape: {_shape}, dygraph shape: {dygraph_shapes[_name]}"
+                )
+                all_right = False
+        self.assertTrue(all_right)
+
+    def dygraph_prune(self, net, ratios):
+        paddle.disable_static()
+        model = net(pretrained=False)
+        pruner = L1NormFilterPruner(model, [1, 3, 16, 16])
+        pruner.prune_vars(ratios, [0])
+        shapes = {}
+        for param in model.parameters():
+            shapes[param.name] = param.shape
+        return shapes
+
+    def static_prune(self, net, ratios):
+        paddle.enable_static()
+        main_program = fluid.Program()
+        startup_program = fluid.Program()
+        with fluid.unique_name.guard():
+            with fluid.program_guard(main_program, startup_program):
+                input = fluid.data(name="image", shape=[None, 3, 16, 16])
+                model = net(pretrained=False)
+                out = model(input)
+
+        place = fluid.CPUPlace()
+        exe = fluid.Executor(place)
+        scope = fluid.Scope()
+        exe.run(startup_program, scope=scope)
+        pruner = Pruner()
+        main_program, _, _ = pruner.prune(
+            main_program,
+            scope,
+            params=ratios.keys(),
+            ratios=ratios.values(),
+            place=place,
+            lazy=False,
+            only_graph=False,
+            param_backup=None,
+            param_shape_backup=None)
+
+        shapes = {}
+        for param in main_program.global_block().all_parameters():
+            shapes[param.name] = param.shape
+        return shapes
+
+
+def add_cases(suite):
+    suite.addTest(
+        TestPrune(
+            net=mobilenet_v1,
+            ratios={"conv2d_22.w_0": 0.5,
+                    "conv2d_8.w_0": 0.6}))
+    suite.addTest(
+        TestPrune(
+            net=resnet50, ratios={"conv2d_22.w_0": 0.5,
+                                  "conv2d_8.w_0": 0.6}))
+
+
+def load_tests(loader, standard_tests, pattern):
+    suite = unittest.TestSuite()
+    add_cases(suite)
+    return suite
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/dygraph/test_sensitivity.py

+# Copyright (c) 2019  PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import sys
+sys.path.append("../../")
+import unittest
+import time
+import numpy as np
+import paddle
+import paddle.fluid as fluid
+from paddleslim.prune import sensitivity
+import paddle.vision.transforms as T
+from paddle.static import InputSpec as Input
+from paddleslim.dygraph import L1NormFilterPruner
+
+
+class TestSensitivity(unittest.TestCase):
+    def __init__(self, methodName='runTest', pruner=None, param_names=[]):
+        super(TestSensitivity, self).__init__(methodName)
+        self._pruner = pruner
+        self._param_names = param_names
+        transform = T.Compose([T.Transpose(), T.Normalize([127.5], [127.5])])
+        self.train_dataset = paddle.vision.datasets.MNIST(
+            mode="train", backend="cv2", transform=transform)
+        self.val_dataset = paddle.vision.datasets.MNIST(
+            mode="test", backend="cv2", transform=transform)
+
+        def _reader():
+            for data in self.val_dataset:
+                yield data
+
+        self.val_reader = _reader
+
+    def runTest(self):
+        dygraph_sen, params = self.dygraph_sen()
+        static_sen = self.static_sen(params)
+        all_right = True
+        for _name, _value in dygraph_sen.items():
+            _losses = {}
+            for _ratio, _loss in static_sen[_name].items():
+                _losses[round(_ratio, 2)] = _loss
+            for _ratio, _loss in _value.items():
+                if not np.allclose(_losses[_ratio], _loss, atol=1e-2):
+                    print(
+                        f'static loss: {static_sen[_name][_ratio]}; dygraph loss: {_loss}'
+                    )
+                    all_right = False
+        self.assertTrue(all_right)
+
+    def dygraph_sen(self):
+        paddle.disable_static()
+        net = paddle.vision.models.LeNet()
+        optimizer = paddle.optimizer.Adam(
+            learning_rate=0.001, parameters=net.parameters())
+        inputs = [Input([None, 1, 28, 28], 'float32', name='image')]
+        labels = [Input([None, 1], 'int64', name='label')]
+        model = paddle.Model(net, inputs, labels)
+        model.prepare(
+            optimizer,
+            paddle.nn.CrossEntropyLoss(),
+            paddle.metric.Accuracy(topk=(1, 5)))
+
+        model.fit(self.train_dataset, epochs=1, batch_size=128, verbose=1)
+        result = model.evaluate(self.val_dataset, batch_size=128, verbose=1)
+        pruner = None
+        if self._pruner == 'l1norm':
+            pruner = L1NormFilterPruner(net, [1, 1, 28, 28])
+        elif self._pruner == 'fpgm':
+            pruner = FPGMFilterPruner(net, [1, 1, 28, 28])
+
+        def eval_fn():
+            result = model.evaluate(self.val_dataset, batch_size=128)
+            return result['acc_top1']
+
+        sen = pruner.sensitive(
+            eval_func=eval_fn,
+            sen_file="_".join(["./dygraph_sen_", str(time.time())]),
+            #sen_file="sen.pickle",
+            target_vars=self._param_names)
+        params = {}
+        for param in net.parameters():
+            params[param.name] = np.array(param.value().get_tensor())
+        print(f'dygraph sen: {sen}')
+        return sen, params
+
+    def static_sen(self, params):
+        paddle.enable_static()
+        main_program = fluid.Program()
+        startup_program = fluid.Program()
+        with fluid.unique_name.guard():
+            with fluid.program_guard(main_program, startup_program):
+                input = fluid.data(name="image", shape=[None, 1, 28, 28])
+                label = fluid.data(name="label", shape=[None, 1], dtype="int64")
+                model = paddle.vision.models.LeNet()
+                out = model(input)
+                acc_top1 = fluid.layers.accuracy(input=out, label=label, k=1)
+        eval_program = main_program.clone(for_test=True)
+        place = fluid.CUDAPlace(0)
+        scope = fluid.global_scope()
+        exe = fluid.Executor(place)
+        exe.run(startup_program)
+
+        val_reader = paddle.fluid.io.batch(self.val_reader, batch_size=128)
+
+        def eval_func(program):
+            feeder = fluid.DataFeeder(
+                feed_list=['image', 'label'], place=place, program=program)
+            acc_set = []
+            for data in val_reader():
+                acc_np = exe.run(program=program,
+                                 feed=feeder.feed(data),
+                                 fetch_list=[acc_top1])
+                acc_set.append(float(acc_np[0]))
+            acc_val_mean = np.array(acc_set).mean()
+            return acc_val_mean
+
+        for _name, _value in params.items():
+            t = scope.find_var(_name).get_tensor()
+            t.set(_value, place)
+        print(f"static base: {eval_func(eval_program)}")
+        criterion = None
+        if self._pruner == 'l1norm':
+            criterion = 'l1_norm'
+        elif self._pruner == 'fpgm':
+            criterion = 'geometry_median'
+        sen = sensitivity(
+            eval_program,
+            place,
+            self._param_names,
+            eval_func,
+            sensitivities_file="_".join(
+                ["./sensitivities_file", str(time.time())]),
+            criterion=criterion)
+        return sen
+
+
+def add_cases(suite):
+    suite.addTest(
+        TestSensitivity(
+            pruner="l1norm", param_names=["conv2d_0.w_0"]))
+
+
+def load_tests(loader, standard_tests, pattern):
+    suite = unittest.TestSuite()
+    add_cases(suite)
+    return suite
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_group_param.py

@@ -45,8 +45,9 @@ class TestPrune(StaticCase):
             ["conv1_weights", "conv2_weights", "conv3_weights"], main_program)
         while [] in groups:
             groups.remove([])
+        print(groups)
         self.assertTrue(len(groups) == 2)
-        self.assertTrue(len(groups[0]) == 18)
+        self.assertTrue(len(groups[0]) == 20)
         self.assertTrue(len(groups[1]) == 6)