-
目录#
--
-
- 量化原理介绍 -
- 剪裁原理介绍 -
- 蒸馏原理介绍 -
- 轻量级模型结构搜索原理介绍 -
1. Quantization Aware Training量化介绍#
+1. Quantization Aware Training量化介绍#
1.1 背景#
近年来，定点量化使用更少的比特数（如8-bit、3-bit、2-bit等）表示神经网络的权重和激活已被验证是有效的。定点量化的优点包括低内存带宽、低功耗、低计算资源占用以及低模型存储需求等。
@@ -348,7 +338,7 @@ Y_{dq} = \frac{Y_q}{(n - 1) * (n - 1)} * X_m * W_m \ 在剪裁一个卷积核之前，按l1_norm对filter从高到低排序，越靠后的filter越不重要，优先剪掉靠后的filter.
2.3 基于敏感度剪裁卷积网络#
根据每个卷积层敏感度的不同，剪掉不同比例的卷积核。
-两个假设#
+两个假设#
- 在一个conv layer的parameter内部，按l1_norm对filter从高到低排序，越靠后的filter越不重要。
- 两个layer剪裁相同的比例的filters，我们称对模型精度影响更大的layer的敏感度相对高。 @@ -358,7 +348,7 @@ Y_{dq} = \frac{Y_q}{(n - 1) * (n - 1)} * X_m * W_m \
- layer的剪裁比例与其敏感度成反比
- 优先剪裁layer内l1_norm相对低的filter
敏感度的理解#
+敏感度的理解#
图7
@@ -366,7 +356,7 @@ Y_{dq} = \frac{Y_q}{(n - 1) * (n - 1)} * X_m * W_m \
如**图7**所示，横坐标是将filter剪裁掉的比例，竖坐标是精度的损失，每条彩色虚线表示的是网络中的一个卷积层。 以不同的剪裁比例**单独**剪裁一个卷积层，并观察其在验证数据集上的精度损失，并绘出**图7**中的虚线。虚线上升较慢的，对应的卷积层相对不敏感，我们优先剪不敏感的卷积层的filter.
-选择最优的剪裁率组合#
+选择最优的剪裁率组合#
我们将**图7**中的折线拟合为**图8**中的曲线，每在竖坐标轴上选取一个精度损失值，就在横坐标轴上对应着一组剪裁率，如**图8**中黑色实线所示。 用户给定一个模型整体的剪裁率，我们通过移动**图5**中的黑色实线来找到一组满足条件的且合法的剪裁率。
@@ -374,7 +364,7 @@ Y_{dq} = \frac{Y_q}{(n - 1) * (n - 1)} * X_m * W_m \ 图8
-迭代剪裁#
+迭代剪裁#
考虑到多个卷积层间的相关性，一个卷积层的修改可能会影响其它卷积层的敏感度，我们采取了多次剪裁的策略，步骤如下：
- step1: 统计各卷积层的敏感度信息 diff --git a/api/nas_api/index.html b/api/nas_api/index.html index bd9457a887497ef73a8b4bc13e12d6b257c84df6..e5aedde7b91317c89d7433c9edcee17857e6e88f 100644 --- a/api/nas_api/index.html +++ b/api/nas_api/index.html @@ -183,7 +183,7 @@
- block_mask(list|None)：-
block_mask是一组由0、1组成的列表，0表示当前block是normal block，1表示当前block是reduction block。如果设置了block_mask，则主要以block_mask为主要配置，input_size，output_size和block_num三种配置是无效的。
-
note
+Note
- reduction block表示经过这个block之后的feature map大小下降为之前的一半，normal block表示经过这个block之后feature map大小不变。
input_size和output_size用来计算整个模型结构中reduction block数量。
@@ -220,7 +220,7 @@
- 通过一组token得到实际的模型结构，一般用来把搜索到最优的token转换为模型结构用来做最后的训练。
-
note
+Note
tokens是一个列表，token映射到搜索空间转换成相应的网络结构，一组token对应唯一的一个网络结构。
参数：
diff --git a/index.html b/index.html index 4a539005bfc0eee506470144f11f313d620d0f90..3e12f8b918352b0e254aaae3a0ab985213a07ad6 100644 --- a/index.html +++ b/index.html @@ -293,5 +293,5 @@ diff --git a/search/search_index.json b/search/search_index.json index 95ff433eea420bc6fa32902124bdf566dc8a00c1..4b005e55a6ccac9675aa6a367f148c921ec8f594 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"PaddleSlim # PaddleSlim\u662fPaddlePaddle\u6846\u67b6\u7684\u4e00\u4e2a\u5b50\u6a21\u5757\uff0c\u4e3b\u8981\u7528\u4e8e\u538b\u7f29\u56fe\u50cf\u9886\u57df\u6a21\u578b\u3002\u5728PaddleSlim\u4e2d\uff0c\u4e0d\u4ec5\u5b9e\u73b0\u4e86\u76ee\u524d\u4e3b\u6d41\u7684\u7f51\u7edc\u526a\u679d\u3001\u91cf\u5316\u3001\u84b8\u998f\u4e09\u79cd\u538b\u7f29\u7b56\u7565\uff0c\u8fd8\u5b9e\u73b0\u4e86\u8d85\u53c2\u6570\u641c\u7d22\u548c\u5c0f\u6a21\u578b\u7f51\u7edc\u7ed3\u6784\u641c\u7d22\u529f\u80fd\u3002\u5728\u540e\u7eed\u7248\u672c\u4e2d\uff0c\u4f1a\u6dfb\u52a0\u66f4\u591a\u7684\u538b\u7f29\u7b56\u7565\uff0c\u4ee5\u53ca\u5b8c\u5584\u5bf9NLP\u9886\u57df\u6a21\u578b\u7684\u652f\u6301\u3002 \u529f\u80fd # \u6a21\u578b\u526a\u88c1 \u652f\u6301\u901a\u9053\u5747\u5300\u6a21\u578b\u526a\u88c1\uff08uniform pruning) \u57fa\u4e8e\u654f\u611f\u5ea6\u7684\u6a21\u578b\u526a\u88c1 \u57fa\u4e8e\u8fdb\u5316\u7b97\u6cd5\u7684\u81ea\u52a8\u6a21\u578b\u526a\u88c1\u4e09\u79cd\u65b9\u5f0f \u91cf\u5316\u8bad\u7ec3 \u5728\u7ebf\u91cf\u5316\u8bad\u7ec3\uff08training aware\uff09 \u79bb\u7ebf\u91cf\u5316\uff08post training\uff09 \u652f\u6301\u5bf9\u6743\u91cd\u5168\u5c40\u91cf\u5316\u548cChannel-Wise\u91cf\u5316 \u84b8\u998f \u8f7b\u91cf\u795e\u7ecf\u7f51\u7edc\u7ed3\u6784\u81ea\u52a8\u641c\u7d22\uff08Light-NAS\uff09 \u652f\u6301\u57fa\u4e8e\u8fdb\u5316\u7b97\u6cd5\u7684\u8f7b\u91cf\u795e\u7ecf\u7f51\u7edc\u7ed3\u6784\u81ea\u52a8\u641c\u7d22\uff08Light-NAS\uff09 \u652f\u6301 FLOPS / \u786c\u4ef6\u5ef6\u65f6\u7ea6\u675f \u652f\u6301\u591a\u5e73\u53f0\u6a21\u578b\u5ef6\u65f6\u8bc4\u4f30 \u5b89\u88c5 # \u5b89\u88c5PaddleSlim\u524d\uff0c\u8bf7\u786e\u8ba4\u5df2\u6b63\u786e\u5b89\u88c5Paddle1.6\u7248\u672c\u6216\u66f4\u65b0\u7248\u672c\u3002Paddle\u5b89\u88c5\u8bf7\u53c2\u8003\uff1a Paddle\u5b89\u88c5\u6559\u7a0b \u3002 \u5b89\u88c5develop\u7248\u672c 1 2 3 git clone https : // github . com / PaddlePaddle / PaddleSlim . git cd PaddleSlim python setup . py install \u5b89\u88c5\u5b98\u65b9\u53d1\u5e03\u7684\u6700\u65b0\u7248\u672c 1 pip install paddleslim - i https : // pypi . org / simple \u5b89\u88c5\u5386\u53f2\u7248\u672c \u8bf7\u70b9\u51fb pypi.org \u67e5\u770b\u53ef\u5b89\u88c5\u5386\u53f2\u7248\u672c\u3002 \u4f7f\u7528 # API\u6587\u6863 \uff1aAPI\u4f7f\u7528\u4ecb\u7ecd\uff0c\u5305\u62ec \u84b8\u998f \u3001 \u526a\u88c1 \u3001 \u91cf\u5316 \u548c \u6a21\u578b\u7ed3\u6784\u641c\u7d22 \u3002 \u793a\u4f8b \uff1a\u57fa\u4e8emnist\u548ccifar10\u7b49\u7b80\u5355\u5206\u7c7b\u4efb\u52a1\u7684\u6a21\u578b\u538b\u7f29\u793a\u4f8b\uff0c\u60a8\u53ef\u4ee5\u901a\u8fc7\u8be5\u90e8\u5206\u5feb\u901f\u4f53\u9a8c\u548c\u4e86\u89e3PaddleSlim\u7684\u529f\u80fd\u3002 \u5b9e\u8df5\u6559\u7a0b \uff1a\u7ecf\u5178\u6a21\u578b\u7684\u5206\u6790\u548c\u538b\u7f29\u5b9e\u9a8c\u6559\u7a0b\u3002 \u6a21\u578b\u5e93 \uff1a\u7ecf\u8fc7\u538b\u7f29\u7684\u5206\u7c7b\u3001\u68c0\u6d4b\u3001\u8bed\u4e49\u5206\u5272\u6a21\u578b\uff0c\u5305\u62ec\u6743\u91cd\u6587\u4ef6\u3001\u7f51\u7edc\u7ed3\u6784\u6587\u4ef6\u548c\u6027\u80fd\u6570\u636e\u3002 Paddle\u68c0\u6d4b\u5e93 \uff1a\u4ecb\u7ecd\u5982\u4f55\u5728\u68c0\u6d4b\u5e93\u4e2d\u4f7f\u7528PaddleSlim\u3002 Paddle\u5206\u5272\u5e93 \uff1a\u4ecb\u7ecd\u5982\u4f55\u5728\u5206\u5272\u5e93\u4e2d\u4f7f\u7528PaddleSlim\u3002 PaddleLite \uff1a\u4ecb\u7ecd\u5982\u4f55\u4f7f\u7528\u9884\u6d4b\u5e93PaddleLite\u90e8\u7f72PaddleSlim\u4ea7\u51fa\u7684\u6a21\u578b\u3002 \u8d21\u732e\u4e0e\u53cd\u9988 #","title":"Home"},{"location":"#paddleslim","text":"PaddleSlim\u662fPaddlePaddle\u6846\u67b6\u7684\u4e00\u4e2a\u5b50\u6a21\u5757\uff0c\u4e3b\u8981\u7528\u4e8e\u538b\u7f29\u56fe\u50cf\u9886\u57df\u6a21\u578b\u3002\u5728PaddleSlim\u4e2d\uff0c\u4e0d\u4ec5\u5b9e\u73b0\u4e86\u76ee\u524d\u4e3b\u6d41\u7684\u7f51\u7edc\u526a\u679d\u3001\u91cf\u5316\u3001\u84b8\u998f\u4e09\u79cd\u538b\u7f29\u7b56\u7565\uff0c\u8fd8\u5b9e\u73b0\u4e86\u8d85\u53c2\u6570\u641c\u7d22\u548c\u5c0f\u6a21\u578b\u7f51\u7edc\u7ed3\u6784\u641c\u7d22\u529f\u80fd\u3002\u5728\u540e\u7eed\u7248\u672c\u4e2d\uff0c\u4f1a\u6dfb\u52a0\u66f4\u591a\u7684\u538b\u7f29\u7b56\u7565\uff0c\u4ee5\u53ca\u5b8c\u5584\u5bf9NLP\u9886\u57df\u6a21\u578b\u7684\u652f\u6301\u3002","title":"PaddleSlim"},{"location":"#_1","text":"\u6a21\u578b\u526a\u88c1 \u652f\u6301\u901a\u9053\u5747\u5300\u6a21\u578b\u526a\u88c1\uff08uniform pruning) \u57fa\u4e8e\u654f\u611f\u5ea6\u7684\u6a21\u578b\u526a\u88c1 \u57fa\u4e8e\u8fdb\u5316\u7b97\u6cd5\u7684\u81ea\u52a8\u6a21\u578b\u526a\u88c1\u4e09\u79cd\u65b9\u5f0f \u91cf\u5316\u8bad\u7ec3 \u5728\u7ebf\u91cf\u5316\u8bad\u7ec3\uff08training aware\uff09 \u79bb\u7ebf\u91cf\u5316\uff08post training\uff09 \u652f\u6301\u5bf9\u6743\u91cd\u5168\u5c40\u91cf\u5316\u548cChannel-Wise\u91cf\u5316 \u84b8\u998f \u8f7b\u91cf\u795e\u7ecf\u7f51\u7edc\u7ed3\u6784\u81ea\u52a8\u641c\u7d22\uff08Light-NAS\uff09 \u652f\u6301\u57fa\u4e8e\u8fdb\u5316\u7b97\u6cd5\u7684\u8f7b\u91cf\u795e\u7ecf\u7f51\u7edc\u7ed3\u6784\u81ea\u52a8\u641c\u7d22\uff08Light-NAS\uff09 \u652f\u6301 FLOPS / \u786c\u4ef6\u5ef6\u65f6\u7ea6\u675f \u652f\u6301\u591a\u5e73\u53f0\u6a21\u578b\u5ef6\u65f6\u8bc4\u4f30","title":"\u529f\u80fd"},{"location":"#_2","text":"\u5b89\u88c5PaddleSlim\u524d\uff0c\u8bf7\u786e\u8ba4\u5df2\u6b63\u786e\u5b89\u88c5Paddle1.6\u7248\u672c\u6216\u66f4\u65b0\u7248\u672c\u3002Paddle\u5b89\u88c5\u8bf7\u53c2\u8003\uff1a Paddle\u5b89\u88c5\u6559\u7a0b \u3002 \u5b89\u88c5develop\u7248\u672c 1 2 3 git clone https : // github . com / PaddlePaddle / PaddleSlim . git cd PaddleSlim python setup . py install \u5b89\u88c5\u5b98\u65b9\u53d1\u5e03\u7684\u6700\u65b0\u7248\u672c 1 pip install paddleslim - i https : // pypi . org / simple \u5b89\u88c5\u5386\u53f2\u7248\u672c \u8bf7\u70b9\u51fb pypi.org \u67e5\u770b\u53ef\u5b89\u88c5\u5386\u53f2\u7248\u672c\u3002","title":"\u5b89\u88c5"},{"location":"#_3","text":"API\u6587\u6863 \uff1aAPI\u4f7f\u7528\u4ecb\u7ecd\uff0c\u5305\u62ec \u84b8\u998f \u3001 \u526a\u88c1 \u3001 \u91cf\u5316 \u548c \u6a21\u578b\u7ed3\u6784\u641c\u7d22 \u3002 \u793a\u4f8b \uff1a\u57fa\u4e8emnist\u548ccifar10\u7b49\u7b80\u5355\u5206\u7c7b\u4efb\u52a1\u7684\u6a21\u578b\u538b\u7f29\u793a\u4f8b\uff0c\u60a8\u53ef\u4ee5\u901a\u8fc7\u8be5\u90e8\u5206\u5feb\u901f\u4f53\u9a8c\u548c\u4e86\u89e3PaddleSlim\u7684\u529f\u80fd\u3002 \u5b9e\u8df5\u6559\u7a0b \uff1a\u7ecf\u5178\u6a21\u578b\u7684\u5206\u6790\u548c\u538b\u7f29\u5b9e\u9a8c\u6559\u7a0b\u3002 \u6a21\u578b\u5e93 \uff1a\u7ecf\u8fc7\u538b\u7f29\u7684\u5206\u7c7b\u3001\u68c0\u6d4b\u3001\u8bed\u4e49\u5206\u5272\u6a21\u578b\uff0c\u5305\u62ec\u6743\u91cd\u6587\u4ef6\u3001\u7f51\u7edc\u7ed3\u6784\u6587\u4ef6\u548c\u6027\u80fd\u6570\u636e\u3002 Paddle\u68c0\u6d4b\u5e93 \uff1a\u4ecb\u7ecd\u5982\u4f55\u5728\u68c0\u6d4b\u5e93\u4e2d\u4f7f\u7528PaddleSlim\u3002 Paddle\u5206\u5272\u5e93 \uff1a\u4ecb\u7ecd\u5982\u4f55\u5728\u5206\u5272\u5e93\u4e2d\u4f7f\u7528PaddleSlim\u3002 PaddleLite \uff1a\u4ecb\u7ecd\u5982\u4f55\u4f7f\u7528\u9884\u6d4b\u5e93PaddleLite\u90e8\u7f72PaddleSlim\u4ea7\u51fa\u7684\u6a21\u578b\u3002","title":"\u4f7f\u7528"},{"location":"#_4","text":"","title":"\u8d21\u732e\u4e0e\u53cd\u9988"},{"location":"model_zoo/","text":"1. \u56fe\u50cf\u5206\u7c7b # \u6570\u636e\u96c6\uff1aImageNet1000\u7c7b 1.1 \u91cf\u5316 # \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Top-1/Top-5 Acc \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNetV1 FP32 baseline 70.99%/89.68% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 quant_post xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 quant_aware xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 FP32 baseline 72.15%/90.65% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 quant_post xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 quant_aware xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 ResNet50 FP32 baseline 76.50%/93.00% xx \u4e0b\u8f7d\u94fe\u63a5 ResNet50 quant_post xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 ResNet50 quant_aware xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 1.2 \u526a\u88c1 # \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Top-1/Top-5 Acc \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 GFLOPs \u4e0b\u8f7d MobileNetV1 baseline 70.99%/89.68% 17 1.11 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 uniform -50% 69.4%/88.66% (-1.59%/-1.02%) 9 0.56 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 sensitive -30% 70.4%/89.3% (-0.59%/-0.38%) 12 0.74 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 sensitive -50% 69.8% / 88.9% (-1.19%/-0.78%) 9 0.56 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 baseline 72.15%/90.65% 15 0.59 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 uniform -50% 65.79%/86.11% (-6.35%/-4.47%) 11 0.296 \u4e0b\u8f7d\u94fe\u63a5 ResNet34 baseline 72.15%/90.65% 84 7.36 \u4e0b\u8f7d\u94fe\u63a5 ResNet34 uniform -50% 70.99%/89.95% (-1.36%/-0.87%) 41 3.67 \u4e0b\u8f7d\u94fe\u63a5 ResNet34 auto -55.05% 70.24%/89.63% (-2.04%/-1.06%) 33 3.31 \u4e0b\u8f7d\u94fe\u63a5 1.3 \u84b8\u998f # \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Top-1/Top-5 Acc \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNetV1 student 70.99%/89.68% 17 \u4e0b\u8f7d\u94fe\u63a5 ResNet50_vd teacher 79.12%/94.44% 99 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 ResNet50_vd 1 distill 72.77%/90.68% (+1.78%/+1.00%) 17 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 student 72.15%/90.65% 15 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 ResNet50_vd distill 74.28%/91.53% (+2.13%/+0.88%) 15 \u4e0b\u8f7d\u94fe\u63a5 ResNet50 student 76.50%/93.00% 99 \u4e0b\u8f7d\u94fe\u63a5 ResNet101 teacher 77.56%/93.64% 173 \u4e0b\u8f7d\u94fe\u63a5 ResNet50 ResNet101 distill 77.29%/93.65% (+0.79%/+0.65%) 99 \u4e0b\u8f7d\u94fe\u63a5 Note [1] \uff1a\u5e26_vd\u540e\u7f00\u4ee3\u8868\u8be5\u9884\u8bad\u7ec3\u6a21\u578b\u4f7f\u7528\u4e86Mixup\uff0cMixup\u76f8\u5173\u4ecb\u7ecd\u53c2\u8003 mixup: Beyond Empirical Risk Minimization 2. \u76ee\u6807\u68c0\u6d4b # 2.1 \u91cf\u5316 # \u6570\u636e\u96c6\uff1a COCO 2017 \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 \u6570\u636e\u96c6 Image/GPU \u8f93\u5165608 Box AP \u8f93\u5165416 Box AP \u8f93\u5165320 Box AP \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNet-V1-YOLOv3 FP32 baseline COCO 8 29.3 29.3 27.1 xx \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 quant_post COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 quant_aware COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain FP32 baseline COCO 8 41.4 xx xx xx \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain quant_post COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain quant_aware COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 \u6570\u636e\u96c6\uff1aWIDER-FACE \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Image/GPU \u8f93\u5165\u5c3a\u5bf8 Easy/Medium/Hard \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d BlazeFace FP32 baseline 8 640 0.915/0.892/0.797 xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace quant_post 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace quant_aware 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-Lite FP32 baseline 8 640 0.909/0.885/0.781 xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-Lite quant_post 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-Lite quant_aware 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-NAS FP32 baseline 8 640 0.837/0.807/0.658 xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-NAS quant_post 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-NAS quant_aware 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 2.2 \u526a\u88c1 # \u6570\u636e\u96c6\uff1aPasacl VOC & COCO 2017 \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 \u6570\u636e\u96c6 Image/GPU \u8f93\u5165608 Box AP \u8f93\u5165416 Box AP \u8f93\u5165320 Box AP \u6a21\u578b\u4f53\u79ef(MB) GFLOPs (608*608) \u4e0b\u8f7d MobileNet-V1-YOLOv3 baseline Pascal VOC 8 76.2 76.7 75.3 94 40.49 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 sensitive -52.88% Pascal VOC 8 77.6 (+1.4) 77.7 (1.0) 75.5 (+0.2) 31 19.08 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 baseline COCO 8 29.3 29.3 27.0 95 41.35 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 sensitive -51.77% COCO 8 26.0 (-3.3) 25.1 (-4.2) 22.6 (-4.4) 32 19.94 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 baseline COCO 8 39.1 - - 177 89.60 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 sensitive -9.37% COCO 8 39.3 (+0.2) - - 150 81.20 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 sensitive -24.68% COCO 8 37.3 (-1.8) - - 113 67.48 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain baseline COCO 8 41.4 - - 177 89.60 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain sensitive -9.37% COCO 8 40.5 (-0.9) - - 150 81.20 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain sensitive -24.68% COCO 8 37.8 (-3.3) - - 113 67.48 \u4e0b\u8f7d\u94fe\u63a5 2.3 \u84b8\u998f # \u6570\u636e\u96c6\uff1aPasacl VOC & COCO 2017 \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 \u6570\u636e\u96c6 Image/GPU \u8f93\u5165608 Box AP \u8f93\u5165416 Box AP \u8f93\u5165320 Box AP \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNet-V1-YOLOv3 student Pascal VOC 8 76.2 76.7 75.3 94 \u4e0b\u8f7d\u94fe\u63a5 ResNet34-YOLOv3 teacher Pascal VOC 8 82.6 81.9 80.1 162 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 ResNet34-YOLOv3 distill Pascal VOC 8 79.0 (+2.8) 78.2 (+1.5) 75.5 (+0.2) 94 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 student COCO 8 29.3 29.3 27.0 95 \u4e0b\u8f7d\u94fe\u63a5 ResNet34-YOLOv3 teacher COCO 8 36.2 34.3 31.4 163 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 ResNet34-YOLOv3 distill COCO 8 31.4 (+2.1) 30.0 (+0.7) 27.1 (+0.1) 95 \u4e0b\u8f7d\u94fe\u63a5 3. \u56fe\u50cf\u5206\u5272 # \u6570\u636e\u96c6\uff1aCityscapes 3.1 \u91cf\u5316 # \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 mIoU \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d DeepLabv3+/MobileNetv1 FP32 baseline 63.26 xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv1 quant_post xx xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv1 quant_aware xx xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv2 FP32 baseline 69.81 xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv2 quant_post xx xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv2 quant_aware xx xx \u4e0b\u8f7d\u94fe\u63a5 3.2 \u526a\u88c1 # \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 mIoU \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 GFLOPs \u4e0b\u8f7d fast-scnn baseline 69.64 11 14.41 \u4e0b\u8f7d\u94fe\u63a5 fast-scnn uniform -17.07% 69.58 (-0.06) 8.5 11.95 \u4e0b\u8f7d\u94fe\u63a5 fast-scnn sensitive -47.60% 66.68 (-2.96) 5.7 7.55 \u4e0b\u8f7d\u94fe\u63a5","title":"\u6a21\u578b\u5e93"},{"location":"model_zoo/#1","text":"\u6570\u636e\u96c6\uff1aImageNet1000\u7c7b","title":"1. \u56fe\u50cf\u5206\u7c7b"},{"location":"model_zoo/#11","text":"\u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Top-1/Top-5 Acc \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNetV1 FP32 baseline 70.99%/89.68% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 quant_post xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 quant_aware xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 FP32 baseline 72.15%/90.65% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 quant_post xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 quant_aware xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 ResNet50 FP32 baseline 76.50%/93.00% xx \u4e0b\u8f7d\u94fe\u63a5 ResNet50 quant_post xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5 ResNet50 quant_aware xx%/xx% xx \u4e0b\u8f7d\u94fe\u63a5","title":"1.1 \u91cf\u5316"},{"location":"model_zoo/#12","text":"\u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Top-1/Top-5 Acc \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 GFLOPs \u4e0b\u8f7d MobileNetV1 baseline 70.99%/89.68% 17 1.11 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 uniform -50% 69.4%/88.66% (-1.59%/-1.02%) 9 0.56 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 sensitive -30% 70.4%/89.3% (-0.59%/-0.38%) 12 0.74 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 sensitive -50% 69.8% / 88.9% (-1.19%/-0.78%) 9 0.56 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 baseline 72.15%/90.65% 15 0.59 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 uniform -50% 65.79%/86.11% (-6.35%/-4.47%) 11 0.296 \u4e0b\u8f7d\u94fe\u63a5 ResNet34 baseline 72.15%/90.65% 84 7.36 \u4e0b\u8f7d\u94fe\u63a5 ResNet34 uniform -50% 70.99%/89.95% (-1.36%/-0.87%) 41 3.67 \u4e0b\u8f7d\u94fe\u63a5 ResNet34 auto -55.05% 70.24%/89.63% (-2.04%/-1.06%) 33 3.31 \u4e0b\u8f7d\u94fe\u63a5","title":"1.2 \u526a\u88c1"},{"location":"model_zoo/#13","text":"\u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Top-1/Top-5 Acc \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNetV1 student 70.99%/89.68% 17 \u4e0b\u8f7d\u94fe\u63a5 ResNet50_vd teacher 79.12%/94.44% 99 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV1 ResNet50_vd 1 distill 72.77%/90.68% (+1.78%/+1.00%) 17 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 student 72.15%/90.65% 15 \u4e0b\u8f7d\u94fe\u63a5 MobileNetV2 ResNet50_vd distill 74.28%/91.53% (+2.13%/+0.88%) 15 \u4e0b\u8f7d\u94fe\u63a5 ResNet50 student 76.50%/93.00% 99 \u4e0b\u8f7d\u94fe\u63a5 ResNet101 teacher 77.56%/93.64% 173 \u4e0b\u8f7d\u94fe\u63a5 ResNet50 ResNet101 distill 77.29%/93.65% (+0.79%/+0.65%) 99 \u4e0b\u8f7d\u94fe\u63a5 Note [1] \uff1a\u5e26_vd\u540e\u7f00\u4ee3\u8868\u8be5\u9884\u8bad\u7ec3\u6a21\u578b\u4f7f\u7528\u4e86Mixup\uff0cMixup\u76f8\u5173\u4ecb\u7ecd\u53c2\u8003 mixup: Beyond Empirical Risk Minimization","title":"1.3 \u84b8\u998f"},{"location":"model_zoo/#2","text":"","title":"2. \u76ee\u6807\u68c0\u6d4b"},{"location":"model_zoo/#21","text":"\u6570\u636e\u96c6\uff1a COCO 2017 \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 \u6570\u636e\u96c6 Image/GPU \u8f93\u5165608 Box AP \u8f93\u5165416 Box AP \u8f93\u5165320 Box AP \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNet-V1-YOLOv3 FP32 baseline COCO 8 29.3 29.3 27.1 xx \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 quant_post COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 quant_aware COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain FP32 baseline COCO 8 41.4 xx xx xx \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain quant_post COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain quant_aware COCO 8 xx xx xx xx \u4e0b\u8f7d\u94fe\u63a5 \u6570\u636e\u96c6\uff1aWIDER-FACE \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 Image/GPU \u8f93\u5165\u5c3a\u5bf8 Easy/Medium/Hard \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d BlazeFace FP32 baseline 8 640 0.915/0.892/0.797 xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace quant_post 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace quant_aware 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-Lite FP32 baseline 8 640 0.909/0.885/0.781 xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-Lite quant_post 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-Lite quant_aware 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-NAS FP32 baseline 8 640 0.837/0.807/0.658 xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-NAS quant_post 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5 BlazeFace-NAS quant_aware 8 640 xx/xx/xx xx \u4e0b\u8f7d\u94fe\u63a5","title":"2.1 \u91cf\u5316"},{"location":"model_zoo/#22","text":"\u6570\u636e\u96c6\uff1aPasacl VOC & COCO 2017 \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 \u6570\u636e\u96c6 Image/GPU \u8f93\u5165608 Box AP \u8f93\u5165416 Box AP \u8f93\u5165320 Box AP \u6a21\u578b\u4f53\u79ef(MB) GFLOPs (608*608) \u4e0b\u8f7d MobileNet-V1-YOLOv3 baseline Pascal VOC 8 76.2 76.7 75.3 94 40.49 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 sensitive -52.88% Pascal VOC 8 77.6 (+1.4) 77.7 (1.0) 75.5 (+0.2) 31 19.08 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 baseline COCO 8 29.3 29.3 27.0 95 41.35 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 sensitive -51.77% COCO 8 26.0 (-3.3) 25.1 (-4.2) 22.6 (-4.4) 32 19.94 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 baseline COCO 8 39.1 - - 177 89.60 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 sensitive -9.37% COCO 8 39.3 (+0.2) - - 150 81.20 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 sensitive -24.68% COCO 8 37.3 (-1.8) - - 113 67.48 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain baseline COCO 8 41.4 - - 177 89.60 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain sensitive -9.37% COCO 8 40.5 (-0.9) - - 150 81.20 \u4e0b\u8f7d\u94fe\u63a5 R50-dcn-YOLOv3 obj365_pretrain sensitive -24.68% COCO 8 37.8 (-3.3) - - 113 67.48 \u4e0b\u8f7d\u94fe\u63a5","title":"2.2 \u526a\u88c1"},{"location":"model_zoo/#23","text":"\u6570\u636e\u96c6\uff1aPasacl VOC & COCO 2017 \u6a21\u578b \u538b\u7f29\u65b9\u6cd5 \u6570\u636e\u96c6 Image/GPU \u8f93\u5165608 Box AP \u8f93\u5165416 Box AP \u8f93\u5165320 Box AP \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d MobileNet-V1-YOLOv3 student Pascal VOC 8 76.2 76.7 75.3 94 \u4e0b\u8f7d\u94fe\u63a5 ResNet34-YOLOv3 teacher Pascal VOC 8 82.6 81.9 80.1 162 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 ResNet34-YOLOv3 distill Pascal VOC 8 79.0 (+2.8) 78.2 (+1.5) 75.5 (+0.2) 94 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 student COCO 8 29.3 29.3 27.0 95 \u4e0b\u8f7d\u94fe\u63a5 ResNet34-YOLOv3 teacher COCO 8 36.2 34.3 31.4 163 \u4e0b\u8f7d\u94fe\u63a5 MobileNet-V1-YOLOv3 ResNet34-YOLOv3 distill COCO 8 31.4 (+2.1) 30.0 (+0.7) 27.1 (+0.1) 95 \u4e0b\u8f7d\u94fe\u63a5","title":"2.3 \u84b8\u998f"},{"location":"model_zoo/#3","text":"\u6570\u636e\u96c6\uff1aCityscapes","title":"3. \u56fe\u50cf\u5206\u5272"},{"location":"model_zoo/#31","text":"\u6a21\u578b \u538b\u7f29\u65b9\u6cd5 mIoU \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 \u4e0b\u8f7d DeepLabv3+/MobileNetv1 FP32 baseline 63.26 xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv1 quant_post xx xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv1 quant_aware xx xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv2 FP32 baseline 69.81 xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv2 quant_post xx xx \u4e0b\u8f7d\u94fe\u63a5 DeepLabv3+/MobileNetv2 quant_aware xx xx \u4e0b\u8f7d\u94fe\u63a5","title":"3.1 \u91cf\u5316"},{"location":"model_zoo/#32","text":"\u6a21\u578b \u538b\u7f29\u65b9\u6cd5 mIoU \u6a21\u578b\u4f53\u79ef\uff08MB\uff09 GFLOPs \u4e0b\u8f7d fast-scnn baseline 69.64 11 14.41 \u4e0b\u8f7d\u94fe\u63a5 fast-scnn uniform -17.07% 69.58 (-0.06) 8.5 11.95 \u4e0b\u8f7d\u94fe\u63a5 fast-scnn sensitive -47.60% 66.68 (-2.96) 5.7 7.55 \u4e0b\u8f7d\u94fe\u63a5","title":"3.2 \u526a\u88c1"},{"location":"search_space/","text":"\u641c\u7d22\u7a7a\u95f4\u7b80\u4ecb # \u641c\u7d22\u7a7a\u95f4\u662f\u795e\u7ecf\u7f51\u7edc\u641c\u7d22\u4e2d\u7684\u4e00\u4e2a\u6982\u5ff5\u3002\u641c\u7d22\u7a7a\u95f4\u662f\u4e00\u7cfb\u5217\u6a21\u578b\u7ed3\u6784\u7684\u6c47\u96c6, SANAS\u4e3b\u8981\u662f\u5229\u7528\u6a21\u62df\u9000\u706b\u7684\u601d\u60f3\u5728\u641c\u7d22\u7a7a\u95f4\u4e2d\u641c\u7d22\u5230\u4e00\u4e2a\u6bd4\u8f83\u5c0f\u7684\u6a21\u578b\u7ed3\u6784\u6216\u8005\u4e00\u4e2a\u7cbe\u5ea6\u6bd4\u8f83\u9ad8\u7684\u6a21\u578b\u7ed3\u6784\u3002 paddleslim.nas \u63d0\u4f9b\u7684\u641c\u7d22\u7a7a\u95f4 # \u6839\u636e\u521d\u59cb\u6a21\u578b\u7ed3\u6784\u6784\u9020\u641c\u7d22\u7a7a\u95f4 # MobileNetV2Space \u2003 MobileNetV2\u7684\u7f51\u7edc\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \uff0c \u8bba\u6587 MobileNetV1Space \u2003 MobilNetV1\u7684\u7f51\u7edc\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \uff0c \u8bba\u6587 ResNetSpace \u2003 ResNetSpace\u7684\u7f51\u7edc\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \uff0c \u8bba\u6587 \u6839\u636e\u76f8\u5e94\u6a21\u578b\u7684block\u6784\u9020\u641c\u7d22\u7a7a\u95f4 # MobileNetV1BlockSpace \u2003 MobileNetV1Block\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 MobileNetV2BlockSpace \u2003 MobileNetV2Block\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 ResNetBlockSpace \u2003 ResNetBlock\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 InceptionABlockSpace \u2003 InceptionABlock\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 InceptionCBlockSpace \u2003 InceptionCBlock\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \u641c\u7d22\u7a7a\u95f4\u793a\u4f8b # \u4f7f\u7528paddleslim\u4e2d\u63d0\u4f9b\u7528\u521d\u59cb\u7684\u6a21\u578b\u7ed3\u6784\u6765\u6784\u9020\u641c\u7d22\u7a7a\u95f4\u7684\u8bdd\uff0c\u4ec5\u9700\u8981\u6307\u5b9a\u641c\u7d22\u7a7a\u95f4\u540d\u5b57\u5373\u53ef\u3002\u4f8b\u5982\uff1a\u5982\u679c\u4f7f\u7528\u539f\u672c\u7684MobileNetV2\u7684\u641c\u7d22\u7a7a\u95f4\u8fdb\u884c\u641c\u7d22\u7684\u8bdd\uff0c\u4f20\u5165SANAS\u4e2d\u7684config\u76f4\u63a5\u6307\u5b9a\u4e3a[('MobileNetV2Space')]\u3002 \u4f7f\u7528paddleslim\u4e2d\u63d0\u4f9b\u7684block\u641c\u7d22\u7a7a\u95f4\u6784\u9020\u641c\u7d22\u7a7a\u95f4\uff1a 2.1 \u4f7f\u7528 input_size , output_size \u548c block_num \u6765\u6784\u9020\u641c\u7d22\u7a7a\u95f4\u3002\u4f8b\u5982\uff1a\u4f20\u5165SANAS\u7684config\u53ef\u4ee5\u6307\u5b9a\u4e3a[('MobileNetV2BlockSpace', {'input_size': 224, 'output_size': 32, 'block_num': 10})]\u3002 2.2 \u4f7f\u7528 block_mask \u6784\u9020\u641c\u7d22\u7a7a\u95f4\u3002\u4f8b\u5982\uff1a\u4f20\u5165SANAS\u7684config\u53ef\u4ee5\u6307\u5b9a\u4e3a[('MobileNetV2BlockSpace', {'block_mask': [0, 1, 1, 1, 1, 0, 1, 0]})]\u3002 \u81ea\u5b9a\u4e49\u641c\u7d22\u7a7a\u95f4(search space) # \u81ea\u5b9a\u4e49\u641c\u7d22\u7a7a\u95f4\u7c7b\u9700\u8981\u7ee7\u627f\u641c\u7d22\u7a7a\u95f4\u57fa\u7c7b\u5e76\u91cd\u5199\u4ee5\u4e0b\u51e0\u90e8\u5206\uff1a \u2003 1. \u521d\u59cb\u5316\u7684tokens( init_tokens \u51fd\u6570)\uff0c\u53ef\u4ee5\u8bbe\u7f6e\u4e3a\u81ea\u5df1\u60f3\u8981\u7684tokens\u5217\u8868, tokens\u5217\u8868\u4e2d\u7684\u6bcf\u4e2a\u6570\u5b57\u6307\u7684\u662f\u5f53\u524d\u6570\u5b57\u5728\u76f8\u5e94\u7684\u641c\u7d22\u5217\u8868\u4e2d\u7684\u7d22\u5f15\u3002\u4f8b\u5982\u672c\u793a\u4f8b\u4e2d\u82e5tokens=[0, 3, 5]\uff0c\u5219\u4ee3\u8868\u5f53\u524d\u6a21\u578b\u7ed3\u6784\u641c\u7d22\u5230\u7684\u901a\u9053\u6570\u4e3a[8, 40, 128]\u3002 \u2003 2. token\u4e2d\u6bcf\u4e2a\u6570\u5b57\u7684\u641c\u7d22\u5217\u8868\u957f\u5ea6( range_table \u51fd\u6570)\uff0ctokens\u4e2d\u6bcf\u4e2atoken\u7684\u7d22\u5f15\u8303\u56f4\u3002 \u2003 3. \u6839\u636etoken\u4ea7\u751f\u6a21\u578b\u7ed3\u6784( token2arch \u51fd\u6570)\uff0c\u6839\u636e\u641c\u7d22\u5230\u7684tokens\u5217\u8868\u4ea7\u751f\u6a21\u578b\u7ed3\u6784\u3002 \u4ee5\u65b0\u589ereset block\u4e3a\u4f8b\u8bf4\u660e\u5982\u4f55\u6784\u9020\u81ea\u5df1\u7684search space\u3002\u81ea\u5b9a\u4e49\u7684search space\u4e0d\u80fd\u548c\u5df2\u6709\u7684search space\u540c\u540d\u3002 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 ### \u5f15\u5165\u641c\u7d22\u7a7a\u95f4\u57fa\u7c7b\u51fd\u6570\u548csearch space\u7684\u6ce8\u518c\u7c7b\u51fd\u6570 from .search_space_base import SearchSpaceBase from .search_space_registry import SEARCHSPACE import numpy as np ### \u9700\u8981\u8c03\u7528\u6ce8\u518c\u51fd\u6570\u628a\u81ea\u5b9a\u4e49\u641c\u7d22\u7a7a\u95f4\u6ce8\u518c\u5230space space\u4e2d @SEARCHSPACE.register ### \u5b9a\u4e49\u4e00\u4e2a\u7ee7\u627fSearchSpaceBase\u57fa\u7c7b\u7684\u641c\u7d22\u7a7a\u95f4\u7684\u7c7b\u51fd\u6570 class ResNetBlockSpace2 ( SearchSpaceBase ): def __init__ ( self , input_size , output_size , block_num , block_mask ): ### \u5b9a\u4e49\u4e00\u4e9b\u5b9e\u9645\u60f3\u8981\u641c\u7d22\u7684\u5185\u5bb9\uff0c\u4f8b\u5982\uff1a\u901a\u9053\u6570\u3001\u6bcf\u4e2a\u5377\u79ef\u7684\u91cd\u590d\u6b21\u6570\u3001\u5377\u79ef\u6838\u5927\u5c0f\u7b49\u7b49 ### self.filter_num \u4ee3\u8868\u901a\u9053\u6570\u7684\u641c\u7d22\u5217\u8868 self . filter_num = np . array ([ 8 , 16 , 32 , 40 , 64 , 128 , 256 , 512 ]) ### \u5b9a\u4e49\u521d\u59cb\u5316token\uff0c\u521d\u59cb\u5316token\u7684\u957f\u5ea6\u6839\u636e\u4f20\u5165\u7684block_num\u6216\u8005block_mask\u7684\u957f\u5ea6\u6765\u5f97\u5230\u7684 def init_tokens ( self ): return [ 0 ] * 3 * len ( self . block_mask ) ### \u5b9a\u4e49token\u7684index\u7684\u53d6\u503c\u8303\u56f4 def range_table ( self ): return [ len ( self . filter_num )] * 3 * len ( self . block_mask ) ### \u628atoken\u8f6c\u6362\u6210\u6a21\u578b\u7ed3\u6784 def token2arch ( self , tokens = None ): if tokens == None : tokens = self . init_tokens () self . bottleneck_params_list = [] for i in range ( len ( self . block_mask )): self . bottleneck_params_list . append ( self . filter_num [ tokens [ i * 3 + 0 ]], self . filter_num [ tokens [ i * 3 + 1 ]], self . filter_num [ tokens [ i * 3 + 2 ]], 2 if self . block_mask [ i ] == 1 else 1 ) def net_arch ( input ): for i , layer_setting in enumerate ( self . bottleneck_params_list ): channel_num , stride = layer_setting [: - 1 ], layer_setting [ - 1 ] input = self . _resnet_block ( input , channel_num , stride , name = 'resnet_layer{}' . format ( i + 1 )) return input return net_arch ### \u6784\u9020\u5177\u4f53block\u7684\u64cd\u4f5c def _resnet_block ( self , input , channel_num , stride , name = None ): shortcut_conv = self . _shortcut ( input , channel_num [ 2 ], stride , name = name ) input = self . _conv_bn_layer ( input = input , num_filters = channel_num [ 0 ], filter_size = 1 , act = 'relu' , name = name + '_conv0' ) input = self . _conv_bn_layer ( input = input , num_filters = channel_num [ 1 ], filter_size = 3 , stride = stride , act = 'relu' , name = name + '_conv1' ) input = self . _conv_bn_layer ( input = input , num_filters = channel_num [ 2 ], filter_size = 1 , name = name + '_conv2' ) return fluid . layers . elementwise_add ( x = shortcut_conv , y = input , axis = 0 , name = name + '_elementwise_add' ) def _shortcut ( self , input , channel_num , stride , name = None ): channel_in = input . shape [ 1 ] if channel_in != channel_num or stride != 1 : return self . conv_bn_layer ( input , num_filters = channel_num , filter_size = 1 , stride = stride , name = name + '_shortcut' ) else : return input def _conv_bn_layer ( self , input , num_filters , filter_size , stride = 1 , padding = 'SAME' , act = None , name = None ): conv = fluid . layers . conv2d ( input , num_filters , filter_size , stride , name = name + '_conv' ) bn = fluid . layers . batch_norm ( conv , act = act , name = name + '_bn' ) return bn","title":"\u641c\u7d22\u7a7a\u95f4"},{"location":"search_space/#_1","text":"\u641c\u7d22\u7a7a\u95f4\u662f\u795e\u7ecf\u7f51\u7edc\u641c\u7d22\u4e2d\u7684\u4e00\u4e2a\u6982\u5ff5\u3002\u641c\u7d22\u7a7a\u95f4\u662f\u4e00\u7cfb\u5217\u6a21\u578b\u7ed3\u6784\u7684\u6c47\u96c6, SANAS\u4e3b\u8981\u662f\u5229\u7528\u6a21\u62df\u9000\u706b\u7684\u601d\u60f3\u5728\u641c\u7d22\u7a7a\u95f4\u4e2d\u641c\u7d22\u5230\u4e00\u4e2a\u6bd4\u8f83\u5c0f\u7684\u6a21\u578b\u7ed3\u6784\u6216\u8005\u4e00\u4e2a\u7cbe\u5ea6\u6bd4\u8f83\u9ad8\u7684\u6a21\u578b\u7ed3\u6784\u3002","title":"\u641c\u7d22\u7a7a\u95f4\u7b80\u4ecb"},{"location":"search_space/#paddleslimnas","text":"","title":"paddleslim.nas \u63d0\u4f9b\u7684\u641c\u7d22\u7a7a\u95f4"},{"location":"search_space/#_2","text":"MobileNetV2Space \u2003 MobileNetV2\u7684\u7f51\u7edc\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \uff0c \u8bba\u6587 MobileNetV1Space \u2003 MobilNetV1\u7684\u7f51\u7edc\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \uff0c \u8bba\u6587 ResNetSpace \u2003 ResNetSpace\u7684\u7f51\u7edc\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 \uff0c \u8bba\u6587","title":"\u6839\u636e\u521d\u59cb\u6a21\u578b\u7ed3\u6784\u6784\u9020\u641c\u7d22\u7a7a\u95f4"},{"location":"search_space/#block","text":"MobileNetV1BlockSpace \u2003 MobileNetV1Block\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 MobileNetV2BlockSpace \u2003 MobileNetV2Block\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 ResNetBlockSpace \u2003 ResNetBlock\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 InceptionABlockSpace \u2003 InceptionABlock\u7684\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801 InceptionCBlockSpace \u2003 InceptionCBlock\u7ed3\u6784\u53ef\u4ee5\u53c2\u8003\uff1a \u4ee3\u7801","title":"\u6839\u636e\u76f8\u5e94\u6a21\u578b\u7684block\u6784\u9020\u641c\u7d22\u7a7a\u95f4"},{"location":"search_space/#_3","text":"\u4f7f\u7528paddleslim\u4e2d\u63d0\u4f9b\u7528\u521d\u59cb\u7684\u6a21\u578b\u7ed3\u6784\u6765\u6784\u9020\u641c\u7d22\u7a7a\u95f4\u7684\u8bdd\uff0c\u4ec5\u9700\u8981\u6307\u5b9a\u641c\u7d22\u7a7a\u95f4\u540d\u5b57\u5373\u53ef\u3002\u4f8b\u5982\uff1a\u5982\u679c\u4f7f\u7528\u539f\u672c\u7684MobileNetV2\u7684\u641c\u7d22\u7a7a\u95f4\u8fdb\u884c\u641c\u7d22\u7684\u8bdd\uff0c\u4f20\u5165SANAS\u4e2d\u7684config\u76f4\u63a5\u6307\u5b9a\u4e3a[('MobileNetV2Space')]\u3002 \u4f7f\u7528paddleslim\u4e2d\u63d0\u4f9b\u7684block\u641c\u7d22\u7a7a\u95f4\u6784\u9020\u641c\u7d22\u7a7a\u95f4\uff1a 2.1 \u4f7f\u7528 input_size , output_size \u548c block_num \u6765\u6784\u9020\u641c\u7d22\u7a7a\u95f4\u3002\u4f8b\u5982\uff1a\u4f20\u5165SANAS\u7684config\u53ef\u4ee5\u6307\u5b9a\u4e3a[('MobileNetV2BlockSpace', {'input_size': 224, 'output_size': 32, 'block_num': 10})]\u3002 2.2 \u4f7f\u7528 block_mask \u6784\u9020\u641c\u7d22\u7a7a\u95f4\u3002\u4f8b\u5982\uff1a\u4f20\u5165SANAS\u7684config\u53ef\u4ee5\u6307\u5b9a\u4e3a[('MobileNetV2BlockSpace', {'block_mask': [0, 1, 1, 1, 1, 0, 1, 0]})]\u3002","title":"\u641c\u7d22\u7a7a\u95f4\u793a\u4f8b"},{"location":"search_space/#search-space","text":"\u81ea\u5b9a\u4e49\u641c\u7d22\u7a7a\u95f4\u7c7b\u9700\u8981\u7ee7\u627f\u641c\u7d22\u7a7a\u95f4\u57fa\u7c7b\u5e76\u91cd\u5199\u4ee5\u4e0b\u51e0\u90e8\u5206\uff1a \u2003 1. \u521d\u59cb\u5316\u7684tokens( init_tokens \u51fd\u6570)\uff0c\u53ef\u4ee5\u8bbe\u7f6e\u4e3a\u81ea\u5df1\u60f3\u8981\u7684tokens\u5217\u8868, tokens\u5217\u8868\u4e2d\u7684\u6bcf\u4e2a\u6570\u5b57\u6307\u7684\u662f\u5f53\u524d\u6570\u5b57\u5728\u76f8\u5e94\u7684\u641c\u7d22\u5217\u8868\u4e2d\u7684\u7d22\u5f15\u3002\u4f8b\u5982\u672c\u793a\u4f8b\u4e2d\u82e5tokens=[0, 3, 5]\uff0c\u5219\u4ee3\u8868\u5f53\u524d\u6a21\u578b\u7ed3\u6784\u641c\u7d22\u5230\u7684\u901a\u9053\u6570\u4e3a[8, 40, 128]\u3002 \u2003 2. token\u4e2d\u6bcf\u4e2a\u6570\u5b57\u7684\u641c\u7d22\u5217\u8868\u957f\u5ea6( range_table \u51fd\u6570)\uff0ctokens\u4e2d\u6bcf\u4e2atoken\u7684\u7d22\u5f15\u8303\u56f4\u3002 \u2003 3. \u6839\u636etoken\u4ea7\u751f\u6a21\u578b\u7ed3\u6784( token2arch \u51fd\u6570)\uff0c\u6839\u636e\u641c\u7d22\u5230\u7684tokens\u5217\u8868\u4ea7\u751f\u6a21\u578b\u7ed3\u6784\u3002 \u4ee5\u65b0\u589ereset block\u4e3a\u4f8b\u8bf4\u660e\u5982\u4f55\u6784\u9020\u81ea\u5df1\u7684search space\u3002\u81ea\u5b9a\u4e49\u7684search space\u4e0d\u80fd\u548c\u5df2\u6709\u7684search space\u540c\u540d\u3002 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 ### \u5f15\u5165\u641c\u7d22\u7a7a\u95f4\u57fa\u7c7b\u51fd\u6570\u548csearch space\u7684\u6ce8\u518c\u7c7b\u51fd\u6570 from .search_space_base import SearchSpaceBase from .search_space_registry import SEARCHSPACE import numpy as np ### \u9700\u8981\u8c03\u7528\u6ce8\u518c\u51fd\u6570\u628a\u81ea\u5b9a\u4e49\u641c\u7d22\u7a7a\u95f4\u6ce8\u518c\u5230space space\u4e2d @SEARCHSPACE.register ### \u5b9a\u4e49\u4e00\u4e2a\u7ee7\u627fSearchSpaceBase\u57fa\u7c7b\u7684\u641c\u7d22\u7a7a\u95f4\u7684\u7c7b\u51fd\u6570 class ResNetBlockSpace2 ( SearchSpaceBase ): def __init__ ( self , input_size , output_size , block_num , block_mask ): ### \u5b9a\u4e49\u4e00\u4e9b\u5b9e\u9645\u60f3\u8981\u641c\u7d22\u7684\u5185\u5bb9\uff0c\u4f8b\u5982\uff1a\u901a\u9053\u6570\u3001\u6bcf\u4e2a\u5377\u79ef\u7684\u91cd\u590d\u6b21\u6570\u3001\u5377\u79ef\u6838\u5927\u5c0f\u7b49\u7b49 ### self.filter_num \u4ee3\u8868\u901a\u9053\u6570\u7684\u641c\u7d22\u5217\u8868 self . filter_num = np . array ([ 8 , 16 , 32 , 40 , 64 , 128 , 256 , 512 ]) ### \u5b9a\u4e49\u521d\u59cb\u5316token\uff0c\u521d\u59cb\u5316token\u7684\u957f\u5ea6\u6839\u636e\u4f20\u5165\u7684block_num\u6216\u8005block_mask\u7684\u957f\u5ea6\u6765\u5f97\u5230\u7684 def init_tokens ( self ): return [ 0 ] * 3 * len ( self . block_mask ) ### \u5b9a\u4e49token\u7684index\u7684\u53d6\u503c\u8303\u56f4 def range_table ( self ): return [ len ( self . filter_num )] * 3 * len ( self . block_mask ) ### \u628atoken\u8f6c\u6362\u6210\u6a21\u578b\u7ed3\u6784 def token2arch ( self , tokens = None ): if tokens == None : tokens = self . init_tokens () self . bottleneck_params_list = [] for i in range ( len ( self . block_mask )): self . bottleneck_params_list . append ( self . filter_num [ tokens [ i * 3 + 0 ]], self . filter_num [ tokens [ i * 3 + 1 ]], self . filter_num [ tokens [ i * 3 + 2 ]], 2 if self . block_mask [ i ] == 1 else 1 ) def net_arch ( input ): for i , layer_setting in enumerate ( self . bottleneck_params_list ): channel_num , stride = layer_setting [: - 1 ], layer_setting [ - 1 ] input = self . _resnet_block ( input , channel_num , stride , name = 'resnet_layer{}' . format ( i + 1 )) return input return net_arch ### \u6784\u9020\u5177\u4f53block\u7684\u64cd\u4f5c def _resnet_block ( self , input , channel_num , stride , name = None ): shortcut_conv = self . _shortcut ( input , channel_num [ 2 ], stride , name = name ) input = self . _conv_bn_layer ( input = input , num_filters = channel_num [ 0 ], filter_size = 1 , act = 'relu' , name = name + '_conv0' ) input = self . _conv_bn_layer ( input = input , num_filters = channel_num [ 1 ], filter_size = 3 , stride = stride , act = 'relu' , name = name + '_conv1' ) input = self . _conv_bn_layer ( input = input , num_filters = channel_num [ 2 ], filter_size = 1 , name = name + '_conv2' ) return fluid . layers . elementwise_add ( x = shortcut_conv , y = input , axis = 0 , name = name + '_elementwise_add' ) def _shortcut ( self , input , channel_num , stride , name = None ): channel_in = input . shape [ 1 ] if channel_in != channel_num or stride != 1 : return self . conv_bn_layer ( input , num_filters = channel_num , filter_size = 1 , stride = stride , name = name + '_shortcut' ) else : return input def _conv_bn_layer ( self , input , num_filters , filter_size , stride = 1 , padding = 'SAME' , act = None , name = None ): conv = fluid . layers . conv2d ( input , num_filters , filter_size , stride , name = name + '_conv' ) bn = fluid . layers . batch_norm ( conv , act = act , name = name + '_bn' ) return bn","title":"\u81ea\u5b9a\u4e49\u641c\u7d22\u7a7a\u95f4(search space)"},{"location":"table_latency/","text":"\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868 # \u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u7528\u4e8e\u5feb\u901f\u8bc4\u4f30\u4e00\u4e2a\u6a21\u578b\u5728\u7279\u5b9a\u786c\u4ef6\u73af\u5883\u548c\u63a8\u7406\u5f15\u64ce\u4e0a\u7684\u63a8\u7406\u901f\u5ea6\u3002 \u8be5\u6587\u6863\u4e3b\u8981\u7528\u4e8e\u5b9a\u4e49PaddleSlim\u652f\u6301\u7684\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u7684\u683c\u5f0f\u3002 \u6982\u8ff0 # \u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u4e2d\u5b58\u653e\u7740\u6240\u6709\u53ef\u80fd\u7684\u64cd\u4f5c\u5bf9\u5e94\u7684\u5ef6\u65f6\u4fe1\u606f\uff0c\u8be5\u8868\u4e2d\u7684\u4e00\u4e2a\u64cd\u4f5c\u5305\u62ec\u64cd\u4f5c\u7c7b\u578b\u548c\u64cd\u4f5c\u53c2\u6570\uff0c\u6bd4\u5982\uff1a\u64cd\u4f5c\u7c7b\u578b\u53ef\u4ee5\u662f conv2d \uff0c\u5bf9\u5e94\u7684\u64cd\u4f5c\u53c2\u6570\u6709\u8f93\u5165\u7279\u5f81\u56fe\u7684\u5927\u5c0f\u3001\u5377\u79ef\u6838\u4e2a\u6570\u3001\u5377\u79ef\u6838\u5927\u5c0f\u7b49\u3002 \u7ed9\u5b9a\u64cd\u4f5c\u7684\u5ef6\u65f6\u4f9d\u8d56\u4e8e\u786c\u4ef6\u73af\u5883\u548c\u63a8\u7406\u5f15\u64ce\u3002 \u6574\u4f53\u683c\u5f0f # \u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u4ee5\u6587\u4ef6\u6216\u591a\u884c\u5b57\u7b26\u4e32\u7684\u5f62\u5f0f\u4fdd\u5b58\u3002 \u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u7b2c\u4e00\u884c\u4fdd\u5b58\u7248\u672c\u4fe1\u606f\uff0c\u540e\u7eed\u6bcf\u884c\u4e3a\u4e00\u4e2a\u64cd\u4f5c\u548c\u5bf9\u5e94\u7684\u5ef6\u65f6\u4fe1\u606f\u3002 \u7248\u672c\u4fe1\u606f # \u7248\u672c\u4fe1\u606f\u4ee5\u82f1\u6587\u5b57\u7b26\u9017\u53f7\u5206\u5272\uff0c\u5185\u5bb9\u4f9d\u6b21\u4e3a\u786c\u4ef6\u73af\u5883\u540d\u79f0\u3001\u63a8\u7406\u5f15\u64ce\u540d\u79f0\u548c\u65f6\u95f4\u6233\u3002 \u786c\u4ef6\u73af\u5883\u540d\u79f0\uff1a \u7528\u4e8e\u6807\u8bc6\u786c\u4ef6\u73af\u5883\uff0c\u53ef\u4ee5\u5305\u542b\u8ba1\u7b97\u67b6\u6784\u7c7b\u578b\u3001\u7248\u672c\u53f7\u7b49\u4fe1\u606f\u3002 \u63a8\u7406\u5f15\u64ce\u540d\u79f0\uff1a \u7528\u4e8e\u6807\u8bc6\u63a8\u7406\u5f15\u64ce\uff0c\u53ef\u4ee5\u5305\u542b\u63a8\u7406\u5f15\u64ce\u540d\u79f0\u3001\u7248\u672c\u53f7\u3001\u4f18\u5316\u9009\u9879\u7b49\u4fe1\u606f\u3002 \u65f6\u95f4\u6233\uff1a \u8be5\u8bc4\u4f30\u8868\u7684\u521b\u5efa\u65f6\u95f4\u3002 \u64cd\u4f5c\u4fe1\u606f # \u64cd\u4f5c\u4fe1\u606f\u5b57\u6bb5\u4e4b\u95f4\u4ee5\u9017\u53f7\u5206\u5272\u3002\u64cd\u4f5c\u4fe1\u606f\u4e0e\u5ef6\u8fdf\u4fe1\u606f\u4e4b\u95f4\u4ee5\u5236\u8868\u7b26\u5206\u5272\u3002 conv2d # \u683c\u5f0f 1 op_type , flag_bias , flag_relu , n_in , c_in , h_in , w_in , c_out , groups , kernel , padding , stride , dilation \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 flag_bias (int) - \u662f\u5426\u6709 bias\uff080\uff1a\u65e0\uff0c1\uff1a\u6709\uff09\u3002 flag_relu (int) - \u662f\u5426\u6709 relu\uff080\uff1a\u65e0\uff0c1\uff1a\u6709\uff09\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 c_out (int) - \u8f93\u51fa Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 groups (int) - \u5377\u79ef\u4e8c\u7ef4\u5c42\uff08Conv2D Layer\uff09\u7684\u7ec4\u6570\u3002 kernel (int) - \u5377\u79ef\u6838\u5927\u5c0f\u3002 padding (int) - \u586b\u5145 (padding) \u5927\u5c0f\u3002 stride (int) - \u6b65\u957f (stride) \u5927\u5c0f\u3002 dilation (int) - \u81a8\u80c0 (dilation) \u5927\u5c0f\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4 activation # \u683c\u5f0f 1 op_type , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4 batch_norm # \u683c\u5f0f 1 op_type , active_type , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 active_type (string|None) - \u6fc0\u6d3b\u51fd\u6570\u7c7b\u578b\uff0c\u5305\u542b\uff1arelu, prelu, sigmoid, relu6, tanh\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4 eltwise # \u683c\u5f0f 1 op_type , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4 pooling # \u683c\u5f0f 1 op_type , flag_global_pooling , n_in , c_in , h_in , w_in , kernel , padding , stride , ceil_mode , pool_type \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 flag_global_pooling (int) - \u662f\u5426\u4e3a\u5168\u5c40\u6c60\u5316\uff080\uff1a\u4e0d\u662f\uff0c1\uff1a\u662f\uff09\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 kernel (int) - \u5377\u79ef\u6838\u5927\u5c0f\u3002 padding (int) - \u586b\u5145 (padding) \u5927\u5c0f\u3002 stride (int) - \u6b65\u957f (stride) \u5927\u5c0f\u3002 ceil_mode (int) - \u662f\u5426\u7528 ceil \u51fd\u6570\u8ba1\u7b97\u8f93\u51fa\u9ad8\u5ea6\u548c\u5bbd\u5ea6\u30020 \u8868\u793a\u4f7f\u7528 floor \u51fd\u6570\uff0c1 \u8868\u793a\u4f7f\u7528 ceil \u51fd\u6570\u3002 pool_type (int) - \u6c60\u5316\u7c7b\u578b\uff0c\u5176\u4e2d 1 \u8868\u793a pooling_max\uff0c2 \u8868\u793a pooling_average_include_padding\uff0c3 \u8868\u793a pooling_average_exclude_padding\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4 softmax # \u683c\u5f0f 1 op_type , axis , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 axis (int) - \u6267\u884c softmax \u8ba1\u7b97\u7684\u7ef4\u5ea6\u7d22\u5f15\uff0c\u5e94\u8be5\u5728 [\u22121\uff0crank \u2212 1] \u8303\u56f4\u5185\uff0c\u5176\u4e2d rank \u662f\u8f93\u5165\u53d8\u91cf\u7684\u79e9\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868"},{"location":"table_latency/#_1","text":"\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u7528\u4e8e\u5feb\u901f\u8bc4\u4f30\u4e00\u4e2a\u6a21\u578b\u5728\u7279\u5b9a\u786c\u4ef6\u73af\u5883\u548c\u63a8\u7406\u5f15\u64ce\u4e0a\u7684\u63a8\u7406\u901f\u5ea6\u3002 \u8be5\u6587\u6863\u4e3b\u8981\u7528\u4e8e\u5b9a\u4e49PaddleSlim\u652f\u6301\u7684\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u7684\u683c\u5f0f\u3002","title":"\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868"},{"location":"table_latency/#_2","text":"\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u4e2d\u5b58\u653e\u7740\u6240\u6709\u53ef\u80fd\u7684\u64cd\u4f5c\u5bf9\u5e94\u7684\u5ef6\u65f6\u4fe1\u606f\uff0c\u8be5\u8868\u4e2d\u7684\u4e00\u4e2a\u64cd\u4f5c\u5305\u62ec\u64cd\u4f5c\u7c7b\u578b\u548c\u64cd\u4f5c\u53c2\u6570\uff0c\u6bd4\u5982\uff1a\u64cd\u4f5c\u7c7b\u578b\u53ef\u4ee5\u662f conv2d \uff0c\u5bf9\u5e94\u7684\u64cd\u4f5c\u53c2\u6570\u6709\u8f93\u5165\u7279\u5f81\u56fe\u7684\u5927\u5c0f\u3001\u5377\u79ef\u6838\u4e2a\u6570\u3001\u5377\u79ef\u6838\u5927\u5c0f\u7b49\u3002 \u7ed9\u5b9a\u64cd\u4f5c\u7684\u5ef6\u65f6\u4f9d\u8d56\u4e8e\u786c\u4ef6\u73af\u5883\u548c\u63a8\u7406\u5f15\u64ce\u3002","title":"\u6982\u8ff0"},{"location":"table_latency/#_3","text":"\u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u4ee5\u6587\u4ef6\u6216\u591a\u884c\u5b57\u7b26\u4e32\u7684\u5f62\u5f0f\u4fdd\u5b58\u3002 \u786c\u4ef6\u5ef6\u65f6\u8bc4\u4f30\u8868\u7b2c\u4e00\u884c\u4fdd\u5b58\u7248\u672c\u4fe1\u606f\uff0c\u540e\u7eed\u6bcf\u884c\u4e3a\u4e00\u4e2a\u64cd\u4f5c\u548c\u5bf9\u5e94\u7684\u5ef6\u65f6\u4fe1\u606f\u3002","title":"\u6574\u4f53\u683c\u5f0f"},{"location":"table_latency/#_4","text":"\u7248\u672c\u4fe1\u606f\u4ee5\u82f1\u6587\u5b57\u7b26\u9017\u53f7\u5206\u5272\uff0c\u5185\u5bb9\u4f9d\u6b21\u4e3a\u786c\u4ef6\u73af\u5883\u540d\u79f0\u3001\u63a8\u7406\u5f15\u64ce\u540d\u79f0\u548c\u65f6\u95f4\u6233\u3002 \u786c\u4ef6\u73af\u5883\u540d\u79f0\uff1a \u7528\u4e8e\u6807\u8bc6\u786c\u4ef6\u73af\u5883\uff0c\u53ef\u4ee5\u5305\u542b\u8ba1\u7b97\u67b6\u6784\u7c7b\u578b\u3001\u7248\u672c\u53f7\u7b49\u4fe1\u606f\u3002 \u63a8\u7406\u5f15\u64ce\u540d\u79f0\uff1a \u7528\u4e8e\u6807\u8bc6\u63a8\u7406\u5f15\u64ce\uff0c\u53ef\u4ee5\u5305\u542b\u63a8\u7406\u5f15\u64ce\u540d\u79f0\u3001\u7248\u672c\u53f7\u3001\u4f18\u5316\u9009\u9879\u7b49\u4fe1\u606f\u3002 \u65f6\u95f4\u6233\uff1a \u8be5\u8bc4\u4f30\u8868\u7684\u521b\u5efa\u65f6\u95f4\u3002","title":"\u7248\u672c\u4fe1\u606f"},{"location":"table_latency/#_5","text":"\u64cd\u4f5c\u4fe1\u606f\u5b57\u6bb5\u4e4b\u95f4\u4ee5\u9017\u53f7\u5206\u5272\u3002\u64cd\u4f5c\u4fe1\u606f\u4e0e\u5ef6\u8fdf\u4fe1\u606f\u4e4b\u95f4\u4ee5\u5236\u8868\u7b26\u5206\u5272\u3002","title":"\u64cd\u4f5c\u4fe1\u606f"},{"location":"table_latency/#conv2d","text":"\u683c\u5f0f 1 op_type , flag_bias , flag_relu , n_in , c_in , h_in , w_in , c_out , groups , kernel , padding , stride , dilation \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 flag_bias (int) - \u662f\u5426\u6709 bias\uff080\uff1a\u65e0\uff0c1\uff1a\u6709\uff09\u3002 flag_relu (int) - \u662f\u5426\u6709 relu\uff080\uff1a\u65e0\uff0c1\uff1a\u6709\uff09\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 c_out (int) - \u8f93\u51fa Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 groups (int) - \u5377\u79ef\u4e8c\u7ef4\u5c42\uff08Conv2D Layer\uff09\u7684\u7ec4\u6570\u3002 kernel (int) - \u5377\u79ef\u6838\u5927\u5c0f\u3002 padding (int) - \u586b\u5145 (padding) \u5927\u5c0f\u3002 stride (int) - \u6b65\u957f (stride) \u5927\u5c0f\u3002 dilation (int) - \u81a8\u80c0 (dilation) \u5927\u5c0f\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"conv2d"},{"location":"table_latency/#activation","text":"\u683c\u5f0f 1 op_type , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"activation"},{"location":"table_latency/#batch_norm","text":"\u683c\u5f0f 1 op_type , active_type , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 active_type (string|None) - \u6fc0\u6d3b\u51fd\u6570\u7c7b\u578b\uff0c\u5305\u542b\uff1arelu, prelu, sigmoid, relu6, tanh\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"batch_norm"},{"location":"table_latency/#eltwise","text":"\u683c\u5f0f 1 op_type , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"eltwise"},{"location":"table_latency/#pooling","text":"\u683c\u5f0f 1 op_type , flag_global_pooling , n_in , c_in , h_in , w_in , kernel , padding , stride , ceil_mode , pool_type \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 flag_global_pooling (int) - \u662f\u5426\u4e3a\u5168\u5c40\u6c60\u5316\uff080\uff1a\u4e0d\u662f\uff0c1\uff1a\u662f\uff09\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 kernel (int) - \u5377\u79ef\u6838\u5927\u5c0f\u3002 padding (int) - \u586b\u5145 (padding) \u5927\u5c0f\u3002 stride (int) - \u6b65\u957f (stride) \u5927\u5c0f\u3002 ceil_mode (int) - \u662f\u5426\u7528 ceil \u51fd\u6570\u8ba1\u7b97\u8f93\u51fa\u9ad8\u5ea6\u548c\u5bbd\u5ea6\u30020 \u8868\u793a\u4f7f\u7528 floor \u51fd\u6570\uff0c1 \u8868\u793a\u4f7f\u7528 ceil \u51fd\u6570\u3002 pool_type (int) - \u6c60\u5316\u7c7b\u578b\uff0c\u5176\u4e2d 1 \u8868\u793a pooling_max\uff0c2 \u8868\u793a pooling_average_include_padding\uff0c3 \u8868\u793a pooling_average_exclude_padding\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"pooling"},{"location":"table_latency/#softmax","text":"\u683c\u5f0f 1 op_type , axis , n_in , c_in , h_in , w_in \\ tlatency \u5b57\u6bb5\u89e3\u91ca op_type(str) - \u5f53\u524dop\u7c7b\u578b\u3002 axis (int) - \u6267\u884c softmax \u8ba1\u7b97\u7684\u7ef4\u5ea6\u7d22\u5f15\uff0c\u5e94\u8be5\u5728 [\u22121\uff0crank \u2212 1] \u8303\u56f4\u5185\uff0c\u5176\u4e2d rank \u662f\u8f93\u5165\u53d8\u91cf\u7684\u79e9\u3002 n_in (int) - \u8f93\u5165 Tensor \u7684\u6279\u5c3a\u5bf8 (batch size)\u3002 c_in (int) - \u8f93\u5165 Tensor \u7684\u901a\u9053 (channel) \u6570\u3002 h_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u9ad8\u5ea6\u3002 w_in (int) - \u8f93\u5165 Tensor \u7684\u7279\u5f81\u5bbd\u5ea6\u3002 latency (float) - \u5f53\u524dop\u7684\u5ef6\u65f6\u65f6\u95f4","title":"softmax"},{"location":"algo/algo/","text":"\u76ee\u5f55 # \u91cf\u5316\u539f\u7406\u4ecb\u7ecd \u526a\u88c1\u539f\u7406\u4ecb\u7ecd \u84b8\u998f\u539f\u7406\u4ecb\u7ecd \u8f7b\u91cf\u7ea7\u6a21\u578b\u7ed3\u6784\u641c\u7d22\u539f\u7406\u4ecb\u7ecd 1. Quantization Aware Training\u91cf\u5316\u4ecb\u7ecd # 1.1 \u80cc\u666f # \u8fd1\u5e74\u6765\uff0c\u5b9a\u70b9\u91cf\u5316\u4f7f\u7528\u66f4\u5c11\u7684\u6bd4\u7279\u6570\uff08\u59828-bit\u30013-bit\u30012-bit\u7b49\uff09\u8868\u793a\u795e\u7ecf\u7f51\u7edc\u7684\u6743\u91cd\u548c\u6fc0\u6d3b\u5df2\u88ab\u9a8c\u8bc1\u662f\u6709\u6548\u7684\u3002\u5b9a\u70b9\u91cf\u5316\u7684\u4f18\u70b9\u5305\u62ec\u4f4e\u5185\u5b58\u5e26\u5bbd\u3001\u4f4e\u529f\u8017\u3001\u4f4e\u8ba1\u7b97\u8d44\u6e90\u5360\u7528\u4ee5\u53ca\u4f4e\u6a21\u578b\u5b58\u50a8\u9700\u6c42\u7b49\u3002 \u88681: \u4e0d\u540c\u7c7b\u578b\u64cd\u4f5c\u7684\u5f00\u9500\u5bf9\u6bd4 \u7531\u88681\u53ef\u77e5\uff0c\u4f4e\u7cbe\u5ea6\u5b9a\u70b9\u6570\u64cd\u4f5c\u7684\u786c\u4ef6\u9762\u79ef\u5927\u5c0f\u53ca\u80fd\u8017\u6bd4\u9ad8\u7cbe\u5ea6\u6d6e\u70b9\u6570\u8981\u5c11\u51e0\u4e2a\u6570\u91cf\u7ea7\u3002 \u4f7f\u7528\u5b9a\u70b9\u91cf\u5316\u53ef\u5e26\u67654\u500d\u7684\u6a21\u578b\u538b\u7f29\u30014\u500d\u7684\u5185\u5b58\u5e26\u5bbd\u63d0\u5347\uff0c\u4ee5\u53ca\u66f4\u9ad8\u6548\u7684cache\u5229\u7528(\u5f88\u591a\u786c\u4ef6\u8bbe\u5907\uff0c\u5185\u5b58\u8bbf\u95ee\u662f\u4e3b\u8981\u80fd\u8017)\u3002\u9664\u6b64\u4e4b\u5916\uff0c\u8ba1\u7b97\u901f\u5ea6\u4e5f\u4f1a\u66f4\u5feb(\u901a\u5e38\u5177\u67092x-3x\u7684\u6027\u80fd\u63d0\u5347)\u3002\u7531\u88682\u53ef\u77e5\uff0c\u5728\u5f88\u591a\u573a\u666f\u4e0b\uff0c\u5b9a\u70b9\u91cf\u5316\u64cd\u4f5c\u5bf9\u7cbe\u5ea6\u5e76\u4e0d\u4f1a\u9020\u6210\u635f\u5931\u3002\u53e6\u5916\uff0c\u5b9a\u70b9\u91cf\u5316\u5bf9\u795e\u7ecf\u7f51\u7edc\u4e8e\u5d4c\u5165\u5f0f\u8bbe\u5907\u4e0a\u7684\u63a8\u65ad\u6765\u8bf4\u662f\u6781\u5176\u91cd\u8981\u7684\u3002 \u88682\uff1a\u6a21\u578b\u91cf\u5316\u524d\u540e\u7cbe\u5ea6\u5bf9\u6bd4 \u76ee\u524d\uff0c\u5b66\u672f\u754c\u4e3b\u8981\u5c06\u91cf\u5316\u5206\u4e3a\u4e24\u5927\u7c7b\uff1a Post Training Quantization \u548c Quantization Aware Training \u3002 Post Training Quantization \u662f\u6307\u4f7f\u7528KL\u6563\u5ea6\u3001\u6ed1\u52a8\u5e73\u5747\u7b49\u65b9\u6cd5\u786e\u5b9a\u91cf\u5316\u53c2\u6570\u4e14\u4e0d\u9700\u8981\u91cd\u65b0\u8bad\u7ec3\u7684\u5b9a\u70b9\u91cf\u5316\u65b9\u6cd5\u3002 Quantization Aware Training \u662f\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u5bf9\u91cf\u5316\u8fdb\u884c\u5efa\u6a21\u4ee5\u786e\u5b9a\u91cf\u5316\u53c2\u6570\uff0c\u5b83\u4e0e Post Training Quantization \u6a21\u5f0f\u76f8\u6bd4\u53ef\u4ee5\u63d0\u4f9b\u66f4\u9ad8\u7684\u9884\u6d4b\u7cbe\u5ea6\u3002 1.2 \u91cf\u5316\u539f\u7406 # 1.2.1 \u91cf\u5316\u65b9\u5f0f # \u76ee\u524d\uff0c\u5b58\u5728\u7740\u8bb8\u591a\u65b9\u6cd5\u53ef\u4ee5\u5c06\u6d6e\u70b9\u6570\u91cf\u5316\u6210\u5b9a\u70b9\u6570\u3002\u4f8b\u5982\uff1a r = min(max(x, a), b) s = \\frac{b - a}{n - 1} q = \\left \\lfloor \\frac{r - a}{s} \\right \\rceil \u5f0f\u4e2d\uff0c x x \u662f\u5f85\u91cf\u5316\u7684\u6d6e\u70b9\u503c\uff0c [a, b] [a, b] \u662f\u91cf\u5316\u8303\u56f4\uff0c a a \u662f\u5f85\u91cf\u5316\u6d6e\u70b9\u6570\u4e2d\u7684\u6700\u5c0f\u503c\uff0c b b \u662f\u5f85\u91cf\u5316\u6d6e\u70b9\u6570\u4e2d\u7684\u6700\u5927\u503c\u3002 \\left \\lfloor \\right \\rceil \\left \\lfloor \\right \\rceil \u8868\u793a\u5c06\u7ed3\u679c\u56db\u820d\u4e94\u5165\u5230\u6700\u8fd1\u7684\u6574\u6570\u3002\u5982\u679c\u91cf\u5316\u7ea7\u522b\u4e3a k k \uff0c\u5219 n n \u4e3a 2^k 2^k \u3002\u4f8b\u5982\uff0c\u82e5 k k \u4e3a8\uff0c\u5219 n n \u4e3a256\u3002 q q \u662f\u91cf\u5316\u5f97\u5230\u7684\u6574\u6570\u3002 PaddleSlim\u6846\u67b6\u4e2d\u9009\u62e9\u7684\u91cf\u5316\u65b9\u6cd5\u4e3a\u6700\u5927\u7edd\u5bf9\u503c\u91cf\u5316( max-abs )\uff0c\u5177\u4f53\u63cf\u8ff0\u5982\u4e0b\uff1a M = max(abs(x)) q = \\left \\lfloor \\frac{x}{M} * (n - 1) \\right \\rceil \u5f0f\u4e2d\uff0c x x \u662f\u5f85\u88ab\u91cf\u5316\u7684\u6d6e\u70b9\u503c\uff0c M M \u662f\u5f85\u91cf\u5316\u6d6e\u70b9\u6570\u4e2d\u7684\u7edd\u5bf9\u503c\u6700\u5927\u503c\u3002 \\left \\lfloor \\right \\rceil \\left \\lfloor \\right \\rceil \u8868\u793a\u5c06\u7ed3\u679c\u56db\u820d\u4e94\u5165\u5230\u6700\u8fd1\u7684\u6574\u6570\u3002\u5bf9\u4e8e8bit\u91cf\u5316\uff0cPaddleSlim\u91c7\u7528 int8_t \uff0c\u5373 n=2^7=128 n=2^7=128 \u3002 q q \u662f\u91cf\u5316\u5f97\u5230\u7684\u6574\u6570\u3002 \u65e0\u8bba\u662f min-max\u91cf\u5316 \u8fd8\u662f max-abs\u91cf\u5316 \uff0c\u4ed6\u4eec\u90fd\u53ef\u4ee5\u8868\u793a\u4e3a\u5982\u4e0b\u5f62\u5f0f\uff1a q = scale * r + b q = scale * r + b \u5176\u4e2d min-max \u548c max-abs \u88ab\u79f0\u4e3a\u91cf\u5316\u53c2\u6570\u6216\u8005\u91cf\u5316\u6bd4\u4f8b\u6216\u8005\u91cf\u5316\u8303\u56f4\u3002 1.2.2 \u91cf\u5316\u8bad\u7ec3 # 1.2.2.1 \u524d\u5411\u4f20\u64ad # \u524d\u5411\u4f20\u64ad\u8fc7\u7a0b\u91c7\u7528\u6a21\u62df\u91cf\u5316\u7684\u65b9\u5f0f\uff0c\u5177\u4f53\u63cf\u8ff0\u5982\u4e0b\uff1a \u56fe1\uff1a\u57fa\u4e8e\u6a21\u62df\u91cf\u5316\u8bad\u7ec3\u7684\u524d\u5411\u8fc7\u7a0b \u7531\u56fe1\u53ef\u77e5\uff0c\u57fa\u4e8e\u6a21\u62df\u91cf\u5316\u8bad\u7ec3\u7684\u524d\u5411\u8fc7\u7a0b\u53ef\u88ab\u63cf\u8ff0\u4e3a\u4ee5\u4e0b\u56db\u4e2a\u90e8\u5206\uff1a 1) \u8f93\u5165\u548c\u6743\u91cd\u5747\u88ab\u91cf\u5316\u62108-bit\u6574\u6570\u3002 2) \u57288-bit\u6574\u6570\u4e0a\u6267\u884c\u77e9\u9635\u4e58\u6cd5\u6216\u8005\u5377\u79ef\u64cd\u4f5c\u3002 3) \u53cd\u91cf\u5316\u77e9\u9635\u4e58\u6cd5\u6216\u8005\u5377\u79ef\u64cd\u4f5c\u7684\u8f93\u51fa\u7ed3\u679c\u4e3a32-bit\u6d6e\u70b9\u578b\u6570\u636e\u3002 4) \u572832-bit\u6d6e\u70b9\u578b\u6570\u636e\u4e0a\u6267\u884c\u504f\u7f6e\u52a0\u6cd5\u64cd\u4f5c\u3002\u6b64\u5904\uff0c\u504f\u7f6e\u5e76\u672a\u88ab\u91cf\u5316\u3002 \u5bf9\u4e8e\u901a\u7528\u77e9\u9635\u4e58\u6cd5( GEMM )\uff0c\u8f93\u5165 X X \u548c\u6743\u91cd W W \u7684\u91cf\u5316\u64cd\u4f5c\u53ef\u88ab\u8868\u8ff0\u4e3a\u5982\u4e0b\u8fc7\u7a0b\uff1a X_q = \\left \\lfloor \\frac{X}{X_m} * (n - 1) \\right \\rceil W_q = \\left \\lfloor \\frac{W}{W_m} * (n - 1) \\right \\rceil \u6267\u884c\u901a\u7528\u77e9\u9635\u4e58\u6cd5\uff1a Y_q = X_q * W_q \u5bf9\u91cf\u5316\u4e58\u79ef\u7ed3\u679c Yq Yq \u8fdb\u884c\u53cd\u91cf\u5316: \\begin{align} Y_{dq} = \\frac{Y_q}{(n - 1) * (n - 1)} * X_m * W_m \\ =\\frac{X_q * W_q}{(n - 1) * (n - 1)} * X_m * W_m \\ =(\\frac{X_q}{n - 1} * X_m) * (\\frac{W_q}{n - 1} * W_m) \\ \\end{align} \u4e0a\u8ff0\u516c\u5f0f\u8868\u660e\u53cd\u91cf\u5316\u64cd\u4f5c\u53ef\u4ee5\u88ab\u79fb\u52a8\u5230 GEMM \u4e4b\u524d\uff0c\u5373\u5148\u5bf9 Xq Xq \u548c Wq Wq \u6267\u884c\u53cd\u91cf\u5316\u64cd\u4f5c\u518d\u505a GEMM \u64cd\u4f5c\u3002\u56e0\u6b64\uff0c\u524d\u5411\u4f20\u64ad\u7684\u5de5\u4f5c\u6d41\u4ea6\u53ef\u8868\u793a\u4e3a\u5982\u4e0b\u65b9\u5f0f\uff1a \u56fe2\uff1a\u57fa\u4e8e\u6a21\u62df\u91cf\u5316\u8bad\u7ec3\u524d\u5411\u8fc7\u7a0b\u7684\u7b49\u4ef7\u5de5\u4f5c\u6d41 \u8bad\u7ec3\u8fc7\u7a0b\u4e2d\uff0cPaddleSlim\u4f7f\u7528\u56fe2\u4e2d\u6240\u793a\u7684\u7b49\u4ef7\u5de5\u4f5c\u6d41\u3002\u5728\u8bbe\u8ba1\u4e2d\uff0c\u91cf\u5316Pass\u5728IrGraph\u4e2d\u63d2\u5165\u91cf\u5316op\u548c\u53cd\u91cf\u5316op\u3002\u56e0\u4e3a\u5728\u8fde\u7eed\u7684\u91cf\u5316\u3001\u53cd\u91cf\u5316\u64cd\u4f5c\u4e4b\u540e\u8f93\u5165\u4ecd\u7136\u4e3a32-bit\u6d6e\u70b9\u578b\u6570\u636e\u3002\u56e0\u6b64\uff0cPaddleSlim\u91cf\u5316\u8bad\u7ec3\u6846\u67b6\u6240\u91c7\u7528\u7684\u91cf\u5316\u65b9\u5f0f\u88ab\u79f0\u4e3a\u6a21\u62df\u91cf\u5316\u3002 1.2.2.2 \u53cd\u5411\u4f20\u64ad # \u7531\u56fe3\u53ef\u77e5\uff0c\u6743\u91cd\u66f4\u65b0\u6240\u9700\u7684\u68af\u5ea6\u503c\u53ef\u4ee5\u7531\u91cf\u5316\u540e\u7684\u6743\u91cd\u548c\u91cf\u5316\u540e\u7684\u6fc0\u6d3b\u6c42\u5f97\u3002\u53cd\u5411\u4f20\u64ad\u8fc7\u7a0b\u4e2d\u7684\u6240\u6709\u8f93\u5165\u548c\u8f93\u51fa\u5747\u4e3a32-bit\u6d6e\u70b9\u578b\u6570\u636e\u3002\u6ce8\u610f\uff0c\u68af\u5ea6\u66f4\u65b0\u64cd\u4f5c\u9700\u8981\u5728\u539f\u59cb\u6743\u91cd\u4e0a\u8fdb\u884c\uff0c\u5373\u8ba1\u7b97\u51fa\u7684\u68af\u5ea6\u5c06\u88ab\u52a0\u5230\u539f\u59cb\u6743\u91cd\u4e0a\u800c\u975e\u91cf\u5316\u540e\u6216\u53cd\u91cf\u5316\u540e\u7684\u6743\u91cd\u4e0a\u3002 \u56fe3\uff1a\u57fa\u4e8e\u6a21\u62df\u91cf\u5316\u8bad\u7ec3\u7684\u53cd\u5411\u4f20\u64ad\u548c\u6743\u91cd\u66f4\u65b0\u8fc7\u7a0b \u56e0\u6b64\uff0c\u91cf\u5316Pass\u4e5f\u4f1a\u6539\u53d8\u76f8\u5e94\u53cd\u5411\u7b97\u5b50\u7684\u67d0\u4e9b\u8f93\u5165\u3002 1.2.2.3 \u786e\u5b9a\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570 # \u5b58\u5728\u7740\u4e24\u79cd\u7b56\u7565\u53ef\u4ee5\u8ba1\u7b97\u6c42\u53d6\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570\uff0c\u5373\u52a8\u6001\u7b56\u7565\u548c\u9759\u6001\u7b56\u7565\u3002\u52a8\u6001\u7b56\u7565\u4f1a\u5728\u6bcf\u6b21\u8fed\u4ee3\u8fc7\u7a0b\u4e2d\u8ba1\u7b97\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570\u7684\u503c\u3002\u9759\u6001\u7b56\u7565\u5219\u5bf9\u4e0d\u540c\u7684\u8f93\u5165\u91c7\u7528\u76f8\u540c\u7684\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570\u3002 \u5bf9\u4e8e\u6743\u91cd\u800c\u8a00\uff0c\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u91c7\u7528\u52a8\u6001\u7b56\u7565\u3002\u6362\u53e5\u8bdd\u8bf4\uff0c\u5728\u6bcf\u6b21\u8fed\u4ee3\u8fc7\u7a0b\u4e2d\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570\u5747\u4f1a\u88ab\u91cd\u65b0\u8ba1\u7b97\u5f97\u5230\u76f4\u81f3\u8bad\u7ec3\u8fc7\u7a0b\u7ed3\u675f\u3002 \u5bf9\u4e8e\u6fc0\u6d3b\u800c\u8a00\uff0c\u53ef\u4ee5\u9009\u62e9\u52a8\u6001\u7b56\u7565\u4e5f\u53ef\u4ee5\u9009\u62e9\u9759\u6001\u7b56\u7565\u3002\u82e5\u9009\u62e9\u4f7f\u7528\u9759\u6001\u7b56\u7565\uff0c\u5219\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570\u4f1a\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u88ab\u8bc4\u4f30\u6c42\u5f97\uff0c\u4e14\u5728\u63a8\u65ad\u8fc7\u7a0b\u4e2d\u88ab\u4f7f\u7528(\u4e0d\u540c\u7684\u8f93\u5165\u5747\u4fdd\u6301\u4e0d\u53d8)\u3002\u9759\u6001\u7b56\u7565\u4e2d\u7684\u91cf\u5316\u6bd4\u4f8b\u7cfb\u6570\u53ef\u4e8e\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u901a\u8fc7\u5982\u4e0b\u4e09\u79cd\u65b9\u5f0f\u8fdb\u884c\u8bc4\u4f30\uff1a \u5728\u4e00\u4e2a\u7a97\u53e3\u4e2d\u8ba1\u7b97\u6fc0\u6d3b\u6700\u5927\u7edd\u5bf9\u503c\u7684\u5e73\u5747\u503c\u3002 \u5728\u4e00\u4e2a\u7a97\u53e3\u4e2d\u8ba1\u7b97\u6fc0\u6d3b\u6700\u5927\u7edd\u5bf9\u503c\u7684\u6700\u5927\u503c\u3002 \u5728\u4e00\u4e2a\u7a97\u53e3\u4e2d\u8ba1\u7b97\u6fc0\u6d3b\u6700\u5927\u7edd\u5bf9\u503c\u7684\u6ed1\u52a8\u5e73\u5747\u503c\uff0c\u8ba1\u7b97\u516c\u5f0f\u5982\u4e0b\uff1a Vt = (1 - k) * V + k * V_{t-1} Vt = (1 - k) * V + k * V_{t-1} \u5f0f\u4e2d\uff0c V V \u662f\u5f53\u524dbatch\u7684\u6700\u5927\u7edd\u5bf9\u503c\uff0c Vt Vt \u662f\u6ed1\u52a8\u5e73\u5747\u503c\u3002 k k \u662f\u4e00\u4e2a\u56e0\u5b50\uff0c\u4f8b\u5982\u5176\u503c\u53ef\u53d6\u4e3a0.9\u3002 1.2.4 \u8bad\u7ec3\u540e\u91cf\u5316 # \u8bad\u7ec3\u540e\u91cf\u5316\u662f\u57fa\u4e8e\u91c7\u6837\u6570\u636e\uff0c\u91c7\u7528KL\u6563\u5ea6\u7b49\u65b9\u6cd5\u8ba1\u7b97\u91cf\u5316\u6bd4\u4f8b\u56e0\u5b50\u7684\u65b9\u6cd5\u3002\u76f8\u6bd4\u91cf\u5316\u8bad\u7ec3\uff0c\u8bad\u7ec3\u540e\u91cf\u5316\u4e0d\u9700\u8981\u91cd\u65b0\u8bad\u7ec3\uff0c\u53ef\u4ee5\u5feb\u901f\u5f97\u5230\u91cf\u5316\u6a21\u578b\u3002 \u8bad\u7ec3\u540e\u91cf\u5316\u7684\u76ee\u6807\u662f\u6c42\u53d6\u91cf\u5316\u6bd4\u4f8b\u56e0\u5b50\uff0c\u4e3b\u8981\u6709\u4e24\u79cd\u65b9\u6cd5\uff1a\u975e\u9971\u548c\u91cf\u5316\u65b9\u6cd5 ( No Saturation) \u548c\u9971\u548c\u91cf\u5316\u65b9\u6cd5 (Saturation)\u3002\u975e\u9971\u548c\u91cf\u5316\u65b9\u6cd5\u8ba1\u7b97FP32\u7c7b\u578bTensor\u4e2d\u7edd\u5bf9\u503c\u7684\u6700\u5927\u503c abs_max \uff0c\u5c06\u5176\u6620\u5c04\u4e3a127\uff0c\u5219\u91cf\u5316\u6bd4\u4f8b\u56e0\u5b50\u7b49\u4e8e abs_max/127 \u3002\u9971\u548c\u91cf\u5316\u65b9\u6cd5\u4f7f\u7528KL\u6563\u5ea6\u8ba1\u7b97\u4e00\u4e2a\u5408\u9002\u7684\u9608\u503c T ( 0