diff --git a/docs/user_guide/quantization_usage.rst b/docs/user_guide/quantization_usage.rst index 2b2f150991c4450ffb92caaa6a9953d1a9b8b90c..0e4246db8a43939b24d8c382a2afa3aa6e48bbfd 100644 --- a/docs/user_guide/quantization_usage.rst +++ b/docs/user_guide/quantization_usage.rst @@ -29,29 +29,42 @@ MACE supports post-training quantization if you want to take a chance to quantiz This method requires developer to calculate tensor range of each activation layer statistically using sample inputs. MACE provides tools to do statistics with following steps: - 1. Convert original model to run on CPU host without obfuscation (by setting `target_abis` to `host`, `runtime` to `cpu`, and `obfuscate` to `0`, appending `:0` to `output_tensors` if missing in yaml config). - E.g., + 1. Convert original model to run on CPU host without obfuscation (by setting `target_abis` to `host`, `runtime` to `cpu`, + and `obfuscate` to `0`, appending `:0` to `output_tensors` if missing in yaml config). E.g., .. code:: sh python tools/converter.py convert --config ../mace-models/inception-v3/inception-v3.yml - 2. Log tensor range of each activation layer by inferring several samples. + 2. Log tensor range of each activation layer by inferring several samples on CPU host. Sample inputs should be + representative to calculate the ranges of each layer properly. .. code:: sh - python tools/converter.py run --config ../mace-models/inception-v3/inception-v3.yml --example --quantize_stat --input_dir samples > range_log + # convert images to input tensors for MACE + python tools/image/image_to_tensor.py --input /path/to/directory/of/input/images + --output_dir /path/to/directory/of/input/tensors --image_shape=299,299,3 + # rename input tensors to start with input tensor name + rename 's/^/input/' * - 3. Calculate overall range of each activation layer by specifying percentage cutoff. + # run with input tensors + python tools/converter.py run --config ../mace-models/inception-v3/inception-v3.yml --example + --quantize_stat --input_dir /path/to/directory/of/input/tensors > range_log + + + 3. Calculate overall range of each activation layer. You may specify `--percentile` or `--enhance` and `--enhance_ratio` + to try different ranges and see which is better. Experimentation shows that the default `percentile` and `enhance_ratio` + works fine for several common models. .. code:: sh - python mace/python/tools/quantization/quantize_stat.py --log_file range_log --percentile 5 > overall_range + python mace/python/tools/quantization/quantize_stat.py --log_file range_log > overall_range - 4. Convert quantized model (by setting `quantize` to `1` and `quantize_range_file` to the overall_range file path in yaml config). + 4. Convert quantized model (by setting `target_abis` to the final target abis, e.g., `armeabi-v7a`, + `quantize` to `1` and `quantize_range_file` to the overall_range file path in yaml config). .. note:: diff --git a/third_party/nnlib/libhexagon_controller.so b/third_party/nnlib/libhexagon_controller.so index ee742a36cd466845e97d50871ae749bf1ec97769..a5d25ee49ada8f23ce5314ce7e5dea9ac361036e 100755 Binary files a/third_party/nnlib/libhexagon_controller.so and b/third_party/nnlib/libhexagon_controller.so differ diff --git a/third_party/nnlib/v60/libhexagon_nn_skel.so b/third_party/nnlib/v60/libhexagon_nn_skel.so index 96645ccae17161cdb5aa4efbe5fa814cb4012c8b..b8e0c3b1685420df46a93b115cbba947ddba8b6d 100755 Binary files a/third_party/nnlib/v60/libhexagon_nn_skel.so and b/third_party/nnlib/v60/libhexagon_nn_skel.so differ diff --git a/third_party/nnlib/v66/libhexagon_nn_skel.so b/third_party/nnlib/v66/libhexagon_nn_skel.so index 4f86b5a87519d674372b8cf23150475a29969214..054455a8b81eaa2ef72311428727b29e230751f0 100755 Binary files a/third_party/nnlib/v66/libhexagon_nn_skel.so and b/third_party/nnlib/v66/libhexagon_nn_skel.so differ