From 6f222cc27cac91852d2e06681324ccbc9d23f539 Mon Sep 17 00:00:00 2001 From: Bin Li Date: Tue, 19 Nov 2019 15:47:45 +0800 Subject: [PATCH] Update docs for transformation after model loading --- docs/development/how_to_debug.rst | 8 +- docs/installation/env_requirement.rst | 4 +- docs/installation/manual_setup.rst | 8 +- docs/installation/using_docker.rst | 8 +- docs/user_guide/advanced_usage.rst | 99 +++++++++++++++++++----- docs/user_guide/advanced_usage_cmake.rst | 89 +++++++++++++++++---- docs/user_guide/basic_usage.rst | 28 +++---- docs/user_guide/basic_usage_cmake.rst | 16 ++-- docs/user_guide/benchmark.rst | 12 +-- docs/user_guide/quantization_usage.rst | 4 +- docs/zh/installation/env_requirement.rst | 2 +- docs/zh/user_guide/basic_usage.rst | 16 ++-- 12 files changed, 208 insertions(+), 86 deletions(-) diff --git a/docs/development/how_to_debug.rst b/docs/development/how_to_debug.rst index 4c4642c5..0ec3c338 100644 --- a/docs/development/how_to_debug.rst +++ b/docs/development/how_to_debug.rst @@ -39,7 +39,7 @@ You can validate it by specifying `--validate` option while running the model. MACE automatically validate these metrics by running models with synthetic inputs. If you want to specify input data to use, you can add an option in yaml config under 'subgraphs', e.g., -.. code:: yaml +.. code-block:: yaml models: mobilenet_v1: @@ -82,13 +82,13 @@ For CMake users, modify the cmake configuration located in ``tools/cmake/`` as ` For Bazel users, - .. code:: sh + .. code-block:: sh python tools/converter.py run --config=/path/to/config.yml --debug_mode For android, you can use `ndk-stack tools `__ to symbolize stack trace, e.g., - .. code:: sh + .. code-block:: sh adb logcat | $ANDROID_NDK_HOME/ndk-stack -sym /path/to/local/binary/directory/ @@ -140,7 +140,7 @@ Debug engine using GDB GDB can be used as the last resort, as it is powerful that it can trace stacks of your process. If you run models on android, things may be a little bit complicated. - .. code:: sh + .. code-block:: sh # push gdbserver to your phone adb push $ANDROID_NDK_HOME/prebuilt/android-arm64/gdbserver/gdbserver /data/local/tmp/ diff --git a/docs/installation/env_requirement.rst b/docs/installation/env_requirement.rst index e0a571fe..5fce3955 100644 --- a/docs/installation/env_requirement.rst +++ b/docs/installation/env_requirement.rst @@ -36,7 +36,7 @@ Required dependencies For Bazel, install it following installation guide. For python dependencies, - .. code:: sh + .. code-block:: sh pip install -U --user -r setup/requirements.txt @@ -78,7 +78,7 @@ Optional dependencies For python dependencies, - .. code:: sh + .. code-block:: sh pip install -U --user -r setup/optionals.txt diff --git a/docs/installation/manual_setup.rst b/docs/installation/manual_setup.rst index 0c564b76..beadee55 100644 --- a/docs/installation/manual_setup.rst +++ b/docs/installation/manual_setup.rst @@ -10,7 +10,7 @@ Install Bazel Recommend bazel with version larger than ``0.13.0`` (Refer to `Bazel documentation `__). -.. code:: sh +.. code-block:: sh export BAZEL_VERSION=0.13.1 mkdir /bazel && \ @@ -27,7 +27,7 @@ Install Android NDK The recommended Android NDK versions includes r15b, r15c and r16b (Refers to `NDK installation guide `__). -.. code:: sh +.. code-block:: sh # Download NDK r15c cd /opt/ && \ @@ -45,7 +45,7 @@ The recommended Android NDK versions includes r15b, r15c and r16b (Refers to Install extra tools -------------------- -.. code:: sh +.. code-block:: sh apt-get install -y --no-install-recommends \ cmake \ @@ -63,7 +63,7 @@ Install extra tools Install TensorFlow (Optional) ------------------------------ -.. code:: sh +.. code-block:: sh pip install -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com tensorflow==1.8.0 diff --git a/docs/installation/using_docker.rst b/docs/installation/using_docker.rst index bbfcf933..90104421 100644 --- a/docs/installation/using_docker.rst +++ b/docs/installation/using_docker.rst @@ -13,7 +13,7 @@ In most cases, the ``lite edition`` image can satisfy developer's basic needs. - ``lite edition`` docker image. -.. code:: sh +.. code-block:: sh # You can pull lite edition docker image from docker repo (recommended) docker pull registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev-lite @@ -22,7 +22,7 @@ In most cases, the ``lite edition`` image can satisfy developer's basic needs. - ``full edition`` docker image (which contains multiple NDK versions and other dev tools). -.. code:: sh +.. code-block:: sh # You can pull full edition docker image from docker repo (recommended) docker pull registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev @@ -39,7 +39,7 @@ Using the image Create container with the following command -.. code:: sh +.. code-block:: sh # Create a container named `mace-dev` docker run -it --privileged -d --name mace-dev \ @@ -58,7 +58,7 @@ Update images to repository If you are mace inner developer and need update images in remote repository, it can be achieved by `docker/update_images.sh` script. -.. code:: sh +.. code-block:: sh cd docker/ ./update_images.sh diff --git a/docs/user_guide/advanced_usage.rst b/docs/user_guide/advanced_usage.rst index b9eb7543..f2d96b89 100644 --- a/docs/user_guide/advanced_usage.rst +++ b/docs/user_guide/advanced_usage.rst @@ -106,7 +106,7 @@ in one deployment file. Some command tools: - .. code:: bash + .. code-block:: bash # Get device's soc info. adb shell getprop | grep platform @@ -129,7 +129,7 @@ Run you model on the embedded device(ARM Linux) The way to run your model on the ARM Linux is nearly same as with android, except you need specify a device config file. -.. code:: bash +.. code-block:: bash python tools/converter.py run --config=/path/to/your/model_deployment_file.yml --device_yml=/path/to/devices.yml @@ -139,7 +139,7 @@ There are two steps to do before run: MACE use ssh to connect embedded device, you should copy your public key to embedded device with the blow command. - .. code:: bash + .. code-block:: bash cat ~/.ssh/id_rsa.pub | ssh -q {user}@{ip} "cat >> ~/.ssh/authorized_keys" @@ -176,7 +176,7 @@ There are two steps to do before run: Convert model(s) to C++ code --------------------------------- +---------------------------- * **1. Change the model deployment file(.yml)** @@ -184,14 +184,14 @@ Convert model(s) to C++ code * convert model graph to code and model weight to file with below model configuration. - .. code:: sh + .. code-block:: sh model_graph_format: code model_data_format: file * convert both model graph and model weight to code with below model configuration. - .. code:: sh + .. code-block:: sh model_graph_format: code model_data_format: code @@ -202,14 +202,14 @@ Convert model(s) to C++ code * **2. Convert model(s) to code** - .. code:: sh + .. code-block:: sh python tools/converter.py convert --config=/path/to/model_deployment_file.yml The command will generate **${library_name}.a** in **build/${library_name}/model** directory and ** *.h ** in **build/${library_name}/include** like the following dir-tree. - .. code:: + .. code-block:: none # model_graph_format: code # model_data_format: file @@ -240,7 +240,7 @@ Convert model(s) to C++ code * Link `libmace.a` and `${library_name}.a` to your target. * Refer to \ ``mace/tools/mace_run.cc``\ for full usage. The following list the key steps. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -269,9 +269,70 @@ Convert model(s) to C++ code // ... Same with the code in basic usage -Tuning for specific SoC's GPU +Transform models after conversion --------------------------------- + If ``model_graph_format`` or ``model_data_format`` is specified as `file`, the model or weight file will + be generated as a `.pb` or `.data` file after model conversion. After that, more transformations can be + applied to the generated files, such as compression or encryption. To achieve that, the model loading is + split to two stages: 1) load the file from file system to memory buffer; 2) create the MACE engine from the + model buffer. So between the two stages, transformations can be inserted to decompress or decrypt the model + buffer. The transformations are user defined. The following lists the key steps when both ``model_graph_format`` + and ``model_data_format`` are set as `file`. + + .. code-block:: cpp + + // Load model graph from file system + std::unique_ptr model_graph_data = + make_unique(); + if (FLAGS_model_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_file.c_str(), + &model_graph_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + // Load model data from file system + std::unique_ptr model_weights_data = + make_unique(); + if (FLAGS_model_data_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_data_file.c_str(), + &model_weights_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + if (model_graph_data == nullptr || model_weights_data == nullptr) { + // Report error or fallback + } + + // Add transformations here. + ... + + // Create the MACE engine from the model buffer + std::shared_ptr engine; + MaceStatus create_engine_status; + create_engine_status = + CreateMaceEngineFromProto(reinterpret_cast( + model_graph_data->data()), + model_graph_data->length(), + reinterpret_cast( + model_weights_data->data()), + model_weights_data->length(), + input_names, + output_names, + config, + &engine); + if (create_engine_status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + + +Tuning for specific SoC's GPU +----------------------------- + If you want to use the GPU of a specific device, you can just specify the ``target_socs`` in your YAML file and then tune the MACE lib for it (OpenCL kernels), which may get 1~10% performance improvement. @@ -279,7 +340,7 @@ Tuning for specific SoC's GPU Specify ``target_socs`` in your model deployment file(.yml): - .. code:: sh + .. code-block:: sh target_socs: [sdm845] @@ -289,7 +350,7 @@ Tuning for specific SoC's GPU * **2. Convert model(s)** - .. code:: sh + .. code-block:: sh python tools/converter.py convert --config=/path/to/model_deployment_file.yml @@ -303,13 +364,13 @@ Tuning for specific SoC's GPU You should plug in device(s) with the specific SoC(s). - .. code:: sh + .. code-block:: sh python tools/converter.py run --config=/path/to/model_deployment_file.yml --validate The command will generate two files in `build/${library_name}/opencl`, like the following dir-tree. - .. code:: + .. code-block:: none build └── mobilenet-v2 @@ -336,7 +397,7 @@ Tuning for specific SoC's GPU * Change the names of files generated above for not collision and push them to **your own device's directory**. * Use like the previous procedure, below lists the key steps differently. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -379,7 +440,7 @@ Useful Commands --------------- * **run the model** -.. code:: sh +.. code-block:: sh # Test model run time python tools/converter.py run --config=/path/to/model_deployment_file.yml --round=100 @@ -403,7 +464,7 @@ Useful Commands the detailed information is in :doc:`benchmark`. -.. code:: sh +.. code-block:: sh # Benchmark model, get detailed statistics of each Op. python tools/converter.py run --config=/path/to/model_deployment_file.yml --benchmark @@ -446,7 +507,7 @@ the detailed information is in :doc:`benchmark`. Use ``-h`` to get detailed help. -.. code:: sh +.. code-block:: sh python tools/converter.py -h python tools/converter.py build -h @@ -480,7 +541,7 @@ Reduce Library Size Remove the registration of the ops unused for your models in the ``mace/ops/ops_register.cc``, which will reduce the library size significantly. the final binary just link the registered ops' code. -.. code:: cpp +.. code-block:: cpp #include "mace/ops/ops_register.h" diff --git a/docs/user_guide/advanced_usage_cmake.rst b/docs/user_guide/advanced_usage_cmake.rst index 8133c098..842fab3e 100644 --- a/docs/user_guide/advanced_usage_cmake.rst +++ b/docs/user_guide/advanced_usage_cmake.rst @@ -84,7 +84,7 @@ There are many advanced options supported. Some command tools: - .. code:: bash + .. code-block:: bash # Get device's soc info. adb shell getprop | grep platform @@ -107,7 +107,7 @@ Run you model on the embedded device(ARM Linux) The way to run your model on the ARM Linux is nearly same as with android, except you need specify a device config file. - .. code:: bash + .. code-block:: bash python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --validate --device_yml=/path/to/devices.yml @@ -117,7 +117,7 @@ There are two steps to do before run: MACE use ssh to connect embedded device, you should copy your public key to embedded device with the blow command. - .. code:: bash + .. code-block:: bash cat ~/.ssh/id_rsa.pub | ssh -q {user}@{ip} "cat >> ~/.ssh/authorized_keys" @@ -158,14 +158,14 @@ Model Protection Model can be encrypted by obfuscation. - .. code:: bash + .. code-block:: bash python tools/python/encrypt.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml It will override ``mobilenet_v1.pb`` and ``mobilenet_v1.data``. If you want to compiled the model into a library, you should use options ``--gencode_model --gencode_param`` to generate model code, i.e., - .. code:: bash + .. code-block:: bash python tools/python/encrypt.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --gencode_model --gencode_param @@ -173,7 +173,7 @@ It will generate model code into ``mace/codegen/models`` and also generate a hel After that you can rebuild the engine. - .. code:: bash + .. code-block:: bash RUNTIME=GPU RUNMODE=code bash tools/cmake/cmake-build-armeabi-v7a.sh @@ -181,7 +181,7 @@ After that you can rebuild the engine. When you test the model in code format, you should specify it in the script as follows. - .. code:: bash + .. code-block:: bash python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --gencode_model --gencode_param @@ -192,7 +192,7 @@ When you need to integrate the libraries into your applications, you can link `l Refer to \ ``mace/tools/mace_run.cc``\ for full usage. The following list the key steps. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -221,6 +221,67 @@ Refer to \ ``mace/tools/mace_run.cc``\ for full usage. The following list the ke // ... Same with the code in basic usage +Transform models after conversion +--------------------------------- + +If ``model_graph_format`` or ``model_data_format`` is specified as `file`, the model or weight file will +be generated as a `.pb` or `.data` file after model conversion. After that, more transformations can be +applied to the generated files, such as compression or encryption. To achieve that, the model loading is +split to two stages: 1) load the file from file system to memory buffer; 2) create the MACE engine from the +model buffer. So between the two stages, transformations can be inserted to decompress or decrypt the model +buffer. The transformations are user defined. The following lists the key steps when both ``model_graph_format`` +and ``model_data_format`` are set as `file`. + + .. code-block:: cpp + + // Load model graph from file system + std::unique_ptr model_graph_data = + make_unique(); + if (FLAGS_model_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_file.c_str(), + &model_graph_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + // Load model data from file system + std::unique_ptr model_weights_data = + make_unique(); + if (FLAGS_model_data_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_data_file.c_str(), + &model_weights_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + if (model_graph_data == nullptr || model_weights_data == nullptr) { + // Report error or fallback + } + + // Add transformations here. + ... + + // Create the MACE engine from the model buffer + std::shared_ptr engine; + MaceStatus create_engine_status; + create_engine_status = + CreateMaceEngineFromProto(reinterpret_cast( + model_graph_data->data()), + model_graph_data->length(), + reinterpret_cast( + model_weights_data->data()), + model_weights_data->length(), + input_names, + output_names, + config, + &engine); + if (create_engine_status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + + Tuning for specific SoC's GPU --------------------------------- @@ -228,13 +289,13 @@ If you want to use the GPU of a specific device, you can tune the performance fo You can specify `--tune` option when you want to run and tune the performance at the same time. - .. code:: bash + .. code-block:: bash python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --tune It will generate OpenCL tuned parameter binary file in `build/mobilenet_v1/opencl` directory. - .. code:: bash + .. code-block:: none └── mobilenet_v1_tuned_opencl_parameter.MIX2S.sdm845.bin @@ -243,7 +304,7 @@ It specifies your test platform model and SoC. You can use it in production to r To deploy it, change the names of files generated above for not collision and push them to **your own device's directory**. Use like the previous procedure, below lists the key steps differently. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -275,13 +336,13 @@ Multi Model Support (optional) If multiple models are configured in config file. After you test it, it will generate more than one tuned parameter files. Then you need to merge them together. - .. code:: bash + .. code-block:: bash python tools/python/gen_opencl.py After that, it will generate one set of files into `build/opencl` directory. - .. code:: bash + .. code-block:: none ├── compiled_opencl_kernel.bin └── tuned_opencl_parameter.bin @@ -309,7 +370,7 @@ Reduce Library Size Remove the registration of the ops unused for your models in the ``mace/ops/ops_register.cc``, which will reduce the library size significantly. the final binary just link the registered ops' code. -.. code:: cpp +.. code-block:: cpp #include "mace/ops/ops_register.h" diff --git a/docs/user_guide/basic_usage.rst b/docs/user_guide/basic_usage.rst index 2d5208d0..5aaaf4fd 100644 --- a/docs/user_guide/basic_usage.rst +++ b/docs/user_guide/basic_usage.rst @@ -16,7 +16,7 @@ Here we use the mobilenet-v2 model as an example. 1. Pull `MACE `__ project. - .. code:: sh + .. code-block:: sh git clone https://github.com/XiaoMi/mace.git cd mace/ @@ -33,14 +33,14 @@ Here we use the mobilenet-v2 model as an example. 2. Pull `MACE Model Zoo `__ project. - .. code:: sh + .. code-block:: sh git clone https://github.com/XiaoMi/mace-models.git 3. Build a generic MACE library. - .. code:: sh + .. code-block:: sh cd path/to/mace # Build library @@ -59,7 +59,7 @@ Here we use the mobilenet-v2 model as an example. 4. Convert the pre-trained mobilenet-v2 model to MACE format model. - .. code:: sh + .. code-block:: sh cd path/to/mace # Build library @@ -73,7 +73,7 @@ Here we use the mobilenet-v2 model as an example. If you want to run on phone, please plug in at least one phone. Or if you want to run on embedded device, please give a :doc:`advanced_usage`. - .. code:: sh + .. code-block:: sh # Run python tools/converter.py run --config=/path/to/mace-models/mobilenet-v2/mobilenet-v2.yml @@ -107,7 +107,7 @@ MACE now supports models from TensorFlow and Caffe (more frameworks will be supp If your model is from lower version Caffe, you need to upgrade it by using the Caffe built-in tool before converting. - .. code:: bash + .. code-block:: bash # Upgrade prototxt $CAFFE_ROOT/build/tools/upgrade_net_proto_text MODEL.prototxt MODEL.new.prototxt @@ -123,7 +123,7 @@ MACE now supports models from TensorFlow and Caffe (more frameworks will be supp This tool will improve the efficiency of inference like the `Graph Transform Tool `__ in TensorFlow. - .. code:: bash + .. code-block:: bash # Optimize your model $python MACE_ROOT/tools/onnx_optimizer.py model.onnx model_opt.onnx @@ -165,7 +165,7 @@ More details about model deployment file are in :doc:`advanced_usage`. When the deployment file is ready, you can use MACE converter tool to convert your model(s). -.. code:: bash +.. code-block:: bash python tools/converter.py convert --config=/path/to/your/model_deployment_file.yml @@ -184,7 +184,7 @@ You could Download the prebuilt MACE Library from `Github MACE release page overall_range diff --git a/docs/zh/installation/env_requirement.rst b/docs/zh/installation/env_requirement.rst index b7e6cbec..600483d8 100644 --- a/docs/zh/installation/env_requirement.rst +++ b/docs/zh/installation/env_requirement.rst @@ -68,7 +68,7 @@ MACE 需要安装下列依赖： 对于 Python 依赖，可直接执行, - .. code:: sh + .. code-block:: sh pip install -U --user -r setup/optionals.txt diff --git a/docs/zh/user_guide/basic_usage.rst b/docs/zh/user_guide/basic_usage.rst index b5b65226..3fd04b0b 100644 --- a/docs/zh/user_guide/basic_usage.rst +++ b/docs/zh/user_guide/basic_usage.rst @@ -8,7 +8,7 @@ 构建前，清空工作目录 - .. code:: sh + .. code-block:: sh tools/clear_workspace.sh @@ -18,7 +18,7 @@ 确保 CMake 已安装 - .. code:: sh + .. code-block:: sh RUNTIME=GPU bash tools/cmake/cmake-build-armeabi-v7a.sh @@ -33,7 +33,7 @@ 撰写模型相关的 YAML 配置文件： - .. code:: yaml + .. code-block:: yaml models: mobilenet_v1: @@ -55,13 +55,13 @@ 假设模型配置文件的路径是： ``../mace-models/mobilenet-v1/mobilenet-v1.yml``，执行： - .. code:: yaml + .. code-block:: yaml python tools/python/convert.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml 将会在 ``build/mobilenet_v1/model/`` 中产生 4 个文件 - .. code:: sh + .. code-block:: none ├── mobilenet_v1.pb (模型结构文件) ├── mobilenet_v1.data (模型参数文件) @@ -80,13 +80,13 @@ 模型转换后，执行下面命令进行测试： - .. code:: sh + .. code-block:: sh python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --validate 或下面命令进行性能评测： - .. code:: sh + .. code-block:: sh python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --benchmark @@ -100,7 +100,7 @@ 可以查看源码 \ ``mace/tools/mace_run.cc``\ 了解更多详情。下面简要介绍相关步骤： -.. code:: cpp +.. code-block:: cpp // 添加头文件按 #include "mace/public/mace.h" -- GitLab