diff --git a/docs/development/how_to_debug.rst b/docs/development/how_to_debug.rst index 4c4642c593a00b0da1e964f41e6dbde2a3751117..0ec3c338a77f3eaf56713c3d9aa05c22df1c95b8 100644 --- a/docs/development/how_to_debug.rst +++ b/docs/development/how_to_debug.rst @@ -39,7 +39,7 @@ You can validate it by specifying `--validate` option while running the model. MACE automatically validate these metrics by running models with synthetic inputs. If you want to specify input data to use, you can add an option in yaml config under 'subgraphs', e.g., -.. code:: yaml +.. code-block:: yaml models: mobilenet_v1: @@ -82,13 +82,13 @@ For CMake users, modify the cmake configuration located in ``tools/cmake/`` as ` For Bazel users, - .. code:: sh + .. code-block:: sh python tools/converter.py run --config=/path/to/config.yml --debug_mode For android, you can use `ndk-stack tools `__ to symbolize stack trace, e.g., - .. code:: sh + .. code-block:: sh adb logcat | $ANDROID_NDK_HOME/ndk-stack -sym /path/to/local/binary/directory/ @@ -140,7 +140,7 @@ Debug engine using GDB GDB can be used as the last resort, as it is powerful that it can trace stacks of your process. If you run models on android, things may be a little bit complicated. - .. code:: sh + .. code-block:: sh # push gdbserver to your phone adb push $ANDROID_NDK_HOME/prebuilt/android-arm64/gdbserver/gdbserver /data/local/tmp/ diff --git a/docs/installation/env_requirement.rst b/docs/installation/env_requirement.rst index e0a571fe1d00c6e3ffd8ea1f54528fd1a7bee697..5fce3955b4dec97144407a3e0ca4a6395bb872ce 100644 --- a/docs/installation/env_requirement.rst +++ b/docs/installation/env_requirement.rst @@ -36,7 +36,7 @@ Required dependencies For Bazel, install it following installation guide. For python dependencies, - .. code:: sh + .. code-block:: sh pip install -U --user -r setup/requirements.txt @@ -78,7 +78,7 @@ Optional dependencies For python dependencies, - .. code:: sh + .. code-block:: sh pip install -U --user -r setup/optionals.txt diff --git a/docs/installation/manual_setup.rst b/docs/installation/manual_setup.rst index 0c564b76ae90af1a6e2318da432e7d31c87602dd..beadee5539a6e56dea5a1bfd155e72e4b80c3481 100644 --- a/docs/installation/manual_setup.rst +++ b/docs/installation/manual_setup.rst @@ -10,7 +10,7 @@ Install Bazel Recommend bazel with version larger than ``0.13.0`` (Refer to `Bazel documentation `__). -.. code:: sh +.. code-block:: sh export BAZEL_VERSION=0.13.1 mkdir /bazel && \ @@ -27,7 +27,7 @@ Install Android NDK The recommended Android NDK versions includes r15b, r15c and r16b (Refers to `NDK installation guide `__). -.. code:: sh +.. code-block:: sh # Download NDK r15c cd /opt/ && \ @@ -45,7 +45,7 @@ The recommended Android NDK versions includes r15b, r15c and r16b (Refers to Install extra tools -------------------- -.. code:: sh +.. code-block:: sh apt-get install -y --no-install-recommends \ cmake \ @@ -63,7 +63,7 @@ Install extra tools Install TensorFlow (Optional) ------------------------------ -.. code:: sh +.. code-block:: sh pip install -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com tensorflow==1.8.0 diff --git a/docs/installation/using_docker.rst b/docs/installation/using_docker.rst index bbfcf9336548cdea0bc09fcb9f2d8cbd79ff2a8e..901044214360585c79b06689908c76dc8463fcab 100644 --- a/docs/installation/using_docker.rst +++ b/docs/installation/using_docker.rst @@ -13,7 +13,7 @@ In most cases, the ``lite edition`` image can satisfy developer's basic needs. - ``lite edition`` docker image. -.. code:: sh +.. code-block:: sh # You can pull lite edition docker image from docker repo (recommended) docker pull registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev-lite @@ -22,7 +22,7 @@ In most cases, the ``lite edition`` image can satisfy developer's basic needs. - ``full edition`` docker image (which contains multiple NDK versions and other dev tools). -.. code:: sh +.. code-block:: sh # You can pull full edition docker image from docker repo (recommended) docker pull registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev @@ -39,7 +39,7 @@ Using the image Create container with the following command -.. code:: sh +.. code-block:: sh # Create a container named `mace-dev` docker run -it --privileged -d --name mace-dev \ @@ -58,7 +58,7 @@ Update images to repository If you are mace inner developer and need update images in remote repository, it can be achieved by `docker/update_images.sh` script. -.. code:: sh +.. code-block:: sh cd docker/ ./update_images.sh diff --git a/docs/user_guide/advanced_usage.rst b/docs/user_guide/advanced_usage.rst index b9eb754349be4fa4157560df2c7cb3eab236bfee..f2d96b8970c0970b3e79fae47600345d3443f49c 100644 --- a/docs/user_guide/advanced_usage.rst +++ b/docs/user_guide/advanced_usage.rst @@ -106,7 +106,7 @@ in one deployment file. Some command tools: - .. code:: bash + .. code-block:: bash # Get device's soc info. adb shell getprop | grep platform @@ -129,7 +129,7 @@ Run you model on the embedded device(ARM Linux) The way to run your model on the ARM Linux is nearly same as with android, except you need specify a device config file. -.. code:: bash +.. code-block:: bash python tools/converter.py run --config=/path/to/your/model_deployment_file.yml --device_yml=/path/to/devices.yml @@ -139,7 +139,7 @@ There are two steps to do before run: MACE use ssh to connect embedded device, you should copy your public key to embedded device with the blow command. - .. code:: bash + .. code-block:: bash cat ~/.ssh/id_rsa.pub | ssh -q {user}@{ip} "cat >> ~/.ssh/authorized_keys" @@ -176,7 +176,7 @@ There are two steps to do before run: Convert model(s) to C++ code --------------------------------- +---------------------------- * **1. Change the model deployment file(.yml)** @@ -184,14 +184,14 @@ Convert model(s) to C++ code * convert model graph to code and model weight to file with below model configuration. - .. code:: sh + .. code-block:: sh model_graph_format: code model_data_format: file * convert both model graph and model weight to code with below model configuration. - .. code:: sh + .. code-block:: sh model_graph_format: code model_data_format: code @@ -202,14 +202,14 @@ Convert model(s) to C++ code * **2. Convert model(s) to code** - .. code:: sh + .. code-block:: sh python tools/converter.py convert --config=/path/to/model_deployment_file.yml The command will generate **${library_name}.a** in **build/${library_name}/model** directory and ** *.h ** in **build/${library_name}/include** like the following dir-tree. - .. code:: + .. code-block:: none # model_graph_format: code # model_data_format: file @@ -240,7 +240,7 @@ Convert model(s) to C++ code * Link `libmace.a` and `${library_name}.a` to your target. * Refer to \ ``mace/tools/mace_run.cc``\ for full usage. The following list the key steps. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -269,9 +269,70 @@ Convert model(s) to C++ code // ... Same with the code in basic usage -Tuning for specific SoC's GPU +Transform models after conversion --------------------------------- + If ``model_graph_format`` or ``model_data_format`` is specified as `file`, the model or weight file will + be generated as a `.pb` or `.data` file after model conversion. After that, more transformations can be + applied to the generated files, such as compression or encryption. To achieve that, the model loading is + split to two stages: 1) load the file from file system to memory buffer; 2) create the MACE engine from the + model buffer. So between the two stages, transformations can be inserted to decompress or decrypt the model + buffer. The transformations are user defined. The following lists the key steps when both ``model_graph_format`` + and ``model_data_format`` are set as `file`. + + .. code-block:: cpp + + // Load model graph from file system + std::unique_ptr model_graph_data = + make_unique(); + if (FLAGS_model_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_file.c_str(), + &model_graph_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + // Load model data from file system + std::unique_ptr model_weights_data = + make_unique(); + if (FLAGS_model_data_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_data_file.c_str(), + &model_weights_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + if (model_graph_data == nullptr || model_weights_data == nullptr) { + // Report error or fallback + } + + // Add transformations here. + ... + + // Create the MACE engine from the model buffer + std::shared_ptr engine; + MaceStatus create_engine_status; + create_engine_status = + CreateMaceEngineFromProto(reinterpret_cast( + model_graph_data->data()), + model_graph_data->length(), + reinterpret_cast( + model_weights_data->data()), + model_weights_data->length(), + input_names, + output_names, + config, + &engine); + if (create_engine_status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + + +Tuning for specific SoC's GPU +----------------------------- + If you want to use the GPU of a specific device, you can just specify the ``target_socs`` in your YAML file and then tune the MACE lib for it (OpenCL kernels), which may get 1~10% performance improvement. @@ -279,7 +340,7 @@ Tuning for specific SoC's GPU Specify ``target_socs`` in your model deployment file(.yml): - .. code:: sh + .. code-block:: sh target_socs: [sdm845] @@ -289,7 +350,7 @@ Tuning for specific SoC's GPU * **2. Convert model(s)** - .. code:: sh + .. code-block:: sh python tools/converter.py convert --config=/path/to/model_deployment_file.yml @@ -303,13 +364,13 @@ Tuning for specific SoC's GPU You should plug in device(s) with the specific SoC(s). - .. code:: sh + .. code-block:: sh python tools/converter.py run --config=/path/to/model_deployment_file.yml --validate The command will generate two files in `build/${library_name}/opencl`, like the following dir-tree. - .. code:: + .. code-block:: none build └── mobilenet-v2 @@ -336,7 +397,7 @@ Tuning for specific SoC's GPU * Change the names of files generated above for not collision and push them to **your own device's directory**. * Use like the previous procedure, below lists the key steps differently. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -379,7 +440,7 @@ Useful Commands --------------- * **run the model** -.. code:: sh +.. code-block:: sh # Test model run time python tools/converter.py run --config=/path/to/model_deployment_file.yml --round=100 @@ -403,7 +464,7 @@ Useful Commands the detailed information is in :doc:`benchmark`. -.. code:: sh +.. code-block:: sh # Benchmark model, get detailed statistics of each Op. python tools/converter.py run --config=/path/to/model_deployment_file.yml --benchmark @@ -446,7 +507,7 @@ the detailed information is in :doc:`benchmark`. Use ``-h`` to get detailed help. -.. code:: sh +.. code-block:: sh python tools/converter.py -h python tools/converter.py build -h @@ -480,7 +541,7 @@ Reduce Library Size Remove the registration of the ops unused for your models in the ``mace/ops/ops_register.cc``, which will reduce the library size significantly. the final binary just link the registered ops' code. -.. code:: cpp +.. code-block:: cpp #include "mace/ops/ops_register.h" diff --git a/docs/user_guide/advanced_usage_cmake.rst b/docs/user_guide/advanced_usage_cmake.rst index 8133c098d9d256550b158a2519dd1079efce3787..842fab3e94d241f86bfae57cfb45f28b43db8b15 100644 --- a/docs/user_guide/advanced_usage_cmake.rst +++ b/docs/user_guide/advanced_usage_cmake.rst @@ -84,7 +84,7 @@ There are many advanced options supported. Some command tools: - .. code:: bash + .. code-block:: bash # Get device's soc info. adb shell getprop | grep platform @@ -107,7 +107,7 @@ Run you model on the embedded device(ARM Linux) The way to run your model on the ARM Linux is nearly same as with android, except you need specify a device config file. - .. code:: bash + .. code-block:: bash python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --validate --device_yml=/path/to/devices.yml @@ -117,7 +117,7 @@ There are two steps to do before run: MACE use ssh to connect embedded device, you should copy your public key to embedded device with the blow command. - .. code:: bash + .. code-block:: bash cat ~/.ssh/id_rsa.pub | ssh -q {user}@{ip} "cat >> ~/.ssh/authorized_keys" @@ -158,14 +158,14 @@ Model Protection Model can be encrypted by obfuscation. - .. code:: bash + .. code-block:: bash python tools/python/encrypt.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml It will override ``mobilenet_v1.pb`` and ``mobilenet_v1.data``. If you want to compiled the model into a library, you should use options ``--gencode_model --gencode_param`` to generate model code, i.e., - .. code:: bash + .. code-block:: bash python tools/python/encrypt.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --gencode_model --gencode_param @@ -173,7 +173,7 @@ It will generate model code into ``mace/codegen/models`` and also generate a hel After that you can rebuild the engine. - .. code:: bash + .. code-block:: bash RUNTIME=GPU RUNMODE=code bash tools/cmake/cmake-build-armeabi-v7a.sh @@ -181,7 +181,7 @@ After that you can rebuild the engine. When you test the model in code format, you should specify it in the script as follows. - .. code:: bash + .. code-block:: bash python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --gencode_model --gencode_param @@ -192,7 +192,7 @@ When you need to integrate the libraries into your applications, you can link `l Refer to \ ``mace/tools/mace_run.cc``\ for full usage. The following list the key steps. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -221,6 +221,67 @@ Refer to \ ``mace/tools/mace_run.cc``\ for full usage. The following list the ke // ... Same with the code in basic usage +Transform models after conversion +--------------------------------- + +If ``model_graph_format`` or ``model_data_format`` is specified as `file`, the model or weight file will +be generated as a `.pb` or `.data` file after model conversion. After that, more transformations can be +applied to the generated files, such as compression or encryption. To achieve that, the model loading is +split to two stages: 1) load the file from file system to memory buffer; 2) create the MACE engine from the +model buffer. So between the two stages, transformations can be inserted to decompress or decrypt the model +buffer. The transformations are user defined. The following lists the key steps when both ``model_graph_format`` +and ``model_data_format`` are set as `file`. + + .. code-block:: cpp + + // Load model graph from file system + std::unique_ptr model_graph_data = + make_unique(); + if (FLAGS_model_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_file.c_str(), + &model_graph_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + // Load model data from file system + std::unique_ptr model_weights_data = + make_unique(); + if (FLAGS_model_data_file != "") { + auto fs = GetFileSystem(); + status = fs->NewReadOnlyMemoryRegionFromFile(FLAGS_model_data_file.c_str(), + &model_weights_data); + if (status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + } + if (model_graph_data == nullptr || model_weights_data == nullptr) { + // Report error or fallback + } + + // Add transformations here. + ... + + // Create the MACE engine from the model buffer + std::shared_ptr engine; + MaceStatus create_engine_status; + create_engine_status = + CreateMaceEngineFromProto(reinterpret_cast( + model_graph_data->data()), + model_graph_data->length(), + reinterpret_cast( + model_weights_data->data()), + model_weights_data->length(), + input_names, + output_names, + config, + &engine); + if (create_engine_status != MaceStatus::MACE_SUCCESS) { + // Report error or fallback + } + + Tuning for specific SoC's GPU --------------------------------- @@ -228,13 +289,13 @@ If you want to use the GPU of a specific device, you can tune the performance fo You can specify `--tune` option when you want to run and tune the performance at the same time. - .. code:: bash + .. code-block:: bash python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --tune It will generate OpenCL tuned parameter binary file in `build/mobilenet_v1/opencl` directory. - .. code:: bash + .. code-block:: none └── mobilenet_v1_tuned_opencl_parameter.MIX2S.sdm845.bin @@ -243,7 +304,7 @@ It specifies your test platform model and SoC. You can use it in production to r To deploy it, change the names of files generated above for not collision and push them to **your own device's directory**. Use like the previous procedure, below lists the key steps differently. - .. code:: cpp + .. code-block:: cpp // Include the headers #include "mace/public/mace.h" @@ -275,13 +336,13 @@ Multi Model Support (optional) If multiple models are configured in config file. After you test it, it will generate more than one tuned parameter files. Then you need to merge them together. - .. code:: bash + .. code-block:: bash python tools/python/gen_opencl.py After that, it will generate one set of files into `build/opencl` directory. - .. code:: bash + .. code-block:: none ├── compiled_opencl_kernel.bin └── tuned_opencl_parameter.bin @@ -309,7 +370,7 @@ Reduce Library Size Remove the registration of the ops unused for your models in the ``mace/ops/ops_register.cc``, which will reduce the library size significantly. the final binary just link the registered ops' code. -.. code:: cpp +.. code-block:: cpp #include "mace/ops/ops_register.h" diff --git a/docs/user_guide/basic_usage.rst b/docs/user_guide/basic_usage.rst index 2d5208d07884014b2743d3ba70892536412ee6e3..5aaaf4fd860c4e266855ca09ea7b7cb742776187 100644 --- a/docs/user_guide/basic_usage.rst +++ b/docs/user_guide/basic_usage.rst @@ -16,7 +16,7 @@ Here we use the mobilenet-v2 model as an example. 1. Pull `MACE `__ project. - .. code:: sh + .. code-block:: sh git clone https://github.com/XiaoMi/mace.git cd mace/ @@ -33,14 +33,14 @@ Here we use the mobilenet-v2 model as an example. 2. Pull `MACE Model Zoo `__ project. - .. code:: sh + .. code-block:: sh git clone https://github.com/XiaoMi/mace-models.git 3. Build a generic MACE library. - .. code:: sh + .. code-block:: sh cd path/to/mace # Build library @@ -59,7 +59,7 @@ Here we use the mobilenet-v2 model as an example. 4. Convert the pre-trained mobilenet-v2 model to MACE format model. - .. code:: sh + .. code-block:: sh cd path/to/mace # Build library @@ -73,7 +73,7 @@ Here we use the mobilenet-v2 model as an example. If you want to run on phone, please plug in at least one phone. Or if you want to run on embedded device, please give a :doc:`advanced_usage`. - .. code:: sh + .. code-block:: sh # Run python tools/converter.py run --config=/path/to/mace-models/mobilenet-v2/mobilenet-v2.yml @@ -107,7 +107,7 @@ MACE now supports models from TensorFlow and Caffe (more frameworks will be supp If your model is from lower version Caffe, you need to upgrade it by using the Caffe built-in tool before converting. - .. code:: bash + .. code-block:: bash # Upgrade prototxt $CAFFE_ROOT/build/tools/upgrade_net_proto_text MODEL.prototxt MODEL.new.prototxt @@ -123,7 +123,7 @@ MACE now supports models from TensorFlow and Caffe (more frameworks will be supp This tool will improve the efficiency of inference like the `Graph Transform Tool `__ in TensorFlow. - .. code:: bash + .. code-block:: bash # Optimize your model $python MACE_ROOT/tools/onnx_optimizer.py model.onnx model_opt.onnx @@ -165,7 +165,7 @@ More details about model deployment file are in :doc:`advanced_usage`. When the deployment file is ready, you can use MACE converter tool to convert your model(s). -.. code:: bash +.. code-block:: bash python tools/converter.py convert --config=/path/to/your/model_deployment_file.yml @@ -184,7 +184,7 @@ You could Download the prebuilt MACE Library from `Github MACE release page overall_range diff --git a/docs/zh/installation/env_requirement.rst b/docs/zh/installation/env_requirement.rst index b7e6cbecc41b5bdd8a05a8075ea9047d0602a4cc..600483d8e7acf624ddcc58b39e3bb8722d9d0b9b 100644 --- a/docs/zh/installation/env_requirement.rst +++ b/docs/zh/installation/env_requirement.rst @@ -68,7 +68,7 @@ MACE 需要安装下列依赖: 对于 Python 依赖,可直接执行, - .. code:: sh + .. code-block:: sh pip install -U --user -r setup/optionals.txt diff --git a/docs/zh/user_guide/basic_usage.rst b/docs/zh/user_guide/basic_usage.rst index b5b65226f908a25c184e609605dc7ea07bd2d120..3fd04b0b4e281847007e83ee0b02ad0ff3b3866c 100644 --- a/docs/zh/user_guide/basic_usage.rst +++ b/docs/zh/user_guide/basic_usage.rst @@ -8,7 +8,7 @@ 构建前,清空工作目录 - .. code:: sh + .. code-block:: sh tools/clear_workspace.sh @@ -18,7 +18,7 @@ 确保 CMake 已安装 - .. code:: sh + .. code-block:: sh RUNTIME=GPU bash tools/cmake/cmake-build-armeabi-v7a.sh @@ -33,7 +33,7 @@ 撰写模型相关的 YAML 配置文件: - .. code:: yaml + .. code-block:: yaml models: mobilenet_v1: @@ -55,13 +55,13 @@ 假设模型配置文件的路径是: ``../mace-models/mobilenet-v1/mobilenet-v1.yml``,执行: - .. code:: yaml + .. code-block:: yaml python tools/python/convert.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml 将会在 ``build/mobilenet_v1/model/`` 中产生 4 个文件 - .. code:: sh + .. code-block:: none ├── mobilenet_v1.pb (模型结构文件) ├── mobilenet_v1.data (模型参数文件) @@ -80,13 +80,13 @@ 模型转换后,执行下面命令进行测试: - .. code:: sh + .. code-block:: sh python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --validate 或下面命令进行性能评测: - .. code:: sh + .. code-block:: sh python tools/python/run_model.py --config ../mace-models/mobilenet-v1/mobilenet-v1.yml --benchmark @@ -100,7 +100,7 @@ 可以查看源码 \ ``mace/tools/mace_run.cc``\ 了解更多详情。下面简要介绍相关步骤: -.. code:: cpp +.. code-block:: cpp // 添加头文件按 #include "mace/public/mace.h"