diff --git a/docs/conf.py b/docs/conf.py index aff476bd04532e7132acb5c5857a5743122df5a9..bbd145890a0d56f24ac4a46a8534cbff6c3b2547 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -25,3 +25,10 @@ html_theme = "sphinx_rtd_theme" html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] html_static_path = ['_static'] smartquotes = False + +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.viewcode', + 'sphinx.ext.todo', + 'sphinx.ext.mathjax', +] diff --git a/docs/development/how_to_debug.rst b/docs/development/how_to_debug.rst index 1026f3ff293ac4b46c9f67baa3471b3dd849102e..b0e2131fa6a6d48b69b444e9578f8a9e81bbba93 100644 --- a/docs/development/how_to_debug.rst +++ b/docs/development/how_to_debug.rst @@ -1,7 +1,86 @@ How to debug -============== +========================== -Log debug info +Debug correctness +-------------------------- + +MACE provides tools to examine correctness of model execution by comparing model's output of MACE with output of training platform (e.g., Tensorflow, Caffe). +Three metrics are used as comparison results: + +* **Cosine Similarity**: + +.. math:: + + Cosine\ Similarity = \frac{X \cdot X'}{\|X\| \|X'\|} + +This metric will be approximately equal to 1 if output is correct. + +* **SQNR** (Signal-to-Quantization-Noise Ratio): + +.. math:: + + SQNR = \frac{P_{signal}}{P_{noise}} = \frac{\|X\|^2}{\|X - X'\|^2} + +It is usually used to measure quantization accuracy. The higher SQNR is, the better accuracy will be. + +* **Pixel Accuracy**: + +.. math:: + + Pixel\ Accuracy = \frac{\sum^{batch}_{b=1} equal(\mathrm{argmax} X_b, \mathrm{argmax} X'_b)}{batch} + +It is usually used to measure classification accuracy. The higher the better. + +where :math:`X` is expected output (from training platform) whereas :math:`X'` is actual output (from MACE) . + + +MACE automatically validate these metrics by running models with synthetic inputs. +If you want to specify input data to use, you can add an option in yaml config under 'subgraphs', e.g., + +.. code:: yaml + + models: + mobilenet_v1: + platform: tensorflow + model_file_path: https://cnbj1.fds.api.xiaomi.com/mace/miai-models/mobilenet-v1/mobilenet-v1-1.0.pb + model_sha256_checksum: 71b10f540ece33c49a7b51f5d4095fc9bd78ce46ebf0300487b2ee23d71294e6 + subgraphs: + - input_tensors: + - input + input_shapes: + - 1,224,224,3 + output_tensors: + - MobilenetV1/Predictions/Reshape_1 + output_shapes: + - 1,1001 + validation_inputs_data: + - https://cnbj1.fds.api.xiaomi.com/mace/inputs/dog.npy + + +If model's output is suspected to be incorrect, it might be useful to debug your model layer by layer by specifying an intermediate layer as output, +or use binary search method until suspicious layer is found. + + +Debug memory usage +-------------------------- +The simplest way to debug process memory usage is to use ``top`` command. With ``-H`` option, it can also show thread info. +For android, if you need more memory info, e.g., memory used of all categories, ``adb shell dumpsys meminfo`` will help. +By watching memory usage, you can check if memory usage meets expectations or if any leak happens. + + +Debug performance +-------------------------- +Using MACE, you can benchmark a model by examining each layer's duration as well as total duration. Or you can benchmark a single op. +The detailed information is in :doc:`../user_guide/benchmark`. + + +Debug model conversion +-------------------------- +After model is converted to MACE model, a literal model graph is generated in directory `mace/codegen/models/your_model`. +You can refer to it when debugging model conversion. + + +Debug engine using log -------------------------- Mace defines two sorts of logs: one is for users (LOG), the other is for developers (VLOG). @@ -23,14 +102,7 @@ By using Mace run tool, vlog level can be easily set by option, e.g., If models are run on android, you might need to use ``adb logcat`` to view logs. -Debug memory usage --------------------------- -The simplest way to debug process memory usage is to use ``top`` command. With ``-H`` option, it can also show thread info. -For android, if you need more memory info, e.g., memory used of all categories, ``adb shell dumpsys meminfo`` will help. -By watching memory usage, you can check if memory usage meets expectations or if any leak happens. - - -Debug using GDB +Debug engine using GDB -------------------------- GDB can be used as the last resort, as it is powerful that it can trace stacks of your process. If you run models on android, things may be a little bit complicated.