> - To install and use `Android_NDK`, you need to configure environment variables. The command example is `export ANDROID_NDK={$NDK_PATH}/android-ndk-r20b`.
> - In the `build.sh` script, run the `git clone` command to obtain the code in the third-party dependency library. Ensure that the network settings of Git are correct.
### Compilation Options
MindSpore Lite provides a compilation script `build.sh` for one-click compilation, located in the root directory of MindSpore. This script can be used to compile the code of training and inference. The following describes the compilation options of MindSpore Lite.
| Parameter | Parameter Description | Value Range | Mandatory or Not |
| -------- | ----- | ---- | ---- |
| **-I** | **Selects an applicable architecture. This option is required when compile MindSpore Lite.** | **arm64, arm32, or x86_64** | **Yes** |
| -d | If this parameter is set, the debug version is compiled. Otherwise, the release version is compiled. | None | No |
| -i | If this parameter is set, incremental compilation is performed. Otherwise, full compilation is performed. | None | No |
| -j[n] | Sets the number of threads used during compilation. Otherwise, the number of threads is set to 8 by default. | Integer | No |
| -e | In the Arm architecture, select the backend operator and set the `gpu` parameter. The built-in GPU operator of the framework is compiled at the same time. | GPU | No |
| -h | Displays the compilation help information. | None | No |
> When the `-I` parameter changes, such as `-I x86_64` is converted to `-I arm64`, adding `-i` for parameter compilation does not take effect.
### Compilation Example
First, download source code from the MindSpore code repository.
Then, run the following commands in the root directory of the source code to compile MindSpore Lite of different versions:
- Debug version of the x86_64 architecture:
```bash
bash build.sh -I x86_64 -d
```
- Release version of the x86_64 architecture, with the number of threads set:
```bash
bash build.sh -I x86_64 -j32
```
- Release version of the Arm 64-bit architecture in incremental compilation mode, with the number of threads set:
```bash
bash build.sh -I arm64 -i-j32
```
- Release version of the Arm 64-bit architecture in incremental compilation mode, with the built-in GPU operator compiled:
```bash
bash build.sh -I arm64 -e gpu
```
### Output Description
After the compilation is complete, go to the `mindspore/output` directory of the source code to view the file generated after compilation. The file is divided into two parts.
-`mindspore-lite-{version}-converter-{os}.tar.gz`：Contains model conversion tool.
-`mindspore-lite-{version}-runtime-{os}-{device}.tar.gz`：Contains model inference framework、benchmarking tool and performance analysis tool.
> version: version of the output, consistent with that of the MindSpore.
>
> device: Currently divided into cpu (built-in CPU operator) and gpu (built-in CPU and GPU operator).
>
> os: Operating system on which the output will be deployed.
Execute the decompression command to obtain the compiled output:
#### Description of Converter's Directory Structure
The conversion tool is only available under the `-I x86_64` compilation option, and the content includes the following parts:
```
|
├── mindspore-lite-{version}-converter-{os}
│ └── converter # Model conversion Ttool
│ └── third_party # Header files and libraries of third party libraries
│ ├── protobuf # Dynamic library of Protobuf
```
#### Description of Runtime and Other tools' Directory Structure
The inference framework can be obtained under `-I x86_64`, `-I arm64` and `-I arm32` compilation options, and the content includes the following parts:
- When the compilation option is `-I x86_64`:
```
|
├── mindspore-lite-{version}-runtime-x86-cpu
│ └── benchmark # Benchmarking Tool
│ └── lib # Inference framework dynamic library
│ ├── libmindspore-lite.so # Dynamic library of infernece framework in MindSpore Lite
│ └── third_party # Header files and libraries of third party libraries
│ ├── flatbuffers # Header files of FlatBuffers
│ └── include # Header files of inference framework
│ └── time_profiler # Model network layer time-consuming analysis tool
```
- When the compilation option is `-I arm64`:
```
|
├── mindspore-lite-{version}-runtime-arm64-cpu
│ └── benchmark # Benchmarking Tool
│ └── lib # Inference framework dynamic library
│ ├── libmindspore-lite.so # Dynamic library of infernece framework in MindSpore Lite
│ ├── liboptimize.so # Operator performance optimization library in MindSpore Lite
│ └── third_party # Header files and libraries of third party libraries
│ ├── flatbuffers # Header files of FlatBuffers
│ └── include # Header files of inference framework
│ └── time_profiler # Model network layer time-consuming analysis tool
```
- When the compilation option is `-I arm32`:
```
|
├── mindspore-lite-{version}-runtime-arm64-cpu
│ └── benchmark # Benchmarking Tool
│ └── lib # Inference framework dynamic library
│ ├── libmindspore-lite.so # Dynamic library of infernece framework in MindSpore Lite
│ └── third_party # Header files and libraries of third party libraries
│ ├── flatbuffers # Header files of FlatBuffers
│ └── include # Header files of inference framework
│ └── time_profiler # Model network layer time-consuming analysis tool
```
> 1. `liboptimize.so` only exists in the output package of runtime-arm64 and is only used on ARMv8.2 and CPUs that support fp16.
> 2. Compile ARM64 to get the inference framework output of arm64-cpu by default, if you add `-e gpu`, you will get the inference framework output of arm64-gpu, and the package name is `mindspore-lite-{version}-runtime-arm64-gpu.tar.gz`, compiling ARM32 is in the same way.
> 3. Before running the tools in the converter, benchmark or time_profiler directory, you need to configure environment variables, and configure the path where the dynamic libraries of MindSpore Lite and Protobuf are located to the path where the system searches for dynamic libraries. Take the CPU compiled under version 0.7.0-beta as an example: configure converter: `export LD_LIBRARY_PATH=./output/mindspore-lite-0.7.0-converter-ubuntu/third_party/protobuf/lib:${LD_LIBRARY_PATH}`; configure benchmark and time_profiler: `export LD_LIBRARY_PATH= ./output/mindspore-lite-0.7.0-runtime-x86-cpu/lib:${LD_LIBRARY_PATH}`.
> - `Android_NDK` needs to be installed only when the Arm version is compiled. Skip this dependency when the x86_64 version is compiled.
> - To install and use `Android_NDK`, you need to configure environment variables. The command example is `export ANDROID_NDK={$NDK_PATH}/android-ndk-r20b`.
- Compilation dependencies (additional dependencies required by the MindSpore Lite model conversion tool, which is required only for compilation of the x86_64 version)
MindSpore Lite provides multiple compilation options. You can select different compilation options as required.
| Parameter | Parameter Description | Value Range | Mandatory or Not |
| -------- | ----- | ---- | ---- |
| -d | If this parameter is set, the debug version is compiled. Otherwise, the release version is compiled. | - | No |
| -i | If this parameter is set, incremental compilation is performed. Otherwise, full compilation is performed. | - | No |
| -j[n] | Sets the number of threads used during compilation. Otherwise, the number of threads is set to 8 by default. | - | No |
| -I | Selects an applicable architecture. | arm64, arm32, or x86_64 | Yes |
| -e | In the Arm architecture, select the backend operator and set the `gpu` parameter. The built-in GPU operator of the framework is compiled at the same time. | GPU | No |
| -h | Displays the compilation help information. | - | No |
> When the `-I` parameter changes, that is, the applicable architecture is changed, the `-i` parameter cannot be used for incremental compilation.
## Output Description
After the compilation is complete, go to the `mindspore/output` directory of the source code to view the file generated after compilation. The file is named `mindspore-lite-{version}-{function}-{OS}.tar.gz`. After decompression, the tool package named `mindspore-lite-{version}-{function}-{OS}` can be obtained.
> version: version of the output, consistent with that of the MindSpore.
>
> function: function of the output. `converter` indicates the output of the conversion tool and `runtime` indicates the output of the inference framework.
For the x86 architecture, you can obtain the output of the conversion tool and inference framework；But for the ARM architecture, you only get inference framework.
Generally, the compiled output files include the following types. The architecture selection affects the types of output files.
> For the Arm 64-bit architecture, you can obtain the output of the `arm64-cpu` inference framework. If `-e gpu` is added, you can obtain the output of the `arm64-gpu` inference framework. The compilation for arm 64-bit is the same as that for arm 32-bit.
| Directory | Description | converter | runtime |
| --- | --- | --- | --- |
| include | Inference framework header file | No | Yes |
-`mindspore-lite-0.7.0-runtime-x86-cpu`: `third party`include `flatbuffers` (FlatBuffers header file), `lib`include`libmindspore-lite.so`(Dynamic library of MindSpore Lite inference framework).
-`mindspore-lite-0.7.0-runtime-arm64-cpu`: `third party`include `flatbuffers` (FlatBuffers header file), `lib`include`libmindspore-lite.so`(Dynamic library of MindSpore Lite inference framework) and `liboptimize.so`(Dynamic library of MindSpore Lite advanced operators).
> `liboptimize.so` only exits in runtime-arm64 outputs, and only can be used in the CPU which supports armv8.2 and fp16.
> Before running the tools in the `converter`, `benchmark`, or `time_profiler` directory, you need to configure environment variables and set the paths of the dynamic libraries of MindSpore Lite and Protobuf to the paths of the system dynamic libraries. The following uses the 0.7.0-beta version as an example: `export LD_LIBRARY_PATH=./mindspore-lite-0.7.0/lib:./mindspore-lite-0.7.0/third_party/protobuf/lib:${LD_LIBRARY_PATH}`.
## Compilation Example
First, download source code from the MindSpore code repository.
Then, run the following commands in the root directory of the source code to compile MindSpore Lite of different versions:
- Debug version of the x86_64 architecture:
```bash
bash build.sh -I x86_64 -d
```
- Release version of the x86_64 architecture, with the number of threads set:
```bash
bash build.sh -I x86_64 -j32
```
- Release version of the Arm 64-bit architecture in incremental compilation mode, with the number of threads set:
```bash
bash build.sh -I arm64 -i-j32
```
- Release version of the Arm 64-bit architecture in incremental compilation mode, with the built-in GPU operator compiled:
```bash
bash build.sh -I arm64 -e gpu
```
> - In the `build.sh` script, run the `git clone` command to obtain the code in the third-party dependency library. Ensure that the network settings of Git are correct.
Take the 0.7.0-beta version as an example. After the release version of the x86_64 architecture is compiled, go to the `mindspore/output` directory and run the following decompression command to obtain the output files `include`, `lib`, `benchmark`, `time_profiler`, `converter`, and `third_party`:
The Benchmark tool is used to perform benchmark testing on a MindSpore Lite model and is implemented using the C++ language. It can not only perform quantitative analysis (performance) on the forward inference execution duration of a MindSpore Lite model, but also perform comparative error analysis (accuracy) based on the output of the specified model.
The Benchmark tool is used to perform benchmark testing on a MindSpore Lite model. It can not only perform quantitative analysis (performance) on the forward inference execution duration of a MindSpore Lite model, but also perform comparative error analysis (accuracy) based on the output of the specified model.
## Environment Preparation
To use the Benchmark tool, you need to prepare the environment as follows:
- Compilation: Install compilation dependencies and perform compilation. The code of the Benchmark tool is stored in the `mindspore/lite/tools/benchmark` directory of the MindSpore source code. For details about the compilation operations, see the [Environment Requirements](https://www.mindspore.cn/lite/docs/en/master/deploy.html#id2) and [Compilation Example](https://www.mindspore.cn/lite/docs/en/master/deploy.html#id5) in the compilation document.
- Compilation: Install build dependencies and perform build. The code of the Benchmark tool is stored in the `mindspore/lite/tools/benchmark` directory of the MindSpore source code. For details about the build operations, see the [Environment Requirements](https://www.mindspore.cn/lite/tutorial/en/master/build.html#environment-requirements) and [Compilation Example](https://www.mindspore.cn/lite/tutorial/en/master/build.html#compilation-example) in the build document.
- Run: Obtain the `Benchmark` tool and configure environment variables. For details, see [Output Description](https://www.mindspore.cn/lite/docs/zh-CN/master/compile.html#id4) in the compilation document.
- Run: Obtain the `Benchmark` tool and configure environment variables. For details, see [Output Description](https://www.mindspore.cn/lite/tutorial/en/master/build.html#output-description) in the build document.
## Parameter Description
The command used for benchmark testing based on the compiled Benchmark tool is as follows:
@@ -50,12 +50,9 @@ The following describes the parameters in detail.
| `--cpuBindMode=<CPUBINDMODE>` | Optional | Specifies the type of the CPU core bound to the model inference program. | Integer | 1 | −1: medium core<br/>1: large core<br/>0: not bound |
| `--device=<DEVICE>` | Optional | Specifies the type of the device on which the model inference program runs. | String | CPU | CPU or GPU |
| `--help` | Optional | Displays the help information about the `benchmark` command. | - | - | - |
| `--inDataPath=<INDATAPATH>` | Optional | Specifies the file path of the input data of the tested model. If this parameter is not set, a random value will be used. | String | Null | - |
| `--inDataType=<INDATATYPE>` | Optional | Specifies the file type of the input data of the tested model. | String | Bin | Img: The input data is an image. Bin: The input data is a binary file.|
| `--inDataPath=<INDATAPATH>` | Optional | Specifies the file path of the input data of the tested model. If this parameter is not set, a random value will be used. | String | Null | - |
| `--loopCount=<LOOPCOUNT>` | Optional | Specifies the number of forward inference times of the tested model when the Benchmark tool is used for the benchmark testing. The value is a positive integer. | Integer | 10 | - |
| `--numThreads=<NUMTHREADS>` | Optional | Specifies the number of threads for running the model inference program. | Integer | 2 | - |
| `--omModelPath=<OMMODELPATH>` | Optional | Specifies the file path of the OM model. This parameter is optional only when the `device` type is NPU. | String | Null | - |
| `--resizeDims=<RESIZEDIMS>` | Optional | Specifies the size to be adjusted for the input data of the tested model. | String | Null | - |
| `--warmUpLoopCount=<WARMUPLOOPCOUNT>` | Optional | Specifies the number of preheating inference times of the tested model before multiple rounds of the benchmark test are executed. | Integer | 3 | - |
| `--fp16Priority=<FP16PIORITY>` | Optional | Specifies whether the float16 operator is preferred. | Bool | false | true, false |
@@ -18,15 +19,17 @@ MindSpore Lite provides a tool for offline model conversion. It supports convers
Currently, the following input formats are supported: MindSpore, TensorFlow Lite, Caffe, and ONNX.
## Environment Preparation
## Linux Environment Instructions
### Environment Preparation
To use the MindSpore Lite model conversion tool, you need to prepare the environment as follows:
- Compilation: Install basic and additional compilation dependencies and perform compilation. The compilation version is x86_64. The code of the model conversion tool is stored in the `mindspore/lite/tools/converter` directory of the MindSpore source code. For details about the compilation operations, see the [Environment Requirements] (https://www.mindspore.cn/lite/docs/zh-CN/master/compile.html#id2) and [Compilation Example] (https://www.mindspore.cn/lite/docs/zh-CN/master/deploy.html#id5) in the compilation document.
- Compilation: Install basic and additional build dependencies and perform build. The build version is x86_64. The code of the model conversion tool is stored in the `mindspore/lite/tools/converter` directory of the MindSpore source code. For details about the build operations, see the [Environment Requirements] (https://www.mindspore.cn/lite/tutorial/en/master/build.html#environment-requirements) and [Compilation Example] (https://www.mindspore.cn/lite/tutorial/en/master/build.html#compilation-example) in the build document.
- Run: Obtain the `converter` tool and configure environment variables by referring to [Output Description](https://www.mindspore.cn/lite/docs/zh-CN/master/compile.html#id4) in the compilation document.
- Run: Obtain the `converter` tool and configure environment variables by referring to [Output Description](https://www.mindspore.cn/lite/tutorial/en/master/build.html#output-description) in the build document.
## Parameter Description
### Parameter Description
You can use `./converter_lite ` to complete the conversion. In addition, you can set multiple parameters as required.
You can enter `./converter_lite --help` to obtain help information in real time.
...
...
@@ -51,7 +54,7 @@ The following describes the parameters in detail.
> - The Caffe model is divided into two files: model structure `*.prototxt`, corresponding to the `--modelFile` parameter; model weight `*.caffemodel`, corresponding to the `--weightFile` parameter
## Example
### Example
First, in the root directory of the source code, run the following command to perform compilation. For details, see `compile.md`.
```bash
...
...
@@ -96,6 +99,7 @@ The following describes how to use the conversion command by using several commo
@@ -105,4 +109,73 @@ The following describes how to use the conversion command by using several commo
```
INFO [converter/converter.cc:190] Runconverter] CONVERTER RESULT: SUCCESS!
```
\ No newline at end of file
## Windows Environment Instructions
### Environment Preparation
To use the MindSpore Lite model conversion tool, the following environment preparations are required.
-Compile: The model conversion tool code is in the `mindspore/lite/tools/converter` directory of the MindSpore source code, refer to the [Environment Requirements](https://www.mindspore.cn/lite/tutorials/en) in the deployment document /master/compile.html#id6) and [Compile Example](https://www.mindspore.cn/lite/tutorials/en/master/deploy.html#id8) to compile the Windows version.
-Run: Refer to [Compile Output](https://www.mindspore.cn/lite/tutorials/en/master/compile.html#id9) in the deployment document to obtain the `converter` tool, and set MinGW/ Copy several dependent files (libgcc_s_seh-1.dll, libwinpthread-1.dll, libssp-0.dll, libstdc++-6.dll) in the bin directory to the main directory of the `converter` tool.
### Parameter Description
Reference description Linux environment model conversion tool [parameter description](https://www.mindspore.cn/lite/tutorials/en/master/converter_tool.html#id4)
### Example
First, use the cmd tool to enter the command to compile in the root directory of the source code, refer to `compile.md`.
```bash
call build.bat lite
```
Then, set the log printing level to INFO.
```bash
set MSLOG=INFO
```
Several common examples are selected below to illustrate the use of conversion commands.
-Take Caffe model LeNet as an example to execute the conversion command.
In this example, because the Caffe model is used, two input files of model structure and model weight are required. Then plus fmk type and output path two parameters which are required, you can successfully execute.
The result is shown as:
```
INFO [converter/converter.cc:190] Runconverter] CONVERTER RESULT: SUCCESS!
```
This means that the Caffe model has been successfully converted to the MindSpore Lite model and the new file `lenet.ms` has been obtained.
- Take MindSpore, TensorFlow Lite, ONNX model format and perceptual quantization model as examples to execute conversion commands.
After model conversion using MindSpore Lite, the model inference process needs to be completed in Runtime.
The procedure for using Runtime is shown in the following figure:
Its components and their functions are described as follows:
-`Model`: model used by MindSpore Lite, which instantiates the list of operator prototypes through image composition or direct network loading.
-`Lite Session`: provides the graph compilation function and calls the graph executor for inference.
-`Scheduler`: operator heterogeneous scheduler. It can select a proper kernel for each operator based on the heterogeneous scheduling policy, construct a kernel list, and split a graph into subgraphs.
-`Executor`: graph executor, which executes the kernel list to dynamically allocate and release tensors.
-`Operator`: operator prototype, including operator attributes and methods for inferring the shape, data type, and format.
-`Kernel`: operator, which provides specific operator implementation and the operator forwarding function.
-`Tensor`: tensor used by MindSpore Lite, which provides functions and APIs for tensor memory operations.
## Reading Models
In MindSpore Lite, a model file is an `.ms` file converted using the model conversion tool. During model inference, the model needs to be loaded from the file system and parsed. Related operations are mainly implemented in the Model component. The Model component holds model data such as weight data and operator attributes.
A model is created based on memory data using the static `Import` method of the Model class. The `Model` instance returned by the function is a pointer, which is created by using `new`. If the pointer is not required, you need to release it by using `delete`.
## Session Creation
When MindSpore Lite is used for inference, sessions are the main entrance of inference. You can compile and execute graphs through sessions.
### Creating Contexts
Contexts save some basic configuration parameters required by sessions to guide graph compilation and execution. The definition of context is as follows:
MindSpore Lite supports heterogeneous inference. The preferred backend for inference is specified by `device_ctx_` in `Context` and is CPU by default. During graph compilation, operator selection and scheduling are performed based on the preferred backend.
MindSpore Lite has a built-in thread pool shared by processes. During inference, `thread_num_` is used to specify the maximum number of threads in the thread pool. The default maximum number is 2. It is recommended that the maximum number be no more than 4. Otherwise, the performance may be affected.
MindSpore Lite supports dynamic memory allocation and release. If `allocator` is not specified, a default `allocator` is generated during inference. You can also use the `Context` method to allow multiple `Context` to share the memory allocator.
### Creating Sessions
Use the `Context` created in the previous step to call the static `CreateSession` method of LiteSession to create `LiteSession`. The `LiteSession` instance returned by the function is a pointer, which is created by using `new`. If the pointer is not required, you need to release it by using `delete`.
### Example
The following sample code demonstrates how to create a `Context` and how to allow two `LiteSession` to share a memory pool.
```cpp
autocontext=new(std::nothrow)lite::Context;
if(context==nullptr){
MS_LOG(ERROR)<<"New context failed while running %s",modelName.c_str();
returnRET_ERROR;
}
// The preferred backend is GPU, which means, if there is a GPU operator, it will run on the GPU first, otherwise it will run on the CPU.
context->device_ctx_.type=lite::DT_GPU;
// The medium core takes priority in thread and core binding methods. This parameter will work in the BindThread interface. For specific binding effect, see the "Run Graph" section.
context->cpu_bind_mode_=MID_CPU;
// Configure the number of worker threads in the thread pool to 2, including the main thread.
context->thread_num_=2;
// Allocators can be shared across multiple Contexts.
MS_LOG(ERROR)<<"CreateSession failed while running %s",modelName.c_str();
returnRET_ERROR;
}
```
## Graph Compilation
### Variable Dimension
When using MindSpore Lite for inference, after the session creation and graph compilation have been completed, if you need to resize the input shape, you can reset the shape of the input tensor, and then call the session's Resize() interface.
### Example
The following code demonstrates how to resize the input of MindSpore Lite:
```cpp
// Assume we have created a LiteSession instance named session.
autoinputs=session->GetInputs();
std::vector<int>resize_shape={1,128,128,3};
// Assume the model has only one input,resize input shape to [1, 128, 128, 3]
inputs[0]->set_shape(resize_shape);
session->Resize(inputs);
```
### Compiling Graphs
Before graph execution, call the `CompileGraph` API of the `LiteSession` to compile graphs and further parse the Model instance loaded from the file, mainly for subgraph split and operator selection and scheduling. This process takes a long time. Therefore, it is recommended that `ListSession` achieve multiple executions with one creation and one compilation.
### Example
The following code demonstrates how to compile graph of MindSpore Lite:
```cpp
// Assume we have created a LiteSession instance named session and a Model instance named model before.
// The methods of creating model and session can refer to "Import Model" and "Create Session" two sections.
autoret=session->CompileGraph(model);
if(ret!=RET_OK){
std::cerr<<"CompileGraph failed"<<std::endl;
// session and model need to be released by users manually.
delete(session);
delete(model);
returnret;
}
```
## Data Input
### Obtaining Input Tensors
Before graph execution, you need to copy the input data to model input tensors.
MindSpore Lite provides the following methods to obtain model input tensors.
1. Use the `GetInputsByName` method to obtain vectors of the model input tensors that are connected to the model input node based on the node name.
2. Use the `GetInputs` method to directly obtain the vectors of all model input tensors.
### Copying Data
After model input tensors are obtained, you need to enter data into the tensors. Use the `Size` method of `MSTensor` to obtain the size of the data to be entered into tensors, use the `data_type` method to obtain the data type of tensors, and use the `MutableData` method of `MSTensor` to obtain the writable pointer.
### Example
The following sample code shows how to obtain the entire graph input `MSTensor` from `LiteSession` and enter the model input data to `MSTensor`.
```cpp
// Assume we have created a LiteSession instance named session.
autoinputs=session->GetInputs();
// Assume that the model has only one input tensor.
autoin_tensor=inputs.front();
if(in_tensor==nullptr){
std::cerr<<"Input tensor is nullptr"<<std::endl;
return-1;
}
// It is omitted that users have read the model input file and generated a section of memory buffer: input_buf, as well as the byte size of input_buf: data_size.
if(in_tensor->Size()!=data_size){
std::cerr<<"Input data size is not suit for model input"<<std::endl;
return-1;
}
auto*in_data=in_tensor->MutableData();
if(in_data==nullptr){
std::cerr<<"Data of in_tensor is nullptr"<<std::endl;
return-1;
}
memcpy(in_data,input_buf,data_size);
// Users need to free input_buf.
// The elements in the inputs are managed by MindSpore Lite so that users do not need to free inputs.
```
Note:
- The data layout in the model input tensors of MindSpore Lite must be NHWC.
- The model input `input_buf` is read from disks. After it is copied to model input tensors, you need to release `input_buf`.
- Vectors returned by using the `GetInputs` and `GetInputsByName` methods do not need to be released by users.
## Graph Execution
### Executing Sessions
After a MindSpore Lite session performs graph compilation, you can use `RunGraph` of `LiteSession` for model inference.
### Core Binding
The built-in thread pool of MindSpore Lite supports core binding and unbinding. By calling the `BindThread` API, you can bind working threads in the thread pool to specified CPU cores for performance analysis. The core binding operation is related to the context specified when `LiteSession` is created. The core binding operation sets the affinity between a thread and CPU based on the core binding policy in the context.
Note that core binding is an affinity operation, which is affected by system scheduling. Therefore, successful binding to the specified CPU core cannot be ensured. After executing the code of core binding, you need to perform the unbinding operation. The following is an example:
```cpp
// Assume we have created a LiteSession instance named session.
session->BindThread(true);
autoret=session->RunGraph();
if(ret!=mindspore::lite::RET_OK){
std::cerr<<"RunGraph failed"<<std::endl;
deletesession;
return-1;
}
session->BindThread(false);
```
> Core binding parameters can be used to bind big cores first or middle cores first.
> The rule for determining big core or middle core is based on the CPU core frequency instead of CPU architecture. For the CPU architecture where big, middle, and little cores are not distinguished, this rule can be used.
> Big core first indicates that threads in the thread pool are bound to cores according to core frequency. The first thread is bound to the core with the highest frequency, and the second thread is bound to the core with the second highest frequency. This rule also applies to other threads.
> Middle cores are defined based on experience. By default, middle cores are cores with the third and fourth highest frequency. Middle core first indicates that threads are bound to middle cores preferentially. When there are no available middle cores, threads are bound to little cores.
### Callback Running
MindSpore Lite can transfer two `KernelCallBack` function pointers to call back the inference model when calling `RunGraph`. Compared with common graph execution, callback running can obtain extra information during the running process to help developers analyze performance and fix bugs. The extra information includes:
- Name of the running node
- Input and output tensors before inference of the current node
- Input and output tensors after inference of the current node
### Example
The following sample code demonstrates how to use `LiteSession` to compile a graph, defines two callback functions as the before-callback pointer and after-callback pointer, transfers them to the `RunGraph` API for callback inference, and demonstrates the scenario of multiple graph executions with one graph compilation.
```cpp
// Assume we have created a LiteSession instance named session and a Model instance named model before.
// The methods of creating model and session can refer to "Import Model" and "Create Session" two sections.
autoret=session->CompileGraph(model);
if(ret!=RET_OK){
std::cerr<<"CompileGraph failed"<<std::endl;
// session and model need to be released by users manually.
delete(session);
delete(model);
returnret;
}
// Copy input data into the input tensor. Users can refer to the "Input Data" section. We uses random data here.
autoinputs=session->GetInputs();
for(autoin_tensor:inputs){
in_tensor=inputs.front();
if(in_tensor==nullptr){
std::cerr<<"Input tensor is nullptr"<<std::endl;
return-1;
}
// When calling the MutableData method, if the data in MSTensor is not allocated, it will be malloced. After allocation, the data in MSTensor can be considered as random data.
(void)in_tensor->MutableData();
}
// Definition of callback function before forwarding operator.
// CompileGraph would cost much time, a better solution is calling CompileGraph only once and RunGraph much more times.
for(size_ti=0;i<10;i++){
autoret=session_->RunGraph();
if(ret!=RET_OK){
MS_LOG(ERROR)<<"Run graph failed.";
returnRET_ERROR;
}
}
// session and model needs to be released by users manually.
delete(session);
delete(model);
```
## Obtaining Outputs
### Obtaining Output Tensors
After performing inference, MindSpore Lite can obtain the model inference result.
MindSpore Lite provides the following methods to obtain the model output `MSTensor`.
1. Use the `GetOutputsByName` method to obtain vectors of the model output `MSTensor` that is connected to the model output node based on the node name.
2. Use the `GetOutputMapByNode` method to directly obtain the mapping between the names of all model output nodes and the model output `MSTensor` connected to the nodes.
3. Use the `GetOutputByTensorName` method to obtain the model output `MSTensor` based on the tensor name.
4. Use the `GetOutputMapByTensor` method to directly obtain the mapping between the names of all model output tensors and the model output `MSTensor`.
After model output tensors are obtained, you need to enter data into the tensors. Use the `Size` method of `MSTensor` to obtain the size of the data to be entered into tensors, use the `data_type` method to obtain the data type of `MSTensor`, and use the `MutableData` method of `MSTensor` to obtain the writable pointer.
### Example
The following sample code shows how to obtain the output `MSTensor` from `LiteSession` using the `GetOutputMapByNode` method and print the first ten data or all data records of each output `MSTensor`.
```cpp
// Assume we have created a LiteSession instance named session before.
autooutput_map=session->GetOutputMapByNode();
// Assume that the model has only one output node.
autoout_node_iter=output_map.begin();
std::stringname=out_node_iter->first;
// Assume that the unique output node has only one output tensor.
autoout_tensor=out_node_iter->second.front();
if(out_tensor==nullptr){
std::cerr<<"Output tensor is nullptr"<<std::endl;
return-1;
}
// Assume that the data format of output data is float 32.
// The elements in outputs do not need to be free by users, because outputs are managed by the MindSpore Lite.
```
Note that the vectors or map returned by the `GetOutputsByNodeName`, `GetOutputMapByNode`, `GetOutputByTensorName` and `GetOutputMapByTensor` methods do not need to be released by users.
The following sample code shows how to obtain the output `MSTensor` from `LiteSession` using the `GetOutputsByNodeName` method.
```cpp
// Assume we have created a LiteSession instance named session before.
// Assume that model has a output node named output_node_name_0.
@@ -20,9 +20,9 @@ The TimeProfiler tool can be used to analyze the time consumption of forward inf
To use the TimeProfiler tool, you need to prepare the environment as follows:
- Compilation: Install compilation dependencies and perform compilation. The code of the TimeProfiler tool is stored in the `mindspore/lite/tools/time_profiler` directory of the MindSpore source code. For details about the compilation operations, see the [Environment Requirements](https://www.mindspore.cn/lite/docs/en/master/compile.html#id2) and [Compilation Example](https://www.mindspore.cn/lite/docs/en/master/compile.html#id5) in the compilation document.
- Compilation: Install build dependencies and perform build. The code of the TimeProfiler tool is stored in the `mindspore/lite/tools/time_profiler` directory of the MindSpore source code. For details about the build operations, see the [Environment Requirements](https://www.mindspore.cn/lite/tutorial/en/master/compile.html#environment-requirements) and [Compilation Example](https://www.mindspore.cn/lite/tutorial/en/master/compile.html#compilation-example) in the build document.
- Run: Obtain the `time_profiler` tool and configure environment variables by referring to [Output Description](https://www.mindspore.cn/lite/docs/zh-CN/master/compile.html#id4) in the compilation document.
- Run: Obtain the `time_profiler` tool and configure environment variables by referring to [Output Description](https://www.mindspore.cn/lite/tutorial/en/master/build.html#output-description) in the build document.
