diff --git a/lite/tutorials/source_en/deploy.md b/lite/tutorials/source_en/deploy.md index 86496baa68c949ee53c8c6aa5beb15ddd5f5c5bb..b6d16cd30f3beee3342f91a1acab25b951536533 100644 --- a/lite/tutorials/source_en/deploy.md +++ b/lite/tutorials/source_en/deploy.md @@ -66,22 +66,25 @@ After the compilation is complete, go to the `mindspore/output` directory of the tar -xvf mindspore-lite-{version}-{function}-{OS}.tar.gz ``` -Generally, the compiled output files include the following types. The architecture selection affects the types of output files. - -> For the x86 architecture, you can obtain the output of the conversion tool; for the Arm 64-bit architecture, you can obtain the output of the `arm64-cpu` inference framework. If `-e gpu` is added, you can obtain the output of the `arm64-cpu` inference framework. The compilation for arm 64-bit is the same as that for arm 32-bit. +For the x86 architecture, you can obtain the output of the conversion tool and inference frameworkï¼›But for the ARM architecture, you only get inference framework. -| Directory | Description | x86_64 | Arm 64-bit | Arm 32-bit | -| --- | --- | --- | --- | --- | -| include | Inference framework header file | No | Yes | Yes | -| lib | Inference framework dynamic library | Yes | Yes | Yes | -| benchmark | Benchmark test tool | Yes | Yes | Yes | -| time_profiler | Time consumption analysis tool at the model network layer | Yes | Yes | Yes | -| converter | Model conversion tool | Yes | No | No | -| third_party | Header file and library of the third-party library | Yes | Yes | Yes | +Generally, the compiled output files include the following types. The architecture selection affects the types of output files. -The contents of `third party` vary depending on the architecture as follows: -- x86_64: `protobuf` (Protobuf dynamic library). -- arm: `flatbuffers` (FlatBuffers header file). +> For the Arm 64-bit architecture, you can obtain the output of the `arm64-cpu` inference framework. If `-e gpu` is added, you can obtain the output of the `arm64-gpu` inference framework. The compilation for arm 64-bit is the same as that for arm 32-bit. + +| Directory | Description | converter | runtime | +| --- | --- | --- | --- | +| include | Inference framework header file | No | Yes | +| lib | Inference framework dynamic library | No | Yes | +| benchmark | Benchmark test tool | No | Yes | +| time_profiler | Time consumption analysis tool at the model network layer| No | Yes | +| converter | Model conversion tool | Yes | No | No | +| third_party | Header file and library of the third-party library | Yes | Yes | + +Take the 0.7.0-beta version and CPU as an example. The contents of `third party` and `lib` vary depending on the architecture as follows: +- `mindspore-lite-0.7.0-converter-ubuntu`: include `protobuf` (Protobuf dynamic library). +- `mindspore-lite-0.7.0-runtime-x86-cpu`: include `flatbuffers` (FlatBuffers header file). +TODO: Add document content. > Before running the tools in the `converter`, `benchmark`, or `time_profiler` directory, you need to configure environment variables and set the paths of the dynamic libraries of MindSpore Lite and Protobuf to the paths of the system dynamic libraries. The following uses the 0.7.0-beta version as an example: `export LD_LIBRARY_PATH=./mindspore-lite-0.7.0/lib:./mindspore-lite-0.7.0/third_party/protobuf/lib:${LD_LIBRARY_PATH}`. diff --git a/lite/tutorials/source_en/use/runtime.md b/lite/tutorials/source_en/use/runtime.md new file mode 100644 index 0000000000000000000000000000000000000000..fe1fa8694aeb3750f199f251f86e68839128dafe --- /dev/null +++ b/lite/tutorials/source_en/use/runtime.md @@ -0,0 +1,3 @@ +# Runtime + +<a href="https://gitee.com/mindspore/docs/blob/master/lite/tutorials/source_en/use/runtime.md" target="_blank"><img src="../_static/logo_source.png"></a> diff --git a/lite/tutorials/source_en/use/runtime_lite.md b/lite/tutorials/source_en/use/runtime_lite.md deleted file mode 100644 index 834347308ff5fe95b14d70c2720c0c161522414c..0000000000000000000000000000000000000000 --- a/lite/tutorials/source_en/use/runtime_lite.md +++ /dev/null @@ -1,3 +0,0 @@ -# Runtime (Lite) - -<a href="https://gitee.com/mindspore/docs/blob/master/lite/tutorials/source_en/use/runtime_lite.md" target="_blank"><img src="../_static/logo_source.png"></a> diff --git a/lite/tutorials/source_zh_cn/deploy.md b/lite/tutorials/source_zh_cn/deploy.md index a12ccea3f77987e653114203583ce065d8bb7979..dec4cbe357e1bf40809dac285234bc70ef6b6784 100644 --- a/lite/tutorials/source_zh_cn/deploy.md +++ b/lite/tutorials/source_zh_cn/deploy.md @@ -73,25 +73,27 @@ MindSpore Liteæ供多ç§ç¼–译方å¼ï¼Œç”¨æˆ·å¯æ ¹æ®éœ€è¦é€‰æ‹©ä¸åŒçš„ç¼– ```bash tar -xvf mindspore-lite-{version}-{function}-{OS}.tar.gz ``` +编译x86å¯èŽ·å¾—转æ¢å·¥å…·`converter`与推ç†æ¡†æž¶`runtime`功能的输出件,编译ARM仅能获得推ç†æ¡†æž¶`runtime`。 -编译åŽçš„输出件一般包å«ä»¥ä¸‹å‡ ç§ï¼Œæž¶æž„的选择会影å“输出件的ç§ç±»ã€‚ +输出件ä¸åŒ…å«ä»¥ä¸‹å‡ ç±»å项,功能ä¸åŒæ‰€å«å†…容也会有所区别。 -> 编译x86å¯èŽ·å¾—转æ¢å·¥å…·çš„输出件,编译ARM64默认å¯èŽ·å¾—`arm64-cpu`的推ç†æ¡†æž¶è¾“å‡ºä»¶ï¼Œè‹¥æ·»åŠ `-e gpu`则获得`arm64-gpu`的推ç†æ¡†æž¶è¾“出件,编译ARM32åŒç†ã€‚ +> 编译ARM64默认å¯èŽ·å¾—`arm64-cpu`的推ç†æ¡†æž¶è¾“å‡ºä»¶ï¼Œè‹¥æ·»åŠ `-e gpu`则获得`arm64-gpu`的推ç†æ¡†æž¶è¾“出件,编译ARM32åŒç†ã€‚ -编译åŽçš„输出件一般包å«ä»¥ä¸‹å‡ ç§ï¼Œæž¶æž„的选择会影å“输出件的ç§ç±»ã€‚ +| 目录 | 说明 | converter | runtime | +| --- | --- | --- | --- | +| include | 推ç†æ¡†æž¶å¤´æ–‡ä»¶ | æ— | 有 | +| lib | 推ç†æ¡†æž¶åŠ¨æ€åº“ | æ— | 有 | +| benchmark | 基准测试工具 | æ— | 有 | +| time_profiler | 模型网络层耗时分æžå·¥å…· | æ— | 有 | +| converter | 模型转æ¢å·¥å…· | 有 | æ— | +| third_party | 第三方库头文件和库 | 有 | 有 | -| 目录 | 说明 | x86_64 | arm64 | arm32 | -| --- | --- | --- | --- | --- | -| include | 推ç†æ¡†æž¶å¤´æ–‡ä»¶ | æ— | 有 | 有 | -| lib | 推ç†æ¡†æž¶åŠ¨æ€åº“ | 有 | 有 | 有 | -| benchmark | 基准测试工具 | 有 | 有 | 有 | -| time_profiler | 模型网络层耗时分æžå·¥å…· | 有 | 有 | 有 | -| converter | 模型转æ¢å·¥å…· | 有 | æ— | æ— | -| third_party | 第三方库头文件和库 | 有 | 有 | 有 | - -在x86_64ã€ARM两ç§æž¶æž„下,`third party`的内容ä¸åŒã€‚å…¶ä¸ï¼š -- x86_64:`protobuf`(Protobuf的动æ€åº“)。 -- ARM:`flatbuffers`(FlatBuffers头文件)。 +以0.7.0-beta版本,CPU编译为例,ä¸åŒåŒ…å下,`third party`与`lib`的内容ä¸åŒï¼š + +- `mindspore-lite-0.7.0-converter-ubuntu`:包å«`protobuf`(Protobuf的动æ€åº“)。 +- `mindspore-lite-0.7.0-runtime-x86-cpu`:`third party`包å«`flatbuffers`(FlatBuffers头文件),`lib`包å«`libmindspore-lite.so`(MindSpore Lite的动æ€åº“)。 +- `mindspore-lite-0.7.0-runtime-arm64-cpu`:`third party`包å«`flatbuffers`(FlatBuffers头文件),`lib`包å«`libmindspore-lite.so`(MindSpore Lite的动æ€åº“)和`liboptimize.so`。 +TODO:补全文件内容 > è¿è¡Œconverterã€benchmark或time_profiler目录下的工具å‰ï¼Œéƒ½éœ€é…置环境å˜é‡ï¼Œå°†MindSpore Liteå’ŒProtobuf的动æ€åº“所在的路径é…置到系统æœç´¢åŠ¨æ€åº“的路径ä¸ã€‚以0.7.0-beta版本为例:`export LD_LIBRARY_PATH=./mindspore-lite-0.7.0/lib:./mindspore-lite-0.7.0/third_party/protobuf/lib:${LD_LIBRARY_PATH}`。 diff --git a/lite/tutorials/source_zh_cn/use/runtime.md b/lite/tutorials/source_zh_cn/use/runtime.md new file mode 100644 index 0000000000000000000000000000000000000000..f1615bf27713c1f5cfc8236251e4e5e44353d51f --- /dev/null +++ b/lite/tutorials/source_zh_cn/use/runtime.md @@ -0,0 +1,354 @@ +# Runtimeä½¿ç”¨æŒ‡å— + +<!-- TOC --> + +- [Runtime使用指å—](#runtime使用指å—) + - [概述](#概述) + - [读å–模型](#读å–模型) + - [创建会è¯](#创建会è¯) + - [创建上下文](#创建上下文) + - [创建会è¯](#创建会è¯-1) + - [使用示例](#使用示例) + - [图编译](#图编译) + - [å¯å˜ç»´åº¦](#å¯å˜ç»´åº¦) + - [图编译](#图编译-1) + - [输入数æ®](#输入数æ®) + - [获å–输入Tensor](#获å–输入tensor) + - [æ•°æ®æ‹·è´](#æ•°æ®æ‹·è´) + - [使用示例](#使用示例-1) + - [图执行](#图执行) + - [执行会è¯](#执行会è¯) + - [ç»‘æ ¸](#ç»‘æ ¸) + - [回调è¿è¡Œ](#回调è¿è¡Œ) + - [使用示例](#使用示例-2) + - [获å–输出](#获å–输出) + - [获å–输出Tensor](#获å–输出tensor) + - [使用示例](#使用示例-3) + - [获å–版本å·](#获å–版本å·) + - [使用示例](#使用示例-4) + +<!-- /TOC --> + +<a href="https://gitee.com/mindspore/docs/blob/master/lite/tutorials/source_zh_cn/use/runtime.md" target="_blank"><img src="../_static/logo_source.png"></a> + +## 概述 + +通过MindSpore Lite模型转æ¢åŽï¼Œéœ€åœ¨Runtimeä¸å®Œæˆæ¨¡åž‹çš„推ç†æ‰§è¡Œæµç¨‹ã€‚ + +Runtime总体使用æµç¨‹å¦‚下图所示: + +![img](../images/side_infer_process.png) + +包å«çš„组件åŠåŠŸèƒ½å¦‚下所述: +- `Model`:MindSpore Liteä½¿ç”¨çš„æ¨¡åž‹ï¼Œé€šè¿‡ç”¨æˆ·æž„å›¾æˆ–ç›´æŽ¥åŠ è½½ç½‘ç»œï¼Œæ¥å®žä¾‹åŒ–ç®—å原型的列表。 +- `Lite Session`:æ供图编译的功能,并调用图执行器进行推ç†ã€‚ +- `Scheduler`:算åå¼‚æž„è°ƒåº¦å™¨ï¼Œæ ¹æ®å¼‚构调度ç–略,为æ¯ä¸€ä¸ªç®—å选择åˆé€‚çš„kernelï¼Œæž„é€ kernel list,并切分å图。 +- `Executor`:图执行器,执行kernel list,动æ€åˆ†é…和释放Tensor。 +- `Operator`:算å原型,包å«ç®—å的属性,以åŠshapeã€data typeå’Œformat的推导方法。 +- `Kernel`:算å库æ供算å的具体实现,æ供算åforward的能力。 +- `Tensor`:MindSpore Lite使用的Tensor,æ供了Tensor内å˜æ“作的功能和接å£ã€‚ + +## 读å–模型 + +在MindSpore Liteä¸ï¼Œæ¨¡åž‹æ–‡ä»¶æ˜¯ä»Žæ¨¡åž‹è½¬æ¢å·¥å…·è½¬æ¢å¾—到的`.ms`文件。进行模型推ç†æ—¶ï¼Œéœ€è¦ä»Žæ–‡ä»¶ç³»ç»ŸåŠ 载模型,并进行模型解æžï¼Œè¿™éƒ¨åˆ†æ“作主è¦åœ¨Modelä¸å®žçŽ°ã€‚ModelæŒæœ‰æƒé‡æ•°æ®ã€ç®—å属性ç‰æ¨¡åž‹æ•°æ®ã€‚ + +模型通过Model类的é™æ€`Import`方法从内å˜æ•°æ®ä¸åˆ›å»ºã€‚函数返回的`Model`实例是一个指针,通过`new`创建,ä¸å†éœ€è¦æ—¶ï¼Œéœ€è¦ç”¨æˆ·é€šè¿‡`delete`释放。 + +## åˆ›å»ºä¼šè¯ + +使用MindSpore Lite执行推ç†æ—¶ï¼ŒSession是推ç†çš„主入å£ï¼Œé€šè¿‡Session我们å¯ä»¥è¿›è¡Œå›¾ç¼–译ã€å›¾æ‰§è¡Œã€‚ + +### 创建上下文 + +上下文会ä¿å˜ä¼šè¯æ‰€éœ€çš„一些基本é…ç½®å‚数,用于指导图编译和图执行,其定义如下: + +MindSpore Lite支æŒå¼‚构推ç†ï¼ŒæŽ¨ç†æ—¶çš„主选åŽç«¯ç”±`Context`ä¸çš„`device_ctx_`指定,默认为CPUã€‚åœ¨è¿›è¡Œå›¾ç¼–è¯‘æ—¶ï¼Œä¼šæ ¹æ®ä¸»é€‰åŽç«¯è¿›è¡Œç®—å选型调度。 + +MindSpore Liteå†…ç½®ä¸€ä¸ªè¿›ç¨‹å…±äº«çš„çº¿ç¨‹æ± ï¼ŒæŽ¨ç†æ—¶é€šè¿‡`thread_num_`æŒ‡å®šçº¿ç¨‹æ± çš„æœ€å¤§çº¿ç¨‹æ•°ï¼Œé»˜è®¤ä¸º2线程,推è最多ä¸è¶…过4个线程,å¦åˆ™å¯èƒ½ä¼šå½±å“性能。 + +MindSpore Lite支æŒåŠ¨æ€å†…å˜åˆ†é…和释放,如果没有指定`allocator`,推ç†æ—¶ä¼šç”Ÿæˆä¸€ä¸ªé»˜è®¤çš„`allocator`,也å¯ä»¥é€šè¿‡`Context`方法在多个`Context`ä¸å…±äº«å†…å˜åˆ†é…器。 + +如果用户通过`new`创建`Context`,ä¸å†éœ€è¦æ—¶ï¼Œéœ€è¦ç”¨æˆ·é€šè¿‡`delete`释放。一般在创建完SessionåŽï¼ŒContextå³å¯é‡Šæ”¾ã€‚ + +### åˆ›å»ºä¼šè¯ + +用上一æ¥åˆ›å»ºå¾—到的`Context`,调用LiteSessionçš„é™æ€`CreateSession`方法æ¥åˆ›å»º`LiteSession`。函数返回的`LiteSession`实例是一个指针,通过`new`创建,ä¸å†éœ€è¦æ—¶ï¼Œéœ€è¦ç”¨æˆ·é€šè¿‡`delete`释放。 + +### 使用示例 + +下é¢ç¤ºä¾‹ä»£ç 演示了`Context`的创建,以åŠåœ¨ä¸¤ä¸ª`LiteSession`间共享内å˜æ± 的功能: + +```cpp +auto context = new (std::nothrow) lite::Context; +if (context == nullptr) { + MS_LOG(ERROR) << "New context failed while running %s", modelName.c_str(); + return RET_ERROR; +} +// The preferred backend is GPU, which means, if there is a GPU operator, it will run on the GPU first, otherwise it will run on the CPU. +context->device_ctx_.type = lite::DT_GPU; +// The medium core takes priority in thread and core binding methods. This parameter will work in the BindThread interface. For specific binding effect, see the "Run Graph" section. +context->cpu_bind_mode_ = MID_CPU; +// Configure the number of worker threads in the thread pool to 2, including the main thread. +context->thread_num_ = 2; +// Allocators can be shared across multiple Contexts. +auto *context2 = new Context(context->thread_num_, context->allocator, context->device_ctx_); +context2->cpu_bind_mode_ = context->cpu_bind_mode_; +// Use Context to create Session. +auto session1 = session::LiteSession::CreateSession(context); +// After the LiteSession is created, the Context can be released. +delete (context); +if (session1 == nullptr) { + MS_LOG(ERROR) << "CreateSession failed while running %s", modelName.c_str(); + return RET_ERROR; +} +// session1 and session2 can share one memory pool. +auto session2 = session::LiteSession::CreateSession(context2); +delete (context2); +if (session == nullptr) { + MS_LOG(ERROR) << "CreateSession failed while running %s", modelName.c_str(); + return RET_ERROR; +} +``` + +## 图编译 + +### å¯å˜ç»´åº¦ + +TODO:该功能还在开å‘ä¸ã€‚ + +### 图编译 + +在图执行å‰ï¼Œéœ€è¦è°ƒç”¨`LiteSession`çš„`CompileGraph`接å£è¿›è¡Œå›¾ç¼–译,进一æ¥è§£æžä»Žæ–‡ä»¶ä¸åŠ 载的Model实例,主è¦è¿›è¡Œå图切分ã€ç®—å选型调度。这部分会耗费较多时间,所以建议`ListSession`创建一次,编译一次,多次执行。 + +## è¾“å…¥æ•°æ® + +### 获å–输入Tensor + +在图执行å‰ï¼Œéœ€è¦å°†è¾“入数æ®æ‹·è´åˆ°æ¨¡åž‹çš„输入Tensor。 + +MindSpore Liteæ供两ç§æ–¹æ³•æ¥èŽ·å–模型的输入Tensor。 + +1. 使用`GetInputsByName`æ–¹æ³•ï¼Œæ ¹æ®æ¨¡åž‹è¾“入节点的å称æ¥èŽ·å–模型输入Tensorä¸è¿žæŽ¥åˆ°è¯¥èŠ‚点的Tensorçš„vector。 +2. 使用`GetInputs`方法,直接获å–所有的模型输入Tensorçš„vector。 + +### æ•°æ®æ‹·è´ + +当获å–到模型的输入,就需è¦å‘Tensorä¸å¡«å…¥æ•°æ®ã€‚通过`MSTensor`çš„`Size`方法æ¥èŽ·å–Tensor应该填入的数æ®å¤§å°ï¼Œé€šè¿‡`data_type`方法æ¥èŽ·å–Tensorçš„æ•°æ®ç±»åž‹ï¼Œé€šè¿‡`MSTensor`çš„`MutableData`方法æ¥èŽ·å–å¯å†™çš„指针。 + +### 使用示例 + +下é¢ç¤ºä¾‹ä»£ç 演示了从`LiteSession`ä¸èŽ·å–整图输入`MSTensor`,并且å‘å…¶ä¸çŒå…¥æ¨¡åž‹è¾“入数æ®çš„过程: + +```cpp +// Assume we have created a LiteSession instance named session. +auto inputs = session->GetInputs(); +// Assume that the model has only one input tensor. +auto in_tensor = inputs.front(); +if (in_tensor == nullptr) { + std::cerr << "Input tensor is nullptr" << std::endl; + return -1; +} +// It is omitted that users have read the model input file and generated a section of memory buffer: input_buf, as well as the byte size of input_buf: data_size. +if (in_tensor->Size() != data_size) { + std::cerr << "Input data size is not suit for model input" << std::endl; + return -1; +} +auto *in_data = in_tensor->MutableData(); +if (in_data == nullptr) { + std::cerr << "Data of in_tensor is nullptr" << std::endl; + return -1; +} +memcpy(in_data, input_buf, data_size); +// Users need to free input_buf. +// The elements in the inputs are managed by MindSpore Lite so that users do not need to free inputs. +``` + +需è¦æ³¨æ„的是: +- MindSpore Lite的模型输入Tensorä¸çš„æ•°æ®æŽ’布必须是NHWC。 +- 模型的输入`input_buf`是用户从ç£ç›˜è¯»å–的,当拷è´ç»™æ¨¡åž‹è¾“å…¥Tensor以åŽï¼Œç”¨æˆ·éœ€è¦è‡ªè¡Œé‡Šæ”¾`input_buf`。 +- `GetInputs`å’Œ`GetInputsByName`方法返回的vectorä¸éœ€è¦ç”¨æˆ·é‡Šæ”¾ã€‚ + +## 图执行 + +### æ‰§è¡Œä¼šè¯ + +MindSpore Lite会è¯åœ¨è¿›è¡Œå›¾ç¼–译以åŽï¼Œå³å¯ä½¿ç”¨`LiteSession`çš„`RunGraph`进行模型推ç†ã€‚ + +### ç»‘æ ¸ + +MindSpore Liteå†…ç½®çº¿ç¨‹æ± æ”¯æŒç»‘æ ¸ã€è§£ç»‘æ“作,通过调用`BindThread`接å£ï¼Œå¯ä»¥å°†çº¿ç¨‹æ± ä¸çš„工作线程绑定到指定CPUæ ¸ï¼Œç”¨äºŽæ€§èƒ½åˆ†æžã€‚ç»‘æ ¸æ“作与创建`LiteSession`æ—¶ç”¨æˆ·æŒ‡å®šçš„ä¸Šä¸‹æ–‡æœ‰å…³ï¼Œç»‘æ ¸æ“ä½œä¼šæ ¹æ®ä¸Šä¸‹æ–‡ä¸çš„ç»‘æ ¸ç–略进行线程与CPU的亲和性设置。 + +需è¦æ³¨æ„çš„æ˜¯ï¼Œç»‘æ ¸æ˜¯ä¸€ä¸ªäº²å’Œæ€§æ“作,ä¸ä¿è¯ä¸€å®šèƒ½ç»‘定到指定的CPUæ ¸ï¼Œä¼šå—到系统调度的影å“ã€‚è€Œä¸”ç»‘æ ¸åŽï¼Œéœ€è¦åœ¨æ‰§è¡Œå®Œä»£ç åŽè¿›è¡Œè§£ç»‘æ“作,示例如下: + +```cpp +// Assume we have created a LiteSession instance named session. +session->BindThread(true); +auto ret = session->RunGraph(); +if (ret != mindspore::lite::RET_OK) { + std::cerr << "RunGraph failed" << std::endl; + delete session; + return -1; +} +session->BindThread(false); +``` + +> ç»‘æ ¸å‚数有两ç§é€‰æ‹©ï¼šå¤§æ ¸ä¼˜å…ˆå’Œä¸æ ¸ä¼˜å…ˆã€‚ +> åˆ¤å®šå¤§æ ¸å’Œä¸æ ¸çš„è§„åˆ™å…¶å®žæ˜¯æ ¹æ®CPUæ ¸çš„é¢‘çŽ‡è€Œä¸æ˜¯æ ¹æ®CPU的架构,对于没有大ä¸å°æ ¸ä¹‹åˆ†çš„CPU架构,在该规则下也å¯ä»¥åŒºåˆ†å¤§æ ¸å’Œä¸æ ¸ã€‚ +> ç»‘å®šå¤§æ ¸ä¼˜å…ˆæ˜¯æŒ‡çº¿ç¨‹æ± ä¸çš„çº¿ç¨‹ä»Žé¢‘çŽ‡æœ€é«˜çš„æ ¸å¼€å§‹ç»‘å®šï¼Œç¬¬ä¸€ä¸ªçº¿ç¨‹ç»‘å®šåœ¨é¢‘çŽ‡æœ€é«˜çš„æ ¸ä¸Šï¼Œç¬¬äºŒä¸ªçº¿ç¨‹ç»‘å®šåœ¨é¢‘çŽ‡ç¬¬äºŒé«˜çš„æ ¸ä¸Šï¼Œä»¥æ¤ç±»æŽ¨ã€‚ +> 对于ä¸æ ¸ä¼˜å…ˆï¼Œä¸æ ¸çš„å®šä¹‰æ˜¯æ ¹æ®ç»éªŒæ¥å®šä¹‰çš„,默认设定ä¸æ ¸æ˜¯ç¬¬ä¸‰å’Œç¬¬å››é«˜é¢‘çŽ‡çš„æ ¸ï¼Œå½“ç»‘å®šç–略为ä¸æ ¸ä¼˜å…ˆæ—¶ï¼Œä¼šä¼˜å…ˆç»‘定到ä¸æ ¸ä¸Šï¼Œå½“ä¸æ ¸ä¸å¤Ÿç”¨æ—¶ï¼Œä¼šå¾€å°æ ¸ä¸Šè¿›è¡Œç»‘定。 + +### 回调è¿è¡Œ + +Mindspore Liteå¯ä»¥åœ¨è°ƒç”¨`RunGraph`æ—¶ï¼Œä¼ å…¥ä¸¤ä¸ª`KernelCallBack`函数指针æ¥å›žè°ƒæŽ¨ç†æ¨¡åž‹ï¼Œç›¸æ¯”于一般的图执行,回调è¿è¡Œå¯ä»¥åœ¨è¿è¡Œè¿‡ç¨‹ä¸èŽ·å–é¢å¤–çš„ä¿¡æ¯ï¼Œå¸®åŠ©å¼€å‘者进行性能分æžã€Bug调试ç‰ã€‚é¢å¤–çš„ä¿¡æ¯åŒ…括: +- 当å‰è¿è¡Œçš„节点å称 +- 推ç†å½“å‰èŠ‚点å‰çš„输入输出Tensor +- 推ç†å½“å‰èŠ‚点åŽçš„输入输出Tensor + +### 使用示例 + +下é¢ç¤ºä¾‹ä»£ç 演示了使用`LiteSession`进行图编译,并定义了两个回调函数作为å‰ç½®å›žè°ƒæŒ‡é’ˆå’ŒåŽç½®å›žè°ƒæŒ‡é’ˆï¼Œä¼ 入到`RunGraph`接å£è¿›è¡Œå›žè°ƒæŽ¨ç†ï¼Œå¹¶æ¼”示了一次图编译,多次图执行的使用场景: + +```cpp +// Assume we have created a LiteSession instance named session and a Model instance named model before. +// The methods of creating model and session can refer to "Import Model" and "Create Session" two sections. +auto ret = session->CompileGraph(model); +if (ret != RET_OK) { + std::cerr << "CompileGraph failed" << std::endl; + // session and model need to be released by users manually. + delete (session); + delete (model); + return ret; +} +// Copy input data into the input tensor. Users can refer to the "Input Data" section. We uses random data here. +auto inputs = session->GetInputs(); +for (auto in_tensor : inputs) { + in_tensor = inputs.front(); + if (in_tensor == nullptr) { + std::cerr << "Input tensor is nullptr" << std::endl; + return -1; + } + // When calling the MutableData method, if the data in MSTensor is not allocated, it will be malloced. After allocation, the data in MSTensor can be considered as random data. + (void) in_tensor->MutableData(); +} +// Definition of callback function before forwarding operator. +auto before_call_back_ = [&](const std::vector<mindspore::tensor::MSTensor *> &before_inputs, + const std::vector<mindspore::tensor::MSTensor *> &before_outputs, + const session::CallBackParam &call_param) { + std::cout << "Before forwarding " << call_param.name_callback_param << std::endl; + return true; +}; +// Definition of callback function after forwarding operator. +auto after_call_back_ = [&](const std::vector<mindspore::tensor::MSTensor *> &after_inputs, + const std::vector<mindspore::tensor::MSTensor *> &after_outputs, + const session::CallBackParam &call_param) { + std::cout << "After forwarding " << call_param.name_callback_param << std::endl; + return true; +}; +// Call the callback function when performing the model inference process. +ret = session_->RunGraph(before_call_back_, after_call_back_); +if (ret != RET_OK) { + MS_LOG(ERROR) << "Run graph failed."; + return RET_ERROR; +} +// CompileGraph would cost much time, a better solution is calling CompileGraph only once and RunGraph much more times. +for (size_t i = 0; i < 10; i++) { + auto ret = session_->RunGraph(); + if (ret != RET_OK) { + MS_LOG(ERROR) << "Run graph failed."; + return RET_ERROR; + } +} +// session and model needs to be released by users manually. +delete (session); +delete (model); +``` + +## 获å–输出 + +### 获å–输出Tensor + +MindSpore Lite在执行完推ç†åŽï¼Œå°±å¯ä»¥èŽ·å–模型的推ç†ç»“果。 + +MindSpore Liteæ供四ç§æ–¹æ³•æ¥èŽ·å–模型的输出`MSTensor`。 +1. 使用`GetOutputsByNodeName`æ–¹æ³•ï¼Œæ ¹æ®æ¨¡åž‹è¾“出节点的å称æ¥èŽ·å–模型输出`MSTensor`ä¸è¿žæŽ¥åˆ°è¯¥èŠ‚点的Tensorçš„vector。 +2. 使用`GetOutputMapByNode`方法,直接获å–所有的模型输出节点的å称和连接到该节点的模型输出`MSTensor`的一个map。 +3. 使用`GetOutputByTensorName`æ–¹æ³•ï¼Œæ ¹æ®æ¨¡åž‹è¾“出Tensorçš„å称æ¥èŽ·å–对应的模型输出`MSTensor`。 +4. 使用`GetOutputMapByTensor`方法,直接获å–所有的模型输出`MSTensor`çš„å称和`MSTensor`指针的一个map。 + +当获å–到模型的输出Tensor,就需è¦å‘Tensorä¸å¡«å…¥æ•°æ®ã€‚通过`MSTensor`çš„`Size`方法æ¥èŽ·å–Tensor应该填入的数æ®å¤§å°ï¼Œé€šè¿‡`data_type`方法æ¥èŽ·å–`MSTensor`çš„æ•°æ®ç±»åž‹ï¼Œé€šè¿‡`MSTensor`çš„`MutableData`方法æ¥èŽ·å–å¯è¯»å†™çš„内å˜æŒ‡é’ˆã€‚ + +### 使用示例 + +下é¢ç¤ºä¾‹ä»£ç 演示了使用`GetOutputMapByNode`接å£èŽ·å–输出`MSTensor`,并打å°äº†æ¯ä¸ªè¾“出`MSTensor`çš„å‰å个数æ®æˆ–所有数æ®ï¼š + +```cpp +// Assume we have created a LiteSession instance named session before. +auto output_map = session->GetOutputMapByNode(); +// Assume that the model has only one output node. +auto out_node_iter = output_map.begin(); +std::string name = out_node_iter->first; +// Assume that the unique output node has only one output tensor. +auto out_tensor = out_node_iter->second.front(); +if (out_tensor == nullptr) { + std::cerr << "Output tensor is nullptr" << std::endl; + return -1; +} +// Assume that the data format of output data is float 32. +if (out_tensor->data_type() != mindspore::TypeId::kNumberTypeFloat32) { + std::cerr << "Output of lenet should in float32" << std::endl; + return -1; +} +auto *out_data = reinterpret_cast<float *>(out_tensor->MutableData()); +if (out_data == nullptr) { + std::cerr << "Data of out_tensor is nullptr" << std::endl; + return -1; +} +// Print the first 10 float data or all output data of the output tensor. +std::cout << "Output data: "; +for (size_t i = 0; i < 10 & i < out_tensor->ElementsNum(); i++) { + std::cout << " " << out_data[i]; +} +std::cout << std::endl; +// The elements in outputs do not need to be free by users, because outputs are managed by the MindSpore Lite. +``` + +需è¦æ³¨æ„的是,`GetOutputsByNodeName`ã€`GetOutputMapByNode`ã€`GetOutputByTensorName`å’Œ`GetOutputMapByTensor`方法返回的vector或mapä¸éœ€è¦ç”¨æˆ·é‡Šæ”¾ã€‚ + +下é¢ç¤ºä¾‹ä»£ç 演示了使用`GetOutputsByNodeName`接å£èŽ·å–输出`MSTensor`的方法: + +```cpp +// Assume we have created a LiteSession instance named session before. +// Assume that model has a output node named output_node_name_0. +auto output_vec = session->GetOutputsByNodeName("output_node_name_0"); +// Assume that output node named output_node_name_0 has only one output tensor. +auto out_tensor = output_vec.front(); +if (out_tensor == nullptr) { + std::cerr << "Output tensor is nullptr" << std::endl; + return -1; +} +``` + +下é¢ç¤ºä¾‹ä»£ç 演示了使用`GetOutputMapByTensor`接å£èŽ·å–输出`MSTensor`的方法: + +```cpp +// Assume we have created a LiteSession instance named session before. +auto output_map = session->GetOutputMapByTensor(); +// Assume that output node named output_node_name_0 has only one output tensor. +auto out_tensor = output_vec.front(); +if (out_tensor == nullptr) { + std::cerr << "Output tensor is nullptr" << std::endl; + return -1; +} +``` + +## 获å–ç‰ˆæœ¬å· +MindSpore Liteæ供了`Version`方法å¯ä»¥èŽ·å–版本å·ï¼ŒåŒ…å«åœ¨`include/version.h`头文件ä¸ï¼Œè°ƒç”¨è¯¥æ–¹æ³•å¯ä»¥å¾—到版本å·å—符串。 + +### 使用示例 + +下é¢ä»£ç 演示如何获å–MindSpore Lite的版本å·ï¼š +```cpp +#include "include/version.h" +std::string version = mindspore::lite::Version(); +``` + diff --git a/lite/tutorials/source_zh_cn/use/runtime_lite.md b/lite/tutorials/source_zh_cn/use/runtime_lite.md deleted file mode 100644 index 3c5162b116c3bc64c3b136abf21453d5c934855c..0000000000000000000000000000000000000000 --- a/lite/tutorials/source_zh_cn/use/runtime_lite.md +++ /dev/null @@ -1,11 +0,0 @@ -# Runtime使用指å—(Lite) - -<!-- TOC --> - -- [Runtime使用指å—(Lite)](#runtime使用指å—lite) - -<!-- /TOC --> - -<a href="https://gitee.com/mindspore/docs/blob/master/lite/tutorials/source_zh_cn/use/runtime_lite.md" target="_blank"><img src="../_static/logo_source.png"></a> - -