OpenHarmony provides native distributed AI capabilities. The AI subsystem consists of the following components:
- MindSpore Lite: an AI inference framework that provides unified APIs for AI inference.
- Neural Network Runtime (NNRt): an intermediate bridge that connects the inference framework and AI hardware.
## MindSpore Lite
MindSpore Lite is a built-in AI inference framework of OpenHarmony. It provides AI model inference capabilities for different hardware devices and end-to-end AI model inference solutions for developers to empower intelligent applications in all scenarios. Currently, MindSpore Lite has been widely used in applications such as image classification, target recognition, facial recognition, and character recognition.
**Figure 1** Development process for MindSpore Lite model inference
The MindSpore Lite development process consists of two phases:
- Model conversion
MindSpore Lite uses models in `.ms` format for inference. You can use the model conversion tool provided by MindSpore Lite to convert third-party framework models, such as TensorFlow, TensorFlow Lite, Caffe, and ONNX, into `.ms` models. For details, see [Converting Models for Inference](https://www.mindspore.cn/lite/docs/en/r1.8/use/converter_tool.html).
- Model inference
You can call the MindSpore Lite runtime APIs to implement model inference. The procedure is as follows:
1. Create an inference context by setting the inference hardware and number of threads.
2. Load the **.ms** model file.
3. Set the model input data.
4. Perform model inference, and read the output.
MindSpore Lite is built in the OpenHarmony standard system as a system component. You can develop AI applications based on MindSpore Lite in the following ways:
- Method 1: [Using MindSpore Lite JavaScript APIs to develop AI applications](./mindspore-guidelines-based-js.md). You directly call MindSpore Lite JavaScript APIs in the UI code to load the AI model and perform model inference. An advantage of this method is the quick verification of the inference effect.
- Method 2: [Using MindSpore Lite native APIs to develop AI applications](./mindspore-guidelines-based-native.md). You encapsulate the algorithm models and the code for calling MindSpore Lite native APIs into a dynamic library, and then use N-API to encapsulate the dynamic library into JavaScript APIs for the UI to call.
## Neural Network Runtime
Neural Network Runtime (NNRt) functions as a bridge to connect the upper-layer AI inference framework and bottom-layer acceleration chip, implementing cross-chip inference computing of AI models.
MindSpore Lite supports configuration of the NNRt backend, and therefore you can directly configure MindSpore Lite to use the NNRt hardware. The focus of this topic is about how to develop AI applications using MindSpore Lite. For details about how to use NNRt, see [Connecting the Neural Network Runtime to an AI Inference Framework](../napi/neural-network-runtime-guidelines.md).
# Using MindSpore Lite JavaScript APIs to Develop AI Applications
## Scenarios
## Scenarios
MindSpore Lite is an AI engine that implements AI model inference for different hardware devices. It has been used in a wide range of fields, such as image classification, target recognition, facial recognition, and character recognition.
You can use the JavaScript APIs provided by MindSpore Lite to directly integrate MindSpore Lite capabilities into the UI code. This way, you can quickly deploy AI algorithms for AI model inference.
This document describes the general development process for implementing MindSpore Lite model inference. For details about how to use native APIs to implement model inference, see [Using MindSpore Lite for Model Inference](../napi/mindspore-lite-guidelines.md).
## Basic Concepts
## Basic Concepts
...
@@ -27,16 +25,14 @@ APIs involved in MindSpore Lite model inference are categorized into context API
...
@@ -27,16 +25,14 @@ APIs involved in MindSpore Lite model inference are categorized into context API
## How to Develop
## How to Develop
The development process consists of the following main steps:
Assume that you have prepared a model in the **.ms** format. The key steps in model inference are model reading, model building, model inference, and memory release. The development procedure is described as follows:
1. Create a context, and set parameters such as the number of runtime threads and device type.
2. Load the model. In this example, the model is read from the file.
3. Load data. Before executing a model, you need to obtain the model input and then fill data in the input tensor.
4. Perform model inference by calling **predict**, and read the output.
1. Prepare the required model. You can download the required model directly or obtain the model by using the model conversion tool. The required data is read from the `bin` file.
- If the downloaded model is in the `.ms` format, you can use it directly for inference. This document uses `mnet.caffemodel.ms` as an example.
- If the downloaded model uses a third-party framework, such as TensorFlow, TensorFlow Lite, Caffe, or ONNX, you can use the [model conversion tool](https://www.mindspore.cn/lite/docs/en/r2.0/use/downloads.html#1-8-1) to convert it to the `.ms` format.
2. Create a context, and set parameters such as the number of runtime threads and device type.
3. Load the model. In this example, the model is read from the file.
4. Load data. Before executing a model, you need to obtain the model input and then fill data in the input tensor.
5. Perform inference and print the output. Call the **predict** API to perform model inference.
$ hdc shell aa start -a EntryAbility -b com.example.myapptfjs
$ hdc shell aa start -a EntryAbility -b com.example.myapptfjs
```
```
2. Use the hdc tool to connect to the rk3568 development board and push `mnet.caffemodel.ms` to the sandbox directory on the device. `mnet\_caffemodel\_nhwc.bin` is stored in the `rawfile` directory of the local project.
2. Use hdc to connect to the device, and push **mnet.caffemodel.ms** to the sandbox directory on the device. **mnet\_caffemodel\_nhwc.bin** is stored in the **rawfile** directory of the local project.
3. Click **Test\_MSLiteModel\_predict** on the screen of the rk3568 development board to run the test case. The following information is displayed in the HiLog printing result:
3. Click **Test\_MSLiteModel\_predict** on the device screen to run the test case. The following information is displayed in the HiLog printing result:
```shell
```shell
08-27 23:25:50.278 31782-31782/? I C03d00/JSAPP: =========MSLITE predict start=====
08-27 23:25:50.278 31782-31782/? I C03d00/JSAPP: =========MSLITE predict start=====
# Using MindSpore Lite Native APIs to Develop AI Applications
## Scenarios
You can use the native APIs provided by MindSpore Lite to deploy AI algorithms and provides APIs for the UI layer to invoke the algorithms for model inference. A typical scenario is the AI SDK development.
## Basic concepts
-[N-API](../reference/native-lib/third_party_napi/napi.md): a set of native APIs used to build JavaScript components. N-APIs can be used to encapsulate libraries developed using C/C++ into JavaScript modules.
## Preparing the Environment
- Install DevEco Studio 3.1.0.500 or later, and update the SDK to API version 10 or later.
## How to Develop
1. Create a native C++ project.
Open DevEco Studio, choose **File** > **New** > **Create Project** to create a native C++ template project. By default, the **entry/src/main/** directory of the created project contains the **cpp/** directory. You can store C/C++ code in this directory and provide JavaScript APIs for the UI layer to call the code.
2. Compile the C++ inference code.
Assume that you have prepared a model in the **.ms** format.
Before using the Native APIs provided by MindSpore Lite for development, you need to reference the corresponding header files.
target_link_libraries(mslite_napi PUBLIC mindspore_lite_ndk)# MindSpore Lite dynamic library to link
target_link_libraries(mslite_napi PUBLIC hilog_ndk.z)
target_link_libraries(mslite_napi PUBLIC rawfile.z)
target_link_libraries(mslite_napi PUBLIC ace_napi.z)
```
3. Use N-APIs to encapsulate C++ dynamic libraries into JavaScript modules.
Create the **libmslite_api/** subdirectory in **entry/src/main/cpp/types/**, and create the **index.d.ts** file in the subdirectory. The file content is as follows:
```js
exportconstrunDemo:(a:String,b:Object)=>number;
```
Use the preceding code to define the JavaScript API `runDemo()`.
In addition, add the **oh-package.json5** file to associate the API with the **.so** file to form a complete JavaScript module.
```json
{
"name":"libmslite_napi.so",
"types":"./index.d.ts"
}
```
4. Invoke the encapsulated MindSpore module in the UI code.
In **entry/src/ets/MainAbility/pages/index.ets**, define the **onClick()** event and call the encapsulated **runDemo()** API in the event callback.
```js
importmsliteNapifrom'libmslite_napi.so'// Import the msliteNapi module.
// Certain code omitted
// Trigger the event when the text on the UI is tapped.
The traffic management module allows you to query real-time or historical data traffic by the specified network interface card (NIC) or user ID (UID).
Its functions include:
- Obtaining real-time traffic data by NIC or UID
- Obtaining historical traffic data by NIC or UID
- Subscribing to traffic change events by NIC or UID
> **NOTE**
> To maximize the application running efficiency, most API calls are called asynchronously in callback or promise mode. The following code examples use the callback mode. For details about the APIs, see [Traffic Management](../reference/apis/js-apis-net-statistics.md).
The following describes the development procedure specific to each application scenario.
## Available APIs
For the complete list of APIs and example code, see [Traffic Management](../reference/apis/js-apis-net-statistics.md).
| Type| API| Description|
| ---- | ---- | ---- |
| ohos.net.statistics | getIfaceRxBytes(nic: string, callback: AsyncCallback\<number>): void; |Obtains the real-time downlink data traffic of the specified NIC. |
| ohos.net.statistics | getIfaceTxBytes(nic: string, callback: AsyncCallback\<number>): void; |Obtains the real-time uplink data traffic of the specified NIC. |
| ohos.net.statistics | getCellularRxBytes(callback: AsyncCallback\<number>): void; |Obtains the real-time downlink data traffic of the cellular network.|
| ohos.net.statistics | getCellularTxBytes(callback: AsyncCallback\<number>): void; |Obtains the real-time uplink data traffic of the cellular network.|
| ohos.net.statistics | getAllRxBytes(callback: AsyncCallback\<number>): void; |Obtains the real-time downlink data traffic of the all NICs. |
| ohos.net.statistics | getAllTxBytes(callback: AsyncCallback\<number>): void; |Obtains the real-time uplink data traffic of the all NICs. |
| ohos.net.statistics | getUidRxBytes(uid: number, callback: AsyncCallback\<number>): void; |Obtains the real-time downlink data traffic of the specified application. |
| ohos.net.statistics | getUidTxBytes(uid: number, callback: AsyncCallback\<number>): void; |Obtains the real-time uplink data traffic of the specified application. |
| ohos.net.statistics | getTrafficStatsByIface(ifaceInfo: IfaceInfo, callback: AsyncCallback\<NetStatsInfo>): void; |Obtains the historical data traffic of the specified NIC. |
| ohos.net.statistics | getTrafficStatsByUid(uidInfo: UidInfo, callback: AsyncCallback\<NetStatsInfo>): void; |Obtains the historical data traffic of the specified application. |
| ohos.net.statistics | on(type: 'netStatsChange', callback: Callback\<{ iface: string, uid?: number }>): void |Subscribes to traffic change events.|
| ohos.net.statistics | off(type: 'netStatsChange', callback?: Callback\<{ iface: string, uid?: number }>): void; |Unsubscribes from traffic change events.|
## Obtaining Real-Time Traffic Data by NIC or UID
1. Obtain the real-time data traffic of the specified NIC.
2. Obtain the real-time data traffic of the cellular network.
3. Obtain the real-time data traffic of all NICs.
4. Obtain the real-time data traffic of the specified application.
```js
// Import the statistics namespace from @ohos.net.statistics.
importstatisticsfrom'@ohos.net.statistics'
// Obtain the real-time downlink data traffic of the specified NIC.
// Unsubscribe from traffic change events. You can pass the callback of the **on** function if you want to unsubscribe from a certain type of event. If you do not pass the callback, you will unsubscribe from all events.
The **statistics** module provides APIs to query real-time or historical data traffic by the specified network interface card (NIC) or user ID (UID).
The **statistics** module provides APIs to query real-time or historical data traffic by the specified network interface card (NIC) or user ID (UID).
> **NOTE**
> **NOTE**
>
> The initial APIs of this module are supported since API version 10. Newly added APIs will be marked with a superscript to indicate their earliest API version.
> The initial APIs of this module are supported since API version 10. Newly added APIs will be marked with a superscript to indicate their earliest API version.
## Modules to Import
## Modules to Import
...
@@ -589,7 +588,7 @@ For details about the error codes, see [Traffic Management Error Codes](../error
...
@@ -589,7 +588,7 @@ For details about the error codes, see [Traffic Management Error Codes](../error
on(type: 'netStatsChange', callback: Callback\<{ iface: string, uid?: number }>): void
on(type: 'netStatsChange', callback: Callback\<{ iface: string, uid?: number }>): void
Subscribes to data traffic change events.
Subscribes to traffic change events.
**System API**: This is a system API.
**System API**: This is a system API.
...
@@ -602,7 +601,7 @@ Subscribes to data traffic change events.
...
@@ -602,7 +601,7 @@ Subscribes to data traffic change events.
| type | string | Yes | Event type. This field has a fixed value of **netStatsChange**.|
| type | string | Yes | Event type. This field has a fixed value of **netStatsChange**.|
| callback | Callback\<{ iface: string, uid?: number }\> | Yes | Callback invoked when the data traffic changes.<br>**iface**: NIC name.<br>**uid**: application UID.|
| callback | Callback\<{ iface: string, uid?: number }\> | Yes | Callback invoked when the traffic changes.<br>**iface**: NIC name.<br>**uid**: application UID.|
**Error codes**
**Error codes**
...
@@ -628,7 +627,7 @@ For details about the error codes, see [Traffic Management Error Codes](../error
...
@@ -628,7 +627,7 @@ For details about the error codes, see [Traffic Management Error Codes](../error
off(type: 'netStatsChange', callback?: Callback\<{ iface: string, uid?: number }>): void;
off(type: 'netStatsChange', callback?: Callback\<{ iface: string, uid?: number }>): void;
Unsubscribes from data traffic change events.
Unsubscribes from traffic change events.
**System API**: This is a system API.
**System API**: This is a system API.
...
@@ -641,7 +640,7 @@ Unsubscribes from data traffic change events.
...
@@ -641,7 +640,7 @@ Unsubscribes from data traffic change events.
| type | string | Yes | Event type. This field has a fixed value of **netStatsChange**.|
| type | string | Yes | Event type. This field has a fixed value of **netStatsChange**.|
| callback | Callback\<{ iface: string, uid?: number }\> | No | Callback invoked when the data traffic changes.<br>**iface**: NIC name.<br>uid: application UID.|
| callback | Callback\<{ iface: string, uid?: number }\> | No | Callback invoked when the traffic changes.<br>**iface**: NIC name.<br>uid: application UID.|
## About OpenHarmony<a name="section6370143622110"></a>
OpenHarmony is an open-source project launched by the OpenAtom Foundation. The purpose of this project is to build an open, distributed operating system \(OS\) framework for smart IoT devices in the full-scenario, full-connectivity, and full-intelligence era.
## About OpenHarmony
The open-source code repositories are available at [https://openharmony.gitee.com](https://openharmony.gitee.com).
OpenHarmony is an open source project launched by the OpenAtom Foundation. The purpose of this project is to build an open, distributed operating system (OS) framework for smart IoT devices in the full-scenario, full-connectivity, and full-intelligence era.
## Overview of Source Code Acquisition<a name="section12763342204"></a>
The OpenHarmony source code is open to you as [HPM parts](../hpm-part/hpm-part-about.md), which can be obtained in any of the following ways:
-**Method 1**: Acquire the source code from the Gitee code repository. You can use the **repo** or **git** tool to download the latest code from the code repository.
The open source code repositories are available at [https://openharmony.gitee.com](https://openharmony.gitee.com).
-**Method 2**: Acquire the source code from [DevEco Marketplace](https://repo.harmonyos.com/#/en/home). Visit [DevEco Marketplace](https://repo.harmonyos.com/#/en/home), search for your desired open-source distribution, and download the bundle list \(or customize bundles and download the bundle list\). Then use the **hpm-cli** tool to download and install the bundles and compilation toolchain on your local PC.
-**Method 3**: Download the compressed file of a distribution from a mirror site. This method provides a fast download speed, so you can also use this method for obtaining the source code of an earlier version.
-**Method 4**: Acquire the source code from the GitHub image repository. You can use the **repo** or **git** tool to download the latest code from the code repository.
## Method 1: Acquiring Source Code from the Gitee Code Repository<a name="section537312010229"></a>
### When to Use<a name="section10881513459"></a>
## Overview of Source Code Acquisition
- You want to establish a baseline based on stable OpenHarmony releases and distribute the baseline to your customers.
This document describes how to acquire OpenHarmony source code and provides its directory structure. The OpenHarmony source code is open to you as [HPM parts](../hpm-part/hpm-part-about.md), which can be obtained in any of the following ways:
- You have interconnected your software with OpenHarmony and need official certification from OpenHarmony.
-**Method 1**: Acquire the source code from the Gitee code repository. You can use the **repo** or **git** tool to download the latest code from the code repository.
- You want to contribute code to the OpenHarmony community after obtaining official OpenHarmony certification for chips, modules, and applications.
-**Method 2**: Acquire the source code from [DevEco Marketplace](https://repo.harmonyos.com/#/en/home). Visit [DevEco Marketplace](https://repo.harmonyos.com/#/en/home), search for your desired open source distribution, and download the component list (or customize components and download the component list). Then use the **hpm-cli** tool to download and install the components and compilation toolchain on your local PC.
- You need to rectify OpenHarmony issues.
-**Method 3**: Download the compressed file of a distribution from a mirror site. This method provides a fast download speed, so you can also use this method for obtaining the source code of an earlier version.
- You want to learn OpenHarmony source code.
-**Method 4**: Acquire the source code from the GitHub image repository. You can use the **repo** or **git** tool to download the latest code from the code repository.
## Method 1: Acquiring Source Code from the Gitee Code Repository
1. Register your account with Gitee.
2. Register an SSH public key for access to Gitee.
3. Install the [git client](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and [git-lfs](https://gitee.com/vcs-all-in-one/git-lfs?_from=gitee_search#downloading), and configure basic user information.
4. Run the following commands to install the **repo** tool:
- You want to establish a baseline based on stable OpenHarmony releases and distribute the baseline to your customers.
- You have interconnected your software with OpenHarmony and need official certification from OpenHarmony.
- You want to contribute code to the OpenHarmony community after obtaining official OpenHarmony certification for chips, modules, and applications.
- You need to rectify OpenHarmony issues.
- You want to learn OpenHarmony source code.
### Prerequisites
1. Register your account with Gitee.
2. Register an SSH public key for access to Gitee.
3. Install the [Git client](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and [git-lfs](https://gitee.com/vcs-all-in-one/git-lfs?_from=gitee_search#downloading), and configure basic user information.
>Download the release code, which is more stable, if you want to develop commercial functionalities. Download the master code if you want to get quick access to the latest features for your development.
> Download the release code, which is more stable, if you want to develop commercial functionalities. Download the master code if you want to get quick access to the latest features for your development.
-**Obtaining OpenHarmony release code**
-**Obtaining OpenHarmony release code**
For details about how to obtain the source code of an OpenHarmony release, see the [Release Notes](../../release-notes/Readme.md).
For details about how to obtain the source code of an OpenHarmony release, see the [Release Notes](../../release-notes/Readme.md).
-**Obtaining OpenHarmony master code**
-**Obtaining OpenHarmony master code**
Method 1 (recommended): Use the **repo** tool to download the source code over SSH. (You must have registered an SSH public key for access to Gitee.)
Method 1 \(recommended\): Use the **repo** tool to download the source code over SSH. \(You must have registered an SSH public key for access to Gitee.\)
Method 2: Use the **repo** tool to download the source code over HTTPS.
## Method 2: Acquiring Source Code from DevEco Marketplace<a name="section463013147412"></a>
### When to Use
### When to Use<a name="section26661067443"></a>
If OpenHarmony is new to you, sample solutions are helpful to your development. You can obtain an open source distribution from [DevEco Marketplace](https://repo.harmonyos.com/#/en/home), or customize a distribution by adding or deleting components of an open source distribution. Then use the **hpm-cli** tool to download and install the components and compilation toolchain on your local PC.
If OpenHarmony is new to you, sample solutions are helpful to your development. You can obtain an open-source distribution from [DevEco Marketplace](https://repo.harmonyos.com/#/en/home), or customize a distribution by adding or deleting bundles of an open-source distribution. Then use the **hpm-cli** tool to download and install the bundles and compilation toolchain on your local PC.
1. Read carefully the information about the distribution to learn its application scenarios, features, components, usage, and customization methods.
2. Click **Download** if you want to download the distribution to your local PC.
3. Click **Device component tailoring** if you want to add or delete components of the distribution.
2. Learn more about the distribution.
**Figure 2** Example distribution
1. Read carefully the information about the distribution to learn its application scenarios, features, bundles, usage, and customization methods, as shown in the following figure.
1. Decompress the downloaded code structure file using CMD on Windows (or shell in Linux).
1. Decompress the downloaded code structure file using CMD on Windows (or shell in Linux).
2. In the generated directory, run the **hpm install** command to download and install bundles. If the **Install successful** message is displayed, the command has been executed successfully.
2. In the generated directory, run the **hpm install** command to download and install components. If the **Install successful** message is displayed, the command has been executed successfully.
3. Obtain the bundles. The downloaded bundles are stored in the **ohos\_bundles** folder under the project directory. \(The source code of some bundles will be copied to a specified directory after the bundles are installed.\)
3. The downloaded components will be stored in the **ohos_bundles** folder under the project directory. (The source code of some components will be copied to a specified directory after the components are installed.)
## Method 3: Acquiring Source Code from a Mirror Site<a name="section1186691118430"></a>
## Method 3: Acquiring Source Code from a Mirror Site
To ensure the download performance, you are advised to download the source code or the corresponding solution from the image library of the respective site listed in the table below.
To ensure the download performance, you are advised to download the source code or the corresponding solution from the image library of the respective site listed in the table below.
The table below provides only the sites for downloading the latest OpenHarmony LTS code. For details about how to obtain the source code of earlier versions, see the [Release Notes](../../release-notes/Readme.md).
The table below provides only the sites for downloading the latest OpenHarmony LTS code. For details about how to obtain the source code of earlier versions, see the [Release Notes](../../release-notes/Readme.md).
## Method 4: Acquiring Source Code from the GitHub Image Repository
>
> **NOTE**<br>
> The image repository is synchronized at 23:00 (UTC +8:00) every day.
> The image repository is synchronized at 23:00 (UTC +8:00) every day.
Method 1 \(recommended\): Use the **repo** tool to download the source code over SSH. \(You must have registered an SSH public key for access to GitHub. For details, see [Adding a new SSH key to your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account).\)
Method 1 (recommended): Use the **repo** tool to download the source code over SSH. (You must have registered an SSH public key for access to GitHub. For details, see [Adding a new SSH key to your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account).)
In versions earlier than 4.0.6.2, only the values of string resources can be directly obtained. In 4.0.6.2 or later, the values of string resources can be formatted based on the input arguments for enhanced query.
The following error codes are added:
9001007 If the resource obtained by resId formatting error.
9001008 If the resource obtained by resName formatting error.
**Sample Code**
The following uses **getStringSync** as an example. Before the change, only example 1 is supported. After the change, both example 1 and example 2 are supported.