1. Add the KWS configuration file to the **//foundation/ai/engine/services/common/protocol/plugin\_config/plugin\_config\_ini/** directory.
```
[base]
supported_boards = hi3516dv300
related_sessions = asr_keyword_spotting+20001002
// Naming rule: [algorithm name+algorithm version], for example, [asr_keyword_spotting+20001002]
[asr_keyword_spotting+20001002]
AID = asr_keyword_spotting
VersionCode = 20001002
VersionName = 2.00.01.002
XPU = NNIE
District = China
// Location of the complied .so file of the plug-in
FullPath = /usr/lib/libasr_keyword_spotting.so
Chipset = ALL
ChkSum = ''
Key = ''
```
2. Add the type ID of the KWS algorithm to the **aie\_algorithm\_type.h** file in the **//foundation/ai/engine/services/common/protocol/plugin\_config/** directory.
```
// Make sure that the type ID maps to the sequence number in ALGORITHM_TYPE_ID_LITS.
const int ALGORITHM_TYPE_KWS = 3;
```
3. Add the name of the KWS algorithm and its sequence number in **ALGORITHM\_TYPE\_ID\_LITS** to the **aie\_plugin\_info.h** file in the **//foundation/ai/engine/services/server/plugin\_manager/include/** directory.
// Add the sequence number of the KWS algorithm to ALGORITHM_TYPE_ID_LITS, so that the name of the KWS algorithm can be obtained based on the sequence number.
// Make sure that the algorithm name maps to the sequence number in ALGORITHM_TYPE_ID_LITS.
1. Add the API of the KWS plug-in to the **//foundation/ai/engine/services/server/plugin** directory. This API is used to call AI capabilities. The following code snippet is an example API implementation of the KWS plug-in. The reference code is available at the **//foundation/ai/engine/services/server/plugin/asr/keyword\_spotting** directory.
```
#include "plugin/i_plugin.h
class KWSPlugin : public IPlugin {
public:
KWSPlugin();
~KWSPlugin();
const long long GetVersion() const override;
const char* GetName() const override;
const char* GetInferMode() const override;
int32_t Prepare(long long transactionId, const DataInfo &inputInfo, DataInfo &outputInfo) override;
int32_t Release(bool isFullUnload, long long transactionId, const DataInfo &inputInfo) override;
};
```
The preceding code implements the **IPlugin** API provided by the server. The following table shows the mapping between the client APIs and the plug-in APIs.
**Table 1** Mapping between the client APIs and the plug-in APIs
| Client API | Plug-in API | Description |
| ---------- | ----------- | ----------- |
| AieClientPrepare | Prepare | Initializes the inference algorithm plug-in. For KWS, this API loads the KWS model from the fixed location (/sdcard/wenwen_inst.wk) to the memory. |
| AieClientSyncProcess | SyncProcess | Executes the inference algorithm synchronously. For KWS, this API synchronously executes the audio inference algorithm to determine whether the specified wakeup keyword exists in the audio. |
| AieClientAsyncProcess | AsyncProcess | Executes the inference algorithm asynchronously. Currently, this API is not used in KWS. However, you can implement the API based on your use case. |
| AieClientSetOption | SetOption | Sets algorithm-related configuration items, such as the confidence threshold and delay. Currently, this API is not used in KWS. However, you can implement the API based on your use case. |
| AieClientGetOption | GetOption | Obtains algorithm-related configuration items. For KWS, this API can obtain the input and output scale of the KWS model. The input scale is the MFCC feature (fixed value: 4000) required by the KWS model, and the output scale is the confidence (fixed value: 2) of the result. |
| AieClientRelease | Release | Releases the algorithm model. For KWS, this API releases the specified algorithm model and clears the dynamic memory in the feature processor. |
>- The **AieClientInit** and **AieClientDestroy** APIs are used to connect to and disconnect from the server, respectively. They are not called in the plug-in algorithm and therefore do not need to be defined in the plug-in.
>- The KWS plug-in needs to use the **PLUGIN\_INTERFACE\_IMPL** statement to expose the function pointer. Otherwise, the plug-in cannot be properly loaded.
1. Add the API of the KWS SDK to the **//foundation/ai/engine /interfaces/kits** directory. This API can be called by third-party applications. The following code snippet is an example API for the KWS SDK. The reference code is available at the **//foundation/ai/engine /interfaces/kits/asr/keyword\_spotting** directory.
// Destroy the KWS SDK instance to release the session engaged with the plug-in.
int32_t Destroy();
};
```
2. Add the API implementation of the SDK to the **//foundation/ai/engine/services/client/algorithm\_sdk** directory and call the APIs provided by the client to use the algorithm plug-in capabilities. The following code snippet is an example implementation of the **create** method in the API of the KWS SDK. For more details, see the reference code at **//foundation/ai/engine/services/client/algorithm\_sdk/asr/keyword\_spotting**.
```
int32_t KWSSdk::KWSSdkImpl::Create()
{
if (kwsHandle_ != INVALID_KWS_HANDLE) {
HILOGE("[KWSSdkImpl]The SDK has been created");
return KWS_RETCODE_FAILURE;
}
if (InitComponents() != RETCODE_SUCCESS) {
HILOGE("[KWSSdkImpl]Fail to init sdk components");
return KWS_RETCODE_FAILURE;
}
// Call the AieClientInit API provided by the client to initialize the engine service and activate IPC call.
HILOGE("[KWSSdkImpl]Get handle from inputInfo failed");
return KWS_RETCODE_FAILURE;
}
return KWS_RETCODE_SUCCESS;
}
```
The preceding code is the specific API implementation. The **create** function in the API of the KWS SDK calls the open **AieClientInit** and **AieClientPrepare** APIs provided by the client to connect to the server and load the algorithm model. For details, see the implementation of the **create** method in following sections.
>The sequence for the SDK to call client APIs: **AieClientInit** -\> **AieClientPrepare** -\> **AieClientSyncProcess** or **AieClientAsyncProcess** -\> **AieClientRelease** -\> **AieClientDestroy**. An exception will be thrown if the call sequence is violated.
# Development Examples<a name="EN-US_TOPIC_0000001077767514"></a>
For your better understanding, a KWS application is used as an example to walk you through the development process. You can develop the KWS SDK and plug-in based on the AI engine framework on the Hi3516D V300 development board, compile an image for the new version, and burn the image into the version. Then, develop an application with the KWS function. The application can receive external audio and pass the audio to the SDK API. If the audio contains specified keywords, the application will be able to recognize these keywords and print them in the command line.
This example uses a fixed keyword **Hi, xiaowen** for illustration. If the input audio contains **Hi, xiaowen**, the application prints **\[Hi, xiaowen\]**; otherwise, the application prints **\[UNKNOWN\]**.
The SDK identifies the plug-in type based on **algorithmVersion** and **algorithmType** in the **AlgorithmInfo** structure so it can call the plug-in capabilities. Therefore, you need to develop the plug-in configuration file as follows:
1. Add the plug-in configuration file to the **//foundation/ai/engine/services/common/protocol/plugin\_config/plugin\_config\_ini/** directory.
2. Add the algorithm type to the **aie\_algorithm\_type.h** file in the **//foundation/ai/engine/services/common/protocol/plugin\_config/** directory.
3. Add the name of the KWS algorithm and its sequence number in **ALGORITHM\_TYPE\_ID\_LITS** to the **aie\_plugin\_info.h** file in the **//foundation/ai/engine/services/server/plugin\_manager/include/** directory.
For details about the development process, see the development example for the KWS configuration file.
# Development Guidelines<a name="EN-US_TOPIC_0000001090475723"></a>
To access the AI engine framework, you need to develop the SDKs and plug-ins shown in [Figure 1](subsys-aiframework-guide.md#fig143186187187). In this way, you can call the APIs provided by the SDKs to call the algorithm capabilities of plug-ins to implement lifecycle management and on-demand deployment of AI capabilities.
# AI Engine Framework<a name="EN-US_TOPIC_0000001077309802"></a>
The AI subsystem of OpenHarmony provides native distributed AI capabilities. At the heart of the subsystem is a unified AI engine framework, which implements quick integration of AI algorithm plug-ins. The framework consists of the plug-in management, module management, and communication management modules, fulfilling lifecycle management and on-demand deployment of AI algorithms. Plug-in management implements lifecycle management, on-demand deployment, and quick integration of AI algorithm plug-ins. Module management implements task scheduling and client instance management. Communication management implements inter-process communication \(IPC\) between the client and server and data transmission between the AI engine and plug-ins. Under this framework, AI algorithm APIs will be standardized to facilitate distributed calling of AI capabilities. In addition, unified inference APIs will be provided to adapt to different inference framework hierarchies. [Figure 1](#fig143186187187) shows the AI engine framework.
**Figure 1** AI engine framework<aname="fig143186187187"></a>
Code of the AI engine framework consists of three parts: **client**, **server**, and **common**. The client module provides the server connection management function. The northbound SDK needs to encapsulate and call the public APIs provided by the client in the algorithm's external APIs. The server module provides functions such as plug-in loading and task management. Plug-ins are integrated using the plug-in APIs provided by the server. The common module provides platform-related operation methods, engine protocols, and tool classes for other modules.
[Figure 1](#fig171811112818) shows the code dependency between modules of the AI engine framework.
## Recommendation: Develop plug-ins and northbound SDKs in the directories specified by the AI engine.<a name="section17176374131"></a>
In the overall planning of the AI engine framework, northbound SDKs are a part of the client, and plug-ins are called by the server and are considered a part of the server. Therefore, the following directories have been planned for plug-in and northbound SDK development in the AI engine framework:
## Rule: Store all external APIs provided by plug-ins in the **interfaces/kits** directory of the AI subsystem.<a name="section2551029111312"></a>
The AI subsystem exposes its capabilities through external APIs of northbound SDKs. According to API management requirements of OpenHarmony, store all external APIs of northbound SDKs in the **interfaces/kits** directory of the subsystem. Currently, the external APIs of plug-ins of the AI subsystem are stored in the following directory: **//foundation/ai/engine/interface/kits**. You can add a sub-directory for each newly added plug-in in this directory. For example, if you add a CV plug-in, then store its external APIs in the **//foundation/ai/engine/interfaces/kits/cv** directory.
## Rule: Make sure that plug-in compilation results are stored in the **/usr/lib** directory.<a name="section97021558121310"></a>
Plug-in loading on the server uses the dlopen mode and can only be performed in the **/usr/lib** directory. Therefore, when compiling the **.so** file of a plug-in, set the output directory as **/usr/lib** in the compilation configuration file.
# API Development<a name="EN-US_TOPIC_0000001096100171"></a>
## Rule: Encapsulate the external APIs provided by the client based on the algorithm call sequence. For the SDK of an asynchronous plug-in, implement the **IClientCb** callback API provided by the client.<a name="section15872017171616"></a>
The external APIs provided by the client of the AI engine include **AieClientInit**, **AieClientPrepare**, **AieClientSyncProcess**, **AieClientAsyncProcess**, **AieClientRelease**, **AieClientDestroy**, **AieClientSetOption**, and **AieClientGetOption**. The SDK needs to encapsulate at least the following five APIs in sequence: **AieClientInit**, **AieClientPrepare**, **AieClientSyncProcess**\(or **AieClientAsyncProcess**\), **AieClientRelease**, and **AieClientDestroy**. Otherwise, a call failure or memory leakage may occur. For example, if the **AieClientprepare** API is omitted during encapsulation, the server cannot load the plug-in. As a result, APIs that follow it cannot be called.
For an asynchronous plug-in, the SDK needs to implement the **IClientCb** API to receive the algorithm inference result from the client and return the result to the third-party caller.
## Rule: Save all common data related to client interaction in the SDK during API implementation.<a name="section011283741612"></a>
The client of the AI engine uses the singleton pattern for API implementation. If the client is connecting to multiple SDKs, each SDK needs to store all common data exchanged with the client so that they can connect to the server to perform operations such as task inference and return the result. Common data usually includes **clientInfo**, **algorithmInfo**, and **configInfo**, which are defined in the SDK's member variables.
## Recommendation: Enable the SDK to implement the **IServiceDeadCb** API defined by the client.<a name="section1199125331613"></a>
The processes running on the server are system resident processes. The server provides services for clients by way of system capabilities. The **IServiceDeadCb** API is called if a server process is abnormally killed. The SDK can implement related operations in this API, for example, stopping process call or restarting the server.
The following is an example of **IServiceDeadCb** API implementation:
As shown above, the SDK can implement its own operations in the **OnServiceDead\(\)** function, for example, stopping API call.
## Rule: Convert dedicated algorithm data into common data of the AI engine if the SDK and plug-ins need to use the codec module.<a name="section93139389171"></a>
For plug-ins, inference data is transmitted by the third-party caller to them through the client and server. The required data type varies according to algorithms. For example, the CV algorithm requires image data, and the ASR algorithm requires audio data. To address this issue, the AI engine provides the codec capabilities to convert different types of data into common data that can be used by it.
The encoded data is as follows:
```
struct DataInfo {
unsigned char *data;
int length;
} DataInfo;
```
As shown above, **DataInfo** consists of two variables: a pointer to the data memory, and the data length.
To use the APIs of the AI engine framework, you need to:
1. Add the dependency header file **utils/encdec/include/encdec.h**.
2. Add the dependency items in the **build.gn** file.
Add **"//foundation/ai/engine/services/common"** to **include\_dirs**.
Add **"//foundation/ai/engine/services/common/utils/encdec:encdec"** to **deps**.
3. Convert different types of data through codec. The following is an example:
```
// Example function for encoding: arg1, arg2, and arg3 are variables to be encoded, and dataInfo is the encoding result.
retCode = ProcessEncode(dataInfo, arg1, arg2, arg3) // The number of parameters can be flexible.
// Example function for decoding: dataInfo is the data to be decoded, and arg1, arg2, and arg3 are the decoding result.
retCode = ProcessDecode(dataInfo, arg1, arg2, arg3) // The number of parameters can be flexible.
```
Note:
- The sequence of parameters must be the same during encoding and decoding.
- After encoding, the memory used by **dataInfo** needs to be manually released by the caller.
- The memory is managed and released separately on the server and the client.
- If a pointer contains the shared memory, no extra processing is required.
- If other types of pointers are used, you need to dereference them before using **ProcessEncode** or **ProcessDecode**.
- The codec module has not been adapted to the **class** data type and therefore it is not recommended.
## Rule: Release the memory used by the encoded or decoded parameters in the SDK. Otherwise, a memory leakage occurs.<a name="section1698441814183"></a>
Encoding is essentially a process of encapsulating different types of data in the same memory space and then encapsulating the start address and length of the memory into the body. The plug-in is unable to release the memory that has been allocated to output parameter data returned to the SDK through encoding. To obtain the data, the SDK first needs to release the memory.
The following is an example of releasing the memory:
## Rule: Enable plug-ins to implement the **IPlugin** API defined by the server and use the **PLUGIN\_INTERFACE\_IMPL** statement to provide the function pointer for external systems.<a name="section20850717196"></a>
The server manages a variety of plug-ins, and the API implementation logic varies according to plug-ins. To unify the plug-in loading process, the AI engine provides the **IPlugin** API. In the runtime environment, a plug-in is loaded as a dynamic link library \(DLL\) by the AI engine framework in dlopen mode. Therefore, the plug-in needs to use the **PLUGIN\_INTERFACE\_IMPL** statement to expose the function pointer. Otherwise, the plug-in cannot be properly loaded.
## Rule: Use the unified data channel provided by the AI engine for plug-ins.<a name="section1493821732019"></a>
The AI engine provides a unified data channel between the server and plug-ins to send inference requests from the SDK and returned results from plug-ins. Plug-ins need to obtain the request data and encapsulate the inference result over the data channel when calling the inference API.
The following is an example of using the data channel:
```
int SyncProcess(IRequest *request, IResponse *&response)
{
HILOGI("[IvpPlugin]Begin SyncProcess");
if (request == nullptr) {
HILOGE("[IvpPlugin]SyncProcess request is nullptr");
return RETCODE_NULL_PARAM;
}
DataInfo inputInfo = request->GetMsg();
if (inputInfo.data == nullptr) {
HILOGE("[IvpPlugin]InputInfo data is nullptr");
return RETCODE_NULL_PARAM;
}
...
response = IResponse::Create(request);
response->SetResult(outputInfo);
return RETCODE_SUCCESS;
}
```
In the example, the request and response are the data body sent over the data channel. The server encapsulates data in the request and sends it to the plug-in. After completing algorithm processing, the plug-in encapsulates the result into the response and returns it to the server over the data channel.
## Rule: Name an SDK in the format of **domain\_keyword<\_other information 1\_other information 2\_...\>\_sdk.so**.<a name="section62071110121516"></a>
You are advised to use the commonly known abbreviations for domains. For example, use **cv** for image and video, **asr** for voice recognition, and **translation** for text translation. Add one if there is no available abbreviation for a domain. Use keywords that accurately describe the algorithm capability of the plug-in. For example, use **keyword\_spotting** for wakeup keyword spotting \(KWS\). Add other information, such as the supported chip type and applicable region, between **keyword** and **sdk**, with each of them separated by an underscore \(\_\). Note that the name of a northbound SDK must end with **sdk**.
For example, if the SDK for the KWS plug-in supports only the Kirin 9000 chipset and is applicable only in China, then name the SDK as follows: **asr\_keyword\_spotting\_kirin9000\_china\_sdk.so**.
## Rule: Name a plug-in in the format of **domain\_keyword<\_other information 1\_other information 2\_...\>.so**.<a name="section1665562841519"></a>
Use the same naming rules as the SDK.
A plug-in maps to an SDK. Therefore, they have the same requirements for the domain, keyword, and other information in their names. The only difference is that the name of the SDK ends with **\_sdk** additionally. For example, if the plug-in is named **asr\_keyword\_spotting.so**, the corresponding northbound SDK is named **asr\_keyword\_spotting\_sdk.so**.
For example, if the SDK for the KWS plug-in supports only the Kirin 9000 chipset and is applicable only in China, then name the plug-in as follows: **asr\_keyword\_spotting\_kirin9000\_china.so**.
