The OpenHarmony codec Hardware Device Interface (HDI) driver framework implements the video hardware codec driver based on OpenMAX. It provides APIs for the upper-layer media services to obtain component encoding and decoding capabilities, create a component, set parameters, transfer data, and destroy a component. The codec driver can encode video data in YUV or RGB format to H.264 or H.265 format, and decode raw stream data from H.264 or H.265 format to YUV or RGB format. This document describes the codec functionality developed based on the OpenHarmony Hardware Driver Foundation (HDF).
The codec HDI driver framework is implemented based on the HDF. The figure below shows the codec HDI driver framework.
- Codec HDI Callback Remote Service: an anonymous callback service used to process callbacks.
- Codec HDI: provides standard APIs based on OpenMAX. The upper layer services call the APIs to implement hardware encoding and decoding.
- Codec HDI Adapter: HDI implementation layer, which implements HDI APIs and interacts with OpenMAX Integration layer (IL).
- OpenMAX IL interface: provides OpenMAX IL APIs to directly interact with the codec HDI driver.
- Vendor Impl: vendor adaptation layer, which is the OpenMAX implementation layer adapted by each vendor.
- Codec Hardware: hardware decoding device.
### Basic Concepts
Before you get started, understand the following concepts:
- Sampling rate
The number of samples taken from continuous signals every second to form discrete signals, in Hz.
- OpenMAX IL
A standardized media component interface to enable applications and media frameworks to interact with multimedia codecs and supported components in a unified manner.
- Frame rate
Number of frames of images transmitted per second, or the number of times that a GPU can refresh images per second. A higher frame rate indicates smoother motion, while a lower frame rate means choppier motion and blurry footage.
- Bit rate
Number of bits transmitted or processed per unit of time, generally in kbit/s. A higher bit rate indicates clearer image, while a lower bit rate means blurry image with artifacts.
- Component
An OpenMAX IL component, which is an abstraction of modules in video streams. The components in this document refer to codec components for video encoding and decoding.
### Constraints
The codec HDI applies only to the standard system.
For more details, see [OpenMAX IL](https://www.khronos.org/api/openmax/il).
## Development Guidelines
### When to Use
The codec module implements hardware encoding and decoding of video data. It converts raw stream data such as H.264 data into YUV or RGB data, and converts YUV or RGB data into data formats such as H.264.
| int32_t (*EmptyBufferDone)(struct CodecCallbackType *self, int64_t appData, const struct OmxCodecBuffer *buffer) | Reports an event indicating that the encoding or decoding in the input buffer is complete.|
| int32_t (*FillBufferDone)(struct CodecCallbackType *self, int64_t appData, const struct OmxCodecBuffer *buffer) | Reports an event indicating that the output buffer is filled. |
For more information, see [codec](https://gitee.com/openharmony/drivers_peripheral/tree/master/codec).
### Development Procedure
The codec HDI driver development procedure is as follows:
#### Registering and Initializing the Driver
Define the **HdfDriverEntry** structure (which defines the driver initialization method) and fill in the **g_codecComponentDriverEntry** structure to implement the **Bind()**, **Init()**, and **Release()** pointers.
```c
structHdfDriverEntryg_codecComponentDriverEntry={
.moduleVersion=1,
.moduleName="codec_hdi_omx_server",
.Bind=HdfCodecComponentTypeDriverBind,
.Init=HdfCodecComponentTypeDriverInit,
.Release=HdfCodecComponentTypeDriverRelease,
};
HDF_INIT(g_codecComponentDriverEntry);// Register HdfDriverEntry of the codec HDI with the HDF.
```
-**HdfCodecComponentTypeDriverBind**: binds the device in the HDF to **CodecComponentTypeHost** and registers the codec service with the HDF.
The HCS includes the driver node, loading sequence, and service name. For details about the HCS syntax, see [Configuration Management](driver-hdf-manage.md).
Configuration file Path of the standard system:
vendor/hihope/rk3568/hdf_config/uhdf/
1. Device configuration
Add the **codec_omx_service** configuration to **codec_host** in **device_info.hcs**. The following is an example:
After completing codec module driver adaptation, use the HDI APIs provided by the codec module for further development. The codec HDI provides the following features:
1. Provides codec HDI APIs for video services to implement encoding and decoding of video services.
2. Provides standard interfaces for device developers to ensure that the OEM vendors comply with the HDI adapter standard. This promises a healthy evolution of the ecosystem.
The development procedure is as follows:
1. Initialize the driver, including initializing the instances, callbacks, and component.
2. Set codec parameters and information such as the video width, height, and bit rate.
3. Apply for input and output buffers.
4. Flip codec buffers, enable the component to enter the **OMX_Executing** state, and process the callbacks.
5. Deinitialize the interface instance, destroy the buffers, close the component, and releases all interface objects.
#### Initializing the Driver
Initialize the interface instance and callbacks, and create a component.
```cpp
// Initialize the codec HDI ComponentManager instance.
Automatic framing is not supported in rk OMX decoding. Therefore, you need to manually divide data into frames. Currently, data is divided into frames from code 0x000001 or 0x00000001 and sent to the server for processing. The sample code is as follows:
-**EventHandler**: Called when a command is executed. For example, when the command for changing the component state from **OMX_StateIdle** to **OMX_StateExecuting** is executed, this callback is invoked to return the result.
-**EmptyBufferDone**: Called when the input data is consumed. If the client needs to fill in data to encode or decode, call **EmptyThisBuffer()**.
-**FillBufferDone**: Called when the output data is filled. If the client needs to read the encoded or decoded data, call **FillThisBuffer()**.
HDF_LOGI("OMX_CommandStateSet reached, status is %{public}d",info->data2);
g_core->onStatusChanged();
}
break;
}
default:
break;
}
returnHDF_SUCCESS;
}
```
#### Destroying a Component
Change the component state to IDLE, release the input and output buffers, change the component state to **OMX_StateLoaded**, and call **DestoryComponent** to destroy the component.
// After the buffers are released, the component enters the OMX_StateLoaded state.
if(status!=OMX_StateLoaded){
HDF_LOGI("Wait for OMX_StateLoaded status");
this->WaitForStatusChanged();
}else{
HDF_LOGI(" status is %{public}d",status);
}
```
##### Example of Destroying a Component Instance
```cpp
// Destroy a component instance.
voidOMXCore::Release(){
omxMgr_->DestoryComponent(client_);
client_=nullptr;
CodecComponentManagerRelease();
}
```
# FAQs
## Green Screens Displayed During the Decoding Process
**Symptom**
Green screens are displayed during the decoding process.
**Possible Causes**
OpenMAX does not support framing.
**Solution**
Transfer data frame by frame when **EmptyThisBuffer** is called.
## Only Green Screen Displayed During the Decoding Process
**Symptom**
Decoding fails, and all the frames decoded cannot be played.
**Possible Causes**
For the data in AVCC format, the first frame to be processed must be extra_data.
**Solution**
Write sps and pps to the buffer in extra_data format, and set the buffer flag to **OMX_BUFFERFLAG_EXTRADATA**.
## Failed to Play the Encoded Video
**Symptom**
After the generated video stream (H.264 stream) is written to a file, the video stream cannot be played by FFplay.
**Possible Causes**
- The **xFramerate** parameter of the output port is incorrectly set.
- The **OMX_VIDEO_PARAM_AVCTYPE** parameter is correctly set.
**Solution**
View the **codec_host** log generated during encoding, search for "encode params init settings", and check for incorrect parameters. If **framerate** is **0**, **xFramerate** is incorrectly set. In this case, move the framerate leftwards by 16 bits.
Check the value of **OMX_VIDEO_PARAM_AVCTYPE**, and set it correctly.
# Reference
For more information, see [Codec](https://gitee.com/openharmony/drivers_peripheral/tree/master/codec).
