You can call the native APIs provided by the **AudioDecoder** module to decode audio, that is, to decode media data into PCM streams.
Currently, the following decoding capabilities are supported:
| Container Specification| Audio Decoding Type |
| -------- | :--------------------------- |
| mp4 | AAC, MPEG (MP3), FLAC, Vorbis|
| m4a | AAC |
| flac | FLAC |
| ogg | Vorbis |
| aac | AAC |
| mp3 | MPEG (MP3) |
**Usage Scenario**
- Audio playback
Decode audio and transmit the data to the speaker for playing.
- Audio rendering
Decode audio and transmit the data to the audio processing module for audio rendering.
- Audio editing
Decode audio and transmit the data for audio editing (for example, adjusting the playback speed of a channel). Audio editing is performed based on PCM streams.
## How to Develop
Read [AudioDecoder](../reference/native-apis/_audio_decoder.md) for the API reference.
Refer to the code snippet below to complete the entire audio decoding process, including creating a decoder, setting decoding parameters (such as the sampling rate, bit rate, and number of audio channels), and starting, refreshing, resetting, and destroying the decoder.
During application development, you must call the APIs in the defined sequence. Otherwise, an exception or undefined behavior may occur.
For details about the complete code, see [Sample](https://gitee.com/openharmony/multimedia_av_codec/blob/master/test/nativedemo/audio_demo/avcodec_audio_decoder_demo.cpp).
The figure below shows the call relationship of audio decoding.
![Call relationship of audio decoding](figures/audio-decode.png)
3. Call **OH_AudioDecoder_Configure()** to configure the decoder.
The following options are mandatory: sampling rate, bit rate, and number of audio channels. The maximum input length is optional.
- For AAC decoding, the parameter that specifies whether the data type is Audio Data Transport Stream (ADTS) must be specified. If this parameter is not specified, the data type is considered as Low Overhead Audio Transport Multiplex (LATM).
- For Vorbis decoding, the ID header and setup header must also be specified.
```cpp
enumAudioFormatType:int32_t{
TYPE_AAC=0,
TYPE_FLAC=1,
TYPE_MP3=2,
TYPE_VORBIS=3,
};
// Set the decoding resolution.
int32_tret;
// (Mandatory) Configure the audio sampling rate.
constexpruint32_tDEFAULT_SMAPLERATE=44100;
// (Mandatory) Configure the audio bit rate.
constexpruint32_tDEFAULT_BITRATE=32000;
// (Mandatory) Configure the number of audio channels.
ret = OH_AudioDecoder_FreeOutputData(audioDec, index);
if (ret != AV_ERR_OK) {
// Exception handling.
}
```
8. (Optional) Call **OH_AudioDecoder_Flush()** to refresh the decoder.
After **OH_AudioDecoder_Flush()** is called, the decoder remains in the running state, but the current queue is cleared and the buffer storing the decoded data is freed. To continue decoding, you must call **OH_AudioDecoder_Start()** again.
You need to call **OH_AudioDecoder_Start()** in the following cases:
* The EOS of the file is reached.
* An error with **OH_AudioDecoder_IsValid** set to **true** (indicating that the execution can continue) occurs.
```c++
// Refresh the decoder.
ret = OH_AudioDecoder_Flush(audioDec);
if (ret != AV_ERR_OK) {
// Exception handling.
}
// Start decoding again.
ret = OH_AudioDecoder_Start(audioDec);
if (ret != AV_ERR_OK) {
// Exception handling.
}
```
9. (Optional) Call **OH_AudioDecoder_Reset()** to reset the decoder.
After **OH_AudioDecoder_Reset()** is called, the decoder returns to the initialized state. To continue decoding, you must call **OH_AudioDecoder_Configure()** and then **OH_AudioDecoder_Start()**.
```c++
// Reset the decoder.
ret = OH_AudioDecoder_Reset(audioDec);
if (ret != AV_ERR_OK) {
// Exception handling.
}
// Reconfigure the decoder.
ret = OH_AudioDecoder_Configure(audioDec, format);
if (ret != AV_ERR_OK) {
// Exception handling.
}
```
10. Call **OH_AudioDecoder_Stop()** to stop the decoder.
```c++
// Stop the decoder.
ret = OH_AudioDecoder_Stop(audioDec);
if (ret != AV_ERR_OK) {
// Exception handling.
}
return ret;
```
11. Call **OH_AudioDecoder_Destroy()** to destroy the decoder instance and release resources.
**NOTE**: You only need to call this API once.
```c++
// Call OH_AudioDecoder_Destroy to destroy the decoder.
ret = OH_AudioDecoder_Destroy(audioDec);
if (ret != AV_ERR_OK) {
// Exception handling.
} else {
audioEnc = NULL; // The decoder cannot be destroyed repeatedly.
You can call the native APIs provided by the **AudioEncoder** module to encode audio, that is, to compress audio PCM data into a desired format.
PCM data can be from any source. For example, you can use a microphone to record audio data or import edited PCM data. After audio encoding, you can output streams in the desired format and encapsulate the streams into a target file.
Currently, the following encoding capabilities are supported:
| Container Specification| Audio Encoding Type|
| -------- | :----------- |
| mp4 | AAC, FLAC |
| m4a | AAC |
| flac | FLAC |
| aac | AAC |
**Usage Scenario**
- Audio recording
Record and transfer PCM data, and encode the data into streams in the desired format.
- Audio editing
Import edited PCM data, and encode the data into streams in the desired format.
## How to Develop
Read [AudioEncoder](../reference/native-apis/_audio_encoder.md) for the API reference.
Refer to the code snippet below to complete the entire audio encoding process, including creating an encoder, setting encoding parameters (such as the sampling rate, bit rate, and number of audio channels), and starting, refreshing, resetting, and destroying the encoder.
During application development, you must call the APIs in the defined sequence. Otherwise, an exception or undefined behavior may occur.
For details about the complete code, see [Sample](https://gitee.com/openharmony/multimedia_av_codec/blob/master/test/nativedemo/audio_demo/avcodec_audio_aac_encoder_demo.cpp).
The figure below shows the call relationship of audio encoding.
![Call relationship of audio encoding](figures/audio-encode.png)
int32_t ret = OH_AudioEncoder_SetCallback(audioEnc, cb, userData);
```
3. Call **OH_AudioEncoder_Configure** to configure the encoder.
The following options are mandatory: sampling rate, bit rate, number of audio channels, audio channel type, and bit depth. The maximum input length is optional.
For FLAC encoding, the compliance level and sampling precision are also mandatory.
```cpp
enum AudioFormatType : int32_t {
TYPE_AAC = 0,
TYPE_FLAC = 1,
};
int32_t ret;
// (Mandatory) Configure the audio sampling rate.
constexpr uint32_t DEFAULT_SMAPLERATE = 44100;
// (Mandatory) Configure the audio bit rate.
constexpr uint32_t DEFAULT_BITRATE = 32000;
// (Mandatory) Configure the number of audio channels.
6. Call **OH_AudioEncoder_PushInputData()** to write the data to encode.
To indicate the End of Stream (EOS), pass in the **AVCODEC_BUFFER_FLAGS_EOS** flag.
For AAC encoding, **FRAME_SIZE** (number of sampling points) is fixed at **1024**.
For FLAC encoding, set **FRAME_SIZE** based on the table below.
| Sampling Rate| FRAME_SIZE|
| :----: | :----: |
| 8000 | 576 |
| 16000 | 1152 |
| 22050 | 2304 |
| 24000 | 2304 |
| 32000 | 2304 |
| 44100 | 4608 |
| 48000 | 4608 |
| 88200 | 8192 |
| 96000 | 8192 |
**NOTE**: If **FRAME_SIZE** is not set to **1024** for AAC encoding, an error code is returned. In the case of FLAC encoding, if **FRAME_SIZE** is set to a value greater than the value listed in the table for a given sampling rate, an error code is returned; if **FRAME_SIZE** is set to a value less than the value listed, the encoded file may be damaged.
ret = OH_AudioEncoder_FreeOutputData(audioEnc, index);
if (ret != AV_ERR_OK) {
// Exception handling.
}
if (attr.flags == AVCODEC_BUFFER_FLAGS_EOS) {
cout << "decode eos" << endl;
isRunning_.store(false);
break;
}
```
8. (Optional) Call **OH_AudioEncoder_Flush()** to refresh the encoder.
After **OH_AudioEncoder_Flush()** is called, the current encoding queue is cleared.
To continue encoding, you must call **OH_AudioEncoder_Start()** again.
You need to call **OH_AudioEncoder_Flush()** in the following cases:
* The EOS of the file is reached.
* An error with **OH_AudioEncoder_IsValid** set to **true** (indicating that the execution can continue) occurs.
```c++
// Refresh the encoder.
ret = OH_AudioEncoder_Flush(audioEnc);
if (ret != AV_ERR_OK) {
// Exception handling.
}
// Start encoding again.
ret = OH_AudioEncoder_Start(audioEnc);
if (ret != AV_ERR_OK) {
// Exception handling.
}
```
9. (Optional) Call **OH_AudioEncoder_Reset()** to reset the encoder.
After **OH_AudioEncoder_Reset()** is called, the encoder returns to the initialized state. To continue encoding, you must call **OH_AudioEncoder_Configure()** and then **OH_AudioEncoder_Start()**.
```c++
// Reset the encoder.
ret=OH_AudioEncoder_Reset(audioEnc);
if(ret!=AV_ERR_OK){
// Exception handling.
}
// Reconfigure the encoder.
ret=OH_AudioEncoder_Configure(audioEnc,format);
if(ret!=AV_ERR_OK){
// Exception handling.
}
```
10. Call **OH_AudioEncoder_Stop()** to stop the encoder.
```c++
// Stop the encoder.
ret = OH_AudioEncoder_Stop(audioEnc);
if (ret != AV_ERR_OK) {
// Exception handling.
}
return ret;
```
11. Call **OH_AudioEncoder_Destroy()** to destroy the encoder instance and release resources.
**NOTE**: You only need to call this API once.
```c++
// Call OH_AudioEncoder_Destroy to destroy the encoder.
ret = OH_AudioEncoder_Destroy(audioEnc);
if (ret != AV_ERR_OK) {
// Exception handling.
} else {
audioEnc = NULL; // The encoder cannot be destroyed repeatedly.
You can call the native APIs provided by the **AVDemuxer** module to decapsulate audio and video, that is, to extract audio and video frame data from bit stream data.
Currently, two data input types are supported: remote connection (over HTTP) and File Descriptor (FD).
The following decapsulation formats are supported:
| Media Format | Encapsulation Format |
| -------- | :----------------------------|
| Video | MP4, MPEG TS |
| Audio | M4A, AAC, MP3, OGG, FLAC, WAV|
**Usage Scenario**
- Audio and video playback
Decapsulate audio and video streams, decode the frame data obtained through decapsulation, and play the decoded data.
- Audio and video editing
Decapsulate audio and video streams, and edit the specified frames.
- Media file format conversion
Decapsulate audio and video streams, and encapsulate them into a new file format.
## How to Develop
Read [AVDemuxer](../reference/native-apis/_a_v_demuxer.md) and [AVSource](../reference/native-apis/_a_v_source.md) for the API reference.
> **NOTE**
>
> - To call the decapsulation APIs to parse a network playback path, request the **ohos.permission.INTERNET** permission by following the instructions provided in [Applying for Permissions](../security/accesstoken-guidelines.md).
> - To call the decapsulation APIs to parse a local file, request the **ohos.permission.READ_MEDIA** permission by following the instructions provided in [Applying for Permissions](../security/accesstoken-guidelines.md).
> - You can also use **ResourceManager.getRawFd** to obtain the FD of a file packed in the HAP file. For details, see [ResourceManager API Reference](../reference/apis/js-apis-resource-manager.md#getrawfd9).
1. Create a demuxer instance.
``` c++
// Create the FD. You must have the read permission on the file handle when opening the file.
std::stringfileName="test.mp4";
intfd=open(fileName.c_str(),O_RDONLY);
structstatfileStatus{};
size_tfileSize=0;
if(stat(fileName.c_str(),&fileStatus)==0){
fileSize=static_cast<size_t>(fileStatus.st_size);
}else{
printf("get stat failed");
return;
}
// Create a source resource object for the FD resource file. If offset is not the start position of the file or size is not the actual file size, the data obtained may be incomplete. Consequently, the source resource object may fail to create or subsequent decapsulation may fail.
5. (Optional) Seek to the specified time for the selected track.
``` c++
// Decapsulation is performed from this time.
// Note:
// 1. If OH_AVDemuxer_SeekToTime is called for an MPEG TS file, the target position may be a non-key frame. You can then call OH_AVDemuxer_ReadSample to check whether the current frame is a key frame based on the obtained OH_AVCodecBufferAttr. If it is a non-key frame, which causes display issues on the application side, cyclically read the frames until you reach the first key frame, where you can perform processing such as decoding.
// 2. If OH_AVDemuxer_SeekToTime is called for an OGG file, the file seeks to the start of the time interval (second) where the input parameter millisecond is located, which may cause a certain number of frame errors.
// Manually set the instance to NULL after OH_AVSource_Destroy is called. Do not call this API repeatedly for the same instance; otherwise, a program error occurs.
if(OH_AVSource_Destroy(source)!=AV_ERR_OK){
printf("destroy source pointer error");
}
source=NULL;
// Manually set the instance to NULL after OH_AVDemuxer_Destroy is called. Do not call this API repeatedly for the same instance; otherwise, a program error occurs.
You can call the native APIs provided by the **AVMuxer** module to encapsulate audio and video, that is, to store encoded audio and video data to a file in a certain format.
Currently, the following encapsulation capabilities are supported:
| Encapsulation Format| Video Codec Type | Audio Codec Type | Cover Type |
After you encode audio and video streams, encapsulate them into files.
- Audio and video editing
After you edit audio and video, encapsulate them into files.
- Audio and video transcoding
After transcode audio and video, encapsulate them into files.
## How to Develop
Read [AVMuxer](../reference/native-apis/_a_v_muxer.md) for the API reference.
> **NOTE**
>
> To call the encapsulation APIs to write a local file, request the **ohos.permission.READ_MEDIA** and **ohos.permission.WRITE_MEDIA** permissions by following the instructions provided in [Applying for Permissions](../security/accesstoken-guidelines.md).
The following walks you through how to implement the entire process of audio and video encapsulation. It uses the MP4 format as an example.
1. Call **OH_AVMuxer_Create()** to create an **OH_AVMuxer** instance.
``` c++
// Set the encapsulation format to MP4.
OH_AVOutputFormatformat=AV_OUTPUT_FORMAT_MPEG_4;
// Create a File Descriptor (FD) in read/write mode.
@@ -77,6 +77,7 @@ The table below lists the supported audio playback formats.
| Video Format| Mandatory or Not|
| -------- | -------- |
| H.265<sup>10+</sup> | Yes|
| H.264 | Yes|
| MPEG-2 | No|
| MPEG-4 | No|
...
...
@@ -87,9 +88,9 @@ The table below lists the supported playback formats and mainstream resolutions.
| Video Container Format| Description| Resolution|
| -------- | -------- | -------- |
| MP4| Video formats: H.264, MPEG-2, MPEG-4, and H.263<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
| MKV| Video formats: H.264, MPEG-2, MPEG-4, and H.263<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
| TS| Video formats: H.264, MPEG-2, and MPEG-4<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
| MP4| Video formats: H.265<sup>10+</sup>, H.264, MPEG2, MPEG4, and H.263<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
| MKV| Video formats: H.265<sup>10+</sup>, H.264, MPEG2, MPEG4, and H.263<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
| TS| Video formats: H.265<sup>10+</sup>, H.264, MPEG2, and MPEG4<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
| WebM| Video format: VP8<br>Audio format: VORBIS| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p|
@@ -13,7 +13,7 @@ Read [Image](../reference/apis/js-apis-image.md#imagesource) for APIs related to
```
2. Obtain an image.
- Method 1: Obtain the sandbox path. For details about how to obtain the sandbox path, see [Obtaining the Application Development Path](../application-models/application-context-stage.md#obtaining-the-application-development-path). For details about the application sandbox and how to push files to the application sandbox, see [File Management](../file-management/app-sandbox-directory.md).
- Method 1: Obtain the sandbox path. For details about how to obtain the sandbox path, see [Obtaining Application File Paths](../application-models/application-context-stage.md#obtaining-application-file-paths). For details about the application sandbox and how to push files to the application sandbox, see [File Management](../file-management/app-sandbox-directory.md).
```ts
// Code on the stage model
...
...
@@ -110,6 +110,11 @@ Read [Image](../reference/apis/js-apis-image.md#imagesource) for APIs related to
After the decoding is complete and the pixel map is obtained, you can perform subsequent [image processing](image-transformation.md).
5. Release the **PixelMap** instance.
```ts
pixelMap.release();
```
## Sample Code - Decoding an Image in Resource Files
1. Obtain a resource manager.
...
...
@@ -140,4 +145,7 @@ Read [Image](../reference/apis/js-apis-image.md#imagesource) for APIs related to
**OHAudio** is a set of native APIs introduced in API version 10. These APIs are normalized in design and support both common and low-latency audio channels.
## Prerequisites
To use the playback or recording capability of **OHAudio**, you must first import the corresponding header files.
Specifically, to use APIs for audio playback, import <[native_audiostreambuilder.h](../reference/native-apis/native__audiostreambuilder_8h.md)> and <[native_audiorenderer.h](../reference/native-apis/native__audiorenderer_8h.md)>.
## Audio Stream Builder
**OHAudio** provides the **OH_AudioStreamBuilder** class, which complies with the builder design pattern and is used to build audio streams. You need to specify [OH_AudioStream_Type](../reference/native-apis/_o_h_audio.md#oh_audiostream_type) based on your service scenarios.
**OH_AudioStream_Type** can be set to either of the following:
- AUDIOSTREAM_TYPE_RENDERER
- AUDIOSTREAM_TYPE_CAPTURER
The following code snippet shows how to use [OH_AudioStreamBuilder_Create](../reference/native-apis/_o_h_audio.md#oh_audiostreambuilder_create) to create a builder:
After the audio service is complete, call [OH_AudioStreamBuilder_Destroy](../reference/native-apis/_o_h_audio.md#oh_audiostreambuilder_destroy) to destroy the builder.
```
OH_AudioStreamBuilder_Destroy(builder);
```
## How to Develop
Read [OHAudio](../reference/native-apis/_o_h_audio.md) for the API reference.
The following walks you through how to implement simple playback:
Note that the audio data to play is written through callbacks. You must call **OH_AudioStreamBuilder_SetRendererCallback** to implement the callbacks. For details about the declaration of the callback functions, see [OH_AudioRenderer_Callbacks](../reference/native-apis/_o_h_audio.md#oh_audiorenderer_callbacks).
**OHAudio** is a set of native APIs introduced in API version 10. These APIs are normalized in design and support both common and low-latency audio channels.
## Prerequisites
To use the playback or recording capability of **OHAudio**, you must first import the corresponding header files.
To use APIs for audio recording, import <[native_audiostreambuilder.h](../reference/native-apis/native__audiostreambuilder_8h.md)> and <[native_audiocapturer.h](../reference/native-apis/native__audiocapturer_8h.md)>.
## Audio Stream Builder
**OHAudio** provides the **OH_AudioStreamBuilder** class, which complies with the builder design pattern and is used to build audio streams. You need to specify [OH_AudioStream_Type](../reference/native-apis/_o_h_audio.md#oh_audiostream_type) based on your service scenarios.
**OH_AudioStream_Type** can be set to either of the following:
- AUDIOSTREAM_TYPE_RENDERER
- AUDIOSTREAM_TYPE_CAPTURER
The following code snippet shows how to use [OH_AudioStreamBuilder_Create](../reference/native-apis/_o_h_audio.md#oh_audiostreambuilder_create) to create a builder:
After the audio service is complete, call [OH_AudioStreamBuilder_Destroy](../reference/native-apis/_o_h_audio.md#oh_audiostreambuilder_destroy) to destroy the builder.
```
OH_AudioStreamBuilder_Destroy(builder);
```
## How to Develop
Read [OHAudio](../reference/native-apis/_o_h_audio.md) for the API reference.
The following walks you through how to implement simple recording:
Note that the audio data to record is written through callbacks. You must call **OH_AudioStreamBuilder_SetCapturerCallback** to implement the callbacks. For details about the declaration of the callback functions, see [OH_AudioCapturer_Callbacks](../reference/native-apis/_o_h_audio.md#oh_audiocapturer_callbacks).
You can call the native APIs provided by the **VideoDecoder** module to decode video, that is, to decode media data into a YUV file or send it for display.
Currently, the following decoding capabilities are supported:
| Container Specification| Video Hardware Decoding Type | Video Software Decoding Type |
Video software decoding and hardware decoding are different. When a decoder is created based on the MIME type, only H.264 (video/avc) is supported for software decoding, and H.264 (video/avc) and H.265 (video/hevc) are supported for hardware decoding.
## How to Develop
Read [VideoDecoder](../reference/native-apis/_video_decoder.md) for the API reference.
2. Call **OH_VideoDecoder_SetCallback()** to set callback functions.
Register the **OH_AVCodecAsyncCallback** struct that defines the following callback function pointers:
-**OnError**, a callback used to report a codec operation error
-**OnOutputFormatChanged**, a callback used to report a codec stream change, for example, stream width or height change.
-**OnInputBufferAvailable**, a callback used to report input data required, which means that the decoder is ready for receiving data
-**OnOutputBufferAvailable**, a callback used to report output data generated, which means that decoding is complete (Note: The **data** parameter in the callback function is empty in surface output mode.)
You need to process the callback functions to ensure that the decoder runs properly.
9. (Optional) Call **OH_VideoDecoder_Flush()** to refresh the decoder.
After **OH_VideoDecoder_Flush()** is called, the decoder remains in the running state, but the current queue is cleared and the buffer storing the decoded data is freed.
To continue decoding, you must call **OH_VideoDecoder_Start()** again.
``` c++
int32_tret;
// Refresh the decoder.
ret=OH_VideoDecoder_Flush(videoDec);
if(ret!=AV_ERR_OK){
// Exception handling.
}
// Start decoding again.
ret=OH_VideoDecoder_Start(videoDec);
```
10. (Optional) Call **OH_VideoDecoder_Reset()** to reset the decoder.
After **OH_VideoDecoder_Reset()** is called, the decoder returns to the initialized state. To continue decoding, you must call **OH_VideoDecoder_Configure()** and then **OH_VideoDecoder_Start()**.
``` c++
int32_t ret;
// Reset the decoder.
ret = OH_VideoDecoder_Reset(videoDec);
if (ret != AV_ERR_OK) {
// Exception handling.
}
// Reconfigure the decoder.
ret = OH_VideoDecoder_Configure(videoDec, format);
```
11. Call **OH_VideoDecoder_Stop()** to stop the decoder.
``` c++
int32_t ret;
// Stop the decoder.
ret = OH_VideoDecoder_Stop(videoDec);
if (ret != AV_ERR_OK) {
// Exception handling.
}
return AV_ERR_OK;
```
12. Call **OH_VideoDecoder_Destroy()** to destroy the decoder instance and release resources.
``` c++
int32_t ret;
// Call OH_VideoDecoder_Destroy to destroy the decoder.