Skip to content

D
DeepSpeech

PaddlePaddle / DeepSpeech
大约 2 年 前同步成功

通知 210
Star 8425
Fork 1598
D
DeepSpeech
提交 490300f8 编写于 作者: 小湉湉's avatar 小湉湉
浏览文件
操作

Merge branch 'develop' of github.com:PaddlePaddle/PaddleSpeech into add_ljspeech_hifigan

上级 c36039ce e5e8b8a1
展开全部 隐藏空白更改
内联 并排
Showing with 17034 addition and 2182 deletion
.pre-commit-config.yaml
浏览文件 @ 490300f8
...@@ -50,13 +50,13 @@ repos: ...@@ -50,13 +50,13 @@ repos:
entry: bash .pre-commit-hooks/clang-format.hook -i entry: bash .pre-commit-hooks/clang-format.hook -i
language: system language: system
files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|cuh|proto)$ files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|cuh|proto)$
exclude: (?=speechx/speechx/kaldi).*(\.cpp|\.cc|\.h|\.py)$ exclude: (?=speechx/speechx/kaldi|speechx/patch).*(\.cpp|\.cc|\.h|\.py)$
- id: copyright_checker - id: copyright_checker
name: copyright_checker name: copyright_checker
entry: python .pre-commit-hooks/copyright-check.hook entry: python .pre-commit-hooks/copyright-check.hook
language: system language: system
files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|proto|py)$ files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|proto|py)$
exclude: (?=third_party|pypinyin|speechx/speechx/kaldi).*(\.cpp|\.cc|\.h|\.py)$ exclude: (?=third_party|pypinyin|speechx/speechx/kaldi|speechx/patch).*(\.cpp|\.cc|\.h|\.py)$
- repo: https://github.com/asottile/reorder_python_imports - repo: https://github.com/asottile/reorder_python_imports
rev: v2.4.0 rev: v2.4.0
hooks: hooks:
......
README.md
浏览文件 @ 490300f8
...@@ -178,7 +178,7 @@ Via the easy-to-use, efficient, flexible and scalable implementation, our vision ...@@ -178,7 +178,7 @@ Via the easy-to-use, efficient, flexible and scalable implementation, our vision
<!--- <!---
2021.12.14: We would like to have an online courses to introduce basics and research of speech, as well as code practice with `paddlespeech`. Please pay attention to our [Calendar](https://www.paddlepaddle.org.cn/live). 2021.12.14: We would like to have an online courses to introduce basics and research of speech, as well as code practice with `paddlespeech`. Please pay attention to our [Calendar](https://www.paddlepaddle.org.cn/live).
---> --->
- 🤗 2021.12.14: Our PaddleSpeech [ASR](https://huggingface.co/spaces/KPatrick/PaddleSpeechASR) and [TTS](https://huggingface.co/spaces/akhaliq/paddlespeech) Demos on Hugging Face Spaces are available! - 🤗 2021.12.14: Our PaddleSpeech [ASR](https://huggingface.co/spaces/KPatrick/PaddleSpeechASR) and [TTS](https://huggingface.co/spaces/KPatrick/PaddleSpeechTTS) Demos on Hugging Face Spaces are available!
- 👏🏻 2021.12.10: PaddleSpeech CLI is available for Audio Classification, Automatic Speech Recognition, Speech Translation (English to Chinese) and Text-to-Speech. - 👏🏻 2021.12.10: PaddleSpeech CLI is available for Audio Classification, Automatic Speech Recognition, Speech Translation (English to Chinese) and Text-to-Speech.
### Community ### Community
...@@ -207,6 +207,7 @@ paddlespeech cls --input input.wav ...@@ -207,6 +207,7 @@ paddlespeech cls --input input.wav
```shell ```shell
paddlespeech asr --lang zh --input input_16k.wav paddlespeech asr --lang zh --input input_16k.wav
``` ```
- web demo for Automatic Speech Recognition is integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See Demo: [ASR Demo](https://huggingface.co/spaces/KPatrick/PaddleSpeechASR)
**Speech Translation** (English to Chinese) **Speech Translation** (English to Chinese)
(not support for Mac and Windows now) (not support for Mac and Windows now)
...@@ -218,7 +219,7 @@ paddlespeech st --input input_16k.wav ...@@ -218,7 +219,7 @@ paddlespeech st --input input_16k.wav
```shell ```shell
paddlespeech tts --input "你好，欢迎使用飞桨深度学习框架！" --output output.wav paddlespeech tts --input "你好，欢迎使用飞桨深度学习框架！" --output output.wav
``` ```
- web demo for Text to Speech is integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See Demo: [TTS Demo](https://huggingface.co/spaces/akhaliq/paddlespeech) - web demo for Text to Speech is integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See Demo: [TTS Demo](https://huggingface.co/spaces/KPatrick/PaddleSpeechTTS)
**Text Postprocessing** **Text Postprocessing**
- Punctuation Restoration - Punctuation Restoration
...@@ -397,9 +398,9 @@ PaddleSpeech supports a series of most popular models. They are summarized in [r ...@@ -397,9 +398,9 @@ PaddleSpeech supports a series of most popular models. They are summarized in [r
</tr> </tr>
<tr> <tr>
<td >HiFiGAN</td> <td >HiFiGAN</td>
<td >CSMSC</td> <td >LJSpeech / VCTK / CSMSC / AISHELL-3</td>
<td> <td>
<a href = "./examples/csmsc/voc5">HiFiGAN-csmsc</a> <a href = "./examples/ljspeech/voc5">HiFiGAN-ljspeech</a> / <a href = "./examples/vctk/voc5">HiFiGAN-vctk</a> / <a href = "./examples/csmsc/voc5">HiFiGAN-csmsc</a> / <a href = "./examples/aishell3/voc5">HiFiGAN-aishell3</a>
</td> </td>
</tr> </tr>
<tr> <tr>
...@@ -573,7 +574,6 @@ You are warmly welcome to submit questions in [discussions](https://github.com/P ...@@ -573,7 +574,6 @@ You are warmly welcome to submit questions in [discussions](https://github.com/P
- Many thanks to [yeyupiaoling](https://github.com/yeyupiaoling)/[PPASR](https://github.com/yeyupiaoling/PPASR)/[PaddlePaddle-DeepSpeech](https://github.com/yeyupiaoling/PaddlePaddle-DeepSpeech)/[VoiceprintRecognition-PaddlePaddle](https://github.com/yeyupiaoling/VoiceprintRecognition-PaddlePaddle)/[AudioClassification-PaddlePaddle](https://github.com/yeyupiaoling/AudioClassification-PaddlePaddle) for years of attention, constructive advice and great help. - Many thanks to [yeyupiaoling](https://github.com/yeyupiaoling)/[PPASR](https://github.com/yeyupiaoling/PPASR)/[PaddlePaddle-DeepSpeech](https://github.com/yeyupiaoling/PaddlePaddle-DeepSpeech)/[VoiceprintRecognition-PaddlePaddle](https://github.com/yeyupiaoling/VoiceprintRecognition-PaddlePaddle)/[AudioClassification-PaddlePaddle](https://github.com/yeyupiaoling/AudioClassification-PaddlePaddle) for years of attention, constructive advice and great help.
- Many thanks to [AK391](https://github.com/AK391) for TTS web demo on Huggingface Spaces using Gradio.
- Many thanks to [mymagicpower](https://github.com/mymagicpower) for the Java implementation of ASR upon [short](https://github.com/mymagicpower/AIAS/tree/main/3_audio_sdks/asr_sdk) and [long](https://github.com/mymagicpower/AIAS/tree/main/3_audio_sdks/asr_long_audio_sdk) audio files. - Many thanks to [mymagicpower](https://github.com/mymagicpower) for the Java implementation of ASR upon [short](https://github.com/mymagicpower/AIAS/tree/main/3_audio_sdks/asr_sdk) and [long](https://github.com/mymagicpower/AIAS/tree/main/3_audio_sdks/asr_long_audio_sdk) audio files.
- Many thanks to [JiehangXie](https://github.com/JiehangXie)/[PaddleBoBo](https://github.com/JiehangXie/PaddleBoBo) for developing Virtual Uploader(VUP)/Virtual YouTuber(VTuber) with PaddleSpeech TTS function. - Many thanks to [JiehangXie](https://github.com/JiehangXie)/[PaddleBoBo](https://github.com/JiehangXie/PaddleBoBo) for developing Virtual Uploader(VUP)/Virtual YouTuber(VTuber) with PaddleSpeech TTS function.
- Many thanks to [745165806](https://github.com/745165806)/[PaddleSpeechTask](https://github.com/745165806/PaddleSpeechTask) for contributing Punctuation Restoration model. - Many thanks to [745165806](https://github.com/745165806)/[PaddleSpeechTask](https://github.com/745165806/PaddleSpeechTask) for contributing Punctuation Restoration model.
......
README_cn.md
浏览文件 @ 490300f8
...@@ -392,9 +392,9 @@ PaddleSpeech 的 **语音合成** 主要包含三个模块：文本前端、声 ...@@ -392,9 +392,9 @@ PaddleSpeech 的 **语音合成** 主要包含三个模块：文本前端、声
</tr> </tr>
<tr> <tr>
<td >HiFiGAN</td> <td >HiFiGAN</td>
<td >CSMSC</td> <td >LJSpeech / VCTK / CSMSC / AISHELL-3</td>
<td> <td>
<a href = "./examples/csmsc/voc5">HiFiGAN-csmsc</a> <a href = "./examples/ljspeech/voc5">HiFiGAN-ljspeech</a> / <a href = "./examples/vctk/voc5">HiFiGAN-vctk</a> / <a href = "./examples/csmsc/voc5">HiFiGAN-csmsc</a> / <a href = "./examples/aishell3/voc5">HiFiGAN-aishell3</a>
</td> </td>
</tr> </tr>
<tr> <tr>
......
demos/audio_searching/README.md 0 → 100644
浏览文件 @ 490300f8
([简体中文](./README_cn.md)|English)
# Audio Searching
## Introduction
As the Internet continues to evolve, unstructured data such as emails, social media photos, live videos, and customer service voice calls have become increasingly common. If we want to process the data on a computer, we need to use embedding technology to transform the data into vector and store, index, and query it
However, when there is a large amount of data, such as hundreds of millions of audio tracks, it is more difficult to do a similarity search. The exhaustive method is feasible, but very time consuming. For this scenario, this demo will introduce how to build an audio similarity retrieval system using the open source vector database Milvus
Audio retrieval (speech, music, speaker, etc.) enables querying and finding similar sounds (or the same speaker) in a large amount of audio data. The audio similarity retrieval system can be used to identify similar sound effects, minimize intellectual property infringement, quickly retrieve the voice print library, and help enterprises control fraud and identity theft. Audio retrieval also plays an important role in the classification and statistical analysis of audio data
In this demo, you will learn how to build an audio retrieval system to retrieve similar sound snippets. The uploaded audio clips are converted into vector data using paddlespeech-based pre-training models (audio classification model, speaker recognition model, etc.) and stored in Milvus. Milvus automatically generates a unique ID for each vector, then stores the ID and the corresponding audio information (audio ID, audio speaker ID, etc.) in MySQL to complete the library construction. During retrieval, users upload test audio to obtain vector, and then conduct vector similarity search in Milvus. The retrieval result returned by Milvus is vector ID, and the corresponding audio information can be queried in MySQL by ID
![Workflow of an audio searching system](./img/audio_searching.png)
Note：this demo uses the [CN-Celeb](http://openslr.org/82/) dataset of at least 650,000 audio entries and 3000 speakers to build the audio vector library, which is then retrieved using a preset distance calculation. The dataset can also use other, Adjust as needed, e.g. Librispeech, VoxCeleb, UrbanSound, GloVe, MNIST, etc
## Usage
### 1. Prepare MySQL and Milvus services by docker-compose
The audio similarity search system requires Milvus, MySQL services. We can start these containers with one click through [docker-compose.yaml](./docker-compose.yaml), so please make sure you have [installed Docker Engine](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/) before running. then
```bash
docker-compose -f docker-compose.yaml up -d
```
Then you will see the that all containers are created:
```bash
Creating network "quick_deploy_app_net" with driver "bridge"
Creating milvus-minio ... done
Creating milvus-etcd ... done
Creating audio-mysql ... done
Creating milvus-standalone ... done
Creating audio-webclient ... done
```
And show all containers with `docker ps`, and you can use `docker logs audio-mysql` to get the logs of server container
```bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2bcf279e599 milvusdb/milvus:v2.0.1 "/tini -- milvus run…" 22 hours ago Up 22 hours 0.0.0.0:19530->19530/tcp milvus-standalone
d8ef4c84e25c mysql:5.7 "docker-entrypoint.s…" 22 hours ago Up 22 hours 0.0.0.0:3306->3306/tcp, 33060/tcp audio-mysql
8fb501edb4f3 quay.io/coreos/etcd:v3.5.0 "etcd -advertise-cli…" 22 hours ago Up 22 hours 2379-2380/tcp milvus-etcd
ffce340b3790 minio/minio:RELEASE.2020-12-03T00-03-10Z "/usr/bin/docker-ent…" 22 hours ago Up 22 hours (healthy) 9000/tcp milvus-minio
15c84a506754 iregistry.baidu-int.com/paddlespeech/audio-search-client:1.0 "/bin/bash -c '/usr/…" 22 hours ago Up 22 hours (healthy) 0.0.0.0:8068->80/tcp audio-webclient
```
### 2. Start API Server
Then to start the system server, and it provides HTTP backend services.
- Install the Python packages
```bash
pip install -r requirements.txt
```
- Set configuration
```bash
vim src/config.py
```
Modify the parameters according to your own environment. Here listing some parameters that need to be set, for more information please refer to [config.py](./src/config.py).
| **Parameter** | **Description** | **Default setting** |
| ---------------- | ----------------------------------------------------- | ------------------- |
| MILVUS_HOST | The IP address of Milvus, you can get it by ifconfig. If running everything on one machine, most likely 127.0.0.1 | 127.0.0.1 |
| MILVUS_PORT | Port of Milvus. | 19530 |
| VECTOR_DIMENSION | Dimension of the vectors. | 2048 |
| MYSQL_HOST | The IP address of Mysql. | 127.0.0.1 |
| MYSQL_PORT | Port of Milvus. | 3306 |
| DEFAULT_TABLE | The milvus and mysql default collection name. | audio_table |
- Run the code
Then start the server with Fastapi.
```bash
export PYTHONPATH=$PYTHONPATH:./src
python src/main.py
```
Then you will see the Application is started:
```bash
INFO: Started server process [3949]
2022-03-07 17:39:14,864 ｜ INFO ｜ server.py ｜ serve ｜ 75 ｜ Started server process [3949]
INFO: Waiting for application startup.
2022-03-07 17:39:14,865 ｜ INFO ｜ on.py ｜ startup ｜ 45 ｜ Waiting for application startup.
INFO: Application startup complete.
2022-03-07 17:39:14,866 ｜ INFO ｜ on.py ｜ startup ｜ 59 ｜ Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8002 (Press CTRL+C to quit)
2022-03-07 17:39:14,867 ｜ INFO ｜ server.py ｜ _log_started_message ｜ 206 ｜ Uvicorn running on http://0.0.0.0:8002 (Press CTRL+C to quit)
```
### 3. Usage
- Prepare data
```bash
wget -c https://www.openslr.org/resources/82/cn-celeb_v2.tar.gz && tar -xvf cn-celeb_v2.tar.gz
```
Note: If you want to build a quick demo, you can use ./src/test_main.py:download_audio_data function, it downloads 20 audio files , Subsequent results show this collection as an example
- scripts test (recommend!)
The internal process is downloading data, loading the Paddlespeech model, extracting embedding, storing library, retrieving and deleting library
```bash
python ./src/test_main.py
```
Output：
```bash
Checkpoint path: %your model path%
Extracting feature from audio No. 1 , 20 audios in total
Extracting feature from audio No. 2 , 20 audios in total
...
2022-03-09 17:22:13,870 ｜ INFO ｜ main.py ｜ load_audios ｜ 85 ｜ Successfully loaded data, total count: 20
2022-03-09 17:22:13,898 ｜ INFO ｜ main.py ｜ count_audio ｜ 147 ｜ Successfully count the number of data!
2022-03-09 17:22:13,918 ｜ INFO ｜ main.py ｜ audio_path ｜ 57 ｜ Successfully load audio: ./example_audio/test.wav
...
2022-03-09 17:22:32,580 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 131 ｜ search result http://testserver/data?audio_path=./example_audio/test.wav, distance 0.0
2022-03-09 17:22:32,580 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 131 ｜ search result http://testserver/data?audio_path=./example_audio/knife_chopping.wav, distance 0.021805256605148315
2022-03-09 17:22:32,580 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 131 ｜ search result http://testserver/data?audio_path=./example_audio/knife_cut_into_flesh.wav, distance 0.052762262523174286
...
2022-03-09 17:22:32,582 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 135 ｜ Successfully searched similar audio!
2022-03-09 17:22:33,658 ｜ INFO ｜ main.py ｜ drop_tables ｜ 159 ｜ Successfully drop tables in Milvus and MySQL!
```
- GUI test (optional)
Navigate to 127.0.0.1:8068 in your browser to access the front-end interface
Note: If the browser and the service are not on the same machine, then the IP needs to be changed to the IP of the machine where the service is located, and the corresponding API_URL in docker-compose.yaml needs to be changed and the service can be restarted
- Insert data
Download the data and decompress it to a path named /home/speech/data. Then enter /home/speech/data in the address bar of the upload page to upload the data
![](./img/insert.png)
- Search for similar audio
Select the magnifying glass icon on the left side of the interface. Then, press the "Default Target Audio File" button and upload a .wav sound file you'd like to search. Results will be displayed
![](./img/search.png)
### 4.Result
machine configuration：
- OS: CentOS release 7.6
- kernel：4.17.11-1.el7.elrepo.x86_64
- CPU：Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
- memory：132G
dataset：
- CN-Celeb, train size 650,000, test size 10,000, dimention 192, distance L2
recall and elapsed time statistics are shown in the following figure：
![](./img/result.png)
The retrieval framework based on Milvus takes about 2.9 milliseconds to retrieve on the premise of 90% recall rate, and it takes about 500 milliseconds for feature extraction (testing audio takes about 5 seconds), that is, a single audio test takes about 503 milliseconds in total, which can meet most application scenarios
### 5.Pretrained Models
Here is a list of pretrained models released by PaddleSpeech :
| Model | Sample Rate
| :--- | :---:
| ecapa_tdnn | 16000
| panns_cnn6| 32000
| panns_cnn10| 32000
| panns_cnn14| 32000
demos/audio_searching/README_cn.md 0 → 100644
浏览文件 @ 490300f8
(简体中文|[English](./README.md))
# 音频相似性检索
## 介绍
随着互联网不断发展，电子邮件、社交媒体照片、直播视频、客服语音等非结构化数据已经变得越来越普遍。如果想要使用计算机来处理这些数据，需要使用 embedding 技术将这些数据转化为向量 vector，然后进行存储、建索引、并查询
但是，当数据量很大，比如上亿条音频要做相似度搜索，就比较困难了。穷举法固然可行，但非常耗时。针对这种场景，该demo 将介绍如何使用开源向量数据库 Milvus 搭建音频相似度检索系统
音频检索（如演讲、音乐、说话人等检索）实现了在海量音频数据中查询并找出相似声音（或相同说话人）片段。音频相似性检索系统可用于识别相似的音效、最大限度减少知识产权侵权等，还可以快速的检索声纹库、帮助企业控制欺诈和身份盗用等。在音频数据的分类和统计分析中，音频检索也发挥着重要作用
在本 demo 中，你将学会如何构建一个音频检索系统，用来检索相似的声音片段。使用基于 PaddleSpeech 预训练模型（音频分类模型，说话人识别模型等）将上传的音频片段转换为向量数据，并存储在 Milvus 中。Milvus 自动为每个向量生成唯一的 ID，然后将 ID 和 相应的音频信息（音频id，音频的说话人id等等）存储在 MySQL，这样就完成建库的工作。用户在检索时，上传测试音频，得到向量，然后在 Milvus 中进行向量相似度搜索，Milvus 返回的检索结果为向量 ID，通过 ID 在 MySQL 内部查询相应的音频信息即可
![音频检索流程图](./img/audio_searching.png)
注：该 demo 使用 [CN-Celeb](http://openslr.org/82/) 数据集，包括至少 650000 条音频，3000 个说话人，来建立音频向量库（音频特征，或音频说话人特征），然后通过预设的距离计算方式进行音频（或说话人）检索，这里面数据集也可以使用其他的，根据需要调整，如Librispeech，VoxCeleb，UrbanSound，GloVe，MNIST等
## 使用方法
### 1. MySQL 和 Milvus 安装
音频相似度搜索系统需要用到 Milvus, MySQL 服务。 我们可以通过 [docker-compose.yaml](./docker-compose.yaml) 一键启动这些容器，所以请确保在运行之前已经安装了 [Docker Engine](https://docs.docker.com/engine/install/) 和 [Docker Compose](https://docs.docker.com/compose/install/)。 即
```bash
docker-compose -f docker-compose.yaml up -d
```
然后你会看到所有的容器都被创建:
```bash
Creating network "quick_deploy_app_net" with driver "bridge"
Creating milvus-minio ... done
Creating milvus-etcd ... done
Creating audio-mysql ... done
Creating milvus-standalone ... done
Creating audio-webclient ... done
```
可以采用'docker ps'来显示所有的容器，还可以使用'docker logs audio-mysql'来获取服务器容器的日志：
```bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2bcf279e599 milvusdb/milvus:v2.0.1 "/tini -- milvus run…" 22 hours ago Up 22 hours 0.0.0.0:19530->19530/tcp milvus-standalone
d8ef4c84e25c mysql:5.7 "docker-entrypoint.s…" 22 hours ago Up 22 hours 0.0.0.0:3306->3306/tcp, 33060/tcp audio-mysql
8fb501edb4f3 quay.io/coreos/etcd:v3.5.0 "etcd -advertise-cli…" 22 hours ago Up 22 hours 2379-2380/tcp milvus-etcd
ffce340b3790 minio/minio:RELEASE.2020-12-03T00-03-10Z "/usr/bin/docker-ent…" 22 hours ago Up 22 hours (healthy) 9000/tcp milvus-minio
15c84a506754 iregistry.baidu-int.com/paddlespeech/audio-search-client:1.0 "/bin/bash -c '/usr/…" 22 hours ago Up 22 hours (healthy) 0.0.0.0:8068->80/tcp audio-webclient
```
### 2. 配置并启动 API 服务
启动系统服务程序，它会提供基于 Http 后端服务
- 安装服务依赖的 python 基础包
```bash
pip install -r requirements.txt
```
- 修改配置
```bash
vim src/config.py
```
请根据实际环境进行修改。 这里列出了一些需要设置的参数，更多信息请参考 [config.py](./src/config.py)
| **Parameter** | **Description** | **Default setting** |
| ---------------- | ----------------------------------------------------- | ------------------- |
| MILVUS_HOST | The IP address of Milvus, you can get it by ifconfig. If running everything on one machine, most likely 127.0.0.1 | 127.0.0.1 |
| MILVUS_PORT | Port of Milvus. | 19530 |
| VECTOR_DIMENSION | Dimension of the vectors. | 2048 |
| MYSQL_HOST | The IP address of Mysql. | 127.0.0.1 |
| MYSQL_PORT | Port of Milvus. | 3306 |
| DEFAULT_TABLE | The milvus and mysql default collection name. | audio_table |
- 运行程序
启动用 Fastapi 构建的服务
```bash
export PYTHONPATH=$PYTHONPATH:./src
python src/main.py
```
然后你会看到应用程序启动:
```bash
INFO: Started server process [3949]
2022-03-07 17:39:14,864 ｜ INFO ｜ server.py ｜ serve ｜ 75 ｜ Started server process [3949]
INFO: Waiting for application startup.
2022-03-07 17:39:14,865 ｜ INFO ｜ on.py ｜ startup ｜ 45 ｜ Waiting for application startup.
INFO: Application startup complete.
2022-03-07 17:39:14,866 ｜ INFO ｜ on.py ｜ startup ｜ 59 ｜ Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8002 (Press CTRL+C to quit)
2022-03-07 17:39:14,867 ｜ INFO ｜ server.py ｜ _log_started_message ｜ 206 ｜ Uvicorn running on http://0.0.0.0:8002 (Press CTRL+C to quit)
```
### 3. 测试方法
- 准备数据
```bash
wget -c https://www.openslr.org/resources/82/cn-celeb_v2.tar.gz && tar -xvf cn-celeb_v2.tar.gz
```
注：如果希望快速搭建 demo，可以采用 ./src/test_main.py:download_audio_data 内部的 20 条音频，另外后续结果展示以该集合为例
- 脚本测试（推荐）
```bash
python ./src/test_main.py
```
注：内部将依次下载数据，加载 paddlespeech 模型，提取 embedding，存储建库，检索，删库
输出：
```bash
Checkpoint path: %your model path%
Extracting feature from audio No. 1 , 20 audios in total
Extracting feature from audio No. 2 , 20 audios in total
...
2022-03-09 17:22:13,870 ｜ INFO ｜ main.py ｜ load_audios ｜ 85 ｜ Successfully loaded data, total count: 20
2022-03-09 17:22:13,898 ｜ INFO ｜ main.py ｜ count_audio ｜ 147 ｜ Successfully count the number of data!
2022-03-09 17:22:13,918 ｜ INFO ｜ main.py ｜ audio_path ｜ 57 ｜ Successfully load audio: ./example_audio/test.wav
...
2022-03-09 17:22:32,580 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 131 ｜ search result http://testserver/data?audio_path=./example_audio/test.wav, distance 0.0
2022-03-09 17:22:32,580 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 131 ｜ search result http://testserver/data?audio_path=./example_audio/knife_chopping.wav, distance 0.021805256605148315
2022-03-09 17:22:32,580 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 131 ｜ search result http://testserver/data?audio_path=./example_audio/knife_cut_into_flesh.wav, distance 0.052762262523174286
...
2022-03-09 17:22:32,582 ｜ INFO ｜ main.py ｜ search_local_audio ｜ 135 ｜ Successfully searched similar audio!
2022-03-09 17:22:33,658 ｜ INFO ｜ main.py ｜ drop_tables ｜ 159 ｜ Successfully drop tables in Milvus and MySQL!
```
- 前端测试（可选）
在浏览器中输入 127.0.0.1:8068 访问前端页面
注：如果浏览器和服务不在同一台机器上，那么 IP 需要修改成服务所在的机器 IP，并且docker-compose.yaml 中相应的 API_URL 也要修改，并重新起服务即可
- 上传音频
下载数据并解压到一文件夹，假设为 /home/speech/data，那么在上传页面地址栏输入 /home/speech/data 进行数据上传
![](./img/insert.png)
- 检索相似音频
选择左上角放大镜，点击 “Default Target Audio File” 按钮，上传测试音频，接着你将看到检索结果
![](./img/search.png)
### 4. 结果
机器配置：
- 操作系统: CentOS release 7.6
- 内核：4.17.11-1.el7.elrepo.x86_64
- 处理器：Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
- 内存：132G
数据集：
- CN-Celeb, 训练集 65万, 测试集 1万，向量维度 192，距离计算方式 L2
召回和耗时统计如下图：
![](./img/result.png)
基于 milvus 的检索框架在召回率 90% 的前提下，检索耗时约 2.9 毫秒，加上特征提取(Embedding)耗时约 500毫秒(测试音频时长约 5秒)，即单条音频测试总共耗时约 503 毫秒，可以满足大多数应用场景
### 5. 预训练模型
以下是 PaddleSpeech 提供的预训练模型列表：
| 模型 | 采样率
| :--- | :---:
| ecapa_tdnn| 16000
| panns_cnn6| 32000
| panns_cnn10| 32000
| panns_cnn14| 32000
demos/audio_searching/docker-compose.yaml 0 → 100644
浏览文件 @ 490300f8
version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.0
networks:
app_net:
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2020-12-03T00-03-10Z
networks:
app_net:
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.0.1
networks:
app_net:
ipv4_address: 172.16.23.10
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
ports:
- "19530:19530"
depends_on:
- "etcd"
- "minio"
mysql:
container_name: audio-mysql
image: mysql:5.7
networks:
app_net:
ipv4_address: 172.16.23.11
environment:
- MYSQL_ROOT_PASSWORD=123456
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/mysql:/var/lib/mysql
ports:
- "3306:3306"
webclient:
container_name: audio-webclient
image: qingen1/paddlespeech-audio-search-client:2.3
networks:
app_net:
ipv4_address: 172.16.23.13
environment:
API_URL: 'http://127.0.0.1:8002'
ports:
- "8068:80"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost/"]
interval: 30s
timeout: 20s
retries: 3
networks:
app_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.23.0/24
gateway: 172.16.23.1
demos/audio_searching/requirements.txt 0 → 100644
浏览文件 @ 490300f8
soundfile==0.10.3.post1
librosa==0.8.0
numpy
pymysql
fastapi
uvicorn
diskcache==5.2.1
pymilvus==2.0.1
python-multipart
typing
starlette
pydantic
\ No newline at end of file
demos/audio_searching/src/config.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
############### Milvus Configuration ###############
MILVUS_HOST = os.getenv("MILVUS_HOST", "127.0.0.1")
MILVUS_PORT = int(os.getenv("MILVUS_PORT", "19530"))
VECTOR_DIMENSION = int(os.getenv("VECTOR_DIMENSION", "2048"))
INDEX_FILE_SIZE = int(os.getenv("INDEX_FILE_SIZE", "1024"))
METRIC_TYPE = os.getenv("METRIC_TYPE", "L2")
DEFAULT_TABLE = os.getenv("DEFAULT_TABLE", "audio_table")
TOP_K = int(os.getenv("TOP_K", "10"))
############### MySQL Configuration ###############
MYSQL_HOST = os.getenv("MYSQL_HOST", "127.0.0.1")
MYSQL_PORT = int(os.getenv("MYSQL_PORT", "3306"))
MYSQL_USER = os.getenv("MYSQL_USER", "root")
MYSQL_PWD = os.getenv("MYSQL_PWD", "123456")
MYSQL_DB = os.getenv("MYSQL_DB", "mysql")
############### Data Path ###############
UPLOAD_PATH = os.getenv("UPLOAD_PATH", "tmp/audio-data")
############### Number of Log Files ###############
LOGS_NUM = int(os.getenv("logs_num", "0"))
demos/audio_searching/src/encode.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import librosa
import numpy as np
from logs import LOGGER
def get_audio_embedding(path):
"""
Use vpr_inference to generate embedding of audio
"""
try:
RESAMPLE_RATE = 16000
audio, _ = librosa.load(path, sr=RESAMPLE_RATE, mono=True)
# TODO add infer/python interface to get embedding, now fake it by rand
# vpr = ECAPATDNN(checkpoint_path=None, device='cuda')
# embedding = vpr.inference(audio)
np.random.seed(hash(os.path.basename(path)) % 1000000)
embedding = np.random.rand(1, 2048)
embedding = embedding / np.linalg.norm(embedding)
embedding = embedding.tolist()[0]
return embedding
except Exception as e:
LOGGER.error(f"Error with embedding:{e}")
return None
demos/audio_searching/src/logs.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import codecs
import datetime
import logging
import os
import re
import sys
from config import LOGS_NUM
class MultiprocessHandler(logging.FileHandler):
"""
A handler class which writes formatted logging records to disk files
"""
def __init__(self,
filename,
when='D',
backupCount=0,
encoding=None,
delay=False):
"""
Open the specified file and use it as the stream for logging
"""
self.prefix = filename
self.backupCount = backupCount
self.when = when.upper()
self.extMath = r"^\d{4}-\d{2}-\d{2}"
self.when_dict = {
'S': "%Y-%m-%d-%H-%M-%S",
'M': "%Y-%m-%d-%H-%M",
'H': "%Y-%m-%d-%H",
'D': "%Y-%m-%d"
}
self.suffix = self.when_dict.get(when)
if not self.suffix:
print('The specified date interval unit is invalid: ', self.when)
sys.exit(1)
self.filefmt = os.path.join('.', "logs",
f"{self.prefix}-{self.suffix}.log")
self.filePath = datetime.datetime.now().strftime(self.filefmt)
_dir = os.path.dirname(self.filefmt)
try:
if not os.path.exists(_dir):
os.makedirs(_dir)
except Exception as e:
print('Failed to create log file: ', e)
print("log_path：" + self.filePath)
sys.exit(1)
logging.FileHandler.__init__(self, self.filePath, 'a+', encoding, delay)
def should_change_file_to_write(self):
"""
To write the file
"""
_filePath = datetime.datetime.now().strftime(self.filefmt)
if _filePath != self.filePath:
self.filePath = _filePath
return True
return False
def do_change_file(self):
"""
To change file states
"""
self.baseFilename = os.path.abspath(self.filePath)
if self.stream:
self.stream.close()
self.stream = None
if not self.delay:
self.stream = self._open()
if self.backupCount > 0:
for s in self.get_files_to_delete():
os.remove(s)
def get_files_to_delete(self):
"""
To delete backup files
"""
dir_name, _ = os.path.split(self.baseFilename)
file_names = os.listdir(dir_name)
result = []
prefix = self.prefix + '-'
for file_name in file_names:
if file_name[:len(prefix)] == prefix:
suffix = file_name[len(prefix):-4]
if re.compile(self.extMath).match(suffix):
result.append(os.path.join(dir_name, file_name))
result.sort()
if len(result) < self.backupCount:
result = []
else:
result = result[:len(result) - self.backupCount]
return result
def emit(self, record):
"""
Emit a record
"""
try:
if self.should_change_file_to_write():
self.do_change_file()
logging.FileHandler.emit(self, record)
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
def write_log():
"""
Init a logger
"""
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# formatter = '%(asctime)s ｜ %(levelname)s ｜ %(filename)s ｜ %(funcName)s ｜ %(module)s ｜ %(lineno)s ｜ %(message)s'
fmt = logging.Formatter(
'%(asctime)s ｜ %(levelname)s ｜ %(filename)s ｜ %(funcName)s ｜ %(lineno)s ｜ %(message)s'
)
stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(fmt)
log_name = "audio-searching"
file_handler = MultiprocessHandler(log_name, when='D', backupCount=LOGS_NUM)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(fmt)
file_handler.do_change_file()
logger.addHandler(stream_handler)
logger.addHandler(file_handler)
return logger
LOGGER = write_log()
if __name__ == "__main__":
message = 'test writing logs'
LOGGER.info(message)
LOGGER.debug(message)
LOGGER.error(message)
demos/audio_searching/src/main.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
from typing import Optional
import uvicorn
from config import UPLOAD_PATH
from diskcache import Cache
from fastapi import FastAPI
from fastapi import File
from fastapi import UploadFile
from logs import LOGGER
from milvus_helpers import MilvusHelper
from mysql_helpers import MySQLHelper
from operations.count import do_count
from operations.drop import do_drop
from operations.load import do_load
from operations.search import do_search
from pydantic import BaseModel
from starlette.middleware.cors import CORSMiddleware
from starlette.requests import Request
from starlette.responses import FileResponse
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"])
MODEL = None
MILVUS_CLI = MilvusHelper()
MYSQL_CLI = MySQLHelper()
# Mkdir 'tmp/audio-data'
if not os.path.exists(UPLOAD_PATH):
os.makedirs(UPLOAD_PATH)
LOGGER.info(f"Mkdir the path: {UPLOAD_PATH}")
@app.get('/data')
def audio_path(audio_path):
# Get the audio file
try:
LOGGER.info(f"Successfully load audio: {audio_path}")
return FileResponse(audio_path)
except Exception as e:
LOGGER.error(f"upload audio error: {e}")
return {'status': False, 'msg': e}, 400
@app.get('/progress')
def get_progress():
# Get the progress of dealing with data
try:
cache = Cache('./tmp')
return f"current: {cache['current']}, total: {cache['total']}"
except Exception as e:
LOGGER.error(f"Upload data error: {e}")
return {'status': False, 'msg': e}, 400
class Item(BaseModel):
Table: Optional[str] = None
File: str
@app.post('/audio/load')
async def load_audios(item: Item):
# Insert all the audio files under the file path to Milvus/MySQL
try:
total_num = do_load(item.Table, item.File, MILVUS_CLI, MYSQL_CLI)
LOGGER.info(f"Successfully loaded data, total count: {total_num}")
return {'status': True, 'msg': "Successfully loaded data!"}
except Exception as e:
LOGGER.error(e)
return {'status': False, 'msg': e}, 400
@app.post('/audio/search')
async def search_audio(request: Request,
table_name: str=None,
audio: UploadFile=File(...)):
# Search the uploaded audio in Milvus/MySQL
try:
# Save the upload data to server.
content = await audio.read()
query_audio_path = os.path.join(UPLOAD_PATH, audio.filename)
with open(query_audio_path, "wb+") as f:
f.write(content)
host = request.headers['host']
_, paths, distances = do_search(host, table_name, query_audio_path,
MILVUS_CLI, MYSQL_CLI)
names = []
for path, score in zip(paths, distances):
names.append(os.path.basename(path))
LOGGER.info(f"search result {path}, score {score}")
res = dict(zip(paths, zip(names, distances)))
# Sort results by distance metric, closest distances first
res = sorted(res.items(), key=lambda item: item[1][1], reverse=True)
LOGGER.info("Successfully searched similar audio!")
return res
except Exception as e:
LOGGER.error(e)
return {'status': False, 'msg': e}, 400
@app.post('/audio/search/local')
async def search_local_audio(request: Request,
query_audio_path: str,
table_name: str=None):
# Search the uploaded audio in Milvus/MySQL
try:
host = request.headers['host']
_, paths, distances = do_search(host, table_name, query_audio_path,
MILVUS_CLI, MYSQL_CLI)
names = []
for path, score in zip(paths, distances):
names.append(os.path.basename(path))
LOGGER.info(f"search result {path}, score {score}")
res = dict(zip(paths, zip(names, distances)))
# Sort results by distance metric, closest distances first
res = sorted(res.items(), key=lambda item: item[1][1], reverse=True)
LOGGER.info("Successfully searched similar audio!")
return res
except Exception as e:
LOGGER.error(e)
return {'status': False, 'msg': e}, 400
@app.get('/audio/count')
async def count_audio(table_name: str=None):
# Returns the total number of vectors in the system
try:
num = do_count(table_name, MILVUS_CLI)
LOGGER.info("Successfully count the number of data!")
return num
except Exception as e:
LOGGER.error(e)
return {'status': False, 'msg': e}, 400
@app.post('/audio/drop')
async def drop_tables(table_name: str=None):
# Delete the collection of Milvus and MySQL
try:
status = do_drop(table_name, MILVUS_CLI, MYSQL_CLI)
LOGGER.info("Successfully drop tables in Milvus and MySQL!")
return status
except Exception as e:
LOGGER.error(e)
return {'status': False, 'msg': e}, 400
if __name__ == '__main__':
uvicorn.run(app=app, host='0.0.0.0', port=8002)
demos/audio_searching/src/milvus_helpers.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from config import METRIC_TYPE
from config import MILVUS_HOST
from config import MILVUS_PORT
from config import VECTOR_DIMENSION
from logs import LOGGER
from pymilvus import Collection
from pymilvus import CollectionSchema
from pymilvus import connections
from pymilvus import DataType
from pymilvus import FieldSchema
from pymilvus import utility
class MilvusHelper:
"""
the basic operations of PyMilvus
# This example shows how to:
# 1. connect to Milvus server
# 2. create a collection
# 3. insert entities
# 4. create index
# 5. search
# 6. delete a collection
"""
def __init__(self):
try:
self.collection = None
connections.connect(host=MILVUS_HOST, port=MILVUS_PORT)
LOGGER.debug(
f"Successfully connect to Milvus with IP:{MILVUS_HOST} and PORT:{MILVUS_PORT}"
)
except Exception as e:
LOGGER.error(f"Failed to connect Milvus: {e}")
sys.exit(1)
def set_collection(self, collection_name):
try:
if self.has_collection(collection_name):
self.collection = Collection(name=collection_name)
else:
raise Exception(
f"There is no collection named:{collection_name}")
except Exception as e:
LOGGER.error(f"Failed to set collection in Milvus: {e}")
sys.exit(1)
def has_collection(self, collection_name):
# Return if Milvus has the collection
try:
return utility.has_collection(collection_name)
except Exception as e:
LOGGER.error(f"Failed to check state of collection in Milvus: {e}")
sys.exit(1)
def create_collection(self, collection_name):
# Create milvus collection if not exists
try:
if not self.has_collection(collection_name):
field1 = FieldSchema(
name="id",
dtype=DataType.INT64,
descrition="int64",
is_primary=True,
auto_id=True)
field2 = FieldSchema(
name="embedding",
dtype=DataType.FLOAT_VECTOR,
descrition="speaker embeddings",
dim=VECTOR_DIMENSION,
is_primary=False)
schema = CollectionSchema(
fields=[field1, field2], description="embeddings info")
self.collection = Collection(
name=collection_name, schema=schema)
LOGGER.debug(f"Create Milvus collection: {collection_name}")
else:
self.set_collection(collection_name)
return "OK"
except Exception as e:
LOGGER.error(f"Failed to create collection in Milvus: {e}")
sys.exit(1)
def insert(self, collection_name, vectors):
# Batch insert vectors to milvus collection
try:
self.create_collection(collection_name)
data = [vectors]
self.set_collection(collection_name)
mr = self.collection.insert(data)
ids = mr.primary_keys
self.collection.load()
LOGGER.debug(
f"Insert vectors to Milvus in collection: {collection_name} with {len(vectors)} rows"
)
return ids
except Exception as e:
LOGGER.error(f"Failed to insert data to Milvus: {e}")
sys.exit(1)
def create_index(self, collection_name):
# Create IVF_FLAT index on milvus collection
try:
self.set_collection(collection_name)
default_index = {
"index_type": "IVF_SQ8",
"metric_type": METRIC_TYPE,
"params": {
"nlist": 16384
}
}
status = self.collection.create_index(
field_name="embedding", index_params=default_index)
if not status.code:
LOGGER.debug(
f"Successfully create index in collection:{collection_name} with param:{default_index}"
)
return status
else:
raise Exception(status.message)
except Exception as e:
LOGGER.error(f"Failed to create index: {e}")
sys.exit(1)
def delete_collection(self, collection_name):
# Delete Milvus collection
try:
self.set_collection(collection_name)
self.collection.drop()
LOGGER.debug("Successfully drop collection!")
return "ok"
except Exception as e:
LOGGER.error(f"Failed to drop collection: {e}")
sys.exit(1)
def search_vectors(self, collection_name, vectors, top_k):
# Search vector in milvus collection
try:
self.set_collection(collection_name)
search_params = {
"metric_type": METRIC_TYPE,
"params": {
"nprobe": 16
}
}
res = self.collection.search(
vectors,
anns_field="embedding",
param=search_params,
limit=top_k)
LOGGER.debug(f"Successfully search in collection: {res}")
return res
except Exception as e:
LOGGER.error(f"Failed to search vectors in Milvus: {e}")
sys.exit(1)
def count(self, collection_name):
# Get the number of milvus collection
try:
self.set_collection(collection_name)
num = self.collection.num_entities
LOGGER.debug(
f"Successfully get the num:{num} of the collection:{collection_name}"
)
return num
except Exception as e:
LOGGER.error(f"Failed to count vectors in Milvus: {e}")
sys.exit(1)
demos/audio_searching/src/mysql_helpers.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import pymysql
from config import MYSQL_DB
from config import MYSQL_HOST
from config import MYSQL_PORT
from config import MYSQL_PWD
from config import MYSQL_USER
from logs import LOGGER
class MySQLHelper():
"""
the basic operations of PyMySQL
# This example shows how to:
# 1. connect to MySQL server
# 2. create a table
# 3. insert data to table
# 4. search by milvus ids
# 5. delete table
"""
def __init__(self):
self.conn = pymysql.connect(
host=MYSQL_HOST,
user=MYSQL_USER,
port=MYSQL_PORT,
password=MYSQL_PWD,
database=MYSQL_DB,
local_infile=True)
self.cursor = self.conn.cursor()
def test_connection(self):
try:
self.conn.ping()
except Exception:
self.conn = pymysql.connect(
host=MYSQL_HOST,
user=MYSQL_USER,
port=MYSQL_PORT,
password=MYSQL_PWD,
database=MYSQL_DB,
local_infile=True)
self.cursor = self.conn.cursor()
def create_mysql_table(self, table_name):
# Create mysql table if not exists
self.test_connection()
sql = "create table if not exists " + table_name + "(milvus_id TEXT, audio_path TEXT);"
try:
self.cursor.execute(sql)
LOGGER.debug(f"MYSQL create table: {table_name} with sql: {sql}")
except Exception as e:
LOGGER.error(f"MYSQL ERROR: {e} with sql: {sql}")
sys.exit(1)
def load_data_to_mysql(self, table_name, data):
# Batch insert (Milvus_ids, img_path) to mysql
self.test_connection()
sql = "insert into " + table_name + " (milvus_id,audio_path) values (%s,%s);"
try:
self.cursor.executemany(sql, data)
self.conn.commit()
LOGGER.debug(
f"MYSQL loads data to table: {table_name} successfully")
except Exception as e:
LOGGER.error(f"MYSQL ERROR: {e} with sql: {sql}")
sys.exit(1)
def search_by_milvus_ids(self, ids, table_name):
# Get the img_path according to the milvus ids
self.test_connection()
str_ids = str(ids).replace('[', '').replace(']', '')
sql = "select audio_path from " + table_name + " where milvus_id in (" + str_ids + ") order by field (milvus_id," + str_ids + ");"
try:
self.cursor.execute(sql)
results = self.cursor.fetchall()
results = [res[0] for res in results]
LOGGER.debug("MYSQL search by milvus id.")
return results
except Exception as e:
LOGGER.error(f"MYSQL ERROR: {e} with sql: {sql}")
sys.exit(1)
def delete_table(self, table_name):
# Delete mysql table if exists
self.test_connection()
sql = "drop table if exists " + table_name + ";"
try:
self.cursor.execute(sql)
LOGGER.debug(f"MYSQL delete table:{table_name}")
except Exception as e:
LOGGER.error(f"MYSQL ERROR: {e} with sql: {sql}")
sys.exit(1)
def delete_all_data(self, table_name):
# Delete all the data in mysql table
self.test_connection()
sql = 'delete from ' + table_name + ';'
try:
self.cursor.execute(sql)
self.conn.commit()
LOGGER.debug(f"MYSQL delete all data in table:{table_name}")
except Exception as e:
LOGGER.error(f"MYSQL ERROR: {e} with sql: {sql}")
sys.exit(1)
def count_table(self, table_name):
# Get the number of mysql table
self.test_connection()
sql = "select count(milvus_id) from " + table_name + ";"
try:
self.cursor.execute(sql)
results = self.cursor.fetchall()
LOGGER.debug(f"MYSQL count table:{table_name}")
return results[0][0]
except Exception as e:
LOGGER.error(f"MYSQL ERROR: {e} with sql: {sql}")
sys.exit(1)
demos/audio_searching/src/operations/__init__.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
demos/audio_searching/src/operations/count.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from config import DEFAULT_TABLE
from logs import LOGGER
def do_count(table_name, milvus_cli):
"""
Returns the total number of vectors in the system
"""
if not table_name:
table_name = DEFAULT_TABLE
try:
if not milvus_cli.has_collection(table_name):
return None
num = milvus_cli.count(table_name)
return num
except Exception as e:
LOGGER.error(f"Error attempting to count table {e}")
sys.exit(1)
demos/audio_searching/src/operations/drop.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from config import DEFAULT_TABLE
from logs import LOGGER
def do_drop(table_name, milvus_cli, mysql_cli):
"""
Delete the collection of Milvus and MySQL
"""
if not table_name:
table_name = DEFAULT_TABLE
try:
if not milvus_cli.has_collection(table_name):
return "Collection is not exist"
status = milvus_cli.delete_collection(table_name)
mysql_cli.delete_table(table_name)
return status
except Exception as e:
LOGGER.error(f"Error attempting to drop table: {e}")
sys.exit(1)
demos/audio_searching/src/operations/load.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
from config import DEFAULT_TABLE
from diskcache import Cache
from encode import get_audio_embedding
from logs import LOGGER
def get_audios(path):
"""
List all wav and aif files recursively under the path folder.
"""
supported_formats = [".wav", ".mp3", ".ogg", ".flac", ".m4a"]
return [
item
for sublist in [[os.path.join(dir, file) for file in files]
for dir, _, files in list(os.walk(path))]
for item in sublist if os.path.splitext(item)[1] in supported_formats
]
def extract_features(audio_dir):
"""
Get the vector of audio
"""
try:
cache = Cache('./tmp')
feats = []
names = []
audio_list = get_audios(audio_dir)
total = len(audio_list)
cache['total'] = total
for i, audio_path in enumerate(audio_list):
norm_feat = get_audio_embedding(audio_path)
if norm_feat is None:
continue
feats.append(norm_feat)
names.append(audio_path.encode())
cache['current'] = i + 1
print(
f"Extracting feature from audio No. {i + 1} , {total} audios in total"
)
return feats, names
except Exception as e:
LOGGER.error(f"Error with extracting feature from audio {e}")
sys.exit(1)
def format_data(ids, names):
"""
Combine the id of the vector and the name of the audio into a list
"""
data = []
for i in range(len(ids)):
value = (str(ids[i]), names[i])
data.append(value)
return data
def do_load(table_name, audio_dir, milvus_cli, mysql_cli):
"""
Import vectors to Milvus and data to Mysql respectively
"""
if not table_name:
table_name = DEFAULT_TABLE
vectors, names = extract_features(audio_dir)
ids = milvus_cli.insert(table_name, vectors)
milvus_cli.create_index(table_name)
mysql_cli.create_mysql_table(table_name)
mysql_cli.load_data_to_mysql(table_name, format_data(ids, names))
return len(ids)
demos/audio_searching/src/operations/search.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from config import DEFAULT_TABLE
from config import TOP_K
from encode import get_audio_embedding
from logs import LOGGER
def do_search(host, table_name, audio_path, milvus_cli, mysql_cli):
"""
Search the uploaded audio in Milvus/MySQL
"""
try:
if not table_name:
table_name = DEFAULT_TABLE
feat = get_audio_embedding(audio_path)
vectors = milvus_cli.search_vectors(table_name, [feat], TOP_K)
vids = [str(x.id) for x in vectors[0]]
paths = mysql_cli.search_by_milvus_ids(vids, table_name)
distances = [x.distance for x in vectors[0]]
for i in range(len(paths)):
tmp = "http://" + str(host) + "/data?audio_path=" + str(paths[i])
paths[i] = tmp
distances[i] = (1 - distances[i]) * 100
return vids, paths, distances
except Exception as e:
LOGGER.error(f"Error with search: {e}")
sys.exit(1)
demos/audio_searching/src/test_main.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import zipfile
import gdown
from fastapi.testclient import TestClient
from main import app
client = TestClient(app)
def download_audio_data():
"""
download audio data
"""
url = 'https://drive.google.com/uc?id=1bKu21JWBfcZBuEuzFEvPoAX6PmRrgnUp'
gdown.download(url)
with zipfile.ZipFile('example_audio.zip', 'r') as zip_ref:
zip_ref.extractall('./example_audio')
def test_drop():
"""
Delete the collection of Milvus and MySQL
"""
response = client.post("/audio/drop")
assert response.status_code == 200
def test_load():
"""
Insert all the audio files under the file path to Milvus/MySQL
"""
response = client.post("/audio/load", json={"File": "./example_audio"})
assert response.status_code == 200
assert response.json() == {
'status': True,
'msg': "Successfully loaded data!"
}
def test_progress():
"""
Get the progress of dealing with data
"""
response = client.get("/progress")
assert response.status_code == 200
assert response.json() == "current: 20, total: 20"
def test_count():
"""
Returns the total number of vectors in the system
"""
response = client.get("audio/count")
assert response.status_code == 200
assert response.json() == 20
def test_search():
"""
Search the uploaded audio in Milvus/MySQL
"""
response = client.post(
"/audio/search/local?query_audio_path=.%2Fexample_audio%2Ftest.wav")
assert response.status_code == 200
assert len(response.json()) == 10
def test_data():
"""
Get the audio file
"""
response = client.get("/data?audio_path=.%2Fexample_audio%2Ftest.wav")
assert response.status_code == 200
if __name__ == "__main__":
download_audio_data()
test_load()
test_count()
test_search()
test_drop()
demos/speech_recognition/README.md
浏览文件 @ 490300f8
...@@ -84,5 +84,8 @@ Here is a list of pretrained models released by PaddleSpeech that can be used by ...@@ -84,5 +84,8 @@ Here is a list of pretrained models released by PaddleSpeech that can be used by
| Model | Language | Sample Rate | Model | Language | Sample Rate
| :--- | :---: | :---: | | :--- | :---: | :---: |
| conformer_wenetspeech| zh| 16000 | conformer_wenetspeech| zh| 16k
| transformer_librispeech| en| 16000 | transformer_librispeech| en| 16k
| deepspeech2offline_aishell| zh| 16k
| deepspeech2online_aishell | zh | 16k
|deepspeech2offline_librispeech|en| 16k
demos/speech_recognition/README_cn.md
浏览文件 @ 490300f8
...@@ -81,5 +81,8 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -81,5 +81,8 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
| 模型 | 语言 | 采样率 | 模型 | 语言 | 采样率
| :--- | :---: | :---: | | :--- | :---: | :---: |
| conformer_wenetspeech| zh| 16000 | conformer_wenetspeech | zh | 16k
| transformer_librispeech| en| 16000 | transformer_librispeech | en | 16k
| deepspeech2offline_aishell| zh| 16k
| deepspeech2online_aishell | zh | 16k
| deepspeech2offline_librispeech | en | 16k
demos/speech_server/README.md
浏览文件 @ 490300f8
...@@ -110,21 +110,22 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -110,21 +110,22 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
- Python API - Python API
```python ```python
from paddlespeech.server.bin.paddlespeech_client import ASRClientExecutor from paddlespeech.server.bin.paddlespeech_client import ASRClientExecutor
import json
asrclient_executor = ASRClientExecutor() asrclient_executor = ASRClientExecutor()
asrclient_executor( res = asrclient_executor(
input="./zh.wav", input="./zh.wav",
server_ip="127.0.0.1", server_ip="127.0.0.1",
port=8090, port=8090,
sample_rate=16000, sample_rate=16000,
lang="zh_cn", lang="zh_cn",
audio_format="wav") audio_format="wav")
print(res.json())
``` ```
Output: Output:
```bash ```bash
{'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'transcription': '我认为跑步最重要的就是给我带来了身体健康'}} {'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'transcription': '我认为跑步最重要的就是给我带来了身体健康'}}
time cost 0.604353 s.
``` ```
### 5. TTS Client Usage ### 5. TTS Client Usage
...@@ -146,7 +147,7 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -146,7 +147,7 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
- `speed`: Audio speed, the value should be set between 0 and 3. Default: 1.0 - `speed`: Audio speed, the value should be set between 0 and 3. Default: 1.0
- `volume`: Audio volume, the value should be set between 0 and 3. Default: 1.0 - `volume`: Audio volume, the value should be set between 0 and 3. Default: 1.0
- `sample_rate`: Sampling rate, choice: [0, 8000, 16000], the default is the same as the model. Default: 0 - `sample_rate`: Sampling rate, choice: [0, 8000, 16000], the default is the same as the model. Default: 0
- `output`: Output wave filepath. Default: `output.wav`. - `output`: Output wave filepath. Default: None, which means not to save the audio to the local.
Output: Output:
```bash ```bash
...@@ -160,9 +161,10 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -160,9 +161,10 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
- Python API - Python API
```python ```python
from paddlespeech.server.bin.paddlespeech_client import TTSClientExecutor from paddlespeech.server.bin.paddlespeech_client import TTSClientExecutor
import json
ttsclient_executor = TTSClientExecutor() ttsclient_executor = TTSClientExecutor()
ttsclient_executor( res = ttsclient_executor(
input="您好，欢迎使用百度飞桨语音合成服务。", input="您好，欢迎使用百度飞桨语音合成服务。",
server_ip="127.0.0.1", server_ip="127.0.0.1",
port=8090, port=8090,
...@@ -171,6 +173,11 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -171,6 +173,11 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
volume=1.0, volume=1.0,
sample_rate=0, sample_rate=0,
output="./output.wav") output="./output.wav")
response_dict = res.json()
print(response_dict["message"])
print("Save synthesized audio successfully on %s." % (response_dict['result']['save_path']))
print("Audio duration: %f s." %(response_dict['result']['duration']))
``` ```
Output: Output:
...@@ -178,7 +185,52 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -178,7 +185,52 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
{'description': 'success.'} {'description': 'success.'}
Save synthesized audio successfully on ./output.wav. Save synthesized audio successfully on ./output.wav.
Audio duration: 3.612500 s. Audio duration: 3.612500 s.
Response time: 0.388317 s.
```
### 6. CLS Client Usage
**Note:** The response time will be slightly longer when using the client for the first time
- Command Line (Recommended)
```
paddlespeech_client cls --server_ip 127.0.0.1 --port 8090 --input ./zh.wav
```
Usage:
```bash
paddlespeech_client cls --help
```
Arguments:
- `server_ip`: server ip. Default: 127.0.0.1
- `port`: server port. Default: 8090
- `input`(required): Audio file to be classified.
- `topk`: topk scores of classification result.
Output:
```bash
[2022-03-09 20:44:39,974] [ INFO] - {'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'topk': 1, 'results': [{'class_name': 'Speech', 'prob': 0.9027184844017029}]}}
[2022-03-09 20:44:39,975] [ INFO] - Response time 0.104360 s.
```
- Python API
```python
from paddlespeech.server.bin.paddlespeech_client import CLSClientExecutor
import json
clsclient_executor = CLSClientExecutor()
res = clsclient_executor(
input="./zh.wav",
server_ip="127.0.0.1",
port=8090,
topk=1)
print(res.json())
```
Output:
```bash
{'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'topk': 1, 'results': [{'class_name': 'Speech', 'prob': 0.9027184844017029}]}}
``` ```
...@@ -189,3 +241,6 @@ Get all models supported by the ASR service via `paddlespeech_server stats --tas ...@@ -189,3 +241,6 @@ Get all models supported by the ASR service via `paddlespeech_server stats --tas
### TTS model ### TTS model
Get all models supported by the TTS service via `paddlespeech_server stats --task tts`, where static models can be used for paddle inference inference. Get all models supported by the TTS service via `paddlespeech_server stats --task tts`, where static models can be used for paddle inference inference.
### CLS model
Get all models supported by the CLS service via `paddlespeech_server stats --task cls`, where static models can be used for paddle inference inference.
demos/speech_server/README_cn.md
浏览文件 @ 490300f8
...@@ -80,7 +80,7 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -80,7 +80,7 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
``` ```
### 4. ASR客户端使用方法 ### 4. ASR 客户端使用方法
**注意：** 初次使用客户端时响应时间会略长 **注意：** 初次使用客户端时响应时间会略长
- 命令行 (推荐使用) - 命令行 (推荐使用)
``` ```
...@@ -111,25 +111,26 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -111,25 +111,26 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
- Python API - Python API
```python ```python
from paddlespeech.server.bin.paddlespeech_client import ASRClientExecutor from paddlespeech.server.bin.paddlespeech_client import ASRClientExecutor
import json
asrclient_executor = ASRClientExecutor() asrclient_executor = ASRClientExecutor()
asrclient_executor( res = asrclient_executor(
input="./zh.wav", input="./zh.wav",
server_ip="127.0.0.1", server_ip="127.0.0.1",
port=8090, port=8090,
sample_rate=16000, sample_rate=16000,
lang="zh_cn", lang="zh_cn",
audio_format="wav") audio_format="wav")
print(res.json())
``` ```
输出: 输出:
```bash ```bash
{'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'transcription': '我认为跑步最重要的就是给我带来了身体健康'}} {'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'transcription': '我认为跑步最重要的就是给我带来了身体健康'}}
time cost 0.604353 s.
``` ```
### 5. TTS客户端使用方法 ### 5. TTS 客户端使用方法
**注意：** 初次使用客户端时响应时间会略长 **注意：** 初次使用客户端时响应时间会略长
- 命令行 (推荐使用) - 命令行 (推荐使用)
...@@ -150,7 +151,7 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -150,7 +151,7 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
- `speed`: 音频速度，该值应设置在 0 到 3 之间。 默认值：1.0 - `speed`: 音频速度，该值应设置在 0 到 3 之间。 默认值：1.0
- `volume`: 音频音量，该值应设置在 0 到 3 之间。 默认值： 1.0 - `volume`: 音频音量，该值应设置在 0 到 3 之间。 默认值： 1.0
- `sample_rate`: 采样率，可选 [0, 8000, 16000]，默认与模型相同。 默认值：0 - `sample_rate`: 采样率，可选 [0, 8000, 16000]，默认与模型相同。 默认值：0
- `output`: 输出音频的路径， 默认值：output.wav。 - `output`: 输出音频的路径， 默认值：None，表示不保存音频到本地。
输出: 输出:
```bash ```bash
...@@ -163,9 +164,10 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -163,9 +164,10 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
- Python API - Python API
```python ```python
from paddlespeech.server.bin.paddlespeech_client import TTSClientExecutor from paddlespeech.server.bin.paddlespeech_client import TTSClientExecutor
import json
ttsclient_executor = TTSClientExecutor() ttsclient_executor = TTSClientExecutor()
ttsclient_executor( res = ttsclient_executor(
input="您好，欢迎使用百度飞桨语音合成服务。", input="您好，欢迎使用百度飞桨语音合成服务。",
server_ip="127.0.0.1", server_ip="127.0.0.1",
port=8090, port=8090,
...@@ -174,6 +176,11 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -174,6 +176,11 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
volume=1.0, volume=1.0,
sample_rate=0, sample_rate=0,
output="./output.wav") output="./output.wav")
response_dict = res.json()
print(response_dict["message"])
print("Save synthesized audio successfully on %s." % (response_dict['result']['save_path']))
print("Audio duration: %f s." %(response_dict['result']['duration']))
``` ```
输出: 输出:
...@@ -181,13 +188,63 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee ...@@ -181,13 +188,63 @@ wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespee
{'description': 'success.'} {'description': 'success.'}
Save synthesized audio successfully on ./output.wav. Save synthesized audio successfully on ./output.wav.
Audio duration: 3.612500 s. Audio duration: 3.612500 s.
Response time: 0.388317 s.
``` ```
### 5. CLS 客户端使用方法
**注意：** 初次使用客户端时响应时间会略长
- 命令行 (推荐使用)
```
paddlespeech_client cls --server_ip 127.0.0.1 --port 8090 --input ./zh.wav
```
使用帮助:
```bash
paddlespeech_client cls --help
```
参数:
- `server_ip`: 服务端ip地址，默认: 127.0.0.1。
- `port`: 服务端口，默认: 8090。
- `input`(必须输入): 用于分类的音频文件。
- `topk`: 分类结果的topk。
输出:
```bash
[2022-03-09 20:44:39,974] [ INFO] - {'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'topk': 1, 'results': [{'class_name': 'Speech', 'prob': 0.9027184844017029}]}}
[2022-03-09 20:44:39,975] [ INFO] - Response time 0.104360 s.
```
- Python API
```python
from paddlespeech.server.bin.paddlespeech_client import CLSClientExecutor
import json
clsclient_executor = CLSClientExecutor()
res = clsclient_executor(
input="./zh.wav",
server_ip="127.0.0.1",
port=8090,
topk=1)
print(res.json())
```
输出:
```bash
{'success': True, 'code': 200, 'message': {'description': 'success'}, 'result': {'topk': 1, 'results': [{'class_name': 'Speech', 'prob': 0.9027184844017029}]}}
```
## 服务支持的模型 ## 服务支持的模型
### ASR支持的模型 ### ASR支持的模型
通过 `paddlespeech_server stats --task asr` 获取ASR服务支持的所有模型，其中静态模型可用于 paddle inference 推理。 通过 `paddlespeech_server stats --task asr` 获取ASR服务支持的所有模型，其中静态模型可用于 paddle inference 推理。
### TTS支持的模型 ### TTS支持的模型
通过 `paddlespeech_server stats --task tts` 获取TTS服务支持的所有模型，其中静态模型可用于 paddle inference 推理。 通过 `paddlespeech_server stats --task tts` 获取TTS服务支持的所有模型，其中静态模型可用于 paddle inference 推理。
### CLS支持的模型
通过 `paddlespeech_server stats --task cls` 获取CLS服务支持的所有模型，其中静态模型可用于 paddle inference 推理。
demos/speech_server/cls_client.sh 0 → 100644
浏览文件 @ 490300f8
#!/bin/bash
wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespeech.bj.bcebos.com/PaddleAudio/en.wav
paddlespeech_client cls --server_ip 127.0.0.1 --port 8090 --input ./zh.wav --topk 1
demos/speech_server/conf/application.yaml
浏览文件 @ 490300f8
...@@ -9,12 +9,14 @@ port: 8090 ...@@ -9,12 +9,14 @@ port: 8090
# The task format in the engin_list is: <speech task>_<engine type> # The task format in the engin_list is: <speech task>_<engine type>
# task choices = ['asr_python', 'asr_inference', 'tts_python', 'tts_inference'] # task choices = ['asr_python', 'asr_inference', 'tts_python', 'tts_inference']
engine_list: ['asr_python', 'tts_python'] engine_list: ['asr_python', 'tts_python', 'cls_python']
################################################################################# #################################################################################
# ENGINE CONFIG # # ENGINE CONFIG #
################################################################################# #################################################################################
################################### ASR #########################################
################### speech task: asr; engine_type: python ####################### ################### speech task: asr; engine_type: python #######################
asr_python: asr_python:
model: 'conformer_wenetspeech' model: 'conformer_wenetspeech'
...@@ -46,6 +48,7 @@ asr_inference: ...@@ -46,6 +48,7 @@ asr_inference:
summary: True # False -> do not show predictor config summary: True # False -> do not show predictor config
################################### TTS #########################################
################### speech task: tts; engine_type: python ####################### ################### speech task: tts; engine_type: python #######################
tts_python: tts_python:
# am (acoustic model) choices=['speedyspeech_csmsc', 'fastspeech2_csmsc', # am (acoustic model) choices=['speedyspeech_csmsc', 'fastspeech2_csmsc',
...@@ -105,3 +108,30 @@ tts_inference: ...@@ -105,3 +108,30 @@ tts_inference:
# others # others
lang: 'zh' lang: 'zh'
################################### CLS #########################################
################### speech task: cls; engine_type: python #######################
cls_python:
# model choices=['panns_cnn14', 'panns_cnn10', 'panns_cnn6']
model: 'panns_cnn14'
cfg_path: # [optional] Config of cls task.
ckpt_path: # [optional] Checkpoint file of model.
label_file: # [optional] Label file of cls task.
device: # set 'gpu:id' or 'cpu'
################### speech task: cls; engine_type: inference #######################
cls_inference:
# model_type choices=['panns_cnn14', 'panns_cnn10', 'panns_cnn6']
model_type: 'panns_cnn14'
cfg_path:
model_path: # the pdmodel file of am static model [optional]
params_path: # the pdiparams file of am static model [optional]
label_file: # [optional] Label file of cls task.
predictor_conf:
device: # set 'gpu:id' or 'cpu'
switch_ir_optim: True
glog_info: False # True -> print glog
summary: True # False -> do not show predictor config
docs/source/reference.md
浏览文件 @ 490300f8
...@@ -35,3 +35,7 @@ We borrowed a lot of code from these repos to build `model` and `engine`, thanks ...@@ -35,3 +35,7 @@ We borrowed a lot of code from these repos to build `model` and `engine`, thanks
* [librosa](https://github.com/librosa/librosa/blob/main/LICENSE.md) * [librosa](https://github.com/librosa/librosa/blob/main/LICENSE.md)
- ISC License - ISC License
- Audio feature - Audio feature
* [ThreadPool](https://github.com/progschj/ThreadPool/blob/master/COPYING)
- zlib License
- ThreadPool
docs/source/released_model.md
浏览文件 @ 490300f8
...@@ -49,17 +49,19 @@ Model Type | Dataset| Example Link | Pretrained Models| Static Models|Size (stat ...@@ -49,17 +49,19 @@ Model Type | Dataset| Example Link | Pretrained Models| Static Models|Size (stat
WaveFlow| LJSpeech |[waveflow-ljspeech](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc0)|[waveflow_ljspeech_ckpt_0.3.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/waveflow/waveflow_ljspeech_ckpt_0.3.zip)||| WaveFlow| LJSpeech |[waveflow-ljspeech](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc0)|[waveflow_ljspeech_ckpt_0.3.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/waveflow/waveflow_ljspeech_ckpt_0.3.zip)|||
Parallel WaveGAN| CSMSC |[PWGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc1)|[pwg_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_baker_ckpt_0.4.zip)|[pwg_baker_static_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_baker_static_0.4.zip)|5.1MB| Parallel WaveGAN| CSMSC |[PWGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc1)|[pwg_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_baker_ckpt_0.4.zip)|[pwg_baker_static_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_baker_static_0.4.zip)|5.1MB|
Parallel WaveGAN| LJSpeech |[PWGAN-ljspeech](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc1)|[pwg_ljspeech_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_ljspeech_ckpt_0.5.zip)||| Parallel WaveGAN| LJSpeech |[PWGAN-ljspeech](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc1)|[pwg_ljspeech_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_ljspeech_ckpt_0.5.zip)|||
Parallel WaveGAN|AISHELL-3 |[PWGAN-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/voc1)|[pwg_aishell3_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_aishell3_ckpt_0.5.zip)||| Parallel WaveGAN| AISHELL-3 |[PWGAN-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/voc1)|[pwg_aishell3_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_aishell3_ckpt_0.5.zip)|||
Parallel WaveGAN| VCTK |[PWGAN-vctk](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/vctk/voc1)|[pwg_vctk_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_vctk_ckpt_0.5.zip)||| Parallel WaveGAN| VCTK |[PWGAN-vctk](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/vctk/voc1)|[pwg_vctk_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_vctk_ckpt_0.5.zip)|||
|Multi Band MelGAN | CSMSC |[MB MelGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc3) | [mb_melgan_csmsc_ckpt_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/mb_melgan/mb_melgan_csmsc_ckpt_0.1.1.zip) <br>[mb_melgan_baker_finetune_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/mb_melgan/mb_melgan_baker_finetune_ckpt_0.5.zip)|[mb_melgan_csmsc_static_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/mb_melgan/mb_melgan_csmsc_static_0.1.1.zip) |8.2MB| |Multi Band MelGAN | CSMSC |[MB MelGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc3) | [mb_melgan_csmsc_ckpt_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/mb_melgan/mb_melgan_csmsc_ckpt_0.1.1.zip) <br>[mb_melgan_baker_finetune_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/mb_melgan/mb_melgan_baker_finetune_ckpt_0.5.zip)|[mb_melgan_csmsc_static_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/mb_melgan/mb_melgan_csmsc_static_0.1.1.zip) |8.2MB|
Style MelGAN | CSMSC |[Style MelGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc4)|[style_melgan_csmsc_ckpt_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/style_melgan/style_melgan_csmsc_ckpt_0.1.1.zip)| | | Style MelGAN | CSMSC |[Style MelGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc4)|[style_melgan_csmsc_ckpt_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/style_melgan/style_melgan_csmsc_ckpt_0.1.1.zip)| | |
HiFiGAN | CSMSC |[HiFiGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc5)|[hifigan_csmsc_ckpt_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_csmsc_ckpt_0.1.1.zip)|[hifigan_csmsc_static_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_csmsc_static_0.1.1.zip)|50MB| HiFiGAN | CSMSC |[HiFiGAN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc5)|[hifigan_csmsc_ckpt_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_csmsc_ckpt_0.1.1.zip)|[hifigan_csmsc_static_0.1.1.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_csmsc_static_0.1.1.zip)|50MB|
HiFiGAN | AISHELL-3 |[HiFiGAN-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/voc5)|[hifigan_aishell3_ckpt_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_aishell3_ckpt_0.2.0.zip)|||
HiFiGAN | VCTK |[HiFiGAN-vctk](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/vctk/voc5)|[hifigan_aishell3_ckpt_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_aishell3_ckpt_0.2.0.zip)|||
WaveRNN | CSMSC |[WaveRNN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc6)|[wavernn_csmsc_ckpt_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/wavernn/wavernn_csmsc_ckpt_0.2.0.zip)|[wavernn_csmsc_static_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/wavernn/wavernn_csmsc_static_0.2.0.zip)|18MB| WaveRNN | CSMSC |[WaveRNN-csmsc](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc6)|[wavernn_csmsc_ckpt_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/wavernn/wavernn_csmsc_ckpt_0.2.0.zip)|[wavernn_csmsc_static_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/wavernn/wavernn_csmsc_static_0.2.0.zip)|18MB|
### Voice Cloning ### Voice Cloning
Model Type | Dataset| Example Link | Pretrained Models Model Type | Dataset| Example Link | Pretrained Models
:-------------:| :------------:| :-----: | :-----: :-------------:| :------------:| :-----: | :-----: |
GE2E| AISHELL-3, etc. |[ge2e](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/ge2e)|[ge2e_ckpt_0.3.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/ge2e/ge2e_ckpt_0.3.zip) GE2E| AISHELL-3, etc. |[ge2e](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/ge2e)|[ge2e_ckpt_0.3.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/ge2e/ge2e_ckpt_0.3.zip)
GE2E + Tactron2| AISHELL-3 |[ge2e-tactron2-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/vc0)|[tacotron2_aishell3_ckpt_vc0_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/tacotron2/tacotron2_aishell3_ckpt_vc0_0.2.0.zip) GE2E + Tactron2| AISHELL-3 |[ge2e-tactron2-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/vc0)|[tacotron2_aishell3_ckpt_vc0_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/tacotron2/tacotron2_aishell3_ckpt_vc0_0.2.0.zip)
GE2E + FastSpeech2 | AISHELL-3 |[ge2e-fastspeech2-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/vc1)|[fastspeech2_nosil_aishell3_vc1_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/fastspeech2/fastspeech2_nosil_aishell3_vc1_ckpt_0.5.zip) GE2E + FastSpeech2 | AISHELL-3 |[ge2e-fastspeech2-aishell3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/vc1)|[fastspeech2_nosil_aishell3_vc1_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/fastspeech2/fastspeech2_nosil_aishell3_vc1_ckpt_0.5.zip)
...@@ -67,9 +69,9 @@ GE2E + FastSpeech2 | AISHELL-3 |[ge2e-fastspeech2-aishell3](https://github.com/ ...@@ -67,9 +69,9 @@ GE2E + FastSpeech2 | AISHELL-3 |[ge2e-fastspeech2-aishell3](https://github.com/
## Audio Classification Models ## Audio Classification Models
Model Type | Dataset| Example Link | Pretrained Models Model Type | Dataset| Example Link | Pretrained Models | Static Models
:-------------:| :------------:| :-----: | :-----: :-------------:| :------------:| :-----: | :-----: | :-----:
PANN | Audioset| [audioset_tagging_cnn](https://github.com/qiuqiangkong/audioset_tagging_cnn) | [panns_cnn6.pdparams](https://bj.bcebos.com/paddleaudio/models/panns_cnn6.pdparams), [panns_cnn10.pdparams](https://bj.bcebos.com/paddleaudio/models/panns_cnn10.pdparams), [panns_cnn14.pdparams](https://bj.bcebos.com/paddleaudio/models/panns_cnn14.pdparams) PANN | Audioset| [audioset_tagging_cnn](https://github.com/qiuqiangkong/audioset_tagging_cnn) | [panns_cnn6.pdparams](https://bj.bcebos.com/paddleaudio/models/panns_cnn6.pdparams), [panns_cnn10.pdparams](https://bj.bcebos.com/paddleaudio/models/panns_cnn10.pdparams), [panns_cnn14.pdparams](https://bj.bcebos.com/paddleaudio/models/panns_cnn14.pdparams) | [panns_cnn6_static.tar.gz](https://paddlespeech.bj.bcebos.com/cls/inference_model/panns_cnn6_static.tar.gz)(18M), [panns_cnn10_static.tar.gz](https://paddlespeech.bj.bcebos.com/cls/inference_model/panns_cnn10_static.tar.gz)(19M), [panns_cnn14_static.tar.gz](https://paddlespeech.bj.bcebos.com/cls/inference_model/panns_cnn14_static.tar.gz)(289M)
PANN | ESC-50 |[pann-esc50](../../examples/esc50/cls0)|[esc50_cnn6.tar.gz](https://paddlespeech.bj.bcebos.com/cls/esc50/esc50_cnn6.tar.gz), [esc50_cnn10.tar.gz](https://paddlespeech.bj.bcebos.com/cls/esc50/esc50_cnn10.tar.gz), [esc50_cnn14.tar.gz](https://paddlespeech.bj.bcebos.com/cls/esc50/esc50_cnn14.tar.gz) PANN | ESC-50 |[pann-esc50](../../examples/esc50/cls0)|[esc50_cnn6.tar.gz](https://paddlespeech.bj.bcebos.com/cls/esc50/esc50_cnn6.tar.gz), [esc50_cnn10.tar.gz](https://paddlespeech.bj.bcebos.com/cls/esc50/esc50_cnn10.tar.gz), [esc50_cnn14.tar.gz](https://paddlespeech.bj.bcebos.com/cls/esc50/esc50_cnn14.tar.gz)
## Punctuation Restoration Models ## Punctuation Restoration Models
......
examples/aishell3/tts3/local/synthesize.sh
浏览文件 @ 490300f8
...@@ -4,18 +4,44 @@ config_path=$1 ...@@ -4,18 +4,44 @@ config_path=$1
train_output_path=$2 train_output_path=$2
ckpt_name=$3 ckpt_name=$3
FLAGS_allocator_strategy=naive_best_fit \ stage=0
FLAGS_fraction_of_gpu_memory_to_use=0.01 \ stop_stage=0
python3 ${BIN_DIR}/../synthesize.py \
--am=fastspeech2_aishell3 \ # pwgan
--am_config=${config_path} \ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ FLAGS_allocator_strategy=naive_best_fit \
--am_stat=dump/train/speech_stats.npy \ FLAGS_fraction_of_gpu_memory_to_use=0.01 \
--voc=pwgan_aishell3 \ python3 ${BIN_DIR}/../synthesize.py \
--voc_config=pwg_aishell3_ckpt_0.5/default.yaml \ --am=fastspeech2_aishell3 \
--voc_ckpt=pwg_aishell3_ckpt_0.5/snapshot_iter_1000000.pdz \ --am_config=${config_path} \
--voc_stat=pwg_aishell3_ckpt_0.5/feats_stats.npy \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--test_metadata=dump/test/norm/metadata.jsonl \ --am_stat=dump/train/speech_stats.npy \
--output_dir=${train_output_path}/test \ --voc=pwgan_aishell3 \
--phones_dict=dump/phone_id_map.txt \ --voc_config=pwg_aishell3_ckpt_0.5/default.yaml \
--speaker_dict=dump/speaker_id_map.txt --voc_ckpt=pwg_aishell3_ckpt_0.5/snapshot_iter_1000000.pdz \
--voc_stat=pwg_aishell3_ckpt_0.5/feats_stats.npy \
--test_metadata=dump/test/norm/metadata.jsonl \
--output_dir=${train_output_path}/test \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt
fi
# hifigan
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
FLAGS_allocator_strategy=naive_best_fit \
FLAGS_fraction_of_gpu_memory_to_use=0.01 \
python3 ${BIN_DIR}/../synthesize.py \
--am=fastspeech2_aishell3 \
--am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \
--voc=hifigan_aishell3 \
--voc_config=hifigan_aishell3_ckpt_0.2.0/default.yaml \
--voc_ckpt=hifigan_aishell3_ckpt_0.2.0/snapshot_iter_2500000.pd \
--voc_stat=hifigan_aishell3_ckpt_0.2.0/feats_stats.npy \
--test_metadata=dump/test/norm/metadata.jsonl \
--output_dir=${train_output_path}/test \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt
fi
examples/aishell3/tts3/local/synthesize_e2e.sh
浏览文件 @ 490300f8
...@@ -4,21 +4,50 @@ config_path=$1 ...@@ -4,21 +4,50 @@ config_path=$1
train_output_path=$2 train_output_path=$2
ckpt_name=$3 ckpt_name=$3
FLAGS_allocator_strategy=naive_best_fit \ stage=0
FLAGS_fraction_of_gpu_memory_to_use=0.01 \ stop_stage=0
python3 ${BIN_DIR}/../synthesize_e2e.py \
--am=fastspeech2_aishell3 \ # pwgan
--am_config=${config_path} \ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ FLAGS_allocator_strategy=naive_best_fit \
--am_stat=dump/train/speech_stats.npy \ FLAGS_fraction_of_gpu_memory_to_use=0.01 \
--voc=pwgan_aishell3 \ python3 ${BIN_DIR}/../synthesize_e2e.py \
--voc_config=pwg_aishell3_ckpt_0.5/default.yaml \ --am=fastspeech2_aishell3 \
--voc_ckpt=pwg_aishell3_ckpt_0.5/snapshot_iter_1000000.pdz \ --am_config=${config_path} \
--voc_stat=pwg_aishell3_ckpt_0.5/feats_stats.npy \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--lang=zh \ --am_stat=dump/train/speech_stats.npy \
--text=${BIN_DIR}/../sentences.txt \ --voc=pwgan_aishell3 \
--output_dir=${train_output_path}/test_e2e \ --voc_config=pwg_aishell3_ckpt_0.5/default.yaml \
--phones_dict=dump/phone_id_map.txt \ --voc_ckpt=pwg_aishell3_ckpt_0.5/snapshot_iter_1000000.pdz \
--speaker_dict=dump/speaker_id_map.txt \ --voc_stat=pwg_aishell3_ckpt_0.5/feats_stats.npy \
--spk_id=0 \ --lang=zh \
--inference_dir=${train_output_path}/inference --text=${BIN_DIR}/../sentences.txt \
--output_dir=${train_output_path}/test_e2e \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt \
--spk_id=0 \
--inference_dir=${train_output_path}/inference
fi
# hifigan
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
echo "in hifigan syn_e2e"
FLAGS_allocator_strategy=naive_best_fit \
FLAGS_fraction_of_gpu_memory_to_use=0.01 \
python3 ${BIN_DIR}/../synthesize_e2e.py \
--am=fastspeech2_aishell3 \
--am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=fastspeech2_nosil_aishell3_ckpt_0.4/speech_stats.npy \
--voc=hifigan_aishell3 \
--voc_config=hifigan_aishell3_ckpt_0.2.0/default.yaml \
--voc_ckpt=hifigan_aishell3_ckpt_0.2.0/snapshot_iter_2500000.pdz \
--voc_stat=hifigan_aishell3_ckpt_0.2.0/feats_stats.npy \
--lang=zh \
--text=${BIN_DIR}/../sentences.txt \
--output_dir=${train_output_path}/test_e2e \
--phones_dict=fastspeech2_nosil_aishell3_ckpt_0.4/phone_id_map.txt \
--speaker_dict=fastspeech2_nosil_aishell3_ckpt_0.4/speaker_id_map.txt \
--spk_id=0 \
--inference_dir=${train_output_path}/inference
fi
examples/aishell3/vc0/local/preprocess.sh
浏览文件 @ 490300f8
#!/bin/bash #!/bin/bash
stage=3 stage=0
stop_stage=100 stop_stage=100
config_path=$1 config_path=$1
......
examples/aishell3/voc1/run.sh
浏览文件 @ 490300f8
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
set -e set -e
source path.sh source path.sh
gpus=0 gpus=0,1
stage=0 stage=0
stop_stage=100 stop_stage=100
......
examples/aishell3/voc5/README.md
浏览文件 @ 490300f8
...@@ -135,8 +135,22 @@ optional arguments: ...@@ -135,8 +135,22 @@ optional arguments:
3. `--test-metadata` is the metadata of the test dataset. Use the `metadata.jsonl` in the `dev/norm` subfolder from the processed directory. 3. `--test-metadata` is the metadata of the test dataset. Use the `metadata.jsonl` in the `dev/norm` subfolder from the processed directory.
4. `--output-dir` is the directory to save the synthesized audio files. 4. `--output-dir` is the directory to save the synthesized audio files.
5. `--ngpu` is the number of gpus to use, if ngpu == 0, use cpu. 5. `--ngpu` is the number of gpus to use, if ngpu == 0, use cpu.
## Pretrained Models ## Pretrained Models
The pretrained model can be downloaded here [hifigan_aishell3_ckpt_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_aishell3_ckpt_0.2.0.zip).
Model | Step | eval/generator_loss | eval/mel_loss| eval/feature_matching_loss
:-------------:| :------------:| :-----: | :-----: | :--------:
default| 1(gpu) x 2500000|24.060|0.1068|7.499
HiFiGAN checkpoint contains files listed below.
```text
hifigan_aishell3_ckpt_0.2.0
├── default.yaml # default config used to train hifigan
├── feats_stats.npy # statistics used to normalize spectrogram when training hifigan
└── snapshot_iter_2500000.pdz # generator parameters of hifigan
```
## Acknowledgement ## Acknowledgement
We adapted some code from https://github.com/kan-bayashi/ParallelWaveGAN. We adapted some code from https://github.com/kan-bayashi/ParallelWaveGAN.
examples/csmsc/tts0/local/synthesize_e2e.sh
浏览文件 @ 490300f8
...@@ -7,7 +7,7 @@ ckpt_name=$3 ...@@ -7,7 +7,7 @@ ckpt_name=$3
stage=0 stage=0
stop_stage=0 stop_stage=0
# TODO: tacotron2 动转静的结果没有静态图的响亮, 可能还是 decode 的时候某个函数动静不对齐 # TODO: tacotron2 动转静的结果没有动态图的响亮, 可能还是 decode 的时候某个函数动静不对齐
# pwgan # pwgan
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
FLAGS_allocator_strategy=naive_best_fit \ FLAGS_allocator_strategy=naive_best_fit \
......
examples/csmsc/tts2/local/synthesize.sh
浏览文件 @ 490300f8
...@@ -14,7 +14,7 @@ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then ...@@ -14,7 +14,7 @@ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
--am=speedyspeech_csmsc \ --am=speedyspeech_csmsc \
--am_config=${config_path} \ --am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \ --am_stat=dump/train/feats_stats.npy \
--voc=pwgan_csmsc \ --voc=pwgan_csmsc \
--voc_config=pwg_baker_ckpt_0.4/pwg_default.yaml \ --voc_config=pwg_baker_ckpt_0.4/pwg_default.yaml \
--voc_ckpt=pwg_baker_ckpt_0.4/pwg_snapshot_iter_400000.pdz \ --voc_ckpt=pwg_baker_ckpt_0.4/pwg_snapshot_iter_400000.pdz \
...@@ -34,7 +34,7 @@ if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then ...@@ -34,7 +34,7 @@ if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
--am=speedyspeech_csmsc \ --am=speedyspeech_csmsc \
--am_config=${config_path} \ --am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \ --am_stat=dump/train/feats_stats.npy \
--voc=mb_melgan_csmsc \ --voc=mb_melgan_csmsc \
--voc_config=mb_melgan_csmsc_ckpt_0.1.1/default.yaml \ --voc_config=mb_melgan_csmsc_ckpt_0.1.1/default.yaml \
--voc_ckpt=mb_melgan_csmsc_ckpt_0.1.1/snapshot_iter_1000000.pdz\ --voc_ckpt=mb_melgan_csmsc_ckpt_0.1.1/snapshot_iter_1000000.pdz\
...@@ -53,7 +53,7 @@ if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then ...@@ -53,7 +53,7 @@ if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
--am=speedyspeech_csmsc \ --am=speedyspeech_csmsc \
--am_config=${config_path} \ --am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \ --am_stat=dump/train/feats_stats.npy \
--voc=style_melgan_csmsc \ --voc=style_melgan_csmsc \
--voc_config=style_melgan_csmsc_ckpt_0.1.1/default.yaml \ --voc_config=style_melgan_csmsc_ckpt_0.1.1/default.yaml \
--voc_ckpt=style_melgan_csmsc_ckpt_0.1.1/snapshot_iter_1500000.pdz \ --voc_ckpt=style_melgan_csmsc_ckpt_0.1.1/snapshot_iter_1500000.pdz \
...@@ -73,7 +73,7 @@ if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then ...@@ -73,7 +73,7 @@ if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
--am=speedyspeech_csmsc \ --am=speedyspeech_csmsc \
--am_config=${config_path} \ --am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \ --am_stat=dump/train/feats_stats.npy \
--voc=hifigan_csmsc \ --voc=hifigan_csmsc \
--voc_config=hifigan_csmsc_ckpt_0.1.1/default.yaml \ --voc_config=hifigan_csmsc_ckpt_0.1.1/default.yaml \
--voc_ckpt=hifigan_csmsc_ckpt_0.1.1/snapshot_iter_2500000.pdz \ --voc_ckpt=hifigan_csmsc_ckpt_0.1.1/snapshot_iter_2500000.pdz \
...@@ -93,7 +93,7 @@ if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then ...@@ -93,7 +93,7 @@ if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
--am=speedyspeech_csmsc \ --am=speedyspeech_csmsc \
--am_config=${config_path} \ --am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \ --am_stat=dump/train/feats_stats.npy \
--voc=wavernn_csmsc \ --voc=wavernn_csmsc \
--voc_config=wavernn_csmsc_ckpt_0.2.0/default.yaml \ --voc_config=wavernn_csmsc_ckpt_0.2.0/default.yaml \
--voc_ckpt=wavernn_csmsc_ckpt_0.2.0/snapshot_iter_400000.pdz \ --voc_ckpt=wavernn_csmsc_ckpt_0.2.0/snapshot_iter_400000.pdz \
......
examples/vctk/tts3/local/synthesize.sh
浏览文件 @ 490300f8
...@@ -4,18 +4,43 @@ config_path=$1 ...@@ -4,18 +4,43 @@ config_path=$1
train_output_path=$2 train_output_path=$2
ckpt_name=$3 ckpt_name=$3
FLAGS_allocator_strategy=naive_best_fit \ stage=0
FLAGS_fraction_of_gpu_memory_to_use=0.01 \ stop_stage=0
python3 ${BIN_DIR}/../synthesize.py \
--am=fastspeech2_vctk \ # pwgan
--am_config=${config_path} \ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ FLAGS_allocator_strategy=naive_best_fit \
--am_stat=dump/train/speech_stats.npy \ FLAGS_fraction_of_gpu_memory_to_use=0.01 \
--voc=pwgan_vctk \ python3 ${BIN_DIR}/../synthesize.py \
--voc_config=pwg_vctk_ckpt_0.1.1/default.yaml \ --am=fastspeech2_vctk \
--voc_ckpt=pwg_vctk_ckpt_0.1.1/snapshot_iter_1500000.pdz \ --am_config=${config_path} \
--voc_stat=pwg_vctk_ckpt_0.1.1/feats_stats.npy \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--test_metadata=dump/test/norm/metadata.jsonl \ --am_stat=dump/train/speech_stats.npy \
--output_dir=${train_output_path}/test \ --voc=pwgan_vctk \
--phones_dict=dump/phone_id_map.txt \ --voc_config=pwg_vctk_ckpt_0.1.1/default.yaml \
--speaker_dict=dump/speaker_id_map.txt --voc_ckpt=pwg_vctk_ckpt_0.1.1/snapshot_iter_1500000.pdz \
--voc_stat=pwg_vctk_ckpt_0.1.1/feats_stats.npy \
--test_metadata=dump/test/norm/metadata.jsonl \
--output_dir=${train_output_path}/test \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt
fi
# hifigan
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
FLAGS_allocator_strategy=naive_best_fit \
FLAGS_fraction_of_gpu_memory_to_use=0.01 \
python3 ${BIN_DIR}/../synthesize.py \
--am=fastspeech2_aishell3 \
--am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \
--voc=hifigan_vctk \
--voc_config=hifigan_vctk_ckpt_0.2.0/default.yaml \
--voc_ckpt=hifigan_vctk_ckpt_0.2.0/snapshot_iter_2500000.pdz \
--voc_stat=hifigan_vctk_ckpt_0.2.0/feats_stats.npy \
--test_metadata=dump/test/norm/metadata.jsonl \
--output_dir=${train_output_path}/test \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt
fi
examples/vctk/tts3/local/synthesize_e2e.sh
浏览文件 @ 490300f8
...@@ -4,21 +4,49 @@ config_path=$1 ...@@ -4,21 +4,49 @@ config_path=$1
train_output_path=$2 train_output_path=$2
ckpt_name=$3 ckpt_name=$3
FLAGS_allocator_strategy=naive_best_fit \ stage=0
FLAGS_fraction_of_gpu_memory_to_use=0.01 \ stop_stage=0
python3 ${BIN_DIR}/../synthesize_e2e.py \
--am=fastspeech2_vctk \ # pwgan
--am_config=${config_path} \ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \ FLAGS_allocator_strategy=naive_best_fit \
--am_stat=dump/train/speech_stats.npy \ FLAGS_fraction_of_gpu_memory_to_use=0.01 \
--voc=pwgan_vctk \ python3 ${BIN_DIR}/../synthesize_e2e.py \
--voc_config=pwg_vctk_ckpt_0.1.1/default.yaml \ --am=fastspeech2_vctk \
--voc_ckpt=pwg_vctk_ckpt_0.1.1/snapshot_iter_1500000.pdz \ --am_config=${config_path} \
--voc_stat=pwg_vctk_ckpt_0.1.1/feats_stats.npy \ --am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--lang=en \ --am_stat=dump/train/speech_stats.npy \
--text=${BIN_DIR}/../sentences_en.txt \ --voc=pwgan_vctk \
--output_dir=${train_output_path}/test_e2e \ --voc_config=pwg_vctk_ckpt_0.1.1/default.yaml \
--phones_dict=dump/phone_id_map.txt \ --voc_ckpt=pwg_vctk_ckpt_0.1.1/snapshot_iter_1500000.pdz \
--speaker_dict=dump/speaker_id_map.txt \ --voc_stat=pwg_vctk_ckpt_0.1.1/feats_stats.npy \
--spk_id=0 \ --lang=en \
--inference_dir=${train_output_path}/inference --text=${BIN_DIR}/../sentences_en.txt \
--output_dir=${train_output_path}/test_e2e \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt \
--spk_id=0 \
--inference_dir=${train_output_path}/inference
fi
# hifigan
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
FLAGS_allocator_strategy=naive_best_fit \
FLAGS_fraction_of_gpu_memory_to_use=0.01 \
python3 ${BIN_DIR}/../synthesize_e2e.py \
--am=fastspeech2_vctk \
--am_config=${config_path} \
--am_ckpt=${train_output_path}/checkpoints/${ckpt_name} \
--am_stat=dump/train/speech_stats.npy \
--voc=hifigan_vctk \
--voc_config=hifigan_vctk_ckpt_0.2.0/default.yaml \
--voc_ckpt=hifigan_vctk_ckpt_0.2.0/snapshot_iter_2500000.pdz \
--voc_stat=hifigan_vctk_ckpt_0.2.0/feats_stats.npy \
--lang=en \
--text=${BIN_DIR}/../sentences_en.txt \
--output_dir=${train_output_path}/test_e2e \
--phones_dict=dump/phone_id_map.txt \
--speaker_dict=dump/speaker_id_map.txt \
--spk_id=0 \
--inference_dir=${train_output_path}/inference
fi
examples/vctk/voc5/README.md
浏览文件 @ 490300f8
...@@ -133,7 +133,21 @@ optional arguments: ...@@ -133,7 +133,21 @@ optional arguments:
5. `--ngpu` is the number of gpus to use, if ngpu == 0, use cpu. 5. `--ngpu` is the number of gpus to use, if ngpu == 0, use cpu.
## Pretrained Model ## Pretrained Model
The pretrained model can be downloaded here [hifigan_vctk_ckpt_0.2.0.zip](https://paddlespeech.bj.bcebos.com/Parakeet/released_models/hifigan/hifigan_vctk_ckpt_0.2.0.zip).
Model | Step | eval/generator_loss | eval/mel_loss| eval/feature_matching_loss
:-------------:| :------------:| :-----: | :-----: | :--------:
default| 1(gpu) x 2500000|58.092|0.1234|24.384
HiFiGAN checkpoint contains files listed below.
```text
hifigan_vctk_ckpt_0.2.0
├── default.yaml # default config used to train hifigan
├── feats_stats.npy # statistics used to normalize spectrogram when training hifigan
└── snapshot_iter_2500000.pdz # generator parameters of hifigan
```
## Acknowledgement ## Acknowledgement
We adapted some code from https://github.com/kan-bayashi/ParallelWaveGAN. We adapted some code from https://github.com/kan-bayashi/ParallelWaveGAN.
paddleaudio/CHANGELOG.md
浏览文件 @ 490300f8
# Changelog # Changelog
Date: 2022-3-15, Author: Xiaojie Chen.
- kaldi and librosa mfcc, fbank, spectrogram.
- unit test and benchmark.
Date: 2022-2-25, Author: Hui Zhang. Date: 2022-2-25, Author: Hui Zhang.
- Refactor architecture. - Refactor architecture.
- dtw distance and mcd style dtw - dtw distance and mcd style dtw.
paddleaudio/paddleaudio/backends/soundfile_backend.py
浏览文件 @ 490300f8
...@@ -11,6 +11,7 @@ ...@@ -11,6 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import os
import warnings import warnings
from typing import Optional from typing import Optional
from typing import Tuple from typing import Tuple
...@@ -19,7 +20,6 @@ from typing import Union ...@@ -19,7 +20,6 @@ from typing import Union
import numpy as np import numpy as np
import resampy import resampy
import soundfile as sf import soundfile as sf
from numpy import ndarray as array
from scipy.io import wavfile from scipy.io import wavfile
from ..utils import ParameterError from ..utils import ParameterError
...@@ -38,13 +38,21 @@ RESAMPLE_MODES = ['kaiser_best', 'kaiser_fast'] ...@@ -38,13 +38,21 @@ RESAMPLE_MODES = ['kaiser_best', 'kaiser_fast']
EPS = 1e-8 EPS = 1e-8
def resample(y: array, src_sr: int, target_sr: int, def resample(y: np.ndarray,
mode: str='kaiser_fast') -> array: src_sr: int,
""" Audio resampling target_sr: int,
This function is the same as using resampy.resample(). mode: str='kaiser_fast') -> np.ndarray:
Notes: """Audio resampling.
The default mode is kaiser_fast. For better audio quality, use mode = 'kaiser_fast'
""" Args:
y (np.ndarray): Input waveform array in 1D or 2D.
src_sr (int): Source sample rate.
target_sr (int): Target sample rate.
mode (str, optional): The resampling filter to use. Defaults to 'kaiser_fast'.
Returns:
np.ndarray: `y` resampled to `target_sr`
"""
if mode == 'kaiser_best': if mode == 'kaiser_best':
warnings.warn( warnings.warn(
...@@ -53,7 +61,7 @@ def resample(y: array, src_sr: int, target_sr: int, ...@@ -53,7 +61,7 @@ def resample(y: array, src_sr: int, target_sr: int,
if not isinstance(y, np.ndarray): if not isinstance(y, np.ndarray):
raise ParameterError( raise ParameterError(
'Only support numpy array, but received y in {type(y)}') 'Only support numpy np.ndarray, but received y in {type(y)}')
if mode not in RESAMPLE_MODES: if mode not in RESAMPLE_MODES:
raise ParameterError(f'resample mode must in {RESAMPLE_MODES}') raise ParameterError(f'resample mode must in {RESAMPLE_MODES}')
...@@ -61,9 +69,17 @@ def resample(y: array, src_sr: int, target_sr: int, ...@@ -61,9 +69,17 @@ def resample(y: array, src_sr: int, target_sr: int,
return resampy.resample(y, src_sr, target_sr, filter=mode) return resampy.resample(y, src_sr, target_sr, filter=mode)
def to_mono(y: array, merge_type: str='average') -> array: def to_mono(y: np.ndarray, merge_type: str='average') -> np.ndarray:
""" convert sterior audio to mono """Convert sterior audio to mono.
Args:
y (np.ndarray): Input waveform array in 1D or 2D.
merge_type (str, optional): Merge type to generate mono waveform. Defaults to 'average'.
Returns:
np.ndarray: `y` with mono channel.
""" """
if merge_type not in MERGE_TYPES: if merge_type not in MERGE_TYPES:
raise ParameterError( raise ParameterError(
f'Unsupported merge type {merge_type}, available types are {MERGE_TYPES}' f'Unsupported merge type {merge_type}, available types are {MERGE_TYPES}'
...@@ -101,18 +117,34 @@ def to_mono(y: array, merge_type: str='average') -> array: ...@@ -101,18 +117,34 @@ def to_mono(y: array, merge_type: str='average') -> array:
return y_out return y_out
def _safe_cast(y: array, dtype: Union[type, str]) -> array: def _safe_cast(y: np.ndarray, dtype: Union[type, str]) -> np.ndarray:
""" data type casting in a safe way, i.e., prevent overflow or underflow """Data type casting in a safe way, i.e., prevent overflow or underflow.
This function is used internally.
Args:
y (np.ndarray): Input waveform array in 1D or 2D.
dtype (Union[type, str]): Data type of waveform.
Returns:
np.ndarray: `y` after safe casting.
""" """
return np.clip(y, np.iinfo(dtype).min, np.iinfo(dtype).max).astype(dtype) if 'float' in str(y.dtype):
return np.clip(y, np.finfo(dtype).min,
np.finfo(dtype).max).astype(dtype)
else:
return np.clip(y, np.iinfo(dtype).min,
np.iinfo(dtype).max).astype(dtype)
def depth_convert(y: array, dtype: Union[type, str], def depth_convert(y: np.ndarray, dtype: Union[type, str]) -> np.ndarray:
dithering: bool=True) -> array: """Convert audio array to target dtype safely. This function convert audio waveform to a target dtype, with addition steps of
"""Convert audio array to target dtype safely
This function convert audio waveform to a target dtype, with addition steps of
preventing overflow/underflow and preserving audio range. preventing overflow/underflow and preserving audio range.
Args:
y (np.ndarray): Input waveform array in 1D or 2D.
dtype (Union[type, str]): Data type of waveform.
Returns:
np.ndarray: `y` after safe casting.
""" """
SUPPORT_DTYPE = ['int16', 'int8', 'float32', 'float64'] SUPPORT_DTYPE = ['int16', 'int8', 'float32', 'float64']
...@@ -157,14 +189,20 @@ def depth_convert(y: array, dtype: Union[type, str], ...@@ -157,14 +189,20 @@ def depth_convert(y: array, dtype: Union[type, str],
return y return y
def sound_file_load(file: str, def sound_file_load(file: os.PathLike,
offset: Optional[float]=None, offset: Optional[float]=None,
dtype: str='int16', dtype: str='int16',
duration: Optional[int]=None) -> Tuple[array, int]: duration: Optional[int]=None) -> Tuple[np.ndarray, int]:
"""Load audio using soundfile library """Load audio using soundfile library. This function load audio file using libsndfile.
This function load audio file using libsndfile.
Reference: Args:
http://www.mega-nerd.com/libsndfile/#Features file (os.PathLike): File of waveform.
offset (Optional[float], optional): Offset to the start of waveform. Defaults to None.
dtype (str, optional): Data type of waveform. Defaults to 'int16'.
duration (Optional[int], optional): Duration of waveform to read. Defaults to None.
Returns:
Tuple[np.ndarray, int]: Waveform in ndarray and its samplerate.
""" """
with sf.SoundFile(file) as sf_desc: with sf.SoundFile(file) as sf_desc:
sr_native = sf_desc.samplerate sr_native = sf_desc.samplerate
...@@ -179,9 +217,17 @@ def sound_file_load(file: str, ...@@ -179,9 +217,17 @@ def sound_file_load(file: str,
return y, sf_desc.samplerate return y, sf_desc.samplerate
def normalize(y: array, norm_type: str='linear', def normalize(y: np.ndarray, norm_type: str='linear',
mul_factor: float=1.0) -> array: mul_factor: float=1.0) -> np.ndarray:
""" normalize an input audio with additional multiplier. """Normalize an input audio with additional multiplier.
Args:
y (np.ndarray): Input waveform array in 1D or 2D.
norm_type (str, optional): Type of normalization. Defaults to 'linear'.
mul_factor (float, optional): Scaling factor. Defaults to 1.0.
Returns:
np.ndarray: `y` after normalization.
""" """
if norm_type == 'linear': if norm_type == 'linear':
...@@ -199,12 +245,13 @@ def normalize(y: array, norm_type: str='linear', ...@@ -199,12 +245,13 @@ def normalize(y: array, norm_type: str='linear',
return y return y
def save(y: array, sr: int, file: str) -> None: def save(y: np.ndarray, sr: int, file: os.PathLike) -> None:
"""Save audio file to disk. """Save audio file to disk. This function saves audio to disk using scipy.io.wavfile, with additional step to convert input waveform to int16.
This function saves audio to disk using scipy.io.wavfile, with additional step
to convert input waveform to int16 unless it already is int16 Args:
Notes: y (np.ndarray): Input waveform array in 1D or 2D.
It only support raw wav format. sr (int): Sample rate.
file (os.PathLike): Path of auido file to save.
""" """
if not file.endswith('.wav'): if not file.endswith('.wav'):
raise ParameterError( raise ParameterError(
...@@ -226,7 +273,7 @@ def save(y: array, sr: int, file: str) -> None: ...@@ -226,7 +273,7 @@ def save(y: array, sr: int, file: str) -> None:
def load( def load(
file: str, file: os.PathLike,
sr: Optional[int]=None, sr: Optional[int]=None,
mono: bool=True, mono: bool=True,
merge_type: str='average', # ch0,ch1,random,average merge_type: str='average', # ch0,ch1,random,average
...@@ -236,11 +283,24 @@ def load( ...@@ -236,11 +283,24 @@ def load(
offset: float=0.0, offset: float=0.0,
duration: Optional[int]=None, duration: Optional[int]=None,
dtype: str='float32', dtype: str='float32',
resample_mode: str='kaiser_fast') -> Tuple[array, int]: resample_mode: str='kaiser_fast') -> Tuple[np.ndarray, int]:
"""Load audio file from disk. """Load audio file from disk. This function loads audio from disk using using audio beackend.
This function loads audio from disk using using audio beackend.
Parameters: Args:
Notes: file (os.PathLike): Path of auido file to load.
sr (Optional[int], optional): Sample rate of loaded waveform. Defaults to None.
mono (bool, optional): Return waveform with mono channel. Defaults to True.
merge_type (str, optional): Merge type of multi-channels waveform. Defaults to 'average'.
normal (bool, optional): Waveform normalization. Defaults to True.
norm_type (str, optional): Type of normalization. Defaults to 'linear'.
norm_mul_factor (float, optional): Scaling factor. Defaults to 1.0.
offset (float, optional): Offset to the start of waveform. Defaults to 0.0.
duration (Optional[int], optional): Duration of waveform to read. Defaults to None.
dtype (str, optional): Data type of waveform. Defaults to 'float32'.
resample_mode (str, optional): The resampling filter to use. Defaults to 'kaiser_fast'.
Returns:
Tuple[np.ndarray, int]: Waveform in ndarray and its samplerate.
""" """
y, r = sound_file_load(file, offset=offset, dtype=dtype, duration=duration) y, r = sound_file_load(file, offset=offset, dtype=dtype, duration=duration)
......
paddleaudio/paddleaudio/compliance/kaldi.py
浏览文件 @ 490300f8
...@@ -220,7 +220,7 @@ def spectrogram(waveform: Tensor, ...@@ -220,7 +220,7 @@ def spectrogram(waveform: Tensor,
"""Compute and return a spectrogram from a waveform. The output is identical to Kaldi's. """Compute and return a spectrogram from a waveform. The output is identical to Kaldi's.
Args: Args:
waveform (Tensor): A waveform tensor with shape [C, T]. waveform (Tensor): A waveform tensor with shape `(C, T)`.
blackman_coeff (float, optional): Coefficient for Blackman window.. Defaults to 0.42. blackman_coeff (float, optional): Coefficient for Blackman window.. Defaults to 0.42.
channel (int, optional): Select the channel of waveform. Defaults to -1. channel (int, optional): Select the channel of waveform. Defaults to -1.
dither (float, optional): Dithering constant . Defaults to 0.0. dither (float, optional): Dithering constant . Defaults to 0.0.
...@@ -239,7 +239,7 @@ def spectrogram(waveform: Tensor, ...@@ -239,7 +239,7 @@ def spectrogram(waveform: Tensor,
window_type (str, optional): Choose type of window for FFT computation. Defaults to POVEY. window_type (str, optional): Choose type of window for FFT computation. Defaults to POVEY.
Returns: Returns:
Tensor: A spectrogram tensor with shape (m, padded_window_size // 2 + 1) where m is the number of frames Tensor: A spectrogram tensor with shape `(m, padded_window_size // 2 + 1)` where m is the number of frames
depends on frame_length and frame_shift. depends on frame_length and frame_shift.
""" """
dtype = waveform.dtype dtype = waveform.dtype
...@@ -422,7 +422,7 @@ def fbank(waveform: Tensor, ...@@ -422,7 +422,7 @@ def fbank(waveform: Tensor,
"""Compute and return filter banks from a waveform. The output is identical to Kaldi's. """Compute and return filter banks from a waveform. The output is identical to Kaldi's.
Args: Args:
waveform (Tensor): A waveform tensor with shape [C, T]. waveform (Tensor): A waveform tensor with shape `(C, T)`.
blackman_coeff (float, optional): Coefficient for Blackman window.. Defaults to 0.42. blackman_coeff (float, optional): Coefficient for Blackman window.. Defaults to 0.42.
channel (int, optional): Select the channel of waveform. Defaults to -1. channel (int, optional): Select the channel of waveform. Defaults to -1.
dither (float, optional): Dithering constant . Defaults to 0.0. dither (float, optional): Dithering constant . Defaults to 0.0.
...@@ -451,7 +451,7 @@ def fbank(waveform: Tensor, ...@@ -451,7 +451,7 @@ def fbank(waveform: Tensor,
window_type (str, optional): Choose type of window for FFT computation. Defaults to POVEY. window_type (str, optional): Choose type of window for FFT computation. Defaults to POVEY.
Returns: Returns:
Tensor: A filter banks tensor with shape (m, n_mels). Tensor: A filter banks tensor with shape `(m, n_mels)`.
""" """
dtype = waveform.dtype dtype = waveform.dtype
...@@ -542,7 +542,7 @@ def mfcc(waveform: Tensor, ...@@ -542,7 +542,7 @@ def mfcc(waveform: Tensor,
identical to Kaldi's. identical to Kaldi's.
Args: Args:
waveform (Tensor): A waveform tensor with shape [C, T]. waveform (Tensor): A waveform tensor with shape `(C, T)`.
blackman_coeff (float, optional): Coefficient for Blackman window.. Defaults to 0.42. blackman_coeff (float, optional): Coefficient for Blackman window.. Defaults to 0.42.
cepstral_lifter (float, optional): Scaling of output mfccs. Defaults to 22.0. cepstral_lifter (float, optional): Scaling of output mfccs. Defaults to 22.0.
channel (int, optional): Select the channel of waveform. Defaults to -1. channel (int, optional): Select the channel of waveform. Defaults to -1.
...@@ -571,7 +571,7 @@ def mfcc(waveform: Tensor, ...@@ -571,7 +571,7 @@ def mfcc(waveform: Tensor,
window_type (str, optional): Choose type of window for FFT computation. Defaults to POVEY. window_type (str, optional): Choose type of window for FFT computation. Defaults to POVEY.
Returns: Returns:
Tensor: A mel frequency cepstral coefficients tensor with shape (m, n_mfcc). Tensor: A mel frequency cepstral coefficients tensor with shape `(m, n_mfcc)`.
""" """
assert n_mfcc <= n_mels, 'n_mfcc cannot be larger than n_mels: %d vs %d' % ( assert n_mfcc <= n_mels, 'n_mfcc cannot be larger than n_mels: %d vs %d' % (
n_mfcc, n_mels) n_mfcc, n_mels)
......
paddleaudio/paddleaudio/functional/functional.py
浏览文件 @ 490300f8
...@@ -17,6 +17,7 @@ from typing import Optional ...@@ -17,6 +17,7 @@ from typing import Optional
from typing import Union from typing import Union
import paddle import paddle
from paddle import Tensor
__all__ = [ __all__ = [
'hz_to_mel', 'hz_to_mel',
...@@ -29,19 +30,20 @@ __all__ = [ ...@@ -29,19 +30,20 @@ __all__ = [
] ]
def hz_to_mel(freq: Union[paddle.Tensor, float], def hz_to_mel(freq: Union[Tensor, float],
htk: bool=False) -> Union[paddle.Tensor, float]: htk: bool=False) -> Union[Tensor, float]:
"""Convert Hz to Mels. """Convert Hz to Mels.
Parameters:
freq: the input tensor of arbitrary shape, or a single floating point number. Args:
htk: use HTK formula to do the conversion. freq (Union[Tensor, float]): The input tensor with arbitrary shape.
The default value is False. htk (bool, optional): Use htk scaling. Defaults to False.
Returns: Returns:
The frequencies represented in Mel-scale. Union[Tensor, float]: Frequency in mels.
""" """
if htk: if htk:
if isinstance(freq, paddle.Tensor): if isinstance(freq, Tensor):
return 2595.0 * paddle.log10(1.0 + freq / 700.0) return 2595.0 * paddle.log10(1.0 + freq / 700.0)
else: else:
return 2595.0 * math.log10(1.0 + freq / 700.0) return 2595.0 * math.log10(1.0 + freq / 700.0)
...@@ -58,7 +60,7 @@ def hz_to_mel(freq: Union[paddle.Tensor, float], ...@@ -58,7 +60,7 @@ def hz_to_mel(freq: Union[paddle.Tensor, float],
min_log_mel = (min_log_hz - f_min) / f_sp # same (Mels) min_log_mel = (min_log_hz - f_min) / f_sp # same (Mels)
logstep = math.log(6.4) / 27.0 # step size for log region logstep = math.log(6.4) / 27.0 # step size for log region
if isinstance(freq, paddle.Tensor): if isinstance(freq, Tensor):
target = min_log_mel + paddle.log( target = min_log_mel + paddle.log(
freq / min_log_hz + 1e-10) / logstep # prevent nan with 1e-10 freq / min_log_hz + 1e-10) / logstep # prevent nan with 1e-10
mask = (freq > min_log_hz).astype(freq.dtype) mask = (freq > min_log_hz).astype(freq.dtype)
...@@ -71,14 +73,16 @@ def hz_to_mel(freq: Union[paddle.Tensor, float], ...@@ -71,14 +73,16 @@ def hz_to_mel(freq: Union[paddle.Tensor, float],
return mels return mels
def mel_to_hz(mel: Union[float, paddle.Tensor], def mel_to_hz(mel: Union[float, Tensor],
htk: bool=False) -> Union[float, paddle.Tensor]: htk: bool=False) -> Union[float, Tensor]:
"""Convert mel bin numbers to frequencies. """Convert mel bin numbers to frequencies.
Parameters:
mel: the mel frequency represented as a tensor of arbitrary shape, or a floating point number. Args:
htk: use HTK formula to do the conversion. mel (Union[float, Tensor]): The mel frequency represented as a tensor with arbitrary shape.
htk (bool, optional): Use htk scaling. Defaults to False.
Returns: Returns:
The frequencies represented in hz. Union[float, Tensor]: Frequencies in Hz.
""" """
if htk: if htk:
return 700.0 * (10.0**(mel / 2595.0) - 1.0) return 700.0 * (10.0**(mel / 2595.0) - 1.0)
...@@ -90,7 +94,7 @@ def mel_to_hz(mel: Union[float, paddle.Tensor], ...@@ -90,7 +94,7 @@ def mel_to_hz(mel: Union[float, paddle.Tensor],
min_log_hz = 1000.0 # beginning of log region (Hz) min_log_hz = 1000.0 # beginning of log region (Hz)
min_log_mel = (min_log_hz - f_min) / f_sp # same (Mels) min_log_mel = (min_log_hz - f_min) / f_sp # same (Mels)
logstep = math.log(6.4) / 27.0 # step size for log region logstep = math.log(6.4) / 27.0 # step size for log region
if isinstance(mel, paddle.Tensor): if isinstance(mel, Tensor):
target = min_log_hz * paddle.exp(logstep * (mel - min_log_mel)) target = min_log_hz * paddle.exp(logstep * (mel - min_log_mel))
mask = (mel > min_log_mel).astype(mel.dtype) mask = (mel > min_log_mel).astype(mel.dtype)
freqs = target * mask + freqs * ( freqs = target * mask + freqs * (
...@@ -106,16 +110,18 @@ def mel_frequencies(n_mels: int=64, ...@@ -106,16 +110,18 @@ def mel_frequencies(n_mels: int=64,
f_min: float=0.0, f_min: float=0.0,
f_max: float=11025.0, f_max: float=11025.0,
htk: bool=False, htk: bool=False,
dtype: str=paddle.float32): dtype: str='float32') -> Tensor:
"""Compute mel frequencies. """Compute mel frequencies.
Parameters:
n_mels(int): number of Mel bins. Args:
f_min(float): the lower cut-off frequency, below which the filter response is zero. n_mels (int, optional): Number of mel bins. Defaults to 64.
f_max(float): the upper cut-off frequency, above which the filter response is zero. f_min (float, optional): Minimum frequency in Hz. Defaults to 0.0.
htk(bool): whether to use htk formula. fmax (float, optional): Maximum frequency in Hz. Defaults to 11025.0.
dtype(str): the datatype of the return frequencies. htk (bool, optional): Use htk scaling. Defaults to False.
dtype (str, optional): The data type of the return frequencies. Defaults to 'float32'.
Returns: Returns:
The frequencies represented in Mel-scale Tensor: Tensor of n_mels frequencies in Hz with shape `(n_mels,)`.
""" """
# 'Center freqs' of mel bands - uniformly spaced between limits # 'Center freqs' of mel bands - uniformly spaced between limits
min_mel = hz_to_mel(f_min, htk=htk) min_mel = hz_to_mel(f_min, htk=htk)
...@@ -125,14 +131,16 @@ def mel_frequencies(n_mels: int=64, ...@@ -125,14 +131,16 @@ def mel_frequencies(n_mels: int=64,
return freqs return freqs
def fft_frequencies(sr: int, n_fft: int, dtype: str=paddle.float32): def fft_frequencies(sr: int, n_fft: int, dtype: str='float32') -> Tensor:
"""Compute fourier frequencies. """Compute fourier frequencies.
Parameters:
sr(int): the audio sample rate. Args:
n_fft(float): the number of fft bins. sr (int): Sample rate.
dtype(str): the datatype of the return frequencies. n_fft (int): Number of fft bins.
dtype (str, optional): The data type of the return frequencies. Defaults to 'float32'.
Returns: Returns:
The frequencies represented in hz. Tensor: FFT frequencies in Hz with shape `(n_fft//2 + 1,)`.
""" """
return paddle.linspace(0, float(sr) / 2, int(1 + n_fft // 2), dtype=dtype) return paddle.linspace(0, float(sr) / 2, int(1 + n_fft // 2), dtype=dtype)
...@@ -144,23 +152,21 @@ def compute_fbank_matrix(sr: int, ...@@ -144,23 +152,21 @@ def compute_fbank_matrix(sr: int,
f_max: Optional[float]=None, f_max: Optional[float]=None,
htk: bool=False, htk: bool=False,
norm: Union[str, float]='slaney', norm: Union[str, float]='slaney',
dtype: str=paddle.float32): dtype: str='float32') -> Tensor:
"""Compute fbank matrix. """Compute fbank matrix.
Parameters:
sr(int): the audio sample rate. Args:
n_fft(int): the number of fft bins. sr (int): Sample rate.
n_mels(int): the number of Mel bins. n_fft (int): Number of fft bins.
f_min(float): the lower cut-off frequency, below which the filter response is zero. n_mels (int, optional): Number of mel bins. Defaults to 64.
f_max(float): the upper cut-off frequency, above which the filter response is zero. f_min (float, optional): Minimum frequency in Hz. Defaults to 0.0.
htk: whether to use htk formula. f_max (Optional[float], optional): Maximum frequency in Hz. Defaults to None.
return_complex(bool): whether to return complex matrix. If True, the matrix will htk (bool, optional): Use htk scaling. Defaults to False.
be complex type. Otherwise, the real and image part will be stored in the last norm (Union[str, float], optional): Type of normalization. Defaults to 'slaney'.
axis of returned tensor. dtype (str, optional): The data type of the return matrix. Defaults to 'float32'.
dtype(str): the datatype of the returned fbank matrix.
Returns: Returns:
The fbank matrix of shape (n_mels, int(1+n_fft//2)). Tensor: Mel transform matrix with shape `(n_mels, n_fft//2 + 1)`.
Shape:
output: (n_mels, int(1+n_fft//2))
""" """
if f_max is None: if f_max is None:
...@@ -199,27 +205,20 @@ def compute_fbank_matrix(sr: int, ...@@ -199,27 +205,20 @@ def compute_fbank_matrix(sr: int,
return weights return weights
def power_to_db(magnitude: paddle.Tensor, def power_to_db(spect: Tensor,
ref_value: float=1.0, ref_value: float=1.0,
amin: float=1e-10, amin: float=1e-10,
top_db: Optional[float]=None) -> paddle.Tensor: top_db: Optional[float]=None) -> Tensor:
"""Convert a power spectrogram (amplitude squared) to decibel (dB) units. """Convert a power spectrogram (amplitude squared) to decibel (dB) units. The function computes the scaling `10 * log10(x / ref)` in a numerically stable way.
The function computes the scaling ``10 * log10(x / ref)`` in a numerically
stable way. Args:
Parameters: spect (Tensor): STFT power spectrogram.
magnitude(Tensor): the input magnitude tensor of any shape. ref_value (float, optional): The reference value. If smaller than 1.0, the db level of the signal will be pulled up accordingly. Otherwise, the db level is pushed down. Defaults to 1.0.
ref_value(float): the reference value. If smaller than 1.0, the db level amin (float, optional): Minimum threshold. Defaults to 1e-10.
of the signal will be pulled up accordingly. Otherwise, the db level top_db (Optional[float], optional): Threshold the output at `top_db` below the peak. Defaults to None.
is pushed down.
amin(float): the minimum value of input magnitude, below which the input
magnitude is clipped(to amin).
top_db(float): the maximum db value of resulting spectrum, above which the
spectrum is clipped(to top_db).
Returns: Returns:
The spectrogram in log-scale. Tensor: Power spectrogram in db scale.
shape:
input: any shape
output: same as input
""" """
if amin <= 0: if amin <= 0:
raise Exception("amin must be strictly positive") raise Exception("amin must be strictly positive")
...@@ -227,8 +226,8 @@ def power_to_db(magnitude: paddle.Tensor, ...@@ -227,8 +226,8 @@ def power_to_db(magnitude: paddle.Tensor,
if ref_value <= 0: if ref_value <= 0:
raise Exception("ref_value must be strictly positive") raise Exception("ref_value must be strictly positive")
ones = paddle.ones_like(magnitude) ones = paddle.ones_like(spect)
log_spec = 10.0 * paddle.log10(paddle.maximum(ones * amin, magnitude)) log_spec = 10.0 * paddle.log10(paddle.maximum(ones * amin, spect))
log_spec -= 10.0 * math.log10(max(ref_value, amin)) log_spec -= 10.0 * math.log10(max(ref_value, amin))
if top_db is not None: if top_db is not None:
...@@ -242,15 +241,17 @@ def power_to_db(magnitude: paddle.Tensor, ...@@ -242,15 +241,17 @@ def power_to_db(magnitude: paddle.Tensor,
def create_dct(n_mfcc: int, def create_dct(n_mfcc: int,
n_mels: int, n_mels: int,
norm: Optional[str]='ortho', norm: Optional[str]='ortho',
dtype: Optional[str]=paddle.float32) -> paddle.Tensor: dtype: str='float32') -> Tensor:
"""Create a discrete cosine transform(DCT) matrix. """Create a discrete cosine transform(DCT) matrix.
Parameters: Args:
n_mfcc (int): Number of mel frequency cepstral coefficients. n_mfcc (int): Number of mel frequency cepstral coefficients.
n_mels (int): Number of mel filterbanks. n_mels (int): Number of mel filterbanks.
norm (str, optional): Normalizaiton type. Defaults to 'ortho'. norm (Optional[str], optional): Normalizaiton type. Defaults to 'ortho'.
dtype (str, optional): The data type of the return matrix. Defaults to 'float32'.
Returns: Returns:
Tensor: The DCT matrix with shape (n_mels, n_mfcc). Tensor: The DCT matrix with shape `(n_mels, n_mfcc)`.
""" """
n = paddle.arange(n_mels, dtype=dtype) n = paddle.arange(n_mels, dtype=dtype)
k = paddle.arange(n_mfcc, dtype=dtype).unsqueeze(1) k = paddle.arange(n_mfcc, dtype=dtype).unsqueeze(1)
......
paddleaudio/paddleaudio/functional/window.py
浏览文件 @ 490300f8
...@@ -20,24 +20,11 @@ from paddle import Tensor ...@@ -20,24 +20,11 @@ from paddle import Tensor
__all__ = [ __all__ = [
'get_window', 'get_window',
# windows
'taylor',
'hamming',
'hann',
'tukey',
'kaiser',
'gaussian',
'exponential',
'triang',
'bohman',
'blackman',
'cosine',
] ]
def _cat(a: List[Tensor], data_type: str) -> Tensor: def _cat(x: List[Tensor], data_type: str) -> Tensor:
l = [paddle.to_tensor(_a, data_type) for _a in a] l = [paddle.to_tensor(_, data_type) for _ in x]
return paddle.concat(l) return paddle.concat(l)
...@@ -48,7 +35,7 @@ def _acosh(x: Union[Tensor, float]) -> Tensor: ...@@ -48,7 +35,7 @@ def _acosh(x: Union[Tensor, float]) -> Tensor:
def _extend(M: int, sym: bool) -> bool: def _extend(M: int, sym: bool) -> bool:
"""Extend window by 1 sample if needed for DFT-even symmetry""" """Extend window by 1 sample if needed for DFT-even symmetry. """
if not sym: if not sym:
return M + 1, True return M + 1, True
else: else:
...@@ -56,7 +43,7 @@ def _extend(M: int, sym: bool) -> bool: ...@@ -56,7 +43,7 @@ def _extend(M: int, sym: bool) -> bool:
def _len_guards(M: int) -> bool: def _len_guards(M: int) -> bool:
"""Handle small or incorrect window lengths""" """Handle small or incorrect window lengths. """
if int(M) != M or M < 0: if int(M) != M or M < 0:
raise ValueError('Window length M must be a non-negative integer') raise ValueError('Window length M must be a non-negative integer')
...@@ -64,15 +51,15 @@ def _len_guards(M: int) -> bool: ...@@ -64,15 +51,15 @@ def _len_guards(M: int) -> bool:
def _truncate(w: Tensor, needed: bool) -> Tensor: def _truncate(w: Tensor, needed: bool) -> Tensor:
"""Truncate window by 1 sample if needed for DFT-even symmetry""" """Truncate window by 1 sample if needed for DFT-even symmetry. """
if needed: if needed:
return w[:-1] return w[:-1]
else: else:
return w return w
def general_gaussian(M: int, p, sig, sym: bool=True, def _general_gaussian(M: int, p, sig, sym: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Compute a window with a generalized Gaussian shape. """Compute a window with a generalized Gaussian shape.
This function is consistent with scipy.signal.windows.general_gaussian(). This function is consistent with scipy.signal.windows.general_gaussian().
""" """
...@@ -86,8 +73,8 @@ def general_gaussian(M: int, p, sig, sym: bool=True, ...@@ -86,8 +73,8 @@ def general_gaussian(M: int, p, sig, sym: bool=True,
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def general_cosine(M: int, a: float, sym: bool=True, def _general_cosine(M: int, a: float, sym: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Compute a generic weighted sum of cosine terms window. """Compute a generic weighted sum of cosine terms window.
This function is consistent with scipy.signal.windows.general_cosine(). This function is consistent with scipy.signal.windows.general_cosine().
""" """
...@@ -101,31 +88,23 @@ def general_cosine(M: int, a: float, sym: bool=True, ...@@ -101,31 +88,23 @@ def general_cosine(M: int, a: float, sym: bool=True,
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def general_hamming(M: int, alpha: float, sym: bool=True, def _general_hamming(M: int, alpha: float, sym: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Compute a generalized Hamming window. """Compute a generalized Hamming window.
This function is consistent with scipy.signal.windows.general_hamming() This function is consistent with scipy.signal.windows.general_hamming()
""" """
return general_cosine(M, [alpha, 1. - alpha], sym, dtype=dtype) return _general_cosine(M, [alpha, 1. - alpha], sym, dtype=dtype)
def taylor(M: int, def _taylor(M: int,
nbar=4, nbar=4,
sll=30, sll=30,
norm=True, norm=True,
sym: bool=True, sym: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Compute a Taylor window. """Compute a Taylor window.
The Taylor window taper function approximates the Dolph-Chebyshev window's The Taylor window taper function approximates the Dolph-Chebyshev window's
constant sidelobe level for a parameterized number of near-in sidelobes. constant sidelobe level for a parameterized number of near-in sidelobes.
Parameters:
M(int): window size
nbar, sil, norm: the window-specific parameter.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
if _len_guards(M): if _len_guards(M):
return paddle.ones((M, ), dtype=dtype) return paddle.ones((M, ), dtype=dtype)
...@@ -171,46 +150,25 @@ def taylor(M: int, ...@@ -171,46 +150,25 @@ def taylor(M: int,
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def hamming(M: int, sym: bool=True, dtype: str='float64') -> Tensor: def _hamming(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a Hamming window. """Compute a Hamming window.
The Hamming window is a taper formed by using a raised cosine with The Hamming window is a taper formed by using a raised cosine with
non-zero endpoints, optimized to minimize the nearest side lobe. non-zero endpoints, optimized to minimize the nearest side lobe.
Parameters:
M(int): window size
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
return general_hamming(M, 0.54, sym, dtype=dtype) return _general_hamming(M, 0.54, sym, dtype=dtype)
def hann(M: int, sym: bool=True, dtype: str='float64') -> Tensor: def _hann(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a Hann window. """Compute a Hann window.
The Hann window is a taper formed by using a raised cosine or sine-squared The Hann window is a taper formed by using a raised cosine or sine-squared
with ends that touch zero. with ends that touch zero.
Parameters:
M(int): window size
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
return general_hamming(M, 0.5, sym, dtype=dtype) return _general_hamming(M, 0.5, sym, dtype=dtype)
def tukey(M: int, alpha=0.5, sym: bool=True, dtype: str='float64') -> Tensor: def _tukey(M: int, alpha=0.5, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a Tukey window. """Compute a Tukey window.
The Tukey window is also known as a tapered cosine window. The Tukey window is also known as a tapered cosine window.
Parameters:
M(int): window size
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
if _len_guards(M): if _len_guards(M):
return paddle.ones((M, ), dtype=dtype) return paddle.ones((M, ), dtype=dtype)
...@@ -237,32 +195,18 @@ def tukey(M: int, alpha=0.5, sym: bool=True, dtype: str='float64') -> Tensor: ...@@ -237,32 +195,18 @@ def tukey(M: int, alpha=0.5, sym: bool=True, dtype: str='float64') -> Tensor:
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def kaiser(M: int, beta: float, sym: bool=True, dtype: str='float64') -> Tensor: def _kaiser(M: int, beta: float, sym: bool=True,
dtype: str='float64') -> Tensor:
"""Compute a Kaiser window. """Compute a Kaiser window.
The Kaiser window is a taper formed by using a Bessel function. The Kaiser window is a taper formed by using a Bessel function.
Parameters:
M(int): window size.
beta(float): the window-specific parameter.
sym(bool)：whether to return symmetric window.
The default value is True
Returns:
Tensor: the window tensor
""" """
raise NotImplementedError() raise NotImplementedError()
def gaussian(M: int, std: float, sym: bool=True, def _gaussian(M: int, std: float, sym: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Compute a Gaussian window. """Compute a Gaussian window.
The Gaussian widows has a Gaussian shape defined by the standard deviation(std). The Gaussian widows has a Gaussian shape defined by the standard deviation(std).
Parameters:
M(int): window size.
std(float): the window-specific parameter.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
if _len_guards(M): if _len_guards(M):
return paddle.ones((M, ), dtype=dtype) return paddle.ones((M, ), dtype=dtype)
...@@ -275,21 +219,12 @@ def gaussian(M: int, std: float, sym: bool=True, ...@@ -275,21 +219,12 @@ def gaussian(M: int, std: float, sym: bool=True,
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def exponential(M: int, def _exponential(M: int,
center=None, center=None,
tau=1., tau=1.,
sym: bool=True, sym: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Compute an exponential (or Poisson) window. """Compute an exponential (or Poisson) window. """
Parameters:
M(int): window size.
tau(float): the window-specific parameter.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
"""
if sym and center is not None: if sym and center is not None:
raise ValueError("If sym==True, center must be None.") raise ValueError("If sym==True, center must be None.")
if _len_guards(M): if _len_guards(M):
...@@ -305,15 +240,8 @@ def exponential(M: int, ...@@ -305,15 +240,8 @@ def exponential(M: int,
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def triang(M: int, sym: bool=True, dtype: str='float64') -> Tensor: def _triang(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a triangular window. """Compute a triangular window.
Parameters:
M(int): window size.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
if _len_guards(M): if _len_guards(M):
return paddle.ones((M, ), dtype=dtype) return paddle.ones((M, ), dtype=dtype)
...@@ -330,16 +258,9 @@ def triang(M: int, sym: bool=True, dtype: str='float64') -> Tensor: ...@@ -330,16 +258,9 @@ def triang(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def bohman(M: int, sym: bool=True, dtype: str='float64') -> Tensor: def _bohman(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a Bohman window. """Compute a Bohman window.
The Bohman window is the autocorrelation of a cosine window. The Bohman window is the autocorrelation of a cosine window.
Parameters:
M(int): window size.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
if _len_guards(M): if _len_guards(M):
return paddle.ones((M, ), dtype=dtype) return paddle.ones((M, ), dtype=dtype)
...@@ -353,32 +274,18 @@ def bohman(M: int, sym: bool=True, dtype: str='float64') -> Tensor: ...@@ -353,32 +274,18 @@ def bohman(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
def blackman(M: int, sym: bool=True, dtype: str='float64') -> Tensor: def _blackman(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a Blackman window. """Compute a Blackman window.
The Blackman window is a taper formed by using the first three terms of The Blackman window is a taper formed by using the first three terms of
a summation of cosines. It was designed to have close to the minimal a summation of cosines. It was designed to have close to the minimal
leakage possible. It is close to optimal, only slightly worse than a leakage possible. It is close to optimal, only slightly worse than a
Kaiser window. Kaiser window.
Parameters:
M(int): window size.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
return general_cosine(M, [0.42, 0.50, 0.08], sym, dtype=dtype) return _general_cosine(M, [0.42, 0.50, 0.08], sym, dtype=dtype)
def cosine(M: int, sym: bool=True, dtype: str='float64') -> Tensor: def _cosine(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
"""Compute a window with a simple cosine shape. """Compute a window with a simple cosine shape.
Parameters:
M(int): window size.
sym(bool)：whether to return symmetric window.
The default value is True
dtype(str): the datatype of returned tensor.
Returns:
Tensor: the window tensor
""" """
if _len_guards(M): if _len_guards(M):
return paddle.ones((M, ), dtype=dtype) return paddle.ones((M, ), dtype=dtype)
...@@ -388,19 +295,20 @@ def cosine(M: int, sym: bool=True, dtype: str='float64') -> Tensor: ...@@ -388,19 +295,20 @@ def cosine(M: int, sym: bool=True, dtype: str='float64') -> Tensor:
return _truncate(w, needs_trunc) return _truncate(w, needs_trunc)
## factory function
def get_window(window: Union[str, Tuple[str, float]], def get_window(window: Union[str, Tuple[str, float]],
win_length: int, win_length: int,
fftbins: bool=True, fftbins: bool=True,
dtype: str='float64') -> Tensor: dtype: str='float64') -> Tensor:
"""Return a window of a given length and type. """Return a window of a given length and type.
Parameters:
window(str|(str,float)): the type of window to create. Args:
win_length(int): the number of samples in the window. window (Union[str, Tuple[str, float]]): The window function applied to the signal before the Fourier transform. Supported window functions: 'hamming', 'hann', 'kaiser', 'gaussian', 'exponential', 'triang', 'bohman', 'blackman', 'cosine', 'tukey', 'taylor'.
fftbins(bool): If True, create a "periodic" window. Otherwise, win_length (int): Number of samples.
create a "symmetric" window, for use in filter design. fftbins (bool, optional): If True, create a "periodic" window. Otherwise, create a "symmetric" window, for use in filter design. Defaults to True.
dtype (str, optional): The data type of the return window. Defaults to 'float64'.
Returns: Returns:
The window represented as a tensor. Tensor: The window represented as a tensor.
""" """
sym = not fftbins sym = not fftbins
...@@ -420,7 +328,7 @@ def get_window(window: Union[str, Tuple[str, float]], ...@@ -420,7 +328,7 @@ def get_window(window: Union[str, Tuple[str, float]],
str(type(window))) str(type(window)))
try: try:
winfunc = eval(winstr) winfunc = eval('_' + winstr)
except KeyError as e: except KeyError as e:
raise ValueError("Unknown window type.") from e raise ValueError("Unknown window type.") from e
......
paddleaudio/paddleaudio/metric/dtw.py
浏览文件 @ 490300f8
...@@ -20,9 +20,7 @@ __all__ = [ ...@@ -20,9 +20,7 @@ __all__ = [
def dtw_distance(xs: np.ndarray, ys: np.ndarray) -> float: def dtw_distance(xs: np.ndarray, ys: np.ndarray) -> float:
"""dtw distance """Dynamic Time Warping.
Dynamic Time Warping.
This function keeps a compact matrix, not the full warping paths matrix. This function keeps a compact matrix, not the full warping paths matrix.
Uses dynamic programming to compute: Uses dynamic programming to compute:
......
paddleaudio/setup.py
浏览文件 @ 490300f8
...@@ -11,19 +11,46 @@ ...@@ -11,19 +11,46 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import glob
import os
import setuptools import setuptools
from setuptools.command.install import install
from setuptools.command.test import test
# set the version here # set the version here
VERSION = '0.2.0' VERSION = '0.2.0'
# Inspired by the example at https://pytest.org/latest/goodpractises.html
class TestCommand(test):
def finalize_options(self):
test.finalize_options(self)
self.test_args = []
self.test_suite = True
def run(self):
self.run_benchmark()
super(TestCommand, self).run()
def run_tests(self):
# Run nose ensuring that argv simulates running nosetests directly
import nose
nose.run_exit(argv=['nosetests', '-w', 'tests'])
def run_benchmark(self):
for benchmark_item in glob.glob('tests/benchmark/*py'):
os.system(f'pytest {benchmark_item}')
class InstallCommand(install):
def run(self):
install.run(self)
def write_version_py(filename='paddleaudio/__init__.py'): def write_version_py(filename='paddleaudio/__init__.py'):
import paddleaudio
if hasattr(paddleaudio,
"__version__") and paddleaudio.__version__ == VERSION:
return
with open(filename, "a") as f: with open(filename, "a") as f:
f.write(f"\n__version__ = '{VERSION}'\n") f.write(f"__version__ = '{VERSION}'")
def remove_version_py(filename='paddleaudio/__init__.py'): def remove_version_py(filename='paddleaudio/__init__.py'):
...@@ -35,6 +62,7 @@ def remove_version_py(filename='paddleaudio/__init__.py'): ...@@ -35,6 +62,7 @@ def remove_version_py(filename='paddleaudio/__init__.py'):
f.write(line) f.write(line)
remove_version_py()
write_version_py() write_version_py()
setuptools.setup( setuptools.setup(
...@@ -61,6 +89,16 @@ setuptools.setup( ...@@ -61,6 +89,16 @@ setuptools.setup(
'colorlog', 'colorlog',
'dtaidistance >= 2.3.6', 'dtaidistance >= 2.3.6',
'mcd >= 0.4', 'mcd >= 0.4',
], ) ],
extras_require={
'test': [
'nose', 'librosa==0.8.1', 'soundfile==0.10.3.post1',
'torchaudio==0.10.2', 'pytest-benchmark'
],
},
cmdclass={
'install': InstallCommand,
'test': TestCommand,
}, )
remove_version_py() remove_version_py()
paddleaudio/tests/backends/__init__.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
paddleaudio/tests/backends/base.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import unittest
import urllib.request
mono_channel_wav = 'https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav'
multi_channels_wav = 'https://paddlespeech.bj.bcebos.com/PaddleAudio/cat.wav'
class BackendTest(unittest.TestCase):
def setUp(self):
self.initWavInput()
def initWavInput(self):
self.files = []
for url in [mono_channel_wav, multi_channels_wav]:
if not os.path.isfile(os.path.basename(url)):
urllib.request.urlretrieve(url, os.path.basename(url))
self.files.append(os.path.basename(url))
def initParmas(self):
raise NotImplementedError
paddleaudio/tests/backends/soundfile/__init__.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
paddleaudio/tests/backends/soundfile/test_io.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import filecmp
import os
import unittest
import numpy as np
import soundfile as sf
import paddleaudio
from ..base import BackendTest
class TestIO(BackendTest):
def test_load_mono_channel(self):
sf_data, sf_sr = sf.read(self.files[0])
pa_data, pa_sr = paddleaudio.load(
self.files[0], normal=False, dtype='float64')
self.assertEqual(sf_data.dtype, pa_data.dtype)
self.assertEqual(sf_sr, pa_sr)
np.testing.assert_array_almost_equal(sf_data, pa_data)
def test_load_multi_channels(self):
sf_data, sf_sr = sf.read(self.files[1])
sf_data = sf_data.T # Channel dim first
pa_data, pa_sr = paddleaudio.load(
self.files[1], mono=False, normal=False, dtype='float64')
self.assertEqual(sf_data.dtype, pa_data.dtype)
self.assertEqual(sf_sr, pa_sr)
np.testing.assert_array_almost_equal(sf_data, pa_data)
def test_save_mono_channel(self):
waveform, sr = np.random.randint(
low=-32768, high=32768, size=(48000), dtype=np.int16), 16000
sf_tmp_file = 'sf_tmp.wav'
pa_tmp_file = 'pa_tmp.wav'
sf.write(sf_tmp_file, waveform, sr)
paddleaudio.save(waveform, sr, pa_tmp_file)
self.assertTrue(filecmp.cmp(sf_tmp_file, pa_tmp_file))
for file in [sf_tmp_file, pa_tmp_file]:
os.remove(file)
def test_save_multi_channels(self):
waveform, sr = np.random.randint(
low=-32768, high=32768, size=(2, 48000), dtype=np.int16), 16000
sf_tmp_file = 'sf_tmp.wav'
pa_tmp_file = 'pa_tmp.wav'
sf.write(sf_tmp_file, waveform.T, sr)
paddleaudio.save(waveform.T, sr, pa_tmp_file)
self.assertTrue(filecmp.cmp(sf_tmp_file, pa_tmp_file))
for file in [sf_tmp_file, pa_tmp_file]:
os.remove(file)
if __name__ == '__main__':
unittest.main()
paddleaudio/tests/benchmark/README.md 0 → 100644
浏览文件 @ 490300f8
# 1. Prepare
First, install `pytest-benchmark` via pip.
```sh
pip install pytest-benchmark
```
# 2. Run
Run the specific script for profiling.
```sh
pytest melspectrogram.py
```
Result:
```sh
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.7.7, pytest-7.0.1, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /ssd3/chenxiaojie06/PaddleSpeech/DeepSpeech/paddleaudio
plugins: typeguard-2.12.1, benchmark-3.4.1, anyio-3.5.0
collected 4 items
melspectrogram.py .... [100%]
-------------------------------------------------------------------------------------------------- benchmark: 4 tests -------------------------------------------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_melspect_gpu_torchaudio 202.0765 (1.0) 360.6230 (1.0) 218.1168 (1.0) 16.3022 (1.0) 214.2871 (1.0) 21.8451 (1.0) 40;3 4,584.7001 (1.0) 286 1
test_melspect_gpu 657.8509 (3.26) 908.0470 (2.52) 724.2545 (3.32) 106.5771 (6.54) 669.9096 (3.13) 113.4719 (5.19) 1;0 1,380.7300 (0.30) 5 1
test_melspect_cpu_torchaudio 1,247.6053 (6.17) 2,892.5799 (8.02) 1,443.2853 (6.62) 345.3732 (21.19) 1,262.7263 (5.89) 221.6385 (10.15) 56;53 692.8637 (0.15) 399 1
test_melspect_cpu 20,326.2549 (100.59) 20,607.8682 (57.15) 20,473.4125 (93.86) 63.8654 (3.92) 20,467.0429 (95.51) 68.4294 (3.13) 8;1 48.8438 (0.01) 29 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
========================================================================== 4 passed in 21.12s ===========================================================================
```
paddleaudio/tests/benchmark/log_melspectrogram.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import urllib.request
import librosa
import numpy as np
import paddle
import torch
import torchaudio
import paddleaudio
wav_url = 'https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav'
if not os.path.isfile(os.path.basename(wav_url)):
urllib.request.urlretrieve(wav_url, os.path.basename(wav_url))
waveform, sr = paddleaudio.load(os.path.abspath(os.path.basename(wav_url)))
waveform_tensor = paddle.to_tensor(waveform).unsqueeze(0)
waveform_tensor_torch = torch.from_numpy(waveform).unsqueeze(0)
# Feature conf
mel_conf = {
'sr': sr,
'n_fft': 512,
'hop_length': 128,
'n_mels': 40,
}
mel_conf_torchaudio = {
'sample_rate': sr,
'n_fft': 512,
'hop_length': 128,
'n_mels': 40,
'norm': 'slaney',
'mel_scale': 'slaney',
}
def enable_cpu_device():
paddle.set_device('cpu')
def enable_gpu_device():
paddle.set_device('gpu')
log_mel_extractor = paddleaudio.features.LogMelSpectrogram(
**mel_conf, f_min=0.0, top_db=80.0, dtype=waveform_tensor.dtype)
def log_melspectrogram():
return log_mel_extractor(waveform_tensor).squeeze(0)
def test_log_melspect_cpu(benchmark):
enable_cpu_device()
feature_paddleaudio = benchmark(log_melspectrogram)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
feature_librosa = librosa.power_to_db(feature_librosa, top_db=80.0)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
def test_log_melspect_gpu(benchmark):
enable_gpu_device()
feature_paddleaudio = benchmark(log_melspectrogram)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
feature_librosa = librosa.power_to_db(feature_librosa, top_db=80.0)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=2)
mel_extractor_torchaudio = torchaudio.transforms.MelSpectrogram(
**mel_conf_torchaudio, f_min=0.0)
amplitude_to_DB = torchaudio.transforms.AmplitudeToDB('power', top_db=80.0)
def melspectrogram_torchaudio():
return mel_extractor_torchaudio(waveform_tensor_torch).squeeze(0)
def log_melspectrogram_torchaudio():
mel_specgram = mel_extractor_torchaudio(waveform_tensor_torch)
return amplitude_to_DB(mel_specgram).squeeze(0)
def test_log_melspect_cpu_torchaudio(benchmark):
global waveform_tensor_torch, mel_extractor_torchaudio, amplitude_to_DB
mel_extractor_torchaudio = mel_extractor_torchaudio.to('cpu')
waveform_tensor_torch = waveform_tensor_torch.to('cpu')
amplitude_to_DB = amplitude_to_DB.to('cpu')
feature_paddleaudio = benchmark(log_melspectrogram_torchaudio)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
feature_librosa = librosa.power_to_db(feature_librosa, top_db=80.0)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
def test_log_melspect_gpu_torchaudio(benchmark):
global waveform_tensor_torch, mel_extractor_torchaudio, amplitude_to_DB
mel_extractor_torchaudio = mel_extractor_torchaudio.to('cuda')
waveform_tensor_torch = waveform_tensor_torch.to('cuda')
amplitude_to_DB = amplitude_to_DB.to('cuda')
feature_torchaudio = benchmark(log_melspectrogram_torchaudio)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
feature_librosa = librosa.power_to_db(feature_librosa, top_db=80.0)
np.testing.assert_array_almost_equal(
feature_librosa, feature_torchaudio.cpu(), decimal=2)
paddleaudio/tests/benchmark/melspectrogram.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import urllib.request
import librosa
import numpy as np
import paddle
import torch
import torchaudio
import paddleaudio
wav_url = 'https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav'
if not os.path.isfile(os.path.basename(wav_url)):
urllib.request.urlretrieve(wav_url, os.path.basename(wav_url))
waveform, sr = paddleaudio.load(os.path.abspath(os.path.basename(wav_url)))
waveform_tensor = paddle.to_tensor(waveform).unsqueeze(0)
waveform_tensor_torch = torch.from_numpy(waveform).unsqueeze(0)
# Feature conf
mel_conf = {
'sr': sr,
'n_fft': 512,
'hop_length': 128,
'n_mels': 40,
}
mel_conf_torchaudio = {
'sample_rate': sr,
'n_fft': 512,
'hop_length': 128,
'n_mels': 40,
'norm': 'slaney',
'mel_scale': 'slaney',
}
def enable_cpu_device():
paddle.set_device('cpu')
def enable_gpu_device():
paddle.set_device('gpu')
mel_extractor = paddleaudio.features.MelSpectrogram(
**mel_conf, f_min=0.0, dtype=waveform_tensor.dtype)
def melspectrogram():
return mel_extractor(waveform_tensor).squeeze(0)
def test_melspect_cpu(benchmark):
enable_cpu_device()
feature_paddleaudio = benchmark(melspectrogram)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
def test_melspect_gpu(benchmark):
enable_gpu_device()
feature_paddleaudio = benchmark(melspectrogram)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
mel_extractor_torchaudio = torchaudio.transforms.MelSpectrogram(
**mel_conf_torchaudio, f_min=0.0)
def melspectrogram_torchaudio():
return mel_extractor_torchaudio(waveform_tensor_torch).squeeze(0)
def test_melspect_cpu_torchaudio(benchmark):
global waveform_tensor_torch, mel_extractor_torchaudio
mel_extractor_torchaudio = mel_extractor_torchaudio.to('cpu')
waveform_tensor_torch = waveform_tensor_torch.to('cpu')
feature_paddleaudio = benchmark(melspectrogram_torchaudio)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
def test_melspect_gpu_torchaudio(benchmark):
global waveform_tensor_torch, mel_extractor_torchaudio
mel_extractor_torchaudio = mel_extractor_torchaudio.to('cuda')
waveform_tensor_torch = waveform_tensor_torch.to('cuda')
feature_torchaudio = benchmark(melspectrogram_torchaudio)
feature_librosa = librosa.feature.melspectrogram(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_torchaudio.cpu(), decimal=3)
paddleaudio/tests/benchmark/mfcc.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import urllib.request
import librosa
import numpy as np
import paddle
import torch
import torchaudio
import paddleaudio
wav_url = 'https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav'
if not os.path.isfile(os.path.basename(wav_url)):
urllib.request.urlretrieve(wav_url, os.path.basename(wav_url))
waveform, sr = paddleaudio.load(os.path.abspath(os.path.basename(wav_url)))
waveform_tensor = paddle.to_tensor(waveform).unsqueeze(0)
waveform_tensor_torch = torch.from_numpy(waveform).unsqueeze(0)
# Feature conf
mel_conf = {
'sr': sr,
'n_fft': 512,
'hop_length': 128,
'n_mels': 40,
}
mfcc_conf = {
'n_mfcc': 20,
'top_db': 80.0,
}
mfcc_conf.update(mel_conf)
mel_conf_torchaudio = {
'sample_rate': sr,
'n_fft': 512,
'hop_length': 128,
'n_mels': 40,
'norm': 'slaney',
'mel_scale': 'slaney',
}
mfcc_conf_torchaudio = {
'sample_rate': sr,
'n_mfcc': 20,
}
def enable_cpu_device():
paddle.set_device('cpu')
def enable_gpu_device():
paddle.set_device('gpu')
mfcc_extractor = paddleaudio.features.MFCC(
**mfcc_conf, f_min=0.0, dtype=waveform_tensor.dtype)
def mfcc():
return mfcc_extractor(waveform_tensor).squeeze(0)
def test_mfcc_cpu(benchmark):
enable_cpu_device()
feature_paddleaudio = benchmark(mfcc)
feature_librosa = librosa.feature.mfcc(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
def test_mfcc_gpu(benchmark):
enable_gpu_device()
feature_paddleaudio = benchmark(mfcc)
feature_librosa = librosa.feature.mfcc(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
del mel_conf_torchaudio['sample_rate']
mfcc_extractor_torchaudio = torchaudio.transforms.MFCC(
**mfcc_conf_torchaudio, melkwargs=mel_conf_torchaudio)
def mfcc_torchaudio():
return mfcc_extractor_torchaudio(waveform_tensor_torch).squeeze(0)
def test_mfcc_cpu_torchaudio(benchmark):
global waveform_tensor_torch, mfcc_extractor_torchaudio
mel_extractor_torchaudio = mfcc_extractor_torchaudio.to('cpu')
waveform_tensor_torch = waveform_tensor_torch.to('cpu')
feature_paddleaudio = benchmark(mfcc_torchaudio)
feature_librosa = librosa.feature.mfcc(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_paddleaudio, decimal=3)
def test_mfcc_gpu_torchaudio(benchmark):
global waveform_tensor_torch, mfcc_extractor_torchaudio
mel_extractor_torchaudio = mfcc_extractor_torchaudio.to('cuda')
waveform_tensor_torch = waveform_tensor_torch.to('cuda')
feature_torchaudio = benchmark(mfcc_torchaudio)
feature_librosa = librosa.feature.mfcc(waveform, **mel_conf)
np.testing.assert_array_almost_equal(
feature_librosa, feature_torchaudio.cpu(), decimal=3)
paddleaudio/tests/features/__init__.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
paddleaudio/tests/features/base.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import unittest
import urllib.request
import numpy as np
import paddle
from paddleaudio import load
wav_url = 'https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav'
class FeatTest(unittest.TestCase):
def setUp(self):
self.initParmas()
self.initWavInput()
self.setUpDevice()
def setUpDevice(self, device='cpu'):
paddle.set_device(device)
def initWavInput(self, url=wav_url):
if not os.path.isfile(os.path.basename(url)):
urllib.request.urlretrieve(url, os.path.basename(url))
self.waveform, self.sr = load(os.path.abspath(os.path.basename(url)))
self.waveform = self.waveform.astype(
np.float32
) # paddlespeech.s2t.transform.spectrogram only supports float32
dim = len(self.waveform.shape)
assert dim in [1, 2]
if dim == 1:
self.waveform = np.expand_dims(self.waveform, 0)
def initParmas(self):
raise NotImplementedError
paddleaudio/tests/features/test_istft.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import unittest
import numpy as np
import paddle
from .base import FeatTest
from paddleaudio.functional.window import get_window
from paddlespeech.s2t.transform.spectrogram import IStft
from paddlespeech.s2t.transform.spectrogram import Stft
class TestIstft(FeatTest):
def initParmas(self):
self.n_fft = 512
self.hop_length = 128
self.window_str = 'hann'
def test_istft(self):
ps_stft = Stft(self.n_fft, self.hop_length)
ps_res = ps_stft(
self.waveform.T).squeeze(1).T # (n_fft//2 + 1, n_frmaes)
x = paddle.to_tensor(ps_res)
ps_istft = IStft(self.hop_length)
ps_res = ps_istft(ps_res.T)
window = get_window(
self.window_str, self.n_fft, dtype=self.waveform.dtype)
pd_res = paddle.signal.istft(
x, self.n_fft, self.hop_length, window=window)
np.testing.assert_array_almost_equal(ps_res, pd_res, decimal=5)
if __name__ == '__main__':
unittest.main()
paddleaudio/tests/features/test_kaldi.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import unittest
import numpy as np
import paddle
import torch
import torchaudio
import paddleaudio
from .base import FeatTest
class TestKaldi(FeatTest):
def initParmas(self):
self.window_size = 1024
self.dtype = 'float32'
def test_window(self):
t_hann_window = torch.hann_window(
self.window_size, periodic=False, dtype=eval(f'torch.{self.dtype}'))
t_hamm_window = torch.hamming_window(
self.window_size,
periodic=False,
alpha=0.54,
beta=0.46,
dtype=eval(f'torch.{self.dtype}'))
t_povey_window = torch.hann_window(
self.window_size, periodic=False,
dtype=eval(f'torch.{self.dtype}')).pow(0.85)
p_hann_window = paddleaudio.functional.window.get_window(
'hann',
self.window_size,
fftbins=False,
dtype=eval(f'paddle.{self.dtype}'))
p_hamm_window = paddleaudio.functional.window.get_window(
'hamming',
self.window_size,
fftbins=False,
dtype=eval(f'paddle.{self.dtype}'))
p_povey_window = paddleaudio.functional.window.get_window(
'hann',
self.window_size,
fftbins=False,
dtype=eval(f'paddle.{self.dtype}')).pow(0.85)
np.testing.assert_array_almost_equal(t_hann_window, p_hann_window)
np.testing.assert_array_almost_equal(t_hamm_window, p_hamm_window)
np.testing.assert_array_almost_equal(t_povey_window, p_povey_window)
def test_fbank(self):
ta_features = torchaudio.compliance.kaldi.fbank(
torch.from_numpy(self.waveform.astype(self.dtype)))
pa_features = paddleaudio.compliance.kaldi.fbank(
paddle.to_tensor(self.waveform.astype(self.dtype)))
np.testing.assert_array_almost_equal(
ta_features, pa_features, decimal=4)
def test_mfcc(self):
ta_features = torchaudio.compliance.kaldi.mfcc(
torch.from_numpy(self.waveform.astype(self.dtype)))
pa_features = paddleaudio.compliance.kaldi.mfcc(
paddle.to_tensor(self.waveform.astype(self.dtype)))
np.testing.assert_array_almost_equal(
ta_features, pa_features, decimal=4)
if __name__ == '__main__':
unittest.main()
paddleaudio/tests/features/test_librosa.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddleaudio/tests/features/test_log_melspectrogram.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import unittest
import numpy as np
import paddle
import paddleaudio
from .base import FeatTest
from paddlespeech.s2t.transform.spectrogram import LogMelSpectrogram
class TestLogMelSpectrogram(FeatTest):
def initParmas(self):
self.n_fft = 512
self.hop_length = 128
self.n_mels = 40
def test_log_melspect(self):
ps_melspect = LogMelSpectrogram(self.sr, self.n_mels, self.n_fft,
self.hop_length)
ps_res = ps_melspect(self.waveform.T).squeeze(1).T
x = paddle.to_tensor(self.waveform)
# paddlespeech.s2t的特征存在幅度谱和功率谱滥用的情况
ps_melspect = paddleaudio.features.LogMelSpectrogram(
self.sr,
self.n_fft,
self.hop_length,
power=1.0,
n_mels=self.n_mels,
f_min=0.0)
pa_res = (ps_melspect(x) / 10.0).squeeze(0).numpy()
np.testing.assert_array_almost_equal(ps_res, pa_res, decimal=5)
if __name__ == '__main__':
unittest.main()
paddleaudio/tests/features/test_spectrogram.py 0 → 100644
浏览文件 @ 490300f8
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import unittest
import numpy as np
import paddle
import paddleaudio
from .base import FeatTest
from paddlespeech.s2t.transform.spectrogram import Spectrogram
class TestSpectrogram(FeatTest):
def initParmas(self):
self.n_fft = 512
self.hop_length = 128
def test_spectrogram(self):
ps_spect = Spectrogram(self.n_fft, self.hop_length)
ps_res = ps_spect(self.waveform.T).squeeze(1).T # Magnitude
x = paddle.to_tensor(self.waveform)
pa_spect = paddleaudio.features.Spectrogram(
self.n_fft, self.hop_length, power=1.0)
pa_res = pa_spect(x).squeeze(0).numpy()
np.testing.assert_array_almost_equal(ps_res, pa_res, decimal=5)
if __name__ == '__main__':
unittest.main()
paddleaudio/tests/features/test_stft.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/cli/cls/infer.py
浏览文件 @ 490300f8
...@@ -193,7 +193,8 @@ class CLSExecutor(BaseExecutor): ...@@ -193,7 +193,8 @@ class CLSExecutor(BaseExecutor):
sr=feat_conf['sample_rate'], sr=feat_conf['sample_rate'],
mono=True, mono=True,
dtype='float32') dtype='float32')
logger.info("Preprocessing audio_file:" + audio_file) if isinstance(audio_file, (str, os.PathLike)):
logger.info("Preprocessing audio_file:" + audio_file)
# Feature extraction # Feature extraction
feature_extractor = LogMelSpectrogram( feature_extractor = LogMelSpectrogram(
......
paddlespeech/cli/executor.py
浏览文件 @ 490300f8
...@@ -178,7 +178,8 @@ class BaseExecutor(ABC): ...@@ -178,7 +178,8 @@ class BaseExecutor(ABC):
Returns: Returns:
bool: return `True` for job input, `False` otherwise. bool: return `True` for job input, `False` otherwise.
""" """
return input_ and os.path.isfile(input_) and input_.endswith('.job') return input_ and os.path.isfile(input_) and (input_.endswith('.job') or
input_.endswith('.txt'))
def _get_job_contents( def _get_job_contents(
self, job_input: os.PathLike) -> Dict[str, Union[str, os.PathLike]]: self, job_input: os.PathLike) -> Dict[str, Union[str, os.PathLike]]:
......
paddlespeech/cli/tts/infer.py
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/cli/utils.py
浏览文件 @ 490300f8
...@@ -192,7 +192,7 @@ class ConfigCache: ...@@ -192,7 +192,7 @@ class ConfigCache:
try: try:
cfg = yaml.load(file, Loader=yaml.FullLoader) cfg = yaml.load(file, Loader=yaml.FullLoader)
self._data.update(cfg) self._data.update(cfg)
except: except Exception as e:
self.flush() self.flush()
@property @property
......
paddlespeech/server/__init__.py
浏览文件 @ 490300f8
...@@ -18,6 +18,7 @@ from .base_commands import ClientHelpCommand ...@@ -18,6 +18,7 @@ from .base_commands import ClientHelpCommand
from .base_commands import ServerBaseCommand from .base_commands import ServerBaseCommand
from .base_commands import ServerHelpCommand from .base_commands import ServerHelpCommand
from .bin.paddlespeech_client import ASRClientExecutor from .bin.paddlespeech_client import ASRClientExecutor
from .bin.paddlespeech_client import CLSClientExecutor
from .bin.paddlespeech_client import TTSClientExecutor from .bin.paddlespeech_client import TTSClientExecutor
from .bin.paddlespeech_server import ServerExecutor from .bin.paddlespeech_server import ServerExecutor
......
paddlespeech/server/engine/cls/__init__.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/server/engine/cls/python/__init__.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/server/engine/cls/python/cls_engine.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/server/restful/api.py
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/server/restful/cls_api.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/t2s/exps/csmsc_test.txt 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/t2s/exps/inference.py
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/t2s/exps/syn_utils.py 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
paddlespeech/t2s/exps/synthesize.py
浏览文件 @ 490300f8
此差异已折叠。
setup.py
浏览文件 @ 490300f8
此差异已折叠。
speechx/CMakeLists.txt
浏览文件 @ 490300f8
此差异已折叠。
speechx/README.md 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/build.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/EnableCMP0048.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/absl.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/boost.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/eigen.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/gflags.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/glog.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/gtest.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/kenlm.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/libsndfile.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/openblas.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/cmake/external/openfst.cmake 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/.gitignore 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/CMakeLists.txt 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/README.md 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/decoder/CMakeLists.txt 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/decoder/offline_decoder_main.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/decoder/path.sh 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/decoder/run.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/decoder/valgrind.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/feat/CMakeLists.txt 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/feat/feature-mfcc-test.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/feat/linear_spectrogram_main.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/feat/path.sh 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/feat/run.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/feat/valgrind.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/nnet/CMakeLists.txt 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/nnet/path.sh 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/nnet/pp-model-test.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/nnet/run.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/examples/nnet/valgrind.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
speechx/patch/CPPLINT.cfg 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/patch/openfst/src/include/fst/flags.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/patch/openfst/src/include/fst/log.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/patch/openfst/src/lib/flags.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/CMakeLists.txt
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/base/basic_types.h
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/base/common.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/base/flags.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/base/log.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/base/macros.h
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/base/thread_pool.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/codelab/README.md 已删除 100644 → 0
浏览文件 @ c36039ce
此差异已折叠。
speechx/speechx/decoder/common.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/decoder/ctc_beam_search_decoder.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/decoder/ctc_beam_search_decoder.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/decoder/ctc_decoders 0 → 120000
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/fbank.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/feature_cache.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/feature_cache.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/feature_extractor_controller.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/feature_extractor_interface.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/linear_spectrogram.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/linear_spectrogram.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/mfcc.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/normalizer.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/normalizer.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/raw_audio.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/raw_audio.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/frontend/window.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/decoder/lattice-faster-decoder.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/decoder/lattice-faster-decoder.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/feat/cmvn.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/feat/cmvn.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/lat/determinize-lattice-pruned.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/lat/determinize-lattice-pruned.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/lat/kaldi-lattice.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/lat/kaldi-lattice.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/lat/lattice-functions.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/lat/lattice-functions.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/kaldi/matrix/BUILD 已删除 100644 → 0
浏览文件 @ c36039ce
此差异已折叠。
speechx/speechx/nnet/CMakeLists.txt 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/nnet/decodable-itf.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/nnet/decodable.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/nnet/decodable.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/nnet/nnet_interface.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/nnet/paddle_nnet.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/nnet/paddle_nnet.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/utils/file_utils.cc 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/speechx/utils/file_utils.h 0 → 100644
浏览文件 @ 490300f8
此差异已折叠。
speechx/tools/setup_valgrind.sh 0 → 100755
浏览文件 @ 490300f8
此差异已折叠。
tests/test_tipc/benchmark_train.sh
浏览文件 @ 490300f8
此差异已折叠。
tests/test_tipc/prepare.sh
浏览文件 @ 490300f8
此差异已折叠。
tests/unit/cli/test_cli.sh
浏览文件 @ 490300f8
此差异已折叠。
tests/unit/server/change_yaml.py
浏览文件 @ 490300f8
此差异已折叠。
utils/generate_infer_yaml.py
浏览文件 @ 490300f8
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或