{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# 1. PP-TTS Introduction\n", "\n", "## 1.1 Introduction\n", "\n", "PP-TTS is a streaming speech synthesis system developed by PaddleSpeech. Based on the implementation of [SOTA Algorithms](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/released_model.md#text-to-speech-models), a faster inference engine is used to realize streaming speech synthesis technology to meet the needs of commercial speech interaction scenarios.\n", "\n", "\n", "#### PP-TTS\n", "
\n", "\n", "PP-TTS provides a Chinese streaming speech synthesis system based on FastSpeech2 and HiFiGAN by default:\n", "\n", "- Text Frontend： The rule-based Chinese text frontend system is adopted to optimize Chinese text such as text normalization, polyphony, and tone sandhi.\n", "- Acoustic Model: The decoder of FastSpeech2 is improved so that it can be stream synthesized\n", "- Vocoder: Streaming synthesis of GAN vocoder is supported\n", "- Inference Engine： Using ONNXRuntime to optimize the inference of TTS models, so that the TTS system can also achieve RTF < 1 on low-voltage, meeting the requirements of streaming synthesis\n", "\n", "\n", "## 1.2 Characteristic\n", "- Open source leading Chinese TTS system\n", "- Using ONNXRuntime to optimize the inference of TTS models\n", "- The only open-source streaming TTS system\n", "- Easy disassembly: Developers can easily replace different acoustic models and vocoders in different languages, use different inference engines (Paddle dynamic graph, PaddleInference, ONNXRuntime, etc.), and use different network services (HTTP, WebSocket)\n", "\n", "\n", "# 2. Model Effects and Application Scenarios\n", "## 2.1 TTS\n", "## 2.1.1 Datasets：\n", "\n", "Common TTS datasets are shown in the following table:\n", "\n", "| language | dataset |audio info | describtion |\n", "| -------- | -------- | -------- | -------- |\n", "| Chinese | [CSMSC](https://www.data-baker.com/open_source.html) | 48KHz, 16bit | single speaker，female，12 h|\n", "| Chinese | [AISHELL-3](http://www.aishelltech.com/aishell_3) | 44.1kHz，16bit |multi-speakers，85 h|\n", "| English | [LJSpeech-1.1](https://keithito.com/LJ-Speech-Dataset/) | 22050Hz, 16bit | single speaker，female，24 h|\n", "| English | [VCTK](https://datashare.ed.ac.uk/handle/10283/3443) | 48kHz, 16bit | multi-speakers，44 h|\n", "\n", "## 2.1.2 Model Effects\n", "Click [link](https://paddlespeech.readthedocs.io/en/latest/tts/demo.html) to listen synthetisized audios.\n", "\n", "# 3. How to Use the Model\n", "## 3.1 Model Inference\n", "### Install paddlespeech" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "!pip install paddlespeech" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "!wget https://paddlespeech.bj.bcebos.com/Parakeet/tools/nltk_data.tar.gz\n", "!tar zxvf nltk_data.tar.gz" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from paddlespeech.cli.tts import TTSExecutor\n", "tts_executor = TTSExecutor()\n", "wav_file = tts_executor(\n", " text=\"热烈欢迎您在 Discussions 中提交问题，并在 Issues 中指出发现的 bug。此外，我们非常希望您参与到 Paddle Speech 的开发中！\",\n", " output='output.wav',\n", " am='fastspeech2_mix',\n", " voc='hifigan_csmsc',\n", " lang='mix',\n", " spk_id=174)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import IPython.display as dp\n", "dp.Audio('output.wav')" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## 3.2 Model Training\n", "- [train FastSpeech2 with CSMCS dataset](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/tts3)\n", "- [train HiFiGAN with CSMCS dataset](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/voc5)\n", "# 4. Model Principles\n", "\n", "\n", "### 4.1 Acoustic Model FastSpeech2\n", "We use `FastSpeech2` as acoustic model.\n", "
\n", "
FastSpeech2 Model Structure

\n", "\n", "\n", "PaddleSpeech TTS's FastSpeech2 is different with paper，we use phone level `pitch` and `energy`(which is more like FastPitch)，this can make the synthesized audio more **stable**.\n", "
\n", "
FastPitch Model Structure

\n", "\n", "### 4.2 Vocoder HiFiGAN\n", "We use `HiFiGAN` as Vocoder.\n", "\n", "1. Introduced MPD（Multi-Period Discriminator）。HiFiGAN both have MSD（Multi-Scale Discriminator）and MPD（Multi-Period Discriminator），The goal is to enhance the ability of GAN discriminator to distinguish synthetic or real audio as much as possible.\n", "2. Generator introduced Multi receptive field fusion module. In order to increase the receptive field, WaveNet superimposes hole convolution and generates by sample point. The sound quality is really good, but it also makes the model larger and the reasoning speed slower. HiFiGAN proposed a residual structure, which uses the dilated convolution and vanilla convolution alternately to increase the receptive field, ensure the synthetic sound quality and improve the reasoning speed.\n", "\n", "\"hifigan\"\n", "\n", "
HiFiGAN Model Structure

\n", "\n", "\n", "# 5. Attention\n", "# 6. Related papers and citations\n", "```text\n", "@article{ren2020fastspeech,\n", " title={Fastspeech 2: Fast and high-quality end-to-end text to speech},\n", " author={Ren, Yi and Hu, Chenxu and Tan, Xu and Qin, Tao and Zhao, Sheng and Zhao, Zhou and Liu, Tie-Yan},\n", " journal={arXiv preprint arXiv:2006.04558},\n", " year={2020}\n", "}\n", "\n", "@article{kong2020hifi,\n", " title={Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis},\n", " author={Kong, Jungil and Kim, Jaehyeon and Bae, Jaekyoung},\n", " journal={Advances in Neural Information Processing Systems},\n", " volume={33},\n", " pages={17022--17033},\n", " year={2020}\n", "}\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "py35-paddle1.2.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 1 }