Skip to content

D
DeepSpeech

PaddlePaddle / DeepSpeech
大约 2 年 前同步成功

通知 210
Star 8425
Fork 1598
D
DeepSpeech
“291ff97cd2302661e6e007b587f49a656a257a6d”上不存在“paddle/scripts/docker/Dockerfile.gpu-demo”
未验证 提交 8fd97642 编写于 作者: H Hui Zhang 提交者： GitHub
浏览文件
操作

Merge pull request #1016 from yt605155624/mbmelgan_fine 

[TTS]add aishell3/vc1 readme, add csmsc/voc1 readme
上级 d2a05df0 bdd2fb8f
隐藏空白更改
内联 并排
Showing with 188 addition and 103 deletion
examples/aishell3/tts3/README.md
浏览文件 @ 8fd97642
...@@ -24,7 +24,7 @@ Assume the path to the dataset is `~/datasets/data_aishell3`. ...@@ -24,7 +24,7 @@ Assume the path to the dataset is `~/datasets/data_aishell3`.
Assume the path to the MFA result of AISHELL-3 is `./aishell3_alignment_tone`. Assume the path to the MFA result of AISHELL-3 is `./aishell3_alignment_tone`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
examples/aishell3/vc0/README.md
浏览文件 @ 8fd97642
# Tacotron2 + AISHELL-3 Voice Cloning # Tacotron2 + AISHELL-3 Voice Cloning
This example contains code used to train a [Tacotron2 ](https://arxiv.org/abs/1712.05884) model with [AISHELL-3](http://www.aishelltech.com/aishell_3). The trained model can be used in Voice Cloning Task, We refer to the model structure of [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://arxiv.org/pdf/1806.04558.pdf) . The general steps are as follows: This example contains code used to train a [Tacotron2 ](https://arxiv.org/abs/1712.05884) model with [AISHELL-3](http://www.aishelltech.com/aishell_3). The trained model can be used in Voice Cloning Task, We refer to the model structure of [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://arxiv.org/pdf/1806.04558.pdf) . The general steps are as follows:
1. Speaker Encoder: We use a Speaker Verification to train a speaker encoder. Datasets used in this task are different from those used in Tacotron2, because the transcriptions are not needed, we use more datasets, refer to [ge2e](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/ge2e). 1. Speaker Encoder: We use a Speaker Verification to train a speaker encoder. Datasets used in this task are different from those used in Tacotron2, because the transcriptions are not needed, we use more datasets, refer to [ge2e](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/ge2e).
2. Synthesizer: Then, we use the trained speaker encoder to generate utterance embedding for each sentence in AISHELL-3. This embedding is a extra input of Tacotron2 which will be concated with encoder outputs. 2. Synthesizer: We use the trained speaker encoder to generate speaker embedding for each sentence in AISHELL-3. This embedding is an extra input of Tacotron2 which will be concated with encoder outputs.
3. Vocoder: We use WaveFlow as the neural Vocoder, refer to [waveflow](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc0). 3. Vocoder: We use WaveFlow as the neural Vocoder, refer to [waveflow](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc0).
## Get Started ## Get Started
...@@ -10,7 +10,7 @@ Assume the path to the MFA result of AISHELL-3 is `./alignment`. ...@@ -10,7 +10,7 @@ Assume the path to the MFA result of AISHELL-3 is `./alignment`.
Assume the path to the pretrained ge2e model is `ge2e_ckpt_path=./ge2e_ckpt_0.3/step-3000000` Assume the path to the pretrained ge2e model is `ge2e_ckpt_path=./ge2e_ckpt_0.3/step-3000000`
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. start a voice cloning inference. 4. start a voice cloning inference.
```bash ```bash
...@@ -20,8 +20,8 @@ Run the command below to ...@@ -20,8 +20,8 @@ Run the command below to
```bash ```bash
CUDA_VISIBLE_DEVICES=${gpus} ./local/preprocess.sh ${input} ${preprocess_path} ${alignment} ${ge2e_ckpt_path} CUDA_VISIBLE_DEVICES=${gpus} ./local/preprocess.sh ${input} ${preprocess_path} ${alignment} ${ge2e_ckpt_path}
``` ```
#### generate utterance embedding #### generate speaker embedding
Use pretrained GE2E (speaker encoder) to generate utterance embedding for each sentence in AISHELL-3, which has the same file structure with wav files and the format is `.npy`. Use pretrained GE2E (speaker encoder) to generate speaker embedding for each sentence in AISHELL-3, which has the same file structure with wav files and the format is `.npy`.
```bash ```bash
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
...@@ -33,8 +33,8 @@ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then ...@@ -33,8 +33,8 @@ if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
fi fi
``` ```
The computing time of utterance embedding can be x hours. The computing time of utterance embedding can be x hours.
#### process wav #### process wav
There are silence in the edge of AISHELL-3's wavs, and the audio amplitude is very small, so, we need to remove the silence and normalize the audio. You can the silence remove method based on volume or energy, but the effect is not very good, We use [MFA](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) to get the alignment of text and speech, then utilize the alignment results to remove the silence. There are silence in the edge of AISHELL-3's wavs, and the audio amplitude is very small, so, we need to remove the silence and normalize the audio. You can the silence remove method based on volume or energy, but the effect is not very good, We use [MFA](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) to get the alignment of text and speech, then utilize the alignment results to remove the silence.
We use Montreal Force Aligner 1.0. The label in aishell3 include pinyin，so the lexicon we provided to MFA is pinyin rather than Chinese characters. And the prosody marks(`$` and `%`) need to be removed. You shoud preprocess the dataset into the format which MFA needs, the texts have the same name with wavs and have the suffix `.lab`. We use Montreal Force Aligner 1.0. The label in aishell3 include pinyin，so the lexicon we provided to MFA is pinyin rather than Chinese characters. And the prosody marks(`$` and `%`) need to be removed. You shoud preprocess the dataset into the format which MFA needs, the texts have the same name with wavs and have the suffix `.lab`.
...@@ -73,7 +73,7 @@ if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then ...@@ -73,7 +73,7 @@ if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
fi fi
``` ```
### Train the model ### Train the model
```bash ```bash
CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${preprocess_path} ${train_output_path} CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${preprocess_path} ${train_output_path}
``` ```
...@@ -81,7 +81,7 @@ CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${preprocess_path} ${train_output_ ...@@ -81,7 +81,7 @@ CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${preprocess_path} ${train_output_
Our model remve stop token prediction in Tacotron2, because of the problem of extremely unbalanced proportion of positive and negative samples of stop token prediction, and it's very sensitive to the clip of audio silence. We use the last symbol from the highest point of attention to the encoder side as the termination condition. Our model remve stop token prediction in Tacotron2, because of the problem of extremely unbalanced proportion of positive and negative samples of stop token prediction, and it's very sensitive to the clip of audio silence. We use the last symbol from the highest point of attention to the encoder side as the termination condition.
In addition, in order to accelerate the convergence of the model, we add `guided attention loss` to induce the alignment between encoder and decoder to show diagonal lines faster. In addition, in order to accelerate the convergence of the model, we add `guided attention loss` to induce the alignment between encoder and decoder to show diagonal lines faster.
### Infernece ### Voice Cloning
```bash ```bash
CUDA_VISIBLE_DEVICES=${gpus} ./local/voice_cloning.sh ${ge2e_params_path} ${tacotron2_params_path} ${waveflow_params_path} ${vc_input} ${vc_output} CUDA_VISIBLE_DEVICES=${gpus} ./local/voice_cloning.sh ${ge2e_params_path} ${tacotron2_params_path} ${waveflow_params_path} ${vc_input} ${vc_output}
``` ```
......
examples/aishell3/vc1/README.md
浏览文件 @ 8fd97642
# FastSpeech2 + AISHELL-3 Voice Cloning # FastSpeech2 + AISHELL-3 Voice Cloning
This example contains code used to train a [Tacotron2 ](https://arxiv.org/abs/1712.05884) model with [AISHELL-3](http://www.aishelltech.com/aishell_3). The trained model can be used in Voice Cloning Task, We refer to the model structure of [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://arxiv.org/pdf/1806.04558.pdf) . The general steps are as follows: This example contains code used to train a [FastSpeech2](https://arxiv.org/abs/2006.04558) model with [AISHELL-3](http://www.aishelltech.com/aishell_3). The trained model can be used in Voice Cloning Task, We refer to the model structure of [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://arxiv.org/pdf/1806.04558.pdf) . The general steps are as follows:
1. Speaker Encoder: We use a Speaker Verification to train a speaker encoder. Datasets used in this task are different from those used in Tacotron2, because the transcriptions are not needed, we use more datasets, refer to [ge2e](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/ge2e). 1. Speaker Encoder: We use a Speaker Verification to train a speaker encoder. Datasets used in this task are different from those used in `FastSpeech2`, because the transcriptions are not needed, we use more datasets, refer to [ge2e](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/ge2e).
2. Synthesizer: Then, we use the trained speaker encoder to generate utterance embedding for each sentence in AISHELL-3. This embedding is a extra input of Tacotron2 which will be concated with encoder outputs. 2. Synthesizer: We use the trained speaker encoder to generate speaker embedding for each sentence in AISHELL-3. This embedding is an extra input of `FastSpeech2` which will be concated with encoder outputs.
3. Vocoder: We use WaveFlow as the neural Vocoder, refer to [waveflow](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/ljspeech/voc0). 3. Vocoder: We use [Parallel Wave GAN](http://arxiv.org/abs/1910.11480) as the neural Vocoder, refer to [voc1](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/voc1).
## Dataset
### Download and Extract the datasaet
Download AISHELL-3.
```bash
wget https://www.openslr.org/resources/93/data_aishell3.tgz
```
Extract AISHELL-3.
```bash
mkdir data_aishell3
tar zxvf data_aishell3.tgz -C data_aishell3
```
### Get MFA result of AISHELL-3 and Extract it
We use [MFA2.x](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) to get durations for aishell3_fastspeech2.
You can download from here [aishell3_alignment_tone.tar.gz](https://paddlespeech.bj.bcebos.com/MFA/AISHELL-3/with_tone/aishell3_alignment_tone.tar.gz), or train your own MFA model reference to [use_mfa example](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/use_mfa) (use MFA1.x now) of our repo.
## Pretrained GE2E model
We use pretrained GE2E model to generate spwaker embedding for each sentence.
Download pretrained GE2E model from here [ge2e_ckpt_0.3.zip](https://paddlespeech.bj.bcebos.com/Parakeet/ge2e_ckpt_0.3.zip), and `unzip` it.
## Get Started ## Get Started
Assume the path to the dataset is `~/datasets/data_aishell3`. Assume the path to the dataset is `~/datasets/data_aishell3`.
Assume the path to the MFA result of AISHELL-3 is `./alignment`. Assume the path to the MFA result of AISHELL-3 is `./aishell3_alignment_tone`.
Assume the path to the pretrained ge2e model is `ge2e_ckpt_path=./ge2e_ckpt_0.3/step-3000000` Assume the path to the pretrained ge2e model is `./ge2e_ckpt_0.3`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. start a voice cloning inference. 4. synthesize waveform from `metadata.jsonl`.
5. start a voice cloning inference.
```bash ```bash
./run.sh ./run.sh
``` ```
### Preprocess the dataset ### Preprocess the dataset
```bash ```bash
CUDA_VISIBLE_DEVICES=${gpus} ./local/preprocess.sh ${input} ${preprocess_path} ${alignment} ${ge2e_ckpt_path} CUDA_VISIBLE_DEVICES=${gpus} ./local/preprocess.sh ${conf_path} ${ge2e_ckpt_path}
``` ```
#### generate utterance embedding When it is done. A `dump` folder is created in the current directory. The structure of the dump folder is listed below.
Use pretrained GE2E (speaker encoder) to generate utterance embedding for each sentence in AISHELL-3, which has the same file structure with wav files and the format is `.npy`. ```text
dump
```bash ├── dev
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then │ ├── norm
python3 ${BIN_DIR}/../ge2e/inference.py \ │ └── raw
--input=${input} \ ├── embed
--output=${preprocess_path}/embed \ │ ├── SSB0005
--ngpu=1 \ │ ├── SSB0009
--checkpoint_path=${ge2e_ckpt_path} │ ├── ...
fi │ └── ...
├── phone_id_map.txt
├── speaker_id_map.txt
├── test
│ ├── norm
│ └── raw
└── train
├── energy_stats.npy
├── norm
├── pitch_stats.npy
├── raw
└── speech_stats.npy
``` ```
The `embed` contains the generated speaker embedding for each sentence in AISHELL-3, which has the same file structure with wav files and the format is `.npy`.
The computing time of utterance embedding can be x hours. The computing time of utterance embedding can be x hours.
#### process wav
There are silence in the edge of AISHELL-3's wavs, and the audio amplitude is very small, so, we need to remove the silence and normalize the audio. You can the silence remove method based on volume or energy, but the effect is not very good, We use [MFA](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) to get the alignment of text and speech, then utilize the alignment results to remove the silence.
We use Montreal Force Aligner 1.0. The label in aishell3 include pinyin，so the lexicon we provided to MFA is pinyin rather than Chinese characters. And the prosody marks(`$` and `%`) need to be removed. You shoud preprocess the dataset into the format which MFA needs, the texts have the same name with wavs and have the suffix `.lab`. The dataset is split into 3 parts, namely `train`, `dev` and` test`, each of which contains a `norm` and `raw` sub folder. The raw folder contains speech、pitch and energy features of each utterances, while the norm folder contains normalized ones. The statistics used to normalize features are computed from the training set, which is located in `dump/train/*_stats.npy`.
We use [lexicon.txt](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/paddlespeech/t2s/exps/voice_cloning/tacotron2_ge2e/lexicon.txt) as the lexicon. Also there is a `metadata.jsonl` in each subfolder. It is a table-like file which contains phones, text_lengths, speech_lengths, durations, path of speech features, path of pitch features, path of energy features, speaker and id of each utterance.
You can download the alignment results from here [alignment_aishell3.tar.gz](https://paddlespeech.bj.bcebos.com/Parakeet/alignment_aishell3.tar.gz), or train your own MFA model reference to [use_mfa example](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/use_mfa) (use MFA1.x now) of our repo. The preprocessing step is very similar to that one of [tts3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/tts3), but there is one more `ge2e/inference` step here.
### Train the model
`./local/train.sh` calls `${BIN_DIR}/train.py`.
```bash ```bash
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${conf_path} ${train_output_path}
echo "Process wav ..."
python3 ${BIN_DIR}/process_wav.py \
--input=${input}/wav \
--output=${preprocess_path}/normalized_wav \
--alignment=${alignment}
fi
``` ```
The training step is very similar to that one of [tts3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/tts3), but we should set `--voice-cloning=True` when calling `${BIN_DIR}/train.py`.
#### preprocess transcription ### Synthesize
We revert the transcription into `phones` and `tones`. It is worth noting that our processing here is different from that used for MFA, we separated the tones. This is a processing method, of course, you can only segment initials and vowels. We use [parallel wavegan](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/voc1) as the neural vocoder.
Download pretrained parallel wavegan model from [pwg_aishell3_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/pwg_aishell3_ckpt_0.5.zip) and unzip it.
```bash ```bash
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then unzip pwg_aishell3_ckpt_0.5.zip
python3 ${BIN_DIR}/preprocess_transcription.py \
--input=${input} \
--output=${preprocess_path}
fi
``` ```
The default input is `~/datasets/data_aishell3/train`，which contains `label_train-set.txt`, the processed results are `metadata.yaml` and `metadata.pickle`. the former is a text format for easy viewing, and the latter is a binary format for direct reading. Parallel WaveGAN checkpoint contains files listed below.
#### extract mel ```text
```python pwg_aishell3_ckpt_0.5
if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then ├── default.yaml # default config used to train parallel wavegan
python3 ${BIN_DIR}/extract_mel.py \ ├── feats_stats.npy # statistics used to normalize spectrogram when training parallel wavegan
--input=${preprocess_path}/normalized_wav \ └── snapshot_iter_1000000.pdz # generator parameters of parallel wavegan
--output=${preprocess_path}/mel
fi
``` ```
`./local/synthesize.sh` calls `${BIN_DIR}/synthesize.py`, which can synthesize waveform from `metadata.jsonl`.
### Train the model
```bash ```bash
CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${preprocess_path} ${train_output_path} CUDA_VISIBLE_DEVICES=${gpus} ./local/synthesize.sh ${conf_path} ${train_output_path} ${ckpt_name}
``` ```
The synthesizing step is very similar to that one of [tts3](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/aishell3/tts3), but we should set `--voice-cloning=True` when calling `${BIN_DIR}/synthesize.py`.
Our model remve stop token prediction in Tacotron2, because of the problem of extremely unbalanced proportion of positive and negative samples of stop token prediction, and it's very sensitive to the clip of audio silence. We use the last symbol from the highest point of attention to the encoder side as the termination condition. ### Voice Cloning
Assume there are some reference audios in `./ref_audio`
```text
ref_audio
├── 001238.wav
├── LJ015-0254.wav
└── audio_self_test.mp3
```
`./local/voice_cloning.sh` calls `${BIN_DIR}/voice_cloning.py`
In addition, in order to accelerate the convergence of the model, we add `guided attention loss` to induce the alignment between encoder and decoder to show diagonal lines faster.
### Infernece
```bash ```bash
CUDA_VISIBLE_DEVICES=${gpus} ./local/voice_cloning.sh ${ge2e_params_path} ${tacotron2_params_path} ${waveflow_params_path} ${vc_input} ${vc_output} CUDA_VISIBLE_DEVICES=${gpus} ./local/voice_cloning.sh ${conf_path} ${train_output_path} ${ckpt_name} ${ge2e_params_path} ${ref_audio_dir}
``` ```
## Pretrained Model ## Pretrained Model
[tacotron2_aishell3_ckpt_0.3.zip](https://paddlespeech.bj.bcebos.com/Parakeet/tacotron2_aishell3_ckpt_0.3.zip). [fastspeech2_nosil_aishell3_vc1_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/fastspeech2_nosil_aishell3_vc1_ckpt_0.5.zip)
FastSpeech2 checkpoint contains files listed below.
(There is no need for `speaker_id_map.txt` here )
```text
fastspeech2_nosil_aishell3_ckpt_vc1_0.5
├── default.yaml # default config used to train fastspeech2
├── phone_id_map.txt # phone vocabulary file when training fastspeech2
├── snapshot_iter_96400.pdz # model parameters and optimizer states
└── speech_stats.npy # statistics used to normalize spectrogram when training fastspeech2
```
examples/aishell3/voc1/README.md
浏览文件 @ 8fd97642
...@@ -22,7 +22,7 @@ Assume the path to the dataset is `~/datasets/data_aishell3`. ...@@ -22,7 +22,7 @@ Assume the path to the dataset is `~/datasets/data_aishell3`.
Assume the path to the MFA result of AISHELL-3 is `./aishell3_alignment_tone`. Assume the path to the MFA result of AISHELL-3 is `./aishell3_alignment_tone`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
examples/csmsc/tts2/README.md
浏览文件 @ 8fd97642
...@@ -14,7 +14,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`. ...@@ -14,7 +14,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`.
Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`. Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
...@@ -208,7 +208,8 @@ CUDA_VISIBLE_DEVICES=${gpus} ./local/inference.sh ${train_output_path} ...@@ -208,7 +208,8 @@ CUDA_VISIBLE_DEVICES=${gpus} ./local/inference.sh ${train_output_path}
``` ```
## Pretrained Model ## Pretrained Model
Pretrained SpeedySpeech model with no silence in the edge of audios. [speedyspeech_nosil_baker_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/speedyspeech_nosil_baker_ckpt_0.5.zip) Pretrained SpeedySpeech model with no silence in the edge of audios[speedyspeech_nosil_baker_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/speedyspeech_nosil_baker_ckpt_0.5.zip).
Static model can be downloaded here [speedyspeech_nosil_baker_static_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/speedyspeech_nosil_baker_static_0.5.zip). Static model can be downloaded here [speedyspeech_nosil_baker_static_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/speedyspeech_nosil_baker_static_0.5.zip).
SpeedySpeech checkpoint contains files listed below. SpeedySpeech checkpoint contains files listed below.
......
examples/csmsc/tts3/README.md
浏览文件 @ 8fd97642
...@@ -14,7 +14,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`. ...@@ -14,7 +14,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`.
Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`. Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
...@@ -199,8 +199,9 @@ CUDA_VISIBLE_DEVICES=${gpus} ./local/inference.sh ${train_output_path} ...@@ -199,8 +199,9 @@ CUDA_VISIBLE_DEVICES=${gpus} ./local/inference.sh ${train_output_path}
``` ```
## Pretrained Model ## Pretrained Model
Pretrained FastSpeech2 model with no silence in the edge of audios. [fastspeech2_nosil_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/fastspeech2_nosil_baker_ckpt_0.4.zip) Pretrained FastSpeech2 model with no silence in the edge of audios [fastspeech2_nosil_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/fastspeech2_nosil_baker_ckpt_0.4.zip).
Static model can be downloaded here [fastspeech2_nosil_baker_static_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/fastspeech2_nosil_baker_static_0.4.zip)
Static model can be downloaded here [fastspeech2_nosil_baker_static_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/fastspeech2_nosil_baker_static_0.4.zip).
FastSpeech2 checkpoint contains files listed below. FastSpeech2 checkpoint contains files listed below.
```text ```text
......
examples/csmsc/voc1/README.md
浏览文件 @ 8fd97642
...@@ -13,7 +13,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`. ...@@ -13,7 +13,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`.
Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`. Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
...@@ -123,7 +123,8 @@ optional arguments: ...@@ -123,7 +123,8 @@ optional arguments:
## Pretrained Models ## Pretrained Models
Pretrained model can be downloaded here [pwg_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/pwg_baker_ckpt_0.4.zip). Pretrained model can be downloaded here [pwg_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/pwg_baker_ckpt_0.4.zip).
Static models can be downloaded here [pwg_baker_static_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/pwg_baker_static_0.4.zip).
Static model can be downloaded here [pwg_baker_static_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/pwg_baker_static_0.4.zip).
Parallel WaveGAN checkpoint contains files listed below. Parallel WaveGAN checkpoint contains files listed below.
......
examples/csmsc/voc3/README.md
浏览文件 @ 8fd97642
...@@ -13,7 +13,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`. ...@@ -13,7 +13,7 @@ Assume the path to the dataset is `~/datasets/BZNSYP`.
Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`. Assume the path to the MFA result of CSMSC is `./baker_alignment_tone`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
...@@ -106,8 +106,51 @@ optional arguments: ...@@ -106,8 +106,51 @@ optional arguments:
4. `--output-dir` is the directory to save the synthesized audio files. 4. `--output-dir` is the directory to save the synthesized audio files.
5. `--ngpu` is the number of gpus to use, if ngpu == 0, use cpu. 5. `--ngpu` is the number of gpus to use, if ngpu == 0, use cpu.
## Finetune
Since there are no `noise` in the input of Multi Band MelGAN, the audio quality is not so good (see [espnet issue](https://github.com/espnet/espnet/issues/3536#issuecomment-916035415)), we refer to the method proposed in [HiFiGAN](https://arxiv.org/abs/2010.05646), finetune Multi Band MelGAN with the predicted mel-spectrogram from `FastSpeech2`.
The length of mel-spectrograms should align with the length of wavs, so we should generate mels using ground truth alignment.
But since we are fine-tuning, we should use the statistics computed during training step.
You should first download pretrained `FastSpeech2` model from [fastspeech2_nosil_baker_ckpt_0.4.zip](https://paddlespeech.bj.bcebos.com/Parakeet/fastspeech2_nosil_baker_ckpt_0.4.zip) and `unzip` it.
Assume the path to the dump-dir of training step is `dump`.
Assume the path to the duration result of CSMSC is `durations.txt` (generated during training step's preprocessing).
Assume the path to the pretrained `FastSpeech2` model is `fastspeech2_nosil_baker_ckpt_0.4`.
\
The `finetune.sh` can
1. **source path**.
2. generate ground truth alignment mels.
3. link `*_wave.npy` from `dump` to `dump_finetune` (because we only use new mels, the wavs are the ones used during train step) .
4. copy features' stats from `dump` to `dump_finetune`.
5. normalize the ground truth alignment mels.
6. finetune the model.
Before finetune, make sure that the pretrained model is in `finetune.sh` 's `${output-dir}/checkpoints`, and there is a `records.jsonl` in it to refer to this pretrained model
```text
exp/finetune/checkpoints
├── records.jsonl
└── snapshot_iter_1000000.pdz
```
The content of `records.jsonl` should be as follows (change `"path"` to your own ckpt path):
```
{"time": "2021-11-21 15:11:20.337311", "path": "~/PaddleSpeech/examples/csmsc/voc3/exp/finetune/checkpoints/snapshot_iter_1000000.pdz", "iteration": 1000000}↩
```
Run the command below
```bash
./finetune.sh
```
By default, `finetune.sh` will use `conf/finetune.yaml` as config, the dump-dir is `dump_finetune`, the experiment dir is `exp/finetune`.
TODO:
The hyperparameter of `finetune.yaml` is not good enough, a smaller `learning_rate` should be used (more `milestones` should be set).
## Pretrained Models ## Pretrained Models
Pretrained model can be downloaded here [mb_melgan_baker_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/mb_melgan_baker_ckpt_0.5.zip). Pretrained model can be downloaded here [mb_melgan_baker_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/mb_melgan_baker_ckpt_0.5.zip).
Finetuned model can ben downloaded here [mb_melgan_baker_finetune_ckpt_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/mb_melgan_baker_finetune_ckpt_0.5.zip).
Static model can be downloaded here [mb_melgan_baker_static_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/mb_melgan_baker_static_0.5.zip) Static model can be downloaded here [mb_melgan_baker_static_0.5.zip](https://paddlespeech.bj.bcebos.com/Parakeet/mb_melgan_baker_static_0.5.zip)
Multi Band MelGAN checkpoint contains files listed below. Multi Band MelGAN checkpoint contains files listed below.
......
examples/csmsc/voc3/conf/finetune.yaml
浏览文件 @ 8fd97642
...@@ -128,7 +128,7 @@ discriminator_scheduler_params: ...@@ -128,7 +128,7 @@ discriminator_scheduler_params:
# INTERVAL SETTING # # INTERVAL SETTING #
########################################################### ###########################################################
discriminator_train_start_steps: 200000 # Number of steps to start to train discriminator. discriminator_train_start_steps: 200000 # Number of steps to start to train discriminator.
train_max_steps: 1200000 # Number of training steps. train_max_steps: 2000000 # Number of training steps.
save_interval_steps: 1000 # Interval steps to save checkpoint. save_interval_steps: 1000 # Interval steps to save checkpoint.
eval_interval_steps: 1000 # Interval steps to evaluate the network. eval_interval_steps: 1000 # Interval steps to evaluate the network.
...@@ -136,4 +136,4 @@ eval_interval_steps: 1000 # Interval steps to evaluate the network ...@@ -136,4 +136,4 @@ eval_interval_steps: 1000 # Interval steps to evaluate the network
# OTHER SETTING # # OTHER SETTING #
########################################################### ###########################################################
num_snapshots: 10 # max number of snapshots to keep while training num_snapshots: 10 # max number of snapshots to keep while training
seed: 42 # random seed for paddle, random, and np.random seed: 42 # random seed for paddle, random, and np.random
\ No newline at end of file
examples/csmsc/voc3/finetune.sh
浏览文件 @ 8fd97642
...@@ -9,20 +9,19 @@ stop_stage=100 ...@@ -9,20 +9,19 @@ stop_stage=100
source ${MAIN_ROOT}/utils/parse_options.sh || exit 1 source ${MAIN_ROOT}/utils/parse_options.sh || exit 1
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
python3 ${MAIN_ROOT}/paddlespeech/t2s/exps/fastspeech2/gen_gta_mel.py \ python3 ${MAIN_ROOT}/paddlespeech/t2s/exps/fastspeech2/gen_gta_mel.py \
--fastspeech2-config=fastspeech2_nosil_baker_ckpt_0.4/default.yaml \ --fastspeech2-config=fastspeech2_nosil_baker_ckpt_0.4/default.yaml \
--fastspeech2-checkpoint=fastspeech2_nosil_baker_ckpt_0.4/snapshot_iter_76000.pdz \ --fastspeech2-checkpoint=fastspeech2_nosil_baker_ckpt_0.4/snapshot_iter_76000.pdz \
--fastspeech2-stat=fastspeech2_nosil_baker_ckpt_0.4/speech_stats.npy \ --fastspeech2-stat=fastspeech2_nosil_baker_ckpt_0.4/speech_stats.npy \
--dur-file=durations.txt \ --dur-file=durations.txt \
--output-dir=dump_finetune \ --output-dir=dump_finetune \
--phones-dict=fastspeech2_nosil_baker_ckpt_0.4/phone_id_map.txt --phones-dict=fastspeech2_nosil_baker_ckpt_0.4/phone_id_map.txt
fi fi
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
python3 local/link_wav.py \ python3 local/link_wav.py \
--old-dump-dir=dump \ --old-dump-dir=dump \
--dump-dir=dump_finetune --dump-dir=dump_finetune
fi fi
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
...@@ -51,13 +50,13 @@ if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then ...@@ -51,13 +50,13 @@ if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
fi fi
if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
CUDA_VISIBLE_DEVICES=${gpus} \ CUDA_VISIBLE_DEVICES=${gpus} \
FLAGS_cudnn_exhaustive_search=true \ FLAGS_cudnn_exhaustive_search=true \
FLAGS_conv_workspace_size_limit=4000 \ FLAGS_conv_workspace_size_limit=4000 \
python ${BIN_DIR}/train.py \ python ${BIN_DIR}/train.py \
--train-metadata=dump_finetune/train/norm/metadata.jsonl \ --train-metadata=dump_finetune/train/norm/metadata.jsonl \
--dev-metadata=dump_finetune/dev/norm/metadata.jsonl \ --dev-metadata=dump_finetune/dev/norm/metadata.jsonl \
--config=conf/finetune.yaml \ --config=conf/finetune.yaml \
--output-dir=exp/finetune \ --output-dir=exp/finetune \
--ngpu=1 --ngpu=1
fi fi
\ No newline at end of file
examples/ljspeech/tts0/README.md
浏览文件 @ 8fd97642
...@@ -12,7 +12,7 @@ tar xjvf LJSpeech-1.1.tar.bz2 ...@@ -12,7 +12,7 @@ tar xjvf LJSpeech-1.1.tar.bz2
Assume the path to the dataset is `~/datasets/LJSpeech-1.1`. Assume the path to the dataset is `~/datasets/LJSpeech-1.1`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize mels. 4. synthesize mels.
```bash ```bash
......
examples/ljspeech/tts1/README.md
浏览文件 @ 8fd97642
...@@ -12,7 +12,7 @@ tar xjvf LJSpeech-1.1.tar.bz2 ...@@ -12,7 +12,7 @@ tar xjvf LJSpeech-1.1.tar.bz2
Assume the path to the dataset is `~/datasets/LJSpeech-1.1`. Assume the path to the dataset is `~/datasets/LJSpeech-1.1`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
examples/ljspeech/tts3/README.md
浏览文件 @ 8fd97642
...@@ -14,7 +14,7 @@ Assume the path to the dataset is `~/datasets/LJSpeech-1.1`. ...@@ -14,7 +14,7 @@ Assume the path to the dataset is `~/datasets/LJSpeech-1.1`.
Assume the path to the MFA result of LJSpeech-1.1 is `./ljspeech_alignment`. Assume the path to the MFA result of LJSpeech-1.1 is `./ljspeech_alignment`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
examples/ljspeech/voc0/README.md
浏览文件 @ 8fd97642
...@@ -13,7 +13,7 @@ Assume the path to the dataset is `~/datasets/LJSpeech-1.1`. ...@@ -13,7 +13,7 @@ Assume the path to the dataset is `~/datasets/LJSpeech-1.1`.
Assume the path to the Tacotron2 generated mels is `../tts0/output/test`. Assume the path to the Tacotron2 generated mels is `../tts0/output/test`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs from mels. 4. synthesize wavs from mels.
```bash ```bash
......
examples/ljspeech/voc1/README.md
浏览文件 @ 8fd97642
...@@ -12,7 +12,7 @@ Assume the path to the dataset is `~/datasets/LJSpeech-1.1`. ...@@ -12,7 +12,7 @@ Assume the path to the dataset is `~/datasets/LJSpeech-1.1`.
Assume the path to the MFA result of LJSpeech-1.1 is `./ljspeech_alignment`. Assume the path to the MFA result of LJSpeech-1.1 is `./ljspeech_alignment`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
examples/vctk/tts3/README.md
浏览文件 @ 8fd97642
...@@ -17,7 +17,7 @@ Assume the path to the dataset is `~/datasets/VCTK-Corpus-0.92`. ...@@ -17,7 +17,7 @@ Assume the path to the dataset is `~/datasets/VCTK-Corpus-0.92`.
Assume the path to the MFA result of VCTK is `./vctk_alignment`. Assume the path to the MFA result of VCTK is `./vctk_alignment`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
examples/vctk/voc1/README.md
浏览文件 @ 8fd97642
...@@ -17,7 +17,7 @@ Assume the path to the dataset is `~/datasets/VCTK-Corpus-0.92`. ...@@ -17,7 +17,7 @@ Assume the path to the dataset is `~/datasets/VCTK-Corpus-0.92`.
Assume the path to the MFA result of VCTK is `./vctk_alignment`. Assume the path to the MFA result of VCTK is `./vctk_alignment`.
Run the command below to Run the command below to
1. **source path**. 1. **source path**.
2. preprocess the dataset, 2. preprocess the dataset.
3. train the model. 3. train the model.
4. synthesize wavs. 4. synthesize wavs.
- synthesize waveform from `metadata.jsonl`. - synthesize waveform from `metadata.jsonl`.
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或