Skip to content

M
models

PaddlePaddle / models
大约 2 年 前同步成功

通知 232
Star 6828
Fork 2962
M
models
“7ba3baf89b21ce6905c7b785c4578b9eed9b762b”上不存在“mobile/src/operators/kernel/fpga/V1/feed_kernel.cpp”
未验证 提交 e47730a0 编写于 作者: F Feiyu Chan 提交者： GitHub
浏览文件
操作

update to 1.6 APIs and fix bugs with tensorboard. (#3663) 

* ini commit for deepvoice, add tensorboard to requirements

* fix urls for code we adapted from

* fix makedirs for python2, fix README

* fix open with encoding for python2 compatability

* fix python2's str(), use encode for unicode, and str() for int

* fix python2 encoding issue, add model architecture and project structure for README

* add model structure, add explanation for hyperparameter priority order.

* fix repitiio n in README_cn, and reorder that in README

* README update; fix integer division issues for python2 compatability

* fix data type for input data, specifier the int type as np.int64 to be platform agnostic

* fix README for preprocess.py, use io.open instead of open for python2 compatability.

* update commanline options, use new save/load API

* fix IO conflict bug for data parallel training

* only construct summary writer in process 0 to further avoid conflict
上级 21354316
显示空白变更内容
内联 并排
Showing with 92 addition and 952 deletion
PaddleSpeech/DeepVoice3/README.md
浏览文件 @ e47730a0
...@@ -62,10 +62,11 @@ There are many hyperparameters to be tuned depending on the specification of mod ...@@ -62,10 +62,11 @@ There are many hyperparameters to be tuned depending on the specification of mod
Note that `preprocess.py`, `train.py` and `synthesis.py` all accept a `--preset` parameter. To ensure consistency, you should use the same preset for preprocessing, training and synthesizing. Note that `preprocess.py`, `train.py` and `synthesis.py` all accept a `--preset` parameter. To ensure consistency, you should use the same preset for preprocessing, training and synthesizing.
Note that you can overwrite preset hyperparameters with command line argument `--hparams`, just pass several key-value pair in `${key}=${value}` format seperated by comma （`,`）. For example `--hparams="batch_size=8, nepochs=500"` can overwrite default values in the preset json file. For more details about hyperparameters, see `hparams.py`, which contains the definition of `hparams`. Priority order of hyperparameters is command line option `--hparams` > `--preset` json configuration file > definition of hparams in `hparams.py`. Note that you can overwrite preset hyperparameters with command line argument `--hparams`, just pass several key-value pair in `${key}=${value}` format seperated by comma （`,`）. For example `--hparams="batch_size=8, nepochs=500"` can overwrite default values in the preset json file.
Some hyperparameters are only related to training, like `batch_size`, `checkpoint_interval` and you can use different values for preprocessing and training. But hyperparameters related to data preprocessing, like `num_mels` and `ref_level_db`, should be kept the same for preprocessing and training. Some hyperparameters are only related to training, like `batch_size`, `checkpoint_interval` and you can use different values for preprocessing and training. But hyperparameters related to data preprocessing, like `num_mels` and `ref_level_db`, should be kept the same for preprocessing and training.
For more details about hyperparameters, see `hparams.py`, which contains the definition of `hparams`. Priority order of hyperparameters is command line option `--hparams` > `--preset` json configuration file > definition of hparams in `hparams.py`.
### Dataset ### Dataset
...@@ -85,7 +86,7 @@ python preprocess.py \ ...@@ -85,7 +86,7 @@ python preprocess.py \
${name} ${in_dir} ${out_dir} ${name} ${in_dir} ${out_dir}
``` ```
Now `${dataset_name}$` only supports `ljspeech`. Support for other datasets is pending. Now `${name}$` only supports `ljspeech`. Support for other datasets is pending.
Assuming that you use `presers/deepvoice3_ljspeech.json` for LJSpeech and the path of the unziped dataset is `~/data/LJSpeech-1.1`, then you can preprocess data with the following command. Assuming that you use `presers/deepvoice3_ljspeech.json` for LJSpeech and the path of the unziped dataset is `~/data/LJSpeech-1.1`, then you can preprocess data with the following command.
...@@ -120,9 +121,9 @@ You can load saved checkpoint and resume training with `--checkpoint`, if you wa ...@@ -120,9 +121,9 @@ You can load saved checkpoint and resume training with `--checkpoint`, if you wa
#### train a part of the model #### train a part of the model
You can also train parts of the model while freezing other parts, by passing `--train-seq2seq-only` or `--train-postnet-only`. When training only parts of the model, other parts should be loaded from saved checkpoints. You can also train parts of the model while freezing other parts, by passing `--train-seq2seq-only` or `--train-postnet-only`. When training only parts of the model, other parts should be loaded from saved checkpoint.
To train only the `seq2seq` or `postnet`, you should load from a whole model with `--checkpoint`and keep the same configurations. Note that when training only the `postnet`, you should set `use_decoder_state_for_postnet_input=false`, because when train only the postnet, the postnet takes the ground truth mel-spectrogram as input. To train only the `seq2seq` or `postnet`, you should load from a whole model with `--checkpoint`and keep the same configurations. Note that when training only the `postnet`, you should set `use_decoder_state_for_postnet_input=false`, because when train only the postnet, the postnet takes the ground truth mel-spectrogram as input. Note that the default value for `use_decoder_state_for_postnet_input` is `True`.
example: example:
...@@ -132,7 +133,7 @@ python train.py --data-root=${data-root} --use-gpu \ ...@@ -132,7 +133,7 @@ python train.py --data-root=${data-root} --use-gpu \
--preset=${preset_json_path} \ --preset=${preset_json_path} \
--hparams="parameters you may want to override" \ --hparams="parameters you may want to override" \
--train-seq2seq-only \ --train-seq2seq-only \
--checkpoint=${path_of_the_saved_model} --output=${directory_to_save_results}
``` ```
### Training on multiple GPUs ### Training on multiple GPUs
...@@ -161,14 +162,14 @@ python -m paddle.distributed.launch \ ...@@ -161,14 +162,14 @@ python -m paddle.distributed.launch \
In the example above, we set only GPU `2, 3, 4, 5` to be visible. Then `--selected_gpus="0, 1, 2, 3"` means the logical ids of the selected gpus, which correpond to GPU `2, 3, 4, 5`. In the example above, we set only GPU `2, 3, 4, 5` to be visible. Then `--selected_gpus="0, 1, 2, 3"` means the logical ids of the selected gpus, which correpond to GPU `2, 3, 4, 5`.
Model checkpoints (directory ending with `.model`) are saved in `./checkpoints` per 10000 steps by default. Layer-wise averaged attention alignments (.png) are saved in `.checkpointys/alignment_ave`. And alignments for each attention layer are saved in `.checkpointys/alignment_layer{attention_layer_num}` per 10000 steps for inspection. Model checkpoints (`*.pdparams` for the model and `*.pdoptim` for the optimizer) are saved in `${directory_to_save_results}/checkpoints` per 10000 steps by default. Layer-wise averaged attention alignments (.png) are saved in `${directory_to_save_results}/checkpoints/alignment_ave`. And alignments for each attention layer are saved in `${directory_to_save_results}/checkpoints/alignment_layer{attention_layer_num}` per 10000 steps for inspection.
Synthesis results of 6 sentences (hardcoded in `eval_model.py`) are saved in `checkpoints/eval`, including `step{step_num}_text{text_id}_single_alignment.png` for averaged alignments and `step{step_num}_text{text_id}_single_predicted.wav` for the predicted waveforms. Synthesis results of 6 sentences (hardcoded in `eval_model.py`) are saved in `checkpoints/eval`, including `step{step_num}_text{text_id}_single_alignment.png` for averaged alignments and `step{step_num}_text{text_id}_single_predicted.wav` for the predicted waveforms.
### Monitor with Tensorboard ### Monitor with Tensorboard
Logs with tensorboard are saved in `./log/${datetime}` directory by default. You can monitor logs by tensorboard. Logs with tensorboard are saved in `${directory_to_save_results}/log/` directory by default. You can monitor logs by tensorboard.
```bash ```bash
tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888 tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888
...@@ -179,9 +180,9 @@ tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888 ...@@ -179,9 +180,9 @@ tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888
Given a list of text, `synthesis.py` synthesize audio signals from a trained model. Given a list of text, `synthesis.py` synthesize audio signals from a trained model.
```bash ```bash
python infer.py --use-gpu --preset=${preset_json_path} \ python synthesis.py --use-gpu --preset=${preset_json_path} \
--hparams="parameters you may want to override" \ --hparams="parameters you may want to override" \
${checkpoint} ${text_list_file} ${dst_dir}} ${checkpoint} ${text_list_file} ${dst_dir}
``` ```
Example test_list.txt: Example test_list.txt:
...@@ -198,7 +199,7 @@ generated waveform files and alignment files are saved in `${dst_dir}`. ...@@ -198,7 +199,7 @@ generated waveform files and alignment files are saved in `${dst_dir}`.
According to [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://arxiv.org/abs/1710.07654), the position rate is different for different datasets. There are 2 position rates, one for the query and the other for the key, which are referred to as $\omega_1$ and $\omega_2$ in th paper, and the corresponding names in preset json are `query_position_rate` and `key_position_rate`. According to [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://arxiv.org/abs/1710.07654), the position rate is different for different datasets. There are 2 position rates, one for the query and the other for the key, which are referred to as $\omega_1$ and $\omega_2$ in th paper, and the corresponding names in preset json are `query_position_rate` and `key_position_rate`.
For example, the `query_position_rate` and `key_position_rate` for LJSpeech are `1.0` and `1.385`, respectively. These values are computed with `compute_timestamp_ratio.py`. Run the command below. For example, the `query_position_rate` and `key_position_rate` for LJSpeech are `1.0` and `1.385`, respectively. These values are computed with `compute_timestamp_ratio.py`. Run the command below, where `${data_root}` means the path of the preprocessed dataset.
```bash ```bash
python compute_timestamp_ratio.py --preset=${preset_json_path} \ python compute_timestamp_ratio.py --preset=${preset_json_path} \
......
PaddleSpeech/DeepVoice3/README_cn.md
浏览文件 @ e47730a0
...@@ -62,7 +62,7 @@ nltk.download("cmudict") ...@@ -62,7 +62,7 @@ nltk.download("cmudict")
`preprocess.py`，`train.py`，`synthesis.py` 都接受 `--preset` 参数。为了保持一致性，最好在数据预处理，模型训练和语音合成时使用相同的预设配置。 `preprocess.py`，`train.py`，`synthesis.py` 都接受 `--preset` 参数。为了保持一致性，最好在数据预处理，模型训练和语音合成时使用相同的预设配置。
可以通过 `--hparams` 参数来覆盖预设的超参数配置，参数格式是逗号分隔的键值对 `${key}=${value}`，例如 `--hparams="batch_size=8, nepochs=500"`。关于超参数设置更多细节可以参考 `hparams.py` ，其中定义了 hparams。超参数的优先级序列是：通过命令行参数 `--hparams` 传入的参数优先级高于通过 `--preset` 参数传入的 json 配置文件，高于 `hparams.py` 中的定义。 可以通过 `--hparams` 参数来覆盖预设的超参数配置，参数格式是逗号分隔的键值对 `${key}=${value}`，例如 `--hparams="batch_size=8, nepochs=500"`。
部分参数可以只和训练有关，如 `batch_size`, `checkpoint_interval`, 用户在训练时可以使用不同的值。但部分参数和数据预处理相关，如 `num_mels` 和 `ref_level_db`, 这些参数在数据预处理和训练时候应该保持一致。 部分参数可以只和训练有关，如 `batch_size`, `checkpoint_interval`, 用户在训练时可以使用不同的值。但部分参数和数据预处理相关，如 `num_mels` 和 `ref_level_db`, 这些参数在数据预处理和训练时候应该保持一致。
...@@ -86,7 +86,7 @@ python preprocess.py \ ...@@ -86,7 +86,7 @@ python preprocess.py \
${name} ${in_dir} ${out_dir} ${name} ${in_dir} ${out_dir}
``` ```
目前 `${dataset_name}$` 只支持 `ljspeech`。未来将会支持更多数据集。 目前 `${name}$` 只支持 `ljspeech`。未来将会支持更多数据集。
假设你使用 `presers/deepvoice3_ljspeech.json` 作为处理 LJSpeech 的预设配置文件，并且解压后的数据集位于 `~/data/LJSpeech-1.1`, 那么使用如下的命令进行数据预处理。 假设你使用 `presers/deepvoice3_ljspeech.json` 作为处理 LJSpeech 的预设配置文件，并且解压后的数据集位于 `~/data/LJSpeech-1.1`, 那么使用如下的命令进行数据预处理。
...@@ -123,7 +123,7 @@ python train.py --data-root=${data-root} --use-gpu \ ...@@ -123,7 +123,7 @@ python train.py --data-root=${data-root} --use-gpu \
用户可以通过 `--train-seq2seq-only` 或者 `--train-postnet-only` 来实现固定模型的其他部分，只训练需要训练的部分。但当只训练模型的一部分时，其他的部分需要从保存的模型中加载。 用户可以通过 `--train-seq2seq-only` 或者 `--train-postnet-only` 来实现固定模型的其他部分，只训练需要训练的部分。但当只训练模型的一部分时，其他的部分需要从保存的模型中加载。
当只训练模型的 `seq2seq` 部分或者 `postnet` 部分时，需要使用 `--checkpoint` 加载整个模型并保持相同的配置。注意，当只训练 `postnet` 的时候，需要保证配置中的`use_decoder_state_for_postnet_input=false`，因为在这种情况下，postnet 使用真实的 mel 频谱作为输入。 当只训练模型的 `seq2seq` 部分或者 `postnet` 部分时，需要使用 `--checkpoint` 加载整个模型并保持相同的配置。注意，当只训练 `postnet` 的时候，需要保证配置中的`use_decoder_state_for_postnet_input=false`，因为在这种情况下，postnet 使用真实的 mel 频谱作为输入。注意，`use_decoder_state_for_postnet_input` 的默认值是 `True`。
示例: 示例:
...@@ -133,7 +133,7 @@ python train.py --data-root=${data-root} --use-gpu \ ...@@ -133,7 +133,7 @@ python train.py --data-root=${data-root} --use-gpu \
--preset=${preset_json_path} \ --preset=${preset_json_path} \
--hparams="parameters you may want to override" \ --hparams="parameters you may want to override" \
--train-seq2seq-only \ --train-seq2seq-only \
--checkpoint=${path_of_the_saved_model} --output=${directory_to_save_results}
``` ```
### 使用 GPU 多卡训练 ### 使用 GPU 多卡训练
...@@ -144,7 +144,7 @@ python train.py --data-root=${data-root} --use-gpu \ ...@@ -144,7 +144,7 @@ python train.py --data-root=${data-root} --use-gpu \
python -m paddle.distributed.launch \ python -m paddle.distributed.launch \
--started_port ${port_of_the_first_worker} \ --started_port ${port_of_the_first_worker} \
--selected_gpus ${logical_gpu_ids_to_choose} \ --selected_gpus ${logical_gpu_ids_to_choose} \
--log_dir ${path_of_write_log} \ --log_dir ${path_to_write_log} \
training_script ... training_script ...
``` ```
...@@ -157,19 +157,20 @@ python -m paddle.distributed.launch \ ...@@ -157,19 +157,20 @@ python -m paddle.distributed.launch \
train.py --data-root=${data-root} \ train.py --data-root=${data-root} \
--use-gpu --use-data-parallel \ --use-gpu --use-data-parallel \
--preset=${preset_json_path} \ --preset=${preset_json_path} \
--hparams="parameters you may want to override" --hparams="parameters you may want to override" \
--output=${directory_to_save_results}
``` ```
上述的示例中，设置了 `2, 3, 4, 5` 号显卡为可见的 GPU。然后 `--selected_gpus=0,1,2,3` 选择的是 GPU 的逻辑序号，分别对应于 `2, 3, 4, 5` 号卡。 上述的示例中，设置了 `2, 3, 4, 5` 号显卡为可见的 GPU。然后 `--selected_gpus=0,1,2,3` 选择的是 GPU 的逻辑序号，分别对应于 `2, 3, 4, 5` 号卡。
模型默认被保存为后缀为 `.model`的文件夹，保存在 `./checkpoints` 文件夹中。多层平均的注意力机制对齐结果被保存为 `.png` 图片，默认保存在 `.checkpointys/alignment_ave` 中。每一层的注意力机制对齐结果默认被保存在 `.checkpointys/alignment_layer{attention_layer_num}`文件夹中。默认每 10000 步保存一次用于查看。 模型默认被保存为后缀为 `.model`的文件夹，保存在 `${directory_to_save_results}/checkpoints` 文件夹中。多层平均的注意力机制对齐结果被保存为 `.png` 图片，默认保存在 `${directory_to_save_results}/checkpoints/alignment_ave` 中。每一层的注意力机制对齐结果默认被保存在 `${directory_to_save_results}/checkpoints/alignment_layer{attention_layer_num}`文件夹中。默认每 10000 步保存一次用于查看。
对 6 个给定的句子的语音合成结果保存在 `checkpoints/eval` 中，包含多层平均平均的注意力机制对齐结果，这被保存为名为 `step{step_num}_text{text_id}_single_alignment.png` 的图片；以及合成的音频文件，保存为名为 `step{step_num}_text{text_id}_single_predicted.wav` 的音频。 对 6 个给定的句子的语音合成结果保存在 `checkpoints/eval` 中，包含多层平均平均的注意力机制对齐结果，这被保存为名为 `step{step_num}_text{text_id}_single_alignment.png` 的图片；以及合成的音频文件，保存为名为 `step{step_num}_text{text_id}_single_predicted.wav` 的音频。
### 使用 Tensorboard 查看训练 ### 使用 Tensorboard 查看训练
Tensorboard 训练日志默认被保存在 `./log/${datetime}` 文件夹，可以通过 tensorboard 查看。使用方法如下。 Tensorboard 训练日志被保存在 `${directory_to_save_results}/log/` 文件夹，可以通过 tensorboard 查看。使用方法如下。
```bash ```bash
tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888 tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888
...@@ -180,9 +181,9 @@ tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888 ...@@ -180,9 +181,9 @@ tensorboard --logdir=${log_dir} --host=$HOSTNAME --port=8888
给定一组文本，使用 `synthesis.py` 从一个训练好的模型来合成语音，使用方法如下。 给定一组文本，使用 `synthesis.py` 从一个训练好的模型来合成语音，使用方法如下。
```bash ```bash
python infer.py --use-gpu --preset=${preset_json_path} \ python synthesis.py --use-gpu --preset=${preset_json_path} \
--hparams="parameters you may want to override" \ --hparams="parameters you may want to override" \
${checkpoint} ${text_list_file} ${dst_dir}} ${checkpoint} ${text_list_file} ${dst_dir}
``` ```
示例文本文件如下: 示例文本文件如下:
...@@ -199,7 +200,7 @@ A text-to-speech synthesis system typically consists of multiple stages, such as ...@@ -199,7 +200,7 @@ A text-to-speech synthesis system typically consists of multiple stages, such as
根据 [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://arxiv.org/abs/1710.07654), 对于不同的数据集，会有不同的 position rate. 有两个不同的 position rate，一个用于 query 一个用于 key， 这在论文中称为 $\omega_1$ 和 $\omega_2$ ，在预设配置文件中的名字分别为 `query_position_rate` 和 `key_position_rate`。 根据 [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://arxiv.org/abs/1710.07654), 对于不同的数据集，会有不同的 position rate. 有两个不同的 position rate，一个用于 query 一个用于 key， 这在论文中称为 $\omega_1$ 和 $\omega_2$ ，在预设配置文件中的名字分别为 `query_position_rate` 和 `key_position_rate`。
比如 LJSpeech 数据集的 `query_position_rate` 和 `key_position_rate` 分别为 `1.0` 和 `1.385`。这些值可以 `compute_timestamp_ratio.py`。使用如下命令计算。 比如 LJSpeech 数据集的 `query_position_rate` 和 `key_position_rate` 分别为 `1.0` 和 `1.385`。这些值可以使用 `compute_timestamp_ratio.py` 计算，命令如下，其中 `${data_root}` 是预处理后的数据集路径。
```bash ```bash
python compute_timestamp_ratio.py --preset=${preset_json_path} \ python compute_timestamp_ratio.py --preset=${preset_json_path} \
......
PaddleSpeech/DeepVoice3/compute_timestamp_ratio.py
浏览文件 @ e47730a0
# Part of code was adpated from https://github.com/r9y9/deepvoice3_pytorch/tree/master/compute_timestamp_ratio.py # Part of code was adpated from https://github.com/r9y9/deepvoice3_pytorch/tree/master/compute_timestamp_ratio.py
# Copyright (c) 2017: Ryuichi Yamamoto. # Copyright (c) 2017: Ryuichi Yamamoto.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import argparse import argparse
import sys import sys
import io
import numpy as np import numpy as np
from hparams import hparams, hparams_debug_string from hparams import hparams, hparams_debug_string
from deepvoice3_paddle.data import TextDataSource, MelSpecDataSource from deepvoice3_paddle.data import TextDataSource, MelSpecDataSource
...@@ -34,7 +39,7 @@ if __name__ == "__main__": ...@@ -34,7 +39,7 @@ if __name__ == "__main__":
# Load preset if specified # Load preset if specified
if preset is not None: if preset is not None:
with open(preset) as f: with io.open(preset) as f:
hparams.parse_json(f.read()) hparams.parse_json(f.read())
# Override hyper parameters # Override hyper parameters
hparams.parse(args.hparams) hparams.parse(args.hparams)
......
PaddleSpeech/DeepVoice3/conversion/README.md 已删除 100644 → 0
浏览文件 @ 21354316
# Parameter conversion
## generate name map
To convert a model trained with `https://github.com/r9y9/deepvoice3_pytorch`, we provide a script to generate name map between pytorch model and paddle model for `deepvoice3`. You can provide `--preset` and `--hparams` to specify the model's configuration.
```bash
python generate_name_map.py --preset=${preset_to_use} --haprams="hyper parameters to overwrite"
```
It would print a name map. The format of the name map file looks like this. Each line consists of 3 fields, the first is the name of a parameter in the saved state dict of pytorch model, the second and third is the name and shape of the corresponding parameter in the saved state dict of paddle.
```
seq2seq.encoder.embed_tokens.weight encoder/Encoder_0/Embedding_0.w_0 [149, 256]
seq2seq.encoder.convolutions.0.bias encoder/Encoder_0/ConvProj1D_1/Conv2D_0.b_0 [512]
seq2seq.encoder.convolutions.0.weight_g encoder/Encoder_0/ConvProj1D_1/Conv2D_0.w_1 [512]
```
Redirect the output to a file to save it.
```bash
python generate_name_map.py --preset=${preset_to_use} --haprams="hyper parameters to overwrite" > name_map.txt
```
## convert saved pytorch model to paddle model
Given a name map and a saved pytorch model, you can convert it to a paddle model.
```bash
python convert.py \
--pytorch-model ${pytorch_model.pth} \
--paddle-model ${path_to_save_paddle_model} \
--name-map ${name_map_path}
```
Note that the user should provide the name map file, and ensure the models are equivalent to each other and corresponding parameters have the right shapes.
PaddleSpeech/DeepVoice3/conversion/convert.py 已删除 100644 → 0
浏览文件 @ 21354316
# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import numpy as np
import torch
import paddle
from paddle import fluid
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
"--pytorch-model",
dest='pytorch_model',
type=str,
help="The source pytorch mode.")
parser.add_argument(
"--paddle-model",
dest='paddle_model',
type=str,
help="The directory to save paddle model, now saves model as a folder.")
parser.add_argument(
"--name-map",
dest="name_map",
type=str,
help="name mapping for the source model and the target model.")
def read_name_map(fname):
"""
There should be a 3-column file.
The first comuln is the name of parameter in pytorch model's state dict;
The second column is the name of parameter in paddle model's state dict;
The third column is the shape of the repective parameter in paddle model.
"""
name_map = {}
with open(fname, 'rt') as f:
for line in f:
src_key, tgt_key, tgt_shape = line.strip().split('\t')
tgt_shape = eval(tgt_shape)
name_map[src_key] = (tgt_key, tgt_shape)
return name_map
def torch2paddle(state_dict, name_map, dirname):
"""
state_dict: pytorch model's state dict.
name_map: a text file for name mapping from pytorch model to paddle model.
dirname: path of the paddle model to save.
"""
program = fluid.Program()
global_block = program.global_block()
for k in state_dict.keys():
global_block.create_parameter(
name=name_map[k][0],
shape=[1],
dtype='float32',
initializer=fluid.initializer.Constant(value=0.0))
place = fluid.core.CPUPlace()
exe = fluid.Executor(place)
exe.run(fluid.default_startup_program())
exe.run(program)
# NOTE: transpose the pytorch model's parameter if neccessary
# we do not transpose here because we used conv instead of FC layer to replace Linear in pytorch,
# which does not need us to transpose the paramerters.
# but when you use a FC layer corresponding a torch Linear module, be sure to transpose the weight.
# Other transformations are not concerned, but users should check the data shape to ensure that
# the transformations are what's expected.
for k, v in state_dict.items():
fluid.global_scope().find_var(name_map[k][0]).get_tensor().set(
v.cpu().numpy().reshape(name_map[k][1]), place)
fluid.io.save_params(exe, dirname, main_program=program)
if __name__ == "__main__":
args, _ = parser.parse_known_args()
result = torch.load(args.pytorch_model)
state_dict = result["state_dict"]
name_map = read_name_map(args.name_map)
torch2paddle(state_dict, name_map, args.paddle_model)
PaddleSpeech/DeepVoice3/conversion/generate_name_map.py 已删除 100644 → 0
浏览文件 @ 21354316
# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Generate a name map from configuration
"""
from collections import OrderedDict
import numpy as np
from argparse import ArgumentParser
from deepvoice3_paddle import frontend
from hparams import hparams, hparams_debug_string
def build_arg_parser():
parser = ArgumentParser(description="Train deepvoice 3 model.")
parser.add_argument(
"--preset",
type=str,
required=True,
help="Path of preset parameters in json format.")
parser.add_argument(
"--hparams",
type=str,
default="",
help="Hyper parameters to override preset.")
return parser
def gen_conv(in_channels, out_channels, kernel_size, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name):
# bias
torch_bias_name = "{}.bias".format(torch_base_name)
paddle_bias_name = "{}.b_0".format(paddle_base_name)
torch_bias_shape = (out_channels, )
paddle_bias_shape = [out_channels, ]
torch_state[torch_bias_name] = torch_bias_shape
paddle_state[paddle_bias_name] = paddle_bias_shape
name_map[torch_bias_name] = paddle_bias_name
# wg
torch_wg_name = "{}.weight_g".format(torch_base_name)
paddle_wg_name = "{}.w_1".format(paddle_base_name)
torch_wg_shape = (out_channels, 1, 1)
paddle_wg_shape = [out_channels, ]
torch_state[torch_wg_name] = torch_wg_shape
paddle_state[paddle_wg_name] = paddle_wg_shape
name_map[torch_wg_name] = paddle_wg_name
# wv
torch_wv_name = "{}.weight_v".format(torch_base_name)
paddle_wv_name = "{}.w_0".format(paddle_base_name)
torch_wv_shape = (out_channels, in_channels, kernel_size)
paddle_wv_shape = [out_channels, in_channels, 1, kernel_size]
torch_state[torch_wv_name] = torch_wv_shape
paddle_state[paddle_wv_name] = paddle_wv_shape
name_map[torch_wv_name] = paddle_wv_name
def gen_fc2conv(in_channels, out_channels, torch_state, paddle_state, name_map,
torch_base_name, paddle_base_name):
# bias
torch_bias_name = "{}.bias".format(torch_base_name)
paddle_bias_name = "{}.b_0".format(paddle_base_name)
torch_bias_shape = (out_channels, )
paddle_bias_shape = [out_channels, ]
torch_state[torch_bias_name] = torch_bias_shape
paddle_state[paddle_bias_name] = paddle_bias_shape
name_map[torch_bias_name] = paddle_bias_name
# wg
torch_wg_name = "{}.weight_g".format(torch_base_name)
paddle_wg_name = "{}.w_1".format(paddle_base_name)
torch_wg_shape = (out_channels, 1)
paddle_wg_shape = [out_channels, ]
torch_state[torch_wg_name] = torch_wg_shape
paddle_state[paddle_wg_name] = paddle_wg_shape
name_map[torch_wg_name] = paddle_wg_name
# wv
torch_wv_name = "{}.weight_v".format(torch_base_name)
paddle_wv_name = "{}.w_0".format(paddle_base_name)
torch_wv_shape = (out_channels, in_channels)
paddle_wv_shape = [out_channels, in_channels, 1, 1]
torch_state[torch_wv_name] = torch_wv_shape
paddle_state[paddle_wv_name] = paddle_wv_shape
name_map[torch_wv_name] = paddle_wv_name
def generate_name_map(name_scope):
TTS_model_idx = 0
prefix = "/".join([name_scope, "DeepVoiceTTS_{}".format(TTS_model_idx)])
_frontend = getattr(frontend, hparams.frontend)
torch_state = OrderedDict()
paddle_state = OrderedDict()
name_map = OrderedDict()
# text embedding
torch_name = "seq2seq.encoder.embed_tokens.weight"
paddle_name = "{}/ConvS2S_0/Encoder_0/Embedding_0.w_0".format(prefix)
torch_shape = (_frontend.n_vocab, hparams.text_embed_dim)
paddle_shape = [_frontend.n_vocab, hparams.text_embed_dim]
torch_state[torch_name] = torch_shape
paddle_state[paddle_name] = paddle_shape
name_map[torch_name] = paddle_name
# encoder
Conv1D_idx = 0
Conv1DGLU_idx = 0
if hparams.n_speakers > 1:
for i in [1, 2]:
torch_base_name = "seq2seq.encoder.speaker_fc{}".format(i)
paddle_base_name = "{}/ConvS2S_0/Encoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
Conv1D_idx += 1
gen_fc2conv(hparams.speaker_embed_dim, hparams.text_embed_dim,
torch_state, paddle_state, name_map, torch_base_name,
paddle_base_name)
# encoder convolution specs
encoder_channels = hparams.encoder_channels
kernel_size = hparams.kernel_size
h = encoder_channels
k = kernel_size
convolutions = [(h, k, 1), (h, k, 3), (h, k, 9), (h, k, 27), (h, k, 1),
(h, k, 3), (h, k, 9), (h, k, 27), (h, k, 1), (h, k, 3)]
torch_layer_idx = 0
in_channels = hparams.text_embed_dim
# first conv1d & conv1dglus
for out_channels, kernel_size, dilation in convolutions:
if in_channels != out_channels:
torch_base_name = "seq2seq.encoder.convolutions.{}".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Encoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_channels, out_channels, 1, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 2
Conv1D_idx += 1
in_channels = out_channels
torch_base_name = "seq2seq.encoder.convolutions.{}.conv".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Encoder_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * out_channels, kernel_size, torch_state,
paddle_state, name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "seq2seq.encoder.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Encoder_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, out_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
in_channels = out_channels
# last conv1d
torch_base_name = "seq2seq.encoder.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Encoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_channels, hparams.text_embed_dim, 1, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 2
Conv1D_idx += 1
in_channels = out_channels
# decoder
# position embeddings
PositionEmbedding_idx = 0
torch_name = "seq2seq.decoder.embed_query_positions.weight"
paddle_name = "{}/ConvS2S_0/Decoder_0/PositionEmbedding_{}/Embedding_0.w_0".format(
prefix, PositionEmbedding_idx)
torch_shape = (hparams.max_positions, hparams.decoder_channels)
paddle_shape = [hparams.max_positions, hparams.decoder_channels]
PositionEmbedding_idx += 1
torch_state[torch_name] = torch_shape
paddle_state[paddle_name] = paddle_shape
name_map[torch_name] = paddle_name
torch_name = "seq2seq.decoder.embed_keys_positions.weight"
paddle_name = "{}/ConvS2S_0/Decoder_0/PositionEmbedding_{}/Embedding_0.w_0".format(
prefix, PositionEmbedding_idx)
PositionEmbedding_idx += 1
torch_shape = (hparams.max_positions, hparams.text_embed_dim)
paddle_shape = [hparams.max_positions, hparams.text_embed_dim]
torch_state[torch_name] = torch_shape
paddle_state[paddle_name] = paddle_shape
name_map[torch_name] = paddle_name
Conv1D_idx = 0
Conv1DGLU_idx = 0
if hparams.n_speakers > 1:
for i in [1, 2]:
torch_base_name = "seq2seq.decoder.speaker_proj{}".format(i)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
Conv1D_idx += 1
gen_fc2conv(hparams.speaker_embed_dim, 1, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
# prenet
torch_layer_idx = 0
h = hparams.decoder_channels
k = hparams.kernel_size
prenet_convolutions = [(h, k, 1), (h, k, 3)]
in_channels = hparams.num_mels * hparams.outputs_per_step
for out_channels, kernel_size, dilation in prenet_convolutions:
if in_channels != out_channels:
torch_base_name = "seq2seq.decoder.preattention.{}".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_channels, out_channels, 1, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 2
Conv1D_idx += 1
in_channels = out_channels
torch_base_name = "seq2seq.decoder.preattention.{}.conv".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * out_channels, kernel_size, torch_state,
paddle_state, name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "seq2seq.decoder.preattention.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, out_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
in_channels = out_channels
# conv & attn
torch_layer_idx = 0
convolutions = [(h, k, 1), (h, k, 3), (h, k, 9), (h, k, 27), (h, k, 1)]
for out_channels, kernel_size, dilation in convolutions:
if in_channels != out_channels:
torch_base_name = "seq2seq.decoder.convolutions.{}".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_channels, out_channels, kernel_size, in_channels,
out_channels, kernel_size, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 2
Conv1D_idx += 1
in_channels = out_channels
torch_base_name = "seq2seq.decoder.convolutions.{}.conv".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * out_channels, kernel_size, torch_state,
paddle_state, name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "seq2seq.decoder.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, out_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
in_channels = out_channels
# attention
attention = [True, False, False, False, True]
AttentionLayer_idx = 0
parts = [
"query", "key" if hparams.key_projection else None, "value"
if hparams.value_projection else None, "out"
]
parts = [x for x in parts if x is not None]
for (i, (attn, (out_channels, kernel_size,
dilation))) in enumerate(zip(attention, convolutions)):
if attn is False:
in_channels = out_channels
continue
for ipart, part in enumerate(parts):
torch_base_name = "seq2seq.decoder.attention.{}.{}_projection".format(
i, part)
paddle_base_name = "{}/ConvS2S_0/Decoder_0/AttentionLayer_{}/Conv1D_{}/Conv2D_0".format(
prefix, AttentionLayer_idx, ipart)
if part == "query":
C_in = out_channels
C_out = hparams.text_embed_dim
elif part == "out":
C_in = hparams.text_embed_dim
C_out = out_channels
else:
C_in = hparams.text_embed_dim
C_out = hparams.text_embed_dim
gen_fc2conv(C_in, C_out, torch_state, paddle_state, name_map,
torch_base_name, paddle_base_name)
in_channels = out_channels
AttentionLayer_idx += 1
# last conv
torch_base_name = "seq2seq.decoder.last_conv"
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_channels, hparams.num_mels * hparams.outputs_per_step, 1,
torch_state, paddle_state, name_map, torch_base_name,
paddle_base_name)
Conv1D_idx += 1
# for done output
torch_base_name = "seq2seq.decoder.fc"
paddle_base_name = "{}/ConvS2S_0/Decoder_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_fc2conv(hparams.num_mels * hparams.outputs_per_step, 1, torch_state,
paddle_state, name_map, torch_base_name, paddle_base_name)
# converter
# time_upsampling
if hparams.use_decoder_state_for_postnet_input:
in_dim = hparams.decoder_channels // hparams.outputs_per_step
else:
in_dim = hparams.num_mels
h = hparams.converter_channels
time_upsampling = max(hparams.downsample_step // hparams.outputs_per_step,
1)
assert time_upsampling == hparams.downsample_step, "implementation difference occured"
assert time_upsampling in [1, 2, 4], "other values not supported yet"
torch_layer_idx = 0
Conv1D_idx = 0
Conv1DGLU_idx = 0
Conv1DTranspose_idx = 0
h = hparams.converter_channels
k = hparams.kernel_size
postnet_convolutions = [(h, k, 1), (h, k, 3), (2 * h, k, 1), (2 * h, k, 3)]
in_channels = postnet_convolutions[0][0]
if time_upsampling == 4:
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_dim, in_channels, 1, torch_state, paddle_state, name_map,
torch_base_name, paddle_base_name)
torch_layer_idx += 1
Conv1D_idx += 1
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DTranspose_{}/Conv2DTranspose_0".format(
prefix, Conv1DTranspose_idx)
gen_conv(in_channels, in_channels, 2, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 1
Conv1DTranspose_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DTranspose_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DTranspose_{}/Conv2DTranspose_0".format(
prefix, Conv1DTranspose_idx)
gen_conv(in_channels, in_channels, 2, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 1
Conv1DTranspose_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
elif time_upsampling == 2:
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_dim, in_channels, 1, torch_state, paddle_state, name_map,
torch_base_name, paddle_base_name)
torch_layer_idx += 1
Conv1D_idx += 1
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DTranspose_{}/Conv2DTranspose_0".format(
prefix, Conv1DTranspose_idx)
gen_conv(in_channels, in_channels, 2, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 1
Conv1DTranspose_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DTranspose_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
else:
assert time_upsampling == 1, "other values are not supported"
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_dim, in_channels, 1, torch_state, paddle_state, name_map,
torch_base_name, paddle_base_name)
torch_layer_idx += 1
Conv1D_idx += 1
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * in_channels, 3, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, in_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
for (out_channels, kernel_size, dilation) in postnet_convolutions:
if in_channels != out_channels:
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1D_{}/Conv2D_0".format(
prefix, Conv1D_idx)
gen_conv(in_channels, out_channels, 1, torch_state, paddle_state,
name_map, torch_base_name, paddle_base_name)
torch_layer_idx += 2
Conv1D_idx += 1
in_channels = out_channels
torch_base_name = "postnet.convolutions.{}.conv".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_0/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_conv(in_channels, 2 * out_channels, kernel_size, torch_state,
paddle_state, name_map, torch_base_name, paddle_base_name)
if hparams.n_speakers > 1:
torch_base_name = "postnet.convolutions.{}.speaker_proj".format(
torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1DGLU_{}/Conv1D_1/Conv2D_0".format(
prefix, Conv1DGLU_idx)
gen_fc2conv(hparams.speaker_embed_dim, out_channels, torch_state,
paddle_state, name_map, torch_base_name,
paddle_base_name)
torch_layer_idx += 1
Conv1DGLU_idx += 1
in_channels = out_channels
# last conv
linear_dim = hparams.fft_size // 2 + 1
torch_base_name = "postnet.convolutions.{}".format(torch_layer_idx)
paddle_base_name = "{}/Converter_0/Conv1D_{}/Conv2D_0".format(prefix,
Conv1D_idx)
gen_conv(in_channels, linear_dim, 1, torch_state, paddle_state, name_map,
torch_base_name, paddle_base_name)
torch_layer_idx += 2
Conv1D_idx += 1
in_channels = out_channels
# speaker_embed
if hparams.n_speakers > 1:
torch_name = "embed_speakers.weight"
torch_shape = (hparams.n_speakers, hparams.speaker_embed_dim)
paddle_name = "{}/Embedding_0.w_0".format(prefix)
paddle_shape = [hparams.n_speakers, hparams.speaker_embed_dim]
torch_state[torch_name] = torch_shape
paddle_state[paddle_name] = paddle_shape
name_map[torch_name] = paddle_name
# for k, v in torch_state.items():
# print("{}\t{}".format(k, v))
# for k, v in paddle_state.items():
# print("{}\t{}".format(k, v))
for k in name_map:
assert np.prod(torch_state[k]) == np.prod(paddle_state[name_map[
k]]), "{} does not match".format(k)
print("{}\t{}\t{}".format(k, name_map[k], paddle_state[name_map[k]]))
if __name__ == "__main__":
parser = build_arg_parser()
args, _ = parser.parse_known_args()
if args.preset is not None:
with open(args.preset) as f:
hparams.parse_json(f.read())
# Override hyper parameters
hparams.parse(args.hparams)
# print(hparams_debug_string())
generate_name_map("dv3")
PaddleSpeech/DeepVoice3/deepvoice3_paddle/data.py
浏览文件 @ e47730a0
...@@ -15,6 +15,7 @@ ...@@ -15,6 +15,7 @@
import numpy as np import numpy as np
import random import random
import io
import platform import platform
from os.path import dirname, join from os.path import dirname, join
...@@ -52,15 +53,14 @@ class TextDataSource(FileDataSource): ...@@ -52,15 +53,14 @@ class TextDataSource(FileDataSource):
def collect_files(self): def collect_files(self):
meta = join(self.data_root, "train.txt") meta = join(self.data_root, "train.txt")
with open(meta, "rb") as f: with io.open(meta, "rt", encoding="utf-8") as f:
lines = f.readlines() lines = f.readlines()
l = lines[0].decode("utf-8").split("|") l = lines[0].split("|")
assert len(l) == 4 or len(l) == 5 assert len(l) == 4 or len(l) == 5
self.multi_speaker = len(l) == 5 self.multi_speaker = len(l) == 5
texts = list(map(lambda l: l.decode("utf-8").split("|")[3], lines)) texts = list(map(lambda l: l.split("|")[3], lines))
if self.multi_speaker: if self.multi_speaker:
speaker_ids = list( speaker_ids = list(map(lambda l: int(l.split("|")[-1]), lines))
map(lambda l: int(l.decode("utf-8").split("|")[-1]), lines))
# Filter by speaker_id # Filter by speaker_id
# using multi-speaker dataset as a single speaker dataset # using multi-speaker dataset as a single speaker dataset
if self.speaker_id is not None: if self.speaker_id is not None:
...@@ -106,21 +106,18 @@ class _NPYDataSource(FileDataSource): ...@@ -106,21 +106,18 @@ class _NPYDataSource(FileDataSource):
def collect_files(self): def collect_files(self):
meta = join(self.data_root, "train.txt") meta = join(self.data_root, "train.txt")
with open(meta, "rb") as f: with io.open(meta, "rt", encoding="utf-8") as f:
lines = f.readlines() lines = f.readlines()
l = lines[0].decode("utf-8").split("|") l = lines[0].split("|")
assert len(l) == 4 or len(l) == 5 assert len(l) == 4 or len(l) == 5
multi_speaker = len(l) == 5 multi_speaker = len(l) == 5
self.frame_lengths = list( self.frame_lengths = list(map(lambda l: int(l.split("|")[2]), lines))
map(lambda l: int(l.decode("utf-8").split("|")[2]), lines))
paths = list( paths = list(map(lambda l: l.split("|")[self.col], lines))
map(lambda l: l.decode("utf-8").split("|")[self.col], lines))
paths = list(map(lambda f: join(self.data_root, f), paths)) paths = list(map(lambda f: join(self.data_root, f), paths))
if multi_speaker and self.speaker_id is not None: if multi_speaker and self.speaker_id is not None:
speaker_ids = list( speaker_ids = list(map(lambda l: int(l.split("|")[-1]), lines))
map(lambda l: int(l.decode("utf-8").split("|")[-1]), lines))
# Filter by speaker_id # Filter by speaker_id
# using multi-speaker dataset as a single speaker dataset # using multi-speaker dataset as a single speaker dataset
indices = np.array(speaker_ids) == self.speaker_id indices = np.array(speaker_ids) == self.speaker_id
...@@ -297,7 +294,7 @@ def create_batch(batch): ...@@ -297,7 +294,7 @@ def create_batch(batch):
# text positions # text positions
text_positions = np.array( text_positions = np.array(
[_pad(np.arange(1, len(x[0]) + 1), max_input_len) for x in batch], [_pad(np.arange(1, len(x[0]) + 1), max_input_len) for x in batch],
dtype=np.int) dtype=np.int64)
text_positions = np.expand_dims(text_positions, axis=-1) text_positions = np.expand_dims(text_positions, axis=-1)
max_decoder_target_len = max_target_len // r // downsample_step max_decoder_target_len = max_target_len // r // downsample_step
...@@ -306,7 +303,8 @@ def create_batch(batch): ...@@ -306,7 +303,8 @@ def create_batch(batch):
s, e = 1, max_decoder_target_len + 1 s, e = 1, max_decoder_target_len + 1
frame_positions = np.tile( frame_positions = np.tile(
np.expand_dims( np.expand_dims(
np.arange(s, e), axis=0), (len(batch), 1)) np.arange(
s, e, dtype=np.int64), axis=0), (len(batch), 1))
frame_positions = np.expand_dims(frame_positions, axis=-1) frame_positions = np.expand_dims(frame_positions, axis=-1)
# done flags # done flags
......
PaddleSpeech/DeepVoice3/deepvoice3_paddle/deepvoice3.py
浏览文件 @ e47730a0
...@@ -219,7 +219,6 @@ class Encoder(dg.Layer): ...@@ -219,7 +219,6 @@ class Encoder(dg.Layer):
values (Variable), Shape(B, C_embed, 1, T_enc), the encoded values (Variable), Shape(B, C_embed, 1, T_enc), the encoded
representation for values. representation for values.
""" """
x = self.embed(x) x = self.embed(x)
x = fluid.layers.dropout( x = fluid.layers.dropout(
......
PaddleSpeech/DeepVoice3/deepvoice3_paddle/save_load.py 已删除 100644 → 0
浏览文件 @ 21354316
# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
from os.path import dirname, join
import paddle
from paddle import fluid
import paddle.fluid.dygraph as dg
def _load(checkpoint_path):
"""
Load saved state dict and optimizer state(optional).
"""
state_dict, optimizer_state = dg.load_persistables(dirname=checkpoint_path)
return state_dict, optimizer_state
def load_checkpoint(path, model, optimizer=None, reset_optimizer=True):
"""
layers like FC, Conv*, ... the Layer does not initialize their parameters
before first run.
1. if you want to load only a part of a saved whole model, to part of an
existing model, just pass the part as the target model , and path of the
saved whole model as source path.
2. if you want to load exactly from what is saved, just passed the model
and path as expected.
The rule of thumb is:
1. loading to a model works with name, a unique global name.
2. loading from a directory works with file structure, each parameter is
saved in a file. Loading a file from directory A/ would `create` a
corresponding Variable for each saved parameter, whose name is the file's
relative path from directory A/.
"""
print("Load checkpoint from: {}".format(path))
state_dict, optimizer_state = _load(path)
model.load_dict(state_dict)
if not reset_optimizer and optimizer is not None:
if optimizer_state is not None:
print("[loading] Load optimizer state from {}".format(path))
optimizer.load(optimizer_state)
return model
def _load_embedding(path, model):
print("[loading] Loading embedding from {}".format(path))
state_dict, optimizer_state = _load(path)
key = os.path.join(model.full_name(), "ConvS2S_0/Encoder_0/Embedding_0.w_0")
tensor = model.state_dict()[key]._ivar.value().get_tensor()
tensor.set(state_dict[key], fluid.framework._current_expected_place())
def save_checkpoint(model, optimizer, checkpoint_dir, global_step):
checkpoint_path = join(checkpoint_dir,
"checkpoint_step{:09d}.model".format(global_step))
dg.save_persistables(
model.state_dict(), dirname=checkpoint_path, optimizers=optimizer)
print("[checkpoint] Saved checkpoint:", checkpoint_path)
PaddleSpeech/DeepVoice3/eval_model.py
浏览文件 @ e47730a0
...@@ -23,7 +23,7 @@ from warnings import warn ...@@ -23,7 +23,7 @@ from warnings import warn
from datetime import datetime from datetime import datetime
import matplotlib import matplotlib
# Force matplotlib to not use any Xwindows backend. # Force matplotlib not to use any Xwindows backend.
matplotlib.use("Agg") matplotlib.use("Agg")
from matplotlib import pyplot as plt from matplotlib import pyplot as plt
from matplotlib import cm from matplotlib import cm
...@@ -109,7 +109,7 @@ def prepare_spec_image(spectrogram): ...@@ -109,7 +109,7 @@ def prepare_spec_image(spectrogram):
where T means the time steps of the spectrogram. It is treated where T means the time steps of the spectrogram. It is treated
as the width of the image. And C means the channels of the as the width of the image. And C means the channels of the
spectrogram, which is treated as the height of the image. And 4 spectrogram, which is treated as the height of the image. And 4
means it is a 'ARGP' format. means it is a 'ARGB' format.
""" """
# [0, 1] # [0, 1]
......
PaddleSpeech/DeepVoice3/preprocess.py
浏览文件 @ e47730a0
# coding: utf-8
# Part of code was adpated from https://github.com/r9y9/deepvoice3_pytorch/tree/master/preprocess.py # Part of code was adpated from https://github.com/r9y9/deepvoice3_pytorch/tree/master/preprocess.py
# Copyright (c) 2017: Ryuichi Yamamoto. # Copyright (c) 2017: Ryuichi Yamamoto.
...@@ -78,7 +77,7 @@ if __name__ == "__main__": ...@@ -78,7 +77,7 @@ if __name__ == "__main__":
# Load preset if specified # Load preset if specified
if preset is not None: if preset is not None:
with open(preset) as f: with io.open(preset) as f:
hparams.parse_json(f.read()) hparams.parse_json(f.read())
# Override hyper parameters # Override hyper parameters
hparams.parse(args.hparams) hparams.parse(args.hparams)
......
PaddleSpeech/DeepVoice3/requirements.txt
浏览文件 @ e47730a0
...@@ -9,6 +9,6 @@ tqdm==4.35.0 ...@@ -9,6 +9,6 @@ tqdm==4.35.0
tensorboardX==1.8 tensorboardX==1.8
matplotlib matplotlib
requests==2.22.0 requests==2.22.0
lws lws==1.2.4
nnmnkwii nnmnkwii
tensorboard tensorboard
PaddleSpeech/DeepVoice3/synthesis.py
浏览文件 @ e47730a0
...@@ -19,6 +19,7 @@ from __future__ import print_function ...@@ -19,6 +19,7 @@ from __future__ import print_function
import argparse import argparse
import sys import sys
import os import os
import io
from os.path import dirname, join, basename, splitext, exists from os.path import dirname, join, basename, splitext, exists
from tqdm import tqdm from tqdm import tqdm
import numpy as np import numpy as np
...@@ -34,7 +35,6 @@ from deepvoice3_paddle.dry_run import dry_run ...@@ -34,7 +35,6 @@ from deepvoice3_paddle.dry_run import dry_run
from hparams import hparams from hparams import hparams
from train import make_deepvoice3_from_hparams from train import make_deepvoice3_from_hparams
from eval_model import tts, plot_alignment from eval_model import tts, plot_alignment
from deepvoice3_paddle.save_load import load_checkpoint
def build_parser(): def build_parser():
...@@ -51,14 +51,6 @@ def build_parser(): ...@@ -51,14 +51,6 @@ def build_parser():
"--use-gpu", "--use-gpu",
action="store_true", action="store_true",
help="Whether to use gpu for generation.") help="Whether to use gpu for generation.")
parser.add_argument(
"--checkpoint-seq2seq",
type=str,
help="Load seq2seq model from checkpoint path.")
parser.add_argument(
"--checkpoint-postnet",
type=str,
help="Load postnet model from checkpoint path.")
parser.add_argument( parser.add_argument(
"--file-name-suffix", type=str, default="", help="File name suffix.") "--file-name-suffix", type=str, default="", help="File name suffix.")
parser.add_argument( parser.add_argument(
...@@ -91,8 +83,6 @@ if __name__ == "__main__": ...@@ -91,8 +83,6 @@ if __name__ == "__main__":
text_list_file_path = args.text_list_file text_list_file_path = args.text_list_file
dst_dir = args.dst_dir dst_dir = args.dst_dir
use_gpu = args.use_gpu use_gpu = args.use_gpu
checkpoint_seq2seq_path = args.checkpoint_seq2seq
checkpoint_postnet_path = args.checkpoint_postnet
max_decoder_steps = args.max_decoder_steps max_decoder_steps = args.max_decoder_steps
file_name_suffix = args.file_name_suffix file_name_suffix = args.file_name_suffix
...@@ -107,7 +97,7 @@ if __name__ == "__main__": ...@@ -107,7 +97,7 @@ if __name__ == "__main__":
# Load preset if specified # Load preset if specified
if preset is not None: if preset is not None:
with open(preset) as f: with io.open(preset) as f:
hparams.parse_json(f.read()) hparams.parse_json(f.read())
# Override hyper parameters # Override hyper parameters
hparams.parse(args.hparams) hparams.parse(args.hparams)
...@@ -118,18 +108,19 @@ if __name__ == "__main__": ...@@ -118,18 +108,19 @@ if __name__ == "__main__":
# Model # Model
model = make_deepvoice3_from_hparams(hparams) model = make_deepvoice3_from_hparams(hparams)
dry_run(model) dry_run(model)
load_checkpoint(checkpoint_path, model) model_dict, _ = dg.load_dygraph(args.checkpoint)
model.set_dict(model_dict)
checkpoint_name = splitext(basename(checkpoint_path))[0] checkpoint_name = splitext(basename(checkpoint_path))[0]
model.seq2seq.decoder.max_decoder_steps = max_decoder_steps model.seq2seq.decoder.max_decoder_steps = max_decoder_steps
if not os.path.exists(dst_dir): if not os.path.exists(dst_dir):
os.mkdir(dst_dir) os.makedirs(dst_dir)
with open(text_list_file_path, "rb") as f: with io.open(text_list_file_path, "rt", encoding="utf-8") as f:
lines = f.readlines() lines = f.readlines()
for idx, line in enumerate(lines): for idx, line in enumerate(lines):
text = line.decode("utf-8")[:-1] text = line[:-1]
words = nltk.word_tokenize(text) words = nltk.word_tokenize(text)
waveform, alignment, _, _ = tts(model, waveform, alignment, _, _ = tts(model,
text, text,
......
PaddleSpeech/DeepVoice3/train.py
浏览文件 @ e47730a0
...@@ -16,6 +16,9 @@ from __future__ import absolute_import ...@@ -16,6 +16,9 @@ from __future__ import absolute_import
from __future__ import division from __future__ import division
from __future__ import print_function from __future__ import print_function
import os
import io
from paddle import fluid from paddle import fluid
import paddle.fluid.dygraph as dg import paddle.fluid.dygraph as dg
...@@ -27,18 +30,11 @@ from deepvoice3_paddle.data import (TextDataSource, MelSpecDataSource, ...@@ -27,18 +30,11 @@ from deepvoice3_paddle.data import (TextDataSource, MelSpecDataSource,
LinearSpecDataSource, LinearSpecDataSource,
PartialyRandomizedSimilarTimeLengthSampler, PartialyRandomizedSimilarTimeLengthSampler,
Dataset, make_loader, create_batch) Dataset, make_loader, create_batch)
from deepvoice3_paddle import frontend from deepvoice3_paddle import frontend
from deepvoice3_paddle.builder import deepvoice3, WindowRange from deepvoice3_paddle.builder import deepvoice3, WindowRange
from deepvoice3_paddle.dry_run import dry_run from deepvoice3_paddle.dry_run import dry_run
from deepvoice3_paddle.save_load import load_checkpoint, _load_embedding, save_checkpoint
from train_model import train_model from train_model import train_model
from deepvoice3_paddle.loss import TTSLoss from deepvoice3_paddle.loss import TTSLoss
import platform
from datetime import datetime
from tensorboardX import SummaryWriter from tensorboardX import SummaryWriter
...@@ -57,10 +53,10 @@ def build_arg_parser(): ...@@ -57,10 +53,10 @@ def build_arg_parser():
parser.add_argument( parser.add_argument(
"--use-gpu", action="store_true", help="Whether to use gpu training.") "--use-gpu", action="store_true", help="Whether to use gpu training.")
parser.add_argument( parser.add_argument(
"--checkpoint-dir", "--output",
type=str, type=str,
default="checkpoints", default="result",
help="Directory where to save model checkpoints") help="Directory to save results")
parser.add_argument( parser.add_argument(
"--preset", "--preset",
type=str, type=str,
...@@ -87,9 +83,6 @@ def build_arg_parser(): ...@@ -87,9 +83,6 @@ def build_arg_parser():
"--train-postnet-only", "--train-postnet-only",
action="store_true", action="store_true",
help="Train only postnet model.") help="Train only postnet model.")
parser.add_argument("--log-event-path", type=str, help="Log event path.")
parser.add_argument(
"--load-embedding", type=str, help="Load embedding from checkpoint.")
parser.add_argument( parser.add_argument(
"--speaker-id", "--speaker-id",
type=int, type=int,
...@@ -152,21 +145,6 @@ def make_optimizer_from_hparams(hparams): ...@@ -152,21 +145,6 @@ def make_optimizer_from_hparams(hparams):
return optim, clipper return optim, clipper
def make_writer_from_args(args):
if args.log_event_path is None:
if platform.system() == "Windows":
log_event_path = "log/run-test" + str(datetime.now()).replace(
" ", "_").replace(":", "_")
else:
log_event_path = "log/run-test" + str(datetime.now()).replace(" ",
"_")
else:
log_event_path = args.log_event_path
print("Log event path: {}".format(log_event_path))
writer = SummaryWriter(log_event_path)
return writer
def make_loss_from_hparams(hparams): def make_loss_from_hparams(hparams):
criterion = TTSLoss( criterion = TTSLoss(
hparams.masked_loss_weight, hparams.priority_freq_weight, hparams.masked_loss_weight, hparams.priority_freq_weight,
...@@ -202,12 +180,19 @@ if __name__ == "__main__": ...@@ -202,12 +180,19 @@ if __name__ == "__main__":
# Load preset if specified # Load preset if specified
if args.preset is not None: if args.preset is not None:
with open(args.preset) as f: with io.open(args.preset) as f:
hparams.parse_json(f.read()) hparams.parse_json(f.read())
# Override hyper parameters # Override hyper parameters
hparams.parse(args.hparams) hparams.parse(args.hparams)
print(hparams_debug_string()) print(hparams_debug_string())
checkpoint_dir = os.path.join(args.output, "checkpoints")
tensorboard_dir = os.path.join(args.output, "log")
if not os.path.exists(checkpoint_dir):
os.makedirs(checkpoint_dir)
if not os.path.exists(tensorboard_dir):
os.makedirs(tensorboard_dir)
data_root = args.data_root data_root = args.data_root
speaker_id = args.speaker_id speaker_id = args.speaker_id
X = FileSourceDataset(TextDataSource(data_root, speaker_id)) X = FileSourceDataset(TextDataSource(data_root, speaker_id))
...@@ -239,7 +224,8 @@ if __name__ == "__main__": ...@@ -239,7 +224,8 @@ if __name__ == "__main__":
model = make_deepvoice3_from_hparams(hparams) model = make_deepvoice3_from_hparams(hparams)
optimizer, clipper = make_optimizer_from_hparams(hparams) optimizer, clipper = make_optimizer_from_hparams(hparams)
writer = make_writer_from_args(args) print("Log event path: {}".format(tensorboard_dir))
writer = SummaryWriter(tensorboard_dir) if local_rank == 0 else None
criterion = make_loss_from_hparams(hparams) criterion = make_loss_from_hparams(hparams)
# loading saved model # loading saved model
...@@ -250,17 +236,8 @@ if __name__ == "__main__": ...@@ -250,17 +236,8 @@ if __name__ == "__main__":
assert hparams.use_decoder_state_for_postnet_input is False, \ assert hparams.use_decoder_state_for_postnet_input is False, \
"when training only the postnet, there is no decoder states" "when training only the postnet, there is no decoder states"
dry_run(model)
if args.checkpoint is not None: if args.checkpoint is not None:
load_checkpoint( model_dict, optimizer_dict = dg.load_dygraph(args.checkpoint)
args.checkpoint,
model,
optimizer,
reset_optimizer=args.reset_optimizer)
if args.load_embedding is not None:
_load_embedding(args.load_embedding, model)
if args.use_data_parallel: if args.use_data_parallel:
strategy = dg.parallel.prepare_context() strategy = dg.parallel.prepare_context()
......
PaddleSpeech/DeepVoice3/train_model.py
浏览文件 @ e47730a0
...@@ -23,20 +23,16 @@ from paddle import fluid ...@@ -23,20 +23,16 @@ from paddle import fluid
import paddle.fluid.dygraph as dg import paddle.fluid.dygraph as dg
from tqdm import tqdm from tqdm import tqdm
from deepvoice3_paddle.save_load import save_checkpoint
from eval_model import eval_model, save_states from eval_model import eval_model, save_states
def train_model(model, loader, criterion, optimizer, clipper, writer, args, def train_model(model, loader, criterion, optimizer, clipper, writer, args,
hparams): hparams):
assert fluid.framework.in_dygraph_mode( assert fluid.framework.in_dygraph_mode(
), "this function must be run with dygraph guard" ), "this function must be run within dygraph guard"
local_rank = dg.parallel.Env().local_rank local_rank = dg.parallel.Env().local_rank
if not os.path.exists(args.checkpoint_dir):
os.mkdir(args.checkpoint_dir)
# amount of shifting when compute losses # amount of shifting when compute losses
linear_shift = hparams.outputs_per_step linear_shift = hparams.outputs_per_step
mel_shift = hparams.outputs_per_step mel_shift = hparams.outputs_per_step
...@@ -44,6 +40,8 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -44,6 +40,8 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
global_step = 0 global_step = 0
global_epoch = 0 global_epoch = 0
ismultispeaker = model.n_speakers > 1 ismultispeaker = model.n_speakers > 1
checkpoint_dir = os.path.join(args.output, "checkpoints")
tensorboard_dir = os.path.join(args.output, "log")
for epoch in range(hparams.nepochs): for epoch in range(hparams.nepochs):
epoch_loss = 0. epoch_loss = 0.
...@@ -113,7 +111,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -113,7 +111,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
linear_predicted, linear_target, linear_mask) linear_predicted, linear_target, linear_mask)
lin_loss = criterion.binary_divergence_weight * lin_div \ lin_loss = criterion.binary_divergence_weight * lin_div \
+ (1 - criterion.binary_divergence_weight) * lin_l1_loss + (1 - criterion.binary_divergence_weight) * lin_l1_loss
if writer is not None: if writer is not None and local_rank == 0:
writer.add_scalar("linear_loss", writer.add_scalar("linear_loss",
float(lin_loss.numpy()), global_step) float(lin_loss.numpy()), global_step)
writer.add_scalar("linear_l1_loss", writer.add_scalar("linear_l1_loss",
...@@ -134,7 +132,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -134,7 +132,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
mel_mask) mel_mask)
mel_loss = criterion.binary_divergence_weight * mel_div \ mel_loss = criterion.binary_divergence_weight * mel_div \
+ (1 - criterion.binary_divergence_weight) * mel_l1_loss + (1 - criterion.binary_divergence_weight) * mel_l1_loss
if writer is not None: if writer is not None and local_rank == 0:
writer.add_scalar("mel_loss", writer.add_scalar("mel_loss",
float(mel_loss.numpy()), global_step) float(mel_loss.numpy()), global_step)
writer.add_scalar("mel_l1_loss", writer.add_scalar("mel_l1_loss",
...@@ -143,7 +141,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -143,7 +141,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
float(mel_div.numpy()), global_step) float(mel_div.numpy()), global_step)
done_loss = criterion.done_loss(done_hat, done) done_loss = criterion.done_loss(done_hat, done)
if writer is not None: if writer is not None and local_rank == 0:
writer.add_scalar("done_loss", writer.add_scalar("done_loss",
float(done_loss.numpy()), global_step) float(done_loss.numpy()), global_step)
...@@ -153,7 +151,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -153,7 +151,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
attn_loss = criterion.attention_loss(alignments, attn_loss = criterion.attention_loss(alignments,
input_lengths.numpy(), input_lengths.numpy(),
decoder_length) decoder_length)
if writer is not None: if writer is not None and local_rank == 0:
writer.add_scalar("attention_loss", writer.add_scalar("attention_loss",
float(attn_loss.numpy()), global_step) float(attn_loss.numpy()), global_step)
...@@ -169,7 +167,8 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -169,7 +167,8 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
loss = mel_loss + done_loss loss = mel_loss + done_loss
else: else:
loss = lin_loss loss = lin_loss
if writer is not None:
if writer is not None and local_rank == 0:
writer.add_scalar("loss", float(loss.numpy()), global_step) writer.add_scalar("loss", float(loss.numpy()), global_step)
if isinstance(optimizer._learning_rate, if isinstance(optimizer._learning_rate,
...@@ -177,6 +176,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -177,6 +176,7 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
current_lr = optimizer._learning_rate.step().numpy() current_lr = optimizer._learning_rate.step().numpy()
else: else:
current_lr = optimizer._learning_rate current_lr = optimizer._learning_rate
if writer is not None and local_rank == 0:
writer.add_scalar("learning_rate", current_lr, global_step) writer.add_scalar("learning_rate", current_lr, global_step)
epoch_loss += loss.numpy()[0] epoch_loss += loss.numpy()[0]
...@@ -185,13 +185,15 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -185,13 +185,15 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
global_step % hparams.checkpoint_interval == 0): global_step % hparams.checkpoint_interval == 0):
save_states(global_step, writer, mel_outputs, linear_outputs, save_states(global_step, writer, mel_outputs, linear_outputs,
alignments, mel, linear, alignments, mel, linear,
input_lengths.numpy(), args.checkpoint_dir) input_lengths.numpy(), checkpoint_dir)
save_checkpoint(model, optimizer, args.checkpoint_dir, step_path = os.path.join(
global_step) checkpoint_dir, "checkpoint_{:09d}".format(global_step))
dg.save_dygraph(model.state_dict(), step_path)
dg.save_dygraph(optimizer.state_dict(), step_path)
if (local_rank == 0 and global_step > 0 and if (local_rank == 0 and global_step > 0 and
global_step % hparams.eval_interval == 0): global_step % hparams.eval_interval == 0):
eval_model(global_step, writer, model, args.checkpoint_dir, eval_model(global_step, writer, model, checkpoint_dir,
ismultispeaker) ismultispeaker)
if args.use_data_parallel: if args.use_data_parallel:
...@@ -232,5 +234,9 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args, ...@@ -232,5 +234,9 @@ def train_model(model, loader, criterion, optimizer, clipper, writer, args,
global_step += 1 global_step += 1
average_loss_in_epoch = epoch_loss / (step + 1)
print("Epoch loss: {}".format(average_loss_in_epoch))
if writer is not None and local_rank == 0:
writer.add_scalar("average_loss_in_epoch", average_loss_in_epoch,
global_epoch)
global_epoch += 1 global_epoch += 1
print("Epoch loss: {}".format(epoch_loss / (step + 1)))
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或