Errors when train other DataSet using FastSpeech
Created by: ymzlygw
准备用fastSpeech 训练JUST数据集时,先准备使用tts 教学model 得到aligments文件,结果处理时报出如下错误:
运行程序: python get_alignments_just.py --use_gpu 1 --data '../jsut_ver1.1' --config '../pre_model_tts/ljspeech.yaml' --checkpoint_transformer '../pre_model_tts/step-120000'
输出:
W0728 07:55:57.502537 15534 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.1, Runtime API Version: 10.0
W0728 07:55:57.505069 15534 device_context.cc:260] device: 0, cuDNN Version: 7.6.
[checkpoint] Rank 0: loaded model from ../pre_model_tts/step-120000.pdparams
7%|███████████▏ | 534/7696 [00:16<03:31, 33.82it/s]Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
Error: /paddle/paddle/fluid/operators/lookup_table_v2_op.cu:43 Assertion id < N
failed. Variable value (input) of OP(fluid.layers.embedding) expected >= 0 and < 1024, but got 1053. Please check input value.
........
........
value.
7%|███████████▏ | 537/7696 [00:16<03:36, 33.00it/s]
Traceback (most recent call last):
File "get_alignments_just.py", line 157, in
alignments(args)
File "get_alignments_just.py", line 144, in alignments
network_cfg['decoder_num_head'])
File "/root/AI_/Paddle/Parakeet/parakeet/models/fastspeech/utils.py", line 22, in get_alignment
multi_attn = attn_probs[i].numpy()
paddle.fluid.core_avx.EnforceNotMet:
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::platform::GpuMemcpySync(void*, void const*, unsigned long, cudaMemcpyKind)
Error Message Summary:
ExternalError: Cuda error(4), unspecified launch failure. [Advise: An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. All existing device memory allocations are invalid. To continue using CUDA, theprocess must be terminated and relaunched.] at (/paddle/paddle/fluid/platform/gpu_info.cc:281)
这是哪里出问题了呢?麻烦开发者解答下