Created by: misbullah

Hi, I got the following error when running the librispeech examples.

I use cuDNN v5.5

CUDA_VISIBLE_DEVICES=0,1,2,3

(paddle-env) alim@ctc3:~/paddle-deepspeech/examples/librispeech$ sh run_train.sh ----------- Configuration Arguments ----------- augment_conf_path: conf/augmentation.config batch_size: 16 dev_manifest: data/librispeech/manifest.dev-clean init_model_path: None is_local: 1 learning_rate: 0.0005 max_duration: 27.0 mean_std_path: data/librispeech/mean_std.npz min_duration: 0.0 num_conv_layers: 2 num_iter_print: 100 num_passes: 50 num_proc_data: 16 num_rnn_layers: 3 output_model_dir: ./checkpoints/libri rnn_layer_size: 512 share_rnn_weights: 1 shuffle_method: batch_shuffle_clipped specgram_type: linear test_off: 0 train_manifest: data/librispeech/manifest.train trainer_count: 30 use_gpu: 1 use_gru: 0 use_sortagrad: 1 vocab_path: data/librispeech/vocab.txt

I0105 17:36:05.526255 27647 Util.cpp:166] commandline: --use_gpu=1 --rnn_use_batch=True --log_clipping=True --trainer_count=30 [INFO 2018-01-05 17:36:10,822 layers.py:2689] output for conv_0: c = 32, h = 81, w = 54, size = 139968 [INFO 2018-01-05 17:36:10,824 layers.py:3251] output for batch_norm_0: c = 32, h = 81, w = 54, size = 139968 [INFO 2018-01-05 17:36:10,825 layers.py:7409] output for scale_sub_region_0: c = 32, h = 81, w = 54, size = 139968 [INFO 2018-01-05 17:36:10,826 layers.py:2689] output for conv_1: c = 32, h = 41, w = 54, size = 70848 [INFO 2018-01-05 17:36:10,827 layers.py:3251] output for batch_norm_1: c = 32, h = 41, w = 54, size = 70848 [INFO 2018-01-05 17:36:10,828 layers.py:7409] output for scale_sub_region_1: c = 32, h = 41, w = 54, size = 70848 F0105 17:36:10.859037 27647 hl_gpu_matrix_kernel.cuh:181] Check failed: cudaSuccess == err (0 vs. 8) [hl_gpu_apply_unary_op failed] CUDA error: invalid device function

* Check failure stack trace: *

@ 0x7fb7341cabcd google::LogMessage::Fail() @ 0x7fb7341ce67c google::LogMessage::SendToLog() @ 0x7fb7341ca6f3 google::LogMessage::Flush() @ 0x7fb7341cfb8e google::LogMessageFatal::~LogMessageFatal() @ 0x7fb73403f3eb hl_gpu_apply_unary_op<>() @ 0x7fb73403f75d paddle::BaseMatrixT<>::applyUnary<>() @ 0x7fb73403f9a3 paddle::BaseMatrixT<>::zero() @ 0x7fb733fde375 paddle::GpuMatrix::zeroMem() @ 0x7fb733ec8e72 paddle::BatchNormBaseLayer::init() @ 0x7fb733e820c1 paddle::CudnnBatchNormLayer::init() @ 0x7fb733ed3d7f paddle::NeuralNetwork::init() @ 0x7fb733ef9506 paddle::MultiGradientMachine::MultiGradientMachine() @ 0x7fb733efdd7f paddle::GradientMachine::create() @ 0x7fb7341a7495 GradientMachine::createFromPaddleModelPtr() @ 0x7fb7341a767f GradientMachine::createByConfigProtoStr() @ 0x7fb733d84717 _wrap_GradientMachine_createByConfigProtoStr @ 0x52714b PyEval_EvalFrameEx @ 0x555551 PyEval_EvalCodeEx @ 0x525560 PyEval_EvalFrameEx @ 0x555551 PyEval_EvalCodeEx @ 0x524338 PyEval_EvalFrameEx @ 0x568b3a (unknown) @ 0x4c2604 (unknown) @ 0x4d1c5c (unknown) @ 0x55f6db (unknown) @ 0x5244dd PyEval_EvalFrameEx @ 0x555551 PyEval_EvalCodeEx @ 0x524338 PyEval_EvalFrameEx @ 0x555551 PyEval_EvalCodeEx @ 0x525560 PyEval_EvalFrameEx @ 0x555551 PyEval_EvalCodeEx @ 0x525560 PyEval_EvalFrameEx @ 0x555551 PyEval_EvalCodeEx @ 0x525560 PyEval_EvalFrameEx run_train.sh: line 33: 27647 Aborted (core dumped) CUDA_VISIBLE_DEVICES=0,1,2,3 python -u train.py --batch_size=16 --trainer_count=30 --num_passes=50 --num_proc_data=16 --num_conv_layers=2 --num_rnn_layers=3 --rnn_layer_size=512 --num_iter_print=100 --learning_rate=5e-4 --max_duration=27.0 --min_duration=0.0 --test_off=False --use_sortagrad=True --use_gru=False --use_gpu=True --is_local=True --share_rnn_weights=True --train_manifest='data/librispeech/manifest.train' --dev_manifest='data/librispeech/manifest.dev-clean' --mean_std_path='data/librispeech/mean_std.npz' --vocab_path='data/librispeech/vocab.txt' --output_model_dir='./checkpoints/libri' --augment_conf_path='conf/augmentation.config' --specgram_type='linear' --shuffle_method='batch_shuffle_clipped' Failed in training

(paddle-env) alim@ctc3:~/paddle-deepspeech/examples/librispeech$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2016 NVIDIA Corporation Built on Sun_Sep__4_22:14:01_CDT_2016 Cuda compilation tools, release 8.0, V8.0.44

(paddle-env) alim@ctc3:~/paddle-deepspeech/examples/librispeech$ nvidia-smi Fri Jan 5 17:38:35 2018
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 384.90 Driver Version: 384.90 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX 1080 Off | 00000000:02:00.0 Off | N/A | | 24% 45C P2 44W / 180W | 1117MiB / 8114MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce GTX 1080 Off | 00000000:03:00.0 Off | N/A | | 24% 36C P8 8W / 180W | 10MiB / 8114MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 GeForce GTX 1080 Off | 00000000:81:00.0 Off | N/A | | 24% 36C P8 7W / 180W | 10MiB / 8114MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 GeForce GTX 1080 Off | 00000000:82:00.0 Off | N/A | | 24% 45C P2 134W / 180W | 2438MiB / 8114MiB | 0% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 27612 C nnet3-chain-train 1107MiB | | 3 27473 C nnet3-chain-train 1159MiB | | 3 27479 C nnet3-chain-train 1269MiB | +-----------------------------------------------------------------------------+

Any suggestion?

Thanks, Alim