在docker image中运行examples/tiny/run_train.sh报错:terminate called after throwing an instance of 'Xbyak::Error'
Created by: jayxioresearch
按照“Installation->Running in Docker Container"中步骤安装了Docker image. 按照“Getting Started“ 中的步骤一步步运行,“sh run_data.sh” 成功,在运行“sh run_train.sh”报错。
全部的报错信息: terminate called after throwing an instance of 'Xbyak::Error' what(): internal error Aborted (core dumped) Fail in training!
全部输出如下: root@1ade2c36fbac:/DeepSpeech# cd examples/tiny/ root@1ade2c36fbac:/DeepSpeech/examples/tiny# sh run_data.sh Skip downloading and unpacking. Data already exists in /root/.cache/paddle/dataset/speech/libri/test-clean. Creating manifest data/tiny/manifest.test-clean ... Skip downloading and unpacking. Data already exists in /root/.cache/paddle/dataset/speech/libri/dev-clean. Creating manifest data/tiny/manifest.dev-clean ... ----------- Configuration Arguments ----------- count_threshold: 0 manifest_paths: ['data/tiny/manifest.dev-clean'] vocab_path: data/tiny/vocab.txt
----------- Configuration Arguments ----------- manifest_path: data/tiny/manifest.tiny num_samples: 64 output_path: data/tiny/mean_std.npz specgram_type: linear
Tiny data preparation done. root@1ade2c36fbac:/DeepSpeech/examples/tiny# sh run_train.sh ----------- Configuration Arguments ----------- augment_conf_path: conf/augmentation.config batch_size: 16 dev_manifest: data/tiny/manifest.tiny init_model_path: None is_local: 1 learning_rate: 1e-05 max_duration: 27.0 mean_std_path: data/tiny/mean_std.npz min_duration: 0.0 num_conv_layers: 2 num_iter_print: 100 num_passes: 20 num_proc_data: 1 num_rnn_layers: 3 output_model_dir: ./checkpoints/tiny rnn_layer_size: 2048 share_rnn_weights: 1 shuffle_method: batch_shuffle_clipped specgram_type: linear test_off: 0 train_manifest: data/tiny/manifest.tiny trainer_count: 4 use_gpu: 1 use_gru: 0 use_sortagrad: 1 vocab_path: data/tiny/vocab.txt
terminate called after throwing an instance of 'Xbyak::Error' what(): internal error Aborted (core dumped) Fail in training! root@1ade2c36fbac:/DeepSpeech/examples/tiny#
nvidia-docker信息 (nvidia GPU: 1080TI GTX) : jay@tweed:~$ sudo docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi Wed Oct 31 20:07:40 2018 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 390.87 Driver Version: 390.87 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX 108... Off | 00000000:09:00.0 Off | N/A | | 8% 53C P0 55W / 250W | 0MiB / 11178MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce GTX 108... Off | 00000000:41:00.0 Off | N/A | | 10% 54C P0 53W / 250W | 0MiB / 11176MiB | 2% Default | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+