Created by: GC-Tom

不使用gpu情况下 可正常使用，使用gpu时的报错信息如下： ----------- Configuration Arguments ----------- alpha: 2.5 batch_size: 128 beam_size: 500 beta: 0.3 cutoff_prob: 1.0 cutoff_top_n: 40 decoding_method: ctc_beam_search error_rate_type: wer lang_model_path: models/lm/common_crawl_00.prune01111.trie.klm mean_std_path: models/librispeech/mean_std.npz model_path: checkpoints/libri/epoch_3 num_conv_layers: 2 num_proc_bsearch: 8 num_rnn_layers: 3 rnn_layer_size: 2048 share_rnn_weights: True specgram_type: linear test_manifest: data/librispeech/manifest.test-clean use_gpu: True use_gru: False vocab_path: models/librispeech/vocab.txt

[INFO 2019-12-07 11:40:57,547 model.py:475] begin to initialize the external scorer for decoding [INFO 2019-12-07 11:41:08,305 model.py:485] language model: is_character_based = 0, max_order = 5, dict_size = 400000 [INFO 2019-12-07 11:41:08,522 model.py:486] end initializing scorer [INFO 2019-12-07 11:41:08,522 test.py:109] start evaluation ... W1207 11:41:13.783799 9190 device_context.cc:235] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.1, Runtime API Version: 9.0 W1207 11:41:13.788204 9190 device_context.cc:243] device: 0, cuDNN Version: 7.5. finish initing model from pretrained params from checkpoints/libri/epoch_3 /usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py:774: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "/data/DeepSpeech/test.py", line 149, in main() File "/data/DeepSpeech/test.py", line 145, in main evaluate() File "/data/DeepSpeech/test.py", line 113, in evaluate feeding_dict=data_generator.feeding) File "/data/DeepSpeech/model_utils/model.py", line 424, in infer_batch_probs return_numpy=False) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 775, in run six.reraise(*sys.exc_info()) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 770, in run use_program_cache=use_program_cache) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 817, in _run_impl use_program_cache=use_program_cache) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 894, in _run_program fetch_var_name) paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 paddle::operators::CUDNNConvOpKernel::Compute(paddle::framework::ExecutionContext const&) const 3 std::Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::CUDNNConvOpKernel, paddle::operators::CUDNNConvOpKernel, paddle::operators::CUDNNConvOpKernelpaddle::platform::float16 >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1 (closed)}>::M_invoke(std::Any_data const&, paddle::framework::ExecutionContext const&) 4 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const 5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const 6 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) 7 paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) 8 paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool)

Python Call Stacks (More useful to users):

File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/framework.py", line 2459, in append_op attrs=kwargs.get("attrs", None)) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layers/nn.py", line 2803, in conv2d "data_format": data_format, File "/data/DeepSpeech/model_utils/network.py", line 47, in conv_bn_layer bias_attr=False) File "/data/DeepSpeech/model_utils/network.py", line 299, in conv_group name='layer_0', ) File "/data/DeepSpeech/model_utils/network.py", line 408, in deep_speech_v2_network masks=masks) File "/data/DeepSpeech/model_utils/model.py", line 145, in create_network share_rnn_weights=self._share_rnn_weights) File "/data/DeepSpeech/model_utils/model.py", line 403, in infer_batch_probs feeder, log_probs, _ = self.create_network(is_infer=True) File "/data/DeepSpeech/test.py", line 113, in evaluate feeding_dict=data_generator.feeding) File "/data/DeepSpeech/test.py", line 145, in main evaluate() File "/data/DeepSpeech/test.py", line 149, in main()

Error Message Summary:

PaddleCheckError: CUDNN_STATUS_EXECUTION_FAILED at [/paddle/paddle/fluid/operators/conv_cudnn_op.cu:288] [operator < conv2d > error]

Process finished with exit code 1

此外，请问为什么train.py可以正常使用gpu，test和infer就会报错误信息，如果train时也报错，那么基本原因就是cudnn版本问题，现在困惑的是train是正常的，麻烦告知下原因，谢谢