Created by: wueching

wueching@ubuntu:~/research/ai/DeepSpeech/examples/tiny$ ./run_train.sh----------- Configuration Arguments ----------- augment_conf_path: conf/augmentation.config batch_size: 4 dev_manifest: data/tiny/manifest.tiny init_from_pretrained_model: None is_local: 1 learning_rate: 1e-05 max_duration: 27.0 mean_std_path: data/tiny/mean_std.npz min_duration: 0.0 num_conv_layers: 2 num_epoch: 20 num_iter_print: 1 num_rnn_layers: 3 num_samples: 64 output_model_dir: ./checkpoints/tiny rnn_layer_size: 2048 save_epoch: 1 share_rnn_weights: 1 shuffle_method: batch_shuffle_clipped specgram_type: linear test_off: 0 train_manifest: data/tiny/manifest.tiny use_gpu: 0 use_gru: 0 use_sortagrad: 1 vocab_path: data/tiny/vocab.txt

I0103 11:00:11.385960 15533 parallel_executor.cc:421] The number of CPUPlace, which is used in ParallelExecutor, is 4. And the Program will be copied 4 copies W0103 11:00:12.639942 15533 fuse_all_reduce_op_pass.cc:72] Find all_reduce operators: 29. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 29. I0103 11:00:12.644691 15533 build_strategy.cc:363] SeqOnlyAllReduceOps:0, num_trainers:1 I0103 11:00:12.784705 15533 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0103 11:00:12.845278 15533 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 W0103 11:00:16.228854 15565 dynamic_loader.cc:151] Failed to find dynamic library: /usr/local/lib/python2.7/dist-packages/paddle/libs/libwarpctc.so (dlopen: cannot load any more object with static TLS) W0103 11:00:16.230798 15565 dynamic_loader.cc:120] Can not find library: libwarpctc.so. The process maybe hang. Please try to add the lib path to LD_LIBRARY_PATH. /usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py:779: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "train.py", line 142, in main() File "train.py", line 138, in main train() File "train.py", line 133, in train test_off=args.test_off) File "/home/wueching/research/ai/DeepSpeech/model_utils/model.py", line 337, in train return_numpy=False) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 780, in run six.reraise(*sys.exc_info()) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 775, in run use_program_cache=use_program_cache) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 834, in _run_impl return_numpy=return_numpy) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 674, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::platform::dynload::GetWarpCTCDsoHandle() 3 void std::__once_call_impl<std::_Bind_simple<decltype (get_warpctc_version({parm#1}...)) paddle::platform::dynload::DynLoad__get_warpctc_version::operator()<>()::{lambda()#1 (closed)} ()> >() 4 paddle::operators::WarpCTCFunctorpaddle::platform::CPUDeviceContext::operator()(paddle::framework::ExecutionContext const&, float const*, float*, int const*, int const*, int const*, unsigned long, unsigned long, unsigned long, float*) 5 paddle::operators::WarpCTCKernel<paddle::platform::CPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 6 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::WarpCTCKernel<paddle::platform::CPUDeviceContext, float> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1 (closed)}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 7 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 8 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 9 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 10 paddle::framework::details::ComputationOpHandle::RunImpl() 11 paddle::framework::details::OpHandleBase::Run(bool) 12 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*) 13 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 14 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 15 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 16 ThreadPool::ThreadPool(unsigned long)::{lambda()#1 (closed)}::operator()() const

Python Call Stacks (More useful to users):

File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/framework.py", line 2488, in append_op attrs=kwargs.get("attrs", None)) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layers/nn.py", line 7557, in warpctc 'norm_by_times': norm_by_times, File "/home/wueching/research/ai/DeepSpeech/model_utils/network.py", line 446, in deep_speech_v2_network input=fc, label=text_data, blank=dict_size, norm_by_times=True) File "/home/wueching/research/ai/DeepSpeech/model_utils/model.py", line 145, in create_network share_rnn_weights=self._share_rnn_weights) File "/home/wueching/research/ai/DeepSpeech/model_utils/model.py", line 281, in train train_reader, log_probs, ctc_loss = self.create_network() File "train.py", line 133, in train test_off=args.test_off) File "train.py", line 138, in main train() File "train.py", line 142, in main()

Error Message Summary:

Error: Failed to find dynamic library: libwarpctc.so ( dlopen: cannot load any more object with static TLS ) Please specify its path correctly using following ways: Method. set environment variable LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS. For instance, issue command: export LD_LIBRARY_PATH=... Note: After Mac OS 10.11, using the DYLD_LIBRARY_PATH is impossible unless System Integrity Protection (SIP) is disabled. at (/paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:177) [operator < warpctc > error] Failed in training!

How to fix this issue,thanks a lot.