Created by: Willoiron

您好,我在单个NV M40 12GB 显卡上面跑aishell ,用的是nvidia-docker运行, batch_size从默认的64改成了24,连续两次试着跑aishell都在第15个epoch结束的时候出现了下面的这个问题: Pass: 15, Batch: 4700, TrainCost: 0.866109 ................................................................................................... Pass: 15, Batch: 4800, TrainCost: 0.982023 ................................................................................................... Pass: 15, Batch: 4900, TrainCost: 0.861112 ................................................................................................... Pass: 15, Batch: 5000, TrainCost: 0.930983 ...Exception in thread Thread-32: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/DeepSpeech/data_utils/utility.py", line 153, in flush_worker sample = in_queue.get() File "", line 2, in get File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() IOError: got end of file during message

Process Process-822: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/DeepSpeech/data_utils/utility.py", line 135, in order_handle_worker while order_id != out_order[0]: File "", line 2, in getitem File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() IOError: [Errno 104] Connection reset by peer Process Process-830: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/DeepSpeech/data_utils/utility.py", line 135, in order_handle_worker while order_id != out_order[0]: File "", line 2, in getitem File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() IOError: [Errno 104] Connection reset by peer Process Process-819: Process Process-815: Traceback (most recent call last): Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap Process Process-828: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/DeepSpeech/data_utils/utility.py", line 135, in order_handle_worker while order_id != out_order[0]: File "", line 2, in getitem File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() IOError: [Errno 104] Connection reset by peer self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/DeepSpeech/data_utils/utility.py", line 135, in order_handle_worker self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/DeepSpeech/data_utils/utility.py", line 135, in order_handle_worker while order_id != out_order[0]: File "", line 2, in getitem File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod while order_id != out_order[0]: File "", line 2, in getitem File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() kind, result = conn.recv() IOError: [Errno 104] Connection reset by peer IOError: [Errno 104] Connection reset by peer . . . 请问这个问题是..?