Created by: Redlyliu

你好，我paddle detection用yolov3_r34_voc 训练自己的数据，出现下面的错误，运行环境是windos10 RTX20206 6G，python3.6.10 paddle 1.7.1，CUDA 9.0 ，CUDNN 7.3.1 error: 2020-03-27 14:20:07,666-INFO: iter: 17200, lr: 0.001000, 'loss': '14.405443', time: 0.527, eta: 7:43:53 2020-03-27 14:20:11,634-INFO: fail to map op [RandomFlipImage_545aca] with error: RandomFlipImage_545aca: invalid box, x2 should be greater than x1 and stack: Traceback (most recent call last): File "D:\ai_nvidia\ai_code\Paddle\PaddleDetection-release-0.2\ppdet\data\reader.py", line 45, in call data = f(data, ctx) File "D:\ai_nvidia\ai_code\Paddle\PaddleDetection-release-0.2\ppdet\data\transform\operators.py", line 427, in call raise BboxError(m) ppdet.data.transform.operators.BboxError: RandomFlipImage_545aca: invalid box, x2 should be greater than x1

2020-03-27 14:20:11,646-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]] 2020-03-27 14:20:18,789-INFO: iter: 17220, lr: 0.001000, 'loss': '14.676455', time: 0.595, eta: 8:43:36 2020-03-27 14:20:32,041-INFO: iter: 17240, lr: 0.001000, 'loss': '14.626053', time: 0.643, eta: 9:25:01 2020-03-27 14:20:32,511-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]] 2020-03-27 14:20:40,244-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-3] exits for reason[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]]] 2020-03-27 14:20:41,087-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-4] exits for reason[consumer[consumer-05f-3] exits for reason[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]]]] 2020-03-27 14:20:42,242-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-0] exits for reason[consumer[consumer-05f-4] exits for reason[consumer[consumer-05f-3] exits for reason[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]]]]] 2020-03-27 14:20:42,627-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-5] exits for reason[consumer[consumer-05f-0] exits for reason[consumer[consumer-05f-4] exits for reason[consumer[consumer-05f-3] exits for reason[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]]]]]] 2020-03-27 14:20:42,779-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-7] exits for reason[consumer[consumer-05f-5] exits for reason[consumer[consumer-05f-0] exits for reason[consumer[consumer-05f-4] exits for reason[consumer[consumer-05f-3] exits for reason[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]]]]]]] 2020-03-27 14:20:42,810-WARNING: recv endsignal from outq with errmsg[consumer[consumer-05f-6] exits for reason[consumer[consumer-05f-7] exits for reason[consumer[consumer-05f-5] exits for reason[consumer[consumer-05f-0] exits for reason[consumer[consumer-05f-4] exits for reason[consumer[consumer-05f-3] exits for reason[consumer[consumer-05f-1] exits for reason[consumer[consumer-05f-2] failed to map with error:[RandomFlipImage_545aca: invalid box, x2 should be greater than x1]]]]]]]]] 2020-03-27 14:20:42,810-WARNING: Your reader has raised an exception! Exception in thread Thread-21: Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\paddle\lib\threading.py", line 916, in _bootstrap_inner self.run() File "C:\ProgramData\Anaconda3\envs\paddle\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\reader.py", line 805, in thread_main six.reraise(*sys.exc_info()) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\six.py", line 703, in reraise raise value File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\reader.py", line 785, in thread_main for tensors in self._tensor_reader(): File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\reader.py", line 853, in tensor_reader_impl for slots in paddle_reader(): File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\data_feeder.py", line 488, in reader_creator for item in reader(): File "D:\ai_nvidia\ai_code\Paddle\PaddleDetection-release-0.2\ppdet\data\reader.py", line 417, in _reader reader.reset() File "D:\ai_nvidia\ai_code\Paddle\PaddleDetection-release-0.2\ppdet\data\parallel_map.py", line 253, in reset assert not self._exit, "cannot reset for already stopped dataset" AssertionError: cannot reset for already stopped dataset

C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\executor.py:782: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "my_train.py", line 323, in main() File "my_train.py", line 233, in main outs = exe.run(compiled_train_prog, fetch_list=train_values) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\executor.py", line 783, in run six.reraise(*sys.exc_info()) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\six.py", line 703, in reraise raise value File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\executor.py", line 778, in run use_program_cache=use_program_cache) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\executor.py", line 843, in _run_impl return_numpy=return_numpy) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\executor.py", line 677, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

Windows not support stack backtrace yet.

Python Call Stacks (More useful to users):

File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\framework.py", line 2525, in append_op attrs=kwargs.get("attrs", None)) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\reader.py", line 733, in _init_non_iterable outputs={'Out': self._feed_list}) File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\reader.py", line 646, in init self._init_non_iterable() File "C:\ProgramData\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\reader.py", line 280, in from_generator iterable, return_list) File "D:\ai_nvidia\ai_code\Paddle\PaddleDetection-release-0.2\ppdet\modeling\architectures\yolov3.py", line 152, in build_inputs iterable=iterable) if use_dataloader else None File "my_train.py", line 115, in main feed_vars, train_loader = model.build_inputs(**inputs_def) File "my_train.py", line 323, in main()

Error Message Summary:

Error: Blocking queue is killed because the data reader raises an exception [Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (D:\1.7.1\paddle\paddle/fluid/operators/reader/blocking_queue.h:141) [operator < read > error]