U-Net训练问题
Created by: gxqnba
数据检校全部通过，但是进行训练时，提示reader出错，报错如下： Use multi-thread reader 2020-09-11 08:25:32,431-WARNING: Your reader has raised an exception! Exception in thread Thread-2: Traceback (most recent call last): File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1156, in thread_main six.reraise(*sys.exc_info()) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1136, in thread_main for tensors in self._tensor_reader(): File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 195, in call for each_sample in self.generator(): File "pdseg/train.py", line 207, in data_generator for b in data_gen: File "/home/aistudio/work/PaddleSeg/pdseg/reader.py", line 81, in generator yield self.process_image(line, self.data_dir, self.mode) File "/home/aistudio/work/PaddleSeg/pdseg/reader.py", line 253, in process_image img, grt = aug.resize(img, grt, mode) File "/home/aistudio/work/PaddleSeg/pdseg/data_aug.py", line 44, in resize grt = cv2.resize(grt, target_size, interpolation=cv2.INTER_NEAREST) TypeError: Expected Ptrcv::UMat for argument '%s'
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py:1070: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.")
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >) 3 paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >) 4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&) 5 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 6 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const
Python Call Stacks (More useful to users):
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2610, in append_op attrs=kwargs.get("attrs", None)) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1078, in _init_non_iterable attrs={'drop_last': self._drop_last}) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 976, in init self._init_non_iterable() File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 608, in from_generator iterable, return_list, drop_last) File "/home/aistudio/work/PaddleSeg/pdseg/models/model_builder.py", line 142, in build_model use_double_buffer=True) File "pdseg/train.py", line 237, in train train_prog, startup_prog, phase=ModelPhase.TRAIN) File "pdseg/train.py", line 453, in main train(cfg) File "pdseg/train.py", line 466, in main(args)
Error Message Summary:
Error: Blocking queue is killed because the data reader raises an exception [Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141) [operator < read > error] W0911 08:25:32.449854 315 operator.cc:187] read raises an exception std::future_error, St12future_error F0911 08:25:32.449900 315 exception_holder.h:37] std::exception caught, No associated state