自定义yolov3 backbone voc 数据集下量化训练报错
Created by: justinmeibi
Paddle version: paddlepaddle-gpu==1.7.0.post107 paddleslim==1.0.1 Paddle With CUDA: None OS: debian buster/sid Python version: 3.7.3 CUDA version: 10.1 cuDNN version: 7.5.0 Nvidia driver version: 410.104
配置文件 batch 1,8 线程1,8 bufsize 1,16都试过 fine_gained_loss=True
tools/train.py 是正常的 但是 slim/quantion/train.py量化训练报错
报错信息:Error Message Summary:
Error: Blocking queue is killed because the data reader raises an exception [Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141) [operator < read > error]
补充log信息: 2020-07-15 16:01:26,102-INFO: places would be ommited when DataLoader is not iterable I0715 16:01:26.122622 22133 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0715 16:01:26.143930 22133 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0715 16:01:26.169147 22133 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0715 16:01:26.181007 22133 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 2020-07-15 16:01:26,225-WARNING: recv endsignal from outq with errmsg[consumer[consumer-019-0] failed to map with error:[target0 not in samples]] 2020-07-15 16:01:26,225-WARNING: Your reader has raised an exception! Exception in thread Thread-4: