Created by: Fauny

ppyolo模型，已完成训练和剪枝，测试正常

参考slim/quantization/README.md， 尝试对剪枝后模型做量化：

python slim/quantization/train.py --not_quant_pattern yolo_output \
    --eval \
    -c ./configs/my_ppyolo.yml \
    -o max_iters=30000 \
    save_dir=./output/prune_quant \
    LearningRate.base_lr=0.0001 \
    LearningRate.schedulers="[!PiecewiseDecay {gamma: 0.1, milestones: [10000]}]" \
    pretrain_weights=output/my_ppyolo_pruned/best_model
    # weights=output/my_ppyolo_pruned/best_model 用这个也不行

报错

WARNING: Do not have avx core. You may not build with AVX, but AVX is supported on local machine.
 You could build paddle WITH_AVX=ON to get better performance.
The original error is: No module named 'paddle.fluid.core_avx'
W0911 18:50:31.530455  4074 init.cc:185] AVX is available, Please re-compile on local machine
[09-11 18:50:31 MainThread @logger.py:224] Argv: slim/quantization/train.py --not_quant_pattern yolo_output --eval -c /home/aistudio/work/luobo-prune.yml -o max_iters=30000 save_dir=./output/qd LearningRate.base_lr=0.0001 LearningRate.schedulers=[!PiecewiseDecay {gamma: 0.1, milestones: [10000]}] weights=../output/prune/best_model
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/pandas/core/tools/datetimes.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py:153: UserWarning: PaddlePaddle version 1.8.0 or higher is required, but 0.0.0 installed, Maybe you are using a develop version, please make sure the version is good with your code.
  (min_version, fluid_version.full_version))
2020-09-11 18:50:34,718-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000500] in Optimizer will not take effect, and it will only be applied to other Parameters!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2020-09-11 18:50:42,368-INFO: places would be ommited when DataLoader is not iterable
W0911 18:50:42.421072  4074 device_context.cc:268] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 10.1, Runtime API Version: 9.0
W0911 18:50:42.426079  4074 device_context.cc:276] device: 0, cuDNN Version: 7.6.
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/io.py:2052: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc_0.b_0 fc_0.w_0
  format(" ".join(unused_para_list)))
2020-09-11 18:50:46,380-INFO: quant_aware config {'weight_quantize_type': 'channel_wise_abs_max', 'activation_quantize_type': 'moving_average_abs_max', 'weight_bits': 8, 'activation_bits': 8, 'not_quant_pattern': ['yolo_output'], 'quantize_op_types': ['depthwise_conv2d', 'mul', 'conv2d'], 'dtype': 'int8', 'window_size': 10000, 'moving_rate': 0.9, 'for_tensorrt': False, 'is_full_quantize': False}
2020-09-11 18:51:19,065-INFO: quant_aware config {'weight_quantize_type': 'channel_wise_abs_max', 'activation_quantize_type': 'moving_average_abs_max', 'weight_bits': 8, 'activation_bits': 8, 'not_quant_pattern': ['yolo_output'], 'quantize_op_types': ['depthwise_conv2d', 'mul', 'conv2d'], 'dtype': 'int8', 'window_size': 10000, 'moving_rate': 0.9, 'for_tensorrt': False, 'is_full_quantize': False}
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
2020-09-11 18:51:29,143-INFO: places would be ommited when DataLoader is not iterable
I0911 18:51:30.399793  4074 graph_pattern_detector.cc:100] ---  detected 96 subgraphs
I0911 18:51:30.555467  4074 graph_pattern_detector.cc:100] ---  detected 16 subgraphs
I0911 18:51:30.598870  4074 graph_pattern_detector.cc:100] ---  detected 16 subgraphs
/home/aistudio/work/PaddleDetection/ppdet/data/parallel_map.py:243: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  .format(sample.errmsg))
2020-09-11 18:51:37,084-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-13] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:37,176-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-15] exits for reason[consumer[consumer-5a2-13] failed to map with error:[target0 not in samples]]]
2020-09-11 18:51:37,484-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-2] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:37,874-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-7] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:37,974-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-6] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:38,283-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-3] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:38,873-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-14] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:39,382-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-11] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:39,574-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-4] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:39,786-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-5] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:40,434-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-0] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:40,529-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-10] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:41,022-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-12] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:41,277-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-1] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:41,383-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-9] failed to map with error:[target0 not in samples]]
2020-09-11 18:51:41,870-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5a2-8] failed to map with error:[target0 not in samples]]
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py:1145: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
  logging.warn('Your reader has raised an exception!')
2020-09-11 18:51:41,870-WARNING: Your reader has raised an exception!
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1146, in __thread_main__
    six.reraise(*sys.exc_info())
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1126, in __thread_main__
    for tensors in self._tensor_reader():
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1196, in __tensor_reader_impl__
    for slots in paddle_reader():
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 523, in __reader_creator__
    for item in reader():
  File "/home/aistudio/work/PaddleDetection/ppdet/data/reader.py", line 452, in _reader
    reader.reset()
  File "/home/aistudio/work/PaddleDetection/ppdet/data/parallel_map.py", line 259, in reset
    assert not self._exit, "cannot reset for already stopped dataset"
AssertionError: cannot reset for already stopped dataset

Traceback (most recent call last):
  File "slim/quantization/train.py", line 317, in <module>
    main()
  File "slim/quantization/train.py", line 241, in main
    outs = exe.run(compiled_train_prog, fetch_list=train_values)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 1082, in run
    six.reraise(*sys.exc_info())
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 1080, in run
    return_merged=return_merged)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 1178, in _run_impl
    return_merged=return_merged)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 893, in _run_parallel
    tensors = exe.run(fetch_var_names, return_merged)._move_to_list()
paddle.fluid.core_noavx.EnforceNotMet: 

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2   paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> >*)
3   paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocator<paddle::framework::LoDTensor> >*)
4   std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<unsigned long>, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&)
5   std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
6   ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2798, in append_op
    attrs=kwargs.get("attrs", None))
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1081, in _init_non_iterable
    attrs={'drop_last': self._drop_last})
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 979, in __init__
    self._init_non_iterable()
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 621, in from_generator
    iterable, return_list, drop_last)
  File "/home/aistudio/work/PaddleDetection/ppdet/modeling/architectures/yolo.py", line 155, in build_inputs
    iterable=iterable) if use_dataloader else None
  File "slim/quantization/train.py", line 110, in main
    feed_vars, train_loader = model.build_inputs(**inputs_def)
  File "slim/quantization/train.py", line 317, in <module>
    main()

----------------------
Error Message Summary:
----------------------
FatalError: Blocking queue is killed because the data reader raises an exception.
  [Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:154)
  [operator < read > error]

看错误是 reader 出错， 不明白怎么解决，请大佬指导！

把参数 pretrain_weights 改成 weights 不行 pull了最新版的 PaddleDetection ，不行 换成量化前的，也不行