Created by: sfraczek
This PR adds support for int8 mkldnn prior_box operator.
This operator has the same code. After quantizing prior_box, the layout/dimensions and data type of input input
data changes to int8 nhwc. By introducing this operator we can skip dequantization of input input
because only the dimensions of it are read.
The code worked when I created a separate file for mkldnn int8 kernel but since it was the same code as the reference, I decided to put it in one file. Before calibration the kernel has type <float, float>. After calibration it is <unsigned char, float>. When I added it as a single file, it started crashing after calibration at line 183:
182 framework::Tensor var_t;
183 var_t.mutable_data<K>(
184 framework::make_ddim({1, static_cast<int>(variances.size())}),
185 ctx.GetPlace());
186 auto var_et = framework::EigenTensor<K, 2>::From(var_t);
Digging deeper in the stack it crashes at:
The backtrace file: gdb.txt
Is this a problem with PaddlePaddle or did I do something wrong here?