预测时用load_persistables载入模型出错
Created by: emy-085
- 标题：预测时用load_inference_model可以正常预测，用load_persistables会报错
- 版本、环境信息： 1）PaddlePaddle版本：1.5开发版：8cb48e5a 2）GPU：K40, cuda9.0，cudnn_v7.3 3）系统环境：centos
- 问题描述：训练模型时我同时用save_inference_model和save_persistables保存模型参数，这是得到的参数文件(最后一个是save_persistables的结果)：
如下边代码中所示，用load_inference_model可以正常预测，用load_persistables会报错（因为我save_inference_model时只保存了模型计算图的一部分，所以这里必须要用load_persistables去预测）。
================================================================== 详细日志： define model over ... W0922 12:09:06.581745 29803 device_context.cc:198] Please NOTE: device: 0, CUDA Capability: 35, Driver API Version: 9.2, Runtime API Version: 8.0 W0922 12:09:06.587692 29803 device_context.cc:206] device: 0, cuDNN Version: 7.3. !!!A non-EOF exception is thrown. Traceback (most recent call last): File "test.py", line 324, in main() File "test.py", line 284, in main main_program=None, filename='lomoDR_resnet50_iter_80000') File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/io.py", line 771, in load_persistables filename=filename) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/io.py", line 634, in load_vars filename=filename) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/io.py", line 672, in load_vars executor.run(load_prog) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/executor.py", line 617, in run six.reraise(sys.exc_info()) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/executor.py", line 613, in run use_program_cache=use_program_cache) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/executor.py", line 654, in _run_impl use_program_cache=use_program_cache) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/executor.py", line 731, in _run_program fetch_var_name) paddle.fluid.core_avx.EnforceNotMet: Invoke operator load_combine error. Python Call stacks: File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1824, in append_op attrs=kwargs.get("attrs", None)) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/io.py", line 671, in load_vars attrs={'file_path': os.path.join(load_dirname, filename)}) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/io.py", line 634, in load_vars filename=filename) File "/home/vis/enmengyi/software/anaconda2_paddle1.5_latest_cuda9.0_cudnn7.3_mkl/lib/python2.7/site-packages/paddle/fluid/io.py", line 771, in load_persistables filename=filename) File "test.py", line 284, in main main_program=None, filename='lomoDR_resnet50_iter_80000') File "test.py", line 324, in main() C++ Call stacks: You are not allowed to load partial data via load_combine_op, use load_op instead. at [/home/vis/enmengyi/paddle_env/Paddle/paddle/fluid/operators/load_combine_op.h:105] PaddlePaddle Call Stacks: 0 0x7f1faa32a799p std::string paddle::platform::GetTraceBackString<char const>(char const*&&, char const*, int) + 361 1 0x7f1faa32aacap paddle::platform::EnforceNotMet::EnforceNotMet(std::exception_ptr::exception_ptr, char const*, int) + 154 2 0x7f1faa87d5cfp paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, float>::LoadParamsFromBuffer(paddle::framework::ExecutionContext const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, std::istream*, bool, std::vector<std::string, std::allocatorstd::string > const&) const + 1887 3 0x7f1faa87db00p paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const + 1248 4 0x7f1faa87e043p std::Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, int>, paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, signed char>, paddle::operators::LoadCombineOpKernel<paddle::platform::CUDADeviceContext, long> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1 (closed)}>::M_invoke(std::Any_data const&, paddle::framework::ExecutionContext const&) + 35 5 0x7f1fac31a59bp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 411 6 0x7f1fac31ab91p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529 7 0x7f1fac315403p paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 307 8 0x7f1faa4bbf86p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 294 9 0x7f1faa4beaefp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool) + 143 10 0x7f1faa31b9bdp 11 0x7f1faa35d47cp 12 0x7f1fe80fa577p PyEval_EvalFrameEx + 28695 13 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 14 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 15 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 16 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 17 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 18 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 19 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 20 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 21 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 22 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 23 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 24 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 25 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 26 0x7f1fe80f9f68p PyEval_EvalFrameEx + 27144 27 0x7f1fe80fca99p PyEval_EvalCodeEx + 2025 28 0x7f1fe80fccbap PyEval_EvalCode + 26 29 0x7f1fe811601dp 30 0x7f1fe81171c8p PyRun_FileExFlags + 120 31 0x7f1fe81183e8p PyRun_SimpleFileExFlags + 232 32 0x7f1fe812a67cp Py_Main + 2988 33 0x7f1fe7313b45p __libc_start_main + 245 34 0x7f1fe81fc07fp
=================================================================== 预测代码(已省略无关代码)：
def main(argv=None):
"""Main function."""
try:
os.makedirs(ARGS.result_path)
except OSError as e:
if e.errno != 17:
raise
# model definition #
input_image = fluid.layers.data(name='image', shape=[3, 512, 512], dtype='float32')
# backbone
if ARGS.backbone == 'Resnet50':
f_common, f_score, f_geo = resnet.Resnet_East(input_image, 50)
elif ARGS.backbone == 'Resnet50-official':
f_common, f_score, f_geo = resnet.Resnet_East_new(input_image, 50)
elif ARGS.backbone == 'Inception-V4-fpn':
f_common, f_score, f_geo = inception.InceptionV4_East(input_image, 'fpn')
elif ARGS.backbone == 'Inception-V4-wnet':
f_common, f_score, f_geo = inception.InceptionV4_East(input_image, 'wnet')
else:
print('Unknown backbone type!')
exit(-1)
# DR
det_proposals = create_tmp_var(name='det_proposals_var', dtype="float32", shape=[-1, 9])
det_proposals_noscore = create_tmp_var(name='det_proposals_noscore_var', dtype="float32", shape=[-1, 8])
det_proposals, det_proposals_noscore = fluid.layers.py_func(func=detect,
x=[f_score, f_geo],
out=[det_proposals, det_proposals_noscore])
print("define model over ...")
#load model
place = fluid.CUDAPlace(0) if ARGS.use_gpu else fluid.CPUPlace()
exe = fluid.Executor(place)
if ARGS.model_dir:
#fluid.io.load_inference_model(dirname=ARGS.model_dir,
# executor=exe,
# model_filename=ARGS.model_file,
# params_filename=ARGS.params_file)
fluid.io.load_persistables(executor=exe, dirname=ARGS.model_dir,
main_program=None, filename='lomoDR_resnet50_iter_80000')
print("load model over ...")
#loop for infer test images
im_fn_list = get_images()
print("test image number: {}".format(len(im_fn_list)))
for im_fn in im_fn_list:
print('image_name: {}'.format(im_fn))
src_im = cv2.imread(im_fn)
print("src_im: {}".format(src_im.shape))
start_time = time.time()
im_resized = src_im.copy()
im_resized, (ratio_h, ratio_w) = resize_image(im_resized, max_side_len=ARGS.input_size)
im_resized = im_resized.astype(dtype=np.float32)
print("im_resized: {}".format(im_resized.shape))
im_resized.astype(dtype=np.float32)
#whitening
im_resized[:, :, 2] -= 0.485 * 255
im_resized[:, :, 1] -= 0.456 * 255
im_resized[:, :, 0] -= 0.406 * 255
im_resized[:, :, 2] /= (255.0 * 0.229)
im_resized[:, :, 1] /= (255.0 * 0.224)
im_resized[:, :, 0] /= (255.0 * 0.225)
im_resized = np.swapaxes(im_resized, 1, 2)
im_resized = np.swapaxes(im_resized, 1, 0) # C x H x W
im_resized = im_resized[::-1, :, :]
#img to tensor
img_tensor = core.LoDTensor()
img_data = im_resized[np.newaxis, :, :, :]
img_tensor.set(img_data, place)
#forward
test_program = fluid.default_main_program().clone(for_test=True)
score, boxes = exe.run(test_program,
feed={'image': img_tensor},
fetch_list=[f_score, det_proposals], return_numpy=False)
if __name__ == '__main__':
main()