C++ CreatePaddlePredictor加载模型出core
Created by: HrBlack
-
版本、环境信息: 1)PaddlePaddle版本:paddle@paddle_with_lite_1-7-2-22-xpu-turing_PD_BL@git_tag 2)GPU:Tesla T4, CUDA:V10.2.89
-
问题描述: ernie模型训练完后,利用io.save_inference_model保存了预测模型,然后用c++ CreatePaddlePredictor加载时出core;用python加载模型正常。 训练的paddle版本尝试过1.6.3与1.8.1,两种模型均会出core。
-报错信息: WARNING: Logging before InitGoogleLogging() is written to STDERR I0812 01:19:46.660524 29227 analysis_predictor.cc:140] Profiler is deactivated, and no profiling report will be generated. I0812 01:19:46.666958 29227 analysis_predictor.cc:908] MODEL VERSION: 1.8.1 I0812 01:19:46.667003 29227 analysis_predictor.cc:910] PREDICTOR VERSION: 0.0.0 W0812 01:19:46.667055 29227 analysis_predictor.cc:923] - Version incompatible (1) concat W0812 01:19:46.667075 29227 analysis_predictor.cc:923] - Version incompatible (1) cos_sim W0812 01:19:46.667081 29227 analysis_predictor.cc:923] - Version incompatible (1) dropout W0812 01:19:46.667088 29227 analysis_predictor.cc:923] - Version incompatible (1) elementwise_add W0812 01:19:46.667093 29227 analysis_predictor.cc:923] - Version incompatible (1) elementwise_sub W0812 01:19:46.667109 29227 analysis_predictor.cc:923] - Version incompatible (1) feed W0812 01:19:46.667114 29227 analysis_predictor.cc:923] - Version incompatible (1) fetch W0812 01:19:46.667122 29227 analysis_predictor.cc:923] - Version incompatible (1) fill_constant_batch_size_like W0812 01:19:46.667129 29227 analysis_predictor.cc:923] - Version incompatible (3) layer_norm W0812 01:19:46.667135 29227 analysis_predictor.cc:923] - Version incompatible (1) lookup_table W0812 01:19:46.667141 29227 analysis_predictor.cc:923] - Version incompatible (2) matmul W0812 01:19:46.667147 29227 analysis_predictor.cc:923] - Version incompatible (2) mul W0812 01:19:46.667153 29227 analysis_predictor.cc:923] - Version incompatible (1) relu W0812 01:19:46.667160 29227 analysis_predictor.cc:923] - Version incompatible (2) reshape2 W0812 01:19:46.667166 29227 analysis_predictor.cc:923] - Version incompatible (1) scale W0812 01:19:46.667172 29227 analysis_predictor.cc:923] - Version incompatible (2) slice W0812 01:19:46.667179 29227 analysis_predictor.cc:923] - Version incompatible (1) softmax W0812 01:19:46.667186 29227 analysis_predictor.cc:923] - Version incompatible (1) stack W0812 01:19:46.667191 29227 analysis_predictor.cc:923] - Version incompatible (1) tanh W0812 01:19:46.667197 29227 analysis_predictor.cc:923] - Version incompatible (1) transpose2 W0812 01:19:46.667204 29227 analysis_predictor.cc:196] WARNING: Results may be DIFF! Please use the corresponding version of the model and prediction library, and do not use the develop branch. --- Running analysis [ir_graph_build_pass] --- Running analysis [ir_graph_clean_pass] --- Running analysis [ir_analysis_pass] --- Running IR pass [is_test_pass] --- Running IR pass [simplify_with_basic_ops_pass] --- Running IR pass [conv_affine_channel_fuse_pass] --- Running IR pass [conv_eltwiseadd_affine_channel_fuse_pass] --- Running IR pass [conv_bn_fuse_pass] --- Running IR pass [conv_eltwiseadd_bn_fuse_pass] --- Running IR pass [embedding_eltwise_layernorm_fuse_pass] I0812 01:19:46.781428 29227 graph_pattern_detector.cc:101] --- detected 2 subgraphs I0812 01:19:46.781893 29227 graph_pattern_detector.cc:101] --- detected 2 subgraphs I0812 01:19:46.782658 29227 graph_pattern_detector.cc:101] --- detected 12 subgraphs --- Running IR pass [multihead_matmul_fuse_pass_v2] I0812 01:19:46.795693 29227 graph_pattern_detector.cc:101] --- detected 6 subgraphs Segmentation fault (core dumped)
-core堆栈信息: (gdb) bt #0 0x00007f2c08903104 in paddle::framework::LoDTensor* paddle::framework::Variable::GetMutablepaddle::framework::LoDTensor() () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #1 (closed) 0x00007f2c08a60595 in paddle::framework::ir::patterns::BuildFusionV2(paddle::framework::ir::Graph*, std::string const&, paddle::framework::Scope*)::{lambda(std::unordered_map<paddle::framework::ir::PDNode*, paddle::framework::ir::Node*, std::hashpaddle::framework::ir::PDNode*, std::equal_topaddle::framework::ir::PDNode*, std::allocator<std::pair<paddle::framework::ir::PDNode* const, paddle::framework::ir::Node*> > > const&, paddle::framework::ir::Graph*)#2 (closed)}::operator()(std::unordered_map<paddle::framework::ir::PDNode*, paddle::framework::ir::Node*, std::hashpaddle::framework::ir::PDNode*, std::equal_topaddle::framework::ir::PDNode*, std::allocator<std::pair<paddle::framework::ir::PDNode* const, paddle::framework::ir::Node*> > > const&, paddle::framework::ir::Graph*) const [clone .isra.781] () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #2 (closed) 0x00007f2c0b63a135 in paddle::framework::ir::GraphPatternDetector::operator()(paddle::framework::ir::Graph*, std::function<void (std::unordered_map<paddle::framework::ir::PDNode*, paddle::framework::ir::Node*, std::hashpaddle::framework::ir::PDNode*, std::equal_topaddle::framework::ir::PDNode*, std::allocator<std::pair<paddle::framework::ir::PDNode* const, paddle::framework::ir::Node*> > > const&, paddle::framework::ir::Graph*)>) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #3 (closed) 0x00007f2c08a598ef in paddle::framework::ir::MultiHeadMatmulV2FusePass::ApplyImpl(paddle::framework::ir::Graph*) const () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #4 (closed) 0x00007f2c0b64b325 in paddle::framework::ir::Pass::Apply(paddle::framework::ir::Graph*) const () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #5 (closed) 0x00007f2c0b599ba6 in paddle::inference::analysis::IRPassManager::Apply(std::unique_ptr<paddle::framework::ir::Graph, std::default_deletepaddle::framework::ir::Graph >) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #6 (closed) 0x00007f2c0b580fa9 in paddle::inference::analysis::IrAnalysisPass::RunImpl(paddle::inference::analysis::Argument*) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #7 (closed) 0x00007f2c0b4fc4f5 in paddle::inference::analysis::Analyzer::RunAnalysis(paddle::inference::analysis::Argument*) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #8 (closed) 0x00007f2c08926394 in paddle::AnalysisPredictor::OptimizeInferenceProgram() () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #9 (closed) 0x00007f2c08926cef in paddle::AnalysisPredictor::PrepareProgram(std::shared_ptrpaddle::framework::ProgramDesc const&) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #10 (closed) 0x00007f2c08926e6f in paddle::AnalysisPredictor::Init(std::shared_ptrpaddle::framework::Scope const&, std::shared_ptrpaddle::framework::ProgramDesc const&) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #11 0x00007f2c08927075 in std::unique_ptr<paddle::PaddlePredictor, std::default_deletepaddle::PaddlePredictor > paddle::CreatePaddlePredictor<paddle::AnalysisConfig, (paddle::PaddleEngineKind)2>(paddle::AnalysisConfig const&) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #12 (closed) 0x00007f2c08927231 in std::unique_ptr<paddle::PaddlePredictor, std::default_deletepaddle::PaddlePredictor > paddle::CreatePa--Type for more, q to quit, c to continue without paging-- ddlePredictorpaddle::AnalysisConfig(paddle::AnalysisConfig const&) () from /home/work/liushihao02/output/bin/../lib/libpaddle_fluid.so #13 (closed) 0x0000000000696874 in tarot::agent::PaddleEngine::init (this=, root_path=..., config=...) at /home/opt/compiler/gcc-8.2/gcc-8.2/include/c++/8.2.0/bits/unique_ptr.h:342 #14 (closed) 0x0000000000686979 in tarot::agent::Model::init (this=this@entry=0x7ffdc0ea7b70, root_path=...) at bc_out/baidu/model-service/tarot/src/proto/tarot_config.pb.h:5248 #15 0x0000000000481a75 in tarot::run_infer (batch_size=32) at baidu/model-service/tarot/src/tools/local_inference_tool.cpp:355 #16 (closed) 0x0000000000463378 in main (argc=, argv=) at baidu/model-service/tarot/src/tools/local_inference_tool.cpp:508