训练出core(an illegal memory access)
Created by: zhengzhe97
terminate called after throwing an instance of 'paddle::platform::EnforceNotMet' what():
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::framework::details::OpHandleBase::~OpHandleBase() 3 paddle::framework::details::FetchOpHandle::~FetchOpHandle() 4 paddle::framework::ir::Node::~Node() 5 paddle::framework::ir::Node::~Node() 6 paddle::framework::details::ClearFetchOp(paddle::framework::ir::Graph*, std::vector<paddle::framework::details::OpHandleBase*, std::allocatorpaddle::framework::details::OpHandleBase* >) 7 paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal(std::vector<paddle::framework::details::OpHandleBase, std::allocatorpaddle::framework::details::OpHandleBase* >*) 8 paddle::framework::details::FastThreadedSSAGraphExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&) 9 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&) 10 paddle::framework::ParallelExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&)
Error Message Summary:
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
- New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
- Recommended issue content: all error stack information: an illegal memory access was encountered at (/paddle/paddle/fluid/framework/details/op_handle_base.cc:39)
W0322 21:57:18.223031 11077 init.cc:209] Warning: PaddlePaddle catches a failure signal, it may not work properly W0322 21:57:18.223071 11077 init.cc:211] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0322 21:57:18.223075 11077 init.cc:214] The detail failure signal is:
W0322 21:57:18.223084 11077 init.cc:217] *** Aborted at 1584885438 (unix time) try "date -d @1584885438" if you are using GNU date *** W0322 21:57:18.225430 11077 init.cc:217] PC: @ 0x0 (unknown) W0322 21:57:18.225522 11077 init.cc:217] *** SIGABRT (@0x3e800002b45) received by PID 11077 (TID 0x7ff9706be700) from PID 11077; stack trace: *** W0322 21:57:18.227619 11077 init.cc:217] @ 0x7ff9702a2390 (unknown) W0322 21:57:18.229521 11077 init.cc:217] @ 0x7ff96fefc428 gsignal W0322 21:57:18.231348 11077 init.cc:217] @ 0x7ff96fefe02a abort W0322 21:57:18.232658 11077 init.cc:217] @ 0x7ff930e6784a __gnu_cxx::__verbose_terminate_handler() W0322 21:57:18.233703 11077 init.cc:217] @ 0x7ff930e65f47 __cxxabiv1::__terminate() W0322 21:57:18.234709 11077 init.cc:217] @ 0x7ff930e653a5 __cxa_call_terminate W0322 21:57:18.235827 11077 init.cc:217] @ 0x7ff930e65bd8 __gxx_personality_v0 W0322 21:57:18.236817 11077 init.cc:217] @ 0x7ff931158aab _Unwind_RaiseException_Phase2 W0322 21:57:18.237833 11077 init.cc:217] @ 0x7ff931158f49 _Unwind_Resume W0322 21:57:18.241801 11077 init.cc:217] @ 0x7ff90c6b4cdc paddle::framework::details::OpHandleBase::~OpHandleBase() W0322 21:57:18.244470 11077 init.cc:217] @ 0x7ff90c66f151 paddle::framework::details::FetchOpHandle::~FetchOpHandle() W0322 21:57:18.247952 11077 init.cc:217] @ 0x7ff90a0afee9 paddle::framework::ir::Node::~Node() W0322 21:57:18.252969 11077 init.cc:217] @ 0x7ff90a0b0091 paddle::framework::ir::Node::~Node() W0322 21:57:18.255545 11077 init.cc:217] @ 0x7ff90c671a96 paddle::framework::details::ClearFetchOp() W0322 21:57:18.257541 11077 init.cc:217] @ 0x7ff90c66da8a paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal() W0322 21:57:18.263303 11077 init.cc:217] @ 0x7ff90c66c66a paddle::framework::details::FastThreadedSSAGraphExecutor::Run() W0322 21:57:18.265338 11077 init.cc:217] @ 0x7ff90c5b6c10 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run() W0322 21:57:18.269490 11077 init.cc:217] @ 0x7ff90a1af848 paddle::framework::ParallelExecutor::Run() W0322 21:57:18.269999 11077 init.cc:217] @ 0x7ff909dc9f58 ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9framework16ParallelExecutorERKSt6vectorISsSaISsEEE199_S9_INS6_9LoDTensorESaISF_EEIS8_SD_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESY W0322 21:57:18.271737 11077 init.cc:217] @ 0x7ff909e1eb21 pybind11::cpp_function::dispatcher() W0322 21:57:18.272274 11077 init.cc:217] @ 0x557b56793744 _PyMethodDef_RawFastCallKeywords W0322 21:57:18.272737 11077 init.cc:217] @ 0x557b56793861 _PyCFunction_FastCallKeywords W0322 21:57:18.273205 11077 init.cc:217] @ 0x557b567ff6e8 _PyEval_EvalFrameDefault W0322 21:57:18.273622 11077 init.cc:217] @ 0x557b56743539 _PyEval_EvalCodeWithName W0322 21:57:18.274013 11077 init.cc:217] @ 0x557b56792f57 _PyFunction_FastCallKeywords W0322 21:57:18.274472 11077 init.cc:217] @ 0x557b567fb8cc _PyEval_EvalFrameDefault W0322 21:57:18.274893 11077 init.cc:217] @ 0x557b56743539 _PyEval_EvalCodeWithName W0322 21:57:18.275295 11077 init.cc:217] @ 0x557b56792f57 _PyFunction_FastCallKeywords W0322 21:57:18.275760 11077 init.cc:217] @ 0x557b567fb8cc _PyEval_EvalFrameDefault W0322 21:57:18.276190 11077 init.cc:217] @ 0x557b56743539 _PyEval_EvalCodeWithName W0322 21:57:18.276579 11077 init.cc:217] @ 0x557b56792f57 _PyFunction_FastCallKeywords W0322 21:57:18.277034 11077 init.cc:217] @ 0x557b567fb8cc _PyEval_EvalFrameDefault Aborted (core dumped)