Created by: chenwhql
PR types
Others
PR changes
Others
Describe
-
Exception reset mechanism modification: currently in PE, multiple exceptions may occured in multiple threads, and the exception that occur later will overwrite that occur earlier, but what we want should be the first triggered exception . However, EOF exception is lower priority exception and can be overwritten, but other exceptions should not be prioritized.
-
In the ExceptionFinal function of PE, before the exception is thrown, the ClearFetchOp operation must be performed. However, If a new exception occurs in this ClearFetchOp operation, it will cause the loss of exception triggered firstly not thrown. Instead, the cleanup operation should only be performed when an EOF exception is caught. If other exceptions are triggered, the ClearFetchOp should not be continued.
original: only have the exception triggered in ClearFetchOp
terminate called after throwing an instance of 'paddle::platform::EnforceNotMet'
what():
----
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2 paddle::framework::details::OpHandleBase::~OpHandleBase()
3 paddle::framework::details::FetchOpHandle::~FetchOpHandle()
4 paddle::framework::ir::Node::~Node()
5 paddle::framework::ir::Node::~Node()
6 paddle::framework::details::ClearFetchOp(paddle::framework::ir::Graph*, std::vector<paddle::framework::details::OpHandleBase*, std::allocator<paddle::framework::details::OpHandleBase*> >*)
7 paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal(std::vector<paddle::framework::details::OpHandleBase*, std::allocator<paddle::framework::details::OpHandleBase*> >*)
8 paddle::framework::details::FastThreadedSSAGraphExecutor::Run(std::vector<std::string, std::allocator<std::string> > const&, bool)
9 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run(std::vector<std::string, std::allocator<std::string> > const&, bool)
10 paddle::framework::ParallelExecutor::Run(std::vector<std::string, std::allocator<std::string> > const&, bool)
----------------------
Error Message Summary:
----------------------
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
- New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
- Recommended issue content: all error stack information at (/work/paddle_py2/paddle/fluid/framework/details/op_handle_base.cc:39)
W0708 17:20:34.752036 3615 init.cc:231] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0708 17:20:34.752061 3615 init.cc:233] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0708 17:20:34.752068 3615 init.cc:236] The detail failure signal is:
W0708 17:20:34.752074 3615 init.cc:239] *** Aborted at 1594200034 (unix time) try "date -d @1594200034" if you are using GNU date ***
W0708 17:20:34.753860 3615 init.cc:239] PC: @ 0x0 (unknown)
W0708 17:20:34.754398 3615 init.cc:239] *** SIGABRT (@0x1f900000e1f) received by PID 3615 (TID 0x7f37c5bcb700) from PID 3615; stack trace: ***
W0708 17:20:34.755846 3615 init.cc:239] @ 0x7f37c5382160 (unknown)
W0708 17:20:34.758002 3615 init.cc:239] @ 0x7f37c48f03f7 __GI_raise
W0708 17:20:34.759413 3615 init.cc:239] @ 0x7f37c48f17d8 __GI_abort
W0708 17:20:34.761421 3615 init.cc:239] @ 0x7f37334d6c65 __gnu_cxx::__verbose_terminate_handler()
W0708 17:20:34.762082 3615 init.cc:239] @ 0x7f37334d4e06 __cxxabiv1::__terminate()
W0708 17:20:34.762688 3615 init.cc:239] @ 0x7f37334d3ec9 __cxa_call_terminate
W0708 17:20:34.763335 3615 init.cc:239] @ 0x7f37334d4a7a __gxx_personality_v0
W0708 17:20:34.764647 3615 init.cc:239] @ 0x7f37713bf853 _Unwind_RaiseException_Phase2
W0708 17:20:34.765425 3615 init.cc:239] @ 0x7f37713bfd87 _Unwind_Resume
W0708 17:20:34.769990 3615 init.cc:239] @ 0x7f371b5ebed8 paddle::framework::details::OpHandleBase::~OpHandleBase()
W0708 17:20:34.774236 3615 init.cc:239] @ 0x7f371b56cda1 paddle::framework::details::FetchOpHandle::~FetchOpHandle()
W0708 17:20:34.778376 3615 init.cc:239] @ 0x7f3717bcc819 paddle::framework::ir::Node::~Node()
W0708 17:20:34.784080 3615 init.cc:239] @ 0x7f3717bcc9c1 paddle::framework::ir::Node::~Node()
W0708 17:20:34.786631 3615 init.cc:239] @ 0x7f371b5701c6 paddle::framework::details::ClearFetchOp()
W0708 17:20:34.788537 3615 init.cc:239] @ 0x7f371b56b1da paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal()
W0708 17:20:34.792685 3615 init.cc:239] @ 0x7f371b56a1ea paddle::framework::details::FastThreadedSSAGraphExecutor::Run()
W0708 17:20:34.793941 3615 init.cc:239] @ 0x7f371b4bb4dc _ZZN6paddle9framework7details29ScopeBufferedSSAGraphExecutor3RunERKSt6vectorISsSaISsEEbENKUlvE_clEv
W0708 17:20:34.798692 3615 init.cc:239] @ 0x7f371b4bc5c5 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run()
W0708 17:20:34.802013 3615 init.cc:239] @ 0x7f3717ce2ece paddle::framework::ParallelExecutor::Run()
W0708 17:20:34.802249 3615 init.cc:239] @ 0x7f37178dac78 _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9framework16ParallelExecutorERKSt6vectorISsSaISsEEbE213_NS_6objectEIS8_SD_bEINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESW_
W0708 17:20:34.803839 3615 init.cc:239] @ 0x7f371792e0b1 pybind11::cpp_function::dispatcher()
W0708 17:20:34.807186 3615 init.cc:239] @ 0x7f37c569bbb8 PyEval_EvalFrameEx
W0708 17:20:34.808738 3615 init.cc:239] @ 0x7f37c569f0bd PyEval_EvalCodeEx
W0708 17:20:34.810165 3615 init.cc:239] @ 0x7f37c569c345 PyEval_EvalFrameEx
W0708 17:20:34.811570 3615 init.cc:239] @ 0x7f37c569f0bd PyEval_EvalCodeEx
W0708 17:20:34.812960 3615 init.cc:239] @ 0x7f37c569c345 PyEval_EvalFrameEx
W0708 17:20:34.814452 3615 init.cc:239] @ 0x7f37c569f0bd PyEval_EvalCodeEx
W0708 17:20:34.815867 3615 init.cc:239] @ 0x7f37c569c345 PyEval_EvalFrameEx
W0708 17:20:34.817335 3615 init.cc:239] @ 0x7f37c569f0bd PyEval_EvalCodeEx
W0708 17:20:34.818759 3615 init.cc:239] @ 0x7f37c569c345 PyEval_EvalFrameEx
W0708 17:20:34.820199 3615 init.cc:239] @ 0x7f37c569f0bd PyEval_EvalCodeEx
W0708 17:20:34.821599 3615 init.cc:239] @ 0x7f37c569f1f2 PyEval_EvalCode
new：also have the exception triggered in PE
Traceback (most recent call last):
File "./train.py", line 583, in <module>
train(args)
File "./train.py", line 480, in train
ret = train_exe.run(fetch_list=fetch_list, program=train_program)
File "/home/chenweihang/develop/scripts/erine_cuda_error_77/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1079, in run
six.reraise(*sys.exc_info())
File "/home/chenweihang/develop/scripts/erine_cuda_error_77/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1074, in run
return_merged=return_merged)
File "/home/chenweihang/develop/scripts/erine_cuda_error_77/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 1175, in _run_impl
return_merged=return_merged)
File "/home/chenweihang/develop/scripts/erine_cuda_error_77/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 887, in _run_parallel
tensors = exe.run(fetch_var_names, return_merged)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:
--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2 paddle::platform::stream::CUDAStream::Wait() const
3 paddle::platform::CUDADeviceContext::Wait() const
4 paddle::framework::details::ScopeBufferedSSAGraphExecutor::DropLocalExeScopes()
5 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run(std::vector<std::string, std::allocator<std::string> > const&, bool)
6 paddle::framework::ParallelExecutor::Run(std::vector<std::string, std::allocator<std::string> > const&, bool)
----------------------
Error Message Summary:
----------------------
ExternalError: Cuda error(77), an illegal memory access was encountered.
[Advise: The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] at (/work/paddle_py2/paddle/fluid/platform/stream/cuda_stream.cc:65)
terminate called after throwing an instance of 'paddle::platform::EnforceNotMet'
what():
--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2 paddle::framework::details::OpHandleBase::~OpHandleBase()
3 paddle::framework::details::FetchOpHandle::~FetchOpHandle()
4 paddle::framework::ir::Node::~Node()
5 paddle::framework::ir::Node::~Node()
6 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
7 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
8 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
9 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
10 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
11 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
12 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
13 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
14 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
15 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
16 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
17 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
18 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
19 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
20 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
21 std::_Rb_tree<paddle::framework::ir::Node*, std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > >, std::_Select1st<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >, std::less<paddle::framework::ir::Node*>, std::allocator<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > > >::_M_erase(std::_Rb_tree_node<std::pair<paddle::framework::ir::Node* const, std::unique_ptr<paddle::framework::ir::Node, std::default_delete<paddle::framework::ir::Node> > > >*)
22 paddle::framework::ir::Graph::~Graph()
23 paddle::framework::ir::Graph::~Graph()
24 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
----------------------
Error Message Summary:
----------------------
ExternalError: Cuda error(77), an illegal memory access was encountered.
[Advise: The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] at (/work/paddle_py2/paddle/fluid/framework/details/op_handle_base.cc:39)
W0708 20:21:02.116802 10154 init.cc:231] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0708 20:21:02.116840 10154 init.cc:233] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0708 20:21:02.116847 10154 init.cc:236] The detail failure signal is:
W0708 20:21:02.116852 10154 init.cc:239] *** Aborted at 1594210862 (unix time) try "date -d @1594210862" if you are using GNU date ***
W0708 20:21:02.118594 10154 init.cc:239] PC: @ 0x0 (unknown)
W0708 20:21:02.118677 10154 init.cc:239] *** SIGABRT (@0x1f9000027aa) received by PID 10154 (TID 0x7fb0c7853700) from PID 10154; stack trace: ***
W0708 20:21:02.120041 10154 init.cc:239] @ 0x7fb0c700a160 (unknown)
W0708 20:21:02.121485 10154 init.cc:239] @ 0x7fb0c65783f7 __GI_raise
W0708 20:21:02.122866 10154 init.cc:239] @ 0x7fb0c65797d8 __GI_abort
W0708 20:21:02.123625 10154 init.cc:239] @ 0x7fb03515ec65 __gnu_cxx::__verbose_terminate_handler()
W0708 20:21:02.124253 10154 init.cc:239] @ 0x7fb03515ce06 __cxxabiv1::__terminate()
W0708 20:21:02.124856 10154 init.cc:239] @ 0x7fb03515bec9 __cxa_call_terminate
W0708 20:21:02.125496 10154 init.cc:239] @ 0x7fb03515ca7a __gxx_personality_v0
W0708 20:21:02.126250 10154 init.cc:239] @ 0x7fb073047853 _Unwind_RaiseException_Phase2
W0708 20:21:02.126994 10154 init.cc:239] @ 0x7fb073047d87 _Unwind_Resume
W0708 20:21:02.131428 10154 init.cc:239] @ 0x7fb01d26d5c0 paddle::framework::details::OpHandleBase::~OpHandleBase()
W0708 20:21:02.134122 10154 init.cc:239] @ 0x7fb01d1ed151 paddle::framework::details::FetchOpHandle::~FetchOpHandle()
W0708 20:21:02.137547 10154 init.cc:239] @ 0x7fb01984c7b9 paddle::framework::ir::Node::~Node()
W0708 20:21:02.142817 10154 init.cc:239] @ 0x7fb01984c961 paddle::framework::ir::Node::~Node()
W0708 20:21:02.148023 10154 init.cc:239] @ 0x7fb019835e12 std::_Rb_tree<>::_M_erase()
W0708 20:21:02.153007 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.158121 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.163043 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.167941 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.173005 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.177913 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.182853 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.187737 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.192651 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.197572 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.202462 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.207379 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.212318 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.217224 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.222121 10154 init.cc:239] @ 0x7fb019835dff std::_Rb_tree<>::_M_erase()
W0708 20:21:02.224093 10154 init.cc:239] @ 0x7fb01d4ab26e paddle::framework::ir::Graph::~Graph()
W0708 20:21:02.228607 10154 init.cc:239] @ 0x7fb01d4ab301 paddle::framework::ir::Graph::~Graph()
W0708 20:21:02.230768 10154 init.cc:239] @ 0x7fb01956a649 std::_Sp_counted_base<>::_M_release()