最近 LAC 总挂
Created by: daming-lu
之前跑了至少一个月,没有问题,现在总挂。怀疑是 RowwiseAdd 的问题。 @kuke
*** Aborted at 1565738568 (unix time) try "date -d @1565738568" if you are using GNU date *** [34/1890] PC: @ 0x0 (unknown) *** SIGFPE (@0x7f5928dc75bb) received by PID 24783 (TID 0x7f598b8c3740) from PID 685536699; stack trace: *** @ 0x7f598b4ae890 (unknown) @ 0x7f5928dc75bb paddle::operators::math::RowwiseAdd<>::operator()() @ 0x7f5927caadbf paddle::operators::GRUCPUKernel<>::BatchCompute() @ 0x7f5927cac2b3 _ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelReg$ strarFunctorINS0_8platform8CPUPlaceELb0ELm0EJNS0_9operators12GRUCPUKernelIfEENSA_IdEEEEclEPKcSF_iEUlS4_E_E9_M_i$ vokeERKSt9_Any_dataS4_ @ 0x7f5928d99376 paddle::framework::OperatorWithKernel::RunImpl() @ 0x7f5928d99ae4 paddle::framework::OperatorWithKernel::RunImpl() @ 0x7f5928d9740c paddle::framework::OperatorBase::Run() @ 0x7f592727c3fe paddle::framework::Executor::RunPreparedContext() @ 0x7f592727d23f paddle::framework::Executor::Run() @ 0x7f59270f98de _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL18pybind11_init_coreERNS_6mod$ leEEUlRNS2_9framework8ExecutorERKNS6_11ProgramDescEPNS6_5ScopeEibbRKSt6vectorISsSaISsEEE97_vIS8_SB_SD_ibbSI_EIN$ _4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNES10_ @ 0x7f592713c7ce pybind11::cpp_function::dispatcher() @ 0x562becacf9e4 _PyCFunction_FastCallDict @ 0x562becb5cf4e call_function @ 0x562becb8194a _PyEval_EvalFrameDefault @ 0x562becb56206 _PyEval_EvalCodeWithName @ 0x562becb571cf fast_function @ 0x562becb5ced5 call_function @ 0x562becb82715 _PyEval_EvalFrameDefault @ 0x562becb56206 _PyEval_EvalCodeWithName @ 0x562becb571cf fast_function @ 0x562becb5ced5 call_function @ 0x562becb82715 _PyEval_EvalFrameDefault @ 0x562becb5662e _PyEval_EvalCodeWithName @ 0x562becb57897 _PyFunction_FastCallDict @ 0x562becacfdaf _PyObject_FastCallDict @ 0x562becad4a73 _PyObject_Call_Prepend @ 0x562becacfbcb _PyObject_FastCallDict @ 0x562becbc44b2 partial_call @ 0x562becacfbcb _PyObject_FastCallDict @ 0x562becb5748a _PyObject_FastCallKeywords @ 0x562becb5cf4e call_function @ 0x562becb82715 _PyEval_EvalFrameDefault
lac_error_log.txt