Fatal Python error: GC object already tracked
Created by: 13488862011
单机训练seq2seq模型时,样本为100w条数据,训练3轮已经成功,但在第四轮时报了如下的错误;
有两个问题:1. 这个错误通过设置可以避免吗?2.由于3轮已经有结果,是否可以增量训练呢?多谢
Fatal Python error: GC object already tracked
Current Layer forward/backward stack is
* Aborted at 1479314862 (unix time) try "date -d @1479314862" if you are using GNU date *
Current Layer forward/backward stack is
PC: @ 0x7f56ebf1b965 __GI_raise
Current Layer forward/backward stack is
* SIGABRT (@0x1f400003870) received by PID 14448 (TID 0x7f55b7b82700) from PID 14448; stack trace: *
Current Layer forward/backward stack is
@ 0x7f56ed6d4520 (unknown)
Current Layer forward/backward stack is
@ 0x7f56ebf1b965 __GI_raise
Current Layer forward/backward stack is
@ 0x7f56ebf1cd75 __GI_abort
Current Layer forward/backward stack is
@ 0x7f56ecfe9cf9 Py_FatalError
Current Layer forward/backward stack is
@ 0x7f56ecf860ce PyTuple_New
Current Layer forward/backward stack is
@ 0x7f56ecf86bb1 PyTuple_Pack
Current Layer forward/backward stack is
@ 0x7f56ecf4d098 enum_new
Current Layer forward/backward stack is
@ 0x7f56ecf89463 type_call
Current Layer forward/backward stack is
@ 0x7f56ecf34123 PyObject_Call
Current Layer forward/backward stack is
@ 0x7f56ecfcd8b7 PyEval_EvalFrameEx
Current Layer forward/backward stack is
@ 0x7f56ecf516e7 gen_send_ex
Current Layer forward/backward stack is
@ 0x7f56ecf3300b PyIter_Next
Current Layer forward/backward stack is
@ 0x6a3c0f paddle::PyDataProvider2::loadThread()
Current Layer forward/backward stack is
@ 0x7f56ec8162a8 execute_native_thread_routine
Current Layer forward/backward stack is
@ 0x7f56ed6cbd14 start_thread
Current Layer forward/backward stack is
@ 0x7f56ebfd3bfd clone
/home/.jumbo/bin/paddle_local: line 47: 14448 Aborted ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}