Created by: Termset

我改动过的IMPALA算法，运行出现问题。不知道怎么定位到这个问题。

[03-02 14:47:41 MainThread @machine_info.py:86] nvidia-smi -L found gpu count: 1
[03-02 14:47:41 MainThread @machine_info.py:86] nvidia-smi -L found gpu count: 1
[03-02 14:47:41 MainThread @train.py:154] Waiting for 1 remote actors to connect.
[03-02 14:47:41 MainThread @train.py:158] Remote actor count: 1
Error: /paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:88 Assertion `y_[id] != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide.Please check.

Error: /paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:88 Assertion `y_[id] != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide.Please check.

Error: /paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:88 Assertion `y_[id] != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide.Please check.

Error: /paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:88 Assertion `y_[id] != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide.Please check.

Error: /paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:88 Assertion `y_[id] != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide.Please check.
......
Error: /paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:88 Assertion `y_[id] != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide.Please check.

F0302 14:47:44.609232 23176 device_context.cc:328] cudaStreamSynchronize unspecified launch failure errno: 4
*** Check failure stack trace: ***
    @     0x7f1e600d138d  google::LogMessage::Fail()
    @     0x7f1e600d4e3c  google::LogMessage::SendToLog()
    @     0x7f1e600d0eb3  google::LogMessage::Flush()
    @     0x7f1e600d634e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f1e629ee8f7  paddle::platform::CUDADeviceContext::Wait()
    @     0x7f1e601184fb  paddle::framework::Executor::RunPreparedContext()
    @     0x7f1e60a5c312  paddle::operators::RecurrentOp::RunImpl()
    @     0x7f1e6295783c  paddle::framework::OperatorBase::Run()
    @     0x7f1e601184c6  paddle::framework::Executor::RunPreparedContext()
    @     0x7f1e6011bd0f  paddle::framework::Executor::Run()
    @     0x7f1e5ff5609d  _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9framework8ExecutorERKNS6_11ProgramDescEPNS6_5ScopeEibbRKSt6vectorISsSaISsEEE103_vIS8_SB_SD_ibbSI_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNES10_
    @     0x7f1e5ff9edd1  pybind11::cpp_function::dispatcher()
    @     0x7f1eb6d6a302  _PyCFunction_FastCallDict
    @     0x7f1eb6def95b  call_function
    @     0x7f1eb6df2d40  _PyEval_EvalFrameDefault
    @     0x7f1eb6dee100  _PyEval_EvalCodeWithName
    @     0x7f1eb6defb2a  call_function
    @     0x7f1eb6df32cc  _PyEval_EvalFrameDefault
    @     0x7f1eb6dee100  _PyEval_EvalCodeWithName
    @     0x7f1eb6defb2a  call_function
    @     0x7f1eb6df32cc  _PyEval_EvalFrameDefault
    @     0x7f1eb6dee100  _PyEval_EvalCodeWithName
    @     0x7f1eb6defb2a  call_function
    @     0x7f1eb6df32cc  _PyEval_EvalFrameDefault
    @     0x7f1eb6ded514  _PyFunction_FastCall
    @     0x7f1eb6defc88  call_function
    @     0x7f1eb6df2d40  _PyEval_EvalFrameDefault
    @     0x7f1eb6ded514  _PyFunction_FastCall
    @     0x7f1eb6dee515  _PyFunction_FastCallDict
    @     0x7f1eb6d12ce6  _PyObject_FastCallDict
    @     0x7f1eb6d12f3c  _PyObject_Call_Prepend
    @     0x7f1eb6d12fd6  PyObject_Call
已放弃