SSD模型的GPU训练问题
Created by: zmofei
我在使用GPU训练SSD模型的时候(models/ssd/trainer.py), 遇到如下错误:
F0921 14:24:18.580162 6954 hl_gpu_matrix_kernel.cuh:181] Check failed: cudaSuccess == err (0 vs. 8) [hl_gpu_apply_unary_op failed] CUDA error: invalid device function
*** Check failure stack trace: ***
@ 0x7fcdf3faf5ed google::LogMessage::Fail()
@ 0x7fcdf3fb309c google::LogMessage::SendToLog()
@ 0x7fcdf3faf0e3 google::LogMessage::Flush()
@ 0x7fcdf3fb45ae google::LogMessageFatal::~LogMessageFatal()
@ 0x7fcdf3e39ec4 hl_gpu_apply_unary_op<>()
@ 0x7fcdf3e3a205 paddle::BaseMatrixT<>::applyUnary<>()
@ 0x7fcdf3e3a433 paddle::BaseMatrixT<>::zero()
@ 0x7fcdf3cd58d1 paddle::Parameter::enableType()
@ 0x7fcdf3cd11cc paddle::parameterInitNN()
@ 0x7fcdf3cd391a paddle::NeuralNetwork::init()
@ 0x7fcdf3cfc491 paddle::GradientMachine::create()
@ 0x7fcdf3f8c3b3 GradientMachine::createFromPaddleModelPtr()
@ 0x7fcdf3f8c58f GradientMachine::createByConfigProtoStr()
@ 0x7fcdf3b9b4cd _wrap_GradientMachine_createByConfigProtoStr
@ 0x4b4cb9 PyEval_EvalFrameEx
@ 0x4b6b28 PyEval_EvalCodeEx
@ 0x4b5d10 PyEval_EvalFrameEx
@ 0x4b6b28 PyEval_EvalCodeEx
@ 0x4b5d10 PyEval_EvalFrameEx
@ 0x4b6b28 PyEval_EvalCodeEx
@ 0x52940f function_call
@ 0x422cba PyObject_Call
@ 0x4271ad instancemethod_call
@ 0x422cba PyObject_Call
@ 0x48121f slot_tp_init
@ 0x47eb1a type_call
@ 0x422cba PyObject_Call
@ 0x4b31dd PyEval_EvalFrameEx
@ 0x4b6b28 PyEval_EvalCodeEx
@ 0x4b5d10 PyEval_EvalFrameEx
@ 0x4b6b28 PyEval_EvalCodeEx
@ 0x4b6c52 PyEval_EvalCode
Aborted
另外,该模型修改后在CPU下是可以运行的。 请问如何解决这个问题?我是使用pip源安装的