模型训练加上extra_layers后报错的问题
Created by: HugoLian
我有个推荐的双塔模型,本身的trainer是这么定义的: ''' trainer = paddle.trainer.SGD( cost=cost, parameters=parameters, update_equation=paddle.optimizer.Adam(learning_rate=1e-4)) ''' 启动训练可以正常进行,后来加入了AUC层: extra_layers=paddle.evaluator.auc(input=inference, label=label), 即: ''' trainer = paddle.trainer.SGD( cost=cost, parameters=parameters, extra_layers=paddle.evaluator.auc(input=inference, label=label), update_equation=paddle.optimizer.Adam(learning_rate=1e-4)) ''' 训练一开始就迅速报错: ''' I0411 16:52:58.748926 23710 Util.cpp:166] commandline: --use_gpu=False --trainer_count=4 I0411 16:53:01.049448 23710 GradientMachine.cpp:94] Initing parameters.. I0411 16:53:11.947480 23710 GradientMachine.cpp:101] Init parameters done. F0411 16:53:44.801867 23710 Evaluator.cpp:460] Check failed: binIdx <= kBinNum_ bin index [4293786548] out of range, predict value[-0.0703781] *** Check failure stack trace: *** @ 0x7f97f324ebcd google::LogMessage::Fail() @ 0x7f97f325267c google::LogMessage::SendToLog() @ 0x7f97f324e6f3 google::LogMessage::Flush() @ 0x7f97f3253b8e google::LogMessageFatal::~LogMessageFatal() @ 0x7f97f2f8647d paddle::AucEvaluator::evalImp() @ 0x7f97f2f85c46 paddle::Evaluator::eval() @ 0x7f97f2f59dc8 paddle::CombinedEvaluator::eval() @ 0x7f97f2f77332 paddle::MultiGradientMachine::eval() @ 0x7f97f2df7276 _wrap_GradientMachine_eval @ 0x4b4cb9 PyEval_EvalFrameEx @ 0x4b6b28 PyEval_EvalCodeEx @ 0x4b5d10 PyEval_EvalFrameEx @ 0x4b6b28 PyEval_EvalCodeEx @ 0x4b5d10 PyEval_EvalFrameEx @ 0x4b6b28 PyEval_EvalCodeEx @ 0x4b5d10 PyEval_EvalFrameEx @ 0x4b6b28 PyEval_EvalCodeEx @ 0x4b6c52 PyEval_EvalCode @ 0x4e1c7d PyRun_FileExFlags @ 0x4e3501 PyRun_SimpleFileExFlags @ 0x4159dd Py_Main @ 0x7f982859cbd5 __libc_start_main @ 0x414b71 (unknown) @ (nil) (unknown) Aborted ''' 其中bin index [4293786548] 中括号里的数值,每次和每次都不一样,请问都有什么可能的原因呢?