训练过程中报错PreconditionNotMetError: The predict data must less or equal 1.
Created by: liangzhenduo0608
为使您的问题得到快速解决,在建立Issues前,请您先通过如下方式搜索是否有相似问题:【搜索issue关键字】【使用labels筛选】【官方文档】
如果您没有查询到相似问题,为快速解决您的提问,建立issue时请提供如下细节信息:
- 标题:简洁、精准概括您的问题,例如“Insufficient Memory xxx" ”
- 版本、环境信息: 1)PaddlePaddle版本:1.7.1 2)CPU:Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz 3)GPU:预测若用GPU,请提供GPU型号、CUDA和CUDNN版本号 4)系统环境:CentOS6,Python 3.6.4
- 训练信息 1)单机
- 复现信息:如为报错,请给出复现环境、复现步骤
- 问题描述:
错误日志
/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "train.py", line 717, in main(use_cuda) File "train.py", line 711, in main train(use_cuda) File "train.py", line 686, in train fetch_list=[avg_cost.name, auc_batch.name] File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/executor.py", line 783, in run six.reraise(*sys.exc_info()) File "/home/map/.jumbo/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/executor.py", line 778, in run use_program_cache=use_program_cache) File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/executor.py", line 831, in _run_impl use_program_cache=use_program_cache) File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/executor.py", line 905, in _run_program fetch_var_name) paddle.fluid.core_avx.EnforceNotMet:
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) 2 paddle::operators::AucKernel<paddle::platform::CPUPlace, float>::statAuc(paddle::framework::Tensor const*, paddle::framework::Tensor const*, int, int, long*, long*) 3 paddle::operators::AucKernel<paddle::platform::CPUPlace, float>::Compute(paddle::framework::ExecutionContext const&) const 4 ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform8CPUPlaceELb0ELm0EINS0_9operators9AucKernelIS8_fEEEEclEPKcSE_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4 5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const 6 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const 7 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&) 8 paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) 9 paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool, bool)
Python Call Stacks (More useful to users):
File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/framework.py", line 2525, in append_op attrs=kwargs.get("attrs", None)) File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/home/map/.jumbo/lib/python3.6/site-packages/paddle/fluid/layers/metric_op.py", line 235, in auc "StatNegOut": [batch_stat_neg] File "train.py", line 587, in get_model auc = layers.auc(input=predict, label=label, num_thresholds=2 ** 12) File "train.py", line 646, in train model_args = get_model() File "train.py", line 711, in main train(use_cuda) File "train.py", line 717, in main(use_cuda)
Error Message Summary:
PreconditionNotMetError: The predict data must less or equal 1. [Hint: Expected predict_data <= 1, but received predict_data:nan > 1:1.] at (/paddle/paddle/fluid/operators/metrics/auc_op.h:139) [operator < auc > error]
调试信息 paddle.fluid.layers.Print
BATCH 77 1586773000 The content of input layer: The place is:CPUPlace Tensor[fc_4.tmp_2] shape: [1024,2,] dtype: f data: 0.946299,0.0537008,0.710008,0.289992,0.706162,0.293838,0.97014,0.0298601,0.944264,0.0557359,0.995318,0.00468204,0.997226,0.00277375,0.89308,0.10692,0.390375,0.609625,0.915708,0.0842917, BATCH 78 1586773000 The content of input layer: The place is:CPUPlace Tensor[fc_4.tmp_2] shape: [1024,2,] dtype: f data: 0.993649,0.00635145,0.89751,0.10249,1,1.30941e-07,0.717667,0.282333,0.98908,0.0109196,0.987229,0.0127711,0.97767,0.0223299,0.638005,0.361995,0.926301,0.0736985,0.99981,0.000189584, BATCH 79 1586773000 The content of input layer: The place is:CPUPlace Tensor[fc_4.tmp_2] shape: [1024,2,] dtype: f data: 0.819731,0.180269,0.372814,0.627186,0.972738,0.0272616,0.821131,0.178869,0.9855,0.0144998,0.992382,0.00761771,0.994886,0.00511405,0.894112,0.105888,0.881455,0.118545,0.928715,0.0712853, BATCH 80 1586773001 The content of input layer: The place is:CPUPlace Tensor[fc_4.tmp_2] shape: [1024,2,] dtype: f data: 0.999675,0.000325095,0.994081,0.00591934,0.974014,0.0259864,0.935169,0.0648307,0.996004,0.00399586,0.999818,0.000181847,0.999429,0.000571389,0.939129,0.0608707,0.734311,0.265689,0.99998,2.02711e-05, BATCH 81 1586773002 The content of input layer: The place is:CPUPlace Tensor[fc_4.tmp_2] shape: [1024,2,] dtype: f data: nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,