fluid.evaluator.EditDistance运行报错Tensor holds no memory. Call Tensor::mutable_data first
Created by: nizihan
为使您的问题得到快速解决,在建立Issues前,请您先通过如下方式搜索是否有相似问题:【搜索issue关键字】【使用labels筛选】【官方文档】
如果您没有查询到相似问题,为快速解决您的提问,建立issue时请提供如下细节信息:
- 标题:简洁、精准概括您的问题,例如“Insufficient Memory xxx" ”
- 版本、环境信息: 1)PaddlePaddle版本:1.5 3)GPU:CUDA9.0 和CUDNN7
运行报错:
error_evaluator = fluid.evaluator.EditDistance(
input=maxid,
label=label_out,
ignored_tokens = [sos, eos, pad])
tensor信息: 1572604637 maxid The place is:CUDAPlace(0) Tensor[top_k_0.tmp_1] shape: [64,1,] dtype: l LoD: [[ 0,1,2, ][ 0,32,64, ]] data: 235,5881,5881,2923,2923,2923,2923,2923,2923,2923,
1572604637 label_out The place is:CUDAPlace(0)
Tensor[cast_9.tmp_0]
shape: [64,1,]
dtype: l
LoD: [[ 0,1,2, ][ 0,32,64, ]]
data: 8627,5183,4024,1520,4835,1520,431,2,0,0,
报错信息: Traceback (most recent call last): File "train_end2end.py", line 384, in <module> main() File "train_end2end.py", line 380, in main train(ARGS) File "train_end2end.py", line 331, in train fetch_list=fetch_vars) File "paddle_end2end_version/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 651, in run use_program_cache=use_program_cache) File "env/paddle_end2end_version/python/lib/python2.7/site-packages/paddle/fluid/executor.py", line 749, in _run exe.run(program.desc, scope, 0, True, True, fetch_var_name) paddle.fluid.core_avx.EnforceNotMet: Invoke operator sum error. Python Callstacks: File "/paddle_end2end_version/python/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1771, in append_op attrs=kwargs.get("attrs", None)) File "env/paddle_end2end_version/python/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "env/paddle_end2end_version/python/lib/python2.7/site-packages/paddle/fluid/layers/tensor.py", line 313, in sums attrs={'use_mkldnn': False}) File "env/paddle_end2end_version/python/lib/python2.7/site-packages/paddle/fluid/evaluator.py", line 272, in __init__ out=self.total_distance) File "dfb_code/paddle-textnet-fpn-plate-att-new/thirdparty/model/seq2seq_attention_tf.py", line 508, in seq2seq_recog_head_att ignored_tokens = [sos, eos, pad]) File "dfb_code/paddle-textnet-fpn-plate-att-new/thirdparty/model/network_tf.py", line 944, in resnet_end2end position_encoding) File "train_end2end.py", line 133, in train position_encoding) File "train_end2end.py", line 380, in main train(ARGS) File "train_end2end.py", line 384, in <module> main() C++ Callstacks: holder_ should not be null Tensor holds no memory. Call Tensor::mutable_data first. at [/root/workspace/autoML/Paddle/paddle/fluid/framework/tensor.cc:23] PaddlePaddle Call Stacks: 0 0x7f3f8967ba18p void paddle::platform::EnforceNotMet::Init<std::string>(std::string, char const*, int) + 360 1 0x7f3f8967bd67p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 87 2 0x7f3f8b2e06e9p paddle::framework::Tensor::check_memory_size() const + 185 3 0x7f3f89a8761fp paddle::framework::EigenVector<float, 1, long>::Flatten(paddle::framework::Tensor&) + 127 4 0x7f3f8a313a61p void paddle::operators::SumToLoDTensor<float>(paddle::framework::ExecutionContext const&) + 3041 5 0x7f3f8a31b008p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, int>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, long>, paddle::operators::SumKernel<paddle::platform::CUDADeviceContext, paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 248 6 0x7f3f8b28e027p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 375 7 0x7f3f8b28e401p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529 8 0x7f3f8b28b9fcp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332 9 0x7f3f8980600ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 382 10 0x7f3f898090dfp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocator<std::string> > const&, bool) + 143 11 0x7f3f8966c97dp 12 0x7f3f896ae336p 13 0x7f4040453bb8p PyEval_EvalFrameEx + 25016 14 0x7f40404570bdp PyEval_EvalCodeEx + 2061 15 0x7f4040454345p PyEval_EvalFrameEx + 26949 16 0x7f40404570bdp PyEval_EvalCodeEx + 2061 17 0x7f4040454345p PyEval_EvalFrameEx + 26949 18 0x7f40404570bdp PyEval_EvalCodeEx + 2061 19 0x7f4040454345p PyEval_EvalFrameEx + 26949 20 0x7f4040454460p PyEval_EvalFrameEx + 27232 21 0x7f40404570bdp PyEval_EvalCodeEx + 2061 22 0x7f40404571f2p PyEval_EvalCode + 50 23 0x7f404047ff42p PyRun_FileExFlags + 146 24 0x7f40404812d9p PyRun_SimpleFileExFlags + 217 25 0x7f404049700dp Py_Main + 3149 26 0x7f403f694bd5p __libc_start_main + 245 27 0x4007a1p