Created by: Cristhine

W0725 15:08:35.269969 28685 device_context.cc:259] Please NOTE: device: 0, CUDA Capability: 35, Driver API Version: 9.0, Runtime API Version: 8.0 W0725 15:08:35.272292 28685 device_context.cc:267] device: 0, cuDNN Version: 5.0. W0725 15:08:35.272333 28685 device_context.cc:293] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.0, but CUDNN version in your machine is 5.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version. Traceback (most recent call last): File "train.py", line 218, in train() File "train.py", line 85, in train exe.run(fluid.default_startup_program()) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/executor.py", line 651, in run use_program_cache=use_program_cache) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/executor.py", line 749, in run exe.run(program.desc, scope, 0, True, True, fetch_var_name) paddle.fluid.core_avx.EnforceNotMet: Invoke operator fill_constant error. Python Callstacks: File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1842, in prepend_op attrs=kwargs.get("attrs", None)) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/initializer.py", line 189, in call stop_gradient=True) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1625, in create_var kwargs'initializer' File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/layer_helper_base.py", line 383, in set_variable_initializer initializer=initializer) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/optimizer.py", line 317, in add_accumulator var, initializer=Constant(value=float(fill_value))) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/optimizer.py", line 760, in create_accumulators self.add_accumulator(self.velocity_acc_str, p) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/optimizer.py", line 364, in create_optimization_pass [p[0] for p in parameters_and_grads]) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/optimizer.py", line 532, in apply_gradients optimize_ops = self.create_optimization_pass(params_grads) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/optimizer.py", line 562, in apply_optimize optimize_ops = self.apply_gradients(params_grads) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/optimizer.py", line 601, in minimize loss, startup_program=startup_program, params_grads=params_grads) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/dygraph/base.py", line 87, in impl return func(*args, **kwargs) File "/home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl return wrapped_func(args, kwargs) File "</home/vis/duyuting/app/python-cuda8-paddle-0.15.1/lib/python2.7/site-packages/decorator.pyc:decorator-gen-20>", line 2, in minimize File "train.py", line 81, in train optimizer.minimize(loss) File "train.py", line 218, in train() C++ Callstacks: Enforce failed. Expected allocating <= available, but received allocating:11185625045 > available:1332805376. Insufficient GPU memory to allocation. at [/home/vis/wangjian33/code/Paddle/paddle/fluid/platform/gpu_info.cc:262] PaddlePaddle Call Stacks: 0 0x7f222ee60cb8p void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const, int) + 360 1 0x7f222ee61007p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const, int) + 87 2 0x7f22323fbdf6p paddle::platform::GpuMaxChunkSize() + 630 3 0x7f223232495ap 4 0x7f227ca73973p pthread_once + 83 5 0x7f2232323ffdp paddle::memory::legacy::GetGPUBuddyAllocator(int) + 109 6 0x7f2232324e45p void paddle::memory::legacy::Allocpaddle::platform::CUDAPlace(paddle::platform::CUDAPlace const&, unsigned long) + 37 7 0x7f2232325385p paddle::memory::allocation::LegacyAllocator::AllocateImpl(unsigned long) + 421 8 0x7f22323194a5p paddle::memory::allocation::AllocatorFacade::Alloc(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 181 9 0x7f223231962ap paddle::memory::allocation::AllocatorFacade::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 26 10 0x7f22309138fcp paddle::memory::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 44 11 0x7f2230cc7113p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, paddle::framework::proto::VarType_Type, unsigned long) + 147 12 0x7f222f5bda4ep paddle::operators::FillConstantKernel::Compute(paddle::framework::ExecutionContext const&) const + 494 13 0x7f222f5c0b83p std::Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernel, paddle::operators::FillConstantKernelpaddle::platform::float16 >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1 (closed)}>::M_invoke(std::Any_data const&, paddle::framework::ExecutionContext const&) + 35 14 0x7f2230c387f7p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 375 15 0x7f2230c38bd1p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529 16 0x7f2230c3634cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332 17 0x7f222efe8b2ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 382 18 0x7f222efeb7ffp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool) + 143 19 0x7f222ee51c7dp 20 0x7f222ee9368cp 21 0x4b8ba7p PyEval_EvalFrameEx + 30279 22 0x4baac8p PyEval_EvalCodeEx + 2040 23 0x4b8328p PyEval_EvalFrameEx + 28104 24 0x4baac8p PyEval_EvalCodeEx + 2040 25 0x4b8328p PyEval_EvalFrameEx + 28104 26 0x4baac8p PyEval_EvalCodeEx + 2040 27 0x4b8328p PyEval_EvalFrameEx + 28104 28 0x4baac8p PyEval_EvalCodeEx + 2040 29 0x4babf2p PyEval_EvalCode + 50 30 0x4e8addp PyRun_FileExFlags + 285 31 0x4ea361p PyRun_SimpleFileExFlags + 209 32 0x415c5dp Py_Main + 2925 33 0x7f227bfd0bd5p __libc_start_main + 245 34 0x414df1p