Created by: zhouyongxyz

---

Error Message Summary:

ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 3.125000GB memory on GPU 0, available memory is only 117.750000MB.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.

at (/paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:69)

* Check failure stack trace: *

@ 0x7fd50846780d google::LogMessage::Fail() @ 0x7fd50846b2bc google::LogMessage::SendToLog() @ 0x7fd508467333 google::LogMessage::Flush() @ 0x7fd50846c7ce google::LogMessageFatal::~LogMessageFatal() @ 0x7fd50b633598 paddle::framework::details::ExceptionHolder::Catch() @ 0x7fd50b6d0bbe paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync() @ 0x7fd50b6ce55f paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp() @ 0x7fd50b6ce824 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data @ 0x7fd5084c53e3 std::_Function_handler<>::_M_invoke() @ 0x7fd5082bf0f7 std::__future_base::_State_base::_M_do_set() @ 0x7fd594737a99 __pthread_once_slow @ 0x7fd50b6ca9f2 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv @ 0x7fd5082c1554 _ZZN10ThreadPoolC1EmENKUlvE_clEv @ 0x7fd58dd02421 execute_native_thread_routine_compat @ 0x7fd5947306ba start_thread @ 0x7fd59446641d clone @ (nil) (unknown)

使用 2070 和 2080Ti 显卡都这样，paddle 版本 1.8.4-post107 cuda10.2 。并且gpu 的显存是完全足够的，几乎没有怎么使用。