使用paddlepaddle出现cuda的内存泄漏?
Created by: Jessie-J7
系统环境:Ubuntu 16.04 paddle版本: 1.6 cuda版本: 10.1 cudnn版本: 7.6
压测最终结果报告:单张图片泄漏约5k 使用gperftools内存泄漏检测工具,结果如下:
Leak of 800 bytes in 10 objects allocated from: @ 7f7b8b93cdcd unknown @ 00007f7b8b948295 cuVDPAUCtxCreate ??:0 @ 00007f7b8b949e86 cuVDPAUCtxCreate ??:0 @ 00007f7b8b94a3e7 cuVDPAUCtxCreate ??:0 @ 00007f7b8b742999 cuMemGetAttribute ??:0 @ 00007f7b8b743331 cuMemGetAttribute ??:0 @ 00007f7b8b8a2daa cuMemAlloc_v2 ??:0 @ 00007f7ba4d30712 cudart::driverHelper::mallocPtr :0 @ 00007f7ba4d0bcea cudart::cudaApiMalloc :0 @ 00007f7ba4d4128b cudaMalloc :0 @ 00007f7ba4c31769 paddle::memory::detail::GPUAllocator::Alloc ??:0 @ 00007f7ba4c25d4c paddle::memory::detail::BuddyAllocator::RefillPool ??:0 @ 00007f7ba4c2659a paddle::memory::detail::BuddyAllocator::Alloc ??:0 @ 00007f7ba4c2225d paddle::memory::legacy::Alloc ??:0 @ 00007f7ba4c227f7 paddle::memory::allocation::NaiveBestFitAllocator::AllocateImpl ??:0 @ 00007f7ba4c3d195 paddle::memory::allocation::RetryAllocator::AllocateImpl ??:0 @ 00007f7ba4c1ac41 paddle::memory::allocation::AllocatorFacade::Alloc ??:0 @ 00007f7ba4c1acf9 paddle::memory::allocation::AllocatorFacade::AllocShared ??:0 @ 00007f7ba3e6e46a paddle::memory::AllocShared ??:0 @ 00007f7ba18b17b5 paddle::framework::Tensor::mutable_data ??:0 @ 00007f7ba1b4c162 paddle::framework::Tensor::mutable_data ??:0 @ 00007f7ba2b69595 paddle::operators::BatchNormKernel::Compute ??:0 @ 00007f7ba2b6b45e std::_Function_handler::_M_invoke ??:0 @ 00007f7ba3dc8f26 paddle::framework::OperatorWithKernel::RunImpl ??:0 @ 00007f7ba3dc9df6 paddle::framework::OperatorWithKernel::RunImpl ??:0 @ 00007f7ba3dc1b1d paddle::framework::OperatorBase::Run ??:0 @ 00007f7ba18f2af9 paddle::framework::NaiveExecutor::Run ??:0 @ 00007f7ba173bc27 paddle::AnalysisPredictor::ZeroCopyRun ??:0 @ 00007f7bc9175937 RetinaDlInferenceLayer::pdl_do_inference /data/inference_layer.cpp:235 @ 00007f7bc9175a01 RetinaDlInferenceLayer::pdl_infer /data/inference_layer.cpp:245 @ 00007f7bc9196cea DiscCupDetectionLayer::infer /data/detection_layer.cpp:78 @ 00007f7bc9113a6c AbstractLayer::forward /data/layer.hpp:41 @ 00007f7bc9160949 RetinaSolver::process_layer /data/solver.cpp:166 @ 00007f7bc9160cd6 RetinaSolver::process /data/solver.cpp:189 @ 00007f7bc919e2bc RetinaSolver::solve /data/solver.hpp:52 @ 00007f7bc919dc16 merge_for_report /data/report.cpp:54 @ 000055a1349ac379 main
Leak of 496 bytes in 1 objects allocated from: @ 7f7b8b93cdcd unknown @ 00007f7b8b948295 cuVDPAUCtxCreate ??:0 @ 00007f7b8b949e86 cuVDPAUCtxCreate ??:0 @ 00007f7b8b94a3e7 cuVDPAUCtxCreate ??:0 @ 00007f7b8b805544 cudbgApiDetach ??:0 @ 00007f7b8b8058c3 cudbgApiDetach ??:0 @ 00007f7b8b849571 cuEGLApiInit ??:0 @ 00007f7b8b752201 cuMemGetAttribute_v2 ??:0 @ 00007f7b8b7522bf cuMemGetAttribute_v2 ??:0 @ 00007f7b8b8abd69 cuStreamCreateWithPriority ??:0 @ 00007f7ae9ab7a31 cudnnDropoutForward ??:0 @ 00007f7ae9ab7bfb cudnnDropoutForward ??:0 @ 00007f7ae9ae59d1 cudnnDropoutForward ??:0 @ 00007f7ae8510aad cudnnCreate ??:0 @ 00007f7ba178b3ff paddle::platform::CUDADeviceContext::CUDADeviceContext ??:0 @ 00007f7ba1796658 std::_Function_handler::_M_invoke ??:0 @ 00007f7ba178e44d std::__future_base::_State_baseV2::_M_do_set :0 @ 00007f7bcf35c826 __pthread_once_slow /build/glibc-OTsEL5/glibc-2.27/nptl/pthread_once.c:116 @ 00007f7ba17902e1 std::__future_base::_State_baseV2::_M_set_result :0 @ 00007f7ba1790450 std::__future_base::_Deferred_state::_M_complete_async ??:0 @ 00007f7ba178aaeb paddle::platform::DeviceContextPool::Get ??:0 @ 00007f7ba174e7a1 paddle::ZeroCopyTensor::copy_from_cpu ??:0 @ 00007f7bc91756d4 RetinaDlInferenceLayer::pdl_set_input /data/inference_layer.cpp:221 @ 00007f7bc91759ea RetinaDlInferenceLayer::pdl_infer /data/inference_layer.cpp:244
直接申请内存/未释放内存%/累计未释放内存%/间接申请内存/间接未释放内存%
500.0 64.9% 64.9% 500.0 64.9% paddle::memory::detail::AlignedMalloc 219.6 28.5% 93.4% 264.5 34.3% cuEGLApiInit 36.6 4.8% 98.1% 37.4 4.9% cuVDPAUCtxCreate 7.9 1.0% 99.2% 19.0 2.5% cuMemGetAttribute