Created by: VictorBebnev

Hello. I infer model on 2=< workers. When i use GPU then graphical memory is leaking When i use CPU then RAM is leaking. I found a similar issue here without solution.

paddle version: V2 API, 0.12.00 trainer_count >=2

here is my log:

F0503 14:27:35.951573 23235 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory
*** Check failure stack trace: ***
    @     0x7fb07946cf2d  google::LogMessage::Fail()
    @     0x7fb07946f008  google::LogMessage::SendToLog()
    @     0x7fb07946ca19  google::LogMessage::Flush()
F0503 14:27:35.955708 23236 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory
*** Check failure stack trace: ***
    @     0x7fb07946fb4e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fb07946cf2d  google::LogMessage::Fail()
    @     0x7fb07946f008  google::LogMessage::SendToLog()
    @     0x7fb07941d7e4  hl_create_thread_resources()
    @     0x7fb07946ca19  google::LogMessage::Flush()
F0503 14:27:35.955708 23236 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memoryF0503 14:27:35.959286 23234 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory
*** Check failure stack trace: ***
    @     0x7fb07941deff  hl_init()
    @     0x7fb07946fb4e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fb07919c6be  paddle::TrainerThread::valueDispatchThread()
    @     0x7fb07946cf2d  google::LogMessage::Fail()
    @     0x7fb07941d7e4  hl_create_thread_resources()
    @     0x7fb07946f008  google::LogMessage::SendToLog()
    @     0x7fb0ab25dc30  (unknown)
    @     0x7fb07941deff  hl_init()
    @     0x7fb0b636f184  start_thread
    @     0x7fb07946ca19  google::LogMessage::Flush()
    @     0x7fb0b609c03d  clone
F0503 14:27:35.955708 23236 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memoryF0503 14:27:35.959286 23234 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memoryF0503 14:27:35.964733 23233 hl_cuda_device.cc:437] Check failed: cudaSuccess == cudaStat (0 vs. 2) Cuda Error: out of memory
*** Check failure stack trace: ***
    @     0x7fb07946fb4e  google::LogMessageFatal::~LogMessageFatal()
    @              (nil)  (unknown)

here is my code:

paddle.init(trainer_count=2); 
self._inferer = paddle.inference.Inference(output_layer=self._log_probs, parameters=self._parameters) 
infer_results = self._inferer.infer(input=infer_data, feeding=feeding_dict)

imho: Leak in the MultiGradientMachine. Thanks.