[FLINK-15758][MemManager] Release segment and its unsafe memory in GC Cleaner
After #9747, managed memory is allocated from UNSAFE, not as direct nio buffers as before 1.10. The releasing of segments released also underlying unsafe memory which is dangerous in general as there can be still references to java objects giving access to the released memory. If this reference ever leaks, the illegal memory access can result in memory corruption of other code parts w/o even segmentation fault. The solution can be similar to how Java handles direct memory limit: - underlying byte buffers of segments are registered to phantom reference queue with a Java GC cleaner which releases the unsafe memory - all allocations and reservations are managed by a memory limit and an atomic available memory - if available memory is not enough while reserving, the phantom reference queue processing is triggered to run hopefully discovered by GC cleaners - memory can be released directly or in a GC cleaner The GC is also sped up by nulling out byte buffer reference in `HybridMemorySegment#free` which is inaccessible anyways after freeing. Otherwise also a lot of tests, which hold accidental references to memory segments, have to be fixed to not hold them. The `MemoryManager#verifyEmpty` checks that everything can be GC'ed at the end of the tests and after slot closing in production to detect memory leaks if any other references are held, e.g. from `HybridMemorySegment#wrap`. This closes #11109.
Showing
想要评论请 注册 或 登录