I have a process which continuously allocates memory and will free it after another thread have processed related data. When the data processing rate is slow, I see RES memory grows up; but after all the data have been processed, RES goes down but doesn’t go back to original RES value (even after waiting for over 10 minutes).
e.g. 10 MB (original) => 50 MB (peak) => 30MB (after all data are freed)
I have used valgrind with massif to profile memory, it looks like all data are freed. My question is why RES doesn’t go back to original 10 MB?
Advertisement
Answer
There are so many possible reasons, I’ll just list you the two most common ones:
page-wise allocation and fragmentation: Operating systems allocate memory in terms of pages, at least on modern systems using MMUs. Your standard library’s
malloc()(or the allocator fornewin C++) serves arbitrary sizes and manages them internally. A page can only be returned to the operating system when the last allocation occupying it is returned. If only a single allocation in the page is still active, your process must keep this page.libraries: A lot of libraries do their own dynamic memory allocations, even the C standard library does.
That said, as Ctx commented, you can do direct page allocations yourself on Linux using mmap() and maybe avoid the fragmentation issue by using these pages efficiently. This would of course mean to leave the path of standard C, but at least, mmap() is specified in POSIX, so it would work on many systems.