Relation between memory usage by top and by memray -- where is the gap?

[facundobatista]

Hello!

l;dr: I have a memory leak, I see memory increase in real life, I can measure with top, but I'm failing to understand what memray is showing for it.

Context:

I'm trying to find a leak in a Flask app running under gunicorn. For this purpose, I'm running gunicorn it in sync mode with one worker (so when I trigger the memory leak I know which process handled it).

I'm also running gunicorn under memray:

/.../venv/bin/memray run --follow-fork /.../venv/bin/gunicorn ...

All this in a docker container.

How I measure it in the container:

With top:

top -e M -o -PID -c -b -d 2 -p `pgrep -d "," memray`

See information after server is up and in regimen:

top - 13:09:00 up  1:13,  0 user,  load average: 2.52, 3.02, 2.82
Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.7 us,  5.1 sy,  0.0 ni, 81.4 id,  0.8 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :  47838.9 total,  14846.0 free,  20402.1 used,  16396.0 buff/cache     
MiB Swap:  51200.0 total,  51200.0 free,      0.0 used.  27436.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     87 root      20   0  798.8m 245.4m  92.9m S   0.0   0.5   0:03.75 ...
    100 root      20   0 2080.4m 181.4m  27.4m S   0.0   0.4   0:00.14 ...

So I hit once the endpoint that triggers the memory leak, and after the work top shows:

top - 13:09:27 up  1:14,  0 user,  load average: 3.87, 3.28, 2.91
Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s): 19.9 us,  6.6 sy,  0.0 ni, 72.9 id,  0.1 wa,  0.0 hi,  0.5 si,  0.0 st 
MiB Mem :  47838.9 total,  14788.1 free,  20419.6 used,  16428.3 buff/cache     
MiB Swap:  51200.0 total,  51200.0 free,      0.0 used.  27419.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     87 root      20   0  798.8m 245.4m  92.9m S   1.0   0.5   0:03.95 ...
    100 root      20   0 2296.5m 187.7m  33.1m S   3.5   0.4   0:00.49 ...

Focus on process with id 100. See how all memory indications went up. Particularly, VIRT raised ~200 MB!

Memray:

So then I grab the memray outputs, and create a flamegraph:

memray flamegraph --temporal --leaks memray-gunicorn.87.bin.100

When I see it the step us is there:

However, the amount is weird. It goes up from 192.1MB to 196.8MB.

It seems to correspond to the increment in what top shows as RES (resident memory, makes sense!). However the absolute amounts are totally different.

Beyond that, what about VIRT? How can I find that 200MB gap?

All help/information is welcomed. Thanks in advance!

. Facundo

[godlygeek]

If you haven't already read https://bloomberg.github.io/memray/memory.html you should start there.

The graph you're showing has two lines on it. The blue line is RSS, and should match up with the RES column in top. Memray polls the resident set size periodically and shows it in a graph, but does not, and cannot, tell you why it increased or decreased. The resident set size can change while the program is suspended, or sleeping, or blocked on IO. It can change without the program doing anything at all, based on the whims of the kernel.

The orange line is the one that Memray really proactively tracks. That line is showing the heap memory usage of the program: the amount of memory allocated by calls to malloc or mmap or the like and not yet freed by calls to free or munmap or the like. That does not correspond to any of the columns in top. In the graph you showed, the orange line is hovering at pretty close to 0. That's telling us that nothing much was allocated but not freed.

As for why you see VIRT increasing by 200 MB, that could be any number of things. It could be that you loaded a large shared library, and the VIRT increase is the address space of that shared library. It could be that you spawned a large number of threads, and the VIRT increase is the stack for each of those new threads.

Without a reproducer, it's very hard to know why VIRT has increased, but what we can know for sure is that it wasn't caused by calls to a one of the malloc family allocators that Memray tracks or by calls to the mmap function. If it was caused by one of those, the orange line would have jumped up to 200MB.

[facundobatista]

Thanks @godlygeek for all the info! I'll focus in the heap memory then for chasing a memory leak, if any.

Question, do you know how can I find out regarding VIRT memory? you say that is very hard without a "reproducer", what is that? Maybe you can recommend me a tool, or any indication is welcomed, thanks!

[godlygeek]

By "reproducer" I meant a small program that I can run myself to reproduce the issue that you're seeing.