Large number of memfd:doublemapper (deleted) entries

[ayende]

Description

When looking into our process, we noticed a large number of entries like this:

7fb3b0895000-7fb3b0896000 rw-s 03ac6000 00:01 2062                       /memfd:doublemapper (deleted)
7fb3b0896000-7fb3b08a0000 ---s 03ac7000 00:01 2062                       /memfd:doublemapper (deleted)
7fb3b08a0000-7fb3b08ab000 rw-s 03ad1000 00:01 2062                       /memfd:doublemapper (deleted)
7fb3b08ab000-7fb3b08b0000 ---s 03adc000 00:01 2062                       /memfd:doublemapper (deleted)
7fb3b08b0000-7fb3b08b9000 rw-s 03ae1000 00:01 2062                       /memfd:doublemapper (deleted)
7fb3b08b9000-7fb3b08c0000 ---s 03aea000 00:01 2062                       /memfd:doublemapper (deleted)
7fb427006000-7fb427007000 rw-s 00000000 00:01 2062                       /memfd:doublemapper (deleted)

The process has been running for about 8 hours, and we have:

sudo cat /proc/10459/maps | grep doublemapper | wc -l
3308

That number is not stable and grows over time, but we are just now loading data into the system, not yet trying to stress it.

Looking at the code, I found the source of that here:

runtime/src/coreclr/minipal/Unix/doublemapping.cpp

Line 63 in bd84336

int fd = memfd_create("doublemapper", MFD_CLOEXEC);

But looking at where this freed, I see:

runtime/src/coreclr/minipal/Unix/doublemapping.cpp

Line 99 in bd84336

close((int)(size_t)mapperHandle);

This looks like this will only actually be freed on MaxOS, and not on Linux?

FWIW, I couldn't find where this is called.

Is this number of entries expected? Should we monitor this value?
I understand that this is related to the way the JIT allocate memory?

Related, but we are seeing a large increase in memory usage in some production systems, which is not seen in .NET 6.0 but very noticeable in .NET 7.0

I noticed this:
#80580

And we are investigating whatever we do a lot of dynamic assembly generation (so far we don't think so, but can't rule it out).

Reproduction Steps

When I started writing this post, I had:

 sudo cat /proc/10459/maps | grep doublemapper | wc -l
3308

By the time I got here, I had:

sudo cat /proc/10459/maps | grep doublemapper | wc -l
3312

So I certainly think that there is something that work here.
Note that at this point, the process in question was running for hours, basically in a big loop. So there should be no change in behavior nor would I expect it to run any JIT tiering or some such.

Expected behavior

Not have the runtime allocate indefinitely memory

Actual behavior

We are seeing additional memory mapping over time

Regression?

Yes, we aren't seeing that in .NET 6.0

Known Workarounds

No response

Configuration

No response

Other information

No response

[hoyosjs]

@ayende that's an ifndef - so the doublemapper only deallocates on Linux. It's called from the destructor of ExecutableAllocator, which takes memory that's normally RX and creates an RW mapping as needed for W^X purposes. Lambdas that aren't cached and reflection can also cause such behavior too. cc: @janvorli

[ayende]

I'm sorry, didn't realize that this was ifndef, read that as ifdef.
What do you mean by "uncached lambda"?

Is there an expectation that this will grow without limit? What sort of reflection would cause this?

[hoyosjs]

Actually, even in the lambda case of capturing context, I expect allocations of a managed object, but not jitting of new objects. Essentially, I expect tiering, loading, and some debugger operations to cause RX -> RW paging. You can use dotnet counters to see if jitting method count increases. I am not sure what's causing the growth in this case. @janvorli, does this count against the real memory usage? Any ideas what might be contributing to this? I thought since the mapping is deleted it becomes free for the process. I do expect it to count against the max_map_count though.

[janvorli]

The shared memory that is visible as /memfd:doublemapper is used for allocating all executable code for JIT and also for runtime generated helpers and data that need to be allocated close to code that references them. This is the base of the W^X feature that ensures that no memory in the process is writeable and executable at the same time. We double map executable code blocks are writeable memory temporarily to write or modify the code.

[ayende]

Hi,
For reference, we run the workload with DOTNET_EnableWriteXorExecute=0 and we are seeing 10561 entries in the /proc/PID/maps

It is going up & down a bit but appears to be mostly stable.

Without this flag, we are seeing a lot more mapping, and they are always increasing.

My expectation that with W^X, we'll stop needing those once the system stabilized, but we saw overall increase over time even after hours of running.

[marcovr]

We are currently facing a similar issue, where we found runtime crashes like this:

Fatal error. The RW block to unmap was not found
Repeat 2 times:
--------------------------------
   at System.Runtime.CompilerServices.RuntimeHelpers.CompileMethod(System.RuntimeMethodHandleInternal)
--------------------------------
   at System.Reflection.Emit.DynamicMethod.CreateDelegate(System.Type, System.Object)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Compile(System.Linq.Expressions.LambdaExpression)
   ...

This seems to be the case because we run into the default maximum number of memory maps

$ sysctl vm.max_map_count
vm.max_map_count = 65530

I wanted to find out why this is happening and by examining /proc/1/maps I can see that ~90% of all memorymaps are coming from doublemapper

$  cat /proc/1/maps | grep doublemapper | wc -l
59588
$  cat /proc/1/maps | wc -l
63878

[ayende]

We also hit the limit on the # of maps recently in production.

I'm not sure how to point a finger, but it absolutely feels like there is a leak here.

[ayende]

We looked in more detail on the /proc/self/maps and we see:

``
/memfd:doublemapper 7,462 times
Unknown 34,901 times

Those unknown are looking roughly like: `7f9a8bb2e000-7f9b08000000 ---p 00000000 00:00 0`

We run roughly the same process on Windows as well, and looked at the VM Map results.

We have 268GB (!) of `Thread Execution Block` ? 
For that matter you can see that there is a TEB here that is 90MB in size, which seems.. really high

![image](https://github.com/dotnet/runtime/assets/116915/467c944c-3310-476c-8e97-16329632e1d3)

Here is the vmmap data:

[Raven-VMMap.zip](https://github.com/dotnet/runtime/files/12301035/Raven-VMMap.zip)

Any ideas what we are looking at here?