runtime.dotnet.gc.memory_load is 4/3 too high in containers with a memory limit, and so exceeds 100

[lucacavenaghi97]

Tracer Version(s): 3.40.0 through 3.50.0, and current master.

Operating system and platform: Linux x64, running in a container with a memory limit (Kubernetes or Docker).

Instrumentation Mode: Automatic instrumentation, but the mode is irrelevant: the value comes from the runtime metrics writer.

TFM: net8.0 and later. Also net6.0 and net7.0 when DD_RUNTIME_METRICS_ENABLED is not explicitly set.

---

Bug Report

runtime.dotnet.gc.memory_load is published as a percentage, but in a container with a memory limit every value is a third too high: it divides MemoryLoadBytes, which is relative to the container limit, by TotalAvailableMemoryBytes, which is 75% of that limit. A published 60 is a real 45, and nothing in the metric reveals it. Past a real load of 75% the value also goes above 100, with a ceiling of 133.3.

DiagnosticsMetricsRuntimeMetricsListener.cs:137-141 computes it like this:

var availableBytes = gcInfo.TotalAvailableMemoryBytes;

if (availableBytes > 0)
{
    statsd.Gauge(MetricsNames.GcMemoryLoad, (double)gcInfo.MemoryLoadBytes * 100.0 / availableBytes);
}

Those two properties are not expressed against the same base. Both doc pages state their own base:

GCMemoryInfo.MemoryLoadBytes:

When a process is running in a container with a memory limit or when cgroup limits are set: [...] On Linux, the MemoryLoadBytes is obtained from the used physical memory via the CGroup Memory Usage file from memory.usage_in_bytes for CGroups v1 and memory.current for CGroups v2 divided by the memory limit.

GCMemoryInfo.TotalAvailableMemoryBytes:

If the program is run in a container, this value is an implementation-defined fraction of the container's size.

The source agrees with the docs. The two values are filled in next to each other in GCHeap::GetMemoryInfo, and only one of them is scaled by total_physical_mem (src/coreclr/gc/interface.cpp:2249-2250 on main, same code in src/coreclr/gc/gc.cpp on the release branches):

*totalAvailableMemoryBytes = gc_heap::heap_hard_limit != 0 ? gc_heap::heap_hard_limit : gc_heap::total_physical_mem;
*lastRecordedMemLoadBytes  = (uint64_t)(((double)(last_gc_info->memory_load)) / 100 * gc_heap::total_physical_mem);

The fraction is 75% for any container with a memory limit and no explicit heap hard limit configured (src/coreclr/gc/init.cpp, same code on release/8.0 through release/10.0):

uint64_t physical_mem_for_gc = total_physical_mem * (uint64_t)75 / (uint64_t)100;
heap_hard_limit = (size_t)max((uint64_t)(20 * 1024 * 1024), physical_mem_for_gc);

The arithmetic

Writing L for the container limit and C for the real memory load as a percentage:

MemoryLoadBytes           = C/100 * L
TotalAvailableMemoryBytes = 0.75 * L
published                 = (C/100 * L) * 100 / (0.75 * L) = C / 0.75

L cancels out, so the factor is 4/3 whatever the container size. A real load of 75% publishes as 100, and a real load of 100% publishes as 133.33.

A service with limits.memory: 512Mi on .NET 8, at a real memory load of 91%:

total_physical_mem = 536870912
heap_hard_limit    = 402653184
MemoryLoadBytes    = int(0.91 * 536870912) = 488552529
published          = 488552529 * 100.0 / 402653184 = 121.33333310484886

Because the real load is a whole percentage, the published values land on a 4/3 grid and carry a long decimal tail: a real 90 gives 119.99999980131786. In our environment several unrelated services sit at exactly 131.99999978144965, which is 4/3 * 99, so they are at a real load of 99% rather than at any shared limit of 132.

The same division in the BCL is clamped

PhysicalMemoryMonitor.Unix.cs divides the same two properties and treats the overflow as an expected case:

if (memInfo.TotalAvailableMemoryBytes >= memInfo.MemoryLoadBytes)
{
    int memoryLoad = (int)((float)memInfo.MemoryLoadBytes * 100.0 / (float)memInfo.TotalAvailableMemoryBytes);
    return Math.Max(1, memoryLoad);
}

// It's possible the load was legitimately higher than "available". In that case, return 100.
// Otherwise, return 0 to minimize impact because something was unexpected.
return (memInfo.MemoryLoadBytes > 0) ? 100 : 0;

Impact

Upgrading across 3.40.0 makes the series jump by a third, so thresholds set before the upgrade quietly stop meaning what they meant. We saw a service step from about 0.99x to 1.33x of its container's memory utilization on the day of the tracer upgrade, with runtime version, memory limit and working set all unchanged.

The factor depends on the heap hard limit, so a service that sets GCHeapHardLimitPercent gets a different one and the metric is not comparable across services.

Versions

Before 3.40.0 the metric came from GCGlobalHeapHistory.MemoryPressure through RuntimeEventListener, which is the GC's own percentage and needs no arithmetic. The new listener was present before 3.40.0 but off by default; #8267 made it the default and shipped in 3.40.0 (TracerSettings.cs:203, collector selection in RuntimeMetricsWriter.cs:335-341). Everything before 3.40.0 is unaffected.

Verified on tracer v3.50.0 and master, and on dotnet/runtime release/8.0, release/9.0, release/10.0 and main.

[andrewlock]

Thanks for flagging this @lucacavenaghi97 - we're looking into it, and will keep you updated