improve cgroup memory usage calculation to avoid counting purgeable cache#549
Open
dschaaff wants to merge 1 commit into
Open
improve cgroup memory usage calculation to avoid counting purgeable cache#549dschaaff wants to merge 1 commit into
dschaaff wants to merge 1 commit into
Conversation
…ache When running under a cgroup limit, get_cgroup() reported stat.usage_in_bytes (memory.current on v2, memory.usage_in_bytes on v1) as the process memory usage. That counter includes all page cache, including cold reclaimable cache (inactive_file). For a workload that writes logs and spool to disk, cold cache can dominate: on a real pod memory.current was ~51GB while inactive_file was ~46GB and the true non-cache footprint was ~1.8GB. kumod saw usage near its limit, hit get_headroom() == 0, and ran shrink_ready_queue_due_to_low_mem and other reductions against pressure that was almost entirely reclaimable cache the kernel would drop before any OOM. Report the working set instead: ``` working_set = max(memory.current - inactive_file, anon) ``` This matches container_memory_working_set_bytes (kubelet/cAdvisor) and the kernel's own reclaimability accounting. It still counts everything that can drive an OOM: anonymous memory, dirty page cache, active_file, tmpfs/shm, and slab. Signed-off-by: Daniel Schaaff <daniel@danielschaaff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When running under a cgroup limit, get_cgroup() reported stat.usage_in_bytes (memory.current on v2, memory.usage_in_bytes on v1) as the process memory usage. That counter includes all page cache, including cold reclaimable cache (inactive_file). For a workload that writes logs and spool to disk, cold cache can dominate: on a real pod memory.current was ~51GB while inactive_file was ~46GB and the true non-cache footprint was ~1.8GB. kumod saw usage near its limit, hit get_headroom() == 0, and ran
shrink_ready_queue_due_to_low_mem
and other reductions against pressure that was almost entirely reclaimable cache the kernel would drop before any OOM.
Report the working set instead:
This matches container_memory_working_set_bytes (kubelet/cAdvisor). It still counts everything that can drive an OOM: anonymous memory, dirty page cache, active_file, tmpfs/shm, and slab.
Memory accounting outside of a cgroup is untouched.
Disclosure: I used Claude to aid in the development of this.