Add monitor to track pods terminated due to OOM #4128
+103
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yadneshk
requested review from
jewzaam,
bennerv,
hawkowl,
rogbas,
petrkotas,
jharrington22,
cblecker,
cadenmarchese,
UlrichSchlueter,
SudoBrendan,
yjst2012,
jaitaiwan,
anshulvermapatel,
hlipsig,
tiguelu,
mociarain,
kimorris27,
tsatam,
bitoku,
fahlmant and
sankur-codes
as code owners
yadneshk
force-pushed
the
yadneshk/ARO-7793-oomkilled-pods
branch
2 times, most recently
from
c15f2c0 to
2f4b4ce
Compare
|
LGTM, |
yadneshk
force-pushed
the
yadneshk/ARO-7793-oomkilled-pods
branch
from
2f4b4ce to
c912e13
Compare
Introduce a new monitor that emits metrics "pod.oomkilled" to report pods that were killed due to OOM. This monitor looks for pods that are in terminated state and has reason as "OOMKilled".
yadneshk
force-pushed
the
yadneshk/ARO-7793-oomkilled-pods
branch
from
c912e13 to
94755ce
Compare
edisonLcardenas
approved these changes
Mar 6, 2025
| "name": p.Name, | ||
| "namespace": p.Namespace, | ||
| "nodeName": p.Spec.NodeName, | ||
| "containername": cntrStatus.Name, |
There was a problem hiding this comment.
Do you think it make sense if we add the pod name as a dimension?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue this PR addresses:
Fixes https://issues.redhat.com/browse/ARO-7793
What this PR does / why we need it:
Pods killed by OOM often enter a CrashLoopBackOff state and keep restarting. While the restart counter may reflect these restarts, it doesn’t explicitly report when pods are being terminated due to OOM.
This change introduces a new monitor that emits the metric
pod.oomkilled, specifically tracking pods terminated with the reasonOOMKilled. The monitor identifies pods in a terminated state and reports OOM terminations, providing clearer visibility into OOM-related failures.Test plan for issue:
Is there any documentation that needs to be updated for this PR?
How do you know this will function as expected in production?