rule: log PromQL info annotations at debug instead of warn#8873
Open
Goutham-Annem wants to merge 4 commits into
Open
rule: log PromQL info annotations at debug instead of warn#8873Goutham-Annem wants to merge 4 commits into
Goutham-Annem wants to merge 4 commits into
Conversation
filterOutPromQLWarnings logged every PromQL annotation at warn level, including purely informational ones (e.g. "metric might not be a counter"), causing log spam on every rule evaluation cycle for rules that trigger this heuristic. Add IsPromQLInfoAnnotation to distinguish info from warning annotations, and log info annotations at debug. Fixes thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Goutham-Annem
force-pushed
the
rule-log-promql-info-at-debug
branch
2 times, most recently
from
ad8003b to
0bb7fda
Compare
Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Goutham-Annem
force-pushed
the
rule-log-promql-info-at-debug
branch
from
0bb7fda to
46a09b8
Compare
Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
…ssue thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
filterOutPromQLWarningsincmd/thanos/rule.gologged every PromQL annotation atlevel.Warn, including purely informational ones like "metric might not be a counter" (annotations.PromQLInfo). Since this runs on every rule evaluation cycle for any rule whose query triggers the heuristic, it produces continuous log spam (see Rule: log warning spam from PromQL info "metric might not be a counter" #8229) rather than something an operator needs to act on.IsPromQLInfoAnnotationtopkg/extannotations/annotations.go, mirroring the existingIsPromQLAnnotationbut checking only theannotations.PromQLInfoprefix.filterOutPromQLWarningsnow logs info-type annotations atlevel.Debugand keeps true warning-type annotations (annotations.PromQLWarning) atlevel.Warn. Filtering behavior (which warnings get removed from the response) is unchanged — only the log level for info-type annotations changes.Verification
TestIsPromQLInfoAnnotation(pkg/extannotations/annotations_test.go) covering info vs. warning vs. unrelated strings.TestFilterOutPromQLWarnings_LogLevel(cmd/thanos/rule_test.go), which captures logger output vialog.NewLogfmtLoggerand asserts info annotations producelevel=debugwhile warning annotations producelevel=warn.TestFilterOutPromQLWarnings(filtering behavior) still passes unchanged.make go-lintpasses with 0 issues.Closes #8229