Problem
Pyrra generates multi-window, multi-burn-rate alerts. For a single SLO this means that several ErrorBudgetBurn alerts (fast, medium, slow, long-term) can and often do fire at the same time when an incident happens: a fast burn immediately trips, and the slower windows follow shortly after. The result is multiple notifications for what is, from an operator's point of view, one and the same problem.
The idiomatic way to deal with this in Alertmanager is an inhibition rule: the more urgent alert (fast burn) should inhibit the less urgent ones (medium/slow/long-term) for the same SLO. Today this is hard to express reliably, because there is no stable, semantic label that identifies which burn-rate tier an alert belongs to.
Why the existing labels don't solve this
Pyrra already attaches short, long, severity and exhaustion to each burn-rate alert (see slo/rules.go), but none of them is a good inhibition key:
severity — configurable via AlertingSeverities, and multiple tiers commonly share the same value (critical / warning). Using it as an inhibition key is ambiguous and breaks as soon as a user customizes severities.short / long — these are window durations. They technically differ per tier, but they change whenever the window configuration changes, which makes any inhibit_rules written against them brittle.
Proposed solution
Add a dedicated, semantic label to every generated burn-rate alert whose value identifies the tier, e.g.:
labels:
burnrate_tier: fast # one of: fast | medium | slow | long-term
This maps directly to the existing window index in slo/rules.go (0=fast, 1=medium, 2=slow, 3=long-term, see alertSeverityLabel), so the implementation is small: a helper analogous to alertSeverityLabel(i, w) that maps the window index to the tier name, set alongside the other alert labels.
Inhibition then becomes trivial and stable:
inhibit_rules:
- source_matchers:
- 'alertname="ErrorBudgetBurn"'
- 'burnrate_tier="fast"'
target_matchers:
- 'alertname="ErrorBudgetBurn"'
- 'burnrate_tier=~"medium|slow|long-term"'
equal: ['slo'] # plus any grouping labelsThe label is additive and set unconditionally (like short/long/exhaustion today), so no new configuration surface is needed.
Open questions / points to discuss
- Label name.
burnrate_tier is my suggestion. Alternatives: burnrate, burn_rate, window. Note burnrate overlaps naming-wise with the recording rules (BurnrateName), which is why I lean towards burnrate_tier.
- Value format. Short semantic values (
fast / medium / slow / long-term) for easy matchers vs. matching the CRD field style (fastBurn / mediumBurn / …).
equal labels for inhibition. Should be documented clearly — at minimum slo, plus grouping labels — so users can write correct inhibition rules.- Fingerprint change. Adding a label changes the fingerprint of existing alerts. Low risk since it's purely additive, but worth calling out for existing users.
Alternatives considered
- Reusing
severity / short / long for inhibition — rejected for the reasons above (ambiguous / brittle).
- Emitting four boolean labels (
fastBurn="true", …) — rejected; it bloats the label set and makes matchers more awkward than a single tier-valued label.
I'd really appreciate your feedback, as well as the open questions above. Thanks a lot for taking a look!
Problem
Pyrra generates multi-window, multi-burn-rate alerts. For a single SLO this means that several
ErrorBudgetBurnalerts (fast, medium, slow, long-term) can and often do fire at the same time when an incident happens: a fast burn immediately trips, and the slower windows follow shortly after. The result is multiple notifications for what is, from an operator's point of view, one and the same problem.The idiomatic way to deal with this in Alertmanager is an inhibition rule: the more urgent alert (fast burn) should inhibit the less urgent ones (medium/slow/long-term) for the same SLO. Today this is hard to express reliably, because there is no stable, semantic label that identifies which burn-rate tier an alert belongs to.
Why the existing labels don't solve this
Pyrra already attaches
short,long,severityandexhaustionto each burn-rate alert (seeslo/rules.go), but none of them is a good inhibition key:severity— configurable viaAlertingSeverities, and multiple tiers commonly share the same value (critical/warning). Using it as an inhibition key is ambiguous and breaks as soon as a user customizes severities.short/long— these are window durations. They technically differ per tier, but they change whenever the window configuration changes, which makes anyinhibit_ruleswritten against them brittle.Proposed solution
Add a dedicated, semantic label to every generated burn-rate alert whose value identifies the tier, e.g.:
This maps directly to the existing window index in
slo/rules.go(0=fast, 1=medium, 2=slow, 3=long-term, seealertSeverityLabel), so the implementation is small: a helper analogous toalertSeverityLabel(i, w)that maps the window index to the tier name, set alongside the other alert labels.Inhibition then becomes trivial and stable:
The label is additive and set unconditionally (like
short/long/exhaustiontoday), so no new configuration surface is needed.Open questions / points to discuss
burnrate_tieris my suggestion. Alternatives:burnrate,burn_rate,window. Noteburnrateoverlaps naming-wise with the recording rules (BurnrateName), which is why I lean towardsburnrate_tier.fast/medium/slow/long-term) for easy matchers vs. matching the CRD field style (fastBurn/mediumBurn/ …).equallabels for inhibition. Should be documented clearly — at minimumslo, plus grouping labels — so users can write correct inhibition rules.Alternatives considered
severity/short/longfor inhibition — rejected for the reasons above (ambiguous / brittle).fastBurn="true", …) — rejected; it bloats the label set and makes matchers more awkward than a single tier-valued label.I'd really appreciate your feedback, as well as the open questions above. Thanks a lot for taking a look!