Skip to content

PrometheusRule CRDs with multiple alerts sharing one dash0.com/id silently lose all but the last alert #254

Description

@mmanciop

Problem

A PrometheusRule CRD with two or more alerting rules under one metadata.labels["dash0.com/id"] does not produce one check rule per alert when applied via check-rules create -f/apply -f — it silently collapses to a single check rule, named after whichever alert was processed last. Every earlier alert's check rule is overwritten and lost with no warning.

Verified directly against a live environment:

$ dash0 check-rules create -f rules.yaml   # CRD with alerts A and B, one shared dash0.com/id
Check rule "group - A" created
Check rule "group - B" updated
$ dash0 check-rules list --all -o json | jq '[.[] | select(.id == "<the shared id>")]'
[
  { "id": "<the shared id>", "name": "group - B" }
]

Only "B" exists; "A" is gone.

Root cause: every alert converted from one CRD receives the exact same Id value (ConvertPrometheusRuleToPrometheusAlertRule is called once per rule with the same ruleID, the CRD-level label), and the CLI's check-rule import path (ImportCheckRule) treats Id as the PUT upsert key — so each subsequent alert's create/update silently replaces the previous one under that id.

This is documented today only as an identifier-ambiguity note ("a CRD with multiple alerts shares one identifier — pin a unique label per CRD, or split multi-alert CRDs into one CRD per alert" in docs/commands.md's asset-identifiers table), which reads as an advisory rather than what it actually is: any multi-alert CRD sharing one id loses data on every apply. The suggested workaround ("pin a unique label per CRD") does not actually help — the collision is between alerts within the same CRD, not between CRDs, so a unique-per-CRD label still collides across that CRD's own alerts.

It also undermines apply --since's alerting-rule partial-removal detection (#253), which assumes multiple alerts sharing a CRD's id persist as separate, individually trackable check rules after apply — in practice they never do, so the "one alert removed while others survive" scenario that feature is designed to detect cannot occur through normal use of a shared-id CRD.

Acceptance criteria

  • Applying a PrometheusRule CRD with N alerting rules results in N live check rules, regardless of whether they share one dash0.com/id label.
  • No existing check rule is silently overwritten by an unrelated alert from the same CRD.
  • docs/commands.md's PrometheusRule identifier note is corrected once fixed (currently mischaracterizes this as an ambiguity to route around rather than a data-loss bug).

Reported while writing roundtrip tests for #253's apply --since/--force.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions