Skip to content

Alert rules from charms in destroyed models remain on disk and get triggered #780

@sinapah

Description

@sinapah

Bug Description

Assume a COS Lite is storing telemetry for another model and that Grafana Agent machine 2/stable is the agent collecting telemetry and pushing it into COS Lite. If that model is destroyed before the relation between Grafana Agent and Prometheus is removed, then the alert rules forward through GA to Prom remains on Prom's disk. As a result, since the model is destroyed, alerts such as HostMetricsMissing keep getting triggered.

To Reproduce

  1. Deploy COS Lite.
  2. Deploy a machine model with GA 2/stable in it and relate GA to Prom over a CMR.
  3. Destroy the model.
  4. Alerts such as HostMetricsMissing for the Grafana Agent that was destroyed will remain on disk in Prometheus.

Environment

TBC.

Relevant log output

N/A

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions