context
Flaky alerts and alert validation before alert provisioning are a missing tool in the grafana ecosystem.
A useful tool to add to the grafanactl is to allow it to do alert validation. Where grafana control would 'simulate' the alert manager behavior and run the alerts on historical data. This would make validation of alerts easier and avoid flaky alerts.
possible implementation
To do this grafana has a 'v1/api/eval' which can be used to see the alert results. The eval api has a 'now' which can be set to the past and the alert manager code isn't that complicated.
grafanactl validate alert --from=2025-08-10 --to=2025-08-11 alert-file.yaml
So would you be interested in an implementation of this tool? Or maybe its something that better fits into the 'grafana' UI in the future than the provisioning tool.
appendix: brainstorm of more posibilities
- A 'validation' file could also be pulled from the historical alert data from grafana for existing rules.
- A 'validation' file could also be generated by the tool which can then be used as a 'golden standard'.
Maybe adding labels to alerts like 'false positive' could then get statistics from historical data where the updated alert gives you 'x false alerts, y real alerts, z unknown alerts`. when compared to the real data.
context
Flaky alerts and alert validation before alert provisioning are a missing tool in the grafana ecosystem.
A useful tool to add to the grafanactl is to allow it to do alert validation. Where grafana control would 'simulate' the alert manager behavior and run the alerts on historical data. This would make validation of alerts easier and avoid flaky alerts.
possible implementation
To do this grafana has a 'v1/api/eval' which can be used to see the alert results. The eval api has a 'now' which can be set to the past and the alert manager code isn't that complicated.
So would you be interested in an implementation of this tool? Or maybe its something that better fits into the 'grafana' UI in the future than the provisioning tool.
appendix: brainstorm of more posibilities
Maybe adding labels to alerts like 'false positive' could then get statistics from historical data where the updated alert gives you 'x false alerts, y real alerts, z unknown alerts`. when compared to the real data.