Skip to content

README.md

Latest commit

 

History

History
77 lines (52 loc) · 3.55 KB

README.md

File metadata and controls

77 lines (52 loc) · 3.55 KB
Raw

Grafana AI Assistant Research

Research and experimentation on Grafana Assistant's metric discovery capabilities — focusing on the Context Gap in enterprise environments and a proposed Self-Correcting Query Assistant concept.

Key Finding

When Grafana Assistant encounters opaque metric names (e.g., svc_kpi_007_v2 instead of node_cpu_seconds_total), its discovery rate drops to 0%. The Assistant relies on metric name semantics for understanding — a dependency that becomes critical as it evolves from query assistance to full dashboard authoring.

→ Full experiment writeup and results

What's in This Repo

File Description
experiment-results.md Complete experiment protocol, results, and proposed mitigation
screenshots/ Grafana Assistant test screenshots (6 images)
docker-compose.yaml Local Prometheus + Alloy stack
rules.yml Recording rules that generate opaque metric names
prometheus.yml Prometheus config with remote_write to Grafana Cloud
config.alloy Grafana Alloy forwarding config

The Problem

Enterprise observability environments don't follow community naming conventions. Metrics are often:

  • Auto-generated by provisioning tools (metric_autogen_4821_v3)
  • Legacy system artifacts (legacy_corp_svc_cpu_tick_v2)
  • Internal codenames (svc_kpi_007_v2)

The Assistant's current heuristic — string-matching metric names — works for node_cpu_seconds_total but fails completely for svc_kpi_007_v2, even though both represent CPU data.

This matters more as the mutation surface grows. When the Assistant only helped write queries, a wrong metric was a visible, recoverable error. When it authors dashboards (#118487), a wrong metric produces a panel that looks right but shows wrong data.

Proposed Mitigation: Self-Correcting Knowledge Base

A per-datasource metric knowledge base that maps opaque names → semantic descriptions + common query patterns, bootstrapped from:

  1. Label metadata — automatic extraction of job, team, unit labels
  2. LLM inference — initial semantic guesses from available signals
  3. User corrections — feedback loop where user fixes become permanent knowledge
svc_kpi_007_v2
├── description: "API latency P99 for payment service"
├── unit: milliseconds
├── team: fintech
├── common_queries: histogram_quantile(0.99, rate(...[5m]))
└── source: user_correction

The flywheel: AI guesses → user corrects → system learns → next query is better.

Quick Start

cp .env.example .env
# Edit .env with your Grafana Cloud credentials
docker compose up -d

Then open Grafana Assistant and compare:

  • "Show me CPU usage" → which metrics get discovered?
  • "Query svc_kpi_007_v2" → does the Assistant understand what it is?

Related Issues

Author

Baily Zhang — Product Manager exploring the intersection of AI and observability.

License

MIT