Skip to content

metrics-dashboard.md

Latest commit

 

History

History
203 lines (156 loc) · 5.71 KB

metrics-dashboard.md

File metadata and controls

203 lines (156 loc) · 5.71 KB
Raw

Measurable Outcomes Dashboard

From our teaching data: 89% of teams can't answer "Is AI helping?" This framework provides quantifiable metrics to prove ROI to leadership.

Five Key Metrics

1. Time Saved Per Week (Toil Reduction)

What it measures: Hours of manual troubleshooting replaced by automated agent analysis.

Target: 10-20 hours/week for a team of 5 SREs.

PromQL:

# Average time saved per agent execution (seconds)
avg(agent_execution_time_saved_seconds) by (agent_type)

# Total time saved this week
sum(increase(agent_time_saved_seconds_total[7d]))

2. Incidents Prevented/Detected

What it measures: Anomalies caught by agents before user impact.

Target: 15-30 per month after Phase 1 stabilization.

PromQL:

# Incidents detected per day
sum(increase(agent_incidents_detected_total[24h])) by (severity)

# Prevention rate (detected before user impact)
sum(agent_incidents_prevented_total) / sum(agent_incidents_detected_total) * 100

3. MTTR Improvement

What it measures: Reduction in mean time to resolution with agent-assisted troubleshooting.

Target: 40-50% improvement (e.g., 20min → 11min).

PromQL:

# MTTR with agent assistance
histogram_quantile(0.5, rate(incident_resolution_time_seconds_bucket{assisted="true"}[7d]))

# MTTR without agent assistance (baseline)
histogram_quantile(0.5, rate(incident_resolution_time_seconds_bucket{assisted="false"}[7d]))

4. False Positive Rate

What it measures: Percentage of agent alerts/recommendations that were not actionable.

Target: Below 15%. Above 25% indicates agents are generating noise, not value.

PromQL:

# False positive rate over 7 days
sum(increase(agent_recommendations_dismissed_total[7d]))
/
sum(increase(agent_recommendations_total[7d])) * 100

5. Cost Savings

What it measures: Dollar value of waste eliminated by cost optimization agents.

Target: $30K-60K/year for a mid-size cluster.

PromQL:

# Monthly cost savings from agent recommendations
sum(increase(agent_cost_savings_dollars_total[30d]))

# Over-provisioned resource value identified
sum(opencost_resource_waste_dollars) by (namespace, workload)

Grafana Dashboard Configuration

Panel Layout

Row 1: Executive Summary
  [Time Saved This Week] [Incidents Prevented] [MTTR Improvement %]

Row 2: Agent Activity
  [Executions/Hour]      [Error Rate]          [Avg Response Time]

Row 3: Quality Metrics
  [False Positive Rate]  [Tool Call Success %]  [Token Usage/Cost]

Row 4: Cost Impact
  [Monthly Savings]      [Waste Identified]     [GPU Cost Attribution]

Sample Dashboard JSON (Excerpt)

{
  "dashboard": {
    "title": "Platform Intelligence - AI Operations ROI",
    "tags": ["kagent", "ai-ops", "platform-intelligence"],
    "panels": [
      {
        "title": "Time Saved This Week",
        "type": "stat",
        "targets": [
          {
            "expr": "sum(increase(agent_time_saved_seconds_total[7d])) / 3600",
            "legendFormat": "Hours Saved"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "h",
            "thresholds": {
              "steps": [
                {"color": "red", "value": 0},
                {"color": "yellow", "value": 5},
                {"color": "green", "value": 10}
              ]
            }
          }
        }
      },
      {
        "title": "False Positive Rate",
        "type": "gauge",
        "targets": [
          {
            "expr": "sum(increase(agent_recommendations_dismissed_total[7d])) / sum(increase(agent_recommendations_total[7d])) * 100"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "unit": "percent",
            "max": 100,
            "thresholds": {
              "steps": [
                {"color": "green", "value": 0},
                {"color": "yellow", "value": 15},
                {"color": "red", "value": 25}
              ]
            }
          }
        }
      }
    ]
  }
}

Recording Rules

Add these to your Prometheus configuration to pre-compute dashboard metrics:

groups:
  - name: agent_roi_metrics
    interval: 60s
    rules:
      - record: agent:time_saved:weekly_hours
        expr: sum(increase(agent_time_saved_seconds_total[7d])) / 3600

      - record: agent:false_positive:rate7d
        expr: |
          sum(increase(agent_recommendations_dismissed_total[7d]))
          /
          sum(increase(agent_recommendations_total[7d])) * 100

      - record: agent:cost_savings:monthly_dollars
        expr: sum(increase(agent_cost_savings_dollars_total[30d]))

      - record: agent:mttr:improvement_percent
        expr: |
          (1 - (
            histogram_quantile(0.5, rate(incident_resolution_time_seconds_bucket{assisted="true"}[7d]))
            /
            histogram_quantile(0.5, rate(incident_resolution_time_seconds_bucket{assisted="false"}[7d]))
          )) * 100

Measurement Best Practices

  1. Baseline first. Collect 2-4 weeks of data before deploying agents to establish comparison metrics.
  2. Measure outcomes, not activity. "Agent made 500 tool calls" is not a useful metric. "Agent reduced MTTR by 45%" is.
  3. Track false positives aggressively. High false positive rates erode team trust faster than any benefit agents provide.
  4. Report monthly to leadership. Use the executive summary row for stakeholder updates.
  5. Iterate on thresholds. The targets above are starting points — calibrate to your environment after 30 days.