feat(#87): metrics gear — KPI band, Error Insights, drill-down drawer

[sarg3nt]

Summary

• Adds a 6-card KPI summary band (with sparklines + Δ vs prior window), an Error Insights panel (top backends / source IPs / countries by 4xx+5xx), and a slide-in drill-down drawer with per-backend mini-charts, top sources, and recent 5xx HAProxy log lines parsed live from the agent.
• All new endpoints sit on top of existing stats_history and traffic_flows tables — no new agent collection required.
• Crosshair tooltips and a gradient fill on the 5xx chart for the visual polish the issue asked for.
• Also includes a planning doc (docs/research/metrics-source-agnostic.md) outlining a 9-phase plan to make the gear source-agnostic so it works on plain Linux boxes and any detected web server / proxy. That doc is a follow-up — this PR doesn't implement it.

What changed

New backend (gearbox/internal/framework/handler/, gearbox/internal/framework/database/):

• GET /api/{id}/metrics/summary?range=… — KPI cards
• GET /api/{id}/metrics/error-breakdown?range=… — top backends/sources/countries
• GET /api/{id}/metrics/backend/{name}/details?range=… — drill-down data
• GET /api/{id}/metrics/log-errors?status_min=500&backend=… — recent parsed HAProxy log lines
• DB queries in a new metrics_insights.go file
• 22 new unit tests for KPI math, sparkline downsampling, log-line parsing, and range parsing

Dashboard UI (history.templ):

• KPI band at top, replacing the previous "scroll down to see anything" layout
• Error Insights panel replacing "Recent Incidents" (incidents are still queryable; we just don't surface them here anymore)
• Drill-down drawer (slides in from right) with Esc/backdrop close
• Crosshair tooltips and gradient fill applied to existing charts via the shared option helpers

Docs:

• gearbox/internal/gears/metrics/README.md rewritten to describe the new layout, endpoints, and data sources
• docs/research/metrics-source-agnostic.md — the follow-up plan
• docs/research/README.md — links the new plan

Issue answered

Closes #87.

The original issue asked three things, addressed as follows:

Issue ask	How it's addressed
"Errors skyrocketing — but where?"	Error Insights panel shows top backends by 5xx + 4xx, ranked. Click a row → drawer with per-backend mini-charts + top source IPs + recent 5xx log lines.
"Requests going up — who/what/where?"	KPI band shows requests/min with prior-window delta + sparkline. Drill-down drawer lists top source IPs hitting any backend.
"Traffic gear says 29% error rate to light-hugger — find it"	Clicking the relevant backend row in Error Insights opens the drawer with the live request-vs-error chart, status-code mix, top sources, and 5xx log lines.

Test plan

• make templ-generate && make build — clean build (verified locally)
• go vet ./... — clean (verified locally)
• go test ./... — full suite passes, including 22 new tests (verified locally)
• Visit /history on mjolnir with HAProxy data present — KPI band populates, Error Insights shows backends, click-through opens drawer with log lines
• Hover any chart — crosshair tooltip shows all series at the same x
• Click 5xx chart fullscreen icon — chart expands, gradient fill visible
• Drawer behaviour: Esc closes; backdrop click closes; close (×) button closes; opens cleanly on second/third use without leftover charts
• No regression on existing 7 charts or the time-range/resolution controls

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new "insights" surface to the Metrics gear: a 6-card KPI band, an Error Insights panel (top backends/IPs/countries by 4xx+5xx), and a slide-in drill-down drawer with per-backend mini-charts, top sources, and live-parsed HAProxy 5xx log lines. Backed by four new dashboard endpoints that aggregate over existing stats_history and traffic_flows tables (no agent changes). Also includes a research doc outlining a future source-agnostic plan.

Changes:

• New backend handlers (api_metrics_insights.go + helpers + DB queries) for KPI summary, error breakdown, backend details, and HAProxy log-error parsing, registered under /api/{boxID}/metrics/*.
• Major rework of history.templ: KPI band with sparklines, Error Insights replacing Recent Incidents, slide-in drill-down drawer, gradient fills + crosshair tooltips on existing charts.
• README rewrite for the metrics gear and a new 9-phase research doc on going source-agnostic.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
gearbox/cmd/server/main.go	Registers four new /metrics/* routes.
gearbox/internal/framework/handler/api_metrics_insights.go	KPI/error-breakdown/details/log-errors handlers + HAProxy log line parser.
gearbox/internal/framework/handler/api_metrics_insights_helpers.go	KPI math, sparkline downsampling, error-totals helpers.
gearbox/internal/framework/handler/api_metrics_insights_test.go	Unit tests for range parsing, KPI math, log parsing, downsampling.
gearbox/internal/framework/database/metrics_insights.go	New traffic_flows aggregation queries (top backends/sources/countries, per-backend timeseries, error-rate buckets).
gearbox/internal/framework/templates/pages/history.templ	KPI band, Error Insights panel, drill-down drawer markup, JS, and CSS; chart gradient/crosshair upgrades.
gearbox/internal/gears/metrics/README.md	Documents the new layout, endpoints, and data sources.
docs/research/metrics-source-agnostic.md	Phased plan for evolving Metrics into a source-agnostic surface.
docs/research/README.md	Index entry for the new research doc.

Comments suppressed due to low confidence (1)

gearbox/internal/framework/handler/api_metrics_insights.go:121

• TotalRequests in stats_history is the cumulative HAProxy request counter (lifetime since process start), not a per-bucket count. maxStatField(curr, ...) returns the largest value of that counter in the window, and perMinute(currReqMax, since, until) then divides that absolute counter by the window length. On a long-running HAProxy with, say, 100M lifetime requests, a 24h window will report ~70 000 requests/min regardless of how many requests actually arrived during that day. To get the actual rate, compute (max - min) of the counter in the window (handling counter resets if HAProxy restarted) and divide by the window minutes. The same issue applies to the TotalRequests sparkline — the current sparkline plots the cumulative counter, which is an almost-flat ramp rather than a meaningful "requests per bucket" trend. The previously-existing total_5xx_errors-based KPI has the same shape.

	currReqMax := maxStatField(curr, func(s database.StatsSnapshot) float64 { return float64(s.TotalRequests) })
	prevReqMax := maxStatField(prev, func(s database.StatsSnapshot) float64 { return float64(s.TotalRequests) })
	currReqRate := perMinute(currReqMax, since, until)
	prevReqRate := perMinute(prevReqMax, prevSince, since)
	cards = append(cards, kpiCard{
		Key:         "requests",
		Label:       "Requests / min",
		Value:       currReqRate,
		Unit:        "/min",
		Decimals:    1,
		PrevValue:   prevReqRate,
		DeltaPct:    pctDelta(currReqRate, prevReqRate),
		Status:      "good",
		Sparkline:   statSparkline(curr, func(s database.StatsSnapshot) float64 { return float64(s.TotalRequests) }, 30),
		Description: "Total HTTP requests served per minute, averaged across the window.",
	})