feat(#87): metrics gear — KPI band, Error Insights, drill-down drawer (#90)

* feat(#87): metrics gear — KPI band, Error Insights, drill-down drawer

The Metrics page used to be seven Chart.js panels and a "Recent Incidents"
list. When the 5xx chart spiked the user had no way to find out *which*
backend, *what* path, or *which* source IP was responsible — they had to
hunt through the Logs gear's many sources by hand.

Reworked the page into four stacked surfaces:

1. KPI summary band — six stat cards with sparklines and delta vs. the
   previous window: Requests/min, Avg Response, Error Rate %, 5xx Errors,
   Active Sessions, Healthy Backends N/M. Cards colour by health and
   delta arrows invert so "up" is red for errors, green for traffic.
2. Charts grid — same seven charts, now with crosshair tooltips
   (index-mode hover) and a gradient fill on the 5xx Errors chart so
   spikes pop visually.
3. Error Insights panel (replaces Recent Incidents) — three columns of
   top backends / source IPs / countries by 4xx+5xx count, each row
   clickable. Renders an empty-state pill when the window is quiet.
4. Drill-down drawer — slides in from the right with a per-backend
   summary, requests+errors mini-chart, status-code doughnut, top
   sources hitting the backend, and recent 5xx HAProxy log lines
   parsed live from the agent's haproxy log.

Backend additions (no new agent collection — everything sits on top of
the existing stats_history and traffic_flows tables):

- /api/{id}/metrics/summary?range=…
- /api/{id}/metrics/error-breakdown?range=…
- /api/{id}/metrics/backend/{name}/details?range=…
- /api/{id}/metrics/log-errors?status_min=500&lines=…&backend=…

The log-errors endpoint parses HAProxy access-log lines with a small
regex set and returns structured records (status / source IP / backend /
method / path). It degrades gracefully when the agent's logs aren't
reachable — the drawer shows a "logs unavailable" hint instead of
breaking the rest of the page.

22 new unit tests cover the helpers and the HAProxy log parser.

Closes #87

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(#87): plan for source-agnostic metrics dashboard

After the first-pass implementation, the deeper question is whether the
gear is HAProxy-only by accident. Audited the agent side and confirmed:
the bones are in place (gear.ProbeResult schema exists, discovery/
package exists with HAProxy/Docker/systemd detectors), but the
capability manifest endpoint that manager.go references is unshipped
and discovery isn't wired into the probe phase.

The new doc lays out a 9-phase plan that's strictly additive at each
step:

  0. Ship /api/v1/system/capabilities (the "upcoming" endpoint the
     codebase already references).
  1. Source-attribute every chart/KPI in the UI (wording only).
  2. Graceful no-HAProxy mode — show host metrics on plain boxes.
  3. discovery/{nginx,apache,caddy,traefik}.go probes.
  4. nginx metrics gear — first non-HAProxy source, proves the model.
  5. Promote access-log parsing agent-side with format profiles.
  6. Error Insights panel becomes multi-source.
  7. Apache / Caddy / Docker / Traefik.
  8. Optional cross-source aggregates.

Phases 0–2 are recommended as the next PR — they make the page honest
on non-HAProxy boxes and set up the capability manifest abstraction
that every later phase depends on.

A condensed version of the plan was posted as a comment on #87.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(#87): address Copilot review on metrics insights

Three findings, all valid:

1. The prev-window stats slice was un-trimmed. GetStatsHistory takes
   a lower bound only, so prevRaw spanned both windows; the `prevOnly`
   index list I computed was promptly discarded with `_ = prevOnly`,
   so every prev-window aggregate was actually a union aggregate and
   the resulting delta_pct was ~0 % regardless of actual change.
   Fixed by collapsing prevRaw to entries strictly before `since`.

2. APIMetricsLogErrorsHandler returned HTTP 403 when the user lacked
   `logs:view`, but the README and the frontend's loadLogErrors() both
   expected the structured {available:false, reason:…} envelope, so
   users without log access saw the generic "Failed to load log lines"
   error instead of the documented hint. Switched to the envelope.

3. escapeJSArg only handled the JS string context (\\ and '), but the
   value is interpolated into a double-quoted HTML onclick attribute,
   so an embedded " could break out and a < could start a new tag.
   Hardened to also escape &<>" — defence in depth on the source-IP
   path where values flow from agent-collected data.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: sarg3nt

docs/research/README.md

@@ -23,6 +23,11 @@ This directory contains active design documents and architectural research for t
   - Best-practice survey (Portainer, Lens, ArgoCD, Datadog, HAProxy stats, Grafana, NN/g)
   - Three layout options with ASCII mockups, recommendation, and implementation outline
 
+- **[metrics-source-agnostic.md](metrics-source-agnostic.md)**
+  - Plan to evolve the Metrics gear from "HAProxy with extras" to a source-agnostic dashboard with per-source attribution ([#87](https://github.com/sarg3nt/gearbox/issues/87))
+  - Surveys the existing capability/probe + discovery infrastructure on the agent side
+  - 9-phase rollout, each independently shippable, starting from a capability manifest endpoint and ending at multi-source (HAProxy + nginx + Apache + Caddy + Docker) drill-down
+
 ## Purpose
 
 These documents serve as:

docs/research/metrics-source-agnostic.md

@@ -682,6 +682,12 @@ func main() {
 			r.Get("/{boxID}/history/backend/{backendName}", h.APIBackendHistoryHandler)
 			r.Get("/{boxID}/incidents", h.APIIncidentsHandler)
 
+			// Metrics gear v2 — insights & drill-down endpoints
+			r.Get("/{boxID}/metrics/summary", h.APIMetricsSummaryHandler)
+			r.Get("/{boxID}/metrics/error-breakdown", h.APIMetricsErrorBreakdownHandler)
+			r.Get("/{boxID}/metrics/backend/{backendName}/details", h.APIMetricsBackendDetailsHandler)
+			r.Get("/{boxID}/metrics/log-errors", h.APIMetricsLogErrorsHandler)
+
 			// Disabled entities management
 			r.Get("/{boxID}/disabled-entities", h.APIDisabledEntitiesHandler)
 			r.Post("/{boxID}/disable-entity", h.APIDisableEntityHandler)