Layered Error Insights — host-agnostic signals + source-specific modules (design)

[sarg3nt]

Summary

The current Error Insights panel on the metrics page is HAProxy-shaped: three columns of "top backends / source IPs / countries by 5xx," fed by a traffic_flows SQL query. That framing is wrong for a tool that's meant to monitor any host — a box without a reverse proxy has nothing to render. We need to refactor "Error Insights" into a layered design: a host-agnostic Layer 1 that every monitored box gets for free, plus pluggable Layer 2 source-specific modules (HAProxy, nginx, Apache, Caddy, Traefik, future Postgres/Redis) that light up when the agent detects the source is present.

This issue is a research + design umbrella, not an implementation. The goal is to land an agreed-upon design before any code changes.

Why this matters (the prompting incident)

A real-world investigation surfaced both the framing problem and a useful test case for "what should Error Insights be able to tell me?":

A 5xx storm appeared on HAProxy's chart — ~225 5xx accumulating over 18 hours. Manual diagnosis required SSH'ing to the proxy host, grepping access logs by status code and backend, sampling a 503 line to see the URL path, then reading the dashboard's own container logs to find that the cert/TLS error that was driving the 5xx was buried in docker logs. None of that signal made it into the UI. The Error Insights panel said "Everything's green" the entire time, because it reads a SQL table that's only populated when somebody visits the Traffic Analysis page.

Two distinct deficiencies became clear:

1. The aggregation Error Insights performs already has the right data source available (the agent's accesslog gear, exposed via /api/v1/access-log/{source}/recent and proxied by the dashboard's per-source-errors handler). The existing Error Insights panel just doesn't use it — it reads traffic_flows, populated only on-demand from a different code path.
2. Many of the most useful "where is something broken?" signals aren't HAProxy-shaped at all — they're host-agnostic: a service in restart loop, an OOM-kill, a disk filling up, an agent that can't reach itself. Those should be Layer 1.

Proposed layered design

Layer 1 — host-agnostic signals (every monitored box gets these)

Surfaced as a small set of "what's wrong on this box right now" cards, always rendered on the metrics page even when no reverse proxy is detected.

Candidate signals — pick a starter subset, leave the rest as Layer-1 plugins:

• Agent / collector self-health. Last successful collect per data source, consecutive failure count, last error verbatim. (This is what would have flagged the prompting incident.)
• Systemd unit failures. Units in failed state; recent restart-loop detection.
• Kernel / dmesg events. OOM-kills, filesystem read-only events, hardware faults.
• Auth failures. sshd journal, fail2ban hits.
• Disk pressure. Filesystems near full; growth-rate detection.
• Container errors (if Docker present). Containers in restart loops, exited containers with non-zero codes.
• Clock skew. NTP drift past a threshold.

Layer 2 — source-specific insight modules (light up when source is detected)

The agent already returns capability info (api_capabilities.go) — we extend it so each detected source registers a Layer-2 module. The dashboard asks "what modules apply to this box?" and renders them in priority order under the Layer 1 cards.

Initial modules:

• HAProxy — the current Error Insights logic, rewritten to drive off the agent's /api/v1/access-log/haproxy/recent instead of traffic_flows. Adds a per-URL-path column (which was the clue in the prompting incident — /htmx/mjolnir/metrics made the self-inflicted nature obvious).
• nginx / Apache / Caddy — the existing per-source-errors panel becomes a proper Layer-2 module under the unified shell, instead of a separate section.
• Traefik — Prometheus-only (no access log on the agent today). Module exposes a status-class breakdown derived from traefik_*_total{code=~"5.."} counters.

Future modules — out of scope for this design but the interface should accommodate them: Postgres slow queries, Redis evictions, application-log error aggregation per backend.

Platform pieces this design implies

• Source-capability detection in the agent, registered per gear. Already partially there.
• A module registry in the dashboard, with priority/ordering and an "applies to this box?" predicate.
• A common UI shell: page header → Layer 1 cards → N Layer-2 sections, each module owning its column-shape. Drill-down drawer is shared infrastructure.
• A single canonical fetch path per source. Today HAProxy has both /api/{box}/metrics/log-errors (legacy, dashboard-side parser) and /api/{box}/metrics/source/haproxy/log-errors (newer, agent-side parser). These should converge.
• Honest empty states. If a Layer-1 signal is unavailable (no journalctl access in the container, logs:view permission missing), say so — don't render silence as health.

Open design questions for the research phase

1. Module boundary — do Layer 2 modules live agent-side (each ships its parser, exposes a structured endpoint) or dashboard-side (agent ships raw records, dashboard parses)? Current code is mixed (accesslog parsing is agent-side; GeoIP enrichment is dashboard-side). Worth picking a side.
2. Storage — do Layer 1 errors land in the alerts table, in a new aggregated table, or stay ephemeral and re-fetched per page-load like the current source-errors panel? Each has trade-offs for "show me yesterday's errors" vs. memory footprint.
3. Relationship to /alerts — is Error Insights the read-only "right now" view and /alerts the historical/managed view? Or do they fold together? Currently they're disjoint UIs reading disjoint data.
4. Naming — once it's host-general, "Error Insights" might not be the right top-level label. "What's wrong" / "Issues" / "Diagnostics" all in the running. Want a single name early so the design doc doesn't slip.
5. Layer 1 plugin model — is each Layer 1 signal a fixed-set hard-coded thing, or does it use the same plugin/gear pattern Layer 2 does? (Probably the same pattern, but it's a design choice.)
6. Per-box config — which Layer 1 signals are on by default, which are opt-in (e.g. fail2ban hits get noisy on a box that's actively being scanned), and where does the config live?
7. Permission boundaries — logs:view is currently the gate for log-derived insights. Does it still hold for Layer 1? Should the agent-health card require fewer permissions since it's about the platform, not host content?

Out of scope for this issue

• Any code changes. This is research + design + a sign-off on direction before implementation issues get filed.
• The cert/SAN agent deployment work — separate issue.
• The 5xx-rate anomaly-alert idea I raised earlier ("when 5xx-per-minute jumps above baseline, alert") — folds in once Layer 1 alerting plumbing is decided.

Deliverable

A design doc (committed to docs/ in this repo) that:

1. Picks a side on each open question above.
2. Lists the Layer 1 starter set and which agent-side capabilities each one needs.
3. Specifies the Layer 2 module interface.
4. Sketches the UI shell.
5. Identifies follow-up issues to file for each implementable chunk.

When that's merged, this umbrella issue can close and the work gets broken into concrete implementation issues.