feat(#91): dashboard — multi-source metrics page (nginx/Apache/Caddy/Traefik) - #102
Merged
Conversation
…Traefik) Closes #91. Dashboard-side companion to PR #101's agent work: DB (new source_stats table): - Aggregate rollups per (server_id, source, collected_at). Distinct table from traffic_flows because non-HAProxy sources don't emit per-IP/per-backend detail — they expose top-level counters via stub_status / mod_status / Prometheus. Adding a source column to traffic_flows would mean a forest of NULL columns for every non-HAProxy row. - Stable rollups: requests_total, response_{2xx,3xx,4xx,5xx} (where the source emits per-status-class), active_connections, plus an extra_json column for source-specific fields (nginx reading/writing/waiting; Apache worker pool; Caddy request_errors_total; Traefik entrypoints). - SaveSourceStats / GetLatestSourceStats / GetSourceStatsRange / PruneSourceStats helpers in source_stats.go. Agent client: - GetSourceStats(source) returns the agent's per-source /stats JSON as a flexible map — the four sources have different shapes; the collector normalises into the common SourceStatsSnapshot. - GetAccessLogRecent(source, statusMin, limit) hits the agent's /api/v1/access-log/{source}/recent endpoint and returns the AccessLogResponse envelope verbatim. Collector: - SourceStatsCollector polls every supported source each persistHistory tick. Sources whose gear hasn't probed Available return 503/404 from the agent — silently swallowed via isExpectedSourceMiss so the journal stays clean on HAProxy-only hosts. One source's failure doesn't skip the others. - Per-source normalisers map agent JSON → SourceStatsSnapshot: nginx requests/active + reading/writing/waiting → extra; Apache total_accesses + busy_workers + idle/cpu/uptime → extra; Caddy requests_total + request_errors_total → extra; Traefik full status-class breakdown surfaced as first-class columns. - Wired into manager.persistHistory; lifecycle-safe with the existing stats persistence. Handlers + routes: - /api/{boxID}/metrics/source/{source}/summary — per-source latest snapshot + history range for the chart cards. Gated by the agent capability for that source so empty/unknown render with reason rather than a misleading zero-data card. - /api/{boxID}/metrics/source/{source}/log-errors — Phase 5 refactor. Proxies to the agent's structured access-log endpoint rather than parsing log text dashboard-side. Accepts "haproxy" too — the agent's parser is the unification point. Existing /metrics/log-errors stays untouched for now so older dashboard clients don't break mid-rollout. UI (metrics.templ): - Four new per-source chart cards (nginx, Apache, Caddy, Traefik), each with the same data-source / data-source-card hooks the existing cards use. Hidden by default; applyCapabilities() flips them on per-source based on the agent's probe table. - "Per-Source Recent Errors" section below the existing HAProxy Error Insights panel — Phase 6's multi-block view. One block per Available non-HAProxy source via the new log-errors endpoint; built with createElement + textContent so agent-supplied fields can't inject HTML. - applyCapabilities() now iterates all sources, exposes window._availableSources so the loaders can skip absent sources rather than 404-spamming the agent. - loadSourceMetrics renders per-source Chart.js cards (nginx → stacked connection state, Apache → workers, Caddy → requests vs errors, Traefik → status-class stacked area). Phases addressed (all of remaining #91): - Phase 4 (nginx end-to-end): agent collector + dashboard card. - Phase 5 (generic access-log abstraction): dashboard log-errors now consumes the agent's structured records. - Phase 6 (multi-source Error Insights): new section. - Phase 7 (Apache / Caddy / Traefik end-to-end): same shape as nginx. - Phase 8 (cross-source aggregates): explicitly deferred per #91 — marked optional, best designed once a real second source has driven the UX. - Docker metrics (Phase 7 piece): out of scope here, lives in the feature/containers-gear branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds dashboard-side support for multi-source Metrics cards and recent-error panels for nginx, Apache, Caddy, and Traefik, building on the agent-side collectors/access-log endpoints from the linked work.
Changes:
- Adds source stats persistence, collection, and API handlers.
- Adds dashboard routes and UI cards/sections for per-source metrics and errors.
- Extends source labels/constants and agent client methods for source stats and structured access logs.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
gearbox/internal/framework/templates/pages/metrics.templ |
Adds per-source chart cards, error section, and JS loaders/renderers. |
gearbox/internal/framework/handler/api_metrics_sources.go |
Adds per-source summary and log-error dashboard API handlers. |
gearbox/internal/framework/handler/api_metrics_insights.go |
Adds source constants and label helper. |
gearbox/internal/framework/database/traffic.go |
Adds source_stats schema. |
gearbox/internal/framework/database/source_stats.go |
Adds source stats model and DB helpers. |
gearbox/internal/framework/collector/source_stats.go |
Adds collector to poll agent source stats and normalize snapshots. |
gearbox/internal/framework/collector/source_stats_test.go |
Adds normalizer and expected-miss tests. |
gearbox/internal/framework/collector/manager.go |
Wires source stats collection into history persistence. |
gearbox/internal/framework/agent/client.go |
Adds source stats and access-log client methods/types. |
gearbox/cmd/server/main.go |
Registers new per-source metrics routes. |
Critical (charts wouldn't render): - metrics.templ: per-source cards called a nonexistent chartDefaults() helper and configured Chart.js with `type: 'time'` despite the page not loading a date adapter. Both would have thrown at render. Switch to the existing createChartOptions(0) + formatTime() string-label pattern that the other chart cards already use. No new dependencies, identical visual treatment. Functional bugs: - collector: isExpectedSourceMiss matched substring "status 503" in err.Error(), but agent.APIError.Error() returns the Message body not a status-coded string — so 503/404 from the agent surfaced as warnings every history tick instead of being silently swallowed. Use errors.As(*agent.APIError) and inspect StatusCode directly. - database: source_stats was excluded from ClearMetricsData, CleanupMetricsByAge, and CleanupMetricsBySize. Per-source rows would have grown unbounded and survived "clear metrics data". Wire into all three paths (the column is server_id, not box_id — the traffic-analysis schema predates the box_id naming). - database: GetSourceStatsRange ordered ASC LIMIT, which on long ranges with the 1-minute history interval returns the OLDEST rows in the window, hiding recent data. Switch to DESC LIMIT then reverse to chronological order before return. - metrics.templ: per-source summary URL didn't pass the selected range; every card ignored the hours-select control and used the handler's 24h default. Add hoursSelectToRange() helper + ?range= query param. - metrics.templ: chart datasets plotted monotonic counters directly (requests_total, response_*xx, etc.), so the cards showed ever-rising lines rather than per-interval rates. New counterDeltas() helper diffs successive samples (resets land as 0, matching Prometheus rate() semantics). Applied to Caddy requests + errors, Traefik status-class bands, and nginx's new requests line. - metrics.templ: nginx card was titled "Connections & Requests" but the dataset only rendered reading/writing/waiting. Add a request-rate line on a secondary y-axis so it doesn't get crushed by the connection-state stack. - handlers: log-errors source set included Traefik, but the agent has no Traefik access-log parser (Traefik logs flow through Prometheus, not the access-log endpoint). Listing Traefik always proxied to a 404 from the agent. Split into two sets — supportedSourceSummaries (nginx/apache/caddy/traefik) for the summary endpoint, supportedLogErrorSources (haproxy/nginx/ apache/caddy) for log-errors. - metrics.templ: the visibility gate for the "Per-Source Recent Errors" section used the wider chart set (anyNonHAProxy) but the renderer only walks SOURCES_WITH_ERROR_LOGS. On a Traefik-only host the section would have unhid empty. Pull the source list into two named constants (SOURCES_WITH_CHARTS, SOURCES_WITH_ERROR_LOGS) and gate the section on the renderer set. - metrics.templ: toggleFullscreen's exit branch removed `hidden` from every other chart-card, including capability-gated per-source cards. After fullscreening any card, unavailable source cards would become visible. Check data-source-card + window._availableSources during the un-hide pass; capability- hidden cards stay hidden. - agent client: GetAccessLogRecent omitted status_min when 0, causing the agent to apply its own default of 500. The handler accepts 0 as "disable filter" but couldn't actually request it. Change to pass status_min when >= 0; negative values fall back to the agent default (no caller uses that path today). Doc + a11y: - handlers: dropped a comment claiming a per-source handler test keeps the source lists in sync. No such test exists; the comment would have given a false sense of coverage. Added a cross-reference to collector.SupportedSources() instead. - metrics.templ: per-source error tables had only a tbody. Added a visually-hidden <caption> + visible <thead> with scoped column headers so screen readers announce column context. - metrics.templ: the per-source-errors refresh button was icon- only with a `title` attribute. Added aria-label for assistive technology that doesn't read title attributes reliably. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #91. Dashboard-side companion to #101's agent work. The Metrics page now renders per-source chart cards (nginx connections, Apache workers, Caddy requests/errors, Traefik status-class) and a "Per-Source Recent Errors" section, all behind capability gates so a homelab box that only runs HAProxy looks exactly the same as it does today.
This is the second of the two-PR plan I proposed for finishing #91:
After this lands, all of #91's phases except the explicitly-optional Phase 8 ("cross-source aggregates" — Combined toggle, stacked-area "by source" chart) are done. That phase is best designed once at least one real second source on light-hugger drives the UX.
Phases addressed
gearbox/gearbox/metrics.templgearbox/Docker metrics (Phase 7 piece) stays out — that work lives on
feature/containers-gearwith its own moby/client integration. After this and that both land, #91 closes andfeature/containers-gearrebases against the now-current main (see project memory on the homelab side).Files
New
internal/framework/database/source_stats.goSourceStatsSnapshot+SaveSourceStats/GetLatestSourceStats/GetSourceStatsRange/PruneSourceStats.internal/framework/collector/source_stats.gonginx/apache/caddy/traefikevery persistHistory tick, normalises agent JSON →SourceStatsSnapshot, persists. Silently skips 503/404 (source not Available on host).internal/framework/collector/source_stats_test.goisExpectedSourceMiss.internal/framework/handler/api_metrics_sources.goAPIMetricsSourceSummaryHandler+APIMetricsSourceLogErrorsHandler. The latter is the Phase-5 refactor — proxies to the agent's structured access-log endpoint instead of parsing log text client-side.Modified
internal/framework/database/traffic.gosource_statstable + index added toinitTrafficSchema.internal/framework/agent/client.goGetSourceStats(source)+GetAccessLogRecent(source, statusMin, limit)client methods.internal/framework/collector/manager.gopersistHistory.internal/framework/handler/api_metrics_insights.gosourceLabel()helper.cmd/server/main.go/{boxID}/metrics/source/{source}/summaryand/{boxID}/metrics/source/{source}/log-errors.templates/pages/metrics.templloadSourceMetrics()/loadSourceErrors()+ multi-sourceapplyCapabilities(). UI built viacreateElement+textContentso agent-supplied path / timestamp / method fields can't inject HTML.Schema design note — why a new table instead of extending
traffic_flowsI considered adding a
sourcecolumn totraffic_flowsper #91's original plan. Decided against it because the non-HAProxy sources don't emit per-IP/per-backend detail — they expose top-level rollups via stub_status / mod_status / Prometheus. Adding asourcecolumn would mean every non-HAProxy row carries NULL forsource_ip,backend_name, etc.; HAProxy data would stay shaped as today; and the unique constraint would have to growsourcefor both shapes to coexist.A separate
source_statstable is cleaner: HAProxy's per-IP rollups stay intraffic_flowsuntouched (existing chart cards see no diff); aggregate rollups for the new sources live insource_statswith theextra_jsonoverflow column for per-source nuances. If/when per-IP data lands for the other sources via access-log parsing, that's the right moment to revisit — at that point asourcecolumn ontraffic_flows(or a separate per-source flows table) would be justified.The
extra_jsoncolumn matters: nginx wants reading/writing/waiting/accepts/handled; Apache wants idle_workers/cpu_load/uptime; Caddy wants request_errors_total; Traefik wants entrypoints. Squeezing those into named columns would either commit to a least-common-denominator surface or pile up sparse columns. JSON keeps the schema additive when new sources arrive.Capability handling
caps.Entry("nginx")(etc.) — same pattern PR feat(#91): metrics gear — source-aware, no-HAProxy mode (phases 0-2) #93 established for HAProxy.hidden;applyCapabilities()flips them on per-source.window._availableSourcesis exposed soloadSourceMetrics/loadSourceErrorscan skip absent sources rather than 404-spamming the agent on every render.Test plan
go test ./...— all packages pass ongearbox.go vet ./...clean.gofmt -lclean on every file added or modified in this PR.templ generateclean —metrics_templ.goregenerated and excluded from VCS as expected (*_templ.gogitignored).collected_at(falls back to now), and theisExpectedSourceMissswallow logic for 503/404 responses.light-hugger: HAProxy box; the four per-source cards stay hidden, "Per-Source Recent Errors" section stays hidden, HAProxy data continues to render as before. (HAProxy-only host = no visual change.)Backwards compatibility
source_statstable — additive; existing schemas untouched./source/{source}/...paths; existing/metrics/summary,/metrics/log-errors, etc. behave identically.applyCapabilities()still hides HAProxy cards when haproxy isn't Available, identical to today. The new per-source logic is purely additive.isExpectedSourceMissswallows the 404s so dashboard logs stay clean. The collector simply persists no rows for those sources, the chart cards stay hidden via capability gates.What's NOT in this PR
feature/containers-gearwith its own moby/client integration. Reconcile when both PRs land./metrics/log-errorsendpoint with the new per-source one. The old endpoint stays untouched here so an in-flight rollout can ramp safely — the new front-end calls into/source/{source}/log-errorsfor the multi-source section. Once the rollout fully migrates, the old endpoint can be removed in a follow-up.References
🤖 Generated with Claude Code