feat(#91): agent — phase 4/5/7 metrics collection (nginx, Apache, Caddy, Traefik) + access-log - #101
Merged
Merged
Conversation
…dy, Traefik) + access-log endpoint Builds on PR #100's detection layer with periodic metric scraping for the four web servers plus a structured access-log endpoint. This finishes the agent-side work for issue #91; the dashboard side (per-source chart cards, multi-source Error Insights, DB migration) ships as the next PR. Per-source collectors (each adds a CollectorGear collector + /api/v1/{name}/stats endpoint, cached snapshot, force=true synchronous re-scrape): - nginx: parses stub_status (active/reading/writing/waiting + monotonic accepts/handled/requests). - apache: parses mod_status?auto's key-value format (Total Accesses, worker pool, CPU load, ReqPerSec, etc.). - caddy: scrapes Prometheus at :2019/metrics; sums caddy_http_requests_total + request_errors_total; flags admin status via caddy_admin_http_requests_total presence. - traefik: scrapes Prometheus; buckets traefik_router_requests_total by status-class label so the dashboard gets a real 2xx/3xx/4xx/5xx breakdown; also surfaces the entrypoints list. Access-log endpoint (Phase 5): - New internal/framework/services/accesslog/ package with 5 profile parsers: haproxy, nginx-combined, apache-common, apache-combined, caddy-json. The dashboard's existing parseHAProxyLogLine is ported into the haproxy profile byte-for-byte (with one regex tightening: the syslog [pid] bracket no longer claims the date match). - New internal/gears/accesslog/ gear: GET /api/v1/access-log/{source}/recent?status_min=500&limit=500 reads the last N lines of the source's access log via tail, parses each line with the matching profile, filters by status_min, returns newest-first. - 4 new env vars to override default log paths (HAPROXY_ACCESS_LOG, NGINX_ACCESS_LOG, APACHE_ACCESS_LOG, CADDY_ACCESS_LOG). Apache falls back from /var/log/apache2/ to /var/log/httpd/ for RHEL hosts. Capability manifest reports which sources have a readable log on this host. Shared helper: internal/framework/services/promtext/ — minimal Prometheus exposition-format parser (samples + label maps; counter sums; SumByNameWithLabel for status-class extraction). Scoped to the agent's needs to avoid pulling in prometheus/common's 50+-package transitive footprint just for two scrape routines. Test coverage: - Each new collector has unit tests covering parser correctness, scrape success / failure modes, 503 before first scrape, cached response shape, force=true behaviour, override resolution. - Access-log gear tests cover probe verdict, capabilities map, unknown source 404, no-log available=false envelope, status_min filtering, limit cap, tail-failure surfacing, isReadable's non-regular-file rejection. - 5 parser profiles each have happy-path + reject-noise tests including HAProxy negative-Tt handling and Caddy non-HTTP entries. - promtext tests cover summation, label-value escapes, malformed lines, trailing scrape timestamps. 23 files added, 7 modified. go build / vet / test / gofmt all clean. Out of scope (Phase 6, 8 + dashboard wire-up): - DB migration adding source column to traffic_flows. - Multi-source Error Insights (dashboard refactor). - Cross-source aggregates (Phase 8 — optional). - Source-aware ingest from these endpoints to traffic_flows. - Per-source chart cards / KPIs / capability gates. These all live in the dashboard repo and ship as the next PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the agent-side metrics collection for nginx, Apache, Caddy, and Traefik on top of PR #100's detection layer, plus a new structured access-log endpoint (Phase 5 of #91). Each detector gear gains a periodic Prometheus / status-page scrape with a cached snapshot exposed at /api/v1/{source}/stats, and a new accesslog gear surfaces parsed log records at /api/v1/access-log/{source}/recent. A small promtext package and an accesslog parser package (5 profile parsers) are introduced as shared services.
Changes:
- New per-source collectors (nginx/apache/caddy/traefik) with cached snapshots,
?force=truedebug path, and 503-before-first-scrape semantics. - New
accessloggear +accesslogparser package consolidating HAProxy/nginx/Apache/Caddy log parsing behind one endpoint. - Shared minimal Prometheus text parser (
promtext) and 4 new*_ACCESS_LOGenv vars for log path overrides.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
gearbox-agent/internal/gears/{nginx,apache,caddy,traefik}/collector.go |
Periodic scrape + cached Stats + /api/v1/{source}/stats handler. |
gearbox-agent/internal/gears/{nginx,apache,caddy,traefik}/collector_test.go |
Per-collector unit tests for parse, scrape, override, 503 envelope. |
gearbox-agent/internal/gears/{nginx,apache,caddy,traefik}/plugin.go |
Adds collector-time fields, drops no-op RegisterRoutes, refreshes Description. |
gearbox-agent/internal/gears/accesslog/plugin.go |
New gear: /api/v1/access-log/{source}/recent with override + default-path resolution. |
gearbox-agent/internal/gears/accesslog/plugin_test.go |
Probe / handler / clamp / readability tests. |
gearbox-agent/internal/framework/services/accesslog/{parser,haproxy,nginx_combined,apache,caddy_json}.go |
Common Record + 5 profile parsers; ports HAProxy parser from dashboard. |
gearbox-agent/internal/framework/services/accesslog/{parser,profiles}_test.go |
Profile dispatch + per-profile parsing tests. |
gearbox-agent/internal/framework/services/promtext/promtext.go + test |
Minimal Prometheus exposition-format parser (Parse, SumByName, escape handling). |
gearbox-agent/internal/framework/config/config.go |
4 new *_ACCESS_LOG env-var fields. |
gearbox-agent/internal/framework/gear/dependencies.go |
Matching *AccessLog Dependencies fields. |
gearbox-agent/cmd/gearbox-agent/main.go |
Registers the new accesslog gear and wires the access-log overrides. |
- access-log: implement the documented Apache CLF fallback. Previously the handler used a single parser from sourceProfile (ApacheCombined) and the comments + PR body claimed a per-record fallback to ApacheCommon that didn't exist. RHEL hosts running default CLF would have produced zero parsed records. New parseWithFallback helper + sourceFallbackProfile map drive the actual fallback now; Apache is the only source using it today. - access-log: status_min query parameter now accepts an explicit 0 to disable the filter. Was previously clamped to a 100 minimum, which silently coerced 0 to 100 and broke the "give me all records" intent. Default when the param is absent stays 500 (the dashboard's primary use case). Lock the new defaults in via two new tests. - caddy: drop the AdminRunning field. The previous heuristic relied on caddy_admin_http_requests_total existing, which Prometheus doesn't emit for counters with zero increments. A freshly-started Caddy with admin enabled but no admin traffic yet would have falsely read "admin disconnected." The real signal is "did the scrape succeed?" — which the handler already conveys via 503 before the first successful scrape — so the field was redundant on success and misleading on cold start. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
sarg3nt
added a commit
that referenced
this pull request
May 15, 2026
…Traefik) (#102) * feat(#91): dashboard — multi-source metrics page (nginx/Apache/Caddy/Traefik) Closes #91. Dashboard-side companion to PR #101's agent work: DB (new source_stats table): - Aggregate rollups per (server_id, source, collected_at). Distinct table from traffic_flows because non-HAProxy sources don't emit per-IP/per-backend detail — they expose top-level counters via stub_status / mod_status / Prometheus. Adding a source column to traffic_flows would mean a forest of NULL columns for every non-HAProxy row. - Stable rollups: requests_total, response_{2xx,3xx,4xx,5xx} (where the source emits per-status-class), active_connections, plus an extra_json column for source-specific fields (nginx reading/writing/waiting; Apache worker pool; Caddy request_errors_total; Traefik entrypoints). - SaveSourceStats / GetLatestSourceStats / GetSourceStatsRange / PruneSourceStats helpers in source_stats.go. Agent client: - GetSourceStats(source) returns the agent's per-source /stats JSON as a flexible map — the four sources have different shapes; the collector normalises into the common SourceStatsSnapshot. - GetAccessLogRecent(source, statusMin, limit) hits the agent's /api/v1/access-log/{source}/recent endpoint and returns the AccessLogResponse envelope verbatim. Collector: - SourceStatsCollector polls every supported source each persistHistory tick. Sources whose gear hasn't probed Available return 503/404 from the agent — silently swallowed via isExpectedSourceMiss so the journal stays clean on HAProxy-only hosts. One source's failure doesn't skip the others. - Per-source normalisers map agent JSON → SourceStatsSnapshot: nginx requests/active + reading/writing/waiting → extra; Apache total_accesses + busy_workers + idle/cpu/uptime → extra; Caddy requests_total + request_errors_total → extra; Traefik full status-class breakdown surfaced as first-class columns. - Wired into manager.persistHistory; lifecycle-safe with the existing stats persistence. Handlers + routes: - /api/{boxID}/metrics/source/{source}/summary — per-source latest snapshot + history range for the chart cards. Gated by the agent capability for that source so empty/unknown render with reason rather than a misleading zero-data card. - /api/{boxID}/metrics/source/{source}/log-errors — Phase 5 refactor. Proxies to the agent's structured access-log endpoint rather than parsing log text dashboard-side. Accepts "haproxy" too — the agent's parser is the unification point. Existing /metrics/log-errors stays untouched for now so older dashboard clients don't break mid-rollout. UI (metrics.templ): - Four new per-source chart cards (nginx, Apache, Caddy, Traefik), each with the same data-source / data-source-card hooks the existing cards use. Hidden by default; applyCapabilities() flips them on per-source based on the agent's probe table. - "Per-Source Recent Errors" section below the existing HAProxy Error Insights panel — Phase 6's multi-block view. One block per Available non-HAProxy source via the new log-errors endpoint; built with createElement + textContent so agent-supplied fields can't inject HTML. - applyCapabilities() now iterates all sources, exposes window._availableSources so the loaders can skip absent sources rather than 404-spamming the agent. - loadSourceMetrics renders per-source Chart.js cards (nginx → stacked connection state, Apache → workers, Caddy → requests vs errors, Traefik → status-class stacked area). Phases addressed (all of remaining #91): - Phase 4 (nginx end-to-end): agent collector + dashboard card. - Phase 5 (generic access-log abstraction): dashboard log-errors now consumes the agent's structured records. - Phase 6 (multi-source Error Insights): new section. - Phase 7 (Apache / Caddy / Traefik end-to-end): same shape as nginx. - Phase 8 (cross-source aggregates): explicitly deferred per #91 — marked optional, best designed once a real second source has driven the UX. - Docker metrics (Phase 7 piece): out of scope here, lives in the feature/containers-gear branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#91): address Copilot review findings on PR #102 Critical (charts wouldn't render): - metrics.templ: per-source cards called a nonexistent chartDefaults() helper and configured Chart.js with `type: 'time'` despite the page not loading a date adapter. Both would have thrown at render. Switch to the existing createChartOptions(0) + formatTime() string-label pattern that the other chart cards already use. No new dependencies, identical visual treatment. Functional bugs: - collector: isExpectedSourceMiss matched substring "status 503" in err.Error(), but agent.APIError.Error() returns the Message body not a status-coded string — so 503/404 from the agent surfaced as warnings every history tick instead of being silently swallowed. Use errors.As(*agent.APIError) and inspect StatusCode directly. - database: source_stats was excluded from ClearMetricsData, CleanupMetricsByAge, and CleanupMetricsBySize. Per-source rows would have grown unbounded and survived "clear metrics data". Wire into all three paths (the column is server_id, not box_id — the traffic-analysis schema predates the box_id naming). - database: GetSourceStatsRange ordered ASC LIMIT, which on long ranges with the 1-minute history interval returns the OLDEST rows in the window, hiding recent data. Switch to DESC LIMIT then reverse to chronological order before return. - metrics.templ: per-source summary URL didn't pass the selected range; every card ignored the hours-select control and used the handler's 24h default. Add hoursSelectToRange() helper + ?range= query param. - metrics.templ: chart datasets plotted monotonic counters directly (requests_total, response_*xx, etc.), so the cards showed ever-rising lines rather than per-interval rates. New counterDeltas() helper diffs successive samples (resets land as 0, matching Prometheus rate() semantics). Applied to Caddy requests + errors, Traefik status-class bands, and nginx's new requests line. - metrics.templ: nginx card was titled "Connections & Requests" but the dataset only rendered reading/writing/waiting. Add a request-rate line on a secondary y-axis so it doesn't get crushed by the connection-state stack. - handlers: log-errors source set included Traefik, but the agent has no Traefik access-log parser (Traefik logs flow through Prometheus, not the access-log endpoint). Listing Traefik always proxied to a 404 from the agent. Split into two sets — supportedSourceSummaries (nginx/apache/caddy/traefik) for the summary endpoint, supportedLogErrorSources (haproxy/nginx/ apache/caddy) for log-errors. - metrics.templ: the visibility gate for the "Per-Source Recent Errors" section used the wider chart set (anyNonHAProxy) but the renderer only walks SOURCES_WITH_ERROR_LOGS. On a Traefik-only host the section would have unhid empty. Pull the source list into two named constants (SOURCES_WITH_CHARTS, SOURCES_WITH_ERROR_LOGS) and gate the section on the renderer set. - metrics.templ: toggleFullscreen's exit branch removed `hidden` from every other chart-card, including capability-gated per-source cards. After fullscreening any card, unavailable source cards would become visible. Check data-source-card + window._availableSources during the un-hide pass; capability- hidden cards stay hidden. - agent client: GetAccessLogRecent omitted status_min when 0, causing the agent to apply its own default of 500. The handler accepts 0 as "disable filter" but couldn't actually request it. Change to pass status_min when >= 0; negative values fall back to the agent default (no caller uses that path today). Doc + a11y: - handlers: dropped a comment claiming a per-source handler test keeps the source lists in sync. No such test exists; the comment would have given a false sense of coverage. Added a cross-reference to collector.SupportedSources() instead. - metrics.templ: per-source error tables had only a tbody. Added a visually-hidden <caption> + visible <thead> with scoped column headers so screen readers announce column context. - metrics.templ: the per-source-errors refresh button was icon- only with a `title` attribute. Added aria-label for assistive technology that doesn't read title attributes reliably. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First half of #91's remaining work — all the agent-side metrics collection for the four non-HAProxy web servers, plus the structured access-log endpoint from Phase 5. Builds directly on PR #100's detection layer. The dashboard wire-up (DB migration, source-aware chart cards, multi-source Error Insights) lives in this PR's follow-up.
Each detector gear now also has a collector that periodically scrapes its source's metric surface, caches the normalised snapshot, and exposes it at
/api/v1/{source}/stats. The dashboard's existing capability-driven hide/show pattern means new sources show up in the manifest immediately on Phase-3 land and the dashboard can iterate at its own pace.Docker metrics intentionally deferred — that work overlaps with the in-progress
feature/containers-gearbranch and should land in that PR rather than here, per the discussion on #100.What's in this PR
Per-source metrics collectors (Phase 4 + Phase 7 agent)
nginxstub_status(text)/api/v1/nginx/statsapachemod_status?auto(key:value)/api/v1/apache/statscaddy:2019/metrics/api/v1/caddy/statscaddy_http_requests_total,request_errors_total, admin presencetraefik:8082/metrics(or:8080)/api/v1/traefik/statstraefik_router_requests_totalbucketed by status-class label + entrypointsEach collector:
NGINX_STATUS_URL,APACHE_STATUS_URL,CADDY_ADMIN_URL,TRAEFIK_METRICS_URL) — no override-resolution drift between probe-time and scrape-time.?force=truetriggers a fresh scrape for debugging.traefik_sentinel.Access-log endpoint (Phase 5)
New
internal/framework/services/accesslog/package with five profile parsers (haproxy,nginx-combined,apache-common,apache-combined,caddy-json) producing a commonRecordshape. The dashboard's existingparseHAProxyLogLineis ported into thehaproxyprofile byte-for-byte, with one regex tightening so thesyslog[pid]bracket no longer claims the date match (that was a latent dashboard-side bug; the existing fixtures never carried the syslog wrapper).New
internal/gears/accesslog/gear exposesGET /api/v1/access-log/{source}/recent?status_min=500&limit=500&lines=2000:/var/log/apache2/to/var/log/httpd/for RHEL hosts.tail -n(same pattern the existinglogsgear uses — handles rotation and partial writes for free).limitnewest-first records.available=falseenvelope (200 OK, not 500) when the source has no readable log file on this host — keeps the dashboard's Error Insights panel rendering a friendly "logs unavailable" state instead of breaking.Four new env vars to override defaults:
HAPROXY_ACCESS_LOG,NGINX_ACCESS_LOG,APACHE_ACCESS_LOG,CADDY_ACCESS_LOG.Shared helper
internal/framework/services/promtext/— minimal Prometheus exposition-format parser. Sample + label-map + numeric value,SumByNameandSumByNameWithLabelhelpers, escape handling for\\/\"/\n, tolerates the optional trailing scrape timestamp. Scoped to what Caddy/Traefik scrapers actually need so we don't drag inprometheus/common's 50+-package transitive footprint just for two scrape routines.Files
23 added, 7 modified. ~3400 LOC including tests.
internal/framework/services/accesslog/(new)internal/framework/services/promtext/(new)internal/gears/accesslog/(new)/api/v1/access-log/{source}/recentgear + tests.internal/gears/{nginx,apache,caddy,traefik}/collector.go(new)/statshandler per source.internal/gears/{nginx,apache,caddy,traefik}/collector_test.gointernal/gears/{nginx,apache,caddy,traefik}/plugin.gointernal/framework/config/config.go*_ACCESS_LOGenv vars.internal/framework/gear/dependencies.go*AccessLogfields.cmd/gearbox-agent/main.goaccessloggear; wires the new override fields.Backwards compatibility
Purely additive. All new endpoints are under fresh paths (
/api/v1/{nginx,apache,caddy,traefik}/stats,/api/v1/access-log/{source}/recent). The capability manifest's existing keys are unchanged; the new collectors' gears already appeared in PR #100. Older dashboards talking to this agent simply don't call the new endpoints. The dashboard side of the work (which DOES call them) is staged for the next PR.Test plan
go test ./...— all 26 packages pass.go vet ./...clean.gofmt -lclean on every file added or modified in this PR.isReadable.-body-bytes handling.light-hugger(HAProxy host):/api/v1/nginx/statsreturns 503 (nginx not installed);/api/v1/access-log/haproxy/recent?status_min=500returns recent HAProxy 5xx records parsed by the haproxy profile.Out of scope — the follow-up PR
Per the conversation context, this is the agent side of the remaining #91 work. The dashboard side comes next:
ALTER TABLE traffic_flows ADD COLUMN source TEXT NOT NULL DEFAULT 'haproxy'+ index./api/v1/{source}/statsendpoints on the same cadence as HAProxy and persists per-source rows./metrics/log-errorsendpoint that proxies through to the new/api/v1/access-log/{source}/recent.feature/containers-gearbranch's PR — that work already has a richer Docker client integration and shouldn't compete with a parallel implementation here.Once that follow-up lands, #91 closes.
References
🤖 Generated with Claude Code