feat(#95): agent — metric-source primary selection + per-category overrides - #96
Merged
Conversation
…rrides First foundational PR for issue #95 (Phase 3 of #91). Establishes the "who's the primary metric source on this host" mechanism so future PRs can drop in nginx / Apache / Caddy / Traefik detectors without re- architecting how the dashboard picks whose numbers to render. Background: most hosts have one obvious producer per metric category (HTTP requests, response codes, etc.). Where two coexist — e.g. HAProxy fronting nginx, both genuinely serving traffic — the agent now picks one as primary using a built-in preference list and surfaces the choice in the capability manifest. The dashboard reads primary_sources and renders that source's data; alternatives travel along so a "switch source" UI can offer them. Operators override the pick per-category via env var for the edge case where auto-detection chooses wrong. Adds: - MetricCategory + MetricSourceGear interface (gear/source.go). One category defined today: CategoryHTTPRequests. The interface is opt-in — gears that don't produce metrics simply don't implement it. - preferenceOrder map keyed by category; first-match-wins among Available producers. Documented rationale for the homelab-flavoured ordering (HAProxy first as the L7 entry point). - Manager.ResolvePrimarySources() — resolves the primary for each category, honouring an operator override if present + valid, falling back to auto-detection with a warning otherwise. Producers not in preferenceOrder still surface as alternatives (alphabetised) so newer gears don't disappear from the UI before someone gets around to ranking them. - SourceSelection struct surfaced in CapabilitiesResponse under primary_sources — fields are stable JSON keys: source, reason, alternatives. - Per-category override env var: GEARBOX_AGENT_HTTP_SOURCE. Names a gear; case-insensitive, trimmed, lowercased at load. Validated against registered producers; invalid names log a warning and fall back rather than dropping metrics for the category. - Startup log line per resolved category so operators can confirm their override took effect via journalctl without hitting the API. - HAProxy gear declares CategoryHTTPRequests so the existing source works end-to-end immediately. Tests (8 new in gear/source_test.go, 3 new in config_test.go): - Auto-detect picks first Available from preferenceOrder. - Operator override wins over preference; reason names env var. - Override target unavailable → fall back to auto + warning. - Override target unknown / not a producer for the category → same. - No available producers → category omitted from result entirely. - Unranked producer still surfaces as alternative (alphabetised). - End-to-end /api/v1/system/capabilities includes primary_sources. - normaliseSourceOverride trims + lowercases edge cases. - Load() picks up GEARBOX_AGENT_HTTP_SOURCE; default is empty. Docs: README "Metric-source overrides" section explains the model and override behaviour, with explicit "auto-detection is the default" copy. Out of scope (later phases of #95): - nginx / Apache / Caddy / Traefik / Docker gear stubs. - Additional metric categories (backend health, container metrics). - Dashboard consumption of primary_sources. Refs: docs/research/metrics-source-agnostic.md (#91), issue #95.
Contributor
There was a problem hiding this comment.
Pull request overview
Foundational PR for issue #95 (Phase 3): adds the "primary metric source" mechanism in the gearbox-agent so future detector gears (nginx, Apache, Caddy, Traefik) can declare their categories and the agent can deterministically pick which gear's numbers represent each category on the host. Operators can override the auto-pick per category via env vars; the resolution is exposed in /api/v1/system/capabilities.
Changes:
- New
MetricCategory/MetricSourceGear/SourceSelectiontypes and aResolvePrimarySources()resolver with built-inpreferenceOrderand operator-override fallback semantics. - HAProxy gear declares
CategoryHTTPRequests; agent config +Dependenciescarry per-category override env var (GEARBOX_AGENT_HTTP_SOURCE). - Capability manifest gains a
primary_sourcesfield; startup logs each resolved category; comprehensive resolver tests + config tests added.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
gearbox-agent/README.md |
Documents the new override env var and behaviour. |
gearbox-agent/internal/gears/haproxy/plugin.go |
HAProxy declares CategoryHTTPRequests. |
gearbox-agent/internal/framework/gear/source.go |
New types: MetricCategory, SourceSelection, MetricSourceGear. |
gearbox-agent/internal/framework/gear/source_test.go |
Resolver + capabilities-endpoint integration tests. |
gearbox-agent/internal/framework/gear/manager.go |
ResolvePrimarySources(), preference order, override warnings, manifest field, startup logging. |
gearbox-agent/internal/framework/gear/dependencies.go |
New SourceOverrides field. |
gearbox-agent/internal/framework/config/config.go |
New HTTPSource config + normaliseSourceOverride helper. |
gearbox-agent/internal/framework/config/config_test.go |
Tests for normaliseSourceOverride and env-var loading. |
gearbox-agent/cmd/gearbox-agent/main.go |
buildSourceOverrides packs config into Dependencies.SourceOverrides. |
- manager.go: split ResolvePrimarySources (silent, called on every /api/v1/system/capabilities request) from ValidatePrimarySourceOverrides (logs warnings, called once from ProbeAll). The previous design re-logged the same override warning on every dashboard poll. - manager.go: rename deps_override → overrideForCategory; the snake_case was inconsistent with the rest of the file. - manager.go: replace hand-rolled containsString / withoutString with slices.Contains and slices.Clone + slices.DeleteFunc. Module is on Go 1.25; reaching for stdlib here trims maintenance surface. - manager.go: defensive slices.Clone of the Alternatives slice in SourceSelection so a future change to the local `ordered` slice can't alias into a returned value. - manager.go: replace overrideEnvVarFor's switch with a categoryEnvVar map keyed by MetricCategory; missing entry is a build-time-visible hole rather than a runtime "(unknown category)" placeholder leaking into operator-facing log messages and SourceSelection.Reason. - config_test.go: switch new tests from os.Setenv + t.Cleanup to t.Setenv. Safer under panics, idiomatic since Go 1.17. New tests: - TestEveryCategoryHasOverrideEnvVar: guards categoryEnvVar against forgotten entries when a new MetricCategory is added. - TestValidatePrimarySourceOverridesCountsMisses: confirms the validate-vs-resolve split — Validate reports mis-targeted overrides via its return value, Resolve stays silent.
7 tasks
sarg3nt
added a commit
that referenced
this pull request
May 15, 2026
) * feat(#95): agent — phase 3 source detection (nginx, Apache, Caddy, Traefik, Docker, host) Completes the remaining detection work for issue #95 on top of #96's primary-source selection. Six new probe-only gears land their verdicts in the capability manifest so the dashboard sees a complete picture of which HTTP / container sources exist on each host. No metrics collection yet — that lands in Phase 4+ per-source. The four web-server detectors declare CategoryHTTPRequests via MetricSourceGear, so the resolver from #96 now actually picks between alternatives — set GEARBOX_AGENT_HTTP_SOURCE=nginx on a host with both HAProxy and nginx and the manifest flips. Per-source env overrides for non-default surfaces: NGINX_STATUS_URL / NGINX_CONFIG_FILE APACHE_STATUS_URL / APACHE_CONFIG_FILE CADDY_ADMIN_URL TRAEFIK_METRICS_URL DOCKER_SOCKET Also drops the dead internal/framework/discovery/ package (superseded by the ProbeableGear interface from #93; nothing imported it). The discovery/docker.go detection logic was modest — binary lookup + os.Stat + systemctl is-active — and lives on in the new docker gear with the new manifest plumbing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#95): address Copilot review findings on PR #100 Substantive: - probe.isLoopback: parse URL with net/url + check net.IP.IsLoopback so userinfo-spoofed URLs like https://localhost@evil.com/ can't trick the helper into skipping TLS verification against evil.com. - probe.HTTPGet: validate maxBody > 0; return error instead of silently returning empty body that would hide sentinel mismatches. - docker probe: capture stat's FileInfo and check Mode()&ModeSocket so a regular file or directory at the socket path is flagged Inaccessible with a reason that points at the bind-mount mismatch. - traefik probe: track the last non-matching HTTP response across the fallback URL list; the Inaccessible reason now distinguishes "wrong service on this port (200 without sentinel)" from "no listener (connection refused)" so operators debug the right cause. Comment/text alignment: - caddy/traefik/apache Info().Description: Phase 4+ → Phase 7+ to match each file's package header and the docs. - nginx wellKnownConfigPaths comment: "all four" → "all three" (slice has three entries). - nginx Probe comment: "two `nginx -V` runs" → "one `nginx -v` and one `nginx -V`" — accurate process-invocation count. - docs/source-detection.md: "six-step decision tree" → "four-step" to match the rendered numbered list. New tests: - probe_test.go: isLoopback covers userinfo spoof regression, plain loopback variants, and unparseable input. HTTPGet rejects non-positive maxBody. - docker plugin_probe_test.go: new regression test for "path exists but is not a socket". Existing fakeFileInfo now carries a Mode field. - traefik plugin_probe_test.go: new test asserts the Inaccessible reason distinguishes "200 without Traefik sentinel" from generic "unreachable". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First foundational PR for #95 (Phase 3 of #91). Establishes the "who's the primary metric source on this host" mechanism so future PRs can drop in nginx / Apache / Caddy / Traefik detectors without re-architecting how the dashboard picks whose numbers to render.
The problem this addresses
Most hosts have one obvious producer per metric category (HTTP requests, response codes, etc.). Where two coexist — e.g. HAProxy fronting nginx, both genuinely serving traffic — the agent has no way today to say "use HAProxy's numbers, not nginx's." Without a primary-source concept, a host with multiple HTTP sources would either double-count or force the user to decide ad-hoc which numbers to trust.
The override env var fills the extreme-edge-case gap: when auto-detection picks wrong on a specific box, operators force the choice for that box without per-fleet config sprawl.
Design
Categories:
MetricCategoryis a grouping of metric data multiple sources could produce. One category defined today (CategoryHTTPRequests); more land with future phases (backend_health,container_metrics, etc.).Producer registration: Gears that produce metrics implement an opt-in
MetricSourceGearinterface declaring which categories they cover. HAProxy declaresCategoryHTTPRequests. Future gears will declare their own.Auto-detection: A built-in
preferenceOrderper category encodes a homelab-flavoured opinion — HAProxy first (the L7 entry point in this project), then nginx > Apache > Caddy > Traefik (rough order-of-prevalence). The resolver walks the order and picks the first gear that probed Available.Override: Per-category env var. Today:
GEARBOX_AGENT_HTTP_SOURCE=nginx. When set, the named gear wins — provided it actually probed Available and is registered for the category. If not, the agent logs a warning and falls back to auto-detection (losing HTTP metrics because an override's target isn't installed would be worse than serving auto-picked data).Exposure:
/api/v1/system/capabilitiesgrows aprimary_sourcesfield keyed by category. Each entry hassource,reason(auto-detect vs override), andalternatives(the other available producers for the category, in preference order). Categories with no available producer are omitted entirely — callers distinguish "no primary" via key absence.Startup log: One line per resolved category in the journal so operators can confirm overrides without hitting the API.
Files
gear/source.go(new)MetricCategorytype,CategoryHTTPRequestsconstant,MetricSourceGearinterface,SourceSelectionstruct.gear/source_test.go(new)gear/manager.goResolvePrimarySources(),preferenceOrdermap, override validation,primary_sourcesin the manifest, startup logging.gear/dependencies.goSourceOverrides map[MetricCategory]string.config/config.goHTTPSource stringfield;normaliseSourceOverridetrims + lowercases.config/config_test.gogears/haproxy/plugin.goMetricCategories() → [CategoryHTTPRequests].cmd/gearbox-agent/main.gobuildSourceOverrides(cfg)packs the per-category env-var fields into the Dependencies map.README.mdTest plan
go test ./...— 15 packages pass ongearbox-agent./api/v1/system/capabilitiesincludesprimary_sources.go vet ./...clean.light-hugger(HAProxy host): the manifest'sprimary_sources.http_requests.sourcereads"haproxy"with reason"auto-detected from preference order"and emptyalternatives(since only HAProxy is detected today). Override via env var still no-ops gracefully since nginx isn't installed.Backwards compatibility
Purely additive — no breaking changes.
CapabilitiesResponse.Gearstable unchanged. Older dashboards that ignoreprimary_sourcesget the same data they did before.primary_sources. Dashboard already treats missing fields as "use existing defaults" (PR feat(#91): metrics gear — source-aware, no-HAProxy mode (phases 0-2) #93 fail-open behaviour). No regression.What this PR is NOT
Per the issue #95 PR breakdown, this is PR A — the foundation. Out of scope:
backend_health,container_metrics— added when their second producer lands.primary_sources— gets surfaced in metrics-page UI in a separate PR once a real second source exists for HTTP.Design note vs. issue #95 wording
Issue #95 mentioned
GEARBOX_AGENT_DISABLE_SOURCESfor blocking detected sources wholesale. After clarification, the actual need is primary-source selection — picking which detected source wins for each metric category. This PR implements that. The block/allowlist concept can land separately if and when it becomes useful; it's a different control surface and shouldn't be conflated.I'll update the body of issue #95 to reflect this clarification once the PR lands.
References
docs/research/metrics-source-agnostic.md.🤖 Generated with Claude Code