Capability-driven autoconfig: every gear page must honor the active box's actual capabilities (umbrella)

[sarg3nt]

Summary

The dashboard renders gear pages as if every box looks like light-hugger (a bare-host HAProxy node). On a box like mjolnir — a TrueNAS Scale host whose gearbox-agent runs in a distroless container and advertises only access-log, host, and metrics gears — almost every page is broken:

• Bx grid — Mjolnir shows a red dot even though the agent is healthy.
• Metrics page — HAProxy panels render even though Mjolnir has no HAProxy gear; nothing else renders.
• Logs page — Error loading logs: Unexpected token 'F', "Failed to "... is not valid JSON.
• Services page — same Unexpected token 'F' frontend error.
• Alerts — empty (no signals are wired up to capabilities).
• OS Updates — shows light-hugger's packages regardless of which box is selected in the header pill.

Underlying cause: capability data flows from the agent to the dashboard, but the dashboard only consults it on the Gears settings page (filterGearsByAgentCapabilities). The actual gear pages, their data sources, the HAProxy dashboard tiles, the default landing route, and the OS Updates page do not consult it at all — they assume HAProxy + journalctl + apt + docker are all available everywhere.

This is the umbrella to turn that around: capabilities become the source of truth for what the dashboard renders, on a per-box basis. Related but narrower work lives in #106 (cert SANs + container deployment mounts) and #111 (Test Connection probe audit); this issue is the wider rendering-layer fix that has to land for those to feel coherent.

Evidence from the live deployment

mjolnir agent startup table (relevant excerpt):

GEAR          STATUS    REASON
access-log    enabled
apache        disabled  no apache2 or httpd binary on PATH
caddy         disabled  no caddy binary on PATH
certificates  disabled  neither certbot nor acme.sh found on PATH or common install paths
docker        disabled  no docker binary on PATH
haproxy       disabled  stats socket configured at /run/haproxy/admin.sock but does not exist
host          enabled
logs          disabled  neither journalctl nor tail found on PATH; cannot stream any log source
metrics       enabled
nginx         disabled  no nginx binary on PATH
security      disabled  neither fail2ban-client nor nft found on PATH
traefik       disabled  no traefik binary on PATH
traffic       disabled  stats socket configured at /run/haproxy/admin.sock but does not exist
updates       disabled  no supported package manager found on PATH (apt-get/apt/dnf/yum/zypper/apk)

Capabilities are exposed at GET /api/v1/system/capabilities (gearbox-agent/internal/framework/gear/manager.go:757) and wrapped by the dashboard at GET /api/{boxID}/capabilities (internal/framework/handler/api_capabilities.go). The data is there — nothing downstream uses it.

Per-symptom triage

1. Bx grid red dot — Agent unreachable: ... tls: failed to verify certificate

/bx/api/status for Mjolnir returns level=red, reachable=false because the local dev gearbox isn't running with GEARBOX_INSECURE_TLS=true (only the production compose sets it — see homelab apps/gearbox/docker-compose.yml). #106 is the proper fix (env-configurable SANs + regenerate-on-change). This issue inherits that dependency for the red-dot, but the other symptoms below persist even after TLS verifies.

2. HAProxy tiles render for every box; metrics tiles render nothing useful for Mjolnir

Default landing route is /haproxy. That page polls /htmx/{boxID}/stats and /htmx/{boxID}/metrics for every enabled box including ones whose capabilities table says haproxy: disabled. Returns 503 No stats available (plain text), tiles stay empty.

Expected: render only the per-box stat tiles for boxes whose capability table includes haproxy and traffic. For boxes without HAProxy, render an overview tile sourced from whatever metrics gears the agent did enable (host + access-log for Mjolnir).

3. Error loading logs: Unexpected token 'F', "Failed to "... is not valid JSON

Two layered bugs:

• Backend returns plain-text errors via http.Error(w, "Failed to ...", 500) with content-type: text/plain — e.g. api_certificates.go, api_logs.go, os_updates.go. Some routes (apperrors.WriteHTTPError) already write a JSON envelope; the rest don't.
• Frontend does await response.json() without checking status or content-type, so a 5xx returns "Failed to ..." text and the JSON parser throws.

There's a deeper issue too: APILogSourcesHandler falls back to a hardcoded [haproxy, system] list when no settings are saved (api_logs.go:67-78). On Mjolnir neither source exists, so the JS tries to fetch /api/mjolnir/logs/haproxy, the agent rejects it (no logs gear), and the page degrades to the unhandled error.

Expected: the source list comes from the agent's access-log / logs gear capabilities (resources field — see proposed extension below). Hardcoded fallbacks go away.

4. Services page — same JSON parse error

APIServicesHandler (api_services.go:48-100) requires serverConfig.UsesAgentAPI() and forwards to the agent's GetServices. The agent's services story is currently coupled to systemd (systemctl), which isn't available inside the distroless container. Hits the same plain-text-error → JSON-parse-throw path as logs.

Expected: the page consults the services capability for the box. If unavailable, render an empty-state ("Services monitoring not available — this gear requires systemd/dbus access; see configuration docs") instead of a hostile error toast.

5. Alerts — empty

alerts is tagged as a "dashboard concept" in dashboardGearToAgentGear (gears.go:88-95) and not gated by capability at all. It still needs a per-box capability contract — e.g. "alerts requires metrics gear at minimum, can light up with logs/services/security capabilities as they exist." Today it silently renders nothing.

6. OS Updates shows the wrong box

OSUpdatesPage and every /api/os-updates/* handler resolves the box from r.URL.Query().Get("server") and falls back to h.getDefaultServerID(). The header pill stores the user's active box in the gearbox_active_box cookie, but OS Updates never reads the cookie. Other gear pages have a resolveBoxIDFromRequest helper (handler.go:281-308) that prefers explicit URL param → cookie → default; OS Updates doesn't use it.

Expected: a single canonical "resolve active box" helper, used by every page and API handler. Pages without a ?server= always honor the cookie.

Root-cause framing

There are three independent regressions that compound into the user-visible mess:

1. Capability awareness is shallow. Today it gates only the gears settings list. It needs to gate:
   • Which gears appear in the per-box sidebar.
   • Which tiles render inside each gear page.
   • The default landing route per box (today it's hardcoded /haproxy).
   • The data sources offered by Logs / Services / Metrics / OS-Updates.
2. Capabilities don't carry enough detail. The agent returns { gears: { name: { status, reason, capabilities: map[string]string } } }. That's enough to answer "is this gear available?" but not "what resources does it expose?" — which is what the dashboard needs to render the Logs source picker, the Services list, the Metrics source list, the OS-Updates package-manager column, etc.
3. Active-box resolution is inconsistent. Three different conventions in tree (gearbox_active_box cookie, ?server= query param, getDefaultServerID() fallback) that don't always agree.

Plus a transverse bug: error responses break the frontend because they're plain text and the JS unconditionally JSON.parses.

Proposed plan

The work splits into four phases; phases 2 and 3 are the strategic ones.

Phase 1 — Stop the bleeding

Triage-level fixes that unblock the current Mjolnir deployment without changing architecture.

• JSON error envelope everywhere. Audit all http.Error(w, ..., 4xx|5xx) calls under internal/framework/handler/ and internal/gears/*/handlers.go; replace with apperrors.WriteHTTPError (which already emits JSON for Accept: application/json). Add a frontend wrapper that detects non-JSON responses and surfaces a clean toast instead of Unexpected token 'F'.
• OS Updates honors the active-box cookie. Replace the inline r.URL.Query().Get("server") / getDefaultServerID() pair with resolveBoxIDFromRequest in OSUpdatesPage and every /api/os-updates/* handler. Add a regression test that switching the header pill changes the rendered packages.
• Document the dev GEARBOX_INSECURE_TLS requirement until gearbox-agent: cert SANs and Docker deployment requirements (umbrella) #106 lands so local dev doesn't see the red-dot symptom while running this issue's other fixes.

Phase 2 — Capability-driven UI rendering

The foundation. Every dashboard page consults the active box's capability table before deciding what to render.

• Add a Handler.requireCapability(boxID, gearName) helper that returns the cached capability entry (already cached for 5 min — capabilityCacheTTL).
• Default landing route: instead of hardcoded /haproxy, compute it from the active box's capabilities. Preference order: haproxy → metrics → bx overview. Persist as a per-user / per-box preference once Metrics page: per-user, per-box draggable layout (GridStack) #103 metrics layout work is in (it already has the precedent).
• HAProxy dashboard tiles: filter the per-box tile list by capabilities.haproxy.status == available before issuing the /htmx/{box}/stats|metrics polls. Eliminates the 503 storm.
• Sidebar gear list per box: when the active box's capability table excludes logs, hide the Logs nav entry for that box (or render it disabled with a tooltip). Same for Services, OS Updates, Certificates, Traffic.
• Logs page: the source picker is built from capabilities.access-log.resources + capabilities.logs.resources (see Phase 2 extension below). Drop the hardcoded [haproxy, system] fallback in APILogSourcesHandler.
• Services page: if capabilities.services is unavailable, render an empty-state card with a link to the docs explaining what the gear needs (systemd / dbus / docker socket). No error toasts.
• Metrics page: the source list (host / nginx / apache / caddy / traefik / access-log) is filtered to whichever metric sources the agent actually advertises.
• Bx grid: when a box's capabilities table is reachable but degraded (only host+metrics available), the grid should color the row amber (not red) and show "agent online — limited monitoring" in the hover. Red stays for "agent unreachable."

Phase 2 extension — Richer capability response

The current CapabilityEntry only carries status, reason, and a capabilities: map[string]string (today mostly used for versions). Extend it so the agent advertises the concrete resources each gear can serve:

gears:
  access-log:
    status: available
    reason: "haproxy access log found at /var/log/haproxy/access.log"
    capabilities:
      version: "2.8"
    resources:
      log_sources:
        - { name: "haproxy", display_name: "HAProxy access log", path: "/var/log/haproxy/access.log" }
  logs:
    status: available
    resources:
      log_sources:
        - { name: "fail2ban", display_name: "fail2ban", source: "journalctl" }
        - { name: "sshd",     display_name: "SSH",      source: "journalctl" }
  services:
    status: available
    resources:
      services: ["haproxy", "gearbox-agent", "nftables", "fail2ban"]
      actions:  ["start", "stop", "restart"]
  updates:
    status: available
    capabilities:
      package_manager: "apt"
    resources:
      snapshots_supported: true
      pip_supported: false
  metrics:
    status: available
    resources:
      sources: ["host", "access-log"]

The dashboard then renders source pickers / lists / column visibility directly from resources — no hardcoded HAProxy/systemd assumptions anywhere in the rendering path.

Wire diagram (see implementation hint in the agent's manager.go):

Agent gear plugin
  ↓ implements ProbeResources() (in addition to existing Probe())
Agent CapabilitiesResponse
  ↓ resources field added per gear
Dashboard CapabilitiesCache
  ↓ replays into APICapabilityEntry.Resources
Dashboard handlers / templates
  ↓ render source pickers, sidebar entries, default landing route from resources

Phase 3 — Container deployment as a first-class story

(Substantial overlap with #106 — this phase is what uses the cert/mount fixes that lands there; coordinate scope.)

The Mjolnir agent runs as a distroless container with no host bind mounts, which is why so many gears come up disabled. We need to decide which gears can run with no host access (today: host, metrics, access-log), which can run if specific mounts are provided, and which are fundamentally service-deployment-only.

Deliverables (some land in #106, others here):

• A "container deployment compose" reference in this repo, with each mount documented inline:
  • /var/log ro → log file tailing for logs gear without journalctl
  • /run/systemd + /run/dbus ro → systemd-based services gear
  • /var/run/docker.sock ro → docker gear
  • /etc/letsencrypt ro → certificates gear (already documented)
  • /proc ro + /sys ro → richer host metrics
• Agent capability probe should distinguish "binary not on PATH" (gear can never work in this container) from "binary present, host introspection mount missing" (gear could work if the operator mounts X). Today these collapse into the same disabled row; the dashboard should be able to surface "fixable" reasons in the Add Box flow.
• Update apps/gearbox-agent/docker-compose.yml on the homelab repo to match the reference (it's the canary). Today's compose has zero host mounts beyond the agent's own data dir — that's why nothing introspective works.

Phase 4 — Unify box-selection

Once Phases 1–3 are in, finish the consolidation:

• Single resolveActiveBox(r) (BoxConfig, error) helper used by every page handler and every API handler. Precedence: explicit URL/path param → ?server= query → gearbox_active_box cookie → first-enabled-box default.
• setActiveBoxCookie is called from a single shared header-pill HX handler; the cookie is the only client-visible state.
• WebSocket reconnects with the new box context when the active box changes, so /api/events only streams events for the selected box.
• Add a regression test (Playwright or Go test against templated HTML) that switching boxes via the header pill changes the rendered page contents on every gear page.

Out of scope

• Re-architecting the plugin model (the gear interface already supports the metadata we need; this issue is about plumbing it through).
• ZFS / TrueNAS-middleware / IPMI / SMART gears (own issues, will benefit from the capability-resource extension once it lands).
• Replacing the agent's gear-probe with a periodic re-probe loop (today probe runs only at startup — fine for now; revisit if operators start adding capabilities to a running agent).
• Per-source insight modules (Error Insights design — Layered Error Insights — host-agnostic signals + source-specific modules (design) #107).

Acceptance

• On a box with only host / metrics / access-log gears enabled, the dashboard sidebar shows only the entries that capability table justifies — no broken Logs / Services / OS-Updates links.
• No page in the dashboard shows Unexpected token 'F', "Failed to "... is not valid JSON. Errors render as a clean toast sourced from a JSON envelope.
• The default landing for a box without HAProxy is the Bx overview (or whichever capability-derived destination we settle on), not /haproxy with empty tiles.
• OS Updates honors the active-box cookie. Switching boxes in the header pill changes the rendered packages.
• The Logs source picker is populated from the agent's access-log/logs capability resources, not the [haproxy, system] hardcode.
• The agent's /api/v1/system/capabilities response carries per-gear resources (log sources, services, metrics sources, package manager). Dashboard rendering reads from those fields.
• Homelab apps/gearbox-agent/docker-compose.yml ships with the documented mount set so its capability surface matches what an operator would expect from "monitor this Linux box."

Related

• gearbox-agent: cert SANs and Docker deployment requirements (umbrella) #106 — gearbox-agent cert SANs + container deployment mounts (umbrella for the agent-side fixes Phase 3 depends on).
• Test Connection probe 404s on boxes without HAProxy; audit all probes for server-side API key handling #111 — Test Connection probe assumes HAProxy + audit of server-side API-key handling (narrower symptom of the same "HAProxy is hardcoded everywhere" problem).
• Layered Error Insights — host-agnostic signals + source-specific modules (design) #107 — Layered Error Insights design (consumes the richer capability response from this issue's Phase 2 extension).

[sarg3nt]

Progress: 6 PRs landed against this issue overnight

Phase 1 ("stop the bleeding") and most of Phase 2 ("capability-driven UI") are in PR form, awaiting review.

Phase 1 — Stop the bleeding

PR	Title	What it does
#113	fix(os-updates): honor active-box cookie via resolveBoxIDFromRequest	Replaces 33 open-coded r.URL.Query().Get("server") / getDefaultServerID() fallbacks in os_updates.go with the existing resolveBoxIDFromRequest helper. Switching boxes in the header pill now actually changes what /os-updates and every /api/os-updates/* endpoint act on.
#114	fix(errors): emit JSON envelope from WriteHTTPError + harden fetch callers	errors.WriteHTTPError now emits {"success": false, "message": "..."} with Content-Type: application/json instead of http.Error's plain-text default. Logs / Services / Certificates / Security pages updated to parse the envelope and surface .message cleanly. Kills the Unexpected token 'F' toast on the Logs and Services pages.

Phase 2 — Capability-driven UI

PR	Title	What it does
#115	feat(haproxy): filter dashboard tiles by agent haproxy capability	New ServerAdapter.GetEnabledServersWithGearAvailable(gearName) (parallel cold-cache fetches, fail-open). HAProxy gear's overview + status-grid handlers filter through it, so boxes without HAProxy stop generating the 503 burst on `/htmx/{box}/stats
#116	feat(logs): derive default log sources from agent capabilities	APILogSourcesHandler no longer hardcodes [haproxy, system]. The default source list is derived from the agent's access-log / logs capabilities. Older-agent forward-compat preserved via caps.Has() check.
#118	feat(haproxy): render empty-state when no HAProxy-capable boxes	Stacked on #115. Instead of redirecting to /settings/boxes when no haproxy-capable boxes exist (but other boxes do), render a friendly InfoAlert pointing at Bx and Settings.
#119	feat(sidebar): hide gears whose agent reports them unavailable	Reuses filterGearsByAgentCapabilities in InjectIntegrationStatus so the sidebar reflects per-box capability. Mjolnir-as-active now drops HAProxy / Logs / Services / Certificates / Traffic from the nav.

Review status

All 6 PRs have CI green (or in progress on fix-commits). Copilot left 6 review threads across PRs #114, #115, and #116, all addressed and resolved in follow-up commits:

• fix(errors): emit JSON envelope from WriteHTTPError + harden fetch callers #114: WriteHTTPError JSON envelope broke 3 plain-text callers (services service-control, cert download, security block/unblock) → all now JSON-aware via an extractErrorMessage helper; 2xx-non-JSON success responses now throw protocol errors instead of being silently swallowed.
• feat(haproxy): filter dashboard tiles by agent haproxy capability #115: example agent-gear list corrected (no services gear on the agent); cold-cache fetches parallelized so a render path doesn't degrade to O(N × fetchTimeout).
• feat(logs): derive default log sources from agent capabilities #116: caps.Has() vs caps.IsAvailable() distinction added so an older agent that doesn't surface the gear name at all fails open to legacy defaults instead of returning an empty list.

Not done yet

The acceptance criteria from the original issue are partially covered:

• ☑ Capability filtering for the dashboard sidebar, HAProxy tiles, and Logs source picker.
• ☑ JSON error envelope across the board; no more "Unexpected token 'F'".
• ☑ OS Updates honors the active-box cookie.
• ☐ Default landing route per box when the active box has no HAProxy (today still routes to /haproxy with the new empty-state inline).
• ☐ Phase 2 extension — agent gear ProbeResources() so log sources / services / metric sources are advertised by the agent rather than guessed by the dashboard.
• ☐ Phase 3 — container-deployment mount matrix on the agent side. Still owned by gearbox-agent: cert SANs and Docker deployment requirements (umbrella) #106.
• ☐ Phase 4 — fully unifying ?server= / ?box_id= / cookie into a single resolveActiveBox() helper. OS Updates is the first user; remaining handlers still mix the conventions.

Suggested merge order to avoid stacked-PR churn: #113 → #114 → #116 → #115 → #118 → #119. They don't conflict pairwise, but #118 is stacked on #115 and #119 references the same filterGearsByAgentCapabilities helper that #115 touches.