Test Connection probe 404s on boxes without HAProxy; audit all probes for server-side API key handling

[sarg3nt]

Summary

The Test Connection button on the Box Add/Edit form (/settings/boxes) fails with API request failed: 404 page not found on any agent box that doesn't run HAProxy locally (e.g. the homelab gearbox-agent box, which exposes host / metrics / access-log gears but has no HAProxy stats socket).

Root cause

HAProxyBoxTestConnectionPost (internal/framework/handler/haproxy_config.go:466-485) does two probes:

1. client.Health() → GET /health (unauthenticated; succeeds on every agent)
2. client.GetInfo() → GET /api/v1/haproxy/info (authenticated; assumes every box runs HAProxy)

Since the multi-source-detection work (#100, #102), an agent box can legitimately be an nginx / Apache / Caddy / Traefik box — or a pure metrics/host box like the homelab gearbox-agent — with no HAProxy gear enabled at all. In that case /api/v1/haproxy/info returns 404 and Test Connection reports failure even though the agent is healthy and authentication is working.

Repro (from the homelab gearbox-agent container, gear probe summary)

GEAR     STATUS    REASON
haproxy  disabled  stats socket configured at /run/haproxy/admin.sock but does not exist
host     enabled
metrics  enabled

Resulting requests when Test Connection is clicked:

GET /health                  status=200
GET /api/v1/haproxy/info     status=404   ← causes "API request failed: 404 page not found"

/api/v1/metadata is not a usable fallback — it's also HAProxy-sync-specific and returns 503 when the GitOps sync service isn't configured (internal/api/handlers.go:74-78).

Proposed fix

1. Switch the auth probe to a haproxy-agnostic endpoint

Replace client.GetInfo() with a call to /api/v1/system/capabilities (gearbox-agent/internal/framework/gear/manager.go:757). It's authenticated, present on every agent regardless of which gears are enabled, and returns the exact "what can this box do" info Test Connection arguably ought to be showing the user as part of a successful test ("Connected — agent advertises: host, metrics, access-log").

Requires adding a Capabilities() method on internal/framework/agent/client.go alongside GetInfo().

2. Investigate: Test Connection appears to keep failing after a box is added

When the box is added via "Add Box" and then re-opened via the edit form, clicking Test Connection again continues to fail. Likely the same root cause as above (same 404 endpoint), but worth confirming there isn't a second issue lurking — e.g. the edit path loads the encrypted API key from DB (haproxy_config.go:427-450) and if that load silently produces a bad key, the second-probe failure could mask it. Verify by:

• Confirming a fixed probe (issue chore(deps)(deps): bump github.com/go-chi/chi/v5 from 5.2.3 to 5.2.4 in /gearbox in the go-dependencies group #1 above) passes on the same edit form for the same box that just failed.
• Logging whether the edit-mode DB-loaded key roundtrips correctly through encryptor.Decrypt.

3. Audit: all agent probes/calls must run server-side; API key must never reach the browser

The general rule we want enforced across the dashboard:

• The encrypted API key for each box lives in the dashboard DB. Decryption happens on the server.
• Any call to an agent endpoint — Test Connection, capabilities polling, metrics fetch, log tail, anything — is made by the gearbox backend to the agent, passing the API key as a server-set header.
• The browser should never see, fetch, or transmit a stored API key. The only browser-side exception is the Generate API Key modal flow where the user is choosing a new key — that key is in the form field on purpose so the user can copy it and so the form submit carries it to the backend for storage.

What to verify:

• No JS in the dashboard fetches an agent endpoint directly (e.g. nothing like fetch('https://box.example.com:8405/api/v1/metrics', {headers: {'X-API-Key': ...}})).
• The edit form template (haproxy_settings.templ) does not render an existing API key into any value= attribute, hidden field, or data-* attribute. (Current code uses a placeholder •••••••••••••••• with no value, which is correct — this audit is to make sure no future change regresses it.)
• All agent traffic flows: browser → gearbox handler → agent.Client → agent. The browser only ever talks to gearbox.

If any direct browser→agent call exists, move it behind a gearbox handler that proxies the request with the server-loaded API key.

Acceptance criteria

• Test Connection passes for a box whose agent has no HAProxy gear (e.g. homelab gearbox-agent).
• Test Connection succeeds on repeated probes of an already-added box.
• Audit confirms no agent API key is ever transmitted to or from the browser outside the Generate-API-Key flow, and the audit result is documented in this issue or a follow-up.