gearbox-agent: cert SANs and Docker deployment requirements (umbrella)

[sarg3nt]

Summary

The gearbox-agent's self-signed cert generation hard-codes its SAN list to localhost, the host's os.Hostname(), 127.0.0.1, and ::1. When the agent is deployed as a Docker container behind a bridge network, the IP that clients (e.g. the gearbox dashboard) use to reach it is not in the SAN list, and TLS verification fails. There's no env knob to add extra SANs.

This is one specific manifestation of a broader gap: the agent's "Docker container deployment" story is under-specified. The systemd unit on a bare host gets root + journalctl + /proc + /sys for free; a container deployment loses all of that unless mounts and capabilities are explicitly granted. We don't have a canonical answer for "what does gearbox-agent need to be mounted into it to monitor a host adequately."

This issue is the umbrella to fix both pieces.

Concrete evidence (the failing deployment)

A real-world deployment is the homelab repo's apps/gearbox-agent/docker-compose.yml. It runs the agent on a static IP 172.16.2.3 on a Docker bridge network. The gearbox dashboard, in another container on the same bridge, reaches it at https://172.16.2.3:8405.

Inspecting the running cert:

Subject:  CN=gearbox-agent
SANs:     DNS:localhost, DNS:545e83ac2ebe, IP:127.0.0.1, IP:::1

(545e83ac2ebe is the Docker container hash, returned by os.Hostname() inside the container.)

The gearbox dashboard logs the resulting failure every poll interval:

failed to collect stats from agent: request failed:
  Get "https://172.16.2.3:8405/api/v1/haproxy/stats?format=csv":
  tls: failed to verify certificate:
  x509: certificate is valid for 127.0.0.1, ::1, not 172.16.2.3

Three workarounds today, all bad:

1. GEARBOX_INSECURE_TLS=true on the dashboard side. Works, but disables verification entirely. (This is what the homelab repo is doing right now as a stopgap.)
2. Bind-mount a pre-generated cert that includes the right SANs. Works, but cert rotation is on the operator.
3. Use a real CA-signed cert via TLS_CUSTOM. Works for hosts with a public name; doesn't help for bridge-internal addressing.

Scope of this issue

Part 1 — cert SANs (agent code change)

Make the cert generation accept additional SAN entries via env var.

• New env var, e.g. GEARBOX_AGENT_TLS_HOSTS=172.16.2.3,gearbox-agent.lan — comma-separated list, IPs and DNS names both supported (existing generateSelfSignedCert already discriminates with net.ParseIP).
• Plumb through cmd/gearbox-agent/main.go:223 — append to the existing hosts slice.
• When GEARBOX_AGENT_TLS_HOSTS changes (or the existing cert's SAN list doesn't cover the configured hosts), regenerate. Today the agent reuses any non-expired cert.
• Document the env var alongside TLS_CUSTOM and the existing self-signed behavior.

Part 2 — Docker deployment mount requirements (research + docs + an example compose)

Decide and document what the agent needs from the host to do its general OS monitoring job: CPU, memory, disk usage, network interfaces, process counts, system logs, container/service health. Specifically not in scope here: ZFS / TrueNAS / SMART (future). The bar is "what does it take to monitor a generic Linux box from a container."

Open research questions:

• /proc and /sys — does the agent already read from these, or does it use a Go library that needs them mounted? What's the minimum subset?
• journalctl — when the agent runs in a container, the host's journal is invisible. Options: bind-mount /var/log/journal read-only (works for persistent journals only), bind-mount /run/log/journal (for volatile), or move to log-file tailing where possible. What does the existing log streamer assume?
• /var/log — what log files does the agent need read access to in a containerized deployment? HAProxy access log is the obvious one; what else?
• Docker socket — if we want the agent to surface container-error signals (restart loops, exited containers), it needs /var/run/docker.sock. Is that in scope for this issue or a later one?
• Capabilities — does the agent need CAP_DAC_READ_SEARCH to read root-owned logs without running as root in the container? What's the user/uid story (today the homelab compose uses TrueNAS's apps user 568:568)?
• Host networking vs. bridge — network_mode: host sidesteps the cert-SAN problem entirely and gives the agent native access to host network stats. What are the trade-offs? When should each mode be recommended?
• Read-only host mounts — confirm the agent doesn't write anywhere outside its data dir, so all host mounts can be :ro.

Deliverables:

1. A canonical reference docker-compose.yml in the gearbox-agent repo (or alongside it) for "deploy gearbox-agent on a Linux host with sane defaults for OS monitoring."
2. A docs page explaining each mount, why it's needed, what breaks if it's missing.
3. Update the homelab repo's compose file to match the new reference — that deployment is the canary.

Part 3 — comparison with the systemd deployment (so we know the gap)

The light-hugger deployment runs the agent as a systemd unit (/etc/systemd/system/gearbox-agent.service) as root with ProtectHome=read-only, full filesystem read, ReadWritePaths=/root/.local /root/.cache. It tails (via spawned journalctl -f) the journal for fail2ban, rsyslog, sshd, kernel, and haproxy. A container deployment loses all of those unless we plumb equivalents.

For each signal the systemd unit currently surfaces, this issue should record:

• What it is.
• How the systemd unit accesses it.
• What the container deployment needs to access the same thing — mount? capability? alternative data path?
• Whether it's worth supporting in container mode, or whether some signals are systemd-only by design.

Out of scope

• ZFS / SMART / IPMI / TrueNAS middleware — these are future modules and have their own access stories.
• Re-architecting the agent's plugin/gear model.
• Per-source insight modules (HAProxy / nginx / etc.) — those belong in the layered Error Insights design issue.

Acceptance

• GEARBOX_AGENT_TLS_HOSTS (name TBD) lands and the homelab bridge deployment passes TLS verification without GEARBOX_INSECURE_TLS=true.
• A reference compose file + docs exist for Docker deployments.
• The homelab apps/gearbox-agent/docker-compose.yml is updated to match the reference and is the canary.
• For every general OS-monitoring signal the systemd deployment provides, the docs answer "and here's how you get it in a container deployment, or here's why we don't."