|
| 1 | +# Gear Probes — capability-driven gear loading |
| 2 | + |
| 3 | +**Version:** 0.1.0 | **Last updated:** 2026-05-13 |
| 4 | + |
| 5 | +The gearbox-agent runs on hosts with wildly different software stacks: an HAProxy gateway, a TrueNAS storage box, a Docker host, a workstation. The agent ships every gear it knows about, but a given host can only usefully run a subset of them. The **probe phase** is how the agent figures out, at startup, which gears apply on *this* box. |
| 6 | + |
| 7 | +This document explains: |
| 8 | + |
| 9 | +- What the probe phase does and what it does not do |
| 10 | +- The four-state status enum |
| 11 | +- Per-gear probe contracts |
| 12 | +- The startup summary table |
| 13 | +- How to add `Probe()` to a new gear |
| 14 | + |
| 15 | +For background and the broader capability-driven loading design, see issue [#60](https://github.com/sarg3nt/gearbox/issues/60). |
| 16 | + |
| 17 | +## Lifecycle |
| 18 | + |
| 19 | +The agent's gear lifecycle now has four phases, in this order: |
| 20 | + |
| 21 | +```text |
| 22 | +Probe → Initialize → Start → (later) Stop |
| 23 | +``` |
| 24 | + |
| 25 | +1. **Probe** — each gear is asked `Probe(ctx, deps) ProbeResult`. The result is recorded in the manager and rendered to the journal as a summary table. |
| 26 | +2. **Initialize** — only gears whose probe returned `available` go through `Initialize`. Their `Collector`, `Streamer`, runner, and event-handler objects are constructed here. |
| 27 | +3. **Start** — the same loaded gears have `Start(ctx)` called; background goroutines (log streamers, periodic tickers, etc.) launch. |
| 28 | +4. **Stop** — on shutdown, only gears that were started get `Stop(ctx)`. |
| 29 | + |
| 30 | +Gears that probe non-`available` are skipped at every subsequent stage: |
| 31 | + |
| 32 | +- No `Initialize` (no state objects allocated) |
| 33 | +- No `Start` (no goroutines, no exec'd subprocesses, no periodic collection) |
| 34 | +- No `RegisterRoutes` (their HTTP endpoints simply do not exist on this host) |
| 35 | +- No collector/streamer registration |
| 36 | +- No event-handler registration |
| 37 | + |
| 38 | +### What is *not* saved |
| 39 | + |
| 40 | +Be honest about the boundaries: |
| 41 | + |
| 42 | +- **Binary size is unchanged.** Every gear is blank-imported in [`cmd/gearbox-agent/main.go`](../cmd/gearbox-agent/main.go), so all gear code is linked into the binary regardless of probe verdict. Go does not support module unloading. |
| 43 | +- **Gear `init()` functions still run.** That is how each gear registers itself with `gear.Register(...)`. The empty `&Gear{}` instance is held in the global registry forever. |
| 44 | +- **Probe itself runs on every gear.** That is the point — we need the verdict. |
| 45 | + |
| 46 | +### What is saved |
| 47 | + |
| 48 | +The dominant wins are **CPU/IO and runtime RAM**, not code size: |
| 49 | + |
| 50 | +| Skipped step | Cost avoided | |
| 51 | +|---------------------------------------|-----------------------------------------------------------------------------------------| |
| 52 | +| Construct collectors / streamers | Heap allocations for per-source `tail`/`journalctl` pipelines, parser state, etc. | |
| 53 | +| Start background goroutines | Goroutine stacks + scheduler load; with log streaming this is the largest single win | |
| 54 | +| Spawn `tail` / `journalctl` children | Subprocess RSS + parent-side read pumps; was the source of the original log-spam loops | |
| 55 | +| Periodic collection tickers | Continuous CPU every N seconds, often per-source | |
| 56 | +| Chi route entries + event subscribers | Small but real — keeps the router and event bus pointed at gears that actually answer | |
| 57 | + |
| 58 | +For a TrueNAS host that genuinely has nothing this agent cares about (no HAProxy, no certbot, no fail2ban, no apt), the probe phase reduces the runtime from "five permanently-degraded gears noisily failing every two seconds" to "everything skipped, agent sits idle." That is the regression the probe phase exists to fix. |
| 59 | + |
| 60 | +## Status enum |
| 61 | + |
| 62 | +Each gear reports one of four statuses via `ProbeResult.Status`. The statuses are not interchangeable: each implies a different operator fix, and conflating them is what historically cost debugging time. |
| 63 | + |
| 64 | +| Status | Meaning | Operator action | |
| 65 | +|-----------------|---------------------------------------------------------------|--------------------------------------------------------------------------| |
| 66 | +| `available` | Prereqs present and reachable. Gear loads normally. | None. | |
| 67 | +| `not_installed` | The thing the gear manages isn't on this host at all. | Expected on hosts that don't run this software. | |
| 68 | +| `inaccessible` | Prereqs exist, but the agent can't reach them. | Fix the access — add bind mount, adjust permissions, set correct path. | |
| 69 | +| `disabled` | Forced off by configuration. | Change the config to re-enable. (Reserved; no gear returns this today.) | |
| 70 | + |
| 71 | +The distinction between `not_installed` and `inaccessible` matters most in container mode. An agent running in a Docker container on an HAProxy host that wasn't given the right bind mounts should report `inaccessible` for the haproxy gear — *not* `not_installed`, because that would send the operator to install HAProxy on a box that already has it. |
| 72 | + |
| 73 | +`ProbeResult.Reason` is a human-readable sentence that names what surface was probed and what was wrong. Examples: |
| 74 | + |
| 75 | +- `"stats socket configured at /run/haproxy/admin.sock but does not exist (HAProxy not running, or bind mount missing in container mode)"` |
| 76 | +- `"neither fail2ban-client nor nft found on PATH"` |
| 77 | +- `"no haproxy binary on PATH; no stats socket or URL configured"` |
| 78 | + |
| 79 | +Bad reasons (do not ship these): |
| 80 | + |
| 81 | +- `"prereqs not met"` |
| 82 | +- `"haproxy unavailable"` |
| 83 | +- `"see logs"` |
| 84 | + |
| 85 | +## Per-gear probe contracts |
| 86 | + |
| 87 | +Each gear's `Probe()` determines `available` from a clearly-named host surface. The table below is the current contract; expect this to evolve as #60 lands its container-mode probing work. |
| 88 | + |
| 89 | +| Gear | `available` when | Capabilities reported | |
| 90 | +|----------------|-------------------------------------------------------------------------------|--------------------------------------------------| |
| 91 | +| `certificates` | `certbot` on PATH or in common install paths, **or** `acme.sh` home present | `manager`, `path` / `home` | |
| 92 | +| `haproxy` | Stats URL set, **or** stats socket file exists, **or** `haproxy` on PATH | `stats_url` / `stats_socket` | |
| 93 | +| `logs` | `journalctl` **or** `tail` on PATH | `journalctl`, `tail` | |
| 94 | +| `metrics` | `/proc/stat` readable | — | |
| 95 | +| `security` | `fail2ban-client` **or** `nft` on PATH | `fail2ban`, `nftables` | |
| 96 | +| `traffic` | Same as `haproxy` (stick tables share the stats socket) | `stats_url` / `stats_socket` | |
| 97 | +| `updates` | One of `apt-get` / `apt` / `dnf` / `yum` / `zypper` / `apk` on PATH | `package_manager`, `path` | |
| 98 | + |
| 99 | +The `haproxy` and `traffic` probes deliberately distinguish three states: |
| 100 | + |
| 101 | +- **`available`** — stats URL configured, or stats socket file actually exists. |
| 102 | +- **`inaccessible`** — socket path is configured but missing on disk; or haproxy binary is present but neither stats source is configured. The fix differs from "install HAProxy." |
| 103 | +- **`not_installed`** — no haproxy binary, no socket, no URL. The host genuinely doesn't run HAProxy. |
| 104 | + |
| 105 | +## Startup summary table |
| 106 | + |
| 107 | +After all probes run, the manager writes a single human-readable table to stderr (which lands in the systemd journal on systemd hosts, and in `docker logs` for container deployments): |
| 108 | + |
| 109 | +```text |
| 110 | +Gear probe summary: |
| 111 | +
|
| 112 | + GEAR STATUS REASON |
| 113 | + certificates enabled |
| 114 | + haproxy enabled |
| 115 | + logs enabled |
| 116 | + metrics enabled |
| 117 | + security disabled neither fail2ban-client nor nft found on PATH |
| 118 | + traffic enabled |
| 119 | + updates enabled |
| 120 | +``` |
| 121 | + |
| 122 | +Conventions: |
| 123 | + |
| 124 | +- Status column shows `enabled` (probe returned `available`) or `disabled` (any other status). The distinct `not_installed` / `inaccessible` / `disabled` reasons live in the reason column for `disabled` rows. |
| 125 | +- Reason column is **blank** for enabled rows — clutter-suppressed, since the detected version/path is already in the structured slog stream via `gear probe complete`. |
| 126 | +- Columns are auto-aligned to the widest cell; the table works for one gear or twenty. |
| 127 | + |
| 128 | +In addition to the table, the probe phase emits structured slog lines that operators or log shippers can parse: |
| 129 | + |
| 130 | +```text |
| 131 | +INFO probing host for gear capabilities registered_gears=7 |
| 132 | +INFO gear probe complete registered=7 available=6 unavailable=1 |
| 133 | +``` |
| 134 | + |
| 135 | +The dashboard (gearbox web UI) consumes this information via the upcoming `/api/v1/system/capabilities` endpoint — tracked in [#60](https://github.com/sarg3nt/gearbox/issues/60) §6. |
| 136 | + |
| 137 | +## Adding `Probe()` to a new gear |
| 138 | + |
| 139 | +`ProbeableGear` is a **sub-interface** of `Gear`. Implementing it is optional: gears that don't are treated as always-available, which preserves the pre-probe-phase behaviour. Concretely: |
| 140 | + |
| 141 | +```go |
| 142 | +package mygear |
| 143 | + |
| 144 | +import ( |
| 145 | + "context" |
| 146 | + "os/exec" |
| 147 | + |
| 148 | + "github.com/sarg3nt/gearbox-agent/internal/framework/gear" |
| 149 | +) |
| 150 | + |
| 151 | +func (g *Gear) Probe(ctx context.Context, deps gear.Dependencies) gear.ProbeResult { |
| 152 | + // Detect prereqs side-effect-free. No connections, no state mutation, |
| 153 | + // no loud logging — the manager logs a single summary line per gear. |
| 154 | + path, err := exec.LookPath("the-thing-i-need") |
| 155 | + if err != nil { |
| 156 | + return gear.ProbeNotInstalled( |
| 157 | + "the-thing-i-need binary not found on PATH", |
| 158 | + ) |
| 159 | + } |
| 160 | + return gear.ProbeAvailable( |
| 161 | + "the-thing-i-need found", |
| 162 | + map[string]string{"path": path}, |
| 163 | + ) |
| 164 | +} |
| 165 | +``` |
| 166 | + |
| 167 | +Rules for a good `Probe()`: |
| 168 | + |
| 169 | +1. **Be fast.** Probe runs synchronously before Initialize. Don't open network connections, don't shell out to slow commands (`find /` etc.), don't read multi-megabyte config files. |
| 170 | +2. **Be side-effect-free.** No state mutation on the gear receiver. No log spam. The manager logs once per gear; if your probe is chatty, raise it to `Debug` or remove it. |
| 171 | +3. **Use the helpers.** `ProbeAvailable`, `ProbeNotInstalled`, `ProbeInaccessible`, `ProbeDisabled` exist so reviewers can see the verdict at a glance. |
| 172 | +4. **Reason like a sentence.** "What did I probe, what did I expect, what did I get?" An operator should be able to act on the reason without reading source. |
| 173 | +5. **Distinguish missing-thing from missing-access.** If the configured path doesn't exist *as a file*, that may be `not_installed` (host doesn't run it) or `inaccessible` (bind mount missing). Check the parent directory: if the *parent* doesn't exist, mount is missing → `inaccessible`. If parent exists but the file doesn't, host genuinely lacks it → `not_installed`. |
| 174 | + |
| 175 | +## Testing |
| 176 | + |
| 177 | +Manager-level tests live in [`internal/framework/gear/manager_probe_test.go`](../internal/framework/gear/manager_probe_test.go). They use a `withTestRegistry` helper that swaps the global registry's plugin map for an isolated one, so each test controls exactly which gears are registered. |
| 178 | + |
| 179 | +A typical unit test for per-gear `Probe()` should: |
| 180 | + |
| 181 | +- Stub out filesystem / `exec.LookPath` lookups (use `t.TempDir()` and prepend it to `$PATH` via `t.Setenv("PATH", ...)`). |
| 182 | +- Assert both the `Status` and the substring of `Reason` an operator would search for. |
| 183 | + |
| 184 | +## Related |
| 185 | + |
| 186 | +- [#60 — capability-driven gear loading + containerizable agent](https://github.com/sarg3nt/gearbox/issues/60) — the broader design; §6 covers the future `/api/v1/system/capabilities` endpoint and the container-mode probing rules. |
| 187 | +- [docs/docker.md](docker.md) — current container deployment, including the bind mounts each probe expects to see in container mode. |
| 188 | +- [Top-level gear architecture (docs/gears.md)](../../docs/gears.md) — gear system overview shared by gearbox and gearbox-agent. |
0 commit comments