Commit 50fc32b
feat(rotation): keyring phases 2–5 — recovery PR for orphaned stack (#139)
* feat(rotation): Phase 2 install/use/remove endpoints + rotator service (#72)
Builds on Phase 1's keyring plumbing with the active machinery needed
to actually rotate keys: three new agent endpoints implementing the
controller-orchestrated three-phase dance (install → use → remove),
the dashboard-side rotator service that drives it, and the
`X-Gearbox-Kid` request/response header that Phase 5 will use for
drift detection.
Agent endpoints
---------------
`POST /api/v1/system/keyring/install`
- Body `{kid, secret_b64, role?}`. New entries default to `secondary`;
the controller flips primary in a separate `/use` call.
- Idempotent on `(kid, same_secret)` — re-installing returns 200 with
current state. Same kid with a different secret returns 409 so a
divergence between dashboard and agent surfaces loudly.
- Returns 507 when the keyring is at `MaxKeyRingEntries` (4) — guards
against runaway growth from a buggy rotator.
`POST /api/v1/system/keyring/use`
- Body `{kid}`. Flips named entry to primary, demotes the prior
primary to secondary. Both stay accepted; the agent doesn't
distinguish primary vs secondary for inbound auth.
`DELETE /api/v1/system/keyring/{kid}`
- Refuses 409 on the only remaining entry — agent never bricks itself.
All three serialize through a handler-level mutex so concurrent
controller calls can't race their read-modify-write of the on-disk
keyring. The atomic-tmpfile+rename + `atomic.Pointer` swap from Phase
1 means the auth middleware sees the new keyring on the very next
request after the swap, no restart.
Dashboard side
--------------
`agent.Client`
- New `NewClientWithKID(url, key, kid)` constructor + `WithKID(kid)`
setter. Existing `NewClient` callers untouched.
- All outbound requests now go through a `setAuthHeaders(req)`
helper that sets `Authorization: Bearer …` and, when the client
was built with a kid, the `X-Gearbox-Kid` request header. Agent
middleware echoes the matched kid in the response header of the
same name; Phase 5 compares the two to detect drift.
- New `KeyRingGet`, `KeyRingInstall(kid, secret, role)`,
`KeyRingUse(kid)`, `KeyRingDelete(kid)` methods.
`services/agent_keyring`
- New package containing the `Rotator`, which composes
`RotateBox(boxID, overlap)` from the install/use/remove primitives:
decrypt current primary → build authenticated client → install new
key on agent as secondary → persist new key to box_agent_keys →
flip primary on agent → flip primary in DB (which also stamps
`retired_at` on the old entry).
- `CleanupRetiredKeys(boxID, overlap)` removes any entries whose
`retired_at` is older than the overlap window. Uses `time.Since`
for the cutoff comparison so SQLite-driver timezone behaviour
doesn't bite (modernc.org/sqlite scans bare DATETIMEs in local
time; comparing against `time.Now()` in UTC would otherwise mis-
fire). Removes from agent first, then DB; tolerates the agent
having already lost the entry (404) or refusing to remove the
last key (409) since neither leaves us in a bad state.
- `DefaultOverlapWindow = 24h`. Tunable per-call so Phase 4's
scheduler can pick the operator's configured value.
`SetBoxPrimaryKey` now stamps `retired_at` with `time.Now().UTC()`
explicitly rather than letting SQLite emit `CURRENT_TIMESTAMP`, so
the value round-trips correctly through the driver's date parser.
Tests
-----
- 11 keyring-endpoint integration tests (agent side): install adds
secondary, install is idempotent, KID collision with different
secret returns 409, malformed secret returns 400, use flips
primary, use of unknown kid returns 404, delete works, delete of
only-remaining-entry returns 409, mutations persist across
in-process reload + on-disk reload, keyring file mode is 0600,
install over MaxKeyRingEntries returns 507.
- 4 rotator integration tests against a `httptest` mock of the
agent's keyring API (the dashboard module can't import the agent
module): happy path covers full install → use → DB-flip, cleanup
removes retired keys past the overlap window, cleanup leaves keys
within the overlap window alone, missing-box returns error.
- All existing tests in both modules still pass.
Refs: Phase 2 of the implementation plan posted to #72.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(rotation): Phase 3 manual rotate UI + handlers (#72)
Surfaces the Phase 2 rotator behind two operator-visible buttons —
the bit that makes the keyring work actually usable. No new
back-end abstractions; just two thin handlers that compose the
existing rotator with the dashboard's box management.
Backend
-------
`POST /settings/boxes/{id}/rotate-key`
- Rotates one box. Returns 200 with `{success, new_kid, old_kid,
retire_after}` or 4xx/5xx with `{success: false, message}`.
- Constructs a rotator per request from the handler's existing DB +
encryptor — no new singletons.
`POST /settings/boxes/rotate-key-all`
- Iterates every enabled box and rotates each through the same
rotator. Reports per-box success/failure so the operator sees
exactly which boxes need follow-up. Failures on one box don't
halt the run; this matches the homelab use case better than a
strict circuit-breaker — operator decides whether to investigate
one bad box or move on.
Both routes are wired into the admin-only `/settings` group in
cmd/server/main.go alongside the existing box CRUD routes; same
permission gate as `HAProxyBoxUpdatePost` etc.
UI
--
Box edit form (`HAProxyBoxEditPage`)
- New "Rotate API key" section under the API-Key field with a Rotate
Key button. Visible only on edit (server != nil).
- Click → `showConfirmDialog` (warning style) explaining the 24h
overlap → POST → toast on success or alert dialog on failure.
- Reuses the in-page rotate-spinner SVG to surface in-flight state.
Boxes list (`HAProxyBoxesPageContent`)
- New "Rotate All Keys" button next to the existing "Add Box"
button. Visible only when at least one box exists.
- Click → confirm → POST → success toast or alert dialog with a
per-box failure list when partial.
JS uses the established `showConfirmDialog` / `showAlertDialog` /
`showToast` APIs from `layouts.Base`, per the CLAUDE.md "never use
native confirm/alert/prompt" rule.
Tests
-----
No new tests — the rotator's behaviour is already covered by
`services/agent_keyring/rotator_test.go` (Phase 2). The handlers
are thin enough that adding HTTP-level tests would duplicate the
rotator-side coverage. Browser-level testing of the new UI was not
performed in this commit; operator should smoke-test by hitting
both buttons end-to-end before merging.
Refs: Phase 3 of the implementation plan posted to #72.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(rotation): Phase 4 retired-key cleaner (#72)
Adds a background sweeper that walks every enabled box on a tick and
removes any keys whose retired_at + overlap window has passed —
completing the install→use→remove three-phase rotation cycle without
operator intervention.
Originally Phase 4 in the issue plan also covered auto-rotation on a
schedule (off by default, configurable cadence). That needs a new
global-settings surface in the dashboard (no app-level config table
exists today; only user_preferences), which is its own design pass.
Deferring the auto-rotate scheduler to a follow-up so this PR stays
focused on the piece that's actually needed regardless of whether
auto-rotation is enabled.
What's here
-----------
`services/agent_keyring/cleaner.go`
- `RetiredKeyCleaner` runs as a goroutine off the dashboard's
process-lifetime context.
- Hourly tick (`CleanerInterval`) — short enough that a 24h
rotation cleans up within a few hours of its target, long enough
that the sweep is cheap.
- Immediate sweep on start so a freshly-deployed dashboard catches
up on any retired keys left from manual rotations done while the
prior instance was down.
- Per-box failures are logged but don't halt the sweep; one
unreachable agent shouldn't block cleanup on the others.
`cmd/server/main.go`
- Wires the cleaner into startup alongside the existing alert
evaluator. Cancelled when main returns.
Tests
-----
- `TestCleaner_RemovesRetiredKeyOnTick` — rotates a box, then runs
the cleaner with a 1ms overlap and 20ms interval; verifies the
retired key is removed from both the mock agent and the DB.
- `TestCleaner_NoopWhenNothingRetired` — no rotation happens, so
the cleaner finds nothing to do; verifies the seeded entry
survives a sweep.
Refs: Phase 4 (lite) of the implementation plan posted to #72.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(rotation): Phase 5 drift detection in agent.Client (#72) (#132)
When the dashboard signs an outbound request with kid X but the agent
matches kid Y instead (because rotation propagated on one side but
not the other), drift detection logs the disagreement so an operator
can resync. Closes the observability loop on the install→use→remove
cycle: Phase 2 added the request header, the agent's auth middleware
already echoes the matched kid, and this commit wires the
dashboard-side comparison.
What's here
-----------
`agent.HeaderKID = "X-Gearbox-Kid"` is the shared constant the
dashboard sends and the agent echoes back. The agent's middleware
sets it on every authenticated response (see Phase 1).
`agent.Client`
- `SetDriftHandler(DriftHandler)` installs an optional callback
invoked when `resp.Header.Get(HeaderKID)` differs from the kid the
client was built with. Reads c.onDrift at RoundTrip time, so the
handler can be installed AFTER construction (typical for long-
lived per-box clients held by the dashboard).
- Transport wrap: `kidObservingTransport` sits between the http
client and the underlying TLS transport, calling `c.checkDrift` on
every successful response. One central point of inspection — no
invasive edits to every `doRequest*` method.
- `LogDriftHandler(logger, boxID)` builds a ready-to-use DriftHandler
that emits a structured warn-level log. Lowest-friction wiring for
the long-lived clients held in the WebSocketManager.
Test fix
--------
`TestClientTimeout` was a flaky pre-existing test whose substring
check was case-sensitive — Go's net/http error message capitalises
"Timeout" sometimes and emits "context deadline exceeded" other
times. Fixed by lower-casing the error message before substring
matching. Verified stable across 5 runs.
Tests
-----
`client_drift_test.go` exercises the four corners of the matrix:
- Drift handler fires when kid mismatches.
- Doesn't fire when kid matches.
- Doesn't fire when the agent omits the header (older agents).
- Doesn't fire when the dashboard client has no kid.
Not yet wired into production code
----------------------------------
Adding `agent.LogDriftHandler` is the small API surface; deciding
*where* to call SetDriftHandler is a separate design choice
(WebSocketManager? capability poller? every short-lived
handler.agentClient()?). Deferring that integration so this PR
stays focused on the observability primitive itself. A follow-up
can install LogDriftHandler at every site that constructs a kid-
bearing client.
Refs: Phase 5 of the implementation plan posted to #72.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 265a26a commit 50fc32b
16 files changed
Lines changed: 1990 additions & 35 deletions
File tree
- gearbox-agent
- cmd/gearbox-agent
- internal/api
- gearbox
- cmd/server
- internal/framework
- agent
- database
- handler
- services/agent_keyring
- templates/pages
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
542 | 542 | | |
543 | 543 | | |
544 | 544 | | |
545 | | - | |
| 545 | + | |
546 | 546 | | |
547 | 547 | | |
548 | 548 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
| 6 | + | |
5 | 7 | | |
6 | 8 | | |
| 9 | + | |
| 10 | + | |
7 | 11 | | |
8 | 12 | | |
9 | 13 | | |
10 | 14 | | |
11 | 15 | | |
12 | 16 | | |
13 | | - | |
14 | | - | |
15 | | - | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
16 | 21 | | |
17 | 22 | | |
| 23 | + | |
18 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
19 | 31 | | |
20 | 32 | | |
21 | | - | |
22 | | - | |
23 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
24 | 37 | | |
25 | 38 | | |
26 | 39 | | |
27 | 40 | | |
28 | 41 | | |
29 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
30 | 46 | | |
31 | 47 | | |
32 | | - | |
| 48 | + | |
33 | 49 | | |
34 | | - | |
35 | | - | |
| 50 | + | |
| 51 | + | |
36 | 52 | | |
37 | 53 | | |
38 | | - | |
39 | | - | |
| 54 | + | |
| 55 | + | |
40 | 56 | | |
41 | 57 | | |
42 | 58 | | |
| |||
61 | 77 | | |
62 | 78 | | |
63 | 79 | | |
64 | | - | |
65 | | - | |
66 | | - | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
67 | 262 | | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
68 | 327 | | |
0 commit comments