feat(rotation): Phase 2 install/use/remove endpoints + rotator service (#72)

Builds on Phase 1's keyring plumbing with the active machinery needed
to actually rotate keys: three new agent endpoints implementing the
controller-orchestrated three-phase dance (install → use → remove),
the dashboard-side rotator service that drives it, and the
`X-Gearbox-Kid` request/response header that Phase 5 will use for
drift detection.

Agent endpoints
---------------

`POST /api/v1/system/keyring/install`

- Body `{kid, secret_b64, role?}`. New entries default to `secondary`;
  the controller flips primary in a separate `/use` call.
- Idempotent on `(kid, same_secret)` — re-installing returns 200 with
  current state. Same kid with a different secret returns 409 so a
  divergence between dashboard and agent surfaces loudly.
- Returns 507 when the keyring is at `MaxKeyRingEntries` (4) — guards
  against runaway growth from a buggy rotator.

`POST /api/v1/system/keyring/use`

- Body `{kid}`. Flips named entry to primary, demotes the prior
  primary to secondary. Both stay accepted; the agent doesn't
  distinguish primary vs secondary for inbound auth.

`DELETE /api/v1/system/keyring/{kid}`

- Refuses 409 on the only remaining entry — agent never bricks itself.

All three serialize through a handler-level mutex so concurrent
controller calls can't race their read-modify-write of the on-disk
keyring. The atomic-tmpfile+rename + `atomic.Pointer` swap from Phase
1 means the auth middleware sees the new keyring on the very next
request after the swap, no restart.

Dashboard side
--------------

`agent.Client`

- New `NewClientWithKID(url, key, kid)` constructor + `WithKID(kid)`
  setter. Existing `NewClient` callers untouched.
- All outbound requests now go through a `setAuthHeaders(req)`
  helper that sets `Authorization: Bearer …` and, when the client
  was built with a kid, the `X-Gearbox-Kid` request header. Agent
  middleware echoes the matched kid in the response header of the
  same name; Phase 5 compares the two to detect drift.
- New `KeyRingGet`, `KeyRingInstall(kid, secret, role)`,
  `KeyRingUse(kid)`, `KeyRingDelete(kid)` methods.

`services/agent_keyring`

- New package containing the `Rotator`, which composes
  `RotateBox(boxID, overlap)` from the install/use/remove primitives:
  decrypt current primary → build authenticated client → install new
  key on agent as secondary → persist new key to box_agent_keys →
  flip primary on agent → flip primary in DB (which also stamps
  `retired_at` on the old entry).
- `CleanupRetiredKeys(boxID, overlap)` removes any entries whose
  `retired_at` is older than the overlap window. Uses `time.Since`
  for the cutoff comparison so SQLite-driver timezone behaviour
  doesn't bite (modernc.org/sqlite scans bare DATETIMEs in local
  time; comparing against `time.Now()` in UTC would otherwise mis-
  fire). Removes from agent first, then DB; tolerates the agent
  having already lost the entry (404) or refusing to remove the
  last key (409) since neither leaves us in a bad state.
- `DefaultOverlapWindow = 24h`. Tunable per-call so Phase 4's
  scheduler can pick the operator's configured value.

`SetBoxPrimaryKey` now stamps `retired_at` with `time.Now().UTC()`
explicitly rather than letting SQLite emit `CURRENT_TIMESTAMP`, so
the value round-trips correctly through the driver's date parser.

Tests
-----

- 11 keyring-endpoint integration tests (agent side): install adds
  secondary, install is idempotent, KID collision with different
  secret returns 409, malformed secret returns 400, use flips
  primary, use of unknown kid returns 404, delete works, delete of
  only-remaining-entry returns 409, mutations persist across
  in-process reload + on-disk reload, keyring file mode is 0600,
  install over MaxKeyRingEntries returns 507.
- 4 rotator integration tests against a `httptest` mock of the
  agent's keyring API (the dashboard module can't import the agent
  module): happy path covers full install → use → DB-flip, cleanup
  removes retired keys past the overlap window, cleanup leaves keys
  within the overlap window alone, missing-box returns error.
- All existing tests in both modules still pass.

Refs: Phase 2 of the implementation plan posted to #72.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: sarg3nt

gearbox-agent/cmd/gearbox-agent/main.go

@@ -542,7 +542,7 @@ func main() {
 	// backoff layered on top — see 2026-05 audit P1-7).
 	rateLimiter := middleware.DefaultRateLimiter(logger)
 	authBackoff := middleware.DefaultBackoffTracker(logger)
-	keyRingHandler := api.NewKeyRingHandler(server.KeyRing(), logger)
+	keyRingHandler := api.NewKeyRingHandler(server.KeyRing(), cfg.KeyRingPath, logger)
 	pluginRouter := server.Router().Group(func(r chi.Router) {
 		r.Use(middleware.RateLimitMiddleware(rateLimiter))
 		r.Use(middleware.APIKeyAuth(server.KeyRing(), logger, authBackoff))

gearbox-agent/internal/api/keyring.go

@@ -1,42 +1,58 @@
 package api
 
 import (
+	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"log/slog"
 	"net/http"
+	"strings"
+	"sync"
 
 	"github.com/go-chi/chi/v5"
 
 	"github.com/sarg3nt/gearbox-agent/internal/framework/crypto"
 )
 
-// KeyRingHandler exposes the agent's keyring metadata over the HTTP API.
-// Phase 1 ships only the read endpoint; Phase 2 adds install/use/remove
-// for the controller-driven three-phase rotation flow (issue #72).
+// KeyRingHandler exposes the agent's keyring metadata and rotation
+// endpoints over the HTTP API. The mutation endpoints (install/use/
+// remove) drive the controller-orchestrated three-phase rotation from
+// issue #72; the metadata endpoint is read-only.
 type KeyRingHandler struct {
 	keyring *crypto.KeyRingPointer
+	path    string
 	logger  *slog.Logger
+
+	// mu serializes mutation handlers — keyring writes are tmpfile+
+	// rename atomic on disk, but two concurrent installs racing would
+	// still let the second one's read-modify-write step lose the
+	// first one's changes.
+	mu sync.Mutex
 }
 
-// NewKeyRingHandler wires a handler around the agent's keyring pointer.
-func NewKeyRingHandler(keyring *crypto.KeyRingPointer, logger *slog.Logger) *KeyRingHandler {
-	return &KeyRingHandler{keyring: keyring, logger: logger}
+// NewKeyRingHandler wires a handler around the agent's keyring pointer
+// and the disk path the keyring is persisted to.
+func NewKeyRingHandler(keyring *crypto.KeyRingPointer, path string, logger *slog.Logger) *KeyRingHandler {
+	return &KeyRingHandler{keyring: keyring, path: path, logger: logger}
 }
 
 // RegisterRoutes mounts the keyring endpoints on r. The caller is
 // responsible for putting r behind the API-key auth middleware.
 func (h *KeyRingHandler) RegisterRoutes(r chi.Router) {
 	r.Get("/api/v1/system/keyring", h.handleGet)
+	r.Post("/api/v1/system/keyring/install", h.handleInstall)
+	r.Post("/api/v1/system/keyring/use", h.handleUse)
+	r.Delete("/api/v1/system/keyring/{kid}", h.handleRemove)
 }
 
-// keyRingResponse is what the API returns. Never includes secret bytes.
+// keyRingResponse is what GET returns. Never includes secret bytes.
 type keyRingResponse struct {
-	Version int                       `json:"version"`
-	Entries []crypto.KeyRingMetadata  `json:"entries"`
+	Version int                      `json:"version"`
+	Entries []crypto.KeyRingMetadata `json:"entries"`
 }
 
-// handleGet returns the keyring metadata — kids, roles, creation times,
-// and short fingerprints — but never the actual secrets.
+// handleGet returns keyring metadata — kids, roles, creation times,
+// fingerprints. Secrets are never exposed.
 //
 //	@Summary		Get agent keyring metadata
 //	@Description	Returns the currently-installed API keys' metadata. Secrets are NEVER exposed. The fingerprint is the first 8 hex chars of sha256(secret); useful to verify the dashboard has the same secret without round-tripping the secret itself.
@@ -61,8 +77,251 @@ func (h *KeyRingHandler) handleGet(w http.ResponseWriter, _ *http.Request) {
 		Version: kr.Version,
 		Entries: kr.Snapshot(),
 	}
-	w.Header().Set("Content-Type", "application/json")
-	if err := json.NewEncoder(w).Encode(resp); err != nil {
-		h.logger.Error("encode keyring response failed", "error", err)
+	writeJSON(w, http.StatusOK, resp)
+}
+
+// installRequest is the install endpoint's body. Secret is base64url-
+// encoded raw bytes (the same encoding used in the on-wire token).
+type installRequest struct {
+	KID      string `json:"kid"`
+	SecretB64 string `json:"secret_b64"`
+	Role     string `json:"role"` // optional; defaults to "secondary"
+}
+
+// handleInstall adds a new entry to the keyring. Idempotent: re-
+// installing the same (kid, secret) pair is a no-op and returns 200.
+// Installing a different secret under an existing kid returns 409.
+//
+//	@Summary		Install a new keyring entry
+//	@Description	Adds a new accepted API key to the agent's keyring. New entries default to role=secondary; the controller calls /use to flip primary after confirming installation.
+//	@Tags			system
+//	@Security		BearerAuth
+//	@Accept			json
+//	@Produce		json
+//	@Param			body	body		installRequest	true	"new key"
+//	@Success		200		{object}	keyRingResponse
+//	@Failure		400		{string}	string	"Bad Request"
+//	@Failure		401		{string}	string	"Unauthorized"
+//	@Failure		409		{string}	string	"Conflict"
+//	@Failure		507		{string}	string	"Keyring full"
+//	@Router			/api/v1/system/keyring/install [post]
+func (h *KeyRingHandler) handleInstall(w http.ResponseWriter, r *http.Request) {
+	var req installRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "invalid JSON body")
+		return
+	}
+	req.KID = strings.TrimSpace(req.KID)
+	if req.KID == "" || req.SecretB64 == "" {
+		writeError(w, http.StatusBadRequest, "kid and secret_b64 are required")
+		return
+	}
+	secret, err := base64.RawURLEncoding.DecodeString(req.SecretB64)
+	if err != nil || len(secret) != crypto.SecretLength {
+		writeError(w, http.StatusBadRequest, "secret_b64 must be base64url-encoded 32 random bytes")
+		return
+	}
+	role := req.Role
+	if role == "" {
+		role = "secondary"
+	}
+	if role != "primary" && role != "secondary" {
+		writeError(w, http.StatusBadRequest, "role must be 'primary' or 'secondary'")
+		return
+	}
+
+	h.mu.Lock()
+	defer h.mu.Unlock()
+
+	current := h.keyring.Load()
+	next := current.Clone()
+
+	// Idempotency: same (kid, secret) → 200 with current state. Same
+	// kid + different secret → 409. The controller treats both as
+	// "already done" but the 409 surfaces a state divergence to logs.
+	if existing := findEntry(current, req.KID); existing != nil {
+		if constantTimeEq(existing.Secret, secret) {
+			writeJSON(w, http.StatusOK, keyRingResponse{
+				Version: current.Version,
+				Entries: current.Snapshot(),
+			})
+			return
+		}
+		writeError(w, http.StatusConflict, "kid already exists with a different secret")
+		return
+	}
+
+	entry := crypto.KeyRingEntry{
+		KID:    req.KID,
+		Secret: secret,
+		Role:   role,
+	}
+	if err := next.Add(entry); err != nil {
+		if errors.Is(err, crypto.ErrKeyRingFull) {
+			writeError(w, http.StatusInsufficientStorage, "keyring at maximum capacity")
+			return
+		}
+		writeError(w, http.StatusBadRequest, "add entry: "+err.Error())
+		return
+	}
+
+	if err := crypto.SaveKeyRing(h.path, next); err != nil {
+		h.logger.Error("save keyring", "error", err)
+		writeError(w, http.StatusInternalServerError, "failed to persist keyring")
+		return
+	}
+	h.keyring.Store(next)
+	h.logger.Info("keyring: installed entry", "kid", req.KID, "role", role)
+
+	writeJSON(w, http.StatusOK, keyRingResponse{
+		Version: next.Version,
+		Entries: next.Snapshot(),
+	})
+}
+
+// useRequest is the use endpoint's body.
+type useRequest struct {
+	KID string `json:"kid"`
+}
+
+// handleUse flips the named entry to role=primary and demotes the
+// existing primary to secondary. Both stay accepted; the agent doesn't
+// distinguish primary vs secondary for inbound auth purposes, but the
+// /keyring metadata response and the kid echoed in `X-Gearbox-Kid` let
+// the controller drive the overlap window correctly.
+//
+//	@Summary		Promote a keyring entry to primary
+//	@Description	Flips the named keyring entry's role to "primary" and demotes the prior primary to "secondary". Both keys remain valid for inbound auth.
+//	@Tags			system
+//	@Security		BearerAuth
+//	@Accept			json
+//	@Produce		json
+//	@Param			body	body		useRequest	true	"target kid"
+//	@Success		200		{object}	keyRingResponse
+//	@Failure		400		{string}	string	"Bad Request"
+//	@Failure		401		{string}	string	"Unauthorized"
+//	@Failure		404		{string}	string	"Unknown kid"
+//	@Router			/api/v1/system/keyring/use [post]
+func (h *KeyRingHandler) handleUse(w http.ResponseWriter, r *http.Request) {
+	var req useRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "invalid JSON body")
+		return
+	}
+	req.KID = strings.TrimSpace(req.KID)
+	if req.KID == "" {
+		writeError(w, http.StatusBadRequest, "kid is required")
+		return
+	}
+
+	h.mu.Lock()
+	defer h.mu.Unlock()
+
+	next := h.keyring.Load().Clone()
+	if err := next.SetPrimary(req.KID); err != nil {
+		if errors.Is(err, crypto.ErrUnknownKID) {
+			writeError(w, http.StatusNotFound, "unknown kid")
+			return
+		}
+		writeError(w, http.StatusInternalServerError, "set primary: "+err.Error())
+		return
+	}
+
+	if err := crypto.SaveKeyRing(h.path, next); err != nil {
+		h.logger.Error("save keyring", "error", err)
+		writeError(w, http.StatusInternalServerError, "failed to persist keyring")
+		return
+	}
+	h.keyring.Store(next)
+	h.logger.Info("keyring: promoted to primary", "kid", req.KID)
+
+	writeJSON(w, http.StatusOK, keyRingResponse{
+		Version: next.Version,
+		Entries: next.Snapshot(),
+	})
+}
+
+// handleRemove deletes the named entry. Refuses to remove the only
+// remaining entry — an operator error otherwise bricks the box.
+//
+//	@Summary		Remove a keyring entry
+//	@Description	Deletes the named keyring entry. Refuses to remove the only remaining entry; the agent always retains at least one accepted key.
+//	@Tags			system
+//	@Security		BearerAuth
+//	@Param			kid	path	string	true	"key id"
+//	@Success		200	{object}	keyRingResponse
+//	@Failure		401	{string}	string	"Unauthorized"
+//	@Failure		404	{string}	string	"Unknown kid"
+//	@Failure		409	{string}	string	"Cannot remove last key"
+//	@Router			/api/v1/system/keyring/{kid} [delete]
+func (h *KeyRingHandler) handleRemove(w http.ResponseWriter, r *http.Request) {
+	kid := strings.TrimSpace(chi.URLParam(r, "kid"))
+	if kid == "" {
+		writeError(w, http.StatusBadRequest, "kid is required")
+		return
 	}
+
+	h.mu.Lock()
+	defer h.mu.Unlock()
+
+	next := h.keyring.Load().Clone()
+	if err := next.Remove(kid); err != nil {
+		switch {
+		case errors.Is(err, crypto.ErrUnknownKID):
+			writeError(w, http.StatusNotFound, "unknown kid")
+		case errors.Is(err, crypto.ErrCannotRemoveLast):
+			writeError(w, http.StatusConflict, "cannot remove the only remaining key")
+		default:
+			writeError(w, http.StatusInternalServerError, "remove: "+err.Error())
+		}
+		return
+	}
+
+	if err := crypto.SaveKeyRing(h.path, next); err != nil {
+		h.logger.Error("save keyring", "error", err)
+		writeError(w, http.StatusInternalServerError, "failed to persist keyring")
+		return
+	}
+	h.keyring.Store(next)
+	h.logger.Info("keyring: removed entry", "kid", kid)
+
+	writeJSON(w, http.StatusOK, keyRingResponse{
+		Version: next.Version,
+		Entries: next.Snapshot(),
+	})
+}
+
+// findEntry returns the entry with the given kid, or nil. Reads only
+// — does not lock; caller already holds the handler mutex.
+func findEntry(kr *crypto.KeyRing, kid string) *crypto.KeyRingEntry {
+	for i := range kr.Entries {
+		if kr.Entries[i].KID == kid {
+			return &kr.Entries[i]
+		}
+	}
+	return nil
+}
+
+// constantTimeEq compares two byte slices in constant time. Used by
+// the install-idempotency check so reinstalling under an existing kid
+// doesn't leak timing about the existing secret.
+func constantTimeEq(a, b []byte) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	var v byte
+	for i := range a {
+		v |= a[i] ^ b[i]
+	}
+	return v == 0
+}
+
+func writeJSON(w http.ResponseWriter, status int, payload any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	_ = json.NewEncoder(w).Encode(payload)
+}
+
+func writeError(w http.ResponseWriter, status int, msg string) {
+	writeJSON(w, status, map[string]string{"error": msg})
 }