Context
The nightly REM cycle (specs/FLAIR-NIGHTLY-REM.md, slice 2 in #707) is currently host-scoped: a launchd/systemd timer invokes the runner, which snapshots to the local filesystem and calls the local REST API. That shape is correct for a single-node deploy and silently wrong for a clustered one.
Problems in a multi-node deploy
- N timers, one corpus. Every node running its own nightly timer means N distillation runs over the same replicated memory corpus: duplicate model spend, racing candidate staging, and N audit trails claiming to be "the" nightly run. Exact-duplicate claim skipping makes this converge, but convergence-by-dedup is a backstop, not a design.
- Snapshot locality.
~/.flair/snapshots/ lives on whichever node ran the cycle. The parent spec's inviolable — "every night is reversible" — quietly degrades to "reversible if you're on the right node."
- In-node concurrency (lesser issue): two concurrent
execute calls for the same agent can race between the validate and insert phases. All-or-nothing staging plus pending-dedup bounds the damage; a cheap per-agent in-flight guard closes it.
Direction
Single-runner semantics per agent corpus, in escalating strength:
- v1 (config): nightly enable is a deliberate per-deploy action — exactly one node gets the timer. Document this as the clustered-deploy rule. Zero new machinery.
- v2 (lease): runner acquires a lease row (agent-scoped, TTL) before cycling; skips cleanly when the lease is held. Caveat to design honestly: with async replication a table lease is advisory, not a true distributed lock — combine with jittered start times and the idempotent staging backstop.
- End-state: REM as an explicit org-level singleton process (the org-REM direction in
DESIGN.md) — scheduled in one place by the platform rather than by per-host cron, with snapshots exported somewhere cluster-durable rather than node-local.
Slice 2 (#707) doesn't change this calculus and shouldn't grow scope for it; this issue tracks the clustered-deploy semantics as its own slice (docs rule in the near term, lease + snapshot durability after).
Context
The nightly REM cycle (
specs/FLAIR-NIGHTLY-REM.md, slice 2 in #707) is currently host-scoped: a launchd/systemd timer invokes the runner, which snapshots to the local filesystem and calls the local REST API. That shape is correct for a single-node deploy and silently wrong for a clustered one.Problems in a multi-node deploy
~/.flair/snapshots/lives on whichever node ran the cycle. The parent spec's inviolable — "every night is reversible" — quietly degrades to "reversible if you're on the right node."executecalls for the same agent can race between the validate and insert phases. All-or-nothing staging plus pending-dedup bounds the damage; a cheap per-agent in-flight guard closes it.Direction
Single-runner semantics per agent corpus, in escalating strength:
DESIGN.md) — scheduled in one place by the platform rather than by per-host cron, with snapshots exported somewhere cluster-durable rather than node-local.Slice 2 (#707) doesn't change this calculus and shouldn't grow scope for it; this issue tracks the clustered-deploy semantics as its own slice (docs rule in the near term, lease + snapshot durability after).