Skip to content

REM nightly in clustered deploys: single-runner semantics + snapshot durability #709

Description

@tps-flint

Context

The nightly REM cycle (specs/FLAIR-NIGHTLY-REM.md, slice 2 in #707) is currently host-scoped: a launchd/systemd timer invokes the runner, which snapshots to the local filesystem and calls the local REST API. That shape is correct for a single-node deploy and silently wrong for a clustered one.

Problems in a multi-node deploy

  1. N timers, one corpus. Every node running its own nightly timer means N distillation runs over the same replicated memory corpus: duplicate model spend, racing candidate staging, and N audit trails claiming to be "the" nightly run. Exact-duplicate claim skipping makes this converge, but convergence-by-dedup is a backstop, not a design.
  2. Snapshot locality. ~/.flair/snapshots/ lives on whichever node ran the cycle. The parent spec's inviolable — "every night is reversible" — quietly degrades to "reversible if you're on the right node."
  3. In-node concurrency (lesser issue): two concurrent execute calls for the same agent can race between the validate and insert phases. All-or-nothing staging plus pending-dedup bounds the damage; a cheap per-agent in-flight guard closes it.

Direction

Single-runner semantics per agent corpus, in escalating strength:

  • v1 (config): nightly enable is a deliberate per-deploy action — exactly one node gets the timer. Document this as the clustered-deploy rule. Zero new machinery.
  • v2 (lease): runner acquires a lease row (agent-scoped, TTL) before cycling; skips cleanly when the lease is held. Caveat to design honestly: with async replication a table lease is advisory, not a true distributed lock — combine with jittered start times and the idempotent staging backstop.
  • End-state: REM as an explicit org-level singleton process (the org-REM direction in DESIGN.md) — scheduled in one place by the platform rather than by per-host cron, with snapshots exported somewhere cluster-durable rather than node-local.

Slice 2 (#707) doesn't change this calculus and shouldn't grow scope for it; this issue tracks the clustered-deploy semantics as its own slice (docs rule in the near term, lease + snapshot durability after).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions