Skip to content

[deferred] Freeze every legacy subject's immutable agent id in one catalog transaction - #457

Closed
schickling-assistant wants to merge 3 commits into
schickling-assistant/2026-09-05-delta-003-immutable-idsfrom
schickling-assistant/2026-09-05-delta-003-migrate-ids
Closed

[deferred] Freeze every legacy subject's immutable agent id in one catalog transaction#457
schickling-assistant wants to merge 3 commits into
schickling-assistant/2026-09-05-delta-003-immutable-idsfrom
schickling-assistant/2026-09-05-delta-003-migrate-ids

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Stacked on #456 (schickling-assistant/2026-09-05-delta-003-immutable-ids). Review that first; this
branch's base is its head, not main.

Problem

#456 made an explicit immutable id expressible and taught every reader the target shapes. Nothing
assigns one. Until something does, effective_id answers with the legacy bus identity everywhere,
and step 3 of DELTA-003's reader-first
rollout — "in one catalog transaction, add migrated unique IDs to live and structurally archived
declarations, update archived tombstones, and rewrite every supervisor reference to its
already-resolved migrated ID"
— has no implementation.

That step cannot be a loop over declarations. R24 makes it one transaction for three reasons: an
archived subject may only freeze bytes that are unique across the combined live-and-archived
set; supervisor references must resolve against the pre-migration index, not against a plane
that is half-migrated underneath them; and a reassigned legacy bus identity must be recorded
durably in the same commit, because a version-1 durable message row's from/to bytes then denote
two subjects (docs/vrs/03-message/spec.md:88-107).

Change

st2 catalog migrate-ids [--host <host>] [--dry-run] [--resume] [--json], in
src/catalog_migrate_ids.rs. It is the same transaction shape src/catalog_archive.rs already
uses — one exclusive authoring lock, one strict discovery, a pure plan, every write inside one
generation commit — and reuses the existing primitives rather than adding machinery:
CatalogLock::exclusive, begin_generation_commit, catalog_archive::observe,
catalog_transaction::{validate_full_catalog, persist_tempfile_from_control, open_dir_beneath}, and
the authoring module's insert_node/exact_agent_node/quoted (widened to pub(crate), no
behavior change).

Assignment. Live subjects first, so a live claim is decided before any archived subject can
compete for the same bytes:

subject id why
live its <host>.<identity> bus identity, frozen preserves every runtime identifier, task ID, socket path, and declaration-anchored state path
archived, bytes unused in the combined set the same bytes archived continuity without admitting a duplicate
archived, bytes already claimed generated UUIDv7 global uniqueness; recorded in declaration and tombstone
already carries an explicit id untouched, and its ID joins the uniqueness set a legacy freeze may never take an ID a migrated subject owns

A live subject that cannot freeze its own bytes (because a migrated subject already owns them)
refuses — that is a duplicate-declaration fault for the operator, not something to reassign.

Reassignment record. <catalog>/.st2/agent-id-migration.json
(st2.agent-id-migration.v1) records every reassigned legacy bus identity, the subject that kept
it, and the reassigned subject's new ID. It is written only when something was reassigned. The
module exposes read_migration_record plus attribute_legacy_endpoint, which implements
MESSAGE-R04: an untouched legacy endpoint is its own migrated ID; a reassigned one resolves only
for the row's own state owner (sender for a sender-owned row, recipient for an inbox row) and is
otherwise unattributed, so a reader renders historical bytes rather than addressing the live
replacement. Wiring it into the message reader is PR D3's writer work; the function and its proof
land here with the record that makes them possible.

Supervisor references resolve against the combined pre-migration index — a full bus identity,
or a bare identity on the referring declaration's own resolved host, exactly as
supervisor_chain::resolve_spec reads them — and are rewritten to the parent's migrated ID.
Absence and ambiguity both refuse before any write with legacy-supervisor-unresolved, naming the
declaration and the reference. This matters on real data: 501 of dev3's 714 supervisor references
are bare identities that contain dots, so "which dot is the separator" is a real question and
answering it by precedence would be a guess.

Pre-admission gate. The catalog must already admit before a byte is rewritten. Migration
re-admits the whole live plane after its writes; a plane that already fails admission would fail
that re-admission for a reason migration did not cause, leaving every declaration rewritten, the
generation unmoved, and a marker whose resume can only fail the same way. --dry-run still reports
its complete plan against a non-admitting catalog, which is what makes it useful as a pre-flight.

Crash recovery. <catalog>/.st2/migrate-ids-incomplete records the planned assignment set
before the first write. A plain rerun over a half-migrated plane refuses and names --resume;
--resume re-plans from current bytes, refuses any ID the interrupted transaction did not plan or
any subject an outside writer gave a different ID, then finishes exactly the remainder.

Edits are span-bounded over exact source bytes: comments, formatting, and every unrelated byte
survive. TOML and JSON declarations refuse with unsupported-declaration-format. A Nix-owned
declaration is migrated — the meta { managed-by "nix" } marker guards interactive authoring
(st2 rename, st2 describe), not the one transaction that has to reach the whole plane — and the
receipt lists it under nixOwned so its generator can be taught to emit id before the next
activation re-projects the file without one. This is the single place the verb deliberately differs
from the authoring commands.

Also here: in-tree UUIDv7 (RFC 9562 §5.7, getrandom(2) on Linux and a character-device-checked
/dev/urandom elsewhere, no weak fallback — forty lines instead of a dependency and a lockfile
revision), and Tombstone.agentId plus its unarchive coherence check and graph projection.

Gates and evidence

nix develop -c cargo test --no-fail-fast:

suite result
tests/catalog_migrate_ids.rs 11 passed (new)
src/catalog_migrate_ids.rs unit tests 4 passed (new)
src/uuid_v7.rs unit tests 7 passed (new)
tests/catalog_archive.rs 15 passed
tests/catalog_graph.rs 10 passed
st2 lib 725 passed, 1 ignored
agent-spec (lib + 2 integration) 17 + 75 + 6 passed
tests/agent_publish.rs 9 failed — pre-existing
tests/catalog_apply.rs 14 failed — pre-existing

Both red files are the standing lane #438/#446 tracks on main: an independent worktree at
compoundingtech/st2 main (0d22352) fails the same 9 and the same 14 by name, and the
agent_publish nine were additionally proved identical at this stack's base b30bc54 by stashing
every change and re-running. Nothing in this PR touches publication or apply.

Proof corpus

tests/catalog_migrate_ids.rs, against a fixture shaped like a real catalog at migration time —
one counted root, a five-deep supervisor chain, dotted identities, 40 flat live subjects, 602
structurally archived subjects, and one identity claimed by both planes:

  1. every live subject froze its bus identity; every non-colliding archived subject froze its own;
    the colliding archived subject received a UUIDv7 (length, version nibble, and RFC 9562 variant
    all asserted) while the live claimant kept the bytes; the tombstone carries the same ID; the
    reassignment record names the keeper and the reassigned ID; every supervisor reference — including
    an archived subject supervised by another archived subject — names a migrated ID; a second run is
    a true no-op that advances no generation;
  2. --dry-run reports the whole plan and writes nothing (no ids, no marker, no record, generation
    unchanged);
  3. an unresolved reference refuses with legacy-supervisor-unresolved, naming declaration and
    reference, with nothing written;
  4. an ambiguous reference (a.b readable as host a's b and as host h's dotted identity a.b,
    both present) refuses rather than picking one;
  5. a TOML declaration refuses with unsupported-declaration-format;
  6. a Nix-owned declaration is migrated, reported under nixOwned, and keeps its meta block;
  7. a catalog that does not currently admit refuses before any write;
  8. an explicit id is left alone and blocks a freeze that would claim it;
  9. an interrupted transaction (abort injected after the first declaration is published) leaves its
    marker, refuses a plain rerun, and resumes exactly the remaining four declarations, advancing one
    generation;
  10. --resume with no marker refuses;
  11. an archived directory with no readable tombstone refuses, because a hidden subject could
    otherwise lose its bytes.

Against a copy of dev3's real declaration plane

st2 catalog snapshot --raw-preimage (read-only, shared lock, generation 1034 before and after)
produced a 722-declaration copy; every run below is against that copy, never the live catalog.

  • migrate-ids --host dev3 --dry-run --json: 722 assigned, all frozen-bus-identity, 714
    supervisor rewrites, 0 reassignments, 18 nixOwned, 0 already migrated.
  • migrate-ids --host dev3 --json on the unrepaired copy: refused by the pre-admission gate,
    naming 4 pre-existing socket-path-too-long declarations (fix(validate): reject an unbindable session socket path at admission #442's rule against identities that
    predate it) and one unresolvable profile module (an artifact of the raw preimage, which does not
    capture _profiles/**). Zero declarations were rewritten and no marker was left.
  • After repairing the copy's pty-root and profile envelope: 722 ids frozen, 714 references
    rewritten, generation 0 → 1, marker cleared, 722/722 declarations carrying id; a rerun reports
    unchanged and leaves the generation at 1; catalog graph --json then reads 722 agents,
    complete: true, 0 conflicts, every id equal to its frozen bus identity, and parentId/rootId
    ID-keyed.

Evidence file: tmp/agent-cleanup/dev3-evidence.md (gitignored) carries the before/after
measurements and the full shape characterization.

Operator sequence for the live dev3 migration

Not run by this PR. It needs the dotfiles catalog projection to emit id first, plus a
serialized cutover window. Exact sequence, in order:

  1. Repair the four socket-path-too-long declarations. dev3's live plane does not currently
    admit its own rules, so the gate refuses before anything else:
    2026-09-02-new-agent-sam.schickling-assistant-202.request.{2psw1i,2ynlcp,nguieu} and
    global.coding-agents.session-creation.interview-1785674581013-613809-0. Each needs a shorter
    identity/task id or a shorter declared pty-root. Verify with st2 validate --strict.
  2. Teach the dotfiles projection to emit id for every seat it projects, using each seat's
    existing <host>.<identity> as the frozen value, and to preserve it across re-projection. Until
    this lands, the 18 Nix-owned declarations this verb migrates are re-projected without an id at
    the next activation. Order matters: projection-emits-id before the live migration, or the
    migration is undone for those 18 on the next nix-convergence activate.
  3. Quiesce the writers. Stop the resident supervisor (systemctl --user stop st2) so its
    auto-archive pass (feat(supervisor): archive retired seats after a grace period #449, ≤25 per pass, 7-day grace) cannot take the exclusive authoring lock
    mid-transaction, and hold the st2-catalog activation so no catalog apply races it. This is
    one serialized catalog transaction; nothing else may write the declaration plane.
  4. Snapshot for rollback. st2 catalog snapshot --output <path> --json, and record
    rootSha256 plus .st2/catalog-generation.
  5. Dry run. st2 catalog migrate-ids --host dev3 --dry-run --json > plan.json. Confirm
    reassigned is empty (dev3 has zero archived subjects today, so no UUIDv7 path is exercised),
    assigned equals the declaration count, and nixOwned is the set step 2 covered.
  6. Migrate. st2 catalog migrate-ids --host dev3 --json. On an interruption, rerun with
    --resume; never with a plain rerun.
  7. Verify. st2 catalog graph --host dev3 --json is complete: true with 0 conflicts and every
    id equal to the seat's former bus identity; st2 validate --strict is clean; the generation
    advanced by exactly one; no .st2/migrate-ids-incomplete remains.
  8. Restart the supervisor and release the activation hold. Confirm live PTY sessions kept their
    PID, creation identity, and generation — an ID freeze changes no runtime identifier, and that is
    the property the whole "freeze, do not re-key" design exists to preserve.
  9. Rollback, if needed before writers are activated: st2 catalog apply the step-4 snapshot
    under its recorded digests. The frozen IDs are additive declaration state; removing them returns
    the plane to its pre-migration bytes.

Intentionally left

  • Every writer. ST_AGENT, default task IDs, PTY presentation tags, message provenance, and
    runtime ownership still use the legacy bus identity. Freezing an ID equal to that identity is
    precisely what makes this PR safe to land before them.
  • Wiring attribute_legacy_endpoint into the message reader, and the version-2 writers. PR D3.
  • unarchive refusing an unmigrated archive after activation, and validating ID uniqueness
    against the prospective live-and-archived set. Both are activation-gated behavior; the tombstone
    coherence check that does not depend on activation is here.
  • UUIDv7 for new subjects in new-agent/authoring paths. The generator exists and is proved;
    the creation paths move in D3.
  • Address migration. Nothing assigns an address. Positional identity remains every
    subject's effective address, exactly as R24's fallback specifies.
  • The live dev3 migration itself, per the sequence above.
  • INVARIANTS rows and the DELTA-003 status, which move with the writer PR.

Stack created with GitHub Stacks CLIGive Feedback 💬

Posted on behalf of @schickling
field value
agent_identity dev3.direct.omp.v6c4mkm2
session dev3.v6c4mkm2
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.1.2
agent_runtime OMP 18.1.2
tooling_profile dotfiles@7534055

R24 gives every new agent subject a UUIDv7, and ID migration gives one to an
archived subject whose legacy bus identity another subject already claimed. The
layout is RFC 9562 section 5.7 and the whole implementation is forty lines, so it
lives here rather than adding a dependency and a lockfile revision.

Entropy is `getrandom(2)` on Linux and a character-device-checked
`/dev/urandom` elsewhere. There is deliberately no clock- or pid-derived
fallback: immutable agent ID uniqueness rests on these bits, and a caller inside
a catalog transaction needs a refusal before its first write, not a weak id.
An archived subject's immutable ID has to survive outside the live plane: R24
gives an archived collision a UUIDv7 "in its declaration and tombstone", and
`st2 catalog graph --json` must be able to show an archived subject's ID.

- `Tombstone` gains optional `agentId`, and `ArchivedEntry` carries it in the
  archive receipt. The tombstone schema stays v1: an added optional field
  changes no existing field's meaning, and the record does not deny unknown
  fields, so readers of both vintages parse both shapes. `id` keeps its meaning
  as the legacy `<host>.<identity>` key, and its coherence assertion stays.
- Archival never mints identity. A legacy declaration archives with no
  `agentId`; freezing one is the migration verb's job.
- `unarchive` refuses when a tombstone's recorded ID and the archived
  declaration's explicit `id` disagree, before anything moves.
- The graph's archived row appends nullable `agentId`.

Refusing an unmigrated archive after activation, and validating ID uniqueness
against the prospective live-and-archived set, land with the writer PR.
One atomic transaction that makes an existing catalog expressible in decision
0015's identity model without re-keying any durable state.

- A live subject freezes its existing `<host>.<identity>` bus identity as its
  explicit `id`, so every runtime identifier, task ID, socket path, and
  declaration-anchored state path keeps its exact bytes.
- A structurally archived subject freezes the same bytes when they remain unique
  across the combined live-and-archived set. An archived collision — the shape a
  re-projecting generator and an archiving supervisor produce together — receives
  a UUIDv7 in both its declaration and its tombstone, and the reassignment is
  recorded in `.st2/agent-id-migration.json` so a reader of a version-1 durable
  record never retypes colliding bytes into the wrong subject.
- Supervisor references resolve against the combined *pre-migration* index and
  are rewritten to the parent's migrated ID in the same transaction. A missing or
  ambiguous reference refuses before any write with `legacy-supervisor-unresolved`.
- The catalog must already admit: migration re-admits the whole live plane after
  its writes, and a plane that already fails admission would fail that
  re-admission for a reason migration did not cause, leaving every declaration
  rewritten and the generation unmoved. A dry run still reports its full plan.
- `--resume` finishes an interrupted run and may only apply work the interrupted
  transaction planned. A plain rerun over a half-migrated plane refuses.

Edits are span-bounded over exact source bytes through the authoring module's
existing insertion helper, so comments, formatting, and every unrelated byte
survive. TOML and JSON declarations refuse. A Nix-owned declaration IS migrated —
the marker guards interactive authoring, not the one transaction that has to
reach the whole plane — and the receipt names it so its generator can be taught
to emit `id` before the next activation re-projects the file without one.
@schickling-assistant schickling-assistant changed the title schickling assistant/2026 09 05 delta 003 migrate ids Freeze every legacy subject's immutable agent id in one catalog transaction Sep 5, 2026
@schickling-assistant
schickling-assistant marked this pull request as ready for review September 5, 2026 13:04
@schickling-assistant

Copy link
Copy Markdown
Contributor Author

@codex review

Posted on behalf of @schickling
field value
agent_identity dev3.direct.omp.aetn947t
session dev3.aetn947t
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.1.2
agent_runtime OMP 18.1.2
tooling_profile dotfiles@7534055

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@schickling-assistant

Copy link
Copy Markdown
Contributor Author

Superseded by #464, which keeps the reader, migration, writer, runtime, and routing transition in one coherent change so no independently mergeable partial identity state exists.

Posted on behalf of @schickling
field value
agent_identity dev3.direct.omp.2ahzpbs3
session dev3.2ahzpbs3
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.1.2
agent_runtime OMP 18.1.2
tooling_profile dotfiles@7534055

@schickling-assistant

Copy link
Copy Markdown
Contributor Author

Deferred, not rejected. The branch stays; nothing here is contradicted by what shipped.

Why. The #453 problem statement — a provisional semantic route becomes permanent because the route and the durable key are the same string — is answered by the mutable address alone, with no migration, no record version, and no activation gate. That half ships in #456 (grammar, dup-id/dup-address, roster/graph) and #467 (st2 agent address, AgentSelector, fail-closed resolution). The freeze migration buys two further properties — host-move-invariant identity and live/archive collision attribution — both real, both pre-existing, and both with zero observed instances: this PR's own dev3 dry run reports 0 reassignments over 722 declarations, and dev3 has no archived subjects at all.

Re-entry conditions (now written into docs/vrs/.decisions/0015-… as Amendment 1): a completed cross-host seat move, a live/archive identity collision on any admitted host, or a UUIDv7 creation call site.

Two defects to fix before this lands, both found by audit and recorded in the amendment and in the narrowed DELTA-003:

  1. supervisor_chain::resolve_spec (src/supervisor_chain.rs:30-33) matches spec.bus_id(host) == reference || spec.identity == reference and is byte-untouched by this stack, so a fully migrated catalog whose parent was born with UUIDv7 loses its org-chart edge (graph complete=false, child parentId=null, supervisor-missing). That is exactly the clause this migration exists to satisfy (0015:61-64), and it is unmet. It needs an effective_id arm.
  2. agent-id-missing (src/validate.rs:587, added by [deferred] Activate the immutable-id identity model behind a catalog migration gate #463) makes a partially migrated catalog inadmissible, and this verb's pre-write admission gate refuses to write into a non-admitting catalog — so the rollout order this PR prescribes (teach the projection to emit id, then migrate) deadlocks, and the diagnostic recommends the command that just refused. migrate-ids must exempt agent-id-missing from its own gate, since it is the only writer that can clear it.

Also worth revisiting when this returns: src/uuid_v7.rs (175 lines, one call site in the archived-collision arm) versus uuid = { version = "1", features = ["v7"] }, which Cargo.lock:2973-2982 already carries transitively.

Posted on behalf of @schickling
field value
agent_identity dev3.direct.omp.v6c4mkm2
session dev3.v6c4mkm2
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.1.2
agent_runtime OMP 18.1.2
tooling_profile dotfiles@7534055

@schickling-assistant schickling-assistant changed the title Freeze every legacy subject's immutable agent id in one catalog transaction [deferred] Freeze every legacy subject's immutable agent id in one catalog transaction Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant