Skip to content

refactor(lock): typed file-lock guard for every hand-rolled flock site - #511

Merged
schickling-assistant merged 3 commits into
mainfrom
schickling-assistant/2026-09-07-flock-transport
Sep 7, 2026
Merged

refactor(lock): typed file-lock guard for every hand-rolled flock site#511
schickling-assistant merged 3 commits into
mainfrom
schickling-assistant/2026-09-07-flock-transport

Conversation

@schickling-assistant

Copy link
Copy Markdown
Contributor

Applies issue #504 §T1-LOCK. Two commits: extract the transport, then migrate the sites.

Eight places in this tree hand-rolled the same three system calls — open a lock file, flock it,
close it — each with its own unsafe block, and four of them with their own impl Drop. This PR
puts those three calls in one module and routes seven of the eight through it.

Not a pure dedupe. The diff carries three deliberate behaviour changes, listed below with the
test that pins each. If you only read one section, read that one.

What landed

src/flock.rsFileLock guard, Mode { Shared, Exclusive }, Wait { Block, Now },
Open { Create, CreateNew, Existing }. Design rules that made byte-identity achievable:

  • It returns std::io::Result and never anyhow, so every callsite keeps the exact context string
    it reports today. Eight distinct diagnostics stay eight.
  • Ok(None) means contention and nothing else. Every other failure is Err, so a caller cannot
    mistake a lock file it may not open for a lock file somebody else holds.
  • It does not create directories, does not fsync, and does not name lock files. The callers
    legitimately disagree about all three (one creates the parent, one requires the parent to be
    0700 first, one names its lock by suffixing a foreign config path), so absorbing those would
    hide the disagreement rather than resolve it.
  • open and hold stay split, because the window between them is load-bearing:
    catalog_lock writes its two debug contention checkpoints there (five tests read
    ST2_TEST_CATALOG_LOCK_ATTEMPT), and resource_profile proves its opened lock is a regular
    file there.
  • Defaults are resource_profile::lock_publication's — the one site already correct on every axis,
    so they are house style rather than invention: O_RDWR, O_NOFOLLOW, O_CLOEXEC (already
    implied by Rust's OpenOptions; named so a reader does not go hunting for it), create mode
    0600.

CatalogLock is the first consumer, byte-identically: same two context strings, same checkpoint
placement, its private Mode/Wait and its hand-rolled impl Drop are now the transport's.

Then the seven migrations: catalog_transaction::initialize_bootstrap_control,
codex_app_server::acquire_owner_lock, event::StreamLock, harness_state::lock_exclusive
(rippling through context::lock_now and its 3 call sites), message::SentLock,
pretrust::ConfigLock, resource_observe::lock_request_scope. Four impl Drop deleted
(CatalogLock, StreamLock, SentLock, ConfigLock). SentLock's file: Option<File>, which
was never None, collapses to one guard.

Exactly one unsafe flock block is left in production code: resource_profile::lock_publication,
deliberately not migrated. It opens via openat against a retained directory capability, returns
a CatchUpError taxonomy the publication state machine matches on, and proves the opened lock is a
regular file between the open and the flock. Also untouched: host_lock.rs (a pid file that
must survive process death, not a flock site) and every test-side raw flock (those are
adversaries taking the lock from outside the process under test).

Behaviour changes, explicitly

1. O_NOFOLLOW newly hardens five lock files

event, the harness_state acquisition (shared by the harness-state record lock, the
harness-context record lock, and context's now.md lock — three paths, one acquisition),
message, pretrust and resource_observe set neither O_CLOEXEC nor O_NOFOLLOW today. A
symlink planted at any of those lock paths currently redirects the lock domain — and the write that
follows — to its target. After this PR the open fails with ELOOP.

No current test and no live path reaches those lock files through a symlink, so nothing observes
the change today. Pins, mirroring the existing catalog_lock_refuses_a_symlinked_lock_file:

pin file
flock::tests::a_symlinked_lock_path_is_refused_instead_of_locking_its_target src/flock.rs
harness_state::tests::a_symlinked_record_lock_is_refused_instead_of_locking_its_target src/harness_state.rs
context::tests::a_symlinked_now_lock_refuses_the_write_instead_of_locking_its_target src/context.rs
message::tests::a_symlinked_ledger_lock_is_refused_instead_of_locking_its_target src/message.rs
pretrust::tests::a_symlinked_trust_lock_is_refused_instead_of_locking_its_target src/pretrust.rs
resource_observe::tests::a_symlinked_scope_lock_is_refused_instead_of_locking_its_target src/resource_observe.rs

Each asserts the errno is ELOOP, that the symlink target's bytes are unchanged, and (where the
entry point publishes) that nothing was published.

2. Four of those five lock files start being created 0600 instead of 0644

event, harness_state (×3 paths), message, pretrust passed no mode and so created their
lock files 0644 & ~umask. resource_observe already used 0600 and additionally re-tightens a
pre-existing lock file with set_permissions — that call is kept at the callsite, because the
transport only applies mode at creation, so an older build's 0644 lock file would otherwise
stay 0644.

Existing lock files are not re-moded by this PR (except resource_observe's, unchanged from
today). The live catalog does have 0644 .lock files from older builds; they keep that mode and
keep working, since their owner has write access.

Pins: pretrust::tests::the_trust_lock_is_created_private_and_excludes_a_second_holder,
harness_state::tests::a_symlinked_record_lock_is_refused_instead_of_locking_its_target (asserts
0600 on a freshly created record lock),
resource_observe::tests::the_scope_lock_excludes_a_second_holder_and_releases_on_drop,
flock::tests::a_created_lock_file_is_private_to_its_owner.

3. message::shared_existing opens O_RDWR where it opened O_RDONLY

This is the one that needs a human's eye. SentLock::shared_existing is doctor's read-only path
into a sender's ledger (message::inspect_sent, reached from st2 doctor). It opens the ledger
lock O_RDONLY today; the transport has one open shape, so it becomes O_RDWR.

Stated plainly: a doctor running as a different uid against a 0644 ledger lock would start
failing where it read successfully today.
Read access is no longer sufficient; write access is
required.

Evidence that nothing in this repo or on the fleet does that:

  • Every file in the live catalog (~/.local/state/st2/default/catalog, including the archive) is
    owned by a single uid, schicklingfind -printf '%u' over the tree returns exactly one name.
  • st2.service is a systemd user unit with User= empty, so the supervisor and everything it
    launches run as schickling.
  • The repo contains no uid-switching at all: setuid, seteuid, sudo -u, runuser and
    User= do not appear in any .rs, .nix, .kdl or .toml.
  • Root is the only other reader and bypasses the mode check entirely.

So there is no non-root, non-schickling reader, and the narrowing is unobservable on this fleet.
The reasoning is recorded on shared_existing in the source, not just here.

Smaller diagnostic notes

  • codex_app_server::acquire_owner_lock: contention is now Ok(None) mapped to the same
    operator-visible message, "Codex runtime already has an owner at {path}". Previously that
    message carried the EWOULDBLOCK io::Error as its source; now the contention arm carries no
    source (the non-contention error arm still does). Pinned unchanged by the existing
    codex_app_server::tests::runtime_owner_lock_is_nonblocking_and_released_on_close.
  • harness_state::lock_exclusive, event::StreamLock::exclusive and message::SentLock keep
    their exact top-level messages and now attach the underlying io::Error as a source where they
    previously discarded it. pretrust::ConfigLock keeps its "locking {path}: {errno}" string
    byte-for-byte.
  • harness_state::lock_exclusive also widens O_WRONLY to O_RDWR, same single-uid argument as
    above; it creates its own lock file 0600.

Inert, but stated anyway

Two changes that the diff carries and that observably change nothing:

  • Four sites — catalog_transaction.rs:1439, codex_app_server.rs:2957, harness_state.rs:697,
    resource_observe.rs:609 — previously handed back a bare File and released the lock only as a
    side effect of close(2). They now hand back a FileLock, whose Drop issues an explicit
    LOCK_UN before the same close. That is identical unless the descriptor has been duplicated,
    because close on the last descriptor of the open file description releases the lock anyway. No
    site dups: none calls try_clone/dup, none passes the descriptor to a child (all carry
    O_CLOEXEC), so the explicit unlock is redundant rather than new behaviour.
  • src/lib.rs gains a module line for flock. Every item in it is pub(crate) and the module
    itself is declared mod flock; (private, like the sibling fsatomic transport), so the crate's
    public surface is unchanged in both directions.

The thinnest safety net, filled in

harness_state, harness_context, context, message, pretrust and resource_observe had
no dedicated lock test anywhere, and this PR changes their open mode, permissions and symlink
behaviour. The regression oracle is cheap and deterministic and needs no threads: flock locks the
open file description, so a second open of the same lock file in the same process contends
with a live guard.

  • message::tests::inspect_sent_leaves_no_guard_held_and_creates_no_sender_state — the
    load-bearing lifetime in the set. inspect_sent takes up to two sequential shared guards around
    one unlocked read; neither may outlive the call, or a doctor read would block every subsequent
    publication by that sender. Scope, precisely: the test pins the first guard only. Once the
    lock file exists, shared_existing answers Some and inspect_sent early-returns holding that
    first guard, which is therefore the guard whose release is proven. Reaching the second guard
    needs shared_existing to answer None and then Some — a race a single-threaded test cannot
    arrange — so it is left unpinned rather than fake-covered, and the assertion text and doc
    comment say so. The test also asserts the read-only path creates no sender state.
  • message::tests::the_ledger_lock_excludes_a_second_holder_and_releases_on_drop
  • pretrust::tests::the_trust_lock_is_created_private_and_excludes_a_second_holder
  • resource_observe::tests::the_scope_lock_excludes_a_second_holder_and_releases_on_drop
  • flock::tests::contention_under_wait_now_is_ok_none_and_a_dropped_guard_releases,
    flock::tests::a_shared_holder_admits_a_second_reader_but_not_a_writer,
    flock::tests::create_new_refuses_an_existing_lock_file_and_existing_refuses_a_missing_one

Every one of these was mutation-checked: with O_NOFOLLOW removed from flock::open,
SentLock::exclusive weakened to Mode::Shared, and inspect_sent's early-return guard (the one
the test actually exercises) leaked via std::mem::forget, all eight relevant pins fail; with the
mutations reverted (diff verified identical) all eleven pass.

Not in this PR

  • resource_profile::lock_publication — see above.
  • host_lock.rs — not a flock site.
  • Test-side raw flock in src/run/tests.rs, tests/agent_presentation.rs,
    crates/st2-resource-providers/src/vista.rs — adversaries, deliberately raw.
  • No lock file's path or name changed.

Independent review

A reviewer with a clean context was asked to falsify the exclusivity claim above. Verdict: the
three stated behaviour changes plus the four diagnostic notes are the only ones
— no bug, no
deadlock, no blocking/non-blocking flip, no guard-lifetime regression. It re-derived CatalogLock's
ordering line by line (canonicalize, .st2/ creation, root fsync, control openat, open, BOTH
debug checkpoints, flock, apply marker, generation intent, struct build, intent recovery — all
unchanged, both context strings unchanged, and the checkpoints still strictly between the open and
the hold, so all six ST2_TEST_CATALOG_LOCK_ATTEMPT consumers observe what they observed), mapped
every site's Mode/Wait/Open 1:1 against origin/main, and confirmed every migrated guard is a
named binding rather than a temporary.

It surfaced two changes worth stating, both inert, now in the section above; and one honesty fix,
applied in the last commit: the inspect_sent pin's assertion claimed to cover both sequential
guards, but the test's own setup creates the lock file first, so shared_existing answers Some and
the function early-returns holding the FIRST guard — the second is never reached. Reaching it needs
None then Some, a race a single-threaded test cannot arrange, so the test now says exactly what
it proves instead of faking the coverage.

Posted on behalf of @schickling
field value
agent_identity dev3.direct.omp.43sz6ujq
session dev3.43sz6ujq
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.1.7
agent_runtime OMP 18.1.7
tooling_profile dotfiles@39a19af

Eight sites in this tree hand-rolled the same three system calls — open a lock
file, `flock` it, close it — each with its own `unsafe` block and, for four of
them, its own `impl Drop`. `src/flock.rs` owns those three calls and nothing
else: it returns `std::io::Result` so each caller keeps its own context string,
`Ok(None)` means contention and nothing else, and it neither creates
directories, nor fsyncs, nor names lock files, because the callers legitimately
disagree about all three. `open` and `hold` stay split because the window
between them is load-bearing.

Defaults are `resource_profile::lock_publication`'s — the one site already
correct on every axis — so they are house style rather than invention:
`O_RDWR`, `O_NOFOLLOW`, `O_CLOEXEC`, create mode `0600`.

`catalog_lock` is the first consumer, byte-identically: its two debug
contention checkpoints still sit exactly between the open and the `flock`, its
two context strings are unchanged, and its private `Mode`/`Wait` plus its
hand-rolled `impl Drop` are now the transport's.

agent-identity: dev3.direct.omp.43sz6ujq
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.1.7
agent-runtime: OMP 18.1.7
tooling-profile: dotfiles@39a19af
…uard

The seven remaining hand-rolled sites now open and hold through
`crate::flock`, leaving exactly one `unsafe` flock block in production code:
`resource_profile::lock_publication`, which opens through `openat` on a
retained directory capability, proves the opened lock is a regular file between
the open and the `flock`, and returns a `CatchUpError` taxonomy the publication
state machine matches on. `event::StreamLock`, `message::SentLock` and
`pretrust::ConfigLock` lose their hand-rolled `impl Drop`; `SentLock`'s
`file: Option<File>`, which was never `None`, collapses to one guard.

Three deliberate behaviour changes ride along, each now pinned:

- `O_NOFOLLOW` newly hardens five lock files that set neither flag before —
  `event`, the `harness_state` acquisition shared by `harness-state`,
  `harness-context` and `context`'s `now.md`, `message`, `pretrust` and
  `resource_observe`. Six new symlink-refusal tests, mirroring the existing
  `catalog_lock_refuses_a_symlinked_lock_file`.
- Four of those five start being created `0600` instead of `0644`. Existing
  lock files keep their mode; `resource_observe` already re-tightened its own.
- `message::shared_existing` opens `O_RDWR` where it opened `O_RDONLY`, so a
  reader with read-but-not-write access to a ledger lock would newly fail.
  Every file in the live catalog is owned by one uid and st2 runs as a systemd
  user unit with no `User=`, so no such reader exists.

`harness_state`, `harness_context`, `context`, `message`, `pretrust` and
`resource_observe` had no lock test at all. They get one each, using the
oracle that `flock` locks the open file description: a second open of the same
lock file in the same process observes a live holder without threads. The
`message::inspect_sent` pin is the load-bearing one — its two sequential
guards must both be released before it returns, or a doctor read would block
every later publication by that sender.

agent-identity: dev3.direct.omp.43sz6ujq
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.1.7
agent-runtime: OMP 18.1.7
tooling-profile: dotfiles@39a19af
… real scope

The flock module has no external consumer: every caller is a sibling module in
this crate, so it is declared `mod flock;` and matches the fsatomic transport's
shape rather than adding an empty public surface.

The inspect_sent guard test claimed to prove that both of the function's shared
guards are released. It cannot: by the time it runs the lock file exists, so
`shared_existing` answers `Some` and inspect_sent early-returns on its first
guard. That first guard is the one the test's mutation check covers. Reaching the
second needs `shared_existing` to answer `None` then `Some`, a race a
single-threaded test cannot arrange, so the assertion text and doc comment now
state exactly that instead of overstating coverage.

agent-identity: dev3.direct.omp.43sz6ujq
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.1.7
agent-runtime: OMP 18.1.7
tooling-profile: dotfiles@39a19af
@schickling-assistant
schickling-assistant marked this pull request as ready for review September 7, 2026 12:14
@schickling-assistant
schickling-assistant merged commit 05b1890 into main Sep 7, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant