Flaky: single_instance drop_releases_so_next_acquire_succeeds fails ci-success on main (blocks releases)

[charliek]

roost-engine's single_instance::tests::drop_releases_so_next_acquire_succeeds fails intermittently, and when it does it fails rust-build → ci-success on main.

This gates releases. release.yml's ci-gate polls ci-success on the tagged commit and refuses to release unless it is green. A flake on that particular run stops a release until CI is re-run.

Observed failures

Same test, three separate main runs, both platforms — so it is not macOS-specific:

Date	Runner	Run
2026-08-07	macos-latest	31135056075
2026-08-07	ubuntu-latest	31138315777
2026-08-09	macos-latest	on the 98f0987 merge (plan 023 PR #316)

Each time: 152–157 passed; 1 failed, and the same assertion.

The failure

thread 'single_instance::tests::drop_releases_so_next_acquire_succeeds' (50817)
panicked at crates/roost-engine/src/single_instance.rs:181:
called `Result::unwrap()` on an `Err` value: AlreadyHeld(18263)

The test is:

let dir = tempdir().unwrap();
let first = acquire(dir.path().join("roost.lock")).unwrap();
drop(first);
let second = acquire(dir.path().join("roost.lock")).unwrap();  // <- panics here

Note the number in parentheses after the thread name is Rust's thread id, not a pid. The reported holder — AlreadyHeld(18263) — is the test process's own pid, read back out of the lock file it just wrote. So at the moment of the second acquire, flock(LOCK_EX | LOCK_NB) returned WouldBlock against a lock file in a freshly created, uniquely named tempdir(), whose only writer was this same process a few microseconds earlier.

In other words: drop(first) did not reliably release the flock before the next acquire in the same process.

What is not the cause

• Not a stale file from another run — tempdir() is unique per test.
• Not another process — the pid in the file is this process's own.
• Not platform-specific — reproduced on both ubuntu and macOS runners.
• Not caused by plan 023's changes — that PR touched chrome colors, a pixel test, and docs, and the same commit passed on its own PR run and on the next merge to main.

Where to look

crates/roost-engine/src/single_instance.rs. Two candidates worth checking first:

1. Descriptor duplication. read_pid does file.try_clone(). A dup'd descriptor shares the same open file description, and the flock is only released when every copy is closed. That path only runs on the error branch, but cargo test runs these tests concurrently in one process, so it is worth confirming no clone can outlive its InstanceLock.
2. Drop ordering. InstanceLock has a manual Drop impl whose body deliberately does nothing; the _file field is dropped after the body returns. That should be synchronous, but it is the load-bearing assumption the test is asserting, and it is the assumption that appears to fail.

If the release genuinely is asynchronous under some condition, that matters beyond the test: the same "drop then re-acquire" sequence is what a restart-in-place would do.

Suggested handling

Root-cause it rather than retry it — a single-instance lock that does not reliably release is a real property to know about. But if a quick fix is not available, note that this is the sort of flake that will eventually land on a release commit at an inconvenient moment.

Related: #319 (the split XDG_RUNTIME_DIR lock issue) is in the same subsystem, though a different defect.

Refs

• crates/roost-engine/src/single_instance.rs — acquire, read_pid, impl Drop for InstanceLock, and the test at ~:172-183
• .github/workflows/release.yml — ci-gate, which makes this release-blocking

[charliek]

Does not reproduce locally; it is CI-load-shaped (plan 023 follow-up).

150 consecutive runs of cargo test -p roost-engine single_instance on an M-series Mac: 0 failures. So whatever this is, it needs the contention profile of a CI runner (shared vCPUs, the full cargo test --workspace suite running in parallel around it) rather than the test in isolation.

Practical implication for whoever picks this up: the first task is a reproduction, not a fix. Suggested angles, cheapest first — run the whole workspace suite in a loop rather than the single module; pin --test-threads high; add CPU contention (the shed reproduced a different timing bug at 4 hogs on 2 vCPUs, see #297); and try a Linux runner too, since this has failed on both ubuntu-latest and macos-latest.

Frequency data, for prioritisation: 3 of the last 30 main CI runs failed, and all three were this test. Every other main run in that window was green — so this is currently the only recurring cause of a red required check, and release.yml's ci-gate requires ci-success green on the tagged commit.