Skip to content

fix: replace fcntl.flock spawn-mutex with a cross-platform primitive (Windows no-op today) - #2096

Open
KeilerHirsch wants to merge 1 commit into
MemPalace:developfrom
KeilerHirsch:fix/daemon-windows-spawn-mutex
Open

fix: replace fcntl.flock spawn-mutex with a cross-platform primitive (Windows no-op today)#2096
KeilerHirsch wants to merge 1 commit into
MemPalace:developfrom
KeilerHirsch:fix/daemon-windows-spawn-mutex

Conversation

@KeilerHirsch

Copy link
Copy Markdown
Contributor

Problem

start_daemon()'s spawn mutual-exclusion guard is built entirely on fcntl.flock. The comment above it states why it exists:

two concurrent daemon start callers would both observe no running daemon and both spawn a child, double-claiming jobs

But fcntl is POSIX-only, so _fcntl is None on Windows and the whole if _fcntl is not None block is skipped there. On Windows there is no mutual exclusion of any kind — the exact race the comment describes is completely unguarded.

When it fires, two near-simultaneous auto-start callers (two hooks firing back-to-back, or a hook plus a manual daemon start) both call subprocess.Popen. Both children write endpoint.json and pid.json; whichever process is not referenced by the final endpoint.json becomes a permanently orphaned daemon, since stop_daemon() only ever targets the pid recorded there. It holds a Python process, an HTTP port, and a live sqlite connection indefinitely, and there is no supported way to stop it.

Fix

Add _try_lock_start_file() / _wait_lock_start_file(), which use fcntl.flock on POSIX and msvcrt.locking on Windows, and rewire start_daemon() to them.

Both are advisory locks tied to the open file handle and release automatically when it closes, including on abnormal process termination. I measured this on Windows before relying on it:

  • msvcrt.locking(LK_NBLCK, 1) succeeds on a handle opened in "w" mode
  • a second handle to the same file correctly fails with OSError while the first holds the lock
  • the lock is released immediately when the holder is force-killed, so no stale-lock cleanup is needed
  • close() alone releases it, without an explicit LK_UNLCK

msvcrt has no indefinite-blocking mode (LK_LOCK gives up after ~10s), so the Windows wait polls the non-blocking primitive up to a 30s ceiling instead of blocking forever like flock(LOCK_EX).

Error contract

Both helpers raise DaemonError rather than a raw OSError. cli.py's cmd_daemon and _submit_daemon_cli_job catch only DaemonError, so an OSError escaping from here would surface as an unhandled traceback instead of the usual mempalace: daemon error: ... / exit 1 path.

Handle lifetime

This also closes the lock file handle on the two paths that leave start_daemon before the readiness-probe try/finally that owns it: the lock-wait failure, and the "reuse the winner's daemon" return existing. The latter leaked an open handle to start.lock on every contended start before this change, on POSIX too.

Tests

  • _try_lock_start_file actually contends (second handle is refused) and _wait_lock_start_file wakes once the holder releases — exercising whichever primitive the running platform provides, not a mock.
  • Two concurrent start_daemon() callers race through the real on-disk lock, with only subprocess.Popen and the readiness probe faked; asserts exactly one spawn. A threading.Barrier forces both callers to observe "no daemon running" at the same instant, so the race window is deterministic rather than scheduler-dependent. This test fails on the current code with got 2 spawns.
  • The lock handle is released on both early-exit paths.
  • Both platforms translate a lock failure into DaemonError. The POSIX test fakes the fcntl module so it runs on every CI platform rather than only Linux/macOS.

Notes for reviewers

Two adjacent issues are deliberately not included here, to keep this change focused:

  • _detached_kwargs() / subprocess.Popen() sit outside any try that closes lock_fh. If Popen raises, the handle leaks. This is unchanged from develop and affects the uncontended first-spawn path too, so it is orthogonal to this fix.
  • _LOCK_WAIT_TIMEOUT (the Windows-only poll ceiling) is decoupled from start_daemon's caller-supplied timeout. Under contention on Windows, a caller passing a short timeout can still wait up to 30s in the lock phase. Now that this surfaces as a clean DaemonError it is a latency surprise rather than a correctness problem, but plumbing timeout through would be a reasonable follow-up.

Happy to fold either in if you would prefer them handled together.

start_daemon()'s spawn mutual-exclusion guard was built entirely on
fcntl.flock. fcntl is POSIX-only, so `_fcntl` is None on Windows and the
whole `if _fcntl is not None` block was skipped there: no mutual exclusion
of any kind. Two near-simultaneous auto-start callers (two hooks firing
back-to-back, or a hook plus a manual `daemon start`) both observed no
running daemon and both called subprocess.Popen. Both wrote endpoint.json
and pid.json; whichever process did not win that race became a permanently
orphaned daemon, since stop_daemon() only ever targets the pid recorded in
endpoint.json.

Add _try_lock_start_file() / _wait_lock_start_file(), which use fcntl.flock
on POSIX and msvcrt.locking on Windows, and rewire start_daemon() to them.
Both primitives are advisory locks tied to the open file handle and release
automatically when it closes, including on process crash.

msvcrt has no indefinite-blocking mode (LK_LOCK gives up after ~10s), so the
Windows wait polls the non-blocking primitive up to a 30s ceiling. Both
helpers raise DaemonError rather than a raw OSError: cli.py's cmd_daemon and
_submit_daemon_cli_job catch only DaemonError, so an OSError escaping here
would surface as an unhandled traceback instead of the usual
"mempalace: daemon error: ..." exit-1 path.

Also close the lock file handle on the two paths that leave start_daemon
before the readiness-probe try/finally that owns it -- the lock-wait failure
and the "reuse the winner's daemon" return. The latter leaked an open handle
to start.lock on every contended start before this change, on POSIX too.

Tests: assert the primitive actually contends and that a blocked waiter
wakes on release; drive two concurrent start_daemon() callers through the
real on-disk lock (not mocked) and assert exactly one spawn; assert the
handle is released on both early-exit paths; assert both platforms translate
a lock failure into DaemonError. The POSIX translation test fakes the fcntl
module so it runs on every CI platform, not just Linux/macOS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant