Thread-per-backend crash isolation is weaker than process-per-backend and the safety net is untested against torn shared state

[MauricioPerera]

Summary

pgrust uses a thread-per-backend model (one OS thread per connection/auxiliary process, all in a single address space) instead of PostgreSQL's process-per-backend (fork()) model. This is a deliberate, understandable architectural choice, but it changes one of Postgres's most important safety properties — crash isolation — in a way that I don't think is currently proven safe or well-tested, and I want to raise it for discussion/hardening rather than as a single-line bug.

Background: how real Postgres handles this

In C Postgres, if a backend process crashes (segfault, assertion failure, etc.), the postmaster (a separate, unaffected process) detects it via SIGCHLD, assumes shared memory might be corrupted, SIGQUITs every other backend, waits for full quiescence, and resets shared memory from scratch before restarting. Crucially, the postmaster itself is immune to the crash because it never touches the crashing backend's private memory — only shared memory is a shared-fault domain, and even that is unconditionally discarded and rebuilt.

What I found in pgrust

There are two distinct crash classes here, and the code is explicit about the difference (crates/backend/postmaster/launch_backend/src/lib.rs:918-920): "HandleChildCrash. Covers the catchable crash class only (caught panics); memory-safety violations are process-fatal by design."

1. Caught Rust panics (unwrap(), assertions, explicit panic!) are caught via std::panic::catch_unwind per backend thread (launch_backend/src/lib.rs:508), translated into a synthetic crash-exit event, and routed through handle_child_crash → HandleFatalError (crates/backend/postmaster/postmaster/src/statemachine.rs:85-127), which SIGQUITs every other thread, waits for quiescence, and runs ipc::shmem_exit/ipci::ResetShmemAfterCrash/bgworker::BackgroundWorkerShmemInit before restarting from PM_STARTUP. This part structurally mirrors C's contract.

2. However, genuine memory-unsafety crashes (SIGSEGV/SIGBUS/SIGILL) are handled by crates/backend/postmaster/postmaster/src/crash_signals.rs, whose own header comment states: "backends are threads, so a fatal signal kills the whole server... [C's] postmaster survives the child." The handler just logs and re-raises — it does not, and structurally cannot, run the reinit ladder, because the process that would run it is the process that's dying. Recovery depends entirely on an external supervisor (systemd, container orchestrator) restarting the whole binary from scratch. This is a materially weaker guarantee than C's, where the postmaster always survives to relaunch.

3. The catch_unwind path itself is not proven safe against torn shared state. Locking (LWLock/spinlock, crates/backend/storage/lmgr/lwlock/src/lib.rs:849-1144) is manual/C-style, not RAII — nothing releases a held lock on unwind. A panic while holding a lock, or mid-mutation of a shared buffer/freelist/syscache/allocator structure, leaves that structure exactly as torn as it was at the panic point; nothing cleans it up before SIGQUIT reaches siblings. In C, the equivalent scenario (quickdie, which also skips cleanup) only ever risks shared memory, which gets unconditionally rebuilt anyway — sibling processes' private memory is untouchable by construction. Here, all threads share one heap/allocator (the mcx arena system already reviewed separately). A panic triggered by real undefined behavior in unsafe code (buffer overrun, use-after-free, data race — pgrust has plenty of unsafe in hot paths like the JIT deform kernel, buffer manager, memory contexts) can in principle corrupt allocator metadata or shared data structures that other live threads are actively reading/writing at that exact moment, before SIGQUIT even reaches them. This failure mode is structurally impossible under process-per-backend and is, as far as I can tell, structurally possible here.

Why I'm not filing this as a single reproducible bug

I don't have (and don't think static review alone can produce) a concrete repro that demonstrates actual corruption — this is a systemic property of the crash-recovery design, not a specific defect in one function. I want to flag it honestly as a risk worth deliberate attention rather than manufacture a fake one-line "bug."

What I'd suggest

• Document this trade-off explicitly (e.g. in GOAL.md or a dedicated architecture doc) so users/operators understand that pgrust's crash-isolation guarantee for genuine memory-unsafety faults is "the whole server dies, an external supervisor must restart it," rather than C's "the postmaster survives and cleanly resets."
• Consider whether the catch_unwind recovery path needs stronger tests specifically targeting the risk described in point 3 above: e.g. a test that deliberately panics a worker thread while it holds an LWLock or is mid-mutation of a shared structure, and asserts either (a) the reinit ladder still produces a correct, non-corrupted post-restart state, or (b) documents/asserts that this specific scenario is out of scope and relies on the external supervisor.
• I looked at the existing crash-recovery test, crates/backend/postmaster/postmaster/tests/crash_restart.rs: it fabricates a crash by directly calling postmaster_seams::announce_child_exit(VICTIM_PID, SIGABRT) from the test's own thread — no real panic is thrown, no lock is held at fault time, and no shared-state mutation races the sibling's SIGQUIT delivery. It's a solid proof of the state machine's orchestration/bookkeeping correctness, but it does not exercise (and therefore doesn't currently give confidence about) the torn-shared-state scenario in point 3.

Environment / where found

Found via static source review of a fresh clone of this repo, while reviewing the postmaster/connection-handling architecture generally. Have not attempted to build/run pgrust myself, and have not attempted to construct an actual crash-under-lock repro; this is a design-level concern raised from reading crates/backend/postmaster/{postmaster,launch_backend}/src/*.rs and crates/backend/storage/lmgr/lwlock/src/lib.rs directly, not a demonstrated corruption.

[Bodobolero]

Prior discussion https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4845/slides/407/Multi-threaded%20PostgreSQL%20.pdf
and
https://www.postgresql.org/message-id/31cc6df9-53fe-3cd9-af5b-ac0d801163f4@iki.fi

[malisper]

Thank you! I'm going to dig more into this and see what the best solution to address this is