Commit 4421582
authored
fix(engine): close concurrent preempt race in target-state ownership transfer (#1994)
When two components race to own the same target state, the prior fix
relied on tokio scheduling to serialize their `pre_commit` ->
`sink_apply` -> `commit` sequences. That holds for LMDB (microsecond
ops) but breaks under PG latency, where the loser's `delete` lands
after the winner's `upsert` and the target state is lost.
Add a `pending_process_token: Option<u128>` to `TargetStateInfoItem`,
written by `pre_commit` whenever it queues a sink action and cleared
by `commit_in_txn`'s retention pass. A detection sub-pass at the top
of `pre_commit` peeks the token on every preempt-source item: live
(same process token) -> return `PendingRetry` with the unconsumed
declared map so `submit()` can re-invoke after a backoff; dead
(crashed prior process) -> force `prev_may_be_missing=true` on
reconcile.
The detection sub-pass also caches old-owner `tracking_info` bytes
into a per-call `HashMap<StablePath, Vec<u8>>` shared with the Phase 1
preempt branch -- each owner is read once, modified in place across
multiple preempts, and emitted as a single `DeferredWrite` at the end
of Phase 1.
`set_provider_generation` (OnceLock-backed, can't run twice) is
deferred until after the last possible PendingRetry exit. On
`sink_apply` / `commit_in_txn` failure, `rollback_pending_tokens`
re-reads tracking_info and clears every token matching this process's
token; retried indefinitely with logged backoff until success. The
process-exits-mid-rollback case is covered by the dead-token recovery
path on next startup.
`contained_target_state_paths` wrapped in `Arc` to avoid full HashSet
rehash on every retry iteration.
See `specs/target_state_ownership_transfer/concurrent_preempt_race_fix.md`.1 parent 92cf6c3 commit 4421582
3 files changed
Lines changed: 375 additions & 77 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
29 | 35 | | |
30 | 36 | | |
31 | 37 | | |
| |||
47 | 53 | | |
48 | 54 | | |
49 | 55 | | |
| 56 | + | |
50 | 57 | | |
51 | 58 | | |
52 | 59 | | |
53 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
54 | 66 | | |
55 | 67 | | |
56 | 68 | | |
| |||
0 commit comments