Workers self-detect displacement; the Foreman becomes relay and janitor - #179
Open
jallum wants to merge 2 commits into
Open
Workers self-detect displacement; the Foreman becomes relay and janitor#179jallum wants to merge 2 commits into
jallum wants to merge 2 commits into
Conversation
…bedrock-q67.38) FDB has no component that decides another process's retirement: every role self-detects displacement and throws worker_removed on itself — TLogs against ServerDBInfo (isDisplaced), storage servers via rejoin validation through a commit proxy's txnStateStore. The Foreman's reconciliation rule was Bedrock answering a membership question on the workers' behalf; it dies here. - The Foreman relays each newly durable layout push to its running hosted workers (do_relay_tsl) and janitors deliberate retirements (worker_retired -> dispose directory, registration, entry; :transient restart keeps a deliberate shutdown down). do_reconcile_workers and workers_to_retire are deleted. - Shale checks its own id against the pushed log set, guarded by epoch progression: only a layout that progressed past our epoch may retire us (absence during an in-flight recovery is not a death sentence); a never-locked resurrection is judged by any completed layout. - Olivine rejoin-validates: one tag-keyed ask to a pushed commit proxy (CommitProxy.resolve_materializer, answered from RoutingData — the serverList analogue). Absence or another worker's id is authoritative displacement; locked/unavailable are not verdicts and the next push revalidates.
The audit caught the guard defeating the design: the locking phase locks old-layout logs and every advertised materializer into the judging epoch BEFORE the push, so a strict pushed > mine guard meant locked-but- excluded workers were never judged — strays became immortal and displaced logs deferred cleanup a full recovery generation. The guard is now pushed >= locked epoch: a layout may judge every worker it had the chance to include; only a push OLDER than our lock — an in-flight recovery's past — is off-limits. Test matrices now pin the real displaced states (own-epoch push retires an excluded log; the completing push validates a locked stray). Also from the audit: - nil layouts (the coordinator's director-start clear broadcast) are not relayed, and the liveness bound (a worker that cannot run cannot self-detect) is documented on the relay. - the foreman pulls the Link's cached layout once at spin-up, closing the cold-boot push-miss window (the coordinator replays on Link subscription, which can precede the foreman). - stale reconciliation-era comments and guides updated to the self-detection rule.
jallum
pushed a commit
that referenced
this pull request
Aug 21, 2026
…6-08-20/21 as stacked PRs #178-#182 (each cold-audited, findings applied, CI green): A=q67.37 typed pull-source seed + ShardRouter as the single placement site (#178); B=q67.38 workers self-detect displacement (epoch guard: a layout judges every worker it could include, pushed >= locked — the strict guard was caught by audit making strays immortal), Foreman relay+janitor, resolve_materializer rejoin ask (#179), services leaves the TSL — ServerDBInfo parity {epoch, sequencer, proxies, resolvers, logs} (#180); E=q67.41 epoch-constant log wiring (fold deletion; layout/logs stays for introspection), carried start keys consumed, worker ids ride the assignment (#181); D=q67.40 by-key covering-entry resolution, two-tier partial coalescing cache (#182). Remaining in the arc: q67.21 Distributor port (mapping ownership + metadata-shard availability + in-band retirement + overlap-aware client cache), q67.42 bootstrap read continuation, q67.20.7 legacy fallback removal."
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Foreman's reconciliation rule (
do_reconcile_workers) answered a membership question on the workers' behalf: on every layout push it decided which hosted workers to retire. FDB has no component that decides another process's retirement — every role self-detects displacement and throwsworker_removedon itself: TLogs computeisDisplacedagainst ServerDBInfo (rejoinClusterController, TLogServer.actor.cpp), storage servers validate their rejoin against a commit proxy's txnStateStore (absence =worker_removed). The hosting process only observes the exit and deletes files.Course item B, first half (bedrock-q67.38, settled 2026-08-20 after FDB source check). Depends on #178 (the seed had to stop carrying
transaction_servicesfirst).What
worker_retireddisposes the directory, registration, and entry of a worker that decided its own retirement (:transientrestart keeps a deliberate shutdown down).do_reconcile_workers/workers_to_retireare deleted — the Foreman never answers a membership question again.materializers/<tag>), not the wiring push — so rejoin validation is one tag-keyed ask to a pushed commit proxy: newCommitProxy.resolve_materializer/3, answered fromRoutingData(the serverList analogue) under the same cadence rule asfetch_routing. Absence or another worker's id is an authoritative verdict;:locked/:unavailable/:timeoutare not — the next push revalidates.How
Foreman.Impl.do_relay_tsl/2sends{:tsl_updated, tsl}to every running hosted worker;do_worker_retired/2reuses the existing removal path (terminate-if-alive → deregister → rm -rf).RoutingData.resolve_materializer/2is a tag lookup in the same routing view that serves clients — one authority, two readers.shale/displacement_test.exs(epoch-progression guard matrix) andolivine/displacement_test.exs(verdict vs non-verdict matrix, stub proxy).Deferred to q67.21 (recorded on the ticket): in-band privatized-mutation delivery for mid-epoch removals — do not build the private-mutation path before removals exist.
servicesleaving the TSL is the second half of q67.38, next PR in the stack.Full suite (2582 tests), credo --strict, dialyzer green.