fix(postgres): rebuild the datasets already streaming when a join replaces their replication slot (fixes #13229)

[claudespice]

📝 Summary

Two code paths can replace a PostgreSQL logical replication slot, and only one of them tells the datasets already streaming from it that their recorded positions no longer reach it.

slot::ensure_slot — reached from attach_member when a dataset joins — drops a slot the server has invalidated (wal_status = 'lost', from an exhausted max_slot_wal_keep_size or a PostgreSQL 18 idle-timeout) and creates a replacement at the current WAL position. Members already attached to the old slot are not told, so the pump's next connect succeeds against the replacement and streams on as though the history were still there: each already-attached member resumes from its own recorded position, which a slot created at the current WAL position cannot reach. Any row deleted at the source in between has no change event left to carry the deletion, so it survives in the acceleration and in every later query.

The pump's own recovery from a refused stream already did the right thing — replace, then ask every attached member to rebuild and hold it until that rebuild is durable. This makes the join path do the same, through one function the two now share.

Changes

• The setup reports what it did to the slot's history (SlotInfo::history: Kept, ReplacedInvalidated, CreatedFromAbsent, FastForwarded) instead of the caller inferring it from created_fresh. created_fresh does not separate the cases — it is true both where an existing slot's history was thrown away and where there was none to throw away — and acting on it moves floors that are still owed their changes (Shared Postgres replication slot: a member joining a resuming slot second can skip changes, silently losing rows #12609).
• A slot created because none existed is a replacement too, when this source already has members. An operator dropping the slot while the pump is between connections — the only time PostgreSQL permits it — reaches the same end state as an invalidated one, and the members streaming from what is now gone are owed the same rebuild. Whether it counts is the caller's decision, because only the source knows whether anyone was streaming.
• The outcome is latched across setup's retries (HistoryLatch). Setup is retried as a whole, and an attempt that drops or creates a slot has already changed the source when it fails; the next attempt would find an ordinary slot and report Kept.
• install_replacement_slot — the pump's recovery body, now shared with the join path: seed the shared flush at the replacement's start, ask every attached member to rebuild and hold it until that rebuild commits, and reseat the floors held for published tables whose datasets have not joined. Every member is now held before any request is enqueued, without an await in between: the send can park on a full mailbox and the pump reconnects without taking setup_lock, so holding as each member's turn came would leave the rest promotable and creditable for the length of that park.
• SharedSource::slot_generation, so one replacement is not acted on twice. The pump latches a refusal against the slot epoch its connection was streaming and acts on it after its reconnect backoff — the window in which a join can replace that slot. recover_unusable_slot compares the generation under the setup lock and skips a refusal whose slot is already gone; without it the pump would drop a healthy slot and leave every member holding two rebuild requests against one rebuild_pending flag, where the first to commit releases the hold the second still needs. The generation is published only after the fan-out completes, so an adoption that does not finish does not read as one that did.

Test plan

• make lint
• make nextest
• New in-process tests:
  • a_join_that_replaced_the_slot_rebuilds_the_members_already_attached — one rebuild request per attached member, each still held across the pump's next promote, each keeping its own floor, and the unjoined table's held floor reseated to the replacement's start.
  • a_join_on_a_slot_that_kept_its_history_leaves_attached_members_alone — the Shared Postgres replication slot: a member joining a resuming slot second can skip changes, silently losing rows #12609 direction, identical to the above but for the history signal.
  • a_join_that_created_a_slot_under_members_already_attached_rebuilds_them and the_first_member_to_create_a_slot_replaces_nothing — the two sides of the CreatedFromAbsent rule.
  • a_refusal_of_an_already_replaced_slot_does_not_replace_it_again — the generation guard.
  • a_later_attempt_cannot_report_away_what_an_earlier_one_did and a_latch_no_attempt_wrote_reports_the_attempts_own_outcome — the retry latch.
• Each was checked against a neuter matrix — removing the join-path rebuild, keying it on created_fresh, dropping reseat_held_floors, dropping the member hold, removing the generation guard, and forcing the CreatedFromAbsent rule to each constant — every neuter fails at least one test, and every test is failed by at least one neuter.

Not included, and why. The two-dataset integration test in the issue's Done-Done needs the crates/runtime/tests/postgres Docker harness, which is unavailable in the environment this was written in; it is filed as #13306 rather than shipped unverified. FastForwarded discards history for every member in the same way, and the issue excludes it from this signal because keying on it regressed two held-floor tests that also need Docker — #13305. Two residuals the review below surfaced are filed rather than grown into this PR: #13312 (a replacement discovered by a join that then fails or is cancelled is forgotten) and #13311 (the rebuild hold is one flag, so the first of two overlapping rebuilds releases the second's).

Review gate

• Adversarial review: codex — job review-mt0p1ohg-v533hy — run four times as the diff changed; 12 findings across the runs. Acted on in this PR: the retry losing the discard signal, a stale refusal replacing the replacement, the CreatedFromAbsent path, the retry losing that outcome too, the fan-out holding members one at a time, and a refusal tagged at dequeue rather than at connect. Filed rather than grown into this PR: A slot replacement discovered by a join is forgotten if that join fails or is cancelled after the DDL #13312, A member's rebuild hold is a single flag, so the first of two overlapping rebuilds releases the second's hold #13311, A shared replication slot fast-forwarded for a re-bootstrap discards history under members already attached #13305. The final run still returns needs-attention on exactly those three filed residuals.
• /security-review: no findings — the diff adds no input, route, credential, or log surface; the only SQL it touches is the existing parameterized slot-catalog statements.
• /simplify: applied — four findings fixed (a doc comment stranded onto the wrong function by an insertion, adopt_* naming that collided with this module's schema-evolution vocabulary, a doc narrating prior behavior, and rationale duplicated across four comment blocks), light checks and the neuter matrix re-run green afterward.

🔗 Related

Fixes #13229

Follow-ups filed from this work: #13305, #13306, #13311, #13312.

👀 Notes for Reviewers

The signal is deliberately reported rather than inferred, because the issue's second requirement is that it must not overlap created_fresh — and the CreatedFromAbsent case shows why the decision has to be split between the two files: only ensure_slot knows what it did to the slot, and only the source knows whether anyone was streaming from what it replaced.

[github-actions]

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

• kind/bug

Passing checks:

• ✅ Title meets minimum length requirement (10 characters)
• ✅ No banned labels detected
• ✅ Has a label from required category kind/
• ✅ Has a label from required category area/
• ✅ Has at least one assignee: claudespice

[claudespice]

@copilot review

[Copilot]

Pull request overview

Fixes PostgreSQL shared-slot replacement handling so attached datasets rebuild instead of silently retaining stale rows.

Changes:

• Adds explicit, retry-aware slot-history outcomes.
• Shares replacement recovery between join and pump paths.
• Adds generation guards and regression tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
crates/data_components/src/postgres_replication/slot.rs	Reports and latches slot-history changes.
crates/data_components/src/postgres_replication/shared.rs	Rebuilds attached members and guards replacement recovery.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.