test(cluster): promote 4 QA replication/failover/blob-copy regression anchors

[kriszyp]

Three exploratory-QA specs promoted into the tracked cluster suite. Test-only — no product code touched.

What each spec pins

integrationTests/cluster/copyResumeCursorGap.test.mjs (from QA-689, candidate P-468 — source: #537 "Interrupted bulk copies can persist a resume cursor over an undelivered range")
A full-copy receiver is SIGKILLed and restarted repeatedly mid-range (10 interruptions across 5%–88% of a 30k-row / 900-byte-per-row copy, each one hard-asserted to land strictly between 0 and the full row count, so the anchor can never pass vacuously). The oracle is a bidirectional primary-key-set comparison between source and receiver after final convergence — deliberately not a row count and not cluster_status, since the reported failure mode is precisely that the cursor and cluster_status read "caught up" over a silent gap.

This covers the corner copyProgressWedgeRecovery.test.mjs (#453) does not: that one freezes a copy that makes no progress at all; this one interrupts a copy that makes real, committed, cursor-advancing progress.

integrationTests/cluster/auditReplayYieldExcludedTables.test.mjs (from QA-690, candidate P-469 — regression anchor for #535, fixed by 70ec9485 / #536)
Pins that the audit replay loop yields while walking long not-subscribed-table skip runs. qa522-audit-replay-yield-live.test.mjs already exercises this fix, but via a synthetic dropped-table trick (its own comments explain that was a workaround). This spec drives the real production mechanism — static-route excludeTables — at larger scale (60k excluded rows across two runs, interleaved with subscribed-table traffic), with the sender pinned at threads.count: 1 so an external ops-API ping necessarily shares the event loop with the replication send loop. A starved loop (the pre-#536 behavior) shows up as multi-second ping gaps; the assertion is on max gap, plus hard preconditions that the skip run was real (excluded = 60000 on A, exactly 0 on B).

integrationTests/cluster/failoverWriteVisibility.test.mjs (from QA-651, candidate P-442 — source: #1846 "post-failover write-visibility divergence")
Brings up a 3-node full mesh (A–B, A–C, B–C — not a star, so B and C stay able to talk once A is gone), drives a steady write stream into A, kills A mid-stream (precondition hard-asserted: the stream must die with writes acked and more attempted), then takes a fresh batch of writes on survivor B. Every write is then read back through seven surfaces on both survivors: indexed query, primary point-read, full scan, SQL, REST, and — the load-bearing part — direct raw-index and raw-base store scans via a component endpoint.

The raw-store scans are why this isn't redundant with the other survivor tests: search_by_value, REST and SQL all join through the primary record and skip on absence, so none of them can ever prove an index entry is dangling (or that a base row lost its index entry). Only the direct indices.writer.getRange() / primaryStore.getRange() dump can. It then restarts a survivor and re-probes, separating genuine self-heal from restart-only recovery.

On the missing waitForCatchUp: deliberate, and commented in the file. waitForCatchUp derives its target version from the source's highest lastReceivedVersion across all of that source's connections, then checks it against the receiver's single direct link to that source. In a full mesh every edge is direct, so the receiver's link-from-source can only reflect what crossed that one edge and can never reach a target derived from the source's other edges — it times out regardless of the bound. Measured: the zero-wait probe already showed C at 25/25 on every surface, while waitForCatchUp(C, B, {timeoutMs: 15000}) still timed out. Replaced with a bounded count poll whose result is now asserted (the earlier revision of this spec logged a swallowed 60 s timeout as informational, which both cost ~60 s per run and would have masked a real convergence regression).

Gate evidence

All three cleared the QA promotion gate on harper-pro 86f2955e / core cda8d63f6 (2/2 cold reruns each), then were re-verified green in their promoted location on this branch:

spec	result	runtime
auditReplayYieldExcludedTables	3/3 pass	12.3 s
copyResumeCursorGap	1/1 pass	37.1 s
failoverWriteVisibility	1/1 pass	30.8 s

failoverWriteVisibility non-vacuity on the promoted run: A killed after 16 acked / 17 attempted writes; both survivors 25/25 on all seven surfaces with danglingIndex=[] / danglingBase=[], unchanged after C's restart.

All three are standalone files (requires-isolation: each kills and restarts nodes and asserts whole-instance convergence properties, so none can share a booted cluster with co-tenant tests). Plain node:assert, teardownHarper cleanup, no restartHttpWorkers, no fixed sleeps standing in for a convergence wait.

Draft for review — happy to split, rename, or drop any spec.

---

Added 2026-07-24: blobCopyInterruptionIntegrity.test.mjs (QA-692 / P-501)

The fourth anchor, and the one that closes the blob half of gh-pro#537. copyResumeCursorGap (above) cleared the single-source, non-blob shape; this covers the corner the code-read flagged as likeliest — outstandingBlobsToFinish / hasBlobGap make blob commits the plausible source of out-of-order onCommit completion, i.e. a resume cursor advancing past bytes that were never durably written.

A 120-record × 1 MiB blob table (all file-backed, well above FILE_STORAGE_THRESHOLD) is seeded on the source before the receiver joins, so the whole dataset rides the full copy. The receiver is SIGKILLed at seven fractions of copy progress (5–90%), restarted each time with its original config re-passed.

Three-layer oracle, because a blob defect can present as a corrupt file rather than a missing key:

1. bidirectional primary-key-set comparison — not counts, not cluster_status
2. blob content read back and compared byte-for-byte to deterministic expected bytes — catches present-but-empty/truncated
3. direct {dataRootDir}/blobs/data/… walk on both nodes — every live id has a correctly-hashed backing file, and zero files carry content matching no seeded id

Non-duplicative vs the existing blob cluster tests: blobGapDeadlock and blobSaveRejectionContainment never kill a node; replicationBlobResyncOnFailure and replicationBlobRepairAuthoritative each kill once during live replication with an injected save-failure. None interrupts a bulk full copy, and none walks the blob store for orphans across the copy.

Gate evidence: 2/2 cold reruns on harper-pro 86f2955e / core cda8d63f6 (71.1 s / 68.5 s), then green in the promoted location on this branch (69.3 s, 1/1). Interruption precondition armed 7/7, 6/7, 7/7 across those runs — every armed kill landed strictly between 0 and 120 records on the receiver.

Honest note: the interruption loop is progress-driven, so the real kill count varies with machine speed; the 0.9 checkpoint occasionally finds the copy already complete and is skipped by design. The test asserts at least one genuine interruption occurred, so it cannot pass vacuously.

Requires HARPER_RUN_STRESS_TESTS=1. requires-isolation like the other three.

One observation deliberately not asserted: interruption leaves behind structurally-incomplete blob stub files (truncated bodies from writes abandoned at SIGKILL), each superseded by a correctly-resent complete file. No data-integrity impact, and the codebase already treats surplus blob files as benign precedent — but it is a small disk-space leak under sustained restart churn, and worth a look at whether GC reclaims them.

---

Added 2026-07-25: copyReloadLiveSubscribers.test.mjs (QA-578 / P-507)

Regression anchor for #495, fixed by #507 ("emit reload markers for user-DB tables so live subscribers recover copy-applied rows"; commits 0b817f28, 9a54099e, decbca38).

Base-copy rows land on the receiver as audit-less snapshots, and the live-subscription notify path is driven off the audit stream — so those rows fire no per-row events, and without the reload marker a live subscriber never sees them at all. #507 tracks which user tables actually received a copied row this pass (copiedTablesThisPass / shouldEmitCopyReloadMarker) and writes a whole-table reload marker once the copy is durable; Table.ts's subscribe handler turns that into a re-delivery of the table's scope as ordinary put events.

aclConnectCopyReload.test.mjs already covers this over MQTT, with the subscriber attached before the copy. This spec pins the corner it does not:

• the REST/SSE transport, and
• a subscriber that attaches MID-copy — caught by polling the receiver for its 0 -> N transition and attaching at that moment, with the attach-time row count recorded as proof it was genuinely mid-copy — plus
• an ordinary live write after the copy, proving the subscription survived the copy rather than being silently torn down.

Topology is A (leader, seeds 20k rows / ~16 MB) + B (attach-before) + C (attach-mid-copy), each joining A independently so the two orderings don't perturb each other's copy pass.

Non-blind by construction: the receiver's row count is sampled via describe_table on a connection entirely separate from the SSE stream and asserted to transition 0 -> N while the subscriber is attached, and the SSE-observed ids are compared against the source id set (not a count), so a defect delivering the wrong rows is caught too. Revert #507 and the row-count oracle still passes while the id assertion fails — which is the property that makes this an anchor rather than a smoke test.

Gate evidence: 3/3 cold runs green on harper-pro 86f2955e / core cda8d63f6 — two in scratch (13.0 s / 14.5 s) and one in the promoted location on this branch (13.4 s). The mid-copy attach was genuine in all three (attachCount=1, confirmed at 2001–3000 of 20001 rows).

Honest note: mid-copy genuineness is logged, not asserted — if a future machine copies 20k rows faster than the poll can attach, the arm silently degrades to a post-copy attach (still a valid assertion, but the mid-copy coverage claim would be vacuous). Asserting it would trade a silent coverage gap for a CI flake, so it is logged deliberately; if the log ever shows "copy completed before attach landed", raise RECORD_COUNT.

requires-isolation like the other four (it mutates cluster topology via add_node and depends on copy timing). node:assert/strict — the dominant convention in this directory (22 of 41 tracked cluster tests) — per-test timeouts throughout, teardownHarper cleanup, no fixed sleeps standing in for a convergence wait.

---

Added 2026-07-25: connectedBitRestartChurn.test.mjs (QA-587 / P-508)

Regression anchor for the general, black-box-observable guarantees around #431 / #523 ("reconcile connected bit UP from shared-memory truth").

Two nodes, six replicated databases. Three genuine SIGKILL + restart cycles under concurrent write and admin load, then a fourth outage held past WEDGE_RECONCILE_THRESHOLD_MS so the disruptive findWedgedNodeUrls net fires. It asserts the connected bit never wedges stuck-false, never reports a false green (recovery is proven by a real data round-trip, not by the bit), and that the wedge-reconcile net does fire on a long outage — carrying #525's structured truth={...} prior={...} telemetry.

Non-blind: every kill captures the child process's actual exit signal and asserts signal === 'SIGKILL', and every cycle asserts a genuine connected:false was observed before restart. No cycle can pass without a real injected fault.

Scope — what this does NOT pin (also stated at the top of the file, so nobody later "strengthens" it into a flake): #523's actual up-correction path. connectedToNode() only fires on WS 'open' when this.nodeSubscriptions is already populated (replicationConnection.ts:1388), which is only unset for a fresh connection object — built by wedge-reconcile's forceResubscribe, not by a routine reconnect. The line Corrected replication connection state (disconnected -> up) did not appear in any of five runs, despite multi-db fan-out, write/admin load, and deliberately crossing the 30 s threshold: the window is sub-millisecond (WS handshake vs async subscribe()) and there is no fault-injection hook for it. That path's only coverage remains unitTests/replication/reconcileEntryWithTruth.test.mjs. If a hard end-to-end gate on that specific path is wanted, it needs a test-only hook mirroring armReplicationWedgeForTest / maybeStallCopyForTest to delay a fresh connection's subscribe() past its 'open' event — happy to open that separately, it's product code so it's out of scope for a test-only PR. The log evidence here is reported, never asserted.

Not duplicative: replicationReconnect.test.mjs does a single kill→restart→connected:true on one database; replicationWedgeRecovery.test.mjs covers the different #420 open-but-idle shape (socket open, no close event) via the armReplicationWedgeForTest hook. Neither drives repeated crash churn under load across a six-database fan-out, and neither crosses the wedge-reconcile threshold.

Gate evidence: 4/4 tests green, 3/3 cold runs after the margin repair below (~80 s each), including in the promoted location on this branch. Fast-cycle convergence 1.53–1.83 s; long-outage convergence 12.0 s.

Gate note — a real flake, caught and fixed: an earlier revision held the long outage 34 s and passed 2/2 cold, then failed on the 3rd (wedge-reconcile fired during outage: false). The 30 s threshold is measured from the disconnect stamp, which is only set once the peer notices the SIGKILL (WS close 1006), and the sweep ticks every RECONCILE_INTERVAL_MS (5 s) — so the effective margin was ~1–4 s and a late stamp pushed the first qualifying tick past the restart. Widened to threshold + detection lag + two sweep ticks (48 s). Worth flagging to reviewers: two cold reruns is not a sufficient gate for an assertion whose trigger depends on a periodic sweep's phase.

Runtime honesty: ~80 s, the longest file in this directory (the others are 12–37 s), and ~48 s of that is a deliberate outage that cannot be shortened without re-introducing the flake. requires-isolation like the others.

🤖 Generated with Claude Code — authored by Claude Opus 5 (qa-explorer loop)

[gemini-code-assist]

Code Review

This pull request introduces two new integration tests to cover critical replication regression scenarios: one for verifying that the audit replay loop yields during excluded-table skip runs, and another for ensuring that interrupted bulk copies do not persist a resume cursor over undelivered ranges. The reviewer feedback focuses on improving the robustness of process startup within parallel promises to prevent orphaned processes on failure, and replacing CommonJS-specific globals with ESM-safe fallbacks.

[claude]

Reviewed; no blockers found.

[kriszyp]

🤖 This PR promotes two QA scenarios into the regular cluster suite: excluded-table audit replay responsiveness and interrupted bulk-copy convergence.
The bulk-copy test is a useful destructive restart/whole-keyset check, although it does not independently prove that a persisted resume cursor was honored.
The audit-replay test does not exercise the branch changed by #536 and would stay green if that fix were reverted, so this draft should not be marked ready yet.
Static syntax, Prettier, oxlint, and git diff --check pass; runtime tests were not rerun because this detached checkout has no built dist/.