Skip to content

fix(replication): prevent out-of-order full update from reverting newer record - #1172

Merged
kriszyp merged 1 commit into
mainfrom
kris/crdt-ooo-revert
Jun 10, 2026
Merged

fix(replication): prevent out-of-order full update from reverting newer record#1172
kriszyp merged 1 commit into
mainfrom
kris/crdt-ooo-revert

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 8, 2026

Copy link
Copy Markdown
Member

Summary

In the out-of-order resequencing path in resources/Table.ts (precedesExisting <= 0), when audit is enabled and the audit walk finds no succeeding updates to resequence around, a full update fell through to the shared commit line and stored the older record over the newer one (recordToStore = recordUpdate) — leaving the cluster permanently non-convergent.

This skips the losing full update instead: the existing record is newer, so the older update is superseded. It uses a bare return (no writeCommit) to match the existing superseded-by-newer-put branch a few lines above, so no audit record is written referencing this losing update's pre-saved blobs.

Purpose

Fixes #1170. Reproduced via deployment tracking: a burst of same-key full puts replicates to a loaded peer, commits out of order, and the row gets stuck at a non-terminal status forever (origin + other peers converge; one peer does not). Surfaced as the intermittent harper-pro deployTrackingReplication.test.mjs failure on main.

Where to look — please scrutinize

This is a hot, critical replication path; it deserves careful core review:

  • Blob lifecycle (the subtle bit). Codex flagged that write.skipped = true + writeCommit(false) with audit on would write an audit entry referencing the losing update's blobs and then clean them up (orphaning them). The fix avoids this by using a bare return — matching the existing put/delete supersede branch at the top of this block. Please confirm this is the correct discard path and that no blob/audit bookkeeping is skipped that the existing supersede branch relies on.
  • Reachability. The guard triggers only when fullUpdate && !incrementalUpdateToApply at the end of the audit branch — i.e. an out-of-order full update with no reconstructable succeeding updates. Worth confirming there is no legitimate case where we should still write here.

Open items for the reviewer

  • No local regression test yet. This needs a test that drives out-of-order same-key full puts and asserts the newest survives; I could not run the resources/replication suites locally (no native DB build in this environment). Guidance on the right harness (or a follow-up commit) welcome.
  • Verified to compile + lint; not yet run against the replication suite locally — relying on CI.

Cross-model review: Codex (raised the blob concern above; addressed).

🤖 Generated by Claude (Opus 4.x) per the Harper engineering DLC.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@claude

claude Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp
kriszyp marked this pull request as ready for review June 9, 2026 03:49
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@kriszyp
kriszyp force-pushed the kris/crdt-ooo-revert branch from 2f5eecf to e534eca Compare June 9, 2026 23:00
…r record

In the out-of-order resequencing path (precedesExisting <= 0), when audit is on
and the audit walk finds no succeeding updates to resequence around, a full
update fell through to the shared commit and stored the older record over the
newer one (recordToStore = recordUpdate), leaving the cluster non-convergent.

Skip the losing full update instead — the existing record is newer, so this
older update is superseded. Uses a bare return (no writeCommit) to match the
superseded-by-newer-put branch above, so no audit record is written referencing
this losing update's pre-saved blobs.

Reproduced via deployment-tracking: a burst of same-key full puts commits out of
order on a loaded peer and the row gets stuck at a non-terminal status.

Fixes #1170

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the kris/crdt-ooo-revert branch from e534eca to b8d9558 Compare June 9, 2026 23:21
@kriszyp
kriszyp merged commit 976adba into main Jun 10, 2026
37 checks passed
@kriszyp
kriszyp deleted the kris/crdt-ooo-revert branch June 10, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replication: out-of-order full update can revert a newer record (non-convergence)

2 participants