Post-failover write visibility divergence: surviving node's writes appear in indexed queries (without @createdTime) but not in collection scans

[jcohen-hdb]

On a 2-node replicated cluster, in the window after one node dies (observed for 60+ seconds, ~1-in-5 reproduction under the cluster integration harness), the surviving node's own fresh writes intermittently enter a state where all of the following hold simultaneously:

• put() resolves successfully — 50+ consecutive writes at 1–5ms each, verified via trace logging of the writer;
• an indexed-attribute REST query (GET /Table/?attr=value) returns all of the new records — but with their @createdTime field unset;
• a plain collection scan (GET /Table/) remains frozen at pre-failure contents, across repeated polls and fresh requests, for the entire window.

Same node, same instant, two REST query plans, two different answers.

Reproduction

Minimal fixture: a component with a 1s-interval job inserting unique-id records ({ id: unique, node: server.hostname, firedAt: Date @createdTime }) via the scheduler from #1828; boot 2 clustered nodes, deploy, kill the node currently running the writer, observe reads on the survivor. A polling read using the plain scan + firedAt filtering times out ~1 run in 5; the indexed ?node= query sees the rows all along. The failover test being contributed to harper-pro alongside #1828 contains the dual-read-path diagnostics used to capture this (its assertion path has been switched to indexed-count polling so it gates only scheduler behavior).

Representative capture (survivor 127.0.0.4, leader killed at 16:07:53.657Z, snapshot 60s later):

plain scan (poll):   rows=53  newest firedAt=16:07:53.519Z  (pre-kill, frozen)
plain scan (fresh):  rows=54  newest firedAt=16:07:53.519Z  (still frozen)
indexed ?node=...:   rows=50  (the post-kill writes)  firedAt missing on all
writer trace log:    ~55 puts completed in 1-5ms across the same window

SQL was deliberately excluded from the evidence chain.

Hypothesis

Consistent with the documented "put's promise resolves before the commit callback runs" seam (resources/DESIGN.md): inserts reach the secondary index ahead of a commit/stamp stage that appears wedged on the dead peer during replication churn — @createdTime stamping and primary-scan visibility both ride the stage that stalls. Self-heals without intervention (subsequent runs usually pass).

Possibly related mechanisms worth checking during investigation: the committed-watermark pinning in HarperFast/rocksdb-js#668, the point-read/scan coherence class covered by #1760's single-node test, and the previously-fixed null-@createdTime in #329 (same symptom recurring via a different path). Context: #531 (documentation) covers within-transaction read-back semantics; this report is cross-request reads on a surviving node.

Impact

Any application reading its own writes on a surviving node after a peer failure sees stale collection scans for the duration of the window — including the #1828 scheduler's own run-state bookkeeping (its engine tolerates this via in-memory anchors and idempotency, added defensively during review).

🤖 Filed by Claude on behalf of @jcohen-hdb; discovered while validating #1828's failover behavior.