cloneNode: checkSyncStatus passes mid-copy — clone marks itself complete and Available with partial data

[heskew]

Summary

During clone bootstrap, monitorSync()'s checkSyncStatus (cloneNode/cloneNode.ts:523) can return true while the initial full copy is still streaming. Observed on harperfast/harper-pro:5.1.14 (2-node docker rig, no Fabric):

• Follower logged All databases synchronized → Clone from leader node … complete, set cloned: true, and published availability: Available —
• while holding 56,071 of 500,000 records (leader count verified 500,000).
• Replication then died (the credential's user had been removed on the leader as part of the test), so the node stayed Available with ~11% of the data indefinitely. get_status shows every component healthy; nothing retries because cloned: true is durable.

The #279 fix (always-populated availability, timeout-not-success failure branch) is present in 5.1.14 but unreachable when the check false-positives — the gate exists, the comparison lies.

Repro (laptop, docker, no Fabric)

1. Leader seeded with 500k small records (data.items); follower started in clone mode (HDB_LEADER_URL + HDB_LEADER_TOKEN), cpu-capped to slow the copy.
2. Right after Starting to monitor sync status: delete the token's user on the leader (drop_user) and bounce the follower's network for 5s (kills the copy stream and any re-auth).
3. An early poll returns synchronized → clone declares complete → counts diverge as above.

Sequence in the failing run: monitor start → leader-side user delete → 5s network bounce → on reconnect the check passed immediately.

(Same rig also reproduces #579's shape deterministically: remove the user before the follower's first request → Error: Leader request failed: 403 Forbidden at leaderRequest → exit. Internal rig: harper-fabric-lab incidents/clone-admin-delete-2026-07-24/, run.sh, scenarios A/B.)

Where to look

• checkSyncStatus compares the per-db socket's lastReceivedVersion against targetTimestamps captured once up front from describe (getLastUpdatedRecord → max last_updated_record). Nothing consults full-copy progress (the copy/copyCursor machinery from Clone bulk table-copy restarts from zero after ping-timeout disconnect (no mid-copy checkpoint) #241/fix(replication): resumable bulk clone copy (no restart-from-zero) #255) — so any frame that advances lastReceivedVersion to ≥ target (e.g. a handshake/current-head version on reconnect, or copy frames carrying head versions) satisfies the check mid-copy.
• A sync-complete check for a clone probably needs to gate on "full copy finished" per database (copy cursor drained), not just last-received version ≥ a stale describe timestamp.

Impact

Converts "clone stalled" (visible: node stuck Unavailable, #579) into a silent partial clone that reports Available — reads served from a fraction of the dataset, and the gap never closes if replication auth is dead. Strictly worse than the hang for anyone routing traffic by availability.