fix(postgres): incrementalize change-triggered vector reconciliation - #1207
Conversation
roborev: Combined Review (
|
roborev: Combined Review (
|
2dbe4d9 to
bebece9
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
4b87f30 to
b58379a
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
|
This readiness scan is unchanged from the base branch. The PR scopes the two reads behind the 17.1 MB/push ( It's cheap on its own: two index-assisted counts returning a couple of integers, no rows built in Go, nothing read from Postgres. It's not the cost this PR set out to cut, and sizing it fairly needs a profile. I can't just skip it for scoped pushes. A same-fingerprint rebuild clears the active generation's stamps in place, so a half-rebuilt session would look complete and we'd delete valid PG vectors. Doing it safely means a real scope-aware completeness check in |
f220d14 to
8f75e94
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
|
Both roborev findings addressed. The BIGSERIAL id-reuse one is real and fixed in 97b3dd0f: recreating the vector tables restarts the sequence, so a single-generation reset can re-register the same fingerprint under an id a watch process already memoized, slipping past the id check. Keyed the incarnation check on this machine's roborev is right on the coverage count too: it leaves per-event work O(archive) on scoped pushes even though the hash and state reads are now scoped. Folding that fix in here rather than deferring — the readiness count takes the same changed-session scope the hash read already carries, with the full count reserved for generation-wide reconciliation. It's safe for the same reason the scoped push is: writes and eviction are already bounded to that set. Pushing shortly. |
… the changed session
…ive generation fingerprint changes
…n, and skip promotion without a fingerprint memo
…gnal has no unused wrapper
… id so any recreation, by any machine, forces a generation-wide reconcile
…id, so a daemon predating the field cannot leave scoped pushes with a memo the vector phase refuses to promote
…record, since recreating the vector tables restarts the id sequence and can hand the recreated generation the memoized id
8a7bde0 to
467e881
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
Change-triggered
pg push --watchreconciliations can run every 30 to 40 seconds while one session is active, and the vector phase originally read state in proportion to the full generation. The deployment behind #1197 measured 6,034 pushes over 62.66 hours and about 17.1 MB received for one push, on the order of 100 GB on this PostgreSQL path. Those measurements come from @mudssky's issue report.This keeps the branch's changed-session vector scoping and generation-wide maintenance, but rebuilds the local vectors.db export contract underneath it. Change-triggered pushes still derive their candidates from the relational push and read PostgreSQL vector state only for those sessions, while startup, the interval floor, shutdown flush,
pg push --full, generation replacement, and recovery still reconcile the full generation. The local vector source now serves one vector phase through one SQLite read transaction. That snapshot establishes the active generation, the in-place-rebuild marker, and the scoped coverage before exposing hashes or documents, so a same-fingerprint rebuild can no longer be read as complete across separate queries and committed as a partial mirror.A scoped push still promotes itself to generation-wide reconciliation whenever the immutable PostgreSQL generation id changes, a prior vector phase errored, or the watch process has not yet completed one full reconciliation, so a freshly created or recreated generation is never left partially populated. A generation whose local build is mid-flight is refused with
ErrVectorSourceNotReadyand retried after the build settles rather than exported partially. The per-session PostgreSQL ownership and replacement transaction, conflict and deferral rules, watch scheduling, session push phase, schemas, and non-PostgreSQL backends keep their existing behavior. The traffic measurements and the separation between PostgreSQL watch traffic and SSH remote-sync traffic come from @mudssky's issue report.Closes #1197