Description
A fold keeps its write authority after it has lost the advisory lock that was supposed to give it that authority, so two folds of the same repository can run at once and the older one can overwrite the newer one's result.
withRepositoryReconciliation reserves a coordination connection, takes the repository advisory lock on it, and then awaits work(). Nothing connects the loss of that connection's session to the callback: if the session goes away, PostgreSQL releases the advisory lock, but the fold inside the callback carries on, still holding the ordinary work pool and still able to write. The repository's identity write and its materialization carry no lease token and no lock-session identity — the lease token fences the queue row's bookkeeping only.
Once the lock is gone and the job's lease has lapsed, the next claimant acquires a fresh lock immediately. It is not waiting behind anything, so the sixty-second lock acquisition deadline never comes into play. Both folds then run, and whichever finishes last writes last.
Reproduced against a real PostgreSQL 17 container with the real migrations, the real PostgresFoldStore, the real reconcileRepository, the real worker and the real claim SQL. Only the GitHub responses were controlled, so that the two folds could be held and could return different names for the same immutable GitHub repository id. Session loss was injected deterministically with idle_session_timeout on the coordination client, and the lease was expired with a direct update:
ATOMIC {"claimed":1,"idle":3}
AFTER_SESSION_LOSS {"active":1,"locks":0}
OVERLAP {"active":2,"peak":2,"calls":2}
The first line is a healthy control: four concurrent claimNextReconciliationJob calls return exactly one claim and three idle results, so the atomic claim itself is sound. The second shows the original fold still active with zero advisory locks held for that repository. The third shows two folds of the same repository inside their GitHub calls simultaneously. The test then observes registered_repositories.owner_name for that repository holding the OLDER fold's answer after the newer fold had already written its own.
Expected Behavior
Losing the lock should cost the fold its ability to publish. A fold whose coordination session has gone away should not be able to write repository identity or materialization afterwards — either the write is fenced on the same ownership that the lock represents, or the loss of the session cancels the work it was protecting. Two folds of one repository should not be able to publish in an order nobody controls.
Reproduction Steps
- Enqueue and claim a reconciliation job for a repository, and let its fold suspend inside a GitHub call.
- Cause the coordination connection holding that repository's advisory lock to lose its session, for instance with
idle_session_timeout. Confirm with pg_locks that no advisory lock remains for that repository while the fold is still active.
- Expire the job's lease:
update repository_reconciliation_jobs set lease_expires_at = now() - interval '1 second' where repository_id = <id>.
- Deliver another worker tick. The job is reclaimed and a second fold of the same repository starts and runs concurrently with the first.
- Let the second fold complete its identity write, then let the first one complete its own. The first fold's older answer is what remains in
registered_repositories.owner_name.
Environment / Context
The responsible code is unchanged on main at c5dac9997e08fa6c8ac33c3f42a2c1de19c68f5f: src/lib/fold/postgres-store.ts (the coordination scope and the unfenced identity write), src/lib/fold/reconcile.ts (the whole fold inside the callback) and src/lib/fold/reconciliation-worker.ts (renewal stops on a lost lease, the fold does not).
What has changed is how few processes it takes to reach. Before PR 204 a reconciliation worker drained serially and dropped every poll tick while a drain was in flight, so the reclaiming worker had to be a different process — a second server, or the scripts/reconcile.ts CLI. PR 204 lets one worker fold up to four repositories at once, so a single worker process can now be both the stranded fold and the reclaimer. The defect is not introduced there and the writes were never fenced, but the exposure is wider.
Production does not set idle_session_timeout, so the session loss has to come from a network or server-side failure rather than from an idle timer. The frequency of that was not measured.
Discovered During
PR 204, fixing issue 202, by an adversarial review of that branch's production-safety claim. Reproduced independently before filing.
Suggested Fix
Unverified. Fencing the writes on the same ownership the lock represents — carrying the lease token or a lock-session identity into recordVerifiedRepositoryIdentity and the materialization, and rejecting a write whose ownership has lapsed — would stop the stale publication without needing to cancel anything. Wiring the coordination connection's onclose to abort the fold would additionally stop the wasted GitHub work, at the cost of threading cancellation through reconcileRepository.
Separately, runNextReconciliationJob ignores the boolean returned by the completion and retry calls, so a job whose row it no longer owns still reports a successful outcome. That is not required for this defect but is the same missing check.
Description
A fold keeps its write authority after it has lost the advisory lock that was supposed to give it that authority, so two folds of the same repository can run at once and the older one can overwrite the newer one's result.
withRepositoryReconciliationreserves a coordination connection, takes the repository advisory lock on it, and then awaitswork(). Nothing connects the loss of that connection's session to the callback: if the session goes away, PostgreSQL releases the advisory lock, but the fold inside the callback carries on, still holding the ordinary work pool and still able to write. The repository's identity write and its materialization carry no lease token and no lock-session identity — the lease token fences the queue row's bookkeeping only.Once the lock is gone and the job's lease has lapsed, the next claimant acquires a fresh lock immediately. It is not waiting behind anything, so the sixty-second lock acquisition deadline never comes into play. Both folds then run, and whichever finishes last writes last.
Reproduced against a real PostgreSQL 17 container with the real migrations, the real
PostgresFoldStore, the realreconcileRepository, the real worker and the real claim SQL. Only the GitHub responses were controlled, so that the two folds could be held and could return different names for the same immutable GitHub repository id. Session loss was injected deterministically withidle_session_timeouton the coordination client, and the lease was expired with a direct update:The first line is a healthy control: four concurrent
claimNextReconciliationJobcalls return exactly one claim and three idle results, so the atomic claim itself is sound. The second shows the original fold still active with zero advisory locks held for that repository. The third shows two folds of the same repository inside their GitHub calls simultaneously. The test then observesregistered_repositories.owner_namefor that repository holding the OLDER fold's answer after the newer fold had already written its own.Expected Behavior
Losing the lock should cost the fold its ability to publish. A fold whose coordination session has gone away should not be able to write repository identity or materialization afterwards — either the write is fenced on the same ownership that the lock represents, or the loss of the session cancels the work it was protecting. Two folds of one repository should not be able to publish in an order nobody controls.
Reproduction Steps
idle_session_timeout. Confirm withpg_locksthat no advisory lock remains for that repository while the fold is still active.update repository_reconciliation_jobs set lease_expires_at = now() - interval '1 second' where repository_id = <id>.registered_repositories.owner_name.Environment / Context
The responsible code is unchanged on
mainatc5dac9997e08fa6c8ac33c3f42a2c1de19c68f5f:src/lib/fold/postgres-store.ts(the coordination scope and the unfenced identity write),src/lib/fold/reconcile.ts(the whole fold inside the callback) andsrc/lib/fold/reconciliation-worker.ts(renewal stops on a lost lease, the fold does not).What has changed is how few processes it takes to reach. Before PR 204 a reconciliation worker drained serially and dropped every poll tick while a drain was in flight, so the reclaiming worker had to be a different process — a second server, or the
scripts/reconcile.tsCLI. PR 204 lets one worker fold up to four repositories at once, so a single worker process can now be both the stranded fold and the reclaimer. The defect is not introduced there and the writes were never fenced, but the exposure is wider.Production does not set
idle_session_timeout, so the session loss has to come from a network or server-side failure rather than from an idle timer. The frequency of that was not measured.Discovered During
PR 204, fixing issue 202, by an adversarial review of that branch's production-safety claim. Reproduced independently before filing.
Suggested Fix
Unverified. Fencing the writes on the same ownership the lock represents — carrying the lease token or a lock-session identity into
recordVerifiedRepositoryIdentityand the materialization, and rejecting a write whose ownership has lapsed — would stop the stale publication without needing to cancel anything. Wiring the coordination connection'soncloseto abort the fold would additionally stop the wasted GitHub work, at the cost of threading cancellation throughreconcileRepository.Separately,
runNextReconciliationJobignores the boolean returned by the completion and retry calls, so a job whose row it no longer owns still reports a successful outcome. That is not required for this defect but is the same missing check.