fix(auditdb): align locker semantics, split key namespaces, stop nesting retries - #2211
Open
AkramBitar wants to merge 1 commit into
Open
fix(auditdb): align locker semantics, split key namespaces, stop nesting retries#2211AkramBitar wants to merge 1 commit into
AkramBitar wants to merge 1 commit into
Conversation
AkramBitar
marked this pull request as draft
August 12, 2026 15:26
AkramBitar
force-pushed
the
fix-2040-auditdb-locker
branch
from
August 13, 2026 10:33
1f2d770 to
985d315
Compare
AkramBitar
marked this pull request as ready for review
August 13, 2026 10:34
AkramBitar
force-pushed
the
fix-2040-auditdb-locker
branch
from
August 13, 2026 10:37
985d315 to
c720bd7
Compare
Effi-S
force-pushed
the
fix-2040-auditdb-locker
branch
from
August 17, 2026 13:06
c720bd7 to
f69b34e
Compare
Effi-S
reviewed
Aug 17, 2026
Contributor
Author
|
Pushed
Two new conformance cases cover the shrinking and rejected-widening halves of the refresh contract, plus one that deliberately passes no caller deadline — which is what hid the missing budget. |
AkramBitar
force-pushed
the
fix-2040-auditdb-locker
branch
from
August 18, 2026 15:51
1309917 to
37f1eaf
Compare
Contributor
Author
|
@Effi-S could you please have additional look at this PR? |
AkramBitar
force-pushed
the
fix-2040-auditdb-locker
branch
from
August 18, 2026 16:22
37f1eaf to
5aced16
Compare
The auditor takes short-lived locks on the enrollment IDs a request touches, through a Locker chosen from configuration: memory for a single replica, postgres for a cluster. The two were not interchangeable, so the same auditor code was correct on one deployment and broken on the other. The Locker interface now documents the contract its implementations are held to — it had none, which is how they drifted — and locker/conformance_test.go exercises every expectation against both backends. The three findings in #2040: - AssertLocksHeld detects lost locks, not absent ones. An anchor holding nothing succeeds, so a request whose inputs and outputs yield no enrollment IDs can be appended under postgres too, as can an auditor that validates and appends without calling Audit (the dvp and nft views). - The memory locker keeps enrollment-ID semaphores and per-anchor bookkeeping in separate maps. One shared sync.Map, keyed by unconstrained strings of unrelated provenance, let an anchor equal to an enrollment ID return the other namespace's value type and panic on the assertion. - The auditor's retry no longer nests inside the locker's own waiting budget: ErrLockAcquireTimeout is final, so worst-case blocking for one audit is acquireDeadline rather than MaxRetries times it. The inner poll loop is exponential and jittered instead of a flat 100ms, which cuts round trips from hundreds to a few dozen and stops contending replicas retrying in lockstep. And the defects that aligning them surfaced: - A live anchor's EID set may shrink or stay the same, never grow. Deadlock freedom rests on taking shared IDs in one canonical order, and that order only covers the IDs of a single call, so an anchor that kept earlier locks while waiting for new ones held locks outside it: two anchors widening into each other's IDs waited on each other forever, and permanently, since the anchor's lock is held across the blocking acquire and so blocked the release that would have broken the cycle. Widening now fails with ErrLockSetWidened. - Postgres releases the leases a narrowing re-acquisition drops. Its upsert only inserted, and both AssertLocksHeld and the heartbeat require an exact row count per anchor, so each leftover row rejected the next write, killed the heartbeat, and then expired into another replica's hands. - unlockAnchor no longer decides eviction from an emptiness flag sampled before the anchor's lock was released, which let an anchor still holding permits be dropped and those permits be stranded for the process lifetime. - Each backend bounds its own waiting, so a caller that passes no deadline still gets an answer and a spent budget is reported as ErrLockAcquireTimeout. - Failure classification is based on whether an attempt actually lost a race for an ID, not on which context expired first. The underlying error is joined in, so a database outage is reported as itself; a caller's own cancellation on a free ID is not a conflict; a locker's own expired budget is retriable while the caller's context is live; and only ErrLockContention counts towards auditor_audit_lock_conflicts_total. - releaseAnchor detaches from the caller's context so a deferred release on an already-cancelled context still runs, and bounds itself so a stuck DELETE cannot outlive the deadline AcquireLocks promises. A failed re-acquisition no longer releases an anchor that already holds a live session. Fixes #2040 Signed-off-by: AkramBitar <akram@il.ibm.com>
AkramBitar
force-pushed
the
fix-2040-auditdb-locker
branch
from
August 18, 2026 16:32
5aced16 to
e7c992b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In one paragraph
The auditor briefly "locks" each person involved in a transaction, so that two auditors can never write the same person's records at the same time. There are two interchangeable implementations of that lock, chosen by configuration:
memoryfor a single node,postgresfor a cluster. They did not behave the same way, which meant the same auditor code was correct on one deployment and broken on the other. This PR makes them behave identically, and adds a test suite that runs the same expectations against both so they cannot drift apart again.Fixes #2040
What was broken
Audit.In short
Think of two security guards (memory and postgres) with the same job: "make sure nobody edits Alice's records at the same time." They each had their own rulebook, and the rulebooks disagreed in 3 ways — one would crash if Alice's ID matched a transaction ID, one would falsely say "you lost access" for empty transactions, and both would keep each other waiting 10x longer than needed. This PR gives them one shared rulebook and fixes all three inconsistencies.
What this PR changes
AssertLocksHeldreports locks that were lost, not locks that were never taken.ErrLockAcquireTimeout— the locker saying "I already spent the whole budget". Worst case for one audit is now about oneacquireDeadline, notmaxRetries × acquireDeadline, and a contended wait costs a few dozen round trips instead of hundreds.Lockerinterface, andlocker/conformance_test.goruns it against every backend. The interface had no docs at all, which is how the two implementations drifted in the first place.Testing
make checksandmake lint-auto-fixclean.go test -racegreen ontoken/services/storage/auditdb/...andtoken/services/auditor/..., with the Postgres cases running against a real container (not skipped).panic: semaphore: released more than held; the four new Postgres tests fail with the exact misclassifications and deletions described above; the conformance suite failsEmptyEnrollmentIDson Postgres only andContentionIsReportedon both once the masking deadline is corrected.