Fix/attestation last attestation time race - #242
Merged
JamesEjembi merged 6 commits intoAug 28, 2026
Merged
Conversation
added 6 commits
August 28, 2026 07:55
… pipeline Adds the tables the batch processor needs: 'nodes' (holding the monotonic last_attestation_time watermark) and 'attestations' (the work queue that workers claim from). Partial index on pending rows keeps FOR UPDATE SKIP LOCKED scans cheap as the queue grows. Migration: src/database/migrations/014_attestation_pipeline.sql
…EATEST() Root cause: concurrent workers processing attestations for the same node write last_attestation_time in claim order, not timestamp order. A worker holding an older timestamp (T1) that commits after a worker holding a newer one (T3) overwrites the column back to T1, which can trigger a false 'stale node' liveness alert. Fix: AttestationStore.advanceLastAttestationTime() now runs UPDATE nodes SET last_attestation_time = GREATEST(last_attestation_time, $2) WHERE id = $1 so the column can only move forward regardless of write order. Postgres's GREATEST() ignores NULL args, so this is also safe on a node's first-ever attestation. Also adds assignWork() (SELECT ... FOR UPDATE SKIP LOCKED, the work- claim query workers use) and markProcessed()/ensureNode() helpers. File: src/attestation/store.ts (new, 89 lines) Key change: lines 55-71 (advanceLastAttestationTime, the GREATEST update)
Applies the +10 reputation reward (via the existing ReputationStore. applyRewardWithLock, reused as-is) and the monotonic timestamp advance from the same transaction the caller claimed the row in, then marks the attestation processed. File: src/attestation/worker.ts (new, 57 lines)
Runs a pool of workerCount (default 4) concurrent workers, each claiming one attestation at a time via assignWork() (FOR UPDATE SKIP LOCKED) inside its own transaction, until maxBatchSize (default 100) attestations are drained or the pending queue is empty. File: src/attestation/batchProcessor.ts (new, 63 lines)
findStaleNodes() flags nodes whose last_attestation_time is NULL or older than the 48h threshold. Depends on the monotonicity fix in AttestationStore, since a clobbered timestamp here is exactly what produces a false stale-node alert. File: src/monitoring/livenessChecker.ts (new, 40 lines)
Two tests (resolution item 5): 1. testOutOfOrderWritesStayMonotonic - directly reproduces the issue's exact scenario (writes commit T3, T2, T1 in that order) and asserts the column ends at T3, not T1. 2. testFourWorkersTenAttestationsSameNode - runs the real AttestationBatchProcessor with 4 workers over 10 attestations for one node (shuffled timestamps), asserting last_attestation_time == MAX(all timestamps), each attestation processed exactly once, and the reward applied exactly 10 times. Run: npx tsx tests/attestation/batchProcessor.test.ts (both pass) File: tests/attestation/batchProcessor.test.ts (new, 212 lines)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #213
Problem
The attestation batch processor uses a 4-worker pool to process attestations concurrently. When multiple attestations for the same node land in the same batch, workers process them in parallel and each writes
nodes.last_attestation_time. Because writes commit in claim order, not timestamp order, a worker holding an older attestation timestamp (T1) can commit after a worker holding a newer one (T3) — leaving the column at T1 even though T3 is more recent.Downstream,
livenessCheckerreadslast_attestation_timeto flag nodes as stale after 48h of silence. A clobbered timestamp can trigger a false "stale node" alert even though the node attested recently.This repo didn't yet have the attestation pipeline this bug was reported against (no
nodes/attestationstables, no batch processor, no liveness checker), so this PR builds the pipeline and applies the fix in the same pass, following the atomic-update convention already used insrc/reputation/store.ts.Fix
AttestationStore.advanceLastAttestationTime()(src/attestation/store.ts:55-71) updates the column with:GREATEST()ignoresNULLarguments (only returnsNULLif every argument isNULL), so this is also correct on a node's very first attestation. The column can now only move forward, regardless of which worker's transaction commits last.What's included
src/database/migrations/014_attestation_pipeline.sqlnodes+attestationstables, partial index on pending rowssrc/attestation/store.tsassignWork()(FOR UPDATE SKIP LOCKEDclaim query) + theGREATEST()fixsrc/attestation/worker.tsprocessOne()— applies the reputation reward and the monotonic timestamp advance in one transactionsrc/attestation/batchProcessor.tsAttestationBatchProcessor— 4-worker pool draining up to 100 attestations/batchsrc/monitoring/livenessChecker.tsfindStaleNodes()/isStale()— 48h liveness checktests/attestation/batchProcessor.test.tsTests
Two tests in
tests/attestation/batchProcessor.test.ts:testOutOfOrderWritesStayMonotonic— directly reproduces the bug scenario: writes commit in the order T3, T2, T1. Asserts the column ends at T3 (the max), not T1 (the last write in a naive implementation).testFourWorkersTenAttestationsSameNode— runs the realAttestationBatchProcessorwith 4 workers over 10 attestations for one node with shuffled timestamps. Assertslast_attestation_time == MAX(all timestamps), every attestation is processed exactly once, and the reputation reward is applied exactly 10 times.Run directly:
npx tsx tests/attestation/batchProcessor.test.tsVerified locally:
npx tsc --noEmit— 0 errors, project-widenpm test— full suite green, including the new fileNotes for reviewers
last_attestation_seqcounter) butGREATEST()alone satisfies the monotonicity invariant with the least contention, and matches the pattern already established insrc/reputation/store.ts.Database/PoolClient(no live Postgres in the dev environment used to prepare this PR) — a real-DB smoke test before merge is worth doing given that.