fix: make gen-context recency boost deterministic (#440)#444
Merged
Conversation
The token-budget recency boost stamped `mtime = Date.now()` on every recently-committed file. On repos where nearly every file is "recently changed", consecutive files often landed on the same millisecond; which files shared a millisecond (and so fell through to the filePath tie-break instead of sorting by a distinct mtime) shifted run to run, swapping which files survived at the budget cutoff. That made gen-context output non-byte- stable and the retrieval benchmark's hit@5 non-reproducible (86.7/85.6/87.8 over three unchanged runs). Replace Date.now() with a deterministic monotonic counter (nextRecentMtime) that reproduces the exact same processing-order ranking without millisecond collisions. Recent files stay ahead of non-recent ones and keep their walk-order ordering among themselves — so retrieval quality is unchanged (hit@5 holds at a now-reproducible 87.8%) while the output is byte-stable. - all 43 benchmark repos: byte-identical across two clean runs (excluding the Updated: timestamp comment) - retrieval benchmark: identical per-repo results and 87.8% hit@5 across runs - add regression guard (gen-context-determinism.test.js): two clean runs on a committed fixture must be byte-identical; verified it fails on the old Date.now() behaviour - version.json: derived test count 110 -> 111 (new guard)
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
gen-contextoutput was non-byte-stable run-to-run for ~5 large repos (fastapi, laravel, serilog, vapor, svelte), making the retrieval benchmark's hit@5 non-reproducible (86.7 → 85.6 → 87.8% over three unchanged runs). This is the residual tracked in #440 after the first pass (#441).Root cause
The token-budget recency boost stamped
mtime = Date.now()on every recently-committed file. On repos where nearly every file is "recently changed" (svelte: 8743 recent files), consecutive files often landed on the same millisecond. Which files shared a millisecond — and thus fell through to thefilePathtie-break instead of sorting by a distinct mtime — shifted run to run, swapping which equal-priority files survived at the budget cutoff.The
Date.now()value wasn't just a boost: it encoded the (alphabetical) walk order as a monotonic mtime thatbestFirst(mtime desc) relied on. The nondeterminism was only the millisecond collisions.Fix
Replace
Date.now()with a deterministic monotonic counter (nextRecentMtime) that reproduces the exact same processing-order ranking without collisions. Recent files stay ahead of non-recent ones and keep their walk-order ordering among themselves.Results
Updated:timestamp comment).Acceptance criteria
gen-contextoutput byte-identical (excludingUpdated:) across two clean runs for all benchmark repostest/integration/gen-context-determinism.test.js) — verified it fails on the oldDate.now()behaviourUpdated:timestamp — deferred; touches 4 bundled-adapter sites + their source copies, disproportionate for an optional item. Guard already handles it by excluding the line.Tests
test/integration/gen-context-determinism.test.js(2 assertions)version.jsonderived test count 110 → 111Supply-chain gate
Local audit PASS — no shell spawns · no install scripts ·
mainexports API · no fingerprinting. Change is limited to an mtime assignment ingen-context.js.🤖 Generated with Claude Code