Skip to content

fix: make gen-context recency boost deterministic (#440)#444

Merged
manojmallick merged 1 commit into
developfrom
fix/gen-context-determinism-residual-440
Jul 5, 2026
Merged

fix: make gen-context recency boost deterministic (#440)#444
manojmallick merged 1 commit into
developfrom
fix/gen-context-determinism-residual-440

Conversation

@manojmallick

Copy link
Copy Markdown
Owner

Problem

gen-context output was non-byte-stable run-to-run for ~5 large repos (fastapi, laravel, serilog, vapor, svelte), making the retrieval benchmark's hit@5 non-reproducible (86.7 → 85.6 → 87.8% over three unchanged runs). This is the residual tracked in #440 after the first pass (#441).

Root cause

The token-budget recency boost stamped mtime = Date.now() on every recently-committed file. On repos where nearly every file is "recently changed" (svelte: 8743 recent files), consecutive files often landed on the same millisecond. Which files shared a millisecond — and thus fell through to the filePath tie-break instead of sorting by a distinct mtime — shifted run to run, swapping which equal-priority files survived at the budget cutoff.

The Date.now() value wasn't just a boost: it encoded the (alphabetical) walk order as a monotonic mtime that bestFirst (mtime desc) relied on. The nondeterminism was only the millisecond collisions.

Fix

Replace Date.now() with a deterministic monotonic counter (nextRecentMtime) that reproduces the exact same processing-order ranking without collisions. Recent files stay ahead of non-recent ones and keep their walk-order ordering among themselves.

  • An earlier attempt collapsing all recent files to one sentinel value regressed hit@5 to 63.3% (signalQuality dominated). The counter preserves the intended ordering.

Results

  • All 43 benchmark repos: byte-identical across two clean runs (excluding the Updated: timestamp comment).
  • Retrieval benchmark: identical per-repo results and 87.8% hit@5 across runs — quality unchanged, now reproducible. The published headline (87.8%) is no longer stale.

Acceptance criteria

  • gen-context output byte-identical (excluding Updated:) across two clean runs for all benchmark repos
  • Regression guard runs gen-context twice on a fixture and asserts byte-equality (test/integration/gen-context-determinism.test.js) — verified it fails on the old Date.now() behaviour
  • Retrieval benchmark re-baselined to a single reproducible headline: 87.8% (weighted = unweighted over 18 repos)
  • (Optional) omittable Updated: timestamp — deferred; touches 4 bundled-adapter sites + their source copies, disproportionate for an optional item. Guard already handles it by excluding the line.

Tests

  • New: test/integration/gen-context-determinism.test.js (2 assertions)
  • Full integration suite: 105 passed, 0 failed
  • version.json derived test count 110 → 111

Supply-chain gate

Local audit PASS — no shell spawns · no install scripts · main exports API · no fingerprinting. Change is limited to an mtime assignment in gen-context.js.

🤖 Generated with Claude Code

The token-budget recency boost stamped `mtime = Date.now()` on every
recently-committed file. On repos where nearly every file is "recently
changed", consecutive files often landed on the same millisecond; which
files shared a millisecond (and so fell through to the filePath tie-break
instead of sorting by a distinct mtime) shifted run to run, swapping which
files survived at the budget cutoff. That made gen-context output non-byte-
stable and the retrieval benchmark's hit@5 non-reproducible (86.7/85.6/87.8
over three unchanged runs).

Replace Date.now() with a deterministic monotonic counter (nextRecentMtime)
that reproduces the exact same processing-order ranking without millisecond
collisions. Recent files stay ahead of non-recent ones and keep their
walk-order ordering among themselves — so retrieval quality is unchanged
(hit@5 holds at a now-reproducible 87.8%) while the output is byte-stable.

- all 43 benchmark repos: byte-identical across two clean runs (excluding
  the Updated: timestamp comment)
- retrieval benchmark: identical per-repo results and 87.8% hit@5 across runs
- add regression guard (gen-context-determinism.test.js): two clean runs on
  a committed fixture must be byte-identical; verified it fails on the old
  Date.now() behaviour
- version.json: derived test count 110 -> 111 (new guard)
@manojmallick manojmallick merged commit 41a04fc into develop Jul 5, 2026
4 checks passed
@manojmallick manojmallick deleted the fix/gen-context-determinism-residual-440 branch July 5, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant