perf(retrieval): trust unchanged indexes instead of rebuilding and rescanning

[HaD0Yun]

Reference implementation for #1406. Please discuss the direction there, not here.

This is a draft because the open questions are decisions for maintainers, not incomplete work: the exit-code semantics change, and the invalidation guard is a new convention this repo would have to maintain. #1406 states both without requiring anyone to read 4,700 lines. This PR exists to show the direction is feasible and measured, and to be ready if the answer is yes.

Includes #1402 as its first commit. Cross-fork PRs cannot target another fork's branch as a base, so the diff here contains both. Review #1402 first — it is independent and small. If it lands, this drops by that much.

The problem

Two hot paths did redundant work.

Refresh rebuilt an unchanged index. A refresh rewrote the entire BM25 index and re-read every mirror even when nothing on disk had changed. BM25 was 61% of that refresh, all redundant.

Every query rescanned the corpus. The fallback engine re-read and re-tokenized the whole index per query, then walked every chunk to compute document frequencies. Cost was driven by corpus size alone — a query matching 2 documents cost the same as one matching all of them, because the scan was linear rather than an inverted lookup.

	main	this branch
No-op refresh, 2,000 docs	78 ms	44 ms	1.8x
Selective query, 1,200 chunks	19.0 ms	0.021 ms
Selective query cost when corpus grows 8x	8.5x	1.0x

Why these land together

Both are fixed by the same piece of state: a fingerprint sidecar committed after the artifact. Refresh compares it to decide whether to rebuild; the query cache uses it as its cross-process invalidation key. Splitting them would mean introducing the same mechanism twice.

Why there is a lock in a performance PR

Skipping work based on a stored fingerprint is only sound if the artifact and its fingerprint cannot interleave across processes.

The CLI builds a fresh agent per command, so autorag watch in one terminal and autorag refresh in another are two OS processes over one index directory. Without an on-disk lock they can commit a fingerprint describing a newer corpus than the artifact it points at — and every later refresh then trusts it. Permanently.

The lock wraps the existing acquireFileLock primitive (same one memory.ts and subagents/runtime.ts use) and lives at .autorag/refresh.lock, outside every directory index reset removes. index reset and index rebuild take the same lock.

Behavior changes — please push back if you'd prefer otherwise

situation	before	after
autorag refresh refused because another holds the lock	exit 0	exit 1, ok:false, outcome:"busy"
index reset / index rebuild refused	exit 0	exit 1, nothing removed
non-directory file at .autorag/refresh.lock	exit 0, silently did nothing	explicit error

These were all cases where the tool reported success while doing nothing. Anything checking only the exit code read a refused refresh as a successful one. This can break existing set -e CI or cron that treated contention as success; conversely, exit-code-based retry can now detect contention and back off.

watch is unaffected — busy is normal backpressure there and ticks still exit 0.

The part I'd most like your opinion on

Because the fingerprint decides whether a stored artifact is trusted, forgetting to bump INDEX_SEMANTICS_VERSION after changing chunking or serialization would silently serve stale results. So this adds source-region markers around the code that shapes artifact meaning, and a test that digests those regions and fails until the version is bumped or the recorded digest is refreshed deliberately.

It works — adversarial testing broke the guard three times during development (hash(), normalizeMarkdown(), and computeFingerprint() itself were each outside the guarded set at some point, and each let a real semantic change through while the whole bench suite stayed green). But:

• it is a new convention in your codebase, and you'll be the ones maintaining it
• it is a manual allowlist — it does not discover new files or helpers on its own
• pure refactors inside a marked region also trip it, so the recorded digest needs refreshing

If you'd rather not carry that, the alternative is documenting "bump the version when you change these functions" and accepting the risk. I have no attachment to my version.

Query-time code (tokenize, bm25Score, k1, b) is deliberately outside the regions. Changing the tokenizer changes results immediately and correctly against the stored artifact — a test mutates it and asserts the artifact mtime stays unchanged while results change. Guarding it would force needless global rebuilds.

Known limitations

• index rebuild is concurrency-atomic but not rollback-atomic. Delete and rebuild are covered by one lock, so nothing can interleave. But if building fails after the delete, the indexes stay removed and you must rerun.
• artifactBytes only checks size. A same-length tampered artifact passes. Truncation detection, not integrity verification.
• auto engine sticks to fallback after one tantivy failure, since the engine is part of the fingerprint. Results stay correct but quality silently stays lower until index rebuild.
• Cache retention is an empirical budget, not a hard cap. It estimates bytes pinned by the fallback structure; it does not count allocator overhead. The tantivy handle is one per method instance and is not size-bounded.
• Without a fingerprint sidecar the query cache is not held at all — prepared state is rebuilt per query. Deliberate (no cross-process invalidation key exists then), fixed by one normal refresh, but slow until then.
• Parser and dependency upgrades still need --force. An unchanged source with the same mtime/size is not re-mirrored, so a new parser or a chardet/iconv-lite upgrade does not re-parse existing documents. Pre-existing, but worth stating since the fingerprint now depends on mirror content.
• Stale lock reclaim cannot distinguish PID reuse; mkdir atomicity is unverified on NFS.

Verification

biome check (252 files, 0 warnings), tsc --noEmit, and bun run build pass. CI passed on the combined branch before I split it.

Test count 1,249 → 1,365. New coverage: fingerprint skip, artifact integrity, the semantics guard, serial mirror-sync contracts, query cache equivalence across both engines and every scope, cache bounds, lock behavior in-process and across real OS processes, and index reset participation in the lock.

Local vitest run shows 3 failures: two need a Java runtime this machine lacks (PDF parser), one is a load-dependent flake that passes 10/10 in isolation. That last category is not introduced here — running vitest run on unmodified main on this machine produced more failures (12 files / 15 tests) than this branch (10 / 13) in the same session. vitest.config.ts sets no pool or concurrency limits, so wall-clock-budgeted tests compete under load. Happy to open that separately.

Not included

An earlier revision parallelized mirror parsing. I reverted it before opening this: measured gain was 1.01–1.10x on markdown (parsing is dominated by synchronous CPU, not I/O wait), and adversarial testing found a deadlock on a zero concurrency limit, unbounded fan-out on NaN, and a checkpoint/disk digest mismatch that could serve stale results indefinitely. It needs a deterministic commit-prefix design and real PDF/Office measurements, so it belongs in its own change.

Questions

1. Are the exit-code changes acceptable, or would you prefer a distinct code, or a flag to opt in?
2. Do you want the source-region guard, or should this just document the version-bump requirement?
3. Should this be split further? The fingerprint and the query cache share the sidecar, so splitting means introducing it twice — but I'll do it if you'd rather review them apart.