perf: cut index and query hot paths (ASCII decode, fingerprint skip, query cache)

[HaD0Yun]

What

Three hot-path optimizations, measured against this branch's merge base with a separate checkout of main on the same machine.

Corpus	main cold refresh	this branch
200 ASCII docs	175 ms	42 ms	4.2x
200 Korean docs	199 ms	200 ms	1.0x — no gain

Workload	main	this branch
No-op refresh, 2,000 docs	78 ms	44 ms	1.8x
Selective query, 1,200 chunks	19.0 ms	0.021 ms

The Korean row is the honest headline caveat: the parser fast path only applies to pure-ASCII sources, so a Korean-document corpus sees no cold-refresh gain. It still benefits from the index and query work (no-op refresh 2.0x). Given that this project targets Korean documents, please weigh that when scoping.

Commits

Three independent changes, split so they can be reviewed or landed separately. Happy to split into separate PRs on request — the natural boundary is parser alone, and fingerprint skip + lock before query cache (the cache uses the fingerprint sidecar as its invalidation key).

1. perf(parser) — skip chardet.detect() for pure-ASCII buffers. Independent of the other two.
2. perf(retrieval) fingerprint skip + cross-process lock — a no-op refresh no longer rebuilds the BM25 index. The lock is not optional here: skipping is only sound if artifact and fingerprint commits cannot interleave across processes.
3. perf(retrieval) query cache — the fallback engine no longer re-tokenizes the corpus and linearly scans every chunk per query.

Behavior changes

These are deliberate and observable. Please push back if you'd prefer different semantics.

• autorag refresh refused because another refresh holds the lock now exits 1 (ok:false, outcome:"busy") instead of exiting 0. Previously a refused refresh looked successful to anything checking only the exit code. This can break existing set -e CI or cron that treated contention as success; conversely, exit-code-based retry can now detect and back off.
• index reset / index rebuild exit 1 when refused for the same reason, and reset removes nothing in that case.
• A non-directory file at .autorag/refresh.lock is now an explicit error, not a "busy" result. Previously it silently made refresh and reset do nothing.
• watch is unaffected — busy is normal backpressure there and ticks still exit 0.

Correctness

The parser change is the one that touches stored content, so it is argued rather than sampled: for a pure-ASCII buffer the set of encodings chardet can return is enumerable, and every member decodes 0x01-0x7F identically. Sampling backs it up — a test keeps a copy of the previous implementation and compares both across boundary lengths, boundary byte values, BOM, NUL bytes, combining characters and pseudo-random buffers.

Because the fingerprint decides whether a stored artifact is trusted, the code that determines artifact meaning is pinned by source-region markers that a test digests: chunking, chunk id hashing, fallback persistence schema, tantivy schema construction, the fingerprint material itself, and mirror normalization. Changing any of them fails until INDEX_SEMANTICS_VERSION is bumped or the recorded digest is refreshed deliberately.

This guard is a manual allowlist. It does not discover new files or helpers on its own. If a future change puts artifact-shaping logic outside the marked regions, the guard will not notice.

Query-time code (tokenize, bm25Score, k1, b) is intentionally outside those regions. Changing the tokenizer changes query results immediately and correctly against the stored artifact — verified by a test that mutates it and observes the artifact mtime stay unchanged while results change. Guarding it would force needless global rebuilds.

Known limitations

• Korean corpora get no cold-refresh gain (above). Widening the fast path to "any valid UTF-8" is not safe: the current code follows chardet even for valid UTF-8, so short or ambiguous inputs could decode differently. Doing it would be a semantics change requiring a version bump and migration.
• index rebuild is concurrency-atomic but not rollback-atomic. Delete and rebuild are covered by one lock, so no other refresh can interleave. But if parsing or building fails after the delete, the indexes stay removed and you must rerun.
• artifactBytes only checks size. A same-length tampered artifact passes. It is truncation detection, not integrity verification.
• auto engine sticks to fallback after one tantivy failure, because the engine is part of the fingerprint. Results stay correct but quality silently stays lower until index rebuild.
• Cache retention is an empirical budget, not a hard cap. It estimates bytes pinned by the fallback structure; it does not count allocator overhead and does not cover native tantivy handles.
• Without a fingerprint sidecar the query cache is not held at all — prepared state is rebuilt per query. That is deliberate (there is no cross-process invalidation key), and one normal refresh fixes it, but querying a large sidecar-less corpus is slow until then.
• Parser and dependency upgrades still need --force. An unchanged source with the same mtime/size is not re-mirrored, so a new parser or a chardet/iconv-lite upgrade does not re-parse existing documents.
• Stale lock reclaim cannot distinguish PID reuse; mkdir atomicity is unverified on NFS.

Verification

biome check (252 files, 0 warnings), tsc --noEmit, and bun run build all pass.

vitest run reports 3 failures out of 1,365:

File	Cause
test/parser/parser.test.ts	no Java runtime on this machine (PDF parser)
test/agent/parser-mirror.test.ts	same
test/jikji/installer.test.ts	load-dependent flake; passes 10/10 in isolation

That last one is worth flagging separately: the suite is load-dependent regardless of this branch. Running vitest run on the unmodified merge base on this machine produced more failures (12 files / 15 tests) than this branch (10 files / 13 tests) in the same session. vitest.config.ts sets no pool or concurrency limits, so wall-clock-budgeted tests compete under load. Happy to open that as a separate issue.

Test count 1,249 → 1,365. New tests cover fingerprint skip, artifact integrity, semantics guard, mirror sync contracts, query cache equivalence and bounds, lock behavior in-process and across real OS processes, and decode equivalence.

Not included

An earlier revision parallelized mirror parsing. It was reverted before this PR: measured gain was 1.01–1.10x on markdown (parsing is dominated by synchronous CPU, not I/O wait), and adversarial testing found a deadlock on a zero concurrency limit, unbounded fan-out on NaN, and a checkpoint/disk digest mismatch that could serve stale results indefinitely. It needs a deterministic commit-prefix design and real PDF/Office corpus measurements, so it belongs in its own change.

[HaD0Yun]

Closing in favour of a cleaner split — this was one 4,700-line PR from a stranger touching four separate concerns, which is not a fair thing to hand a reviewer.

• perf(parser): skip encoding detection for pure ASCII sources (4.2x cold refresh) #1402 — the ASCII decode fast path. Fully independent, 28 lines of source, 4.2x cold refresh on ASCII corpora, equivalence argued from chardet's candidate set rather than only sampled. Ready for review.
• perf(retrieval): trust unchanged indexes instead of rebuilding and rescanning #1403 — fingerprint skip, cross-process lock and query cache. These share a sidecar so they cannot be split further without introducing the mechanism twice. Draft, because it changes CLI exit-code semantics and adds a new convention to the codebase — both worth agreeing on before a real review.

No code was dropped; the same three commits are redistributed. Sorry for the noise.