feat(search): token-boundary matching and body-text tier [roadmap:v0.10.3]

[tcballard]

Summary

Implements rac/roadmaps/v0.10.x-guide/v0.10.3-search-quality.md.

Adds:

• Token-boundary search matching (ADR-037): matchable text and queries tokenize on non-alphanumeric boundaries and camelCase transitions; terms match tokens by equality or prefix; multi-term queries are AND. The word-boundary false-positive class is gone — lore no longer matches Explorer artifacts.
• Body-text search tier with snippets (ADR-038): the ranking ladder extends to identifier, title, path, section heading, body; heading/body matches carry pinned additive section and snippet fields so an agent can triage matches without retrieving each candidate.
• Revised guide-tool-surface design: the body-text open question is settled and the snippet field shape was pinned before any code landed.
• Re-pinned goldens, a named lore-vs-Explorer regression test, and a token-boundary test battery (+13 tests, 934 total).

Roadmap / ADR Trace

Roadmap:

• rac/roadmaps/v0.10.x-guide/v0.10.3-search-quality.md

Relevant ADRs:

• rac/decisions/adr-037-token-boundary-search-matching.md — the matching semantics
• rac/decisions/adr-038-body-text-search-tier.md — the body tier, snippets, and the permanent rejection of embeddings/fuzzy/scoring
• rac/decisions/adr-007-json-contract-stability.md — additive response rules
• rac/decisions/adr-031, adr-032, adr-033 — consumer boundary, statelessness, response budget

Design: rac/designs/guide-tool-surface.md (revised in this PR, first commit).

Scope

Included

• One Core matching engine in rac.services.resolve (tokenize, five-tier _match_entry, rewritten search_index) serving rac find and search_artifacts identically
• Body/heading text sourced from the corpus snapshot via a new Product.search_sections structure that preserves original heading and line text (the existing sections map is lossy — casefolded/joined — and untouched, since other consumers depend on it); no file re-reads, exactly one corpus walk per search_artifacts call (regression-pinned)
• rac find human output shows snippets indented under heading/body-matched rows
• Index/repository entry models grow search_sections internally; the index JSON contract (to_dict) is unchanged

Excluded

• Semantic, embedding, or RAG-based retrieval; stemming; synonyms; fuzzy matching; opaque scoring (ADR-038, permanent for Core)
• Search configuration flags, pagination, or per-call ranking options
• Tool description changes (pinned verbatim; unchanged)
• The measurement re-run and demo recording (human-only; the roadmap gates the recording on the re-run)

Product / Architecture Decisions

• This is a deliberate, announced behavior change to search semantics, not a bug fix: results differ from v0.10.0 wherever substring matching produced word-boundary hits. Goldens were re-pinned in the same change set and each diff reviewed as a product change (e.g. a fixture roadmap now matches markdown via its body line "Adopt Markdown everywhere", with snippet — intended body-tier behavior).
• Empty and punctuation-only queries now match nothing (they tokenize to zero terms). Previously the empty substring matched everything. One Explorer adapter test relied on a lone dot as an accidental match-all (every path contains .md); it was updated to use a real token — the wildcard was never a documented feature.
• Snippet fields appear only on heading/body matches and are absent (not null) on identifier/title/path matches, which remain byte-identical to v1 — the additive rule of ADR-007 applied strictly.
• Snippet determinism: first matching line in document order, whole stored lines only; snippets ride inside match entries so the existing whole-item truncation covers them without new budget logic.
• Resolution (rac resolve, get_artifact, get_related) is untouched: alias matching stays exact; only search matching changed.

User-Facing Contract

CLI

rac find interface unchanged. Human output gains an indented ↳ Section: line row under heading/body matches.

JSON Output

Search match entries (CLI --json and search_artifacts, identical):

{"id": "...", "type": "...", "title": "...", "path": "...", "section": "...", "snippet": "..."}

section/snippet present only on heading/body matches. All other shapes unchanged.

Exit Codes

Unchanged.

Verification

Ran

python -m pytest -q                                   # 934 passed (+13)
python -m ruff check src/ tests/                      # clean
python -m ruff format --check src/ tests/             # clean
python -m mypy src/                                   # no issues, 64 files
rac validate rac/                                     # exit 0
rac relationships rac/ --validate                     # exit 0
rac review rac/                                       # no priority 1-2 findings
rac find lore rac/                                    # only Lore artifacts, zero Explorer
rac find "delete user" examples/guide                 # demo decision still top match

Plus a live stdio MCP client: search_artifacts("lore") returned exactly the five Lore-related artifacts (title matches without snippets, body matches with), byte-equal to rac find lore --json.

Covered

• Named regression: lore excludes Explorer artifacts on the dogfood corpus
• Tokenization boundaries: prefix matching, camelCase splits, multi-term AND, casefolding
• Tier ordering across all five tiers; body-only matches found with snippets; snippet truncation under small budgets
• CLI/Guide byte-equivalence for identical state and query
• Dogfood demo searchability tests pass UNCHANGED (delete user, delete, soft-delete still surface the demo decision — soft-delete tokenizes to [soft, delete] under AND)
• One corpus walk per search_artifacts call; get_related's one-walk regression still passes

Review Path

1. rac/designs/guide-tool-surface.md — the revised search contract (read first; the code implements this)
2. src/rac/services/resolve.py — tokenizer, tiers, snippet capture
3. src/rac/core/models.py, core/markdown.py, services/index.py, services/repository.py — search_sections plumbing
4. src/rac/output/human.py — snippet display
5. tests/test_resolve.py, test_dogfood.py, test_mcp_tools.py, tests/golden/ — the re-pinned contracts

Notes For Reviewer

• The golden diffs are the semantics change made visible — review them as product behavior, not noise.
• Release notes for v0.10.3 should announce: token-boundary matching (results differ from substring), body-text matching with snippets, and empty/punctuation-only queries returning no matches.
• Per the roadmap, the 10-run grounding measurement re-runs after this merges, and the demo recording happens after that gate passes.

Implementation Process

Implemented with AI assistance under the roadmap contract. Final scope, review, and acceptance decisions were made by the maintainer.