Skip to content

feat(ck-engine): report chunk-level line spans for lexical search results#167

Open
ak2k wants to merge 1 commit into
BeaconBay:mainfrom
ak2k:ck-lexical-spans
Open

feat(ck-engine): report chunk-level line spans for lexical search results#167
ak2k wants to merge 1 commit into
BeaconBay:mainfrom
ak2k:ck-lexical-spans

Conversation

@ak2k

@ak2k ak2k commented Jul 5, 2026

Copy link
Copy Markdown

Lexical (BM25) hits previously spanned the whole file (line 1 to EOF), so a match deep in a large file reported line 1 with a first-lines preview. Results now locate the best-matching chunk — using the same chunker the semantic path uses — and report its real span and text, falling back to the whole-file span when no chunk contains a query term. Document granularity, scoring, and result order are unchanged (covered by a ranking-invariance test).

The refinement runs only after the top-k documents are selected. For each hit the stored content is chunked (language resolved as the indexer does: PDF check, else Language::from_path) and the winning chunk is the one with the most whole-token matches for the query terms. The terms are taken from the parsed tantivy query (its content-field terms), not the raw query string, so field prefixes, quoted phrases, and boolean operators — which tantivy strips before matching — don't leak into span selection. The chunk is tokenized the same way tantivy tokenizes (lowercase, split on non-alphanumeric) and matches are whole-token, so a short term like "in" doesn't tally hits inside larger words. Ties favor the earliest chunk. The tantivy document, its score, and top-k selection are untouched, so an existing index produces identical rankings.

The default lexical preview changes from the file's first three lines to the winning chunk's text (--full-section still returns the whole file). If you'd rather keep the shorter preview as the default, this is easy to gate.

Spans reflect the content the index was built from, so they are only as fresh as the index; the corpus fingerprint triggers a rebuild on ordinary edits, but a span can point at shifted lines after a change that preserves a file's mtime and size.

…ults

Lexical (BM25) hits previously spanned the whole file (line 1 to EOF), so a
match deep in a large file reported line 1 with a first-lines preview. Results
now locate the best-matching chunk — using the same chunker the semantic path
uses — and report its real span and text, falling back to the whole-file span
when no chunk contains a query term. Document granularity, scoring, and result
order are unchanged (covered by a ranking-invariance test).

The refinement runs only after the top-k documents are selected. For each hit
the stored content is chunked (language resolved as the indexer does: PDF check,
else Language::from_path) and the winning chunk is the one with the most
whole-token matches for the query terms. The terms are taken from the parsed
tantivy query (its content-field terms), not the raw query string, so field
prefixes, quoted phrases, and boolean operators — which tantivy strips before
matching — don't leak into span selection. The chunk is tokenized the same way
tantivy tokenizes (lowercase, split on non-alphanumeric) and matches are
whole-token, so a short term like "in" doesn't tally hits inside larger words.
Ties favor the earliest chunk. The tantivy document, its score, and top-k
selection are untouched, so an existing index produces identical rankings.

The default lexical preview changes from the file's first three lines to the
winning chunk's text (--full-section still returns the whole file). If the
shorter preview is preferred as the default, this is easy to gate.

Spans reflect the content the index was built from, so they are only as fresh as
the index; the corpus fingerprint triggers a rebuild on ordinary edits, but a
span can point at shifted lines after a change that preserves a file's mtime and
size.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant