Improve nav-to fuzzy pre-filtering with q-gram bigram bitset and corrected length thresholds

[CyrusNajmabadi]

Followup to #82431

Summary

The NavigateTo pre-filter gates whether fuzzy (edit-distance) matching is attempted for a document. Previously it used only a symbol-name-length bitset with a fixed ±2 delta — too coarse in practice. In a large file with many 6-character symbols, searching for "FooBar" (also 6 characters) would pass the length check for every document, forcing an expensive full-scan fuzzy match against every symbol in those documents even when the character content is completely different.

This PR improves the fuzzy pre-filter in three ways:

1. Correct the length check thresholds to match what WordSimilarityChecker actually enforces: ±1 for pattern lengths 3–4, ±2 for 5+, and no fuzzy matching at all for patterns shorter than 3. Previously, the pre-filter used a blanket ±2 for all lengths and didn't reject short patterns, which meant it was more permissive than the actual fuzzy matcher.

2. Add an exact bigram bitset (38×38 = 1444 bits, 184 bytes per document) storing all lowercased 2-character sliding windows of symbol names. At query time, use the q-gram count lemma (Ukkonen, 1992) to require a minimum number of shared bigrams: min_shared = |pattern| - 1 - 2k, where k is the edit distance threshold. If fewer pattern bigrams match, the document is skipped for fuzzy matching entirely.

3. Share threshold logic by extracting WordSimilarityChecker.GetThreshold(int) and MinFuzzyLength so the pre-filter and the actual fuzzy matcher use the same constants instead of duplicating magic numbers.

Motivation

Consider a codebase with thousands of documents, many containing 6-character symbol names. When the user types "FooBar" in NavigateTo:

• Before: The length check passes for every document that has any symbol of length 4–8 (±2). The caller then creates a PatternMatcher with fuzzy matching enabled and runs it against every symbol in those documents. Most of these comparisons compute edit distance only to conclude there's no match.

• After: The length check passes (length 6, threshold ±2, checks 4–8). But the bigram check then asks: do at least 1 of the 5 bigrams ("fo","oo","ob","ba","ar") exist in this document? For documents with symbols like "XyzWvq", none of these bigrams are stored, so the document is skipped without ever creating a PatternMatcher.

Bigram bitset design

Characters are mapped to a 38-element alphabet:

• a–z → 0..25
• 0–9 → 26..35
• _ → 36
• everything else (Unicode) → 37 ("other" bucket)

This gives exact membership for the 37 most common identifier characters and a single overflow bucket for rare Unicode characters. The bitset is a ulong[23] (184 bytes) — compact enough to store per-document with negligible memory overhead.

Filtering effectiveness by pattern length

Pattern length	k (threshold)	min_shared bigrams	Filtering power
< 3	—	—	Fuzzy disabled entirely
3	1	0	No filtering (always passes)
4	1	1	Need ≥ 1 of 3 bigrams
5	2	0	No filtering (always passes)
6	2	1	Need ≥ 1 of 5 bigrams
7	2	2	Need ≥ 2 of 6 bigrams
8	2	3	Need ≥ 3 of 7 bigrams
10	2	5	Need ≥ 5 of 9 bigrams

Lengths 3 and 5 get no bigram filtering benefit (min_shared = 0). For length 3, we should strongly consider whether fuzzy matching itself is too permissive to even support — with an edit distance threshold of 1, a 3-character pattern like "abc" would fuzzy-match "xbc", "axc", "abx", "ab", "abcd", etc. This is a potential follow-up.

Reference

Ukkonen, E. (1992). "Approximate string-matching with q-grams and maximal matches." Theoretical Computer Science, 92(1), 191–211. DOI: 10.1016/0304-3975(92)90143-4

The q-gram count lemma states: each edit operation can destroy at most q q-grams from a string. Therefore, if edit_distance(s, t) ≤ k, then at least |s| - 1 - q·k of s's q-gram positions must have a matching q-gram in t. For bigrams (q=2): min_shared = |pattern| - 1 - 2k.

Test plan

• Updated LengthCheck theory tests for correct thresholds (pattern < 3 → false, 3–4 → ±1, 5+ → ±2)
• 22-case BigramCountCheck theory covering lengths 3–10, single-edit scenarios, underscore, digits, and Unicode
• BigramCountCheck_MultipleSymbols — bigrams accumulate across all symbols in a document
• BigramCountCheck_RejectsSameLengthDifferentContent — the key scenario: same length but disjoint characters
• BigramCountCheck_UnicodeFallsBackToOtherBucket — Unicode chars share the "other" bucket (documented false positive)
• BigramCountCheck_UnderscoreHasOwnIndex — underscore is exact, not in the "other" bucket
• Updated all end-to-end tests for the new combined length + bigram check
• Separate NavigateToFuzzyPreFilterBenchmarks.cs with 8 benchmarks demonstrating length-only false positives vs. bigram true negatives
• Run benchmarks locally: cd src/Tools/IdeCoreBenchmarks && dotnet run -c Release -- --filter "NavigateToFuzzyPreFilterBenchmarks"

Benchmarks:

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
'SameLen: LengthCheck pass (false positive)'	0.9901 ns	0.1123 ns	0.0292 ns	-	-	-	-
'VariedLen: LengthCheck pass'	0.1914 ns	0.0710 ns	0.0184 ns	-	-	-	-
'SameLen: BigramCheck reject (true negative!)'	7.1073 ns	0.5302 ns	0.1377 ns	-	-	-	-
'SameLen: BigramCheck pass (true positive)'	7.1716 ns	0.4179 ns	0.0647 ns	-	-	-	-
'SameLen: Combined reject (bigram saves)'	28.2908 ns	2.1554 ns	0.3336 ns	-	-	-	-
'SameLen: Combined pass'	28.7838 ns	1.3179 ns	0.2039 ns	-	-	-	-
'SameLen: BigramCheck reject len=8'	10.4243 ns	1.4520 ns	0.2247 ns	-	-	-	-
'VariedLen: BigramCheck reject len=10'	14.9580 ns	0.7782 ns	0.1204 ns	-	-	-	-

[CyrusNajmabadi]

@ToddGrun @dibarbet this can be reviewed starting from commit 635cbc2.

[CyrusNajmabadi]

Future consideration: fuzzy pre-filter effectiveness for short patterns

The bigram pre-filter (q-gram count lemma) has blind spots for short patterns due to the relationship between pattern length, edit distance threshold k, and the minimum shared bigrams formula (|pattern| - 1 - 2k):

Current behavior (GetThreshold: k=1 for length ≤ 4, k=2 for length ≥ 5)

Pattern length	k	Length window	min_shared bigrams	Filtering power
3	1	±1 (2–4)	0	None — always passes
4	1	±1 (3–5)	1	Weak — need ≥1 of 3
5	2	±2 (3–7)	0	None — always passes
6	2	±2 (4–8)	1	Weak — need ≥1 of 5
7	2	±2 (5–9)	2	Moderate
8+	2	±2	3+	Good

Proposed: k=1 for length ≤ 6, k=2 for length ≥ 7

Pattern length	k	Length window	min_shared bigrams	Filtering power
3	1	±1 (2–4)	0	None — always passes
4	1	±1 (3–5)	1	Weak — need ≥1 of 3
5	1	±1 (4–6)	2	Moderate — need ≥2 of 4
6	1	±1 (5–7)	3	Strong — need ≥3 of 5
7	2	±2 (5–9)	2	Moderate
8+	2	±2	3+	Good

The key wins are at lengths 5 and 6, which go from zero/weak filtering to moderate/strong. The trade-off is less fuzzy tolerance (1 edit instead of 2) for 5–6 character patterns, but 2 edits on a 5-character string is 40% different — arguably too aggressive for useful fuzzy matching anyway.

Additionally, we should consider raising MinFuzzyLength from 3 to 4 (or even 5). A 3-letter fuzzy match with k=1 means "Foo" matches "Goo", "Boo", "For", "Fo", "Food", etc. — extremely permissive with zero bigram selectivity. Raising it would reduce noise with minimal loss of useful matches.

These are independent improvements and don't need to block this PR — just noting them for future work.

--

Note: i've implemented this change here: #82459

[davkean]

Be warned, this contributor is a known troll.

[davkean]

I like this.