Commit cfa08a1
committed
feat(search): normalize semantic query (NFKC + casefold)
Semantic find/search queries are passed verbatim to the embedder and
retriever, so colloquial or CJK full/half-width variants of the same
intent do not match indexed content. Examples from real usage:
- "OpenViking" (full-width) vs "OpenViking"
- "OpenVAKING" (mis-spelled case) vs "openvaking"
- "Harms agent" (double space) vs "hermes agent"
Add _normalize_search_query (NFKC + casefold + whitespace collapse) at
the VikingFS.find/search entry points. NFKC folds full/half-width forms
to canonical; casefold handles Unicode case (stronger than str.lower());
whitespace collapse + strip removes accidental gaps. Idempotent and
only widens recall — already-normalized ASCII is unchanged.
Applied only to the semantic path. grep is intentionally left alone: its
pattern is a regular expression, and NFKC/casefold would corrupt explicit
character classes (e.g. [A-Z]); the existing case_insensitive flag covers
case folding there.
Verified at the viking_fs layer (not SearchService) because session/
memory/tools.py calls viking_fs.search directly, bypassing SearchService.
Tests: tests/unit/test_search_query_normalization.py (7 cases, all pass)
cover CJK full-width, casefold, whitespace, empty/None passthrough, and
idempotency.1 parent adb57bd commit cfa08a1
2 files changed
Lines changed: 63 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
87 | 88 | | |
88 | 89 | | |
89 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
90 | 110 | | |
91 | 111 | | |
92 | 112 | | |
| |||
1925 | 1945 | | |
1926 | 1946 | | |
1927 | 1947 | | |
| 1948 | + | |
1928 | 1949 | | |
1929 | 1950 | | |
1930 | 1951 | | |
| |||
2017 | 2038 | | |
2018 | 2039 | | |
2019 | 2040 | | |
| 2041 | + | |
2020 | 2042 | | |
2021 | 2043 | | |
2022 | 2044 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
0 commit comments