Skip to content

Improve Levenshtein search speed and index cleanup#19

Merged
andylokandy merged 1 commit into
andylokandy:masterfrom
M1tsumi:master
Jan 8, 2026
Merged

Improve Levenshtein search speed and index cleanup#19
andylokandy merged 1 commit into
andylokandy:masterfrom
M1tsumi:master

Conversation

@M1tsumi

@M1tsumi M1tsumi commented Jan 5, 2026

Copy link
Copy Markdown
Contributor

Adds a cheap length-based prefilter for Levenshtein mode to avoid unnecessary SIMD Levenshtein calls, and prune empty token entries from the index when removing items.

  • src/lib.rs: add length-difference prefilter in search_tokens() for levenshtein mode; prune empty reverse_map entries in remove().
  • tests/lib.rs: add test remove_prunes_reverse_map_entries() verifying tokens get removed.

levenshtein_simd_k was being called for every stored token; many calls are wasteful when token and pattern lengths differ by more than the allowed edit distance. Skipping those reduces CPU work. Pruning keeps the index compact after deletions.

@andylokandy andylokandy merged commit c2dc9c6 into andylokandy:master Jan 8, 2026
2 checks passed
@andylokandy

Copy link
Copy Markdown
Owner

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants