14 lines (8 loc) · 355 Bytes

Changelog

June 18, 2026

analyze: add Token::byte_range and Token::input_index to recover a token's raw source substring

May 8, 2026

analyze module, support for stopword removal, stemming, ascii_folding, maximum_token_length, case sensitivity

Apr 14, 2026

Initial crate release
UAX #29 compliant word and sentence tokenizer