Skip to content

Add a fast_split module#2150

Open
ArthurZucker wants to merge 20 commits into
feat/train_encode_splitfrom
remove-char-repr-bpe
Open

Add a fast_split module#2150
ArthurZucker wants to merge 20 commits into
feat/train_encode_splitfrom
remove-char-repr-bpe

Conversation

@ArthurZucker

Copy link
Copy Markdown
Collaborator

This removes dependencies on Unicode, Regex, Regex-Automata by unrolling.
On 10K different models only a handful of Rules are actually needed.
Each Pattern is a combination of different Rules. Rules use Matchers to get info on the bytes (is it a letter/digit).

FastLetter vs Letter  (15200 bytes)
  bitmap  : 0.583 ns/byte (  1714 MB/s)
  unicode : 6.293 ns/byte (   159 MB/s)  speedup 10.79x

FastNumber vs Digit  (4000 bytes)
  bitmap  : 0.400 ns/byte (  2503 MB/s)
  unicode : 5.410 ns/byte (   185 MB/s)  speedup 13.54x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant