Skip to content

Pull requests: huggingface/tokenizers

New pull request New
110 Open 876 Closed
110 Open 876 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Do not panic when deserializing an invalid Precompiled normalizer
#2156 opened Jul 5, 2026 by v-code01 Loading…
1
Reject a zero length in the FixedLength pre-tokenizer instead of panicking
#2155 opened Jul 5, 2026 by v-code01 Loading…
Do not panic in the Strip decoder on crafted config or short tokens
#2154 opened Jul 5, 2026 by v-code01 Loading…
Do not panic in BPE builder when a merge exceeds the longest vocab token
#2153 opened Jul 5, 2026 by v-code01 Loading…
Validate TemplateProcessing on deserialization (fix panic on crafted tokenizer.json)
#2152 opened Jul 5, 2026 by v-code01 Loading…
Do not panic in Encoding::truncate when stride >= max_len
#2151 opened Jul 5, 2026 by v-code01 Loading…
1
Add a fast_split module
#2150 opened Jul 5, 2026 by ArthurZucker Collaborator Loading…
Map prepended text to a zero-width span to avoid overlapping offsets (#1775)
#2149 opened Jul 5, 2026 by v-code01 Loading…
1
experiment: reduce normalizer alloc further
#2148 opened Jul 3, 2026 by SBrandeis Contributor • Draft
1
Improve single thread performance of de-tokenization
#2144 opened Jul 2, 2026 by sebpop Contributor Loading…
1
feat: add MultiRegex for common regexes (gpt2,etc) fast path
#2143 opened Jul 2, 2026 by McPatate Member Loading…
3
feat: impl pipeline::PreTokenizer for WhitespaceSplit
#2140 opened Jul 1, 2026 by McPatate Member Loading…
1
feat: handle Merge* variants of the SplitDelimiterBehavior
#2139 opened Jul 1, 2026 by McPatate Member Loading…
1
feat: impl pipeline::PreTokenizer for UnicodeScripts
#2138 opened Jul 1, 2026 by McPatate Member Loading…
2
feat: impl pipeline::PreTokenizer for Digits
#2136 opened Jul 1, 2026 by McPatate Member Loading…
3
feat: impl pipeline::PreTokenizer for CharDelimiterSplit
#2135 opened Jul 1, 2026 by McPatate Member Loading…
1
No alloc normalizers
#2133 opened Jul 1, 2026 by SBrandeis Contributor Loading…
8
chore(deps): bump js-yaml from 3.14.2 to 3.15.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2131 opened Jun 30, 2026 by dependabot Bot Loading…
1
docs: add docstrings to Tokenizer::from_file, from_bytes, and from_petrained
#2124 opened Jun 23, 2026 by BBloggsbott Loading…
feat: train-encode split
#2119 opened Jun 18, 2026 by McPatate Member Loading…
4
chore(deps-dev): bump webpack-dev-server from 5.2.1 to 5.2.5 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2118 opened Jun 18, 2026 by dependabot Bot Loading…
1
refactor: core crate layout
#2114 opened Jun 16, 2026 by McPatate Member • Draft
chore(deps): bump @babel/core from 7.24.3 to 7.29.7 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2113 opened Jun 16, 2026 by dependabot Bot Loading…
1
chore(deps-dev): bump launch-editor from 2.10.0 to 2.14.1 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2112 opened Jun 16, 2026 by dependabot Bot Loading…
1
BPE trainer: inherit continuing_subword_prefix/end_of_word_suffix from the model
#2108 opened Jun 13, 2026 by discobot Loading…
1
ProTip! no:milestone will show everything without a milestone.