Skip to content

Reject a zero length in the FixedLength pre-tokenizer instead of panicking#2155

Open
v-code01 wants to merge 1 commit into
huggingface:mainfrom
v-code01:fixedlength-reject-zero
Open

Reject a zero length in the FixedLength pre-tokenizer instead of panicking#2155
v-code01 wants to merge 1 commit into
huggingface:mainfrom
v-code01:fixedlength-reject-zero

Conversation

@v-code01

@v-code01 v-code01 commented Jul 5, 2026

Copy link
Copy Markdown

What

FixedLength::length is deserialized from the config with no validation (FixedLength::new performs none either). A value of 0 reaches:

for chunk in char_positions.chunks(self.length) { // panics: "chunk size must be non-zero"

so a crafted {"type":"FixedLength","length":0} pre-tokenizer crashes tokenization on any non-empty input.

Fix

Reject length == 0 with a clear error at the start of pre_tokenize:

if self.length == 0 {
    return Err("FixedLength pre-tokenizer length must be greater than 0".into());
}

Tests

Adds zero_length_is_rejected; existing FixedLength tests are unchanged.

cc @ArthurZucker @McPatate

…cking

`FixedLength::length` is deserialized from the config with no validation
(`FixedLength::new` performs none either). A value of 0 reaches
`char_positions.chunks(self.length)`, which panics with "chunk size must
be non-zero" on any non-empty input — a crafted `tokenizer.json` crashes
tokenization.

Reject `length == 0` with a clear error at the start of `pre_tokenize`.
Adds a regression test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant