fix(logprobs): guard convert_ids_list_to_tokens against out-of-range ids#48567
Open
UranusSeven wants to merge 2 commits into
Open
fix(logprobs): guard convert_ids_list_to_tokens against out-of-range ids#48567UranusSeven wants to merge 2 commits into
UranusSeven wants to merge 2 commits into
Conversation
Prompt logprobs handling crashed the async output handler with OverflowError: out of range integral type conversion attempted when an out-of-range token id (negative or beyond u32) reached the fast tokenizer's decode. Such ids can appear in uninitialized slots of the chunked prompt-logprobs tensor. Mirror the 0 <= id < len(tokenizer) guard already used in detokenize_incrementally so these ids decode to "" instead of raising. Co-authored-by: Claude Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai> Signed-off-by: UranusSeven <109661872+UranusSeven@users.noreply.github.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Out-of-range ids (e.g. uninitialized prompt-logprobs slots) make a | ||
| # fast tokenizer's u32 conversion raise OverflowError; guard like | ||
| # detokenize_incrementally does. | ||
| if not 0 <= token_id < vocab_size: |
Member
There was a problem hiding this comment.
Out of range IDs shouldn't get passed here in the first place, right?
cc @njhill
Member
There was a problem hiding this comment.
Yes, @UranusSeven could you explain more about how you encountered this?
Contributor
Author
There was a problem hiding this comment.
This happened in prod, leading to a decoder crash. The whole stack:
ERROR 07-12 06:47:23 [async_llm.py:706] Traceback (most recent call last):
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../vllm/v1/engine/async_llm.py", line 677, in output_handler
ERROR 07-12 06:47:23 [async_llm.py:706] processed_outputs = output_processor.process_outputs(
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../vllm/v1/engine/output_processor.py", line 655, in process_outputs
ERROR 07-12 06:47:23 [async_llm.py:706] req_state.logprobs_processor.update_from_output(engine_core_output)
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../vllm/v1/engine/logprobs.py", line 352, in update_from_output
ERROR 07-12 06:47:23 [async_llm.py:706] self._update_prompt_logprobs(output.new_prompt_logprobs_tensors)
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../vllm/v1/engine/logprobs.py", line 147, in _update_prompt_logprobs
ERROR 07-12 06:47:23 [async_llm.py:706] else convert_ids_list_to_tokens(
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../vllm/tokenizers/detokenizer_utils.py", line 100, in convert_ids_list_to_tokens
ERROR 07-12 06:47:23 [async_llm.py:706] token_str = tokenizer.decode([token_id])
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../vllm/tokenizers/hf.py", line 86, in decode
ERROR 07-12 06:47:23 [async_llm.py:706] return tok.decode(*args, **kwargs)
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../transformers/tokenization_utils_base.py", line 2897, in decode
ERROR 07-12 06:47:23 [async_llm.py:706] return self._decode(
ERROR 07-12 06:47:23 [async_llm.py:706] File ".../transformers/tokenization_utils_tokenizers.py", line 1032, in _decode
ERROR 07-12 06:47:23 [async_llm.py:706] text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
ERROR 07-12 06:47:23 [async_llm.py:706] OverflowError: out of range integral type conversion attempted
ERROR 07-12 06:47:23 [serving.py:1004] Error in chat completion stream generator.
Contributor
Author
There was a problem hiding this comment.
vLLM was v0.23.0.
Contributor
There was a problem hiding this comment.
Drive by comment @DarkLight1337 @njhill #47346 similar issue and fix.
Yes, we shouldn't have out of range IDs here, but if we do have a model regression spitting out invalid token IDs, we currently get a fairly misleading tokenizer crash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
convert_ids_list_to_tokensinvllm/tokenizers/detokenizer_utils.pycallstokenizer.decode([token_id])for every id with no bounds check. When an out-of-range id (negative, or beyond the u32 range) reaches a fast/rust-backed tokenizer, the rust binding raisesOverflowError: out of range integral type conversion attemptedwhile converting the Python int to au32. Such ids can surface in uninitialized slots of the chunked prompt-logprobs tensor (LogprobsTensors.empty_cpuusestorch.empty, filled slice-by-slice across prefill chunks), crashing the async output handler.detokenize_incrementallyin the same file already guards this exact case withif 0 <= new_token_id < len(tokenizer), butconvert_ids_list_to_tokens— used by both sample and prompt logprobs — was missing the equivalent guard.Fix
Mirror the existing
0 <= id < len(tokenizer)guard: out-of-range ids decode to""instead of being passed to the tokenizer. Since the result is empty (no U+FFFD), it does not perturb the downstream byte-fallback UTF-8 correction path.Test plan
test_convert_ids_list_to_tokens_out_of_range(parametrized over-1, -12345, 2**32, 2**63) using a fake rust-like tokenizer that raisesOverflowErroron out-of-range ids; asserts the result is[""].