Certain unicode characters in Pixtral cause "OverflowError: Error in model execution: out of range integral type conversion attempted"

[han-steve]

Python -VV

Python 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]

Pip Freeze

mistral_common==1.4.4
vllm==0.6.2

Reproduction Steps

1. Initialize Pixtral following the vllm tutorial

from vllm import LLM
from vllm.sampling_params import SamplingParams
llm = LLM(model="mistralai/Pixtral-12B-2409", tokenizer_mode="mistral")

2. Create a message with text that contains a unicode character like \u2705.

messages = [{"role": "user", "content": [{"type":"text", "text": "\u2705"}]}]

3. Invoke Pixtral using it

llm.chat(messages, sampling_params=SamplingParams(max_tokens=8192))

4. Observe the error

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1630, in execute_model
[rank0]:     model_input.async_callback()
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/utils.py", line 1102, in weak_bound
[rank0]:     unbound(inst, *args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1014, in _process_model_outputs
[rank0]:     self.output_processor.process_outputs(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/output_processor/single_step.py", line 96, in process_outputs
[rank0]:     return self._process_sequence_group_outputs(sequence_group, outputs[0],
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/output_processor/single_step.py", line 124, in _process_sequence_group_outputs
[rank0]:     new_char_count = self.detokenizer.decode_sequence_inplace(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/detokenizer.py", line 115, in decode_sequence_inplace
[rank0]:     seq.read_offset) = convert_prompt_ids_to_tokens(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/detokenizer.py", line 224, in convert_prompt_ids_to_tokens
[rank0]:     new_tokens = tokenizer.convert_ids_to_tokens(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 227, in convert_ids_to_tokens
[rank0]:     tokens = [self.tokenizer.id_to_byte_piece(id) for id in ids]
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 227, in <listcomp>
[rank0]:     tokens = [self.tokenizer.id_to_byte_piece(id) for id in ids]
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py", line 280, in id_to_byte_piece
[rank0]:     return self._model.decode_single_token_bytes(token_id - self.num_special_tokens)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/tiktoken/core.py", line 272, in decode_single_token_bytes
[rank0]:     return self._core_bpe.decode_single_token_bytes(token)
[rank0]: OverflowError: out of range integral type conversion attempted

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]:   File "<stdin>", line 1, in <module>
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 580, in chat
[rank0]:     return self.generate(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/utils.py", line 1047, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 388, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 877, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1264, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 130, in execute_model
[rank0]:     output = self.driver_worker.execute_model(execute_model_req)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 327, in execute_model
[rank0]:     output = self.model_runner.execute_model(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
[rank0]:     raise type(err)(f"Error in model execution: "
[rank0]: OverflowError: Error in model execution: out of range integral type conversion attempted

Expected Behavior

There should not be an error even though there are unicode characters. Note that other characters like \u2018 don't cause this issue.

Additional Context

No response

Suggested Solutions

No response