Description
Python -VV
...Pip Freeze
annotated-types==0.7.0
attrs==24.2.0
certifi==2024.7.4
charset-normalizer==3.3.2
idna==3.8
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
mistral_common==1.3.4
pydantic==2.6.1
pydantic_core==2.16.2
referencing==0.35.1
regex==2024.7.24
requests==2.32.3
rpds-py==0.20.0
sentencepiece==0.2.0
tiktoken==0.7.0
typing_extensions==4.12.2
urllib3==2.2.2Reproduction Steps
Ive copied the example from: https://github.com/mistralai/mistral-common/blob/main/examples/tokenizer.ipynb
But the tokens aren't properly formatted.
The tokens are like this:
[AVAILABLE_TOOLS]▁[{"type":▁"function",▁"function":▁{"name":▁"get_current_weather",▁"description":▁"Get▁the▁current▁weather",▁"parameters":▁{"type":▁"object",▁"properties":▁{"location":▁{"type":▁"string",▁"description":▁"The▁city▁and▁state,▁e.g.▁San▁Francisco,▁CA"},▁"format":▁{"type":▁"string",▁"enum":▁["celsius",▁"fahrenheit"],▁"description":▁"The▁temperature▁unit▁to▁use.▁Infer▁this▁from▁the▁users▁location."}},▁"required":▁["location",▁"format"]}}}][/AVAILABLE_TOOLS][INST]▁What's▁the▁weather▁like▁today▁in▁Paris[/INST]
First Ive thought that it is intended but with the second message the LLM returns messages like this:
▁The▁current▁weather▁in▁Paris▁is▁72°C.
Expected Behavior
The tokens should have proper formatting with spaces instead of underlines at the most places:
[INST] What's the weather like today in Paris[/INST][TOOL_CALLS] [{"name": "get_current_weather", "arguments": {"location": "Paris, France", "format": "celsius"}, "id": "VvvODy9mT"}][TOOL_RESULTS] {"call_id": "VvvODy9mT", "content": 22}[/TOOL_RESULTS] The current temperature in Paris, France is 22 degrees Celsius.
Additional Context
I am using the mistral-nemo model.
Suggested Solutions
No response