[BUG: cannot import name 'InstructTokenizerBase' from 'mistral_common.tokens.tokenizers.sentencepiece'

[Amit507017]

Python Version

Python 3.11.4

Pip Freeze

absl-py==2.3.1
annotated-types==0.7.0
attrs==25.3.0
certifi==2025.7.9
charset-normalizer==3.4.2
cramjam==2.10.0
docstring_parser==0.16
fastparquet==2024.11.0
filelock==3.18.0
fire==0.7.0
fsspec==2025.5.1
grpcio==1.73.1
idna==3.10
Jinja2==3.1.6
jsonschema==4.24.0
jsonschema-specifications==2025.4.1
Markdown==3.8.2
MarkupSafe==3.0.2
mistral_common==1.7.0
mpmath==1.3.0
networkx==3.5
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.19.3
nvidia-nvjitlink-cu12==12.9.86
nvidia-nvtx-cu12==12.1.105
packaging==25.0
pandas==2.3.1
pillow==11.3.0
protobuf==6.31.1
pyarrow==20.0.0
pydantic==2.11.7
pydantic_core==2.33.2
python-dateutil==2.9.0.post0
pytz==2025.2
PyYAML==6.0.2
referencing==0.36.2
regex==2024.11.6
requests==2.32.4
rpds-py==0.26.0
safetensors==0.5.3
sentencepiece==0.2.0
simple-parsing==0.1.7
six==1.17.0
sympy==1.14.0
tensorboard==2.19.0
tensorboard-data-server==0.7.2
termcolor==3.1.0
tiktoken==0.9.0
torch==2.2.0
tqdm==4.67.1
triton==2.2.0
typing-inspection==0.4.1
typing_extensions==4.14.1
tzdata==2025.2
urllib3==2.5.0
Werkzeug==3.1.3
xformers==0.0.24

Reproduction Steps

While validating data as per readme steps, I am getting the above error

Expected Behavior

The validation of data successfully done

Additional Context

No response

Suggested Solutions

No response

[thathsarani-w]

I'm getting the same error when validating data. Wonder what the issue is. Were you able to resolve this?

[steffi4321]

Hi, I just got the same issue, but I think I found the problem and a fix.
In one commit of mistral-common, they moved the InstructTokenizerBase to another file (mistral-ai/mistral-common@2c9f1762f8824e5d821840414ecb1b9e5267ffb3).
You can fix it by changing two lines in mistral-finetune:

1. In File "mistral-finetune/finetune/data/dataset.py", line 15, change
   from mistral_common.tokens.tokenizers.sentencepiece import InstructTokenizerBase to
   from mistral_common.tokens.tokenizers.instruct import InstructTokenizerBase
2. In File "mistral-finetune/finetune/data/tokenize.py", line 25, change
   from mistral_common.tokens.tokenizers.sentencepiece import InstructTokenizerBase to
   from mistral_common.tokens.tokenizers.instruct import InstructTokenizerBase

Also, afterward, I received another error concerning the updated mistral-common, which I fixed by editing
3. In File "mistral-finetune/finetune/data/tokenize.py", line 180, change validator.validate_messages(messages) to validator.validate_messages(messages, False)
4. In File "mistral-finetune/finetune/data/tokenize.py", line 330, change curr_tokens = instruct_tokenizer.encode_assistant_message(message, is_before_last_user_message=False) to curr_tokens = instruct_tokenizer.encode_assistant_message(message, is_before_last_user_message=False, continue_message=False)

No guarantees given, but maybe it helps :)