MistralAI tokenizer warning persists despite setting HF_TOKEN environment variable

[Somnathumapathi]

When running python ingest.py:

Despite properly setting the HF_TOKEN environment variable, the following warning still appears when running the document ingestion script:

/Users/somnathumapathi/my_projects/aifaq/src/mvt/venv/lib/python3.13/site-packages/langchain_mistralai/embeddings.py:181: UserWarning: Could not download mistral tokenizer from Huggingface for calculating batch sizes. Set a Huggingface token via the HF_TOKEN environment variable to download the real tokenizer. Falling back to a dummy tokenizer that useslen().

[sancheet230]

@Somnathumapathi Did you get the access permissions

[RAWx18]

https://www.linkedin.com/posts/ryanmadhuwala_opensource-hyperledger-aifaq-activity-7338538517431484416-g5PD
Please checkout this video for any issue regarding installation.

Incase of your issue is solved, please close the issue later.

[RAWx18]

@gcapuzzi This isn't an actual issue, as the behavior is expected — it's mentioned that a request to Hugging Face is needed for the Mistral tokenizer. We can go ahead and close this now.