Regarding the language compatibility one way we can achieve this is by calling the hf_api to get the model tags
https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api#huggingface_hub.HfApi.get_model_tags
And validate if the language is present in the list (not sure if it is a fool-proof way to check for all models, since it depends on the contributors adding those tags):
We could build a functionality for this, at the moment I think words from unsupported languages may be treated as unknown tokens.
Originally posted by @HareeshBahuleyan in #1178 (comment)
Regarding the language compatibility one way we can achieve this is by calling the hf_api to get the model tags
https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api#huggingface_hub.HfApi.get_model_tags
And validate if the language is present in the list (not sure if it is a fool-proof way to check for all models, since it depends on the contributors adding those tags):
We could build a functionality for this, at the moment I think words from unsupported languages may be treated as unknown tokens.
Originally posted by @HareeshBahuleyan in #1178 (comment)