[Feature] Add Multilingual Support for Global Languages

[anivar]

The Problem

I've been looking through the codebase and noticed that all the current guardrails are essentially English-only. They use models like DeBERTa, RoBERTa, and others that were trained primarily on English text. This means any-guardrail doesn't really work for most of the world's languages.

For example:

• The Deepset guardrail uses deberta-v3-base-injection - English only
• Mozilla's own off-topic detector is explicitly named jina-embeddings-v2-small-en-off-topic
• ProtectAI models are all English DeBERTa variants

Why This Matters

There are 4.5+ billion people who don't speak English as their first language. Right now, they can't use any-guardrail effectively. If someone sends harmful content in French, Hindi, Arabic, or Chinese, the guardrails might not catch it properly.

The Good News

We don't need to reinvent anything. Multilingual models that handle 100+ languages already exist and work great. They've been battle-tested by big tech companies for years now (as of November 2024).

Simple Solution

We just need to offer multilingual model options alongside the English ones. For example:

# Instead of only having this:
model = "deberta-v3-base-injection"  # English only

# We could also support:
model = "xlm-roberta-base"  # Works for 100 languages!

Two Ways to Do This

Option 1: Add multilingual models to existing guardrails

class Deepset(HuggingFace):
    SUPPORTED_MODELS = [
        "deepset/deberta-v3-base-injection",  # Keep the English one
        "xlm-roberta-base-finetuned",  # Add a multilingual one
    ]

Option 2: Create a simple wrapper

# Take any existing guardrail and make it multilingual
guardrail = make_multilingual(
    AnyGuardrail.create(GuardrailName.DEEPSET)
)

Which Models Should We Use?

Here are some proven multilingual models available today:

• xlm-roberta-base: Handles 100 languages, widely used
• bert-base-multilingual: Covers 104 languages, lightweight
• google/muril-base: Great for Indian languages + English
• google/mt5-base: 101 languages, good for generation

The Best Part

Once we add these, the guardrails would just work for any language automatically. Users wouldn't need to:

• Specify what language they're using
• Install special language packs
• Configure anything

It would just work whether you write in English, Spanish, Hindi, Arabic, or any other language.

How I Can Help

I'm happy to help implement this. I could:

1. Test which multilingual models work best for safety detection
2. Add the multilingual options to existing guardrails
3. Create the wrapper approach if that's preferred
4. Write tests to make sure it works across languages

What do you think? Should we start with adding multilingual models to existing guardrails, or would you prefer the wrapper approach?

[angpt]

Hi @anivar, Thank you for raising this issue! We're committed to making AI safety tools accessible for everyone, including non-English speakers.

We'd welcome a contribution to add a multilingual guardrail, if a suitable one exists.

We don’t train models ourselves. We only use models that have been trained and validated by their original authors. For example, we use deberta-v3-base-injection since it has been specifically trained and validated by the Deepset team as a guardrail.

As you’ve alluded to in your feature request, we’d need to use a specifically trained and validated model like xlm-roberta-base-finetuned to use as a multilingual guardrail. To our knowledge, such a model doesn't currently exist. This training is on the original authors of this model, and is entirely out of our scope.

Could you share if you've found any existing well-researched multilingual guardrail libraries we should consider?

[anivar]

Hey @angpt! Thanks for the response. I found an existing multilingual model that could work.

After looking deeper, I see any-guardrail focuses on detecting prompt injections. Meta released meta-llama/Prompt-Guard-86M which is specifically trained on both English and non-English injections. It uses mDeBERTa as its multilingual base, so it handles multiple languages naturally. The model is small enough to run without GPUs (just 86M parameters) and detects both injections and jailbreaks.

You can check it out at https://huggingface.co/meta-llama/Prompt-Guard-86M

Right now you have English-only models like deepset/deberta-v3-base-injection and ProtectAI/deberta-v3-base-prompt-injection. Adding Meta's Prompt-Guard would immediately enable multilingual injection detection. It's from a reputable source (Meta) and specifically designed for this use case.

The alternative would be fine-tuning xlm-roberta-base on injection datasets, but that's more work when Meta has already done it for us.

Want me to create a PR adding Prompt-Guard as a new multilingual guardrail? It should be straightforward since it follows the same HuggingFace pattern as your existing guardrails.