[Feature] Add Unicode 15.0 and Universal Acceptance (UA) Compliance

[anivar]

Is your feature request related to a problem?

The current implementation doesn't properly handle:

• Non-Latin scripts (Arabic, Devanagari, CJK, etc.)
• Internationalized Domain Names (IDNs) like भारत.भारत
• Email Address Internationalization (EAI) like user@عربي.السعودية
• Unicode normalization (NFC/NFD/NFKC/NFKD)
• Bidirectional text (RTL/LTR mixing)
• Zero-width characters (ZWNJ/ZWJ) needed for Indic scripts

Describe the solution you'd like

Full Unicode and Universal Acceptance compliance:

# Should work correctly
guardrail.validate("प्रयोक्ता@भारत.भारत")  # EAI email
guardrail.validate("https://भारत.भारत")    # IDN domain
guardrail.validate("مرحبا שלום")             # Mixed RTL scripts
guardrail.validate("हिन्‌दी")                # ZWNJ in Hindi

Technical Requirements:

• Unicode 15.0 support
• Grapheme cluster counting (not byte counting)
• Confusable character detection (homograph attacks)
• Zero-width character handling for Indic scripts
• IDN/EAI validation per RFC 5891/6531
• Proper normalization across all forms

Additional context

• Required for global internet standards compliance
• Critical for preventing Unicode-based security attacks (homographs)
• Necessary for Indian language support (22 official languages)
• Enables proper handling of emoji and complex scripts

References:

• Universal Acceptance Steering Group
• Unicode Security Considerations
• RFC 5891 (IDNA 2008)
• RFC 6531 (Internationalized Email)

Implementation Impact

This would make any-guardrail compliant with international standards and usable globally.

[angpt]

Hi @anivar, Thank you for raising issues #107, and #110. We're committed to making AI safety tools accessible globally.

We'd welcome adding guardrails with multilingual, Unicode/UA compliance, and code-mixing support if suitable models exist. However, as I said in #104, we don't train models ourselves, we only host guardrails created by their original authors. These capabilities would need to be built into the underlying models, which is outside our scope.

Could you share if you've found any existing production-ready guardrails with these features?

[anivar]

Hey @angpt! This one's actually different - it's not about finding new models but fixing how the framework handles text.

Here's what's happening: attackers are using Unicode tricks to slip past detection. Even if you have the best models, they can't catch attacks if the text gets scrambled before reaching them.

For example, attackers might use Cyrillic letters that look identical to Latin ones, so "Ѕystem" (with Cyrillic Ѕ) looks like "System" but bypasses keyword detection. They also insert invisible zero-width characters to break pattern matching, or mix right-to-left text to hide malicious parts of their prompts.

The models themselves handle Unicode fine. Meta's Prompt-Guard or any mDeBERTa model understands all these scripts perfectly. But if the preprocessing code breaks on "भारत" or can't handle bidirectional text, the model never gets a clean shot at detecting the injection.

What needs fixing is the text preprocessing pipeline to properly normalize Unicode (NFC/NFD), preserve special characters that matter, and handle international domains and emails correctly. It's basically making sure the text gets to the model intact, not mangled.

This would make all your existing models more robust against Unicode-based bypass attempts. It's engineering work on the framework, not model training.