fix(guardrail): re-land #183, reverted by the 2026-08-13 i4 release (#195)

# Re-land #183, which the 2026-08-13 release reverted

#183 merged as 277c0f1 on 2026-08-13 at 10:52 UTC. The very next commit
on
`main` — **Release 2026-08-13 (i4 b5c36d02) (#194)**, an automated
mirror of the
internal i4 tree cut from a source commit that predates the merge —
rewrote both
files back to their pre-#183 contents.

`main` today is byte-identical to `cd5e450`, the commit *before* #183
landed:

```
$ git diff cd5e450 origin/main -- cosmos_framework/auxiliary/guardrail/blocklist/
(no output)
```

`normalize_for_matching()` is gone, the misaligned
`uncensor_whitelist()` is
back, and `blocklist_test.py` has dropped from 32 tests to 8. Both
defects
described in #183 are live on `main` again — including the `IndexError`
that a
benign prompt like `'Snow White is flat'` raises against the production
lists.

## What this PR does

Restores the #183 state exactly, nothing more:

```
$ git diff 277c0f1 -- cosmos_framework/auxiliary/guardrail/blocklist/
(no output)
```

Two files, 258 insertions, 16 deletions — the same diff that was
reviewed and
approved on #183. #194 touches no other guardrail file, and no caller
outside
`blocklist.py` references either changed method, so there is nothing to
reconcile between the two changes.

## Recap of what comes back

1. **The whitelist restore corrupted the censored prompt.**
`censor_prompt`
walked the input and the censored text side by side *by position* to put
whitelisted words back, but a multi-word blocklist entry is replaced by
a
single censor token, so every such match shifts the censored list one
token
left. `'Snow White is flat'` → `IndexError`; `'a Snow White poster on a
flat
wall'` → the user is quoted `'a **** poster on a flat flat'`, with
`wall`
   silently overwritten. The whitelist is already handed to
`load_censor_words(whitelist_words=...)`, so the restore step is
**removed**
   rather than repaired.

2. **Invisible characters walked past the matcher.**
`normalize_for_matching()` folds them before censoring — NFKC, combining
marks dropped, invisible and bidirectional controls rewritten to a
space,
whitespace runs collapsed. Without it a zero-width space, soft hyphen,
fullwidth letter, or a second space evades a blocked phrase while
rendering
identically to a reader. As stated on #183, this blocks strictly more
than
   before. Homoglyphs remain out of scope.

The full analysis, the corpus measurement (same 4 of 492 Edge reasoner
QA
outputs block before and after, item for item), and the test-by-test
rationale
are on #183 and unchanged.

## Verification

`blocklist_test.py` is back to 32 tests; all 32 pass locally against
this
branch. The tests build the matcher directly, so they need no
checkpoint.

## Note for whoever owns the i4 → OSS sync

This will happen again on the next release unless the change is also
landed in
the internal i4 tree, or the guardrail blocklist files are excluded from
the
mirror. Re-landing it here fixes `main` today; it does not stop the next
automated release from reverting it a second time.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Author: lfengad

cosmos_framework/auxiliary/guardrail/blocklist/blocklist.py

@@ -4,7 +4,7 @@
 import argparse
 import os
 import re
-import string
+import unicodedata
 from difflib import SequenceMatcher
 
 import nltk
@@ -30,6 +30,12 @@
 # never used to decide whether something was censored.
 CENSOR = misc.Color.red("*")
 
+# Characters that occupy no width, or that only reorder what is drawn: zero-width
+# spaces and joiners, the soft hyphen, and the bidirectional controls. They render
+# as nothing, so a blocklist entry split by one still reads as the entry to a human
+# while reaching the matcher as a token it does not recognise.
+INVISIBLE_CHARS = re.compile(r"[­​-‏‪-‮⁠-⁤]")
+
 
 class Blocklist(ContentSafetyGuardrail):
     def __init__(
@@ -61,16 +67,29 @@ def __init__(
         log.debug(f"Whitelisted {len(self.whitelist_words)} words/phrases from whitelist")
         log.debug(f"Loaded {len(self.exact_match_words)} exact match words/phrases from blocklist")
 
-    def uncensor_whitelist(self, input_prompt: str, censored_prompt: str) -> str:
-        """Explicitly uncensor words that are in the whitelist."""
-        input_words = input_prompt.split()
-        censored_words = censored_prompt.split()
-        whitelist_words = set(self.whitelist_words)
-        for i, token in enumerate(input_words):
-            if token.strip(string.punctuation).lower() in whitelist_words:
-                censored_words[i] = token
-        censored_prompt = " ".join(censored_words)
-        return censored_prompt
+    @staticmethod
+    def normalize_for_matching(prompt: str) -> str:
+        """Fold away the ways a word can be written to look normal but not match.
+
+        The matcher compares whitespace-delimited tokens, so anything that splits
+        a word without being visible, or that spells its letters with a different
+        code point, walks past it while the prompt still reads as the blocked word.
+        Three foldings, all of them lossless as far as a reader is concerned:
+
+        * compatibility normalization (NFKC), which maps fullwidth and other
+          presentation forms onto the ASCII letters they are drawn as;
+        * removal of combining marks, so an accent added to a letter does not
+          make a new word;
+        * invisible characters rewritten to a space, then runs of whitespace
+          collapsed, so a word split by any of them is matched as the split it
+          renders as.
+
+        The collapse also closes a plainer hole: a multi-word entry was defeated
+        by typing two spaces between its words.
+        """
+        prompt = unicodedata.normalize("NFKC", prompt)
+        prompt = "".join(c for c in unicodedata.normalize("NFKD", prompt) if not unicodedata.combining(c))
+        return " ".join(INVISIBLE_CHARS.sub(" ", prompt).split())
 
     def censor_prompt(self, input_prompt: str) -> tuple[bool, str]:
         """Censor the prompt using the blocklist with better-profanity fuzzy matching.
@@ -89,9 +108,11 @@ def censor_prompt(self, input_prompt: str) -> tuple[bool, str]:
         # than substituting keeps the stricter reading: "n\x00ike" fuses back to
         # a blocked word instead of being split into two harmless tokens.
         input_prompt = input_prompt.replace(CENSOR_SENTINEL, "")
+        input_prompt = self.normalize_for_matching(input_prompt)
+        # The whitelist is handed to load_censor_words(), so the matcher already
+        # leaves whitelisted words alone. Restoring them a second time here is
+        # what introduced the token misalignment described in the commit message.
         censored_prompt = self.profanity.censor(input_prompt, censor_char=CENSOR_SENTINEL)
-        # Uncensor whitelisted words that were censored from blocklist fuzzy matching
-        censored_prompt = self.uncensor_whitelist(input_prompt, censored_prompt)
         if CENSOR_SENTINEL in censored_prompt:
             display_prompt = censored_prompt.replace(CENSOR_SENTINEL, CENSOR)
             return True, f"Prompt blocked by censorship: Censored Prompt: {display_prompt}"

cosmos_framework/auxiliary/guardrail/blocklist/blocklist_test.py

@@ -1,6 +1,8 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: OpenMDW-1.1
 
+import unicodedata
+
 import pytest
 
 from cosmos_framework.auxiliary.guardrail.blocklist.blocklist import Blocklist
@@ -45,18 +47,19 @@ def test_partial_match_with_threshold():
     assert match is False
 
 
-def _blocklist_with_words(words: list[str]) -> Blocklist:
+def _blocklist_with_words(words: list[str], whitelist: list[str] | None = None) -> Blocklist:
     """Build a Blocklist around a small word list, without downloading a checkpoint.
 
     censor_prompt only needs the profanity matcher and the whitelist, so the
     heavyweight __init__ (checkpoint download, nltk data) is bypassed here.
     """
     from better_profanity.better_profanity import Profanity
 
+    whitelist = whitelist or []
     bl = Blocklist.__new__(Blocklist)
     bl.profanity = Profanity()
-    bl.profanity.load_censor_words(custom_words=words, whitelist_words=[])
-    bl.whitelist_words = []
+    bl.profanity.load_censor_words(custom_words=words, whitelist_words=whitelist)
+    bl.whitelist_words = whitelist
     return bl
 
 
@@ -126,3 +129,221 @@ def test_blocked_word_is_still_detected_alongside_markdown():
     # blocked word is masked.
     assert "**bold**" in message
     assert "badword" not in message
+
+
+@pytest.mark.L1
+def test_multi_word_match_beside_whitelisted_word_does_not_crash():
+    """A multi-word blocklist hit must not make a later whitelisted word crash.
+
+    Regression test: the censored text has one token per match, so a two-word
+    entry made it one token shorter than the input. The whitelist restore walked
+    the two lists by position, so a whitelisted word after the match indexed past
+    the end of the censored list and raised IndexError on an ordinary prompt.
+    """
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("Snow White is flat")
+
+    assert blocked is True
+    assert "is flat" in message
+
+
+@pytest.mark.L1
+def test_multi_word_match_does_not_rewrite_a_later_word():
+    """The reported prompt must quote the input, not a shifted copy of it.
+
+    Same off-by-N as above, but landing inside the list instead of past its end:
+    the write went to the wrong token and silently replaced it.
+    """
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("a Snow White poster on a flat wall")
+
+    assert blocked is True
+    assert "on a flat wall" in message
+    assert "flat flat" not in message
+
+
+@pytest.mark.L1
+def test_several_multi_word_matches_stay_aligned():
+    """Every additional multi-word hit shifts the two lists one token further."""
+    bl = _blocklist_with_words(["snow white", "boston dynamics"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("Snow White and Boston Dynamics on a flat wall")
+
+    assert blocked is True
+    assert "on a flat wall" in message
+
+
+@pytest.mark.L1
+def test_whitelisted_word_is_not_censored():
+    """The whitelist still works: a whitelisted word is reported as written."""
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("a Snow White poster on a flat desk")
+
+    assert blocked is True
+    assert "a flat desk" in message
+
+
+@pytest.mark.L1
+def test_whitelisted_word_alone_does_not_block():
+    """A prompt with only whitelisted words is safe."""
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("the floor is flat")
+
+    assert blocked is False
+    assert message == ""
+
+
+@pytest.mark.L1
+def test_whitelisted_word_inside_a_blocked_phrase_stays_censored():
+    """A whitelisted word must not dissolve a phrase that genuinely matched.
+
+    'snow flat' is on the blocklist and 'flat' is whitelisted. The phrase is the
+    match, so it stays censored; whitelisting a word does not license the phrase
+    that contains it.
+    """
+    bl = _blocklist_with_words(["snow flat"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("a snow flat scene")
+
+    assert blocked is True
+    assert "snow flat" not in message
+
+
+@pytest.mark.L1
+def test_whitelisted_word_at_the_end_of_the_prompt():
+    """The final token is where the positional walk ran off the end."""
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("a snow white poster flat")
+
+    assert blocked is True
+    assert message.endswith("poster flat")
+
+
+@pytest.mark.L1
+def test_repeated_multi_word_matches():
+    """Two hits in a row shift the alignment twice over."""
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("snow white snow white flat wall")
+
+    assert blocked is True
+    assert "flat wall" in message
+
+
+@pytest.mark.L1
+def test_punctuation_around_a_whitelisted_word_is_preserved():
+    """Punctuation belongs to the token and must survive into the message."""
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    blocked, message = bl.censor_prompt("snow white, flat, wall.")
+
+    assert blocked is True
+    assert "flat, wall." in message
+
+
+@pytest.mark.L1
+def test_extra_spaces_do_not_defeat_a_multi_word_entry():
+    """A two-word entry was evaded by typing two spaces between its words."""
+    bl = _blocklist_with_words(["snow white"])
+
+    blocked, _ = bl.censor_prompt("a snow  white poster")
+
+    assert blocked is True
+
+
+@pytest.mark.L1
+@pytest.mark.parametrize(
+    ("name", "prompt"),
+    [
+        ("zero width space", "a snow​white poster"),
+        ("zero width non joiner", "a snow‌white poster"),
+        ("soft hyphen", "a snow­white poster"),
+        ("right to left override", "a snow‮white poster"),
+        ("word joiner", "a snow⁠white poster"),
+        ("byte order mark", "a snowwhite poster"),
+    ],
+)
+def test_invisible_characters_do_not_split_a_blocked_phrase(name, prompt):
+    """An invisible character renders as nothing, so the prompt still reads as the entry.
+
+    Splitting "snow white" with one of these produced a token the matcher did not
+    recognise, while a reader saw the blocked phrase unchanged.
+    """
+    bl = _blocklist_with_words(["snow white"])
+
+    blocked, _ = bl.censor_prompt(prompt)
+
+    assert blocked is True, f"{name} was not folded away"
+
+
+@pytest.mark.L1
+@pytest.mark.parametrize(
+    ("name", "prompt"),
+    [
+        ("non breaking space", "a snow white poster"),
+        ("en quad", "a snow white poster"),
+        ("ideographic space", "a snow　white poster"),
+        ("narrow no break space", "a snow white poster"),
+    ],
+)
+def test_unusual_spaces_do_not_split_a_blocked_phrase(name, prompt):
+    """Any Unicode space between the words of an entry must still match."""
+    bl = _blocklist_with_words(["snow white"])
+
+    blocked, _ = bl.censor_prompt(prompt)
+
+    assert blocked is True, f"{name} was not folded away"
+
+
+@pytest.mark.L1
+def test_fullwidth_letters_are_folded_to_ascii():
+    """Fullwidth forms are drawn as the ASCII letters they normalize to."""
+    bl = _blocklist_with_words(["snow white"])
+
+    blocked, _ = bl.censor_prompt("a ｓｎｏｗ white poster")
+
+    assert blocked is True
+
+
+@pytest.mark.L1
+def test_combining_marks_do_not_make_a_new_word():
+    """An accent added to a letter must not create a word the blocklist misses."""
+    bl = _blocklist_with_words(["snow white"])
+
+    for prompt in ("a snów white poster", unicodedata.normalize("NFD", "a snów white poster")):
+        blocked, _ = bl.censor_prompt(prompt)
+        assert blocked is True, f"{prompt!r} was not folded away"
+
+
+@pytest.mark.L1
+def test_normalization_does_not_block_ordinary_text():
+    """Folding must not invent matches in text that contains none."""
+    bl = _blocklist_with_words(["snow white"], whitelist=["flat"])
+
+    for prompt in (
+        "a robot on a flat desk",
+        "an em dash — and an arrow → in the text",
+        "café scene with Élodie",
+        "snow  and  white  are  separate  words  here",
+        "こんにちは from the model",
+    ):
+        blocked, message = bl.censor_prompt(prompt)
+        assert blocked is False, f"{prompt!r} was wrongly reported as blocked: {message}"
+
+
+@pytest.mark.L1
+def test_normalization_preserves_the_earlier_sentinel_and_markdown_behaviour():
+    """Folding runs after the sentinel strip and must not resurrect the "*" bug."""
+    bl = _blocklist_with_words(["badword"])
+
+    blocked, _ = bl.censor_prompt("A **bold** heading with ​ in it")
+    assert blocked is False
+
+    blocked, message = bl.censor_prompt("a bad\x00word in the text")
+    assert blocked is True
+    assert "badword" not in message