Tokenizer: clarify intended input validation and error handling for invalid inputs

[DEEP-600]

While testing Gemma2Tokenizer.encode(), I noticed that handling of invalid input types is inconsistent.

From the code and documentation, the tokenizer appears to intentionally support:

• str
• list[str]

However, inputs outside these types are not validated early. As a result:

• some invalid inputs are silently accepted
• others fail later with low-level Python errors
• error messages are inconsistent and hard to interpret

Before proposing any fix, I want to better understand the intended behavior here.

---

Observed Behavior

Some examples using invalid inputs:

Input	Current result
123	TypeError: 'int' object is not iterable
["hello", 123]	AttributeError: 'int' object has no attribute 'replace'
{"text": "hello"}	Tokens returned
b"hello"	AttributeError: 'int' object has no attribute 'replace'

The errors vary depending on the input type and often surface deeper in the call stack, exposing Python internals rather than clearly explaining the input mistake.

---

Expected Behavior

For inputs other than the supported types (str, list[str]):

• The tokenizer should fail early
• The error message should be clear and consistent
• The error should indicate what input type was expected and what was actually received

For example:

==> TypeError: tokenizer.encode expects str or list[str], but got int <==

This would make input issues easier to understand and debug, especially in data pipelines and for new users.

---

Question for Maintainers

Could you please clarify the intended behavior of tokenizer.encode() when it receives unsupported input types?

In particular:

• Should all unsupported input types be rejected early with a clear, consistent error?
• Or is silent acceptance of certain structured inputs (e.g. dictionaries) intentional?

Once the intended behavior is confirmed, I’d be happy to work on a fix that aligns with the project’s design.

---

Reproduction

from gemma import gm

tokenizer = gm.text.Gemma2Tokenizer()

tests = [
    123,
    ["hello", 123],
    {"text": "hello"},
    b"hello",
]

for t in tests:
    try:
        print(tokenizer.encode(t))
    except Exception as e:
        print(type(e).__name__, e)

[srikantak-221]

Hi @DEEP-600 ,

The behavior is actually intentional design rather than an oversight. There is an 'isinstance' check which acts purely as a logic dispatcher, toggling between the str path (for raw parsing) and the list[str] path (for mapping pre-split tokens), rather than serving as a safety validator.
The developers explicitly omitted "deep" type checking here because adding an O(N) validation loop in Python would create a significant bottleneck on this critical hot path. Instead, the library relies on the static type signature to define the contract and uses "Duck Typing" for raw speed, which is why invalid inputs trigger native crashes while structured ones like dictionaries might accidentally pass through.

Check this out for reference:

gemma/gemma/gm/text/_tokenizer.py

Line 171 in a7bb060

def encode(

Thank you!

[DEEP-600]

Thanks for the detailed explanation and yep that makes sense I understand now that the behavior is intentional for performance reasons and that encode relies on the static type contract rather than runtime validation.
Appreciate the clarification!