Skip to content

Unigram: honor byte_fallback on encode - #388

Open
RyanFarrar13 wants to merge 1 commit into
huggingface:mainfrom
RyanFarrar13:unigram-byte-fallback
Open

Unigram: honor byte_fallback on encode#388
RyanFarrar13 wants to merge 1 commit into
huggingface:mainfrom
RyanFarrar13:unigram-byte-fallback

Conversation

@RyanFarrar13

Copy link
Copy Markdown

Fixes #387.

UnigramTokenizer never read the model's byte_fallback flag, so any
character absent from the trie became an <unk>, and fuseUnknownTokens
collapsed consecutive 1s into only one token. For vocabularies that declare
byte fallback, those characters have real pieces (<0xNN>) and should use
them, especially newline.

Approach

The viterbi output is a list of sentence substrings, so the original character
survives to the end of tokenize(text:) and only becomes <unk> later in
convertTokenToId. That makes the fix post pass rather than surgery on the
lattice: expand any piece missing from the vocabulary into its UTF-8 byte
pieces.

This copies Unigram::tokenize in HF tokenizers, including 2 details:

  • Byte fallback is all-or-nothing , if the vocabulary is missing any of the
    byte pieces, the piece is left unknown rather than becoming a run of
    unknowns.

Models that do not declare byte_fallback take an early return and are
unaffected.

Tests

Adds Tests/TokenizersTests/UnigramByteFallbackTests.swift, a self contained,
no network connection, built on an 8-entry synthetic vocabulary. Covers single-byte and multi-byte characters, no fusion of consecutive bytes, the disabled-flag path, and the incomplete byte vocabulary case.

No existing test exercises a byte_fallback: true Unigram model, so this went unnoticed.

Verification

Against cyberagent/CAT-Translate-0.8b (Unigram, byte_fallback: true,
102,400 pieces) compared with Python tokenizers 0.22.2 on the same
tokenizer.json, over a 58-case corpus covering emoji, astral characters, CJK, Arabic,
Hebrew, Devanagari, Thai, CRLF, zero-width characters and mixed scripts:

id-exact vs Python
before 21 / 58
after 58 / 58

Full suite: 164/164 passing (159 pre-existing, unchanged).


Found this while building an on-device Japanese learning app, where this
truncated every multi line translation to its first line.

Out-of-vocabulary characters were emitted as <unk> and fused into a single
token, regardless of the model's byte_fallback flag. Expand them to their
UTF-8 byte pieces when the vocabulary declares byte fallback and contains
every required byte piece, matching Unigram::tokenize in HF tokenizers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnigramTokenizer ignores byte_fallback, emitting <unk> for out-of-vocabulary characters

1 participant