Unigram: honor byte_fallback on encode - #388
Open
RyanFarrar13 wants to merge 1 commit into
Open
Conversation
Out-of-vocabulary characters were emitted as <unk> and fused into a single token, regardless of the model's byte_fallback flag. Expand them to their UTF-8 byte pieces when the vocabulary declares byte fallback and contains every required byte piece, matching Unigram::tokenize in HF tokenizers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #387.
UnigramTokenizernever read the model'sbyte_fallbackflag, so anycharacter absent from the trie became an
<unk>, andfuseUnknownTokenscollapsed consecutive 1s into only one token. For vocabularies that declare
byte fallback, those characters have real pieces (
<0xNN>) and should usethem, especially newline.
Approach
The viterbi output is a list of sentence substrings, so the original character
survives to the end of
tokenize(text:)and only becomes<unk>later inconvertTokenToId. That makes the fix post pass rather than surgery on thelattice: expand any piece missing from the vocabulary into its UTF-8 byte
pieces.
This copies
Unigram::tokenizein HFtokenizers, including 2 details:byte pieces, the piece is left unknown rather than becoming a run of
unknowns.
Models that do not declare
byte_fallbacktake an early return and areunaffected.
Tests
Adds
Tests/TokenizersTests/UnigramByteFallbackTests.swift, a self contained,no network connection, built on an 8-entry synthetic vocabulary. Covers single-byte and multi-byte characters, no fusion of consecutive bytes, the disabled-flag path, and the incomplete byte vocabulary case.
No existing test exercises a
byte_fallback: trueUnigram model, so this went unnoticed.Verification
Against
cyberagent/CAT-Translate-0.8b(Unigram,byte_fallback: true,102,400 pieces) compared with Python
tokenizers0.22.2 on the sametokenizer.json, over a 58-case corpus covering emoji, astral characters, CJK, Arabic,Hebrew, Devanagari, Thai, CRLF, zero-width characters and mixed scripts:
Full suite: 164/164 passing (159 pre-existing, unchanged).
Found this while building an on-device Japanese learning app, where this
truncated every multi line translation to its first line.