AMH-Tokenizer v0.2.0 – Syllable-Aware BPE Tokenizer for Amharic

[sefineh-ai]

amharic-tokenizer v0.2.0 – A syllable-aware BPE tokenizer for Amharic (አማርኛ). Decomposes Fidel characters into base consonant–vowel components, reducing vocabulary size while preserving meaning. Supports tokenization with ## subword continuations, end-of-word markers , and spaces, detokenization, and custom training for NLP pipelines.

Usage

"""Test script for basic roundtrip functionality of the AmharicTokenizer."""
from amharic_tokenizer import AmharicTokenizer
def test_roundtrip_basic():
    """Load a trained tokenizer, tokenize text, convert to IDs, and detokenize."""
    tok = AmharicTokenizer.load("amh_bpe_v0.2.0")
    text = (
        "ስዊድን ከኢትዮጵያ ጋር ያላትን ግንኙነት አስመልክቶ "
        "አዲስ የትብብር ስልት መነደፉን አምባሳደሩ ገልጸዋል"
    )

    # Tokenize text
    tokens = tok.tokenize(text)
    print("Tokens:", tokens)

    # Convert tokens to IDs
    ids = tok.convert_tokens_to_ids(tokens)
    print("IDs:", ids)

    # Convert IDs back to tokens
    tokens_from_ids = tok.convert_ids_to_tokens(ids)
    print("Tokens from IDs:", tokens_from_ids)

    # Clean tokens for display
    display_tokens = [t.replace("</w>", "") for t in tokens if t != "</w>"]
    print("Display Tokens:", display_tokens)

    # Detokenize back to original text
    detokenized = tok.detokenize(tokens)
    print("Detokenized:", detokenized)
    print("vocab_size:", tok.vocab_size)

if __name__ == "__main__":
    test_roundtrip_basic()

Sample Output: 

          Tokens: ['ሰእወኢ', '##ደ', '##እነ', '##እ', '</w>', ' ', 'ከአ', '##ኢተእየኦጰእ', '##የ', '##ኣ', '</w>', ' ', 'ገኣ', '##ረ', '##እ', '</w>', ' ', 'የኣለኣ', '##ተ', '##እነ', '##እ', '</w>', ' ', 'ገ', '##እነ', '##እ', '##ኘኡ', '##ነአተእ', '</w>', ' ', 'አሰእመአለእከእ', '##ተኦ', '</w>', ' ', 'አደኢ', '##ሰ', '##እ', '</w>', ' ', 'የአ', '##ተእበእ', '##በ', '##እረ', '##እ', '</w>', ' ', 'ሰእለእ', '##ተእ', '</w>', ' ', 'መአነአ', '##ደአ', '##ፈ', '##ኡነእ', '</w>', ' ', 'አመእበኣሰኣ', '##ደ', '##አረ', '##ኡ', '</w>', ' ', 'ገአለእጸ', '##አወ', '##ኣ', '##ለ', '##እ', '</w>']
          IDs: [56252, 191975, 123541, 121977, 9863, 4, 134750, 119975, 156339, 120755, 9863, 4, 200274, 45529, 121977, 9863, 4, 180994, 83857, 123541, 121977, 9863, 4, 197634, 123541, 121977, 103143, 99749, 9863, 4, 108486, 97157, 9863, 4, 117000, 47785, 121977, 9863, 4, 156340, 96005, 62293, 122375, 121977, 9863, 4, 54490, 95215, 9863, 4, 33128, 191977, 208279, 119387, 9863, 4, 106960, 191975, 107209, 119347, 9863, 4, 197926, 114069, 120755, 17589, 121977, 9863]
          Back tokens from IDs: ['ሰእወኢ', '##ደ', '##እነ', '##እ', '</w>', ' ', 'ከአ', '##ኢተእየኦጰእ', '##የ', '##ኣ', '</w>', ' ', 'ገኣ', '##ረ', '##እ', '</w>', ' ', 'የኣለኣ', '##ተ', '##እነ', '##እ', '</w>', ' ', 'ገ', '##እነ', '##እ', '##ኘኡ', '##ነአተእ', '</w>', ' ', 'አሰእመአለእከእ', '##ተኦ', '</w>', ' ', 'አደኢ', '##ሰ', '##እ', '</w>', ' ', 'የአ', '##ተእበእ', '##በ', '##እረ', '##እ', '</w>', ' ', 'ሰእለእ', '##ተእ', '</w>', ' ', 'መአነአ', '##ደአ', '##ፈ', '##ኡነእ', '</w>', ' ', 'አመእበኣሰኣ', '##ደ', '##አረ', '##ኡ', '</w>', ' ', 'ገአለእጸ', '##አወ', '##ኣ', '##ለ', '##እ', '</w>']
          Display Tokens: ['ሰእወኢ', '##ደ', '##እነ', '##እ', ' ', 'ከአ', '##ኢተእየኦጰእ', '##የ', '##ኣ', ' ', 'ገኣ', '##ረ', '##እ', ' ', 'የኣለኣ', '##ተ', '##እነ', '##እ', ' ', 'ገ', '##እነ', '##እ', '##ኘኡ', '##ነአተእ', ' ', 'አሰእመአለእከእ', '##ተኦ', ' ', 'አደኢ', '##ሰ', '##እ', ' ', 'የአ', '##ተእበእ', '##በ', '##እረ', '##እ', ' ', 'ሰእለእ', '##ተእ', ' ', 'መአነአ', '##ደአ', '##ፈ', '##ኡነእ', ' ', 'አመእበኣሰኣ', '##ደ', '##አረ', '##ኡ', ' ', 'ገአለእጸ', '##አወ', '##ኣ', '##ለ', '##እ']
          Detokenized: ስዊድን ከኢትዮጵያ ጋር ያላትን ግንኙነት አስመልክቶ አዲስ የትብብር ስልት መነደፉን አምባሳደሩ ገልጸዋል
          vocab_size: 5000

[sefineh-ai]

V0.2.0 has been released