AMH-Tokenizer v0.1.2 – Syllable-Aware BPE Tokenizer for Amharic

[sefineh-ai]

AMH-Tokenizer v0.1.2 – A syllable-aware BPE tokenizer for Amharic (አማርኛ). Decomposes Fidel characters into base consonant–vowel components, reducing vocabulary size while preserving meaning. Supports tokenization with ## subword continuations, end-of-word markers , and spaces, detokenization, and custom training for NLP pipelines.

from amharic_tokenizer import AmharicTokenizer

tokenizer = AmharicTokenizer.load("amh_bpe")

text = "የተባለውን የሚያደርገውም በዚህ ምክንያት ነው"
tokens = tokenizer.tokenize(text)
decoded = tokenizer.detokenize(tokens)

print("Tokens:", tokens)
print("Decoded:", decoded)

---

Sample Output

Tokens:

['የአተአ', '##በ', '##ኣለ', '##አ', '</w>', ' ', 
 'ወእ', '##ነ', '##እ', '</w>', ' ', 
 'የአመኢየኣ', '##ደ', '##አረ', '##እ', '##ገ', '##አወእ', '</w>', ' ', 
 'መእ', '</w>', ' ', 
 'በአ', '##ዘኢ', '##ሀ', '##እ', '</w>', ' ', 
 'መእ', '##ከ', '##እነእ', '##የኣ', '##ተእ', '</w>', ' ', 
 'ነ', '##አወእ', '</w>']

Decoded:

የተባለውን የሚያደርገውም በዚህ ምክንያት ነው

---

✅ Notes:

• ' ' represents a space preserved as a token.
• </w> marks end of a word.
• ## marks subword continuation (merged syllables).

This shows round-trip fidelity: the tokenizer can break text into subwords and reconstruct the exact original sentence.

---

This discussion was created from the release AMH-Tokenizer v0.1.2 – Syllable-Aware BPE Tokenizer for Amharic.