Character trigram-based language identifier for 6 European languages (German, English, French, Italian, Dutch and Spanish) using Lidstone smoothing and log-probability scoring.
Given any input text, the system identifies which of the 6 supported languages it belongs to. It achieves 99.89% accuracy on a test set of ~60,000 sentences.
- Preprocessing — lowercasing, digit removal and whitespace normalization
- Feature extraction — character trigram generation using NLTK
- Training — trigram log-probability model per language with Lidstone smoothing (λ=0.3)
- Inference — language with the highest cumulative log-probability score is selected
| Code | Language |
|---|---|
deu |
German |
eng |
English |
fra |
French |
ita |
Italian |
nld |
Dutch |
spa |
Spanish |
| Model | Accuracy | Errors |
|---|---|---|
| Trigram LID (n=3, λ=0.3) | 99.89% | 65 |
| Trigram LID (n=4, λ=0.3) | 99.97% | 19 |
| Linear Interpolation (n=3, α=0.99) | 99.89% | 64 |
pip install nltk scikit-learn pandas seaborn matplotlib ipywidgetsOpen and run LanguageDetector.ipynb in Jupyter Notebook. The notebook includes an interactive demo widget where you can type any text and see the detected language along with the full score ranking.
Training and test data from the Leipzig Corpora Collection:
- 30,000 sentences per language for training
- 10,000 sentences per language for testing
Javier Aranda, Anna Galstyan — Universitat Politècnica de Catalunya (UPC)
Course: Processament del Llenguatge Humà, 2025–2026