Skip to content

README.md

Latest commit

 

History

History
54 lines (37 loc) · 1.76 KB

README.md

File metadata and controls

54 lines (37 loc) · 1.76 KB
Raw

Language Detector

Character trigram-based language identifier for 6 European languages (German, English, French, Italian, Dutch and Spanish) using Lidstone smoothing and log-probability scoring.

Overview

Given any input text, the system identifies which of the 6 supported languages it belongs to. It achieves 99.89% accuracy on a test set of ~60,000 sentences.

How it works

  1. Preprocessing — lowercasing, digit removal and whitespace normalization
  2. Feature extraction — character trigram generation using NLTK
  3. Training — trigram log-probability model per language with Lidstone smoothing (λ=0.3)
  4. Inference — language with the highest cumulative log-probability score is selected

Languages supported

Code Language
deu German
eng English
fra French
ita Italian
nld Dutch
spa Spanish

Results

Model Accuracy Errors
Trigram LID (n=3, λ=0.3) 99.89% 65
Trigram LID (n=4, λ=0.3) 99.97% 19
Linear Interpolation (n=3, α=0.99) 99.89% 64

Requirements

pip install nltk scikit-learn pandas seaborn matplotlib ipywidgets

Usage

Open and run LanguageDetector.ipynb in Jupyter Notebook. The notebook includes an interactive demo widget where you can type any text and see the detected language along with the full score ranking.

Dataset

Training and test data from the Leipzig Corpora Collection:

  • 30,000 sentences per language for training
  • 10,000 sentences per language for testing

Authors

Javier Aranda, Anna Galstyan — Universitat Politècnica de Catalunya (UPC)
Course: Processament del Llenguatge Humà, 2025–2026