Cleanup Dictionnaries - Reduce False Positive

[Terrtia]

Methodology:

graph TD
    A[Language Dictionary] --> A1[Normalize Dictionary<br>Sort and lowercase all entries]
    A1 --> B[Remove words not in kaikki.org dictionary]
    B --> C[Use GlotLID on removed words<br>Re-add if detected language matches]
    C --> D[Remove person/fictional names]
    D --> E[Remove common interjections<br>used in multiple languages]
    E --> F[Remove country names<br>used in multiple languages]
    F --> G[Cleaned Language Dictionary]

Loading

Some names were also present in kaikki.org dictionaries. I manually removed some of them, but some might still remain.

I noticed that the Serbian dictionary isn’t Serbian but Serbo-Croatian

https://kaikki.org/index.html
https://github.com/cisnlp/GlotLID

[pierotofy]

Hey @Terrtia thanks for the PR. This looks like a good effort. When you mention you've reduced false positives, have you benchmarked the results? Can you share those results and your methodology?