One language can be written in multiple scripts, so it will be detected as a different
ScriptLanguage(language + script)
Uses alphabet_detector as a word separator + language prefilter.
Based on chars (1 - 5) and 1 word n-gram language model modified algorithm.
ModelsStorage with all models preloaded uses around 4.1GB of RAM. There can be a way (unimplemented) to unload each language model after use, it will work slower but will use around 300MB of RAM.
This library is a complete rewrite of Lingua: 5x faster, more accuracy, more languages, etc.
Comparison with other language detectors
To use it, you need to patch langram_models in Cargo.toml:
- From Git:
[patch.crates-io]
langram_models = { git = "https://github.com/RoDmitry/langram_models.git" }
- From predownloaded copy (langram_models):
[patch.crates-io]
langram_models = { path = "../langram_models" }
Which is more advanced and allows you to remove model ngrams, so that final executable would be lighter.