fun-langid: The code does lowercase, but that is a locale sensitive operation.

[mihnita]

This method:

def _normalize(self, line:str):
    return line.lower().replace('"', "'").replace("/", " ")

invokes .lower() on the input text, but that is a locale-sensitive operation.
The uppercase I (U+0049) converts i (U+0069) in all languages except Turkish and Azeri, where it should convert to dotless lowercase i (ı, U+0131).

So with the current code lowercase of "LARI" will be "lari", which does not exist in the Turkish n-gram, instead of "ları", which does exist.

This means that the recognition of Turkish and Azeri uppercase text will be problematic.

[icaswell]

Ah, this is indeed an annoying issue :) Thanks for pointing it out! I'm inclined not to worry about it, since this will probably not have a large effect on the accuracy of the model, and the goal of this model is simplicity over accuracy in any case.

[mihnita]

I can't think of a good fix.
But I agree, big chunks of uppercase text are pretty rare.