File encoding

[moi90]

It seems that the encoding of the TSV files generated is not necessarily UTF-8. My opinion is that this should be changed. However, which encoding is used?

[jiho]

Latin-1

[jiho]

But see ecotaxa/ecotaxa_front#256

[moi90]

So what is the safest way to proceed for now? Read and write as ASCII by default so the responsibility lies with the user? Unfortunately, Latin-1 is not a subset of UTF-8...

[jiho]

I know there are ways to snif encoding based on file content but don't know much more.

I just enforce Latin-1 for now and will switch to UTF-8 when we finally make the switch. And it will be incompatible with old files which will force people to upgrade, see the light and use unicode; and that's fine by me.

[moi90]

Yes, maybe I can also include my encoding detection logic here that I use in morphocut. I don't want to include it directly into read_tsv because I don't want to lose the ability of pandas.read_cvs to directly access compressed files, URLs, etc.