consider camino opt-in feature

[cgwalters]

I was thinking of at least adding a feature that adds a dependency on https://docs.rs/camino because IMO really no one should be using non-UTF8 paths for their config files. And commonly one wants to do things like print the file paths to the terminal, and it's just way easier to do that stuff if one knows the path is UTF-8.

[nabijaczleweli]

Paths are defined as byte strings. In most cases, they're locale-encoded (that isn't a given, natch, but it usually holds). That locale doesn't always encode in UTF-8, because it doesn't have to. Don't bite users in the ass just because they named their override file in their native language and they're using a locale with a different encoding than you like. If, in your opinion, calling a config file 10-тяжесть.conf should be invalid just because you're using KOI-8, then that opinion is wrong.

Also, I don't get why you'd want this. It's another dependency, and we don't lose anything by supporting all paths. (Actually, I just looked, we do entry.file_name().into_string() for some reason. This, again, is wrong, and the return signature should be BTreeMap<OsString, PathBuf>. So I guess we do lose something, and that something is 20 minutes needed to make this correct.)

[nabijaczleweli]

#21

[cgwalters]

Also, I don't get why you'd want this. It's another dependency, and we don't lose anything by supporting all paths.

I think it'd be an opt-in feature. The text said this, but I've clarified the issues title.

If, in your opinion, calling a config file 10-тяжесть.conf should be invalid just because you're using KOI-8, then that opinion is wrong.

The thing is that dealing with non-UTF8 can easily pervasively affect the codebase. One example I've hit is that in e.g. rpm-ostree we convert some file paths into systemd tmpfiles.d snippets. But systemd is only expecting UTF-8 there - realistically ASCII even. Basically file paths often end up in file contents or streaming data like terminals, and dealing with mixed encoding in those cases ranges from extremely painful to impossible in practice.

I understand your PoV though, and again I think this would be an opt-in feature.

[nabijaczleweli]

If it's opt-in interface swap then whatever, right, but "not being correctly locale-aware" means you're broken, not that you're doing something right.

In particular, since you mentioned teletypes, they're uniquely positioned to only enter and display text in a format that they understand. A KOI-8 teletype doesn't magically stop being KOI-8 because you gave it UTF-8, just like an IBM 2741 doesn't magically start understanding ASCII because you turned off mode 400 with stty. This is precisely why we have locales, and why V2 did (I'm overgeneralising "locales" to "encodings" here, but the latter is part of the bundle in the former nowadays, so it's sound) – I refer you to my monograph on teletypes, page 11.

If you use any other multibyte encoding, this all (again also by definition IIRC?) just works. systemd-tmpfiles splits a byte string on tabs and newlines, and passes some fields verbatim to mkdir(2). You can put GB18030 in there just as well as UTF-8 or KOI-8 there, and so long as you (systemd) don't try to parse it as text, which paths are not, it just works. If you have the requirement that it's text, then you just have to match the encoding to the locale, which you already must do to correctly process all other data. systemd doesn't have this luxury, normal userland tooling does.