support for invalid UTF-8 bytes

[jtracey]

There are some combinations of options that require ls to display non-UTF-8 byte sequences in a grid. Because this crate only supports String (and spaces), this isn't currently possible. Support should be added for invalid UTF-8 somehow, accepting one of &[u8], OsStr(ing), or AsRef<OsStr>.

[egmontkob]

A fundamental problem here is that there's no universal agreement how terminals (or, in fact, any other UTF-8 device) should handle invalid UTF-8.

Markus Kuhn's test file suggests that each run of characters that form an incomplete UTF-8 (but a valid prefix of a UTF-8 char) should be replaced by a single replacement character; although mentions in the last paragraph before the tests that other strategies are also possible.

WhatWG allegedly says to replace each byte by a replacement character, i.e. incomplete UTF-8 (but valid UTF-8 prefix) might be replaced by multiple replacement symbols. The VTE terminal emulator engine (GNOME Terminal, GNOME Console, Ptyxis, ...) implements this behavior, and includes a modified variant of Markus's test file, with accordingly adjusted margins and an additional paragraph explaining the situation.

Some other circumstances where there's no single correct behavior are: overlong sequences, UTF-8-encoded UTF-16 surrogate characters, UTF-8-encoded 5-byte and 6-byte sequences for characters U+110000 and above which used to be part of Unicode but no longer are.

The behavior in all these cases indeed varies across terminals.

It's not uncommon either to resynchronize too late, dropping a valid single-byte ASCII character right after the invalid sequence (although arguably this is a bug in such terminals).

I'd conclude that the behavior of UTF-8 terminals upon encountering invalid UTF-8 is, by definition, undefined behavior, thereby the concept of measuring the terminal width of invalid UTF-8 sequence does not make sense.

I don't know what those "some combinations of options [of] ls" are that the original report is talking about, but I'd argue that that option of ls should be revised: fixed or removed. ls is not a tool whose output is supposed to be piped to other cmdline tools. It's meant to produce human-readable output in terminals, often in a nicely formatted manner. Emitting invalid UTF-8 when the locale specifies that the terminal speaks UTF-8 cannot be proper behvaior, or at the very least cannot be a behavior where ls can control the layout.

[jtracey]

ls is not a tool whose output is supposed to be piped to other cmdline tools.

I'm not sure where you got that impression, but that's incorrect. The --literal flag exists for a reason (as do many other flags ls supports, such as --zero and --dired), and our opinions on proper handling of character encoding for display don't really matter. GNU is considered ground truth for uutils' coreutils intended behavior, so any criticisms of how ls is used should be brought up with them. We're still failing compatibility with GNU, in part because I can do things like this:

touch "$(printf '\xC0.file')" && diff -q <(ls -w 2 --literal) <(cargo run ls -w 2 --literal)

This is specific example would be fixed by resolving this issue.

[egmontkob]

Pretty much any time anyone brings up a question on some forum about parsing / postprocessing ls's output, the response there is that ls is not the right tool for that, it was primarily designed to produce human-readable output. It's usually recommended to use tools like find, stat etc. instead. I am fairly certain this has come up on the GNU Coreutils mailing list, too. I'm not going to look up examples.

However, you are absolutely right that there are some options of ls designed to make its output parseable, e.g. --literal and --zero, I have indeed missed them.

But that doesn't change the essentials of my point, namely that emitting potentially invalid UTF-8, and trying to count the terminal columns, are fundamentally incompatible.

Based on the circumstances, ls has to pick at most one of these two:

• print raw, unchanged filenames, i.e. potentially invalid UTF-8 output even if the locale is UTF-8,

• produce nicely formatted, aligned output where the terminal widths of fragments of the output (e.g. a filename) does matter.

You can't have both at the same time. It's UB, and accordingly, the actual behavior differs across terminals.

If you changed the method signature as suggested in this issue, it would be semantically incorrect, and whatever implementation you came up with would be exact in a random subset of the terminals and an approximation in others.

If raw filenames need to be printed, it's fundamentally incompatible with trying to wrap after a certain number of visual columns. You're free to print the data literally, but then you cannot count the columns, thus you shouldn't call the said method either.

If you need to keep track of the columns, then first you need to ensure the output is UTF-8, using whatever strategy for escaping or substituting with replacement symbols, and then you can measure the width with the already existing method.

---

We're still failing compatibility with GNU, in part because I can do things like this:

touch "$(printf '\xC0.file')" && diff -q <(ls -w 2 --literal) <(cargo run ls -w 2 --literal)

This is specific example would be fixed by resolving this issue.

What problem do you see here? I've tried it in a new empty directory, containing only that single file this command creates. I've replaced cargo run by the full path to the uutils executable, and added the GNU full path to the other. It works fine: no output or error.

You using the command cargo run suggests to me that you're running this from the project's source tree, containing many files. Could that be the case? The two ls implementations use different sorting order, this results in different output. Irrelevant red herring.

---

I've examined the output of the ls commands in that example. That is, touch "$(printf '\xC0.file')" followed by ls -w 2 --literal. Both with GNU and Uutils, both at git master/main HEAD as of yesterday. Always with full path to the binary, to exclude shell alias. Tried printing to the terminal (catching the data using script) as well as redirecting to a file. The behavior is identical in GNU and Uutils:

If the output goes to a terminal, invalid UTF-8 bits are replaced by ? characters by ls itself. That is, as ls is trying to format to the given width, it always deals with valid UTF-8 strings. (Unless it first measures the width somehow, and then does the ? replacement, which would be very stupid – I haven't checked but I have no reason to doubt it does things in the right order.)

If the output does not go to a terminal (as in your example with diff <()), the specified width (-w) is ignored. That is, the method you're trying to "fix" isn't (or at least shouldn't be) called.

I currently see no issue with ls here, its behavior is fine. And don't see the need for measuring the width of a potentially invalid UTF-8 string. The example you've given does not justify this very feature request / bug report, because, depending on the circumstances (terminal or not), either the data whose width you measure is valid UTF-8, or no such measurement takes place. And that's how it should be.

I'm still absolutely positive that this feature request / bug report is conceptually wrong. If you ever find yourself in the situation where you need to know the width of a potentially invalid UTF-8 string then you've already gone wrong beforehand.

There are some combinations of options that require ls to display non-UTF-8 byte sequences in a grid.

Could you please show such combinations of options, along with other relevant circumstances (most notably: tty or not)?

I don't think ls -w 2 --literal is a good example of this. Depending on tty or not, it either always emits UTF-8, or doesn't need a grid. I presume your diff got confused by the utterly irrelevant issue of different sorting order.

If there is indeed any such circumstance in GNU ls where they try to measure the width of invalid UTF-8, I'd be happy to discuss this with them first.

[egmontkob]

To be honest, I also have doubts about a bigger design decision here.

This same bugreport is forwarded to the underlying ansi-width library (or crate or whatever Rust calls that), whose role is to measure the width in terminal, recognizing a few simple escape sequences (most notably SGR Select Graphic Rendition).

This is a noble goal, I myself have also found myself looking for such a library on a few occasions. It's surely a great, useful library, for when you're given a string that already has SGR color codes.

But... why would it be useful for ls?

If ls itself colors its output then adding the colors first, and then calling a function which skips them, sounds like wrong stacking order to me. Measure the width first, add the colors later. That way a simpler, lower-level library, like unicode-width as mentioned on the ansi-width page, is sufficient.

The other use case is if you do have a filename that contains SGR escape sequences, e.g. after a

touch $'\e[31mLittle Red Riding Hood\e[m'

and you do want to print it as-is including the control characters (i.e. to appear in red), and you do want it align up nicely with all the other filenames.

Which is a highly questionable goal. You need to pass control characters through, and at this point they're not limited to SGR, they can do much nastier operations in the terminal, so you've lost any hope of maintaining a grid layout, unless you count on the user to play "nicely" and only have these kinds of simple SGR-like control characters in filenames. Additionally, I don't think anyone has such filenames and counts on the colors working in them. I believe it's a feature that can't be implemented properly (it'd only work for a highly limited subset of escape sequences, and heavily break for the rest), and a feature probably no one is looking for.

To be fair, it might be somewhat irrelevant here. If you went with a lower level library, e.g. unicode-width rather than ansi-width, the question of invalid UTF-8 would still remain. I don't know what unicode-width does there, but my opinion is surely the same: it only makes sense for such a method to operate on valid UTF-8.