UnicodeError with os.listdir using nightly builds

[OlafM2015]

On an Rpi PIco, I used the uf2 v1.19.1 (2022-06-18) However, I want to use machine.lightsleep. So I start to use the nightly build eg. [v1.19.1-831]. The machine.lightsleep works nicely.

However, a get an UnicodeError using:

    print(os.listdir('/SD/MUZIEK/KLASSIEK/2'))

The files in this dir containts some special characters eg.: André Rieu Boléro.mp3
In a dir withno special characters there is no UnicodeError.
Could not find the solution.

With uf2 v1.19.1 (2022-06-18), there the special characters are handled and no UnicodeErrors occur.

What would be your advise, thanks

[dlech]

I'm guessing that when the file was created, a character encoding other than UTF-8 was used for the file name. MicroPython now checks that all strings are valid UTF-8. So if there is a way you can rename the file using UTF-8, that should fix the problem.

[OlafM2015]

OK, thanks, indeed renaming solved the problem, thanks

[jimmo]

Yes this is exactly the issue. We recently changed this behavior to explicitly raise UnicodeError.

MicroPython's runtime can only support utf-8 encoded strings. So the option we had was to either silently misbehave (and potentially crash), or just refuse to ever instantiate a string from invalid data.

I think in the case of the fileystem, there might be more we can do, especially for FAT filesystems which will use codepages to handle non-ascii characters. Not sure if it's in scope for MicroPython to implement more codepage to unicode handling (beyond what we already do).

[cullinwible]

I am running into this problem in RPI_PICO_W-20241007-v1.24.0-preview.385.g338df1ae3.uf2.

All filenames should be UTF-8, however, the device had a power failure (what appears to be while writing to the card) and now has invalid, inaccessible files on the SD card.

It looks like this was fixed in C-Python back in ~ 2009: "by the way of special escaping for undecodable characters (that is,
os.listdir() will always return strings rather than bytes objects, but some of those strings may have special escapes in them)."

python/cpython#47105 (comment)

Would it be possible to take a similar approach, perhaps replacing them with U+FFFD, the Unicode Replacement Character that should be used in situations such as this?

https://www.unicode.org/glossary/#replacement_character