Pages randomly missing on some file systems, likely Unicode normalisation issue

[amandasystems]

I’ll try to hunt this issue down further, but I’m seeing issues where on some systems links and embedding of pages with Unicode characters don’t find the pages while the exakt same folder on other systems works.

I suspect the issue is that some systems (APFS on macOS) and the web front end performs Unicode normalisation of file names, while some platforms (ZFS on FreeBSD) does not at the file system level.

The correct thing to do is probably to normalise anything that hits the file system and also anything that comes from the file system when performing matching.

[zefhemel]

Ok. I'd need more specifics and examples of what those filesystems could be as well as example file names.

[amandasystems]

I figured you would; I’m working on a reproduction but it’s rather complicated since it requires two installations on two different file systems etc. The way I produced it was by cloud syncing my folder between a macOS and a FreeBSD system and using the character ”ä” in a file name. Others also work though.

What must have happened is that I have a file that contains a link to ä.md with an encoding that isn’t the same as it normalises to. On a normalised file system this link is ok, since the file system normalises the file name and so does match the file, but when the same folder is synced to a non-normalising file system the link is suddenly broken.

[amandasystems]

Ok, so what I get is this: if I create a link [[Ä]], Silverbullet encodes this as [[\\xc4]]. On macOS, this normalises to the file name \u0308.md, while standard Linux file systems (including ZFS in its default configuration) does not. This leads to a situation where if I use the link to create the file on a Linux machine, Silverbullet red-marks the link on macOS (since, I assume, no file with the exact same bytes in its name appears in the directory listing, nor can one ever appear since it will be normalised in the file listings from the OS), but the file resolves at the file system level due to the normalisation and the link works.

This seems to also affect page includes.

What I would expect is that Silverbullet would pick a normalisation to always use when creating files and go with it. Form D seems to be the most used one; it's the default in APFS and it's what users normally suggest for ZFS.

[mjf]

Yeah, I agree with @amandasystems and was also suggesting some similar approach elsewhere. Good UTF-8 support is absolutely essential thing! Not only should it work seamlessly for different filesystems but should also work for tags, attributes, queries etc. which does not (or at least fails sometimes) atm., also all regular expressions should support UTF-8, etc.

Here are funny examples:

• funny filename: w͢͢͝h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡͢t ̧̕h́o̵r͏̵rors̡ ̶͡͠lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡͟e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢̡rt́͏ ̴̷͠ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞ .md,
• funny tag: #<w͢͢͝h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡͢t ̧̕h́o̵r͏̵rors̡ ̶͡͠lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡͟e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢̡rt́͏ ̴̷͠ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞ >,
• funny attribute: [foo: w͢͢͝h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡͢t ̧̕h́o̵r͏̵rors̡ ̶͡͠lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡͟e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢̡rt́͏ ̴̷͠ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞ ].

I use XFS and have native UTF-8 support on Linux for so many years...

mkdir 'w͢͢͝h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡͢t ̧̕h́o̵r͏̵rors̡ ̶͡͠lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡͟e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢̡rt́͏ ̴̷͠ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞'
touch 'w͢͢͝h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡͢t ̧̕h́o̵r͏̵rors̡ ̶͡͠lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡͟e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢̡rt́͏ ̴̷͠ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞.md'
ls -la
drwxr-xr-x.   3 mjf mjf  4096 Jan 23 08:03  .
drwxr-xr-x. 263 mjf mjf 94208 Jan 23 08:02  ..
drwxr-xr-x.   2 mjf mjf     6 Jan 23 08:03 'w͢͢h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡t ̧̕h́o̵r͏̵rors̡ ̶͡lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢rt́͏ ̴̷ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞'
-rw-r--r--.   1 mjf mjf     0 Jan 23 08:02 'w͢͢h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡t ̧̕h́o̵r͏̵rors̡ ̶͡lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢rt́͏ ̴̷ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞.md'

@amandasystems But I would not suggest anything else than just support UTF-8 as well as possible and whenever some user input is involved! I do not thing it's good thing to "encode" the UTF-8 characters for specific filesystem. We should just escape special characters like / or \\ (this is usually OS-dependent) and that's all. Users simply want to see exactly the same names of files and folders they gave them when working with them, IMHO.

If some filesystem does not support UTF-8 names natively, then it's upon users to convert UTF-8 names to something their filesystem can understand. Usualy standard libraries take care of these things too, right?

[amandasystems]

@mjf It's a bit more complicated than supporting UTF-8. Every file system I know of supports UTF-8 in the sense that you can use UTF-8 characters in file names. The problem is that there are different approaches to how to implement it and how to split the responsibility for various complexities it gives rise to. The underlying problem is that the literal same character (not look-alikes, though those are also a problem) can be encoded in multiple ways, as with the letter "ä" in my example above. This means that string comparisons are now non-trivial, because the same string does not necessarily compare equal even if it renders equal. To fix this, strings are normalised to one encoding, which means that you get your nice properties back. Well, some of them.

The question is whose responisbility it is to normalise the strings that make up file and folder names, and the problem is that there are differing opinions on it. Linus Torvalds from what I've heard of the opinion that the bytes you write into the file system should be the ones you read out, i.e. that the two encodings of "Ä" are actually two different files, and it's the responsibility of any client application to handle any normalisation. macOS takes the opposite approach and normalises file names. I'm not aware of what Windows does and at this point I'm afraid to ask.

To make matters even worse, there is also differing opinions on how to normalise.

I tried to look into exactly what XFS does, and I think it performs some sort of normalisation but as far as I can tell it's not mentioned anywhere.

The problem with "users want to see" is see. You can't really see the difference. It's also possible and common that the file name you write to disk from the application, coming straight from the user, is not the one you will read back if you ask the file system for it, if the web browser and file system has different opinions on string normalisation.

What you must probably do is to pick one normalisation and always normalise all incoming strings into that (which, by extension, implies that you also output normalised strings), no matter if they come from the file system or the user. That way the file systems' opinions don't matter and comparisons are always consistent.

[mjf]

@amandasystems

Linus Torvalds from what I've heard of the opinion that the bytes you write into the file system should be the ones you read out, i.e. that the two encodings of "Ä" are actually two different files, and it's the responsibility of any client application to handle any normalisation.

I strongly agree with him and that's the only sensible approach I can imagine, I must say.

I tried to look into exactly what XFS does, and I think it performs some sort of normalisation but as far as I can tell it's not mentioned anywhere.

This is very unlikely. It will behave in the sense of how Linux is supposed to work (on Linux).

What you must probably do is to pick one normalisation and always normalise all incoming strings into that (which, by extension, implies that you also output normalised strings), no matter if they come from the file system or the user.

Wrong approach, IMHO. That will influence how the filesystem objects (folders, files, etc.) look like. If I want to create a directory called
/Příliš žluťoučký kůň úpěl ďábelské ódy!
I should be able to and see the very name using every tool in my OS, consistently. That's where standard libraries step in (or we already use UTF-8). Software should rely on OS. Nobody should implement any encoder/decoder. It makes no sense and it is error prone by nature of it. The Linux's approach is therefor the most straightforward, clean and "burdenless." It just does not encode/decode anything. And everything works as expected. We use UTF-8 primarily everywhere and most applications are either adopted to it or use standard libraries functions. It used to be an issue decade or two ago but it is no longer an issue. The Mac's (and I think that also the Window's) approach is outdated because they still assume things like ISO- encodings, I guess. Nobody should use those today! There is no need for it with proper support of UTF-8. The whole issue could be maybe delivered to the Deno developers, and if the fault is not on their side they can further deliver the issue to Mac standard libraries or whereever... dunno. I think all we need is to take care about escaping special characters in filesystem object names (if there are any in the OS), but that's because of slightly different logic of reasoning about it specific to how SilverBullet works. We wouldn't need to care about it normally... :-)

[amandasystems]

Wrong approach, IMHO.

Fair enough; that's an opinion you can have, but the problem is that a program that does not perform normalisation will break when it encounters a file system that does since it will write bytes for a file name to disk and expect to read the same bytes back, but will potentially get other, equivalent, bytes encoding the same string.

If I want to create a directory called
/Příliš žluťoučký kůň úpěl ďábelské ódy!
I should be able to

Sure, and everyone agrees on that! None of this will stop you from choosing any rendered string. The difference is in how that string is encoded as a file name, and if you can have different but identically-looking strings.

Software should rely on OS.

Sure, but this is about handling a problem that happens downstream from that decision, which is that OS:es do things slightly differently.

The Mac's (and I think that also the Window's) approach is outdated because they still assume things like ISO- encodings, I guess

No, It's all UTF-8. There are multiple UTF-8 encodings of the same string. It looks the same, but the bytes are different. All of them are valid Unicode, encoded as UTF-8. It's not outdated. Normalisation is very much current. It's also implemented in several Unix tools, including Perl.

[mjf]

@amandasystems

program that does not perform normalisation will break when it encounters a file system that does since it will write bytes for a file name to disk and expect to read the same bytes back, but will potentially get other, equivalent, bytes encoding the same string.

Inherently wrong filesystem approach. Yes, and I feel how hard would be to explain that to me who only knows Linux and everything works out of the box all the time. ;-)

Sure, and everyone agrees on that! None of this will stop you from choosing any rendered string. The difference is in how that string is encoded as a file name, and if you can have different but identically-looking strings.

Should not be encoded. If it is by your OS, then Deno should use standard libraries from the OS that would make it seamless to it, and thus to all applications written in it.

Sure, but this is about handling a problem that happens downstream from that decision, which is that OS:es do things slightly differently.

Should be part of, e.g. LibC to handle that for that OS. Nobody should care about it. You write UTF-8 file name to filesystem, you get UTF-8 filename if you read it back. Nobody should cope with these things, because (1) they are difficult to get right, (2) they MUST be the same everywhere (which excludes custom encoding attempts by nature - should be provided by the standard libraries and shared by all applications), etc...

No, It's all UTF-8.

OK, I don't know them at all. I trust You. :-)

There are multiple UTF-8 encodings of the same string. It looks the same, but the bytes are different.

So they are not the same and it's upon the user to know why she used either variant, I think. What can be something in one language can look the same in another but may not be the very same thing. Therefor Linus is right. It's upon the users to choose/know what the want to do. Well, we just settled on UTF-8 for obvious good reasons. But I can have a folder and files named in ISO-8859-2 in it, instead of UTF-8. I just switch and will see them OK.

So you say, some filesystems mangle with the binary representation of the UTF-8? But that would mean that when you copy files between such filesystem and normal Linux filesystem, you have to decode/encode it. That's quite insane! But if it's true, the only way I can see is to use some standard library functions, port them to the other OS and the application (SilverBullet) must be (somehow) aware of where it actually runs to do the transcoding. As I say, it's insane, and I didn't know of the situation before.

Idea: What about this - let SilverBullet make filenames base64 encoded (which is simple ASCII7 and may not cause any issues anywhere). Simple tools can be then written to replace common things like ls which then can be hooked to the specific directory from shell by user, if she want it. In SilverBullet there will be no change, except that we could have literary any names for "files", even containing path delimiters like About I/O optimization.md. There will be just thin layer for proxying back and forth. @zef ?

I'm not sure what's better. :-(

[amandasystems]

I want to stress that normalised strings should have the same semantic meaning and render the same pixels on screen. They will, by definition, look indistinguishable, but they have different representations. Remember that different applications will also encode strings differently, so a file name copied or manipulated via an application, even a command-line application, is not guaranteed to preserve representation in general. You can have a situation where you do e.g. ls | some-script | some-other-script and the file name will have changed and no longer resolve to the same bytes. I really think you want application-level normalisation for these reasons, even if your file system does not normalise. But sure, let's assume you want SilverBullet to preserve the binary representation of what you type or paste as input. You haven't stated a functional reason for this, except that it is aesthetically and/or philosophically pleasing for you, but let's go with it.

For the record, solving the problem at deno's or the libc's level is probably not a good approach since you want those to preserve representation in case you have an application where it matters. That's why SilverBullet needs to decide on whether it's one of the applications where this matters.

If you really want to preserve the binary format, I guess you could apply the "be liberal in what you accept and conservative in what you emit" approach and normalise file names in comparisons as a fallback. This means that when looking up [[Å]] you'd first look up the literal exact binary representation, then a normalised one. The only behaviour this would change on a non-normalising file system is that if you already had a Å.md in one of the encodings and wanted to create one in the other via a link [[Å]], the link would resolve to the one you already had, and you'd have to first create the file outside of SilverBullet using e.g. touch Å.md where you took care to paste the correct Å. Then your [[Å]] and your [[Å]] would link different files. I'm fairly confident this is a situation very few people, if any, would ever be in, and if you are you can pay for the slightly higher complexity because you're already doing Advanced Stuff.

The Base64 option would kind of work, kind of not. If you base64 encoded two strings with a different normalisation, you'd get different Base64 encodings. However, a Base64-encoded string would normalise the same when stored, so what you'd have implemented is a work-around where you nullify the normalisation performed at the file system level by essentially implementing a file system on top of the file system.

Update: to stress my point above, when I wrote this comment, the two Å:s were different. When I paste them into a website that prints out the escaped binary representation they are the same. Something in between, either GitHub, Firefox, or the other website, normalised them. Preserving binary encoding of Unicode is not easy.