Add support for normalizing / hiding hardlinks in the output image

[cipriancraciun]

At the moment, mkdwarfs faithfully reflects the hardlink relation between files from the source file-system (i.e. if file /a and /b are hardlinks to one-another, so they appear in the image.dwarfs when mounted with FUSE). (And this is great.)

However, there are use-cases when one wants to break (or tweak) this relationship in the resulting image, namely:

• (A) either having all files appear with exactly 1 hardlink, although underneath in the resulting image they share exactly the same content;
• (B) either having all duplicate files (based on the hash) in the source file-system appear as hard-links in the resulting image;

---

Why would case (A) (i.e. hardlinks are dissolved) be useful? The hardlinks in the source file-system might be an implementation detail / optimization, and the user wouldn't want that fact shown in the resulting image.

For example there are techniques of using rsync to create snapshots back-ups that use hardlinks. However, the user might want to create a final image, and would like to hide this fact away.

Why would case (B) (i.e. all duplicate files to become hard-links to each-other), some other times, it's beneficial to have deduplicated files share the same hard-links, as this would allow more advanced tools to take benefit of this information in how they operate. (Again, rsync might be one such tool.)

---

As a side-note to point (B), how does DwarFS handle files that have more than 65536 hard-links?

(Where would one encounter such a case: MAME supporting files, namely cheats which are XML files, and for some reason, the developers of MAME decided to copy the same hash-duplicate file ~120K times in the same folder...)

[mhx]

Off the top of my head, (A) seems harder than (B). (B) might be trivial, since duplicate files are already "linked" internally. (A) is something I'd have to think about; presumably the solution would be to build a copy of the dir_entries table with patched inode numbers (if that's all it needs, it shouldn't actually be too hard).

As a side-note to point (B), how does DwarFS handle files that have more than 65536 hard-links?

As long as we're not talking about more than 4,294,967,295 hardlinks, I think DwarFS wouldn't really care. I think FUSE would support 32-bit hardlink counts as well.

[cipriancraciun]

Off the top of my head, (A) seems harder than (B). (B) might be trivial, since duplicate files are already "linked" internally.

I would assumed that option (A) (i.e. having all files appear with 1 hard-link in the image) might have been the simplest choice, meanwhile option (B) I would have assumed would require some extra bookkeeping to identify identical (but not hardlinked in the source) files.

Perhaps there would also be an option (C): have all files that share the same content have the same inode, thus be hardlinks, but when stat-ing it (or perhaps in the data structure inside the image) "lie" that it has only 1 hardlink count.

---

I can't say which is the most useful variant: (A) (or (C)) that make all files seem to have 1 hardlinks, or (B) that is transparent of the fact which files share exactly the same content.

I think there are use-ases for both (A) and (B).

[mhx]

I think there are use-ases for both (A) and (B).

My most important question would be: are there actual use cases? These features can certainly be added, but it all takes time from a budget that is currently only mine. There are features that I think would be useful to a much wider audience than this, and I'd much rather not get distracted by features that are "nice", but that not a lot of users actually need.

I'm happy to be convinced otherwise, but I wouldn't currently prioritize this.

[cipriancraciun]

My most important question would be: are there actual use cases?

I believe there are certain use-cases, but I think these would lie under the category of "mostly obscure usages"...

However, I think implementing option (A) (i.e. all files have exactly one hardlink) might be very useful if one cares about reproducibility of the output image, regardless of how the source folder has been composed from mismatched hard-links.

Meanwhile option (B) (i.e. all files with exactly the same content should be hardlinks) might be very useful for those that use DrawFS as an archival file-system of multiple snapshots that might share the same files over-and-over again. Because it exposes to the user (or tools that the user might use) which files are related, and allows him to easily identify which files are unique and could be perhaps copied, verified, etc.

That being said, I would say this feature isn't blocking some use-case, it's more useful for those that deeply care about having very well organized images.

I'm happy to be convinced otherwise, but I wouldn't currently prioritize this.

With regard to the prioritization, perhaps this isn't such a high priority, and you could leave it as open for whenever there is some spare time.

In fact, if I would to prioritize this issue vs #350 (the "follow the symlinks"), that other issue is more useful as it unblocks some use-cases.

[mhx]

Meanwhile option (B) (i.e. all files with exactly the same content should be hardlinks) might be very useful for those that use DrawFS as an archival file-system of multiple snapshots that might share the same files over-and-over again. Because it exposes to the user (or tools that the user might use) which files are related, and allows him to easily identify which files are unique and could be perhaps copied, verified, etc.

This is exactly one of those features I was talking about.

Having the ability to incrementally add snapshots to an existing DwarFS image and de-duplicating across snapshots has been on my mind for years. Alas, I need more time. :-)

I'd rather support this use case properly than with band-aids.

[cipriancraciun]

Having the ability to incrementally add snapshots to an existing DwarFS image and de-duplicating across snapshots has been on my mind for years. Alas, I need more time. :-)

Just a few moments ago I was thinking at my Restic backup repository, and how if I were to mount it (via FUSE), and point mkdwarfs at it, I could have a single image (that I assume would have roughly the same size as the entire Restic repository), that holds each and every snapshot I took in that repository. (The advantage of DwarFS over Restic would be increased access speed if I were to make lots of read operations on that file-system.)

However, at the moment Restic doesn't expose identical files as deduplicated hardlinks (I think), thus if I were to look at my DwarFS clone of this large Restic repository, it wouldn't help me much to see which files are truly unique and should be indexed, re-saved, etc.