Can you Ignore duplicate inodes?

[bootandy]

Firstly good job, pdu is really fast.

If you create a hard symlink like this: ln file hard_link it appears twice in pdu where as it is only taking up one 'space' on disk. Both file and hard_link have the same Inode - Hence I only 'count' the disk space for one of them. Therefore in dust I chose to ignore the duplicate file as removing either 'file' or 'hard_link' wouldn't recover disk space until they were both removed.

I'm curious to see how you would solve the above problem. I have tried and using channels (slow) and tried locks round a shared hashmap (even slower)

I'm in the middle of a dust re-write to see if I can make it faster based on some of your ideas. :-).

You can see me doing the check for duplicate inodes here:
https://github.com/bootandy/dust/blob/master/src/utils/mod.rs#L253

[KSXGitHub]

Firstly, thank you for creating dust, for despite its limitations, its way of representing data inspired me to create this tool.

I am well aware of pdu's limitations.

There are 2 main reasons why I decided not caring about hardlinks:

• There is no distinguishing between a hardlink and a main inode. I could make some arbitrary rules (such as the first inode is the main inode) but the parallel nature of pdu makes testing impossible.
• I haven't found a way to accurately demonstrate the existence of hardlinks other than assume 1 to be real and others to be fake (which is not completely true). Therefore, I decided that it is better to treat all hardlinks as equal.

I'm curious to see how you would solve the above problem

I'm in the middle of a dust re-write to see if I can make it faster based on some of your ideas. :-).

Not caring about hardlinks was a design choice, not a trade-off choice between speed and utility. But if, hypothetically, I want to detect hardlinks, I wouldn't aggregate all inodes into one HashSet because it would require Mutex lock, which would render parallelization utterly pointless. Instead, what I would turn DataTree::data into a Vec of inodes/sizes pair (by using rayon's into_par_iter().map().collect()), then aggregate and filter them later during visualization (which is single-threaded). Of course, this advice would only work with the data model that I chose (DataTree is a hierarchical tree, not a pseudo-iterator).

When you finish your rewrite, can you notify me? I want to update the benchmarks.

[KSXGitHub]

@bootandy 🎉 Congratulation on the new release! It seems that the new implementation improved a lot as it rises from second-last place to second place in my benchmark.

[KSXGitHub]

@bootandy Hello again! I have just released a new version of pdu (v0.20.0) with a flag --deduplicate-hardlinks to detect the hard links.

There are 2 main reasons why I decided not caring about hardlinks:

• There is no distinguishing between a hardlink and a main inode. I could make some arbitrary rules (such as the first inode is the main inode) but the parallel nature of pdu makes testing impossible.
• I haven't found a way to accurately demonstrate the existence of hardlinks other than assume 1 to be real and others to be fake (which is not completely true). Therefore, I decided that it is better to treat all hardlinks as equal.

I made this feature because I realized that the limitation in representation of hardlinks doesn't actually exist: I can treat all links to the same ino as equally real and not count them toward directory totals. A directory total don't need to be greater than the sum of its children, it only need to be greater than each individual child.

I'm curious to see how you would solve the above problem. I have tried and using channels (slow) and tried locks round a shared hashmap (even slower).

I've solved it. And pdu --dedupe-links is still faster than dust when there are only a few hardlinks. I haven't benchmark the case where every file is a hardlink though.

// pseudocode
use std::os::unix::fs::MetadataExt;
if metadata.nlink() > 1 {
    add_hardlink_to_the_hashmap_mutex();
}

[bootandy]

That's interesting, good find, I might look at using that soon.