add option to disable checksum verification#153
Closed
einsteinx2 wants to merge 1 commit intopauldreik:mainfrom
Closed
add option to disable checksum verification#153einsteinx2 wants to merge 1 commit intopauldreik:mainfrom
einsteinx2 wants to merge 1 commit intopauldreik:mainfrom
Conversation
|
Thanks @einsteinx2 for your option. I have patched rdfind 1.6 with your suggested changes and it works like a charm. Thanks. |
Owner
|
Thanks for the PR! I made it possible disabling the checksumming in #207 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #118
I saw that there was a previous PR from a few years ago addressing this that was never merged. This is an updated solution based off of the top of main, and is a simpler implementation with fewer changes.
Tested on my Debian 12 home NAS server and it behaves as expected.
The drive I was checking is over 16TiB of usable space with over 500k files. It took overnight to run without doing a checksum (only checking names/sizes and then following up with first/last bytes). If I had checksumming on, it would still be running, and for days most likely. The output without the checksum was still very useful as I'll explain below.
To the maintainer(s), I think there is a real use case for this:
In my example, I have a NAS with dozens of TB of storage. I have a lot of files that are significantly large (dozens to hundreds of GB). I've been doing some file reorganization and my goal is just to get a short list of files that might be duplicates that I can then process later.
In my case, if these large files have the same name and size and first/last bytes, I can be basically 100% sure they're the same file. It would take exponentially longer to checksum every file at discovery time only to find out what I already know...they're the same.
Later if needed I can manually checksum files in the results list, or even do something like test/checksum random byte ranges in the files rather than checksumming the whole thing to save a ton of time, but in my case (and I assume others' as well) I already know they're the same so even that isn't needed.
To be clear, I think the default should stay the same (or maybe even go up to sha256/512), but at least having the option to disable checksumming can be very useful and the change is only a few lines.