Improving the "Remove duplicates" feature

[kotchin]

Thank you for your hard work on Ente. I recently discovered it and was quite quickly excited and enamored by the service. My experience has been positive so far, but I would like to touch on just one gripe I had with it:

I wanted to try out the "remove duplicates" feature, which I think worked as advertised (I think? more on that below) but I feel it can be improved.

Currently, the flow is simply the following:

1. Enter the "remove duplicates" option
2. See a list of duplicates and the folder in which each duplicate is located
3. View each duplicate if desired
4. Click the "remove duplicates" button
5. Done

This caused confusion to me for multiple reasons:

1. I was first unable to get any information on what is defined by duplicate. Is it two files with the same name? two files with the same filesize? does it take meta-data into account? Something more/else?
2. I was unable to influence which duplicate I wanted to keep when I saw the list of files and their corresponding folder. I try to have my folders organized so I had some intention of keeping a specific copy
3. Once the "remove duplicates" button was pressed, I had no idea which duplicates were actually removed. What logic did it use to decide which duplicate to remove? Was it only keeping the left-most item? It said all my duplicates were removed, but I can only go by its word as I did not witness the selection of the files to keep.
4. There was no way to revert it, because that button was final, yet the files are apparently stored in the trash and I'm told to clear the trash to claim the space back, so it would be logical to me that there's a way to revert it, but that's maybe not so important if all other concerns are solved.

There are likely several solutions to the above, but I can perhaps share what I think would have made my experience a bit better:

1. Place an "information" button somewhere, so when I click on it, it tells me exactly what information it uses and does not use to determine whether two photos are duplicates.
2. It actually highlights the item that will be kept/removed if I press the "remove duplicates" button
3. It allows me to select another item of the displayed duplicates to manually select which duplicate I want to keep, when I want to change the default selection
4. I would have liked to be able to view more information from the items concerned with the duplication removal, such as filename and exif data. What if one of the items has exif data and the other one does not? if they are considered duplicates, I would want to keep the one with the exif data for example.

Thank you for your time and effort! Best regards.

[vishnukvmd]

Thanks a bunch for the feedback!

For Ente, duplicates are exact duplicates - files who have the exact same bytes. So it takes metadata into account as well.

If you use our mobile app to remove duplicates, it will clear the extra files while retaining a symlink within all albums. So you don't have to pick an instance of the file to retain, your album structure will remain unmodified, but you will reclaim the space occupied by the duplicates.

Adding an "Information" button somewhere definitely makes sense, thanks for this feedback.

Let me know if you have more thoughts / questions!

[kotchin]

Thank you for your reply. That makes perfect sense and clarifies it.
Perhaps you already had this planned, but if you do add this "information" button, mentioning that symlinks are used to keep albums and photos untouched from a user perspective would make it perfect.

On a related note, if symlinks are part of the database, in the case of a "photos takeout", are symlinks provided too? Can one expect to be able to fully re-create the organized photos the exact way they are visible in ente-photos? Thank you!

[vishnukvmd]

Yes, export will reflect the exact album structure onto your local disk. Original files will be downloaded in place of their symlinks.

[gabeklavans]

I'm sure this is already planned, but it would be a great improvement to add a perceptual image hash feature to the deduplication flow. That way even "similar" looking images would be captured by deduplication. Using a method something like this https://github.com/pwlmaciejewski/imghash or something home-grown.

[sneak]

Seconding the request for a perceptual hash comparison. I have tons of non-exact-duplicate files that are very very close to the exact same image (but not pixel perfect - things like jpeg recompression). Identifying the duplicate images with a heuristic for detecting the "best" one (such as the one with the most complete metadata, or largest resolution, or greatest file size) and preselecting all others for deletion would be lovely.

[KestrelWild]

I could REALLY use an option to dedupe by filename alone without hashtag and preferring to keep the image with the larger filesize. I had original photos and the versions reduced by Google Photos. I thought the import would catch the duplicates, but it did not. So now I have a ton of duplicates and no way to remove them :(

[s38b35M5]

That is my use case as well. Most (but not all) of my original photos were uploaded to gPhotos, which reduced quality. After importing my takeout, I uploaded all of my original photos from storage thinking the dedup would catch the duplicates. Now I have between 20GB & 35GB of duplicates with the same filenames.

I didn't know which photos were already uploaded, and after hours of attempting to search by date, I just uploaded all. Reduplication found three duplicates out of hundreds. Now I have to scroll through twenty+ years of photos to compare size on each and every duplicate.