Duplicates when moving photos between libraries

[dylexrocks]

I use icloupd in docker to backup my personal library and the shared library I have with my wife. I run a single container that syncs once every day and downloads both libraries into separate folders.
However I noticed, that when I move a pic from one library to another, after it had already been downloaded by icloudpd, it will be downloaded again to the folder of the library where I moved it to. But it will not be deleted from the origin library folder.
I assume this happens because the pic is not in the "Last deleted" album?
Is there a way to make that work, that already downloaded files, that are moved to another library are deduplicated?

Sorry if this question has been asked again, I looked through the issues and discussion but did not see this question before.

[AndreyNikiforov]

icloudpd sees snapshot of your iCloud the same way you see iCloud through web site. Moving images around does not leave a trace and detecting the move by comparing contents is not implemented. As a result, moving image around albums will create duplicates on your local storage. I assume the same is true for moving between personal and shared folders.

[dylexrocks]

Thanks for the confirmation. So in order to get rid of the duplicates in the current state it would require to wipe both library download directories and to a full sync from time to time, right?

Just thinking if it would be enough if I could identify the duplicates with maybe some external tool and then delete them in both directories so they get redownloaded to only the one where they currently are located in my library.

[AndreyNikiforov]

Thanks for the confirmation. So in order to get rid of the duplicates in the current state it would require to wipe both library download directories and to a full sync from time to time, right?

Just thinking if it would be enough if I could identify the duplicates with maybe some external tool and then delete them in both directories so they get redownloaded to only the one where they currently are located in my library.

Full re-sync sounds like a nuclear option to me. There are plenty of open source and commercial products that can compare to folders and find duplicates.

I would also suggest looking a bit bigger: what is your final objective? Do you use any photo organizing tool over all of your assets? That tool may detect and hide duplicates for you. If your collection is enormous and removing duplicates is essential to controlling cost, then filesystem with automatic de-duplication may be an option to consider for the next upgrade. And so on...

[dylexrocks]

You are right, deleting everything is a bit drastic. I would maybe do it once a year or so. But I guess the better approach would be the second idea. I am thinking about putting together a quick dotnet console app, that does exactly what I need and removes all duplicates, so they get redownloaded again to the right place. This way I could automate the duplicate removal and the overhead of having to redownload an already downloaded file is not too much at once.

My use case currently is only having a backup in case I lose access to my iCloud account for whatever reason. All my photos are in there and it would be devastating to lose them. So my two concerns about the duplicates are wasted storage, which might grow a lot over time and in case of a disaster recovery I would have to manually identify which photo belongs where.

Thanks again for your insights.