Optimize import-url on directories, do not download existing files again

[shcheklein]

The use case is described here:

https://discord.com/channels/485586884165107732/485596304961962003/773882224248225812
https://discord.com/channels/485586884165107732/485596304961962003/776205388654968892

---

I have a bucket with images that is being updated (new images, etc) regularly:

gs://bucket/dataset/images

I don't want to use dvc add, primarily because I have other projects, websites using this bucket directly.
I still want to use these images in the DVC project, download them, download new, etc (basically do sync operation):

dvc run -n download_images 
              --outs-persist-no-cache dataset/images 
              "gsutil -m cp -n -r gs://bucket/dataset/images dataset/"
# -m -> multiprocess
# -n -> not downloading existing files (even though i wouldnt care because i pay the access anyway and the traffic is free) but its faster

This can be done with import-url. This is what import-url was made for:

dvc import-url gs://bucket/dataset/images

The problem is that import-url downloads all the images again on every change in the remote directory.

We should somehow utilize existing knowledge about the local cache to avoid downloading existing images and download only changes.

[aerubanov]

I`m looking at contribution on this issue. So, I check the source code and found that when user run dvc update to get new data:

• dvc firstly remove the stage output
• and then does a re-import of the data

I want to make changes to get this working principle when dvc update runing:

• check if stage dependency is directory. If so, than compare files`s md5 sums in dependency and output. Download files only if sums mismatch.
• If stage dependency is not directory, nothing not changes.

I will try to implement this and open a PR. But I`m not perfectly familiar with codebase, so @shcheklein and @efiop , if i missing something, pleas let me know

[aerubanov]

I tried to implement this optimization in PR #5428, buе faced with problem that remote storage and local machine can use different hash-functions for files. So now it is not always possible to compare files, and this optimization will have much more sense after migrating on unified hash functions with introdusing dvc/objects , as @efiop notice in #5428 (comment).