new command: put-url OR rsync/rclone

[casperdcl]

Summary

An upload equivalent of dvc get-url.

We currently use get-url as a cross-platform replacement for wget. However, together with get-url, put-url will turn DVC into a replacement for rsync/rclone.

Motivation

• we already have get-url so adding put-url seems natural for the same reasons
• put-url will be used by
  • CML internally to sync data
  • LDB internally to sync data
  • the rest of the world
• uses existing functionality of DVC so should be fairly quick to expose
• cross-platform multi-cloud replacement for rsync/rclone. What's not to love?
  • could even create a spin-off thin wrapper (or even abstract the functionality) in a separate Python package

Detailed Design

usage: dvc put-url [-h] [-q | -v] [-j <number>] url targets [targets ...]

Upload or copy files to URL.
Documentation: <https://man.dvc.org/put-url>

positional arguments:
  url                   Destination path to put data to.
                        See `dvc import-url -h` for full list of supported
                        URLs.
  targets               Files/directories to upload.

optional arguments:
  -h, --help            show this help message and exit
  -q, --quiet           Be quiet.
  -v, --verbose         Be verbose.
  -j <number>, --jobs <number>
                        Number of jobs to run simultaneously. The default
                        value is 4 * cpu_count(). For SSH remotes, the default
                        is 4.

How We Teach This

• Name:put-url seems to be in line with the existing get-url (vis. HTTP GET & PUT)
• Idea presentation: continuation of existing DVC patterns
• Docs: simply add https://dvc.org/doc/command-reference/put-url largely based off the existing https://dvc.org/doc/command-reference/get-url
• Teaching: not required

Drawbacks

• can't think of any

Alternatives

• would have to re-implement per-cloud sync options for CML & other products

Unresolved Questions

• minor implementation details
  • CLI naming (put-url)?
  • CLI argument order (url targets [targets...])?
  • Python API (dvc.api.put_url())?

Please do assign me if happy with the proposal.

(dvc get-url + put-url = dvc rsync :))

[dberenbaum]

(dvc get-url + put-url = dvc rsync :))

Does it make sense to have a separate put-url command strictly for uploads, or would it be better to have a combined transfer command? This has both UI and usage implications, and a combined transfer command could enable:

• Transfers that never touch local disk. Downloading files just to upload them could be a waste of time or even impossible if disk space is insufficient.
• dvc get equivalent functionality where remote data in dvc/git (for example, a data/model registry) can be transferred to cloud destinations without cloning the repo or pulling the data.

* Python API (`dvc.api.put_url()`)?

dvc get-url and dvc.api.get_url() don't really do the same thing unfortunately, so I'm unclear what dvc.api.put_url() would do? That confusion might be another reason to prefer a two-way transfer command over separate download and upload commands.

[casperdcl]

dvc get-url and dvc.api.get_url() don't really do the same thing

👀 well that sounds like a problem

would it be better to have a combined transfer command?

Maybe this wouldn't be too difficult to implement. Summarising what I think the most useful rsync options may be:

dvc rsync [-h] [-q | -v] [-j <number>] [--recursive] [--ignore-existing]
          [--remove-source-files] [--include <pattern>] [--exclude <pattern>]
          [--list-only] src [src...] dst

We've endeavoured to provide buffered read/write methods for all remote types so that we can show progress... So simply gluing said BufferedIO methods together should provide the desired functionality.

[dberenbaum]

👀 well that sounds like a problem

😬 Yup, I only realized this pretty recently myself. Extracted to #6494.

[dmpetrov]

The initial motivation for this command was to upload data to storage and preserve a pointer file (*.dvc file) to it for future downloads. This command (and its download equivalent) are supposed to work from no-dvc repositories - it means, no dependencies to .dvc/ dir.

It seems like, we need an "upload equivalent" of import-url, not get-url.

Proposed names: dvc export url [--no-exec] FILE URL and renaming some of the existing commands:

• dvc import-url rename to dvc import url
• dvc import rename to dvc import dvc

--no-exec option is needed for the cases when storage credentials are not set. It means not to upload/download (and not checking for the file existence), only generating a pointer file. In the case of downloading, the pointer file should have empty checksums.

New commands:

• dvc export url FILE URL - the simplest way to upload a file and preserve a pointer file to it
• dvc export model FILE URL - with a set of options to specify meta info for the model (description, model type, input/output type, etc)
• dvc export data FILE URL - with a set of options to specify meta info for the data (description, data type, column name for structured, classes distribution for unstructured, etc... )
• dvc export dvc FILE URL (do we need this one?)
• dvc import model URL FILE
• dvc import data URL FILE

PS: it is related to the idea of lightweight model and data management