dvc add for remote repository (model registry) - GTO

[fortminors]

Hello! It would be quite useful if it was possible to track new artifacts through dvc remotely.

What I mean is the following situation:

1. I have set up a model registry as in here with DVC + GTO
2. Let's say I have the following structure of artifacts and I am working inside this repository:
   $ gto show:

╒═══════════════════════╤══════════╤═════════╕
│ name                  │ latest   │ #prod   │
╞═══════════════════════╪══════════╪═════════╡
│ detection             │ v0.0.1   │ v0.0.1  │
│ segmentation          │ v0.0.2   │ v0.0.2  │
│ classification        │ v0.0.1   │ v0.0.1  │
╘═══════════════════════╧══════════╧═════════╛

3. Now I want to completely change my working directory and work from another place on my PC: cd path/to/other/dir (or even on another machine)
4. I would like to be able to push new artifacts to the model registry that I have created before from the current working directory

Most of the methods in GTO have the optional flag: -r <text>, --repo <text> - Local or remote repository [default: .], which is quite handy.

It would be super useful to make a similar flag for DVC methods (at least for add).

Right now to accomplish this task, I have to do the following:

1. Clone the model registry repository to a temporary location
2. Copy my newly created artifact, say new_model_det.pt, to the cloned repository
3. Track via regular dvc add, gto annotate, dvc push and git push

[dberenbaum]

Hi @fortminors, are the new artifacts updated versions of old artifacts in the repo, or are they completely new artifacts? Are they related to the other artifacts in the repo? Do you actually want them to be part of the existing repo, or are you only really interested in them being part of the model registry?

[fortminors]

Hello! @dberenbaum The new artifacts are the updated version of the previous ones, as in I am retraining the model and pushing it. I also add a timestamp, so in the sense of naming, these are new files.
I am interested in them being part of the model registry. So I can operate on them through dvc and gto, eg. dvc get <repo> <artifact_name>

[dberenbaum]

Do you have separate repos for model training and model registry? If so, do you mind if I ask why?

@aguschin I think we have some related discussions about how to "publish" artifacts, right?

[fortminors]

Yes, this is correct. Basically, for every model I have, I made a separate repository to encapsulate the logic inside (ie train, test, export, etc). Then at the final step for each model, I upload it to model registry, which acts as a storage and quick access place for the end user (ie dvc get from model registry).

[aguschin]

Yep, we discussed this in iterative/gto#307 (and earlier) with @francesco086
Since we'll be moving artifacts.yaml part from GTO to DVC #9219 it can make sense to discuss this re this future workflow.

[fortminors]

Thanks! Looking forward to it.