Using `dvc exp` workflow together with MR

[aguschin]

We'd like to understand how would a MR workflow look like considering dvc exp approach to experiment management. We need to make dvc exp + MR experience feel natural to developers 🙌🏻 Here're my thought, let's discuss.

This is almost the same as if you would branch out and run an experiment without DVC (although we may make UX better by letting dvc exp manage type: model somehow).

So, if you have a separate "feature" branch with experiment.

A. You may want to first assess metrics/reports, then decide to merge branch, then register a Semver version/promote to stage.

1. if that's a manual process, you should be ok using GTO and Studio MR for that
2. if that's automated, you'd be setting up a CI that run some checks (optionally) and registers/promotes
3. as a special case of the previous one, if you're promoting each commit merged to main, you're either:
   i. fine without any promotions/registrations (cause answer to "what's in prod?" will always be "what's in HEAD of main"),
   ii. or you need to register/promote, but if you have more than a single model in repo, you could do that in CI (run CI on merge into main to check what type: model files are updated, register a new version)

B. You may want to deploy the model to preview env before merging (can do register/promote afterwards as well):

1. this may happen without assigning stage (you have CI running on each new commit to the feature branch, that re-deploys the model and run some tests - similar to what happens in dvc.org with feature branches). Then you probably don't need stage assignments, since HEAD of your feature branch is what's assigned to preview
2. this may happen manually - if you don't have a automatic re-deploy on each commit, but want to trigger that manually. Then we may suggest registering a new version and promoting it. After merge, the history will keep this and it won't be garbage collected, which is fine I assume. The tricky part is that rn you can't register the same model Semver twice for different commits. Which means, if you registered few versions in your feature branch (v1.0.0, v1.0.1, v1.0.3), you can't register the same version once you merge the branch. The limitation seems reasonable to me since the last commit in feature branch may not have any specific version registered at all. Instead of considering this a GTO limitation and allowing to register same versions in different commits, i'd suggest thinking that this is a Git limitation that needs to be taken into account. The suggested way to work here would be instead registering "candidate" versions like (v1.0.0-rc0, v1.0.0-rc1, v1.0.0-rc2) as Semver standard assumes.

cc @dberenbaum

[dberenbaum]

There could also be two different CI/automated workflows:

1. To review and merge the latest model.
2. To deploy or release that model.

For example, I might have a nightly build that actually deploys the newest model. I don't want to assume the latest model in main is what's deployed because the nightly build may not have happened yet, or it may have failed. I'd rather have that build process promote the model when deployment happens succesfully. What would be the best way to do this?

[aguschin]

Good point!

I guess we're mixing two things here: "what's actually deployed" and "what's was promoted to be deployed". We just need to keep in mind that Git tags doesn't reflect a real state of deployment - though CI can promote model to stages indicating a deployment was made successfully. The "assignment stages" mechanism can work for both. There even could be a case involving both (by steps):

1. PR with model update was merged into main (via CI or manually, doesn't matter)
2. CI to promote model to #scheduled-to-deploy is triggered at 6am. Git tag mymodel#scheduled-to-deploy#132 is created, which kicks off CD job
3. CD job runs, and upon successful completion, it promotes model to #successfully-deployed by creating Git tag mymodel#successfully-deployed#133, which can kick off next CI workflow (like post an update or file an failure alert somewhere, or you can do that in the same CD job).

If one don't need 2 or 3 in their workflow, one can remove them (If 2 is removed, then 3 will be triggered by cron).

[dberenbaum]

Makes sense @aguschin. Of course, teams will vary, but I would think a typical workflow is that CI may promote to #staging and CD may promote to #prod.

[dberenbaum]

Closing until we see a need for this