140-stage DVC pipeline hard to work with

[dotXem]

I have been advised by @daavoo to create this issue for better tracking of my challenges regarding the usage of DVC. I've put it under Feature Request as what we are looking for may not be possible for current DVC.

Here is the original post I created in the DVC forum detailing our needs.

Basically, our current pipeline is becoming quite big with 30 models and 140 dvc stage instances. We have 3 different stages: create_dataset, train_model, and compute_metrics. Consequently, we use foreach definitions of the 3 stages inside the dvc.yaml file to reduce the code duplication. Still, the params.yaml file is 1300 lines long, which is hard to work with.

Also, all the stage instances have the name "stage_name@number" (e.g., "train_model@0). The names do not hold useful information making the use of selective dvc repro -s hard to work with (which we use a lot). For instance, a common command we use would be dvc repro -s create_dataset@0 create_dataset@1 create_dataset@2 train_model@0 compute_metrics@0 to repro all the stage instances of a given model. To know what stage instances belong to the given model we want to repro, we need to look at the dvc.lock which is super tedious (3k-line long).

Ideally we are looking for a way to:

1. split the params.yaml into smaller ones, each belonging to a given model
2. have better stage instance namings, to better tell them apart

I think point 2. is doable by declaring the stage instances as follows in the params.yaml file:

create_dataset_list:
  model_1_trainset:
    script: create_dataset.py
    dataset_yaml: trainset.yaml
    folder_images: trainset_images
    params: trainset_params.py
    output: trainset.h5

However, as for point 1., I don't have any idea as importing yaml files into other ones is not possible AFAIK.

I have attached a minimal example to better show how our project is organized around DVC.

minimal_dvc.zip

[daavoo]

curious for more ideas here @dberenbaum @skshetry . I didn't find the time of trying to adapt the example to use hydra composition

[dberenbaum]

@dotXem You may want to look at the open PR in #9725. I don't know that it solves any of your issues, but that is the current direction for stage templating, so you may want to check it out.

Do you generally want to run one model pipeline at a time or many/all at once?

[dotXem]

Thank you for the suggestion @dberenbaum. I've looked into it and I don't think it will work well. It does not seem possible to factorize much the different parametrizations of the foreach loop we have into a matrix like presented.

Do you generally want to run one model pipeline at a time or many/all at once?

1 model at a time. It might happen that we want to run several if the model pipelines share some stage together (e.g., multiple finetunings of a general model), or even all at once. However, because it takes a lot of time (total of 50h to run everything I think), we don't do it often.

[dberenbaum]

1 model at a time.

In that case, I agree with @daavoo that hydra sounds like it could be a fit. Have you taken a look at https://dvc.org/doc/user-guide/experiment-management/hydra-composition? There's a basic example in https://github.com/iterative/lstm_seq2seq/tree/main/conf where there are two models (full and small).

I haven't had a chance to try it with your example, but you could have files like conf/models/model_1.yaml that contain the parameters for each model like:

model_name: model_1
create_dataset:
  script: create_dataset.py
  dataset_yaml: trainset.yaml
  folder_images: trainset_images
  params: trainset_creation.py
  output: trainset.h5
train_model:
  ...

[dberenbaum]

To run the pipeline for a particular model, you would run dvc exp run --set-param model=model1