dvc status: shows that params have changed when they haven't

[gonzalojaimovitch]

Bug Report

dvc status shows that params have changed when they haven't

Description

After reproducing a pipeline successfully, where each stage is using the params field and running "dvc status", I get the same message saying that I have a change in my deps with the tag "new" pointing to every parameter file on each stage. When running "dvc repro" again when no changes have happened, it doesn't run the pipeline again, but with a message on each stage that it is using the cached version. When a collaborator pulls the same commit, the pipeline runs one more time (assuming the collaborator doesn't have any cached data). I guess this should not be the expected behaviour. For what I know so far, it should return a message in the style "pipeline up to date", and no execution happening.

Reproduce

1. dvc repro
2. dvc status

Expected

I expect dvc to print a message informing that everything is up to date. Furthermore, I expect this to happen when a collaborator clones the project, pulls the data from remotes and executes dvc status

Environment information

Output of dvc status:

$ dvc status
data_extraction_tags:
        changed deps:
                new:                config/Data_Extraction.yaml
data_extraction_image:
        changed deps:
                new:                config/Data_Extraction.yaml
data_merging:
        changed deps:
                new:                config/Data_Merging.yaml

Additional Information (if any):

This would be the dvc.yaml content I am using in the project:

stages: 
    data_extraction_tags:
        cmd: python -m src.data_extraction.data_extraction --env=./config/.env --config=./config/Data_Extraction.yaml --query_name_param=query_tags_name --query_param=query_tags --query_scheme_param=scheme_tags
        deps:
            - ./src/data_extraction/data_extraction.py
        params:
            - ./config/Data_Extraction.yaml:
                - pathData
                - query_tags_name
                - query_tags
                - scheme_tags
        vars:
            - ./config/Data_Extraction.yaml:query_tags_name
            - ./config/Data_Extraction.yaml:pathData
        outs:
            - ${pathData}/${query_tags_name}.parquet.gzip
            
    data_extraction_image:
        cmd: python -m src.data_extraction.data_extraction --env=./config/.env --config=./config/Data_Extraction.yaml --query_name_param=query_images_name --query_param=query_images --query_scheme_param=scheme_images
        deps:
            - ./src/data_extraction/data_extraction.py
        params:
            - ./config/Data_Extraction.yaml:
                - pathData
                - query_images_name
                - query_images
                - scheme_images
        vars:
            - ./config/Data_Extraction.yaml:query_images_name
            - ./config/Data_Extraction.yaml:pathData
        outs:
            - ${pathData}/${query_images_name}.parquet.gzip

    data_merging:
        cmd: python -m src.data_preparation.data_merging --config=./config/Data_Merging.yaml
        deps:
            - ./src/data_preparation/data_merging.py
            - ${pathDataRaw}
        params:
            - ./config/Data_Merging.yaml:
        vars:
            - ./config/Data_Merging.yaml:pathDataRaw
            - ./config/Data_Merging.yaml:pathDataMerged
        outs:
            - ${pathDataMerged}

[dberenbaum]

Context from discord: https://discord.com/channels/485586884165107732/563406153334128681/1111557378735865856.

It looks like this is limited to the way paths are written in this dvc.yaml with the ./rel/path pattern instead of rel/path.

A simple script to reproduce:

git init -q repo
cd repo
dvc init -q
echo "stages:
  test:
    cmd: echo test
    params:
      - ./my_params.yaml:" > dvc.yaml
echo "params: 1" > my_params.yaml
dvc repro -q
dvc status

The result of dvc status is:

test:
        changed deps:
                new:                my_params.yaml

If I change the script to use my_params.yaml instead of ./my_params.yaml in dvc.yaml, dvc status outputs:

Data and pipelines are up to date.

[skshetry]

Instead of using a def_path, looks like we are using normalized paths. Looking into it.

[sztal]

I have the same problem (for dvc==3.56.0). More concretely, after running one stage the params section used by the previous stage is considered changed, even though it does not change (which is confirmed by the fact that the md5 hash of the entire params.yaml file does not change).

This is my params.yaml.

paths:
  make!:       "newsuse.config:Paths"
  data:        "data"
  proc:        "@data/proc"
  raw:         "@data/raw"
  models:      "models"
  dataset:     "dataset"
  posts:       "@proc/posts.parquet"
  annotations: "@proc/annotations.xlsx"
  postscls:    "@proc/posts-cls.parquet"

gdrive:
  annotations: 158q-O8kkTrXlw1lN7ra_nTswkvIAn-Ch
  inference:
    posts: 1xWhVd98ZKonJvl7UqgqNZGUSAxYQOaRR

annotations:
  labels:
    - OTHER
    - NEGATIVE
  annotators:
    - Magdalena
    - Szymon
    - Maria
  sheet_index_name: country
  content_name: text
  seed: 0

dataset:
  test_size: 0.1
  validation_size: 0.1
  only_valid_snippets: true
  min_words: 3
  balance:
    - country
    - label
  seed: 0

inference:
  batch_size: 32

tokenize:
  field: text
  padding: max_length
  truncation: true
  batched: true

training:
  evaluation:
    make!: newsuse.ml:Evaluation
    metric: f1
  callbacks:
    - make!: transformers:EarlyStoppingCallback
      early_stopping_patience: 2
  arguments:
    make!: newsuse.ml:TrainingArguments
    num_train_epochs: ${eval:${oc.env:NEWSUSE_NUM_TRAIN_EPOCHS,8}}
    per_device_train_batch_size: ${eval:${oc.env:NEWSUSE_PER_DEVICE_TRAIN_BATCH_SIZE,16}}
    gradient_checkpointing: ${eval:${oc.env:NEWSUSE_GRADIENT_CHECKPOINTING,1}}
    fp16: ${eval:${oc.env:NEWSUSE_FP16,1}}
    learning_rate: 5e-6
    seed: 303
    metric_for_best_model: loss
    save_strategy: epoch
    eval_strategy: epoch
    logging_strategy: epoch
    load_best_model_at_end: true
    overwrite_output_dir: true
  hyper:
    n_jobs: ${eval:${oc.env:NEWSUSE_OPTUNA_N_JOBS,1}}
    n_trials: ${eval:${oc.env:NEWSUSE_OPTUNA_N_TRIALS,100}}
    models:
      - distilbert
      - bert
      - xlm-base
      - xlm-large
    space:
      learning_rate:
        type: float
        low: 1e-7
        high: 1e-4
        log: true
      warmup_ratio:
        type: float
        low: 0
        high: 0.5
        step: ${eval:1 / ${....arguments.num_train_epochs}}
      weight_decay:
        type: float
        low: 0
        high: 0.01
        step: 0.002
      lr_scheduler_type:
        type: categorical
        choices:
        - linear
        - cosine
        - cosine_with_restarts
      seed:
        type: categorical
        choices:
        - 2839680617
        - 1047004691
        - 337003538
        - 1884749421
        - 3325526844
        - 2356784760
        - 1445499703
        - 1003721274
        - 3292496752
        - 3153403444

models:
  use: xlm-large

  distilbert:
    name: distilbert
    base: distilbert-base-multilingual-cased
    training: ${training}
    inference: ${inference}
    tokenize: ${tokenize}

  bert:
    name: bert
    base: bert-base-multilingual-cased
    training: ${training}
    inference: ${inference}
    tokenize: ${tokenize}

  xlm-base:
    name: xlm-base
    base: FacebookAI/xlm-roberta-base
    training: ${training}
    inference: ${inference}
    tokenize: ${tokenize}

  xlm-large:
    name: xlm-large
    base: FacebookAI/xlm-roberta-large
    training:
      evaluation: ${training.evaluation}
      callbacks: ${training.callbacks}
      arguments:
        make!: ${training.arguments.make!}
       num_train_epochs: ${training.arguments.num_train_epochs}
        per_device_train_batch_size: ${training.arguments.per_device_train_batch_size}
        gradient_checkpointing: ${training.arguments.gradient_checkpointing}
        fp16: ${training.arguments.fp16}
        learning_rate: 6.977843998506154e-06
        lr_scheduler_type: cosine
        warmup_ratio: 0.25
        weight_decay: 0.004
        seed: 1884749421
        metric_for_best_model: ${training.arguments.metric_for_best_model}
        save_strategy: ${training.arguments.save_strategy}
        eval_strategy: ${training.arguments.eval_strategy}
        logging_strategy: ${training.arguments.logging_strategy}
        load_best_model_at_end: ${training.arguments.load_best_model_at_end}
        overwrite_output_dir: ${training.arguments.overwrite_output_dir}
    inference: ${inference}
    tokenize: ${tokenize}

And this is my dvc.yaml.

stages:

  posts:
    cmd: python scripts/make_posts.py
    deps:
    - data/raw/posts-eu.parquet
    - data/raw/posts-us.parquet
    - scripts/make_posts.py
    outs:
    - data/proc/posts.parquet:
        persist: true

  annotations:
    cmd: python scripts/make_annotations.py --save-to-disk=true
    deps:
    - data/proc/posts.parquet
    - scripts/make_annotations.py
    params:
    - models.use
    - annotations
    outs:
    - data/proc/annotations.xlsx:
        persist: true

  dataset:
    cmd: python scripts/make_dataset.py
    deps:
    - data/proc/posts.parquet
    - data/proc/annotations.xlsx
    - scripts/make_dataset.py
    params:
    - dataset
    outs:
    - dataset:
        persist: true

  model:
    cmd: python scripts/model_train.py --no-eval=true
    deps:
    - dataset
    - scripts/model_train.py
    params:
    - models.use
    - models.${models.use}.training
    outs:
    - models/${models.use}:
        persist: true

  report:
    cmd:
    - >
      cd reports/classifiers &&
      quarto render report.qmd --to pdf --output ${models.use}.pdf &&
      cd -
    - rm -rf reports/classifiers/report_files
    deps:
    - dataset
    - reports/classifiers/report.qmd
    - models/${models.use}
    params:
    - models.use
    outs:
    - reports/classifiers/${models.use}.pdf

  classification:
    cmd: python scripts/model_classify_posts.py
    deps:
    - data/proc/posts.parquet
    - models/${models.use}
    - scripts/model_classify_posts.py
    params:
    - models.use
    outs:
    - data/proc/posts-cls.parquet:
        persist: true

And the exact problem is that once I run dvc repro model everything is fine, that is, I can run it again and it will properly see that no input has changed, so the computations are not repeated. However, once I ran dvc repro report (which is a child of model stage in the DAG) the section models.xlm-large in params.yaml is seen as changed even though the params.yaml file does not change.

Just in case, here is the DAG.

+-------------------------------+         +-------------------------------+  
| data/raw/posts-eu.parquet.dvc |         | data/raw/posts-us.parquet.dvc |  
+-------------------------------+         +-------------------------------+  
                          ***                  ***                           
                             ****          ****                              
                                 **      **                                  
                                 +-------+                                   
                                *| posts |***                                
                             *** +-------+   ***                             
                        *****         *         *****                        
                     ***              *              ***                     
                  ***                 *                 *****                
      +-------------+                 *                      **              
      | annotations |              ***                        *              
      +-------------+             *                           *              
                   ***         ***                            *              
                      *       *                               *              
                       **   **                                *              
                    +---------+                               *              
                    | dataset |                               *              
                    +---------+                               *              
                    **        ***                             *              
                  **             *                            *              
                **                **                          *              
               *                +-------+                    **              
                **              | model |                  **                
                  **            +-------+*               **                  
                    **        ***         ***          **                    
                      **     *               *       **                      
                        *  **                 **   **                        
                    +--------+          +----------------+                   
                    | report |          | classification |                   
                    +--------+          +----------------+ 

UPDATE.
I just did detailed testing, and the problem appears when I mix interpolation with fixed values as in models.xlm-large.training.arguments section. Very weird behavior.

UPDATE 2.
After further testing, it seems that more exactly the problem occurs when I add extra fields to models.xlm-large.training.arguments section that are not present in the default training.arguments section, from which I interpolate default values.

More concretly, everything is fine as long as models.xlm-large.training.arguments uses only the same fields as training.arguments, irrespective of whether the entire training.argument section is copied via dict-interpolation --- models.xlm-large.training.arguments: ${training.arguments} --- or by interpolating all the fields individually.

UPDATE 3.
I confirm that by rewriting the training.arguments and models.xlm-large.training.arguments sections the following way the problem disappears:

training:
  arguments:
    make!: newsuse.ml:TrainingArguments
    num_train_epochs: ${eval:${oc.env:NEWSUSE_NUM_TRAIN_EPOCHS,8}}
    per_device_train_batch_size: ${eval:${oc.env:NEWSUSE_PER_DEVICE_TRAIN_BATCH_SIZE,16}}
    learning_rate: 6.977843998506154e-06
    lr_scheduler_type: cosine
    warmup_ratio: 0.25
    weight_decay: 0.004
    seed: 1884749421
    gradient_checkpointing: ${eval:${oc.env:NEWSUSE_GRADIENT_CHECKPOINTING,1}}
    fp16: ${eval:${oc.env:NEWSUSE_FP16,1}}
    metric_for_best_model: loss
    save_strategy: epoch
    eval_strategy: epoch
    logging_strategy: epoch
    load_best_model_at_end: true
    overwrite_output_dir: true

models:
  xlm-large:
    name: xlm-large
    base: FacebookAI/xlm-roberta-large
    training: ${training}
    inference: ${inference}
    tokenize: ${tokenize}