Minimal modifications to params.yaml when running --set-param

[lucaerculiani]

dvc version: 2.38.1

Hi!

Deep learning guy here.
In our experiments we make use of the configuration files to define the structure of the network.
In particular we have some parameters that are lists where each element is a list of parameters that identify
a particulal sumbodule of the network plus its initialization arguments and its inputs.
This is a quite powerful way to define networks but comes at the cost of having
some parameters with complex values. In order to keep them manageable, we use indentation, inlining
and comments to clarify its structure.
Aside from these, we have other parameters that control other aspects of the network that are usually simple values. These are
the ones we want to change during our batches of experiment to find their optimal values.

Here an example:

activation: "relu"
batch_nomalization: True

network: [ # list of modules : [ name, [*parameters]]
[convolution2d, [16, 2, 2]],
[submodule_x, [32, True, 0.5]],
[submodule_y, [32, False, 1.0, 16, True]],
... 
]

In this case network is the complex parameter whose indentation/inlining is useful, while activation and batch_nomalization are the parameters that are usually explored via batches of experiment.

The problem is that when I try to run an experiment using dvc exp run --set-param activation=swish, the resulting
params.yaml file is something like:

activation: swish
batch_nomalization: True

network: 
- -  convolution2d
- - - 16
- - - 2
- - - 2
- - submodule_x
- - - 32
- - - True
- - - 0.5
- - sumbodule_y
- - - 32
- - - False
- - - 1.0
- - - 16
- - - True
...

The reformatting and dropping of comments makes reading and modifying the network after the fact (while preserving the best set of values for the parameters, discovered during past experiments) more compicated.
What I would prefer is localizing the modifications done to the parameters file only to the keys that actually needs to be changed, thus obtaining (in this case):

activation: swish
batch_nomalization: True

network: [ # list of modules : [ name, [*parameters]]
[convolution2d, [16, 2, 2]],
[submodule_x, [32, True, 0.5]],
[submodule_y, [32, False, 1.0, 16, True]],
...
]

What do you think about it?

[dberenbaum]

Hi @lucaerculiani! There are some related issues like #5736 about preserving the original file format. DVC tries to minimize modifications to params.yaml, but it is harder than it looks because DVC translates the yaml into an internal data structure, which doesn't include any of this formatting, so it's not straightforward to recover the initial formatting when translating it back.

So while this is a legitimate feature request, I'm wondering if you have considered using Hydra? We recently introduced Hydra integration to take advantage of its powerful configuration framework alongside DVC. You could keep your network architecture in a separate file like conf/network/mynetwork.yaml (or keep many different network architectures each in their own file) and then choose a specific architecture for any experiment. DVC would still generate a potentially messy params.yaml, but your original conf/network/mynetwork.yaml file would remain untouched. There's an example in #8093 (comment) of how far you can go with this, even having Hydra directly read your model classes/layers.

[skshetry]

We used to preserve the formatting of params.yaml before. Did anything change with hydra impl? cc @daavoo.

[daavoo]

We used to preserve the formatting of params.yaml before.

We still modified the formatting in some cases (#5736 linked by dave) but it was better than now and preserved the comments.

Did anything change with hydra impl? cc @daavoo.

Yes, I removed the use of our dumper because of #8583 .

There might be a way to try to preserve formatting but it is not as simple as before where we only modified keys of the dict .
We now support operations (adding, composing, removing) that require passing the data to hydra and converting back from the returned omegaconf.

[dberenbaum]

We should save the comments if possible, but I don't think we ever preserved whether lists were formatted with brackets or dashes, which is a major issue here.

[skshetry]

@dberenbaum, we used ruamel.yaml's roundtrip parser for modification which did preserve brackets/dashes. It may not always preserve indentations for individual sequences and might normalize inconsistent indentation, but it did preserve blocks/brackets.

[skshetry]

We now support operations (adding, composing, removing) that require passing the data to hydra and converting back from the returned omegaconf.

Is it possible to get an updated dictionary? If yes, then we can diff it and apply it over the file ourselves. It is complicated, but that might preserve the formatting.

We have this helper that tries to apply the diff and we use it for preserving formatting in .dvc files.

dvc/dvc/utils/collections.py

Line 7 in f386f7c

def apply_diff(src, dest):