Thoughts on DVC for large media (and other large/binary files)

[GregVernon]

I've been working on updating documentation for the isogeometric analysis capabilities in MOOSE and have been working with the large_media submodule for images etc., and wanted to mention a tool we've been using at Coreform for several years: Data Version Control (DVC).

We used to use submodules for our large data files, but it was quite challenging to manage as we found ourselves often needing to track two merge requests - and verifying paths were consistent between code and datafiles... so we migrated to DVC. There are certain large files that we require for our various unit/integration tests, and we've baked those DVC commands into our build scripts - so as an "end-user" I don't concern myself with those DVC operations. Where I do use DVC quite heavily are when I'm wearing my "application engineer" hat on customer projects. A typical workflow for these projects in our repository might look something like this:

mkdir ~/coreform_flex/engineering/CustomerName/ProjectName/CustomerData -p
cd ~/coreform_flex/engineering/CustomerName/ProjectName

# Place files in directory... then:
cd CustomerData
dvc add gearbox_assembly.CATProduct *.CATPart problem_definition.pptx material_test_data.xlsx tensile_test.mp4
dvc push
git commit -m "eng: add customer files"
git push

# Later...
cd ~/coreform_flex/engineering/CustomerName/ProjectName
git add process_material_test_data.py run_sim_workflow.py 
git commit -m "eng: add simulation analysis scripts"
git push
# etc.

# Later...
mkdir ~/coreform_flex/engineering/CustomerName/ProjectName/Report/media -p
cd ~/coreform_flex/engineering/CustomerName/ProjectName/Report/media
# Add report-related files... then:
dvc add *.png *.mp4 *.e *.tar.gz *.pptx *.pdf
dvc push
git commit -m "eng: add report files"
git push

# Create MR and merge it... then:
cd ~/coreform_flex
git checkout master
git clean -xdf  # Cleans out all the dvc'd files, leaving the `.dvc` files in place

# Weeks later when more work is needed
cd ~/coreform_flex/engineering/CustomerName/ProjectName
dvc pull -R . # Just pull (recursively) in the current working directory

The way I would imagine using DVC in, e.g., the example problem I'm currently working on (and its documentation) would be something like:

# Add example large/binary files to DVC
cd ~/projects/moose/modules/solid_mechanics/examples/flex_iga/cframe
dvc add c-frame.step
git commit -m "add input geometry CAD file"
dvc add cframe.cf trimmed_cframe_moose.exo trimmed_cframe_moose.h5
git commit -m "add coreform-computed files required for running example without coreform license"
dvc add gold/*
git commt -m "add gold result files"

# Add documentation media files to DVC
cd ~/projects/moose/modules/solid_mechanics/doc/content/media/solid_mechanics/iga
dvc add *.png
git commit -m "add documentation media files"
 
dvc push
git push

I'm curious to get some thoughts from the MOOSE team on DVC - I'm not sure what all considerations went into choosing to do the large_media submodule, or why one might chose DVC vs Git LFS for MOOSE.

[GiudGiud]

Hello

we don't use git LFS anymore for moose. The large_media submodule is used instead of LFS essentially. This of course means we don't check in huge mesh files in the repo, which makes sense.

For customer projects, we do create separate git repositories, that may have the application as a submodule (to do CI) and use LFS to keep the mesh / large files.

I can't speak to using DVC, I'm less familiar on how these decisions were made. Maybe @permcody or @friedmud can weigh in?