openquake.data

[micheles]

The engine repository contains a lot of data files, like for instance the CSV files used as expectation values for the GSIM tests (they have been there from the beginning) or the .onnx files used for the models using machine learning technologies (a relatively new addition). This causes the repository to grow constantly. Already, a lot of time is spent downloading the repository, both for final users and for our tests, and that gets worse at each release. In particular, 99% of the user do not need 99% of the data, but right now they are forced to download everything, also what they do not need. Also, docker images are large. In the future we expect more .onnx files to be included, and much larger than they are now, so the problem will become worse, even if most people will not use models requiring such files. How can we solve that? Using git support for large files makes things more complicated and it is very expensive (every time a collaborator or a CI/CD runner like GitHub Actions clones the repo or pulls updates, it consumes your bandwidth quota), so it is not a solution. A solution could be to move the data files to a public http server and save the files in directories containing the engine version number (i.e. something like https://downloads.openquake.org/engine-3.23/gsim) and then change the installer to download only the files needed for the current version, possibly with parameters like light or full depending on how much data she wants to download.

Preliminary to such change would be to move the data files in a sensible location in the current repository, for instance we could introduce a package openquake.data for that. That would improve a lot the organization of the repository and would make the installation easier, avoiding errors like the ones we had recently, with the PyPI package missing data files required for the gsims (see #11056).

[mmpagani]

The .onnx files have a different importance than the test file for ground-motion models. I think it's an error to support .onnx files natively, since those models could be represented as tabular GMPEs (which require an .hdf5 file, probably more compact).

[micheles]

@mmpagani: unfortunately, it is a lost battle: even if we decided to reject all the GMPEs using .onnx files (and there are few) the .onnx files are used in the secondary peril models, moreover the risk team plans to use more and more artificial intelligence techniques so we have to support them.

PS: you are still on time to reject #11006 if you don't like it.

[mmpagani]

Ciao Michele, grazie, sorry for missing this. If you are already taking it as a given that OQ will provide support for the use of .onnx file is pointless to reject this PR.