We discovered going from GGG2020 to GGG2020.1 that it's helpful to have a way to migrate netCDF file versions. This should be integrated from the start in the netCDF writer. Ideally, we would avoid duplicating code or manually implementing checks to update an existing file vs. starting from scratch in the run directory. I'm not totally sure yet how to handle that; it's not bad if we just need to add variables or fix values, but changing variable names or removing variables requires copying the netCDF file (since you cannot remove elements from a netCDF file once they are created).
Looking around to see if there are crates that can help. I only found https://crates.io/crates/version-migrate so far - everything else seems much more specific to databases.
It's not even clear yet if I should take inspiration from database migrations and have a variable/attribute that tracks these, or do it like I did for GGG2020.1, and just check if the fix needs applied by looking at the data. So far, the latter has worked, but there could be things that can't easily be figured out that way.
We discovered going from GGG2020 to GGG2020.1 that it's helpful to have a way to migrate netCDF file versions. This should be integrated from the start in the netCDF writer. Ideally, we would avoid duplicating code or manually implementing checks to update an existing file vs. starting from scratch in the run directory. I'm not totally sure yet how to handle that; it's not bad if we just need to add variables or fix values, but changing variable names or removing variables requires copying the netCDF file (since you cannot remove elements from a netCDF file once they are created).
Looking around to see if there are crates that can help. I only found https://crates.io/crates/version-migrate so far - everything else seems much more specific to databases.
It's not even clear yet if I should take inspiration from database migrations and have a variable/attribute that tracks these, or do it like I did for GGG2020.1, and just check if the fix needs applied by looking at the data. So far, the latter has worked, but there could be things that can't easily be figured out that way.