Consider using netcdf4 groups in SWALS to combine multidomain outputs into a single file

[gareth-d-ga]

Currently SWALS writes multiple output files for each domain:

• A netcdf with gridded outputs
• A netcdf with gauge outputs
• Some text metadata

For large multidomains with hundreds of domains, this can lead to > 1000 files for a single model run.

• This can be a problem, e.g., if running hundreds of scenarios then we start approaching file-count limits on some supercomputers (e.g. NCI).

An alternative is to use netcdf groups to combine gauge/grid outputs from each domain in a single netcdf file.

• This would require updates to SWALS including the post-processing scripts.

A potential downside is that netcdf-groups might not play nicely with some other tools (like ncview). So ideally the netcdf-group format could be implemented as a compile-time option, with post-processing routines seamlessly working with either format.

[gareth-d-ga]

While not related to netcdf directly, pull request #22 has reduced the number of output files.

[gareth-d-ga]

While not exactly matching the point above, this can be done in a post-processing step using e.g.

ncecat --gag */Grid*.nc -o test.nc

from inside a multidomain folder.

The resulting file can be viewed with ncview, and read with R's ncdf4 package.

It would be better to improve the group names (to include the domain folder).

The above is at least useful for testing, or, could be an alternative way to manage the file counts. We could also introduce some compression, e.g.

ncecat -4 --deflate 9 --gag */Grid*.nc -o test3.nc

[gareth-d-ga]

ncrename might be able to change groups names

[gareth-d-ga]

This version iteratively adds netcdf files into groups, while using group names to reflect the directory structure. Run from the multidomain directory.

for i in RUN*/Grid*.nc; do echo $i; ncecat -4 --deflate 4 --no_tmp_fl -A -M -G $i --gag $i -o test7.nc; done

We can also put grids and gauges in the one file

for i in RUN*/G*.nc; do echo $i; ncecat -4 --deflate 4 --no_tmp_fl -A -M -G $i --gag $i -o test7.nc; done