-
Notifications
You must be signed in to change notification settings - Fork 8
Description
Extracting grids from forecast datasets to create single-grid, CF-compliant netCDF files to present to MET is a slow process that consumes considerable disk space. As an optimization for advanced users, support an option, either via config value or command-line switch, to specify that the forecast dataset is already CF-compliant and can be read as-is, similar to how wxvx already treats GRIB-formatted local-filesystem forecast datasets, i.e. simply configuring MET to find a particular grid within a GRIB file containing many messages, with many MET invocations ultimately reading from the same GRIB file.
Note that this work does not anticipate configuring MET to process more than a single pair of grids per invocation -- what wxvx currently does -- though an option to do this could be considered for later work. Processing individual grid pairs lets wxvx give granular feedback on what has succeeded or failed, lets it make incremental progress on a large verification graph (including processing newly available, leading-edge forecast data as it becomes available), lets users remove bits of finished work and ask wxvx to regenerate it with an update config, etc.
Later work (also not in-scope for this ticket) could consider adding support for CF-decorating a forecast dataset either in place or as a separate file. The in-place option would conserve both disk space and time, at the cost of modifying the original dataset, which may not be desirable. The separate-file option would not conserve disk space, but may perform considerably less IO vs writing and the reading dozens/hundreds/thousands of single-grid netCDF files.