Skip to content

Support CF-compliant netCDF forecast datasets without grid extraction #96

@maddenp-cu

Description

@maddenp-cu

Extracting grids from forecast datasets to create single-grid, CF-compliant netCDF files to present to MET is a slow process that consumes considerable disk space. As an optimization for advanced users, support an option, either via config value or command-line switch, to specify that the forecast dataset is already CF-compliant and can be read as-is, similar to how wxvx already treats GRIB-formatted local-filesystem forecast datasets, i.e. simply configuring MET to find a particular grid within a GRIB file containing many messages, with many MET invocations ultimately reading from the same GRIB file.

Note that this work does not anticipate configuring MET to process more than a single pair of grids per invocation -- what wxvx currently does -- though an option to do this could be considered for later work. Processing individual grid pairs lets wxvx give granular feedback on what has succeeded or failed, lets it make incremental progress on a large verification graph (including processing newly available, leading-edge forecast data as it becomes available), lets users remove bits of finished work and ask wxvx to regenerate it with an update config, etc.

Later work (also not in-scope for this ticket) could consider adding support for CF-decorating a forecast dataset either in place or as a separate file. The in-place option would conserve both disk space and time, at the cost of modifying the original dataset, which may not be desirable. The separate-file option would not conserve disk space, but may perform considerably less IO vs writing and the reading dozens/hundreds/thousands of single-grid netCDF files.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions