Skip to content

Add optional alternative nexus output format #920

@robertbartel

Description

@robertbartel

Add an optional, alternative format for nexus outputs that writes nexus data to per-formulation files, instead of writing data to per-nexus files. This will facilitate ingest of the nexus data by t-route. It also will help to avoid or mitigate problems with large domains and too many open files, as some compute environments (especially shared research environments) may have strict limits on the number of available file descriptors.

Requirements

  • Configurable option (almost certainly via realization config)
  • When active, alternate format nexus data file(s) are written, and default per-nexus data files are not opened
  • Alternate format nexus data files will be NetCDF
  • Alternate format nexus data files will be per-formulation, with time and nexus id components added to the data
  • Implementation will need to support writing multiple per-formulation files when the formulations section in the realization config has multiple separate formulations configured
    • If ngen does not already fully support such a configuration (needs to be confirmed), then this will need to ensure it is compatible with such functionality if/when available, and that may be addressed at the same time to support ensembling
  • When alternatve format is active, enforce and inform that only global formulation configs are supported
    • Without this, catchments (and nexuses) would have separate formulations, and we'd effectively be back at per-nexus files
    • If this starts to appear to be problematic or unsatisfactory, another issue can be opened to track improving in the future
  • (Implied) During parallel execution, all ranks should write to the same file(s)
    • This might have ramifications that either need to be addressed immediately or noted for later, and in either case clearly documented

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions