Virtual variables

[abkfenris]

In XREDS @mpiannucci implemented a cool pattern to support dynamic generation of new xr.DataArrays in a dataset based on the existing data.

He is using it it to generate Mean Lower Low Water levels based off of NAVD88 in his datasets, and an attribute pointing to the VDatum offsets: https://github.com/asascience-open/xreds/blob/main/xreds/extensions/vdatum.py

To do so he created his own hookspec DatasetExtensionSpec.transform_dataset which he calls in his dataset provider https://github.com/asascience-open/xreds/blob/main/xreds/dataset_extension.py

I had been thinking around the same idea for a while, and then @srstsavage brought it up in the XREDS repo to support generating virtual vector variables. asascience-open/xreds#28

I think we should standardize this pattern into a new type of plugin.

[abkfenris]

Also we can support the flat earthers the -180-180/0-360º problem.

[srstsavage]

Seems like there's some movement in the xpublish-wms space recently so I'll give this a nudge. Support for virtual vector tiles (u/v variables in the data to magnitude/direction tiles) is a significant missing piece in the ecosystem in my mind, and there's some complexity on what should get implemented where.

[abkfenris]

I think a hangup is probably that we need to know when a dataset transformation should be applied.

For some datasets a transformation/virtual variable plugin may be able to decide on CF standard names to convert to other standards (maybe this is something that could be upstream-ed into cf-xarray). But we've all encountered source data that doesn't quite fit into CF that also needs to be converted.

So we need some other way of specifying that kind of thing. XREDS has a extensions key that it passes around with it's dataset loading config, but that's probably not as generalizable (ex: specifying a default color map to WMS for variable).

While we could implement a new Xpublish sidecar to pass around with datasets, datasets already have one with .attrs. I think we should standardize on either a key or prefix for anything that controls parts of the Xpublish ecosystem. That way any transformation or output plugin knows where to look/what to remove (if needed).

ds.attrs = {
  "_xpublish": {
    "virtual_vectors": [
      {
        # component, existing variable (or maybe CF standard name)/new variable name
        "north": "northward_wind",
        "east": "eastward_wind",
        "speed": "wind_speed",
        "to_direction": "wind_to_direction",
      },
      {
        "from_direction": "sea_water_velocity_from_direction",
        "speed": "sea_water_speed",
        "u": "sea_water_u",
        "v": "sea_water_v",
      }
    ],
    "vdatum": {
      "path": "s3://noaa-nodd-kerchunk-pds/nos_vdatums/cbofs_vdatums.nc.zarr",
      "water_level_var": "zeta",
      "vdatum_var": "mllwtomsl",
      "vdatum_name": "mllw",
      "multiplier": -1.0
    },
    "360": {"360_dim": "lon"},
  }
}

ds["funny_variable"].attrs = {
  "_xpublish": {
    "wms": {
      "color_map": "prism"
    }
  }
}

It's not like we'd be the first data server to overload attributes with configuration.

<addAttributes>
  <att name="sourceUrl">(local files)</att>
  <att name="subsetVariables"></att>
  <att name="testOutOfDate">now-1day</att>
</addAttributes>