Propose design doc and roadmap for VirtualiZarr 2.0

[maxrjones]

There's a lot of interconnected technical design issues open. This design doc should help us discuss solutions that impact multiple issues.

The key differences to the current design is using protocols more than ABCs, defining a "Reader" as a protocol required by VirtualiZarr that can be used by backends and the ManifestStore, and orienting backends around ManifestStore creation rather than Dataset creation.

Just a starting point building off ideas shared by @kylebarron and @sharkinsspatial.

[maxrjones]

@kylebarron @chuckwondo @d-v-b I wasn't able to request reviews from you all but it'd be great to get your thoughts on this if you have time

[chuckwondo]

Most of my suggestions are punctuation and syntax fixes, but I do have a couple of more significant comments in here.

[TomNicholas]

Thanks for taking the initiative here @maxrjones ! It's clear from this that there was far too much intention just scattered around various issues - it's nice to discuss it in one place.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.34%. Comparing base (ff1ddb4) to head (81705e9).
Report is 8 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #568      +/-   ##
===========================================
+ Coverage    88.80%   89.34%   +0.53%     
===========================================
  Files           34       34              
  Lines         1948     1943       -5     
===========================================
+ Hits          1730     1736       +6     
+ Misses         218      207      -11     

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[abarciauskas-bgse]

Thanks for putting this together @maxrjones it is super helpful to have these conversations synthesized into this design doc, especially with the associated issues linked. Take my comments with a grain of salt as I haven't been involved directly in these discussions.

[norlandrhagen]

Not sure which Phase in the roadmap it fits into, but having https support for virtual stores would be really clutch for a lot of usecases.

[kylebarron]

I'd suggest considering removing **kwargs from any top-level function because it's often unclear where they go. Instead, if those kwargs are intended for the object reader, they should be passed into the reader before the instance is passed into this function.

[maxrjones]

It provides these arguments to open_virtual_dataset:

    drop_variables: Iterable[str] | None = None,
    loadable_variables: Iterable[str] | None = None,
    decode_times: bool | None = None,
    cftime_variables: Iterable[str] | None = None,
    indexes: Mapping[str, Index] | None = None,

Since I think the motivation was to follow the design of https://docs.xarray.dev/en/stable/generated/xarray.open_mfdataset.html closely, I doubt we'll want to change this even though I also dislike pass-through **kwargs.

[kylebarron]

If you know it's always going to be those 5 arguments, why not re-declare them here so you get accurate type hinting?

[maxrjones]

I defer to Tom on this to decide the preferred balance of mirroring Xarray vs accurate type hinting.

[maxrjones]

We decided to match xarray's **kwargs syntax in today's coordination call.

[kylebarron]

Do you mean we are having **kwargs in our signature or we're going to hoist the specific argument parameters that xarray supports into our signature so that we don't have to accept **kwargs here?

[maxrjones]

Do you mean we are having **kwargs in our signature

Yes, following the design pattern established in https://docs.xarray.dev/en/stable/generated/xarray.open_mfdataset.html

[TomNicholas]

For now, we just want the signature of open_virtual_X to match the signature of open_X as closely as possible. Even though that signature includes untyped **kwargs (for open_mfdataset). We're choosing for now to value consistency with xarray over strong typing.