Working Plan: Support opening and authenticating icechunk with earthaccess.

[jbusecke]

I have been virtualizing several NASA datasets recently (see here, here, and here) with one of them (NLDAS) now existing fully (virtual icechunk store + target chunks in nc files) on the NASA waterinsight bucket.

As discussed in #956 (comment) and with several other folks on slack etc. manually providing the virtual container credentials is a major friction point. In the specific scope of NASA data and earthaccess we can entirely abstract that complexity away from the user!

I want to use this issue to discuss implementation details and make sure that any PR I am opening is aligned with efforts of others like @DeanHenze (see https://github.com/nsidc/earthaccess/pull/964/files).

Scope

This feature will exclusively focus on the ability to open an existing virtual icechunk store including the authentication. For now I would develop something along the lines of

import earthaccess
earthaccess.open_icechunk(<icechunk_store_url>) -> xr.Dataset

This should work for both native icechunk stores and virtual ones

Under the hood this will have to do several things:

1. Match the <icechunk_store_url> with a DAAC and get credentials
2. Open and check the icechunk repository to get all virtual container buckets (if existing) and prefixes
3. (Only if virtual icechunk): Compare all buckets+prefixes from 2 with a central 'allow list' and build credentials for each bucket (similar to 1)
4. Open a fully authenticated icechunk repo and pass it to xarray
5. Return the xarray dataset.

Questions

I ll outline some questions I immediately have here, and will add as things go along.

• The most immediate question to me is, were and how we store the 'allow list' for buckets. What we need here is at the very least a mapping between buckets (and potentially prefixes) and authentication details.
  • Should this information be stored with the code (new locations would be added via new earthaccess versions) or via a config stored elsewhere (would enable admins to allow access to new buckets without requiring new code version). I currently think 2 would be the better choice, but happy to hear opinions on this.
• Ultimately we probably want something like earthaccess.open_virtual() (with either options or keywords to open both kerchunk + icechunk virtual stores? @DeanHenze, can we coordinate to have the same functional signature for earthaccess.open_virtual_icechunk and a potential earthaccess.open_kerchunk? Then .open_virtual could easily dispatch one of them.

Very keen to hear comments, feedback, concerns from the earthaccess community before diving into this!

[DeanHenze]

Thanks for starting this @jbusecke !

I agree with that top level idea you mention where the user never has to know there is an icechunk vs kerchunk (unless they want to). So your proposed earthaccess.open_virtual() function, with a kwarg in there format that could be used to specify "icechunk" or "kerchunk_json", but defaults to some sort of format check if nothing is specified.

One thing I'd like to discuss is whether that top level function should return stores rather than xr.Dataset's, since the store could be passed to any future package which could open zarrs or VDS's. I don't wan't people to be forced to work in Xarray.

So I guess in the way I've proposed, the top level use case would be

earthaccess.get_virtual_reference(short_name, ...) -> url or store
data = xr.opendataset(url or store)

or replace that last line with whatever package.

Another reason I like the earthaccess side of things to return a store or url is that it parallels the netCDF use case. E.g. earthaccess is used to find granule metadata or create file-like objects that get passed into Xarray, but doesn't do the job of actually opening and returning an Xarray.Dataset itself. Whether intentional or not, I like that earthaccess has set themselves up in this way, so they could be used easily with more packages than just Xarray.

[jbusecke]

Yes I like that idea of making it even tighter scoped! Will definitely build it that way.

[jbusecke]

I agree with that top level idea you mention where the user never has to know there is an icechunk vs kerchunk (unless they want to). So your proposed earthaccess.open_virtual() function, with a kwarg in there format that could be used to specify "icechunk" or "kerchunk_json", but defaults to some sort of format check if nothing is specified

We should think about this in the 'finding' logic. Presumably there will not always be both virtual options available? So then maybe the user would pass a list of preferred formats? Some things to discuss tomorrow.