Best way to get a list of unique URLs referenced in a virtual Icechunk v2 repo?

[rsignell]

What would be the best way to get a list (or set) of the unique URLs referenced in a virtual Icechunk v2 repo?

The use case is that I've got virtual FMRC Icechunk repo where I'm creating references for new NetCDF files produced by the forecast.

I'd like to generate one set of NetCDF file names from the existing Icechunk repo, and another set by scanning the forecast NetCDF files in the bucket, and then generate refs and append whatever files are missing.

[TomNicholas]

I swear there was already an issue about this somewhere... I don't think there is a way to do it today, but there could be. It would require iterating through all the chunks in all the relevant manifests, so it could end up being a compute-intensive operation.

[rsignell]

Thanks Tom, that makes sense. I'll try a different strategy of adding a variable with time dimension to my Icechunk repo dataset containing the NetCDF file name and use that as my set of already-processed-files. Or maybe generate the filenames from the time variable data!

[TomNicholas]

variable with time dimension to my Icechunk repo dataset containing the NetCDF file name and use that as my set of already-processed-files.

I think is probably better practice. Generally when virtualizing you quite often need to promote some per-file .attrs to .variables, because otherwise the Zarr/Xarray data model means that these attrs get thrown away during concatenation.