Hi all,
Desired feature
Dean here from PO.DAAC. We are excited to try out icechunk with virtualizarr on one of our data sets and host the virtual data store publicly as a pilot example, but will likely need to access the store via https. I've read on the virtualizarr docs is not possible yet, I think it's potentially in the making? Just wanted to provide some motivation for why it could be worthwhile.
Additional context
So for example, we are currently hosting some Kerchunk'd reference files for several data sets like this one https://podaac.jpl.nasa.gov/dataset/CCMP_WINDS_10M6HR_L4_V3.1 (look under "Data Access" tab, and under "Direct Access" you'll see a link for a virtual reference file). More importantly, the https link to those reference files are in our CMR records, so e.g. the earthaccess package could find those links and use them to open the data set. Here is an example of lazy loading several data sets this way on the TB scale in a few lines of code and a few seconds, using a fork of earthaccess under development. Anyone should be able to install that fork and instantly work with those data sets!
We would love to have similar functionality with a virtual data set backed by an ice chunk store. But as mentioned above, we currently need to have it with an https endpoint. I can go into more detail if needed, but short answer is that ingesting files into our S3 buckets for public use is an extensive process that doesn't allow us to do virtual stores yet. For now, we are uploading the stores to a gitlab repo allowing us to have publicly available https end points. The hope is that if we demo this functionality enough, the right support would come to allow this to be ingested in our S3 buckets (possibly a long way down the road).
Thanks,
Dean
Hi all,
Desired feature
Dean here from PO.DAAC. We are excited to try out icechunk with virtualizarr on one of our data sets and host the virtual data store publicly as a pilot example, but will likely need to access the store via https. I've read on the virtualizarr docs is not possible yet, I think it's potentially in the making? Just wanted to provide some motivation for why it could be worthwhile.
Additional context
So for example, we are currently hosting some Kerchunk'd reference files for several data sets like this one https://podaac.jpl.nasa.gov/dataset/CCMP_WINDS_10M6HR_L4_V3.1 (look under "Data Access" tab, and under "Direct Access" you'll see a link for a virtual reference file). More importantly, the https link to those reference files are in our CMR records, so e.g. the earthaccess package could find those links and use them to open the data set. Here is an example of lazy loading several data sets this way on the TB scale in a few lines of code and a few seconds, using a fork of earthaccess under development. Anyone should be able to install that fork and instantly work with those data sets!
We would love to have similar functionality with a virtual data set backed by an ice chunk store. But as mentioned above, we currently need to have it with an https endpoint. I can go into more detail if needed, but short answer is that ingesting files into our S3 buckets for public use is an extensive process that doesn't allow us to do virtual stores yet. For now, we are uploading the stores to a gitlab repo allowing us to have publicly available https end points. The hope is that if we demo this functionality enough, the right support would come to allow this to be ingested in our S3 buckets (possibly a long way down the road).
Thanks,
Dean