possibility of pre-stage the data before running the model

[uturuncoglu]

I wonder if it is possible to pre-stage the required input data and use it when actual model is running. In my case, I am using GraphCastSmall along with WB2ERA5 dataset and every time when I run the model (for same date), it is fetching data again. So, it would be nice to download data and store in the disk (for certain period) to use it later to run the model. Any example would be helpful at this point.

[NickGeneva]

Apologies for missing this thread @uturuncoglu

The data sources should cache but default in the earth2studio cache folder (by default this is ~/.cache/earth2studio/ so it the data source should automatically handle this to some degree.
But for longer inference jobs, it would typically make the most sense to dump the data using a seperate script into a zarr store. Then use one the xarray data sources to load the data into the pipeline.

I've opened an example to track this request here: #528

Thanks!

[uturuncoglu]

@NickGeneva Sorry for late response. Thanks for the explanation. The feature to use pre-staged data would be very useful. Thanks again.