GEOS-CF Analysis and Forecast Data - Air4US

[siddharth0248]

Dataset Information

Dataset name:
GEOS-CF Analysis and Forecast Data

Data source

Is the dataset in NASA's Common Metadata Repository (CMR)?

• Yes, concept ID is: ________________
• No
• I don't know

Where will the data be located?

• Amazon Public Data Program Bucket
• PO.DAAC AWS Bucket
• GES-DISC AWS Bucket
• LPDAAC AWS Bucket
• ORNL AWS Bucket
• ASF AWS Bucket
• VEDA SMCE AWS Bucket
• Other: GEOS-CF-v2
• I don't know

Please provide any additional details required to discover all the files for this dataset.

Please provide a link to an example data file:

Access Data]

Dataset Extent

How frequently is this dataset updated?

• Sub-daily (hourly or more frequent)
• Daily
• Weekly
• Monthly
• Annually
• Irregularly/As available
• Static (never updated)
• Other: ________________

What is the temporal extent of the data?
Aug 4, 2025 - Ongoing

What is the spatial extent of the data?
Global

Requested services

What portal is requesting this dataset?

• VEDA Earthdata
• MAAP
• GHG Center
• Disasters
• AIR4US
• Water Insights
• Earth Information Explorer (EIE)
• Other: ________________

What capabilities and ODD services are of interest for this dataset? (Select all that apply)

• Assessment - help me decide how to deliver this dataset optimally
• Map visualization
• Timeseries visualization
• Statistics over an area and/or time-frame of interest
• Python access in the VEDA Hub
• QGIS access in the VEDA Hub
• ArcGIS access via EGIS
• Virtualization (aka virtual Zarr store)
• Structural assessment (how should this dataset be chunked and stored)
• Other: ________________
• I don't know

What is the target deadline for integrating this dataset?

• Next 1-3 months
• 3-6 months
• 6-12 months
• No specific deadline
• Other: ________________

What is driving this timeline?
Stakeholder requirements

Progress Tracking

Create Sub-issues

• 1. Access data and exploration

  • Identify data source/location
    • Search and discovery tools for GEOS-CF dataset #312
  • Verify access permissions
  • Initial data inspection

• 2. Integration

  • Define ingestion pipeline
    • Deploy virtualizarr-data-pipelines for GEOS-CF #325
  • Visualize data using one of the tiling services

• 3. Exploring ingested data (Jupyter Notebook)

  • Create notebook for validation
  • Visualize dataset
  • Document findings

• 4. Benchmarking original vs ingested (Jupyter Notebook)

  • Compare performance

• 5. Demo

  • Prepare demo use case
  • Generate visual outputs
  • Share/demo with stakeholders

Other relevant information

1. Existing visualization tool
2. More Details about dataset and requirements

[abarciauskas-bgse]

Noting also this document GEOS-CF Background Information has some dataset details (such as file naming conventions). It includes some use cases and tools used to access the data. My understanding based on this document is that this dataset is used to compare actual observations (analysis "ana" data) with forecasts ("fcst" data).

As far as I am aware the design of AIR4US tools is ongoing but it would be good in designing this dataset to anticipate what they might need.

@tylanderson @siddharth0248 could you point me to the current virtual zarr store, and an example notebook if it exists? Is there a design document for that virtual zarr store? If not I think we should create one, explaining the origin and structure of the virtual zarr store. See https://github.com/earth-mover/icechunk-nasa/tree/main/design-docs for examples.

[siddharth0248]

@abarciauskas-bgse

storage = icechunk.s3_storage(
    bucket='veda-geos-cf-virtualizarr',
    prefix="geos-cf-virtualizarr-data-pipelines/prod",
    anonymous=True,
    region="us-east-1",
)

i have created an initial repo @tylanderson please add other relevant scripts and Dependency management

@abarciauskas-bgse you can access the notebook here. Design docs are missing, and I think we should have them for all the stores that we have created. Happy to assist on it.

[tylanderson]

@siddharth0248 @abarciauskas-bgse is this repo where we'd like any of the pipeline code to generate the stores to go now? Currently it all lives under the nasa impact org

[tylanderson]

On these examples, we should probably add in an example as well of combining the datasets that have different chunk dimensions from earlier in the GEOS-CF production and the main production run (happy to add that in)

[abarciauskas-bgse]

is this repo where we'd like any of the pipeline code to generate the stores to go now? Currently it all lives under the nasa impact org

@tylanderson @siddharth0248 can we combine that notebook with https://github.com/NASA-IMPACT/geos-cf-virtualizarr-data-pipelines. And yes, I think we should move the NASA-IMPACT repo to virtual-zarr. I think we should consolidate all virtual zarr store examples in that virtual-zarr org.