Skip to content

README.md

Latest commit

 

History

History
88 lines (76 loc) · 4.7 KB

README.md

File metadata and controls

88 lines (76 loc) · 4.7 KB
Raw

tempo-no2-maap

End-to-end NASA MAAP processing pipeline for hourly surface NO2 ML/DL model development across CONUS, upgrading the published Colab/ERA5 workflow (Aarons, Ni-Meister & Ramamurthy) to:

  • TEMPO NO2 L3, all four groups (root, product, geolocation, support_data) downloaded weekly; a curated 2-D feature subset feeds the models (vertical profiles are dropped by default to bound storage; pass --keep-profiles to retain them).
  • HRRR 3-km analysis meteorology (Utah s3://hrrrzarr archive, _anl stores only — no forecast files) fully replacing ERA5.
  • NLCD fractional impervious surface (2023 + 2024, year-matched) and TIGER/Line road density (PRISEC default, ALL supported), both on the TEMPO L3 0.02° common grid.
  • Two model-ready products: the paper-style station table (XGBoost TEMPO / no-TEMPO pair) and monthly gridded feature cubes for DL (UNet with station-masked loss).
  • Study window: 2023-08-02 to 2025-06-30 (paper window), configurable.

Everything heavier than orchestration runs as a registered DPS algorithm (interactive workspaces cap at 16 GB); the control notebook only submits and monitors jobs.

Pipeline

01_download_aqs ──────────────┐
02_download_tempo (per month)─┤
03_extract_hrrr   points ─────┼─► 06_merge_table ─► 08_train_xgb (512gb)
                  grid ───────┤
04_road_density ──────────────┼─► 07_build_cubes ─► 09_train_dl (32vcpu-64gb)
05_nlcd_isp ──────────────────┘        (per month)
# Algorithm Queue Granularity Output
01 no2_download_aqs 8gb one job aqs_no2_clean.parquet, station_lat_lon.parquet
02 no2_download_tempo 32gb 1 job/month (weekly zarrs inside) tempo_<wk>.zarr (all groups, CONUS)
03 no2_extract_hrrr 16gb 1 job/month × {points, grid} hrrr_points_YYYYMM.parquet, hrrr_grid_YYYYMM.zarr
04 no2_road_density 64gb one job road_density_tempo_grid.nc, station parquet
05 no2_nlcd_isp 16gb one job station footprints parquet, grid NetCDF
06 no2_merge_table 64gb one job model_table.parquet
07 no2_build_cubes 64gb 1 job/month cube_YYYYMM.zarr
08 no2_train_xgb cuny-512gb one job models, metrics, importances, SHAP, predictions
09 no2_train_dl 32vcpu-64gb one job dl_model.pt, logs (CPU now; GPU-ready)

Setup

  1. Push this repo to a public GitHub/GitLab repo; put the .git URL into every registration/*.yml (repository_url).
  2. In your MAAP workspace, store Earthdata credentials once: maap.secrets.add_secret("EDL_USERNAME", ...), same for EDL_PASSWORD (used by the TEMPO jobs via common/maap_utils.edl_login).
  3. Register: maap.register_algorithm_from_yaml_file(<yml>) for each file in registration/ (or use the Register Algorithm UI with the same values).
  4. Drive everything from notebooks/control_notebook.py (open as a notebook). Fill in the dps_output S3 paths printed by each finished job before launching the next stage.

Testing before scale-up

Use maap-dps-sandbox (8 GB, 10-min cap) to smoke-test each algorithm's build/env/arg handling with a tiny range, e.g. one 2-day TEMPO window or one HRRR day, before submitting the ~23-month batches.

Design notes / assumptions

  • Common grid. All static and gridded products live on the TEMPO L3 0.02° grid (snap to a real weekly zarr with maap_utils.tempo_grid(snap_file=...) for exact coordinates, as the original road-density script did).
  • HRRR regridding is nearest-neighbor via a KDTree built in the HRRR Lambert-conformal projection from s3://hrrrzarr/grid/HRRR_chunk_index.zarr. 3 km → 2 km NN oversampling is intentional; swap in bilinear weights if you prefer smooth fields.
  • HRRR variables (edit common/pipeline_config.HRRR_VARIABLES): HPBL, TMP/2m, RH/2m, UGRD+VGRD/10m (plus derived wspd/wdir), DSWRF, PRES, FRICV (ZUST analog), GUST. Archive gaps are logged per month to hrrr_missing_YYYYMM.json rather than failing the job.
  • TEMPO feature curation lives in pipeline_config.TEMPO_FEATURES; QC mirrors the paper (main_data_quality_flag != 2, tropospheric column below the training-set 98th-percentile cap, signal-to-noise feature).
  • AQS cleaning mirrors the notebook: CAPS-priority de-dup, Hawaii drop, GMT + local datetimes preserved for temporal features.
  • No GPU queue exists in your org today; 09_train_dl is device-agnostic and tuned for CPU patch training. Cubes are portable zarr, so they can be trained elsewhere on GPU unchanged.