Skip to content

SCHEMA.md

Latest commit

 

History

History
114 lines (92 loc) · 5.29 KB

SCHEMA.md

File metadata and controls

114 lines (92 loc) · 5.29 KB
Raw

MARIS-Forecast — per-sample schema

This document describes the on-disk layout of the released dataset. The same constants are available programmatically in maris_forecast/schema.py.

Top-level layout

track_a_short-term_Cross-domain_Datasets/           # Track A: 10 min -> 10 min
  dma_track_v1/  noaa_track_v1/  piraeus_track_v1/  norway_track_v1/
track_b_medium-term_Cross-domain_Datasets/          # Track B: 30 min -> 60 min
  dma/standard_track_v1/  noaa/standard_track_v1/  piraeus/standard_track_v1/  norway/standard_track_v1/

Every leaf directory contains:

<leaf>/
  train/part-000.csv.gz   val/part-000.csv.gz   test/part-000.csv.gz
  context_v1/
    environment/rasters/<split>/{masks.npz, signed_dist_shore.npy, signed_dist_nav.npy, sample_ids.npy}
    environment/features/<split>/environment_descriptors.csv
    environment/vectors/<split>/vectors.jsonl.gz
    social/features/<split>/social_features.csv
    augmented/<split>/part-000.csv.gz      # split + all per-sample fields, single-call loading
  context_v1_2019osm/                       # Piraeus only: historical-OSM (2020-01-01) rebuild
  osm_temporal_consistency/<split>_flags.csv
  sample_ids/  reports/  README.md  summary.json

Window geometry

Track history T_h future T_f interval observation prediction
A 30 30 20 s 10 min 10 min
B 90 180 20 s 30 min 60 min

Positions are in a local planar frame centred on the anchor (the last observed point): x = east, y = north, units of metres.

Split-CSV columns (part-000.csv.gz)

One row per forecasting sample. Groups:

  • Identifiers — sample_id, mmsi, segment_id, ship_type, ship_class, ship_type_id; the Greek and Norwegian feeds add ship_class_unified and ship_type_id_unified (VesselFinder-enriched, 100 % coverage).
  • Timing (UTC) — hist_end_ts, pred_end_ts.
  • Local-metre geometry — hist_x_json, hist_y_json, fut_x_json, fut_y_json (JSON arrays; T_h history and T_f future points).
  • Kinematics / cyclic encodings — hist_sog_json, hist_cog_{sin,cos}_json, hist_heading_{sin,cos}_json, periodic time-of-day / day-of-week encodings.
  • Quality scalars — interp_ratio_*, *_displacement_m, quality_tier (core/…), split.
  • Four inline Stage-17 flags — osm_temporal_consistent (bool), osm_max_inland_depth_m (m), osm_n_inland_points (count), osm_max_consec_inland_run (count).
  • Fifteen anchor-time context scalars — see table below.

The complete column list for each feed is in its summary.json.

Anchor-time context scalars (15)

Group Column Unit Source
Weather met_wind_speed_mps m s⁻¹ ERA5 (Open-Meteo)
Weather met_wind_dir_deg deg (from) ERA5
Weather met_wind_rel_heading_deg deg derived
Weather met_temperature_c °C ERA5
Weather met_pressure_hpa hPa ERA5
Weather met_cloud_cover_pct % ERA5
Sea state sea_wave_height_m m ECMWF WAM
Sea state sea_wave_dir_deg deg (to) ECMWF WAM
Sea state sea_wave_period_s s ECMWF WAM
Sea state sea_swell_wave_height_m m ECMWF WAM
Port port_nearest_dist_km km OSM harbour
Port port_nearest_name text OSM name
Fairway/TSS in_fairway bool OSM fairway
Fairway/TSS dist_to_fairway_m m OSM fairway
Fairway/TSS in_tss bool OSM separation tags

Empty cells denote absent source data (not imputed). The wave fields are empty for the 2019 Piraeus subset (the marine reanalysis begins 2020-01-01).

Context tensors (context_v1/)

File Shape / type Notes
environment/rasters/<split>/masks.npz (N, 6, 128, 128) uint8, key masks channels: land, water, navigable, natural_boundary, manmade_boundary, barrier
environment/rasters/<split>/signed_dist_shore.npy (N, 128, 128) float16 metres, +offshore / −inland
environment/rasters/<split>/signed_dist_nav.npy (N, 128, 128) float16 metres to navigable water
environment/rasters/<split>/sample_ids.npy (N,) row alignment to the split CSV
environment/features/<split>/environment_descriptors.csv 14 scene scalars scene type, water/navigable ratios, boundary densities, env-quality score
social/features/<split>/social_features.csv ≤10 neighbours/sample relative position/velocity, distance, CPA, TCPA; min_cpa_m, min_abs_tcpa_s, interaction_candidate

The patch is a 5-km-radius, 128×128 grid (≈78 m/pixel). Raster/SDF grids are stored north-in-row-0; render with a vertical flip (arr[::-1]) or imshow(..., origin='upper') to display north-up (the figure scripts do this).

Recommended default view

Filter to the OSM-temporally consistent samples:

df = loader.load_split(leaf, "test", consistent_only=True)
# equivalently, on the released inline layout:
#   df = df[df["osm_temporal_consistent"].astype(str).str.lower() == "true"]

Retention: 326,316 / 330,000 Track A (98.88 %) and 99,191 / 106,857 Track B (92.83 %). All curated samples remain available with the flags stored inline, so inconsistent rows can be retained for dock-side / infrastructure studies.