Investigate requirements for UN Decade ODIS/OIH interoperability for IOOS Catalog

[mwengren]

IOOS Catalog should be able to feed an oceanography-focused portal such as the UN Decade's:

Ocean Data Information System (ODIS); https://catalogue.odis.org/, https://oceaninfohub.org/odis/

and OceanInfoHub: https://oceaninfohub.org/

These tools are similar in function to Google Dataset Search in that they harvest JSON-LD from individual data providers. We already have the functionality in place to meet most of their requirements, although we will likely need to enhance our JSON-LD metadata content to meet their schema requirements.

xref:

• OIH metadata documentation: https://book.oceaninfohub.org/index.html
• Draft OIH data provider requirements: https://github.com/iodepo/odis-arch/pull/380/files
• Catalog sitemap.xml: Re-enable sitemap.xml update process from CKAN database ckanext-ioos-theme#241

[benjwadams]

Is there a way to test the schema.org interop with these catalog harvesters?

[MathewBiddle]

The https://data.ioos.us/sitemap.xml seems to be broken. Not sure how to fix it.

XML Parsing Error: not well-formed
Location: https://data.ioos.us/sitemap.xml
Line Number 2, Column 36:
ed-carbonate-system-2016-2018-cb8-1</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/cbp-water-quality-monitoring-subset-1984-2018-ret1-1</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/surfrider-station-452</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/cbp-water-quality-monitoring-subset-1984-2018-cb5-1w</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/observed-carbonate-system-2016-2018-cb4-1c</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/mddnr-station-harris-creek-downstream</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/surfrider-station-796</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/observed-carbonate-system-2016-2018-cb4-4</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/surfrider-station-389</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/great-baynh-great-bay-wq-station</loc><lastmod>2024-08-20</lastmod></url><url><loc>https://data.ioos.us/dataset/mddnr-station-public-landing</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/saildrone-hurricane-monitoring-2021-nrt-data-drone-1060</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/mddnr-station-bishopville-prong</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/surfrider-station-726</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/cbp-water-quality-monitoring-subset-1984-2018-cb4-2w</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/cbp-water-quality-monitoring-subset-1984-2018-cb7-4n</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/mddnr-station-sandy-point-south-beach</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/usgs-hudson-river-01354330</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/autonomous-real-time-marine-mammal-detections-new-york-bight-nw-buoy</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/hwy-80-at-chimney-creek-on-tybee-island-savannah-ga10</loc><lastmod>2024-09-05</lastmod></url><url><loc>https://data.ioos.us/dataset/41110-ilm2wave-ilm2w-wrightsville-beach-nearshore-waves-masonboro-inlet-nc10</loc><lastmod>2024-09-05</lastmod></url><url><loc>https://data.ioos.us/dataset/cbp-water-quality-monitoring-subset-1984-2018-tf3-3</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/weather-research-and-forecasting-wrf-regional-atmospheric-model-oahu</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/pacioos-water-quality-buoy-04-hilo-bay-big-island-hawaii</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/weather-research-and-forecasting-wrf-regional-atmospheric-model-maui-oahu</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/weather-research-and-forecasting-wrf-regional-atmospheric-model-cnmi</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/weather-research-and-forecasting-wrf-regional-atmospheric-model-main-hawaiian-islands</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/aloha-cabled-observatory-aco-acoustic-doppler-current-profiler-adcp-temperature</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/gulf-of-maine-wbts-calanus-abundance-observations-since-2020</loc><lastmod>2024-08-31</lastmod></url><url><loc>https://data.ioos.us/dataset/a01-optode-oxygen</loc><lastmod>2024-08-20</lastmod></url><url><loc>https://data.ioos.us/dataset/bio-ww-iii-latest-forecasts-gulf-of-maine</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/pacioos-weather-station-honolulu-harbor-entrance-oahu-hawaii</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/simulating-waves-nearshore-swan-regional-wave-model-big-island</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/regional-ocean-modeling-system-roms-guam</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/wavewatch-iii-ww3-hawaii-regional-wave-model</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/f01-accelerometer-waves</loc><lastmod>2024-08-20</lastmod></url><url><loc>https://data.ioos.us/dataset/lobo-2-buoy-mid-damariscotta-river</loc><lastmod>2024-08-20</lastmod></url><url><loc>https://data.ioos.us/dataset/e01-aanderaa-surface-currents</loc><lastmod>2024-08-20</lastmod></url><url><loc>https://data.ioos.us/dataset/surface-currents-from-a-diagnostic-model-scud-pacific</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/regional-ocean-modeling-system-roms-western-north-pacific-data-assimilating</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/simulating-waves-nearshore-swan-regional-wave-model-manua-american-samoa</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/noaa-coral-reef-watch-daily-global-5-km-satellite-monitoring-marine-heatwave-watch</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/regional-ocean-modeling-system-roms-oahu-kaneohe</loc><lastmod>2024-09-04</lastmod></url><url><loc>https://data.ioos.us/dataset/regional-ocean-modeling-system-roms-main-hawaiian-islands</loc><lastmod>2024-09-04</lastmod></url></urlset>
-----------------------------------^

[MathewBiddle]

The only addition to the sitemap.xml for ODIS would be to add a <changefreq> node for each dataset. See https://book.odis.org/gettingStarted.html#frequency-of-change.

Add the metadata for each sitemap record, on how often you expect records in your sitemap to change - this will tell systems like ODIS how often to reindex your holdings. Possible values are: always, hourly, daily, weekly, monthly, yearly, never.

[mwengren]

@benjwadams Let's try adding a <changefreq> tag set to weekly for all datasets, assuming we go ahead and change our data provider harvests to weekly as discussed.

It's possible data providers could change harvest frequency on their own once they have accounts on Catalog, but we have no (easy) way of knowing that on a dataset-by-dataset level.

Also, can you test the sitemap.xml to confirm that it's not well formed and the error isn't the result of some sort of harvesting glitch?