Skip to content

[New Notebook]: Reconcile the RA datasets at NCEI vs data available through IOOS Data Catalog #278

Open
0 of 1 issue completed

Description

@MathewBiddle

Programming Language

Python

Module/Software/IOOS data service?

IOOS Data Catalog CKAN, NCEI Geoportal REST API, ISO xml metadata

Module/Software/IOOS data service?

See the following notebooks:

Provide a description.

Do some matching between records at NCEI and records available through the IOOS data catalog. Essentially come up with a list of datasets that aren't at NCEI.

@ocefpaf we have published two notebooks in the CodeLab which is a start to this analysis. Linked below. I would just need to smash them together to see what we get.

  1. https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2017-06-12-NCEI_RA_archive_history.html
  2. https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2024-09-17-CKAN_API_Query.html

I think collecting the metadata will be easy. But, trying to match what is at NCEI to what's available from the RAs in the IOOS data catalog will be tricky. At first glance, metadata doesn't match across the two. And the frequency at which we harvest and records change in the data catalog is a moving target.

Activity

  1. changed the title [-][New Notebook]: Reconcile the datasets at NCEI vs data available through IOOS Data Catalog[/-] [+][New Notebook]: Reconcile the RA datasets at NCEI vs data available through IOOS Data Catalog[/+] on Jul 17, 2025
  2. MathewBiddle commented on Jul 17, 2025

    @MathewBiddle
    ContributorAuthor

    This is going to be messy. Jotting down some notes from an initial dive into tracing one dataset from GCOOS.

    Full stop. Are those data archived at NCEI?? Maybe?

    For reference, this was digging around for one record. There are >43k records in the IOOS Data catalog...

  3. MathewBiddle commented on Jul 17, 2025

    @MathewBiddle
    ContributorAuthor
  4. moved this from To Do to In Progress in IOOS CodeLabon Jul 17, 2025
  5. MathewBiddle commented on Sep 12, 2025

    @MathewBiddle
    ContributorAuthor

    This is somewhat nearly impossible. There is no consistency between what is at NCEI and what is on the IOOS Data Catalog...

    I've tried to do some fuzzy matching between titles and that is hit or miss. I've tried to trace one dataset that I know exists in both and it's still a ~90% confidence that they are 1:1.

    Not sure where to go from here.

  6. mwengren commented on Sep 12, 2025

    @mwengren
    Member

    H @MathewBiddle I'm not sure this is going to help with your immediate goal here, but here's a step in the right direction from my standpoint, as far as IOOS' data providers go anyway:

    1. IOOS data providers include both id and naming_authority attributes across their datasets in a consistent way, following IOOS Metadata Profile guidelines in the Dataset Description section (which are identical to ACDD guidelines for both attributes).
    2. ERDDAP's ISO output code is updated (xref: erddap/erddap#349) to combine both naming_authority and id when generating the <gmd:fileIdentifier></gmd:fileIdentifier> content of resulting ISO XML (for the 'current' version of ISO 19115-2 NOAA endorses, anyway, which includes this field). This concatenated ID still meets the ISO 19115-2 ISO Metadata Workbook guidance NOAA published in 2012 that specifies "fileIdentifier – A unique phrase or string which uniquely identifies the metadata file". Presently, I believe ERDDAP just uses the dataset name as fileIdentifier, which leads to some collisions in metadata catalogs for IOOS anyway (see below).
    3. NCEI archival datasets somehow reference or incorporate some or all of the above concatenated unique identifier in their archival datasets and associated metadata. Not quite sure how this would work, but maybe it would help? At least for datasets that were 1:1 between IOOS Catalog and NCEI/OneStop, you could infer a source dataset ID if the naming_authority + id identifier were preserved in the archived data. I'm sure a 1:1 ratio is definitely not always the case, so how would you map IDs from multiple source datasets aggregated together into a single archived dataset? Comma-separated ID list attribute?

    FWIW, these changes would also help avoid collisions of duplicated ISO fileIdentifier values in ERDDAP datasets in metadata catalogs (an issue we have currently within our own IOOS Catalog with so many providers). This may be a catalog-software specific issue - I'm not sure if OneStop cares if it has two datasets with the same fileIdentifier, but I know CKAN does, which affects Data.gov inventories.

    There's probably a better way to do all of this involving DOIs I'm not aware of, but I thought the above knowledge dump might be useful anyway.

    The main challenge, in any case, rolling out a solution across 43K+ datasets...

  7. MathewBiddle commented on Feb 23, 2026

    @MathewBiddle
    ContributorAuthor

    @apkrelling now that you understand the NCEI pipelines a little more, do you think it's possible to identify records that appear in the IOOS Data Catalog (https://data.ioos.us/) which are not yet archived at NCEI?

    I was previously using the NCEI geoportal API to gather the metadata records from NCEI to compare with what's found in the IOOS Data Catalog.

    The results of this would, ideally, be a list of datasets we need to archive at NCEI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Notebook ideaFor new notebook proposals

Type

No type

Fields

No fields configured for issues without a type.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions