You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[New Notebook]: Reconcile the RA datasets at NCEI vs data available through IOOS Data Catalog #278
Do some matching between records at NCEI and records available through the IOOS data catalog. Essentially come up with a list of datasets that aren't at NCEI.
@ocefpaf we have published two notebooks in the CodeLab which is a start to this analysis. Linked below. I would just need to smash them together to see what we get.
I think collecting the metadata will be easy. But, trying to match what is at NCEI to what's available from the RAs in the IOOS data catalog will be tricky. At first glance, metadata doesn't match across the two. And the frequency at which we harvest and records change in the data catalog is a moving target.
changed the title [-][New Notebook]: Reconcile the datasets at NCEI vs data available through IOOS Data Catalog[/-][+][New Notebook]: Reconcile the RA datasets at NCEI vs data available through IOOS Data Catalog[/+]on Jul 17, 2025
This is somewhat nearly impossible. There is no consistency between what is at NCEI and what is on the IOOS Data Catalog...
I've tried to do some fuzzy matching between titles and that is hit or miss. I've tried to trace one dataset that I know exists in both and it's still a ~90% confidence that they are 1:1.
H @MathewBiddle I'm not sure this is going to help with your immediate goal here, but here's a step in the right direction from my standpoint, as far as IOOS' data providers go anyway:
IOOS data providers include both id and naming_authority attributes across their datasets in a consistent way, following IOOS Metadata Profile guidelines in the Dataset Description section (which are identical to ACDD guidelines for both attributes).
ERDDAP's ISO output code is updated (xref: erddap/erddap#349) to combine both naming_authority and id when generating the <gmd:fileIdentifier></gmd:fileIdentifier> content of resulting ISO XML (for the 'current' version of ISO 19115-2 NOAA endorses, anyway, which includes this field). This concatenated ID still meets the ISO 19115-2 ISO Metadata Workbook guidance NOAA published in 2012 that specifies "fileIdentifier – A unique phrase or string which uniquely identifies the metadata file". Presently, I believe ERDDAP just uses the dataset name as fileIdentifier, which leads to some collisions in metadata catalogs for IOOS anyway (see below).
NCEI archival datasets somehow reference or incorporate some or all of the above concatenated unique identifier in their archival datasets and associated metadata. Not quite sure how this would work, but maybe it would help? At least for datasets that were 1:1 between IOOS Catalog and NCEI/OneStop, you could infer a source dataset ID if the naming_authority + id identifier were preserved in the archived data. I'm sure a 1:1 ratio is definitely not always the case, so how would you map IDs from multiple source datasets aggregated together into a single archived dataset? Comma-separated ID list attribute?
FWIW, these changes would also help avoid collisions of duplicated ISO fileIdentifier values in ERDDAP datasets in metadata catalogs (an issue we have currently within our own IOOS Catalog with so many providers). This may be a catalog-software specific issue - I'm not sure if OneStop cares if it has two datasets with the same fileIdentifier, but I know CKAN does, which affects Data.gov inventories.
There's probably a better way to do all of this involving DOIs I'm not aware of, but I thought the above knowledge dump might be useful anyway.
The main challenge, in any case, rolling out a solution across 43K+ datasets...
@apkrelling now that you understand the NCEI pipelines a little more, do you think it's possible to identify records that appear in the IOOS Data Catalog (https://data.ioos.us/) which are not yet archived at NCEI?
I was previously using the NCEI geoportal API to gather the metadata records from NCEI to compare with what's found in the IOOS Data Catalog.
The results of this would, ideally, be a list of datasets we need to archive at NCEI.
Activity
[-][New Notebook]: Reconcile the datasets at NCEI vs data available through IOOS Data Catalog[/-][+][New Notebook]: Reconcile the RA datasets at NCEI vs data available through IOOS Data Catalog[/+]MathewBiddle commentedon Jul 17, 2025
This is going to be messy. Jotting down some notes from an initial dive into tracing one dataset from GCOOS.
Full stop. Are those data archived at NCEI?? Maybe?
For reference, this was digging around for one record. There are >43k records in the IOOS Data catalog...
MathewBiddle commentedon Jul 17, 2025
CC @ksauby, @apkrelling
MathewBiddle commentedon Sep 12, 2025
This is somewhat nearly impossible. There is no consistency between what is at NCEI and what is on the IOOS Data Catalog...
I've tried to do some fuzzy matching between titles and that is hit or miss. I've tried to trace one dataset that I know exists in both and it's still a ~90% confidence that they are 1:1.
Not sure where to go from here.
mwengren commentedon Sep 12, 2025
H @MathewBiddle I'm not sure this is going to help with your immediate goal here, but here's a step in the right direction from my standpoint, as far as IOOS' data providers go anyway:
idandnaming_authorityattributes across their datasets in a consistent way, following IOOS Metadata Profile guidelines in the Dataset Description section (which are identical to ACDD guidelines for both attributes).naming_authorityandidwhen generating the<gmd:fileIdentifier></gmd:fileIdentifier>content of resulting ISO XML (for the 'current' version of ISO 19115-2 NOAA endorses, anyway, which includes this field). This concatenated ID still meets the ISO 19115-2 ISO Metadata Workbook guidance NOAA published in 2012 that specifies "fileIdentifier – A unique phrase or string which uniquely identifies the metadata file". Presently, I believe ERDDAP just uses the dataset name asfileIdentifier, which leads to some collisions in metadata catalogs for IOOS anyway (see below).naming_authority+ididentifier were preserved in the archived data. I'm sure a 1:1 ratio is definitely not always the case, so how would you map IDs from multiple source datasets aggregated together into a single archived dataset? Comma-separated ID list attribute?FWIW, these changes would also help avoid collisions of duplicated ISO
fileIdentifiervalues in ERDDAP datasets in metadata catalogs (an issue we have currently within our own IOOS Catalog with so many providers). This may be a catalog-software specific issue - I'm not sure if OneStop cares if it has two datasets with the same fileIdentifier, but I know CKAN does, which affects Data.gov inventories.There's probably a better way to do all of this involving DOIs I'm not aware of, but I thought the above knowledge dump might be useful anyway.
The main challenge, in any case, rolling out a solution across 43K+ datasets...
MathewBiddle commentedon Feb 23, 2026
@apkrelling now that you understand the NCEI pipelines a little more, do you think it's possible to identify records that appear in the IOOS Data Catalog (https://data.ioos.us/) which are not yet archived at NCEI?
I was previously using the NCEI geoportal API to gather the metadata records from NCEI to compare with what's found in the IOOS Data Catalog.
The results of this would, ideally, be a list of datasets we need to archive at NCEI.