Harvest metadata of datasets from National Data Platform into Harvard Dataverse

[jggautier]

@sbarbosadataverse let me now that Gary's interested in having Harvard Dataverse harvest or somehow index datasets from the National Data Platform.

The technical report at https://democratizingdata.ai/resources/technical-report might have more helpful information, like using their APIs.

Some additional things we might need to learn about the National Data Platform:

• How often are new datasets added or changed in or removed from the platform? The level of activity might influence how we decide to harvest the metadata, particularly since it looks like we won't be able to harvest using OAI-PMH, which would make it easy to keep the metadata in sync regardless of how often is changes.

[cmbz]

Some potential resources here: https://docs.ckan.org/en/2.9/api/, specifically the API calls related to packages or resources matching a query.

[jggautier]

Today I sent an email to the platform's contact email address, ndp@sdsc.edu, to let them know we're trying to learn how best to harvest from the platform, to learn how often metadata about data on the platform changes, and see if they have any tips.

[jggautier]

@sbarbosadataverse and @jggautier met today with Ilkay Altintas De Callafon, the National Data Platform's PI, and Katie O’Laughlin, its product manager.

We talked about how:

• The platform is in a pilot phase
• There are new records added regularly, so if we do harvest into HDV, we'd want to update records regularly
• They've used ckanext-harvest, a remote harvesting extension described at https://github.com/ckan/ckanext-harvest, to harvest records into their repository, and there might be a way to use that to harvest their records into HDV
• NDP is a "usability layer across multiple repositories", so it's more like a catalog that helps people access and use data, which lives elsewhere. We talked about how it's a little like Binder. And Ilkay talked about how they would be interested in indexing data from Harvard Dataverse and possibly other Dataverse repositories to increase the usability of data published in those repositories.

[jggautier]

I shared this GitHub issue with Ilkay and her colleagues, and @sbarbosadataverse and @cmbz so they could see a summary of our meeting and any next steps.

And wrote that I think the next steps might be to have our developer colleagues here on the Dataverse team review what was shared during the meeting so they can consider how best Harvard Dataverse can harvest from NDP.