| title | Leveraging RDF and CURIE metadata resolution with identifiers.org | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| title_short | Leveraging RDF and CURIE metadata resolution with identifiers.org | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| tags |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| authors |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| affiliations |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| date | 14 May 2025 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| cito-bibliography | paper.bib | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| event | BH24EU | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| biohackathon_name | BioHackathon Europe 2024 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| biohackathon_url | https://biohackathon-europe.org/ | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| biohackathon_location | Barcelona, Spain | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| group | Project 13 - Interconnecting identifiers.org into a broader metadata connectivity | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| git_url | https://github.com/identifiers-org/biohackrxiv2025 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| authors_short | R. C. Juacaba Neto \emph{et al.} |
W3C defined CURIEs [@syntax2004curie] as the means to abbreviate URIs used in languages such as XML and SPARQL. Each CURIE is composed of two parts, a namespace prefix assigned to a collection of records and a unique identifier for a record within that collection (namespace). For example, the Protein Data Bank (namespace ‘pdb’) set of records, where a record identified with the ID 2gc4 can be referenced by the CURIE “pdb:2gc4”. These can be translated to a URI by prepending an appropriate resolver to the CURIE, thus allowing URIs to be written in an abbreviated manner.
This resolution relies on the namespace prefixes being unique globally (for a given resolution service). For this purpose, identifiers.org [@Bernal-Llinares2021-kx] offers a reliable and up-to-date registry of namespaces that contains the required resolution information as well as extra functionality and metadata. For example, the identifiers.org registry records URL patterns for all known resolution sites for a collection, alongside CURIE prefixes. These patterns allow CURIEs to be expanded into multiple access URLs, where available, and, since it uses URL patterns instead of URL prefixes (as in XML and SPARQL), it enables URL conversions where the record ID is placed in the middle of a URL, thereby having a prefix and suffix. Thus, the identifiers.org resolver uses the information from the registry to translate CURIE URIs into resolvable provider URLs. This resolver can be accessed via a REST endpoint that lists and scores (mainly uptime) of alternative URLs for a CURIE or via a redirection endpoint. The endpoint redirects HTTP(S) access to the best-evaluated resource. This can be used via URLs like http://identifiers.org/CURIE. For example: http://identifiers.org/pdb:2gc4. This canonical ‘identifiers.org’ URI thus provides persistent links which can reliably be used in papers and databases, since it will always use the latest URL patterns in the registry to redirect queries, and for canonical URIs provision. This mitigates issues that plague users, such as to systems going offline or changing web domains.
In this paper, we discuss the recent developments on identifiers.org to connect to a broader metadata ecosystem, implemented in two hackathons: the 2024 European BioHackathon and the 2024 German BioHackathon. At the former hackathon, we focused on connecting the registry with RDF and SPARQL services by defining an ontology for the registry information and exposing it by improving our SPARQL server. At the latter hackathon, we focused on expanding our metadata resolution service that collects metadata on specific CURIEs by incorporating different metadata providers and facilitating the addition of new metadata providers.
The registry service is available via a REST API as described in our documentation pages. While this is a widely used API format, it is not compatible with RDF and SPARQL systems. During the 2024 European BioHackathon, we developed a set of RDF terms that map from the identifiers.org relational database, then published them with the triples containing the database information, and finally incorporated them into our SPARQL server to make the data queryable. In the next few subsections, we further explain this work.
This is not the first time that a mapping of the identifiers.org registry into RDF has been proposed. An ontology for identifiers.org was previously proposed during the 46th SPARQLthon in 2016. This work attempts to build upon it. It has also been carried out in collaboration with the community present at the hackathon. The detailed mapping can be found on our SPARQL documentation page. It is mainly based on the DCAT [@dcat2024] and VoID [@void2011] RDF schemas. DCAT is widely used in EU data portals [@EuVocabsDcatAp] and allows for the description of datasets with annotations for elements such as data catalogs, web services, and distributions. Meanwhile, VoID allows for annotations on the ontology itself and the associated SPARQL server. We also utilized well known and accepted terms from Dublin Core [@DublinCore] and W3C. However, we had to define new terms for values (such as URL patterns) that could not be found in well-known RDF vocabularies. These are defined under the identifiers.org ontology terms with the prefix “idot” (identifiers.org terms).
The ontology and the registry triples are publicly available at our ontology repository. The registry triples are updated daily using the current data from our relational database with the help of the Ontop materialisation feature. Ontop is a virtual knowledge graph server that maps relational data into triples. It can operate on a server that generates triples on demand, or it can materialise all the triples to feed a conventional RDF server. We use the latter mode to allow users to access the dataset and to feed our own SPARQL server.
The identifiers.org SPARQL server [@Wimalaratne2015-gb] was originally implemented in 2015. Its objective was to perform translations from identifiers.org URLs to resolved URLs and vice versa. This allowed identifiers.org URLs to act as persistent URIs in knowledge graphs.
This was implemented as a virtual knowledge graph based on RDF4J. It would download the resolution dataset from the registry to record patterns and prefixes allowing it to translate URLs on the fly, rather than actually containing triples for all data entries from namespaces.
During the hackathon, we improved the original server to include the developed ontology and the triples generated by Ontop. It is also integrated with an automated pipeline that runs Ontop daily to update the materialised triples based on the registry contents.
With both the translation features from the original SPARQL server and the registry data integration, the identifiers.org server allows queries to link RDF data via federated queries to translate identifiers.org URLs and collect registry information on these URLs.
Additionally, two further changes were made to the server. First, the SPARQL server now has a frontend based on the sparql-editor provided by the Swiss Institute of Bioinformatics. This includes visualisation tools and examples to improve the user experience. Second, an endpoint was added to list the registry prefixes in the context list format supported by JSON-LD. This allows easier integration of identifiers.org prefixes into JSON-LD files.
Similarly to the resolver service, identifiers.org also offers a metadata resolution service that fetches metadata associated with a CURIE. This was initially based on JSON-LD available on the content behind resolved URLs, defined by schema.org and bioschemas.org, describing different types of data objects. This allows identifiers.org to provide metadata on CURIEs directly from the data providers. However, this approach is limited since not all data providers include JSON-LD annotations in their pages.
We expanded this concept to improve the number of namespaces for which the service is able to retrieve metadata. At the German Hackathon we introduced a more expansive framework capable of retrieving metadata from different sources.
The resulting expansion allows the service to handle different data types and sources via dedicated retrievers. That means that, if a data service is able to provide metadata on one or multiple namespaces in the identifiers.org registry, the metadata resolver is able to collect it. For example, if a new service provides metadata on proteins, we could implement a retriever for it and link it to protein-related namespaces, such as UniProt and InterPro. The only restriction is that the data service must be able to retrieve entries by ID. During the German hackathon we proposed two as proof of concept, one for TogoID and one for EBI Search.
TogoID [@Ikeda2025-tz] is an ID mapping service that contains relationships between IDs in repositories from various biological categories such as gene, protein, chemical compound, etc. This allows users to find CURIEs related to each other, such as how protein P12345 in UniProt has protein domain IPR000796 in InterPro.
EBI Search [@EBISearch2025] is a scalable text search engine that provides easy and uniform access to the biological data resources hosted at the European Bioinformatics Institute (EMBL-EBI), as well as some resources in collaborating institutions. Using the EBI Search API, it is possible to retrieve metadata associated with CURIEs in some namespaces.
The identifiers.org resolution frontend page was also expanded to display data acquired this way. Now, when a CURIE is resolved in the front page of identifiers.org, the resolution page shows a subset of the information acquired this way, for example: uniprot:P12345. Documentation on this API is available in our metadata service documentation page. The system is structured in a way that extra retrievers can be added if needed.
The work from both hackathons aims to improve metadata interoperability. One focuses on metadata for registry entries, while the other focuses on metadata for CURIEs.
Making the registry information queryable via SPARQL substantially improves the options others have to consume its information. Since it was only accessible via REST, users were limited to query the available endpoints and data was only accessible in JSON. Meanwhile, the SPARQL endpoint can return multiple RDF formats (XML, JSON-LD, etc.) and allows for much more complex selections and filtering compared to the current REST endpoints. This, in addition to the previous virtual knowledge graph, makes the identifiers.org SPARQL server a very useful component for services that use its URIs.
In addition, the metadata service is now capable of fetching metadata for specific CURIEs via a unified API. The expansion of retrievers improves this by allowing metadata to be collected from third-party sources, rather than being limited to structured data on the web page of resolved URLs, thus enabling data to be collected for more namespaces. The unified API for metadata collection also makes transparent the API of the metadata sources, allowing users to CURIE metadata on their pages.
Three main tasks have been identified as future work following these hackathons, which may be undertaken at future hackathon events. We discuss them in the following subsections.
Now that registry entries are queryable on our SPARQL server, we aim to link these with the virtual triples originally present. This would enable querying the namespace of an identifiers.org URL and collecting information about it, in addition to resolved URLs. For this, we would need to define a new property under the "idot" prefix or use an existing property, such as http://purl.org/dc/terms/isPartOf. We would then adjust the current server to include triples that link namespaces to identifiers.org CURIE URLs as part of the virtual knowledge graph.
By expanding the current set of retrievers on the metadata API, we would increase the coverage of the service across the registry namespaces. For example, EBI Search covers only a few namespaces related to data under EBI resources. These could be expanded, but the current set was selected as proof of concept based on usage metrics. Meanwhile, TogoID can collect metadata from many other namespaces.
We would prefer to avoid retrievers that are only capable of collecting data from a single namespace, though this may be feasible depending on the use case. For example, UniProt offers a rich API to collect protein information, but a retriever for this API would be available only for the uniprot namespace.
During the hackathons, interest in the search bar was raised. Some users expressed a desire to have this bar on their websites as a way to validate and search CURIEs. Two ideas were proposed: one to make the search bar’s JavaScript code into an embeddable widget, and the other to develop a browser extension to make the bar accessible on every website.
Both ideas could be considered for future hackathons, but further consideration is needed. For example, how to make the search bar easily embeddable on websites, since the current search bar is quite bulky and may not fit with the design of other websites. We could explore the possibility of incorporating the bar as part of a component library, such as TogoStanza [@Katayama2019-bx].
- http://identifiers.org
- https://docs.identifiers.org
- https://sparql.api.identifiers.org
- https://github.com/identifiers-org
- https://togoid.dbcls.jp/
- https://www.ebi.ac.uk/ebisearch
Special thanks to the DBCLS team and Jerven Bolleman for providing guidance on the mapping of the identifiers.org into RDF. RN and HH acknowledge the support of EMBL Core funding and FAIR-IMPACT, which is funded by the European Commission Horizon Europe programme, GA 101057344.