Compatibility to Linked Data Event Streams

[pietercolpaert]

We launched the Linked Data Event Streams (LDES) specification with a focus on making third parties replicate the full history of a dataset, and stay in-sync with it. It is available at https://w3id.org/ldes/specification

It has a different set-up as the delta-notifier’s v0.0.1 format, which is based on additions/updates/deletions of triples. LDES advocates defining a collection of entities (e.g., a collection of mandates). LDES requires you to add version identifiers to the mandates (e.g., ex:MayorofGhent2019v1) and add all triples that you want to expose in this particular collection (you can document this in a shacl shape). We thus only add triples to the system: triples describing a new version of the entity. In order to keep that manageable, you can describe a retention policy and that way remove older data or previous versions from disk.

Some examples of feeds already available:

• https://graph.irail.be/sncb/connections/feed
• https://marineregions.org/feed
• https://smartdata.dev-vlaanderen.be/base (links to all event streams in the address registry)
• https://apidg.gent.be/opendata/adlib2eventstream/v1/ (links to 5 museum catalogues and taxonomies of Ghent)

Components that already exist that allows you to work with LDES:

• LDES client to replicate and sync: https://github.com/TREEcg/event-stream-client/tree/main/packages/actor-init-ldes-client
• LDES to Linked Pipes: https://github.com/district09/ldes-to-linkedpipes
• Load an LDES in a MongoDB, GraphDB or Postgresql: https://github.com/osoc21/ldes2service
• Validator (in progress): https://treecg.github.io/TREE-LDES-visualizer/
• And feel free to look around on our website and github

I wonder whether the delta-notifier could also work with LDES by:

1. Subscribing to an LDES using the LDES client https://github.com/TREEcg/event-stream-client/tree/main/packages/actor-init-ldes-client
2. Sending the new version objects to the right mu-semtech microservice

It is then a responsibility of that microservice to understand what it needs to do with that version object. One option could be to just add triples as-is in a store (this is similar to what we do in LDES2Service). Another option could be to create a version materialization (making the subject’s IRI the IRI of the dcterms:isVersionOf object, and changing dcterms:created into dcterms:modified), removing all triples of the previous version and loading the new version in the store.

[madnificent]

This did not receive a public response yet but: thank you for the issue!

The delta-notifier is a bit more detailed in what it communicates. Constructing LDES feeds is therefore done on a per-case basis with a shared common ground. We may later construct something more generic. Leaving this issue open for the time being to keep that in sight. Related repositories:

• https://github.com/lblod/lpdc-publish-service
• https://github.com/redpencilio/fragmentation-producer-service