feat: add dataset relations support

[vera]

What this PR does / why we need it:

Draft PR: This is a follow-up to the feature proposal DEP004 – Dataset relations: Connecting research outputs in Dataverse and its presentation at the Dataverse community call on 07.07.2026. I'm opening this as a draft PR, although I believe this implementation is quite complete and functional already, I would like to invite further discussion about the implementation here.

Adds first-class dataset relations, allowing datasets to describe relationships with other internal Dataverse datasets or external resources.

Main functionality:

• Introduces version-scoped internal and external dataset relations.
• Adds configurable relation types, including display names, descriptions, inverse types, and a configurable default relation type.
• Adds APIs to create, list, replace, and delete relations, plus superuser-only APIs to manage relation types.
  • The relation list query includes filtering, pagination, deduplication, total counts, and optional facets.
  • By default, listing relations shows those defined at the latest accessible version of the requested dataset, plus any incoming relations defined at the latest published versions of other datasets. For the incoming relations, the relation type is mirrored using the type's inverse (if one exists).
  • Documentation for the APIs was added in doc/sphinx-guides/source/api/native-api.rst.
• Adds an EditDatasetRelations permission (by default, given to any role which previously had the EditDataset permission).
• Adds a replaceable DatasetRelationAlgorithm abstraction for retrieving and deduplicating relations. The current implementation, SqlDirectDatasetRelationAlgorithm, uses native SQL queries. (Alternative implementations could be introduced later without changing the API or service layer.)
  • The retrieval of the list of relations for a dataset is not trivial, especially because we want to include inverse relations which were created at other datasets and only point at our requested dataset. The SQL queries have been optimized, and database indexes were created, to achieve a reasonably good performance. See performance benchmarks below.
• Supports relation data in dataset creation, update, import, harvesting, version summaries, native JSON, and DataCite XML export.
  • Internal relations are exported to DataCite as relatedIdentifier entries when their relation type name matches a supported DataCite relation type.
• Adds relatedDatasetCount to Solr-indexed dataset records.

Special notes for your reviewer:

Some further details:

• The draft update/creation API (PUT /api/datasets/{id}/versions/:draft) now performs metadata and relation replacement in one transaction. Invalid relations therefore roll back the entire submitted update. To that end, a new command (UpdateDatasetVersionWithRelationsCommand) was added which encapsulates existing code of the API endpoint updateDraftVersion and calls the two relevant subcommands (CreateDatasetVersionCommand/UpdateDatasetVersionCommand and ReplaceDatasetRelationsCommand).
• The first relation type that is created becomes the default automatically. The default can subsequently be changed through the API, and is global (not per-dataverse). It is used for relations which are submitted without a type. It makes sense to set some generic type as default, such as "References". This was implemented this way, because relations with no type are difficult to handle during deduplication.
• The database migration creates the new relation tables and constraints, but does not migrate existing relation metadata. A production data migration will require additional work (some relations are stored in structured citation metadata and may be convertible with SQL, while other relations are stored as free text and will require some interpretation or manual mapping).

Suggestions on how to test this:

Run the relevant unit tests and integration tests, especially the newly added/extended test cases:

DatasetRelationsIT
DatasetRelationTypesIT
DatasetRelationIndexingTest
JsonParserTest
JsonPrinterTest
XmlMetadataTemplateTest
PermissionTest

which cover the added and changed functionality, including several edge and error cases. I recommend taking a look at the test cases in DatasetRelationsIT to get an overview of how this feature implementation works in practice.

To test performance of the feature, benchmark scripts are included (see tests/performance/relations/README.md). The following section shows some performance measurement results.

Performance measurements

Using synthetic benchmark data

The synthetic benchmark seed script creates and publishes a temporary collection and target dataset through the API. It then creates a number (-n, default: 10,000) of synthetic copies of that dataset in the database (which is faster than creating n real datasets through the API, and sufficient for the performance test). Finally, for each of the synthetically created datasets, it creates a relation pointing at the target dataset.

Then, a second script measures performance when calling the relation list API for the target dataset, which requests the first page of relations.

Setup:

$ tests/performance/relations/seed.sh -d 'postgresql://dataverse:secret@localhost:5432/dataverse' -b http://localhost:8080 -k '<superuser-api-token>'

Results of performance measurement:

Requesting a page size of 10:

$ tests/performance/relations/measure-list.sh  -b http://localhost:8080 -p perma:nfdi4health-509
...
requests=20 min=0.227s median=0.240s p95=0.254s max=0.256s mean=0.240s

Requesting a page size of 50:

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -l 50
...
requests=20 min=0.227s median=0.234s p95=0.254s max=0.268s mean=0.237s

Requesting a page size of 150:

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -l 150
...
requests=20 min=0.325s median=0.352s p95=0.386s max=0.396s mean=0.357s

Note that the meaningfulness of these numbers is limited, as it is only a relatively simple synthetic benchmark. For more realistic numbers, see the following section.

In real Dataverse instance

For this benchmark, we performed a migration of the relations in our production Dataverse instance to the new model using an SQL script. (The migration script is not generally applicable/included in this PR, since we previously stored relations in our own custom metadata fields, which are used as source data by the migration SQL.)

The following measurements were collected using the relation list API in this real environment.

Overall stats:

Total datasets: 56,547
-- With at least one relation: 56,142
 
Total dataset versions: 61,901
-- Published versions: 61,148
-- Draft versions: 718
-- Deaccessioned versions: 35

Total relations: 135,235
-- Internal relations (to another Dataverse dataset): 5,001
-- External relations (to an external resource): 130,234

Total relation types: 34

Relations by type:
      displayname       | count
------------------------+--------
 Is identical to        | 108253
 References             |  19924
 Is part of             |   1954
 Is described by        |   1440
 Is derived from        |    836
 Is source of           |    683
 Describes              |    488
 Continues              |    423
 Is continued by        |    391
 Has part               |    284
 Is supplemented by     |    210
 Documents              |    130
 Is supplement to       |    112
 Is previous version of |     30
 Is new version of      |     30
 Is documented by       |      9
 Is variant form of     |      7
 Is referenced by       |      7
 Is metadata for        |      6
 Is cited by            |      5
 Has metadata           |      5
 Is original form of    |      4
 Is reviewed by         |      2
 Has version            |      1
 Cites                  |      1

Out of all datasets, we selected the one with the most relations for the performance test:

Stats of the tested dataset (latest published version):

Total relations: 143
-- Internal relations: 140
-- External relations: 3
-- Incoming relations (defined at other datasets): 131
-- Self-defined relations: 12

Results of performance measurement:

Requesting a page size of 10:

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84
...
requests=20 min=0.239s median=0.256s p95=0.264s max=0.275s mean=0.255s

Requesting a page size of 25:

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -l 25
...
requests=20 min=0.302s median=0.326s p95=0.352s max=0.355s mean=0.330s

Requesting a page size of 50:

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -l 50
...
requests=20 min=0.433s median=0.459s p95=0.473s max=0.501s mean=0.459s

Requesting all relations:

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -l 150
...
requests=20 min=0.870s median=0.904s p95=0.935s max=0.946s mean=0.903s

Note that the response is a lot slower when the metadata blocks of all related datasets are included in the response (includeMetadataBlocks=true):

$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -m true
...
requests=20 min=2.024s median=2.096s p95=2.166s max=2.168s mean=2.094s
$ tests/performance/relations/measure-list.sh -b http://feat-search-result-grouping.csh.qa.km.k8s.zbmed.de -p perma:nfdi4health-84 -m true -l 25
...
requests=20 min=4.217s median=4.411s p95=4.540s max=4.595s mean=4.419s

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No direct user-interface changes are introduced. This PR adds backend APIs and response fields that can be consumed by the frontend.

Is there a release notes update needed for this change?:

Yes. This introduces a substantial new feature and several API changes.

Release notes should mention:

• Dataset relation support.
• New relation and relation-type APIs.
• The new EditDatasetRelations permission.
• Relation data in exports, imports, harvesting, and version summaries.
• Any required database migration steps.
• Steps for starting to use the feature, i.e. how to migrate existing relations to the new format.
  • [TODO] Production migration instructions are not yet available. A follow-up task is needed to define and implement migration tooling for structured relation fields and determine how free-text relation metadata should be handled.

Additional documentation:

• Dataset Relations Enhancement Proposal
• Native API documentation: doc/sphinx-guides/source/api/native-api.rst
• Relation type API example: scripts/api/data/relationTypes/relationTypeAllFields.json
• Performance benchmark documentation: tests/performance/relations/README.md

Open questions:

I have compiled a list of open questions in the DEP:

Should relation types be completely installation-specific or share common vocabularies?

Currently, a Dataverse installation starts out without any relation types and they must be configured by the administrator. Should some types come pre-installed?

How should external relation sources (e.g. DataCite) be integrated? The conceptual model/implemented entity model possibly needs to be adapted for this.

Currently, relations are modeled after metadata, i.e. they are versioned in the same way. For relations from external sources such as DataCite, this might not make sense. Some relations may not be directly associated with/defined as part of a specific dataset version.

Should inverse relations always be generated automatically? How should they be displayed in the UI?

Currently, inverse relations are always returned by the API. There's no param to turn that off.

In our UI, we label them with an "Unconfirmed" badge for clarity:

How should relationship approval workflows work?

Currently, there's no defined workflow of how to "approve" an incoming relationship.

(In other words, this is about how to turn an "Unconfirmed" relation as shown in the screenshot above into a confirmed one.)

What workflow would be most user-friendly?

How should transitive relationships be exposed? Performance of retrieval?

Currently, the implementation does not retrieve transitive relations, e.g. if relations like A -> B -> C exist, A and C are transitively related, but this relation is not returned anywhere.

Do we need this? Can the SQL query-based retrieval strategy provide this with reasonable performance?

UI integration strategy?

We have developed JS code for calling the APIs added in this PR, which might be used to integrate the feature into the React SPA.

How could an integration of this feature into React SPA be developed? As long as it is API-only, it's of limited use to users.

How can we migrate existing information from metadata fields?

As mentioned above, we have performed a test migration from our custom metadata fields to the new relation model using an SQL script. For installations who store their relations in the existing citation block fields such as "Related Publication", "Related Dataset" and "Related Material", no migration path has been implemented or tested yet.

Should a migration path be provided as part of this feature's release? (Probably?)

Could the model be extended to cover other types of relations, e.g. between data files?

Currently, only datasets may have relations (to other datasets or external resources).

In theory, the model can be extended to support other types of relation endpoints, if that is wanted.

Related issues:

• As a curator, I want to more easily add metadata about related resources so that it's more discoverable #5277

[vera]

Current IT failures:

/tmp/surefire-reports/TEST-edu.harvard.iq.dataverse.api.FilesIT$IngestSizeLimits.xml:    <failure message="1 expectation failed.&#10;JSON path data.total_count doesn&apos;t match.&#10;Expected: &lt;1&gt;&#10;  Actual: &lt;0&gt;&#10;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError:
/tmp/surefire-reports/TEST-edu.harvard.iq.dataverse.api.FilesIT$IngestSizeLimits.xml:    <failure message="1 expectation failed.&#10;Expected status code &lt;200&gt; but was &lt;400&gt;.&#10;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError:
/tmp/surefire-reports/TEST-edu.harvard.iq.dataverse.api.FilesIT$IngestSizeLimits.xml:    <failure message="1 expectation failed.&#10;Expected status code &lt;200&gt; but was &lt;400&gt;.&#10;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError:
/tmp/surefire-reports/TEST-edu.harvard.iq.dataverse.api.FilesIT$IngestSizeLimits.xml:    <failure message="1 expectation failed.&#10;Expected status code &lt;200&gt; but was &lt;400&gt;.&#10;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError:
/tmp/surefire-reports/TEST-edu.harvard.iq.dataverse.api.SearchIT.xml:    <failure message="1 expectation failed.&#10;JSON path data.total_count_per_object_type.Files doesn&apos;t match.&#10;Expected: is &lt;6&gt;&#10;  Actual: &lt;0&gt;&#10;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError:

Unsure if related to changes in this PR

[pdurbin]

We discussed this at tech hours today.

We gave this PR the 6.13 milestone, the next release, but @vera is flexible about this.

We gave it a size of 50, which is pretty big, but that might be an overestimate since Vera is quite happy to walk any reviewers through the code and docs.

As a reminder, she presented the whole idea back in July during a community call. The recording and her slides are available at https://groups.google.com/g/dataverse-community/c/-Ki-MWvvd2c/m/SGa3-GqjAgAJ