From f16ba1f886e8358ae51e5be8f7718ce2af8bbb9b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 13:56:49 -0400 Subject: [PATCH 01/71] stub out 6.6 release notes #11183 --- doc/release-notes/6.6-release-notes.md | 61 ++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 doc/release-notes/6.6-release-notes.md diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md new file mode 100644 index 00000000000..4517de2de89 --- /dev/null +++ b/doc/release-notes/6.6-release-notes.md @@ -0,0 +1,61 @@ +# Dataverse 6.6 + +Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.6 rather than the [list of releases](https://github.com/IQSS/dataverse/releases), which will cut them off. + +This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project! + +## Release Highlights + +Highlights for Dataverse 6.6 include: + +## Features Added + +## Bugs Fixed + +## API Updates + +## Settings Added + +## Backward Incompatible Changes + +Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. + +## Complete List of Changes + +For the complete list of code changes in this release, see the [6.6 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.6+is%3Aclosed) in GitHub. + +## Getting Help + +For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/g/dataverse-community) or email support@dataverse.org. + +## Installation + +If this is a new installation, please follow our [Installation Guide](https://guides.dataverse.org/en/latest/installation/). Please don't be shy about [asking for help](https://guides.dataverse.org/en/latest/installation/intro.html#getting-help) if you need it! + +Once you are in production, we would be delighted to update our [map of Dataverse installations](https://dataverse.org/installations) around the world to include yours! Please [create an issue](https://github.com/IQSS/dataverse-installations/issues) or email us at support@dataverse.org to join the club! + +You are also very welcome to join the [Global Dataverse Community Consortium](https://www.gdcc.io/) (GDCC). + +## Upgrade Instructions + +Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc. + +These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.5. + +0\. These instructions assume that you are upgrading from the immediate previous version. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version. + +If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. By default, Payara runs as the `dataverse` user. In the commands below, we use sudo to run the commands as a non-root user. + +Also, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. + +```shell +export PAYARA=/usr/local/payara6 +``` + +(or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) + +1\. List deployed applications + +```shell +$PAYARA/bin/asadmin list-applications +``` From 0b8514532701736b5a148433e51320d26a7cf544 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 14:36:35 -0400 Subject: [PATCH 02/71] Payara upgrade #11183 --- doc/release-notes/6.2025.2_update.md | 35 ------------- doc/release-notes/6.6-release-notes.md | 72 ++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 35 deletions(-) delete mode 100644 doc/release-notes/6.2025.2_update.md diff --git a/doc/release-notes/6.2025.2_update.md b/doc/release-notes/6.2025.2_update.md deleted file mode 100644 index 5724612c6e9..00000000000 --- a/doc/release-notes/6.2025.2_update.md +++ /dev/null @@ -1,35 +0,0 @@ -- Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - -## Installation - -The following assumes the standard instructions to `export PAYARA=/usr/local/payara6`, undeploy v6.5, stop payara have been done already. - -### Upgrade to Payara 6.2025.2 - -The steps below reuse your existing domain directory with the new distribution. You may also want to review the Payara upgrade instructions as it could be helpful during any troubleshooting: -[Payara Release Notes](https://docs.payara.fish/community/docs/6.2025.2/Release%20Notes/Release%20Notes%206.2025.2.html). -We also recommend you ensure you followed all update instructions from the past releases regarding Payara. -(The most recent Payara update was for [v6.3](https://github.com/IQSS/dataverse/releases/tag/v6.3).) - -Move the current Payara directory out of the way: - -```shell -mv $PAYARA $PAYARA.6.2024.6 -``` - -Download the new Payara version 6.2025.2 (from https://www.payara.fish/downloads/payara-platform-community-edition/ or https://nexus.payara.fish/repository/payara-community/fish/payara/distributions/payara/6.2025.2/payara-6.2025.2.zip), and unzip it in its place: - -```shell -cd /usr/local -unzip payara-6.2025.2.zip -``` - -Replace the brand new `payara/glassfish/domains/domain1` with your old, preserved domain1: - -```shell -mv payara6/glassfish/domains/domain1 payara6/glassfish/domains/domain1_DIST -mv payara6.6.2024.6/glassfish/domains/domain1 payara6/glassfish/domains/ -``` - -Deploy/install as normal - diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 4517de2de89..8c1cc868935 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,8 +8,12 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- infrastructure updates (Payara) + ## Features Added +- Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. + ## Bugs Fixed ## API Updates @@ -59,3 +63,71 @@ export PAYARA=/usr/local/payara6 ```shell $PAYARA/bin/asadmin list-applications ``` + +2\. Undeploy the previous version (should match "list-applications" above) + +```shell +$PAYARA/bin/asadmin undeploy dataverse-6.4 +``` + +3\. Stop Payara + +```shell +sudo service payara stop +``` + +4\. Upgrade to Payara 6.2025.2 + +The steps below reuse your existing domain directory with the new distribution of Payara. You may also want to review the Payara upgrade instructions as it could be helpful during any troubleshooting: +[Payara Release Notes](https://docs.payara.fish/community/docs/6.2025.2/Release%20Notes/Release%20Notes%206.2025.2.html). +We also recommend you ensure you followed all update instructions from the past releases regarding Payara. +(The most recent Payara update was for [v6.3](https://github.com/IQSS/dataverse/releases/tag/v6.3).) + +Move the current Payara directory out of the way: + +```shell +mv $PAYARA $PAYARA.6.2024.6 +``` + +Download the new Payara version 6.2025.2 (from https://www.payara.fish/downloads/payara-platform-community-edition/ or https://nexus.payara.fish/repository/payara-community/fish/payara/distributions/payara/6.2025.2/payara-6.2025.2.zip), and unzip it in its place: + +```shell +cd /usr/local +unzip payara-6.2025.2.zip +``` + +Replace the brand new `payara/glassfish/domains/domain1` with your old, preserved domain1: + +```shell +mv payara6/glassfish/domains/domain1 payara6/glassfish/domains/domain1_DIST +mv payara6.6.2024.6/glassfish/domains/domain1 payara6/glassfish/domains/ +``` + +5\. Download and deploy this version + +```shell +wget https://github.com/IQSS/dataverse/releases/download/v6.5/dataverse-6.6.war +$PAYARA/bin/asadmin deploy dataverse-6.6.war +``` + +Note: if you have any trouble deploying, stop Payara, remove the following directories, start Payara, and try to deploy again. + +```shell +sudo service payara stop +sudo rm -rf $PAYARA/glassfish/domains/domain1/generated +sudo rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache +sudo rm -rf $PAYARA/glassfish/domains/domain1/lib/databases +sudo service payara start +``` + +6\. For installations with internationalization: + +Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). + +7\. Restart Payara + +```shell +sudo service payara stop +sudo service payara start +``` + From ae5132b97241b7322ae8ab6750e895f175f4e8d9 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 14:47:13 -0400 Subject: [PATCH 03/71] upgrade Solr #11183 --- .../10713-Solr9.8.0 and lib updates.md | 9 ----- doc/release-notes/6.6-release-notes.md | 36 ++++++++++++++++++- 2 files changed, 35 insertions(+), 10 deletions(-) delete mode 100644 doc/release-notes/10713-Solr9.8.0 and lib updates.md diff --git a/doc/release-notes/10713-Solr9.8.0 and lib updates.md b/doc/release-notes/10713-Solr9.8.0 and lib updates.md deleted file mode 100644 index a42960301ef..00000000000 --- a/doc/release-notes/10713-Solr9.8.0 and lib updates.md +++ /dev/null @@ -1,9 +0,0 @@ -Solr 9.8.0 is now the version recommended in our installation guides and used with automated testing. Other libraries Dataverse uses have been updated as well. - -For the upgrade instructions section: - -[note that 6.6 may contain other solr-related changes, so the instructions may need to contain information merged from multiple release notes!] - -If you are upgrading Solr: - - Install solr-9.8.0 following the instructions from the Installation guide. - - Run a full reindex to populate the search catalog. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 8c1cc868935..97c06d3935d 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,11 +8,12 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: -- infrastructure updates (Payara) +- infrastructure updates (Payara and Solr) ## Features Added - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. +- Solr has been upgraded to version 9.8.0. See #10713. ## Bugs Fixed @@ -131,3 +132,36 @@ sudo service payara stop sudo service payara start ``` +8\. Upgrade Solr + +Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. We recommend to upgrading. + +Install Solr 9.8.0 following the [instructions](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr) from the Installation Guide. + +The instructions in the guide suggest to use the config files from the installer zip bundle. When upgrading an existing instance, it may be easier to download them from the source tree: + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/solrconfig.xml +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/schema.xml +cp solrconfig.xml schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf +``` + +8a\. For installations with custom or experimental metadata blocks, update fields + +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr-init-script)). + +- Run the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the correct path of your Solr installation): + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/update-fields.sh +chmod +x update-fields.sh +curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.8.0/server/solr/collection1/conf/schema.xml +``` + +- Start Solr instance (usually `service solr start` depending on Solr/OS). + +9\. Reindex Solr + +```shell +curl http://localhost:8080/api/admin/index +``` From f1332571d1bb61f7f5ca2a3df10cd4bbab9b2e11 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 14:55:49 -0400 Subject: [PATCH 04/71] move collection --- doc/release-notes/10304-add-move-dataverse-collection.md | 5 ----- doc/release-notes/6.6-release-notes.md | 2 ++ 2 files changed, 2 insertions(+), 5 deletions(-) delete mode 100644 doc/release-notes/10304-add-move-dataverse-collection.md diff --git a/doc/release-notes/10304-add-move-dataverse-collection.md b/doc/release-notes/10304-add-move-dataverse-collection.md deleted file mode 100644 index 1ddeb526c61..00000000000 --- a/doc/release-notes/10304-add-move-dataverse-collection.md +++ /dev/null @@ -1,5 +0,0 @@ -### Move a collection from the dashboard - -In addition to the api [Move a Dataverse Collection](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. - -For more information, see #10304. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 97c06d3935d..8f782a8f4c7 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,10 +8,12 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- collections can be moved from the superuser dashboard (API only, previously). - infrastructure updates (Payara and Solr) ## Features Added +- In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. From def16787b8b82edabb94759f0230db4215c3c9fd Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 15:13:09 -0400 Subject: [PATCH 05/71] license enhancements --- .../10883-license improvements.md | 12 --------- doc/release-notes/6.6-release-notes.md | 27 +++++++++++++++++++ 2 files changed, 27 insertions(+), 12 deletions(-) delete mode 100644 doc/release-notes/10883-license improvements.md diff --git a/doc/release-notes/10883-license improvements.md b/doc/release-notes/10883-license improvements.md deleted file mode 100644 index 30b161f9bd5..00000000000 --- a/doc/release-notes/10883-license improvements.md +++ /dev/null @@ -1,12 +0,0 @@ -- License metadata enhancements (#10883): - - Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode - - Updated DataCite metadata export to include rightsIdentifier, rightsIdentifierScheme, and schemeUri consistent with the DataCite 4.5 schema and examples - - Enhanced metadata exports to include all new license fields - - Existing licenses from the example set included with Dataverse will be automatically updated with new fields - - Existing API calls support the new optional fields - - Setup: For existing published datasets, the additional license metadata will not be available from DataCite or in metadata exports until the dataset is republished or - - the /api/admin/metadata/{id}/reExportDataset is run for the dataset - - the api/datasets/{id}/modifyRegistrationMetadata API is run for the dataset, - or the global version of these api calls (/api/admin/metadata/reExportAll, /api/datasets/modifyRegistrationPIDMetadataAll) are used. - \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 8f782a8f4c7..96333e2ceba 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -13,6 +13,18 @@ Highlights for Dataverse 6.6 include: ## Features Added +### License Metadata Enhancements + +- Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode (see JSON files under [Adding Licenses](https://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses) in the guides) +- Updated DataCite metadata export to include rightsIdentifier, rightsIdentifierScheme, and schemeUri consistent with the DataCite 4.5 schema and examples +- Enhanced metadata exports to include all new license fields +- Existing licenses from the example set included with Dataverse will be automatically updated with new fields +- Existing API calls support the new optional fields + +See below for upgrade instructions. See also #10883 and #11232. + +### Other Features + - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. @@ -167,3 +179,18 @@ curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /u ```shell curl http://localhost:8080/api/admin/index ``` + +10\. Run reExportAll to update dataset metadata exports + +For existing published datasets, additional license metadata will not be available from DataCite or in metadata exports until + +- the dataset is republished or +- the /api/admin/metadata/{id}/reExportDataset is run for the dataset or +- the /api/datasets/{id}/modifyRegistrationMetadata API is run for the dataset or +- the global version of these API calls (/api/admin/metadata/reExportAll, /api/datasets/modifyRegistrationPIDMetadataAll) are used. + +For this reason, we recommend reexporting all dataset metadata. For more advanced usage, please see [the guides](http://guides.dataverse.org/en/6.6/admin/metadataexport.html#batch-exports-through-the-api). + +```shell +curl http://localhost:8080/api/admin/metadata/reExportAll +``` From e276680edd2f139214274ccb289ce3136c1eec7c Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 15:30:44 -0400 Subject: [PATCH 06/71] range search --- doc/release-notes/10887-solr-field-types.md | 102 -------------------- doc/release-notes/6.6-release-notes.md | 40 ++++++++ 2 files changed, 40 insertions(+), 102 deletions(-) delete mode 100644 doc/release-notes/10887-solr-field-types.md diff --git a/doc/release-notes/10887-solr-field-types.md b/doc/release-notes/10887-solr-field-types.md deleted file mode 100644 index 1afbae34e90..00000000000 --- a/doc/release-notes/10887-solr-field-types.md +++ /dev/null @@ -1,102 +0,0 @@ -This release enhances how numerical and date fields are indexed in Solr. Previously, all fields were indexed as English text (text_en), but with this update: - -* Integer fields are indexed as `plong` -* Float fields are indexed as `pdouble` -* Date fields are indexed as `date_range` (`solr.DateRangeField`) - -Specifically, the following fields were updated: - -- coverage.Depth -- coverage.ObjectCount -- coverage.ObjectDensity -- coverage.Redshift.MaximumValue -- coverage.Redshift.MinimumValue -- coverage.RedshiftValue -- coverage.SkyFraction -- coverage.Spectral.CentralWavelength -- coverage.Spectral.MaximumWavelength -- coverage.Spectral.MinimumWavelength -- coverage.Temporal.StartTime -- coverage.Temporal.StopTime -- dateOfCollectionEnd -- dateOfCollectionStart -- dateOfDeposit -- distributionDate -- dsDescriptionDate -- journalPubDate -- productionDate -- resolution.Redshift -- targetSampleActualSize -- timePeriodCoveredEnd -- timePeriodCoveredStart - -This change enables range queries when searching from both the UI and the API, such as `dateOfDeposit:[2000-01-01 TO 2014-12-31]` or `targetSampleActualSize:[25 TO 50]`. - -Dataverse administrators must update their Solr schema.xml (manually or by rerunning `update-fields.sh`) and reindex all datasets. - -Additionally, search result highlighting is now more accurate, ensuring that only fields relevant to the query are highlighted in search results. If the query is specifically limited to certain fields, the highlighting is now limited to those fields as well. - -## Upgrade Instructions - -7\. Update Solr schema.xml file. Start with the standard v6.5 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a). - -Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.5/installation/prerequisites.html#solr-init-script)). - -```shell -service solr stop -``` - -Replace schema.xml - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/schema.xml -cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf -``` - -Start Solr (but if you use any custom metadata blocks, perform the next step, 7a first). - -```shell -service solr start -``` - -7a\. For installations with custom or experimental metadata blocks: - -Before starting Solr, update the schema to include all the extra metadata fields that your installation uses. We do this by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the names of the directories, if different): - -```shell - wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/update-fields.sh - chmod +x update-fields.sh - curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml -``` - -Now start Solr. - -8\. Reindex Solr - -Note: these instructions are a little different than usual because we observed a strange error about `DOCS_AND_FREQS_AND_POSITIONS` when testing upgrades (see #11139 for details). Extra steps about explicitly clearing the index and reloading the core are included. If you run into trouble, as a last resort, you could reinstall Solr completely and then reindex. - -Clear the Solr index: - -```shell -curl http://localhost:8080/api/admin/index/clear -``` - -Make sure the Solr index is empty: - -```shell -curl "http://localhost:8983/solr/collection1/select?rows=1000000&wt=json&indent=true&q=*%3A*" -``` - -Reload the Solr core: - -```shell -curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1" -``` - -Below is the simplest way to reindex Solr: - -```shell -curl http://localhost:8080/api/admin/index -``` - -The API above rebuilds the existing index "in place". If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 96333e2ceba..0d110117f87 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,6 +8,8 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- metadata fields now support range searches (dates, integers, etc.) +- more accurate search highlighting - collections can be moved from the superuser dashboard (API only, previously). - infrastructure updates (Payara and Solr) @@ -23,6 +25,44 @@ Highlights for Dataverse 6.6 include: See below for upgrade instructions. See also #10883 and #11232. +### Range Search + +This release enhances how numerical and date fields are indexed in Solr. Previously, all fields were indexed as English text (`text_en`), but with this update: + +* Integer fields are indexed as `plong` +* Float fields are indexed as `pdouble` +* Date fields are indexed as `date_range` (`solr.DateRangeField`) + +This change enables range queries when searching from both the UI and the API, such as `dateOfDeposit:[2000-01-01 TO 2014-12-31]` or `targetSampleActualSize:[25 TO 50]`. See below for a full list of fields that now support range search. + +Additionally, search result highlighting is now more accurate, ensuring that only fields relevant to the query are highlighted in search results. If the query is specifically limited to certain fields, the highlighting is now limited to those fields as well. See #10887. + +Specifically, the following fields were updated: + +- coverage.Depth +- coverage.ObjectCount +- coverage.ObjectDensity +- coverage.Redshift.MaximumValue +- coverage.Redshift.MinimumValue +- coverage.RedshiftValue +- coverage.SkyFraction +- coverage.Spectral.CentralWavelength +- coverage.Spectral.MaximumWavelength +- coverage.Spectral.MinimumWavelength +- coverage.Temporal.StartTime +- coverage.Temporal.StopTime +- dateOfCollectionEnd +- dateOfCollectionStart +- dateOfDeposit +- distributionDate +- dsDescriptionDate +- journalPubDate +- productionDate +- resolution.Redshift +- targetSampleActualSize +- timePeriodCoveredEnd +- timePeriodCoveredStart + ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. From 5590bbe2f1eec47b86684cbf49109ee9e6cc71d4 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 15:54:27 -0400 Subject: [PATCH 07/71] datasets without files --- .../10981-published-datasets-should-contain-files.md | 11 ----------- doc/release-notes/6.6-release-notes.md | 5 +++++ 2 files changed, 5 insertions(+), 11 deletions(-) delete mode 100644 doc/release-notes/10981-published-datasets-should-contain-files.md diff --git a/doc/release-notes/10981-published-datasets-should-contain-files.md b/doc/release-notes/10981-published-datasets-should-contain-files.md deleted file mode 100644 index 964e7ff1937..00000000000 --- a/doc/release-notes/10981-published-datasets-should-contain-files.md +++ /dev/null @@ -1,11 +0,0 @@ -## Feature: Prevent publishing Datasets without files -A new attribute was added to Collections in order to control the publishing of Datasets without files. -Once set to "True", the publishing of a Dataset within a Collection, without files, will be blocked for all non superusers. -In order to configure a Collection to block publishing a superuser must set the attribute "requireFilesToPublishDataset" to "True". -The collection's hierarchy will be checked if the collection's "requireFilesToPublishDataset" attribute is not set explicitly to "True" or "False". - -```shell -curl -X PUT -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/dataverses/$ID/attribute/requireFilesToPublishDataset?value=true" -``` - -See also #10981. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 0d110117f87..fc8a47f1904 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -11,6 +11,7 @@ Highlights for Dataverse 6.6 include: - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting - collections can be moved from the superuser dashboard (API only, previously). +- optionally prevent publishing of datasets without files - infrastructure updates (Payara and Solr) ## Features Added @@ -63,6 +64,10 @@ Specifically, the following fields were updated: - timePeriodCoveredEnd - timePeriodCoveredStart +## Prevent Publishing of Datasets Without Files + +Datasets without files can be optionally prevented from being published though a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. + ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. From 046bca7d7191fd6ff6d42dcaa2d3cbdd104c97e8 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 15:55:15 -0400 Subject: [PATCH 08/71] license enhancements --- doc/release-notes/6.6-release-notes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index fc8a47f1904..8ebc2d9d8d3 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,6 +8,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- license metadata enhancements - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting - collections can be moved from the superuser dashboard (API only, previously). From 2d8e851b370a3f11e084748efe8a7e1bf93ae5d3 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 16:09:49 -0400 Subject: [PATCH 09/71] anon preview url --- doc/release-notes/11159-preview-url-update.md | 7 ------- doc/release-notes/6.6-release-notes.md | 6 ++++++ 2 files changed, 6 insertions(+), 7 deletions(-) delete mode 100644 doc/release-notes/11159-preview-url-update.md diff --git a/doc/release-notes/11159-preview-url-update.md b/doc/release-notes/11159-preview-url-update.md deleted file mode 100644 index 2a84540ae75..00000000000 --- a/doc/release-notes/11159-preview-url-update.md +++ /dev/null @@ -1,7 +0,0 @@ -### Preview URL popup updated - -The Preview URL popup and related documentation has been updated to give the dataset more information about anonymous access including the names of the dataset fields that will be withheld from the url user and suggesting how to review the url before releasing it. See also #11159 and #11164. - -###Bug Fix - -Bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 8ebc2d9d8d3..f07f7f00048 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -69,14 +69,20 @@ Specifically, the following fields were updated: Datasets without files can be optionally prevented from being published though a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. +### Preview URL popup updated + + ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. +- The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. ## Bugs Fixed +- A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. + ## API Updates ## Settings Added From a9c49c2970d9d36a42de133832e61dc78bf18d96 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 16:15:42 -0400 Subject: [PATCH 10/71] ontoportal --- .../11256-news-on-support-for-external-vocabulary.md | 4 ---- doc/release-notes/6.6-release-notes.md | 4 +--- 2 files changed, 1 insertion(+), 7 deletions(-) delete mode 100644 doc/release-notes/11256-news-on-support-for-external-vocabulary.md diff --git a/doc/release-notes/11256-news-on-support-for-external-vocabulary.md b/doc/release-notes/11256-news-on-support-for-external-vocabulary.md deleted file mode 100644 index 180f888e4df..00000000000 --- a/doc/release-notes/11256-news-on-support-for-external-vocabulary.md +++ /dev/null @@ -1,4 +0,0 @@ -### News on Support for External Vocabulary Services - -It is now possible to fill Dataverse keywords metadata using an OntoPortal service. -The code has been shared on [GDCC GitHub Repository](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production). \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index f07f7f00048..4ce57dbc1f5 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -69,13 +69,11 @@ Specifically, the following fields were updated: Datasets without files can be optionally prevented from being published though a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. -### Preview URL popup updated - - ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. +- It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. From 8a694bae9dea0c4891f6c3b6c28fcd38ebc9b587 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 16:24:51 -0400 Subject: [PATCH 11/71] ror --- doc/release-notes/11075-ror.md | 15 --------------- doc/release-notes/6.6-release-notes.md | 23 ++++++++++++++++++----- 2 files changed, 18 insertions(+), 20 deletions(-) delete mode 100644 doc/release-notes/11075-ror.md diff --git a/doc/release-notes/11075-ror.md b/doc/release-notes/11075-ror.md deleted file mode 100644 index de32eebccdf..00000000000 --- a/doc/release-notes/11075-ror.md +++ /dev/null @@ -1,15 +0,0 @@ -### ROR (Research Organization Registry) as Author Identifier Type - -ROR (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. As with all author identifiers, be sure to select the proper identifier type (e.g. "ROR") and enter just the unique identifier (e.g. "03vek6s52") rather than the full URL (e.g. "https://ror.org/03vek6s52"). Like ORCID, ROR will appear in the "Datacite" metadata export format. See also the [ROR](https://ror.org) website, a new [note](https://dataverse-guide--11118.org.readthedocs.build/en/11118/user/dataset-management.html#adding-a-new-dataset) in a guides about entering author identifiers, #11075, and #11118. - -## Upgrade Instructions - -6\. Update metadata blocks - -These changes reflect incremental improvements made to the handling of core metadata fields. Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages). - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/citation.tsv - -curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv -``` diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 4ce57dbc1f5..8f7b38bbe41 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -73,6 +73,7 @@ Datasets without files can be optionally prevented from being published though a - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. +- [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. @@ -196,9 +197,21 @@ sudo service payara stop sudo service payara start ``` -8\. Upgrade Solr +8\. Update metadata blocks -Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. We recommend to upgrading. +These changes reflect incremental improvements made to the handling of core metadata fields. + +Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages). + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/citation.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv +``` + +9\. Upgrade Solr + +Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature, the default `schema.xml` files has changed so you must upgrade. Install Solr 9.8.0 following the [instructions](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr) from the Installation Guide. @@ -210,7 +223,7 @@ wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/schema.xml cp solrconfig.xml schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf ``` -8a\. For installations with custom or experimental metadata blocks, update fields +9a\. For installations with custom or experimental metadata blocks, update fields - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr-init-script)). @@ -224,13 +237,13 @@ curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /u - Start Solr instance (usually `service solr start` depending on Solr/OS). -9\. Reindex Solr +10\. Reindex Solr ```shell curl http://localhost:8080/api/admin/index ``` -10\. Run reExportAll to update dataset metadata exports +11\. Run reExportAll to update dataset metadata exports For existing published datasets, additional license metadata will not be available from DataCite or in metadata exports until From 1e440eff69a1cba3ff0cb932ce114f656e71b274 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 16:48:54 -0400 Subject: [PATCH 12/71] bug fixes --- doc/release-notes/11133-search-fix.md | 3 --- doc/release-notes/11142-more-detailed-file-differences.md | 3 --- doc/release-notes/11234-DOI Publishing Bug.md | 1 - doc/release-notes/6.6-release-notes.md | 5 +++++ doc/release-notes/openaire_fix.md | 1 - 5 files changed, 5 insertions(+), 8 deletions(-) delete mode 100644 doc/release-notes/11133-search-fix.md delete mode 100644 doc/release-notes/11142-more-detailed-file-differences.md delete mode 100644 doc/release-notes/11234-DOI Publishing Bug.md delete mode 100644 doc/release-notes/openaire_fix.md diff --git a/doc/release-notes/11133-search-fix.md b/doc/release-notes/11133-search-fix.md deleted file mode 100644 index 88962b70ea0..00000000000 --- a/doc/release-notes/11133-search-fix.md +++ /dev/null @@ -1,3 +0,0 @@ -### Search fix when using AVOID_EXPENSIVE_SOLR_JOIN=true - -Dataverse v6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the AVOID_EXPENSIVE_SOLR_JOIN feature flag is set to true. This releases fixes the bug. diff --git a/doc/release-notes/11142-more-detailed-file-differences.md b/doc/release-notes/11142-more-detailed-file-differences.md deleted file mode 100644 index d7425b8d2fa..00000000000 --- a/doc/release-notes/11142-more-detailed-file-differences.md +++ /dev/null @@ -1,3 +0,0 @@ -The file page version table now shows more detail, e.g. when there are metadata changes or whether a file has been replaced. -A bug that causes adding free-form provenance to a file to fail has been fixed. -See also #11142 and #11145. diff --git a/doc/release-notes/11234-DOI Publishing Bug.md b/doc/release-notes/11234-DOI Publishing Bug.md deleted file mode 100644 index 2f8743cbe5c..00000000000 --- a/doc/release-notes/11234-DOI Publishing Bug.md +++ /dev/null @@ -1 +0,0 @@ -This release fixes a bug which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 8f7b38bbe41..6971fdebbb8 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -81,6 +81,11 @@ Datasets without files can be optionally prevented from being published though a ## Bugs Fixed - A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. +- The file page version table now shows whether a file has been replaced. See #11142 and #11145. +- The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 +- A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. +- A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. +- Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This releases fixes the bug. See #11133 and #11134. ## API Updates diff --git a/doc/release-notes/openaire_fix.md b/doc/release-notes/openaire_fix.md deleted file mode 100644 index 49a809db14b..00000000000 --- a/doc/release-notes/openaire_fix.md +++ /dev/null @@ -1 +0,0 @@ -bugfix: openaire implementation can now correctly process one or multiple productionPlaces as geolocation \ No newline at end of file From db2ce1d366ff02ff301505587447d1e0123684c2 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 16:58:56 -0400 Subject: [PATCH 13/71] boolean --- doc/release-notes/11064-update-metadata-customization.md | 1 - doc/release-notes/6.6-release-notes.md | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 doc/release-notes/11064-update-metadata-customization.md diff --git a/doc/release-notes/11064-update-metadata-customization.md b/doc/release-notes/11064-update-metadata-customization.md deleted file mode 100644 index bcead3497f8..00000000000 --- a/doc/release-notes/11064-update-metadata-customization.md +++ /dev/null @@ -1 +0,0 @@ -Metadata Customization guide has been updated to explain how to implement a kind of boolean fieldtype (see [Metadata Customization Guide](https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties)) \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 6971fdebbb8..1bf9eed63c8 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -74,6 +74,7 @@ Datasets without files can be optionally prevented from being published though a - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. - [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. +- The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #11064 and #11064. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. From 580d2339dde1eae88ae3009c8b63e31590d6d77d Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 10 Mar 2025 17:05:35 -0400 Subject: [PATCH 14/71] more bugs --- doc/release-notes/11053-metadata-styling-fixes | 1 - doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md | 1 - doc/release-notes/6.6-release-notes.md | 2 ++ 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 doc/release-notes/11053-metadata-styling-fixes delete mode 100644 doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md diff --git a/doc/release-notes/11053-metadata-styling-fixes b/doc/release-notes/11053-metadata-styling-fixes deleted file mode 100644 index c1f2c69c283..00000000000 --- a/doc/release-notes/11053-metadata-styling-fixes +++ /dev/null @@ -1 +0,0 @@ -Minor styling fixes for the Related Publication Field and fields using ORCID or ROR have been made (see #11053, #10964, #11106) diff --git a/doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md b/doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md deleted file mode 100644 index 230d69c9b9f..00000000000 --- a/doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md +++ /dev/null @@ -1 +0,0 @@ -Fix a bug with My Data where listing dataverses for a user with only rights on harvested dataverses would result in a server error response. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 1bf9eed63c8..49a37aba2db 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -87,6 +87,8 @@ Datasets without files can be optionally prevented from being published though a - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This releases fixes the bug. See #11133 and #11134. +- Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. +- Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. ## API Updates From eac8b1eac05c15173d462489e051a28045c59c33 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 09:31:44 -0400 Subject: [PATCH 15/71] typos --- doc/release-notes/6.6-release-notes.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 49a37aba2db..6a7f3bea9fc 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -11,7 +11,7 @@ Highlights for Dataverse 6.6 include: - license metadata enhancements - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting -- collections can be moved from the superuser dashboard (API only, previously). +- collections can be moved from the superuser dashboard - optionally prevent publishing of datasets without files - infrastructure updates (Payara and Solr) @@ -67,14 +67,14 @@ Specifically, the following fields were updated: ## Prevent Publishing of Datasets Without Files -Datasets without files can be optionally prevented from being published though a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. +Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. - [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. -- The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #11064 and #11064. +- The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. @@ -86,7 +86,7 @@ Datasets without files can be optionally prevented from being published though a - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. -- Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This releases fixes the bug. See #11133 and #11134. +- Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. From fdabe382f4c79417c2450373edaa1e661d99c449 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 09:44:31 -0400 Subject: [PATCH 16/71] 3d objects --- .../11120-add-3d-object-metadata-block.md | 60 ------------------- doc/release-notes/6.6-release-notes.md | 15 +++++ 2 files changed, 15 insertions(+), 60 deletions(-) delete mode 100644 doc/release-notes/11120-add-3d-object-metadata-block.md diff --git a/doc/release-notes/11120-add-3d-object-metadata-block.md b/doc/release-notes/11120-add-3d-object-metadata-block.md deleted file mode 100644 index a43be86bc7d..00000000000 --- a/doc/release-notes/11120-add-3d-object-metadata-block.md +++ /dev/null @@ -1,60 +0,0 @@ -### New 3D Object Data Metadata Block - -A new metadata block has been added for describing 3D object data. You can download it from the [guides](https://dataverse-guide--11167.org.readthedocs.build/en/11167/user/appendix.html). See also #11120 and #11167. - -All new Dataverse installations will receive this metadata block by default. We recommend adding it by following the upgrade instructions below. - -## Upgrade Instructions - -### For 6.6-Release-notes.md - -6\. Restart Payara - -7\. Update metadata blocks - -These changes reflect incremental improvements made to the handling of core metadata fields. - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/citation.tsv - -curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv -``` -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/3d_objects.tsv - -curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file 3d_objects.tsv -``` - -8\. Update Solr schema.xml file. Start with the standard v6.6 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 8a). - -Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr-init-script)). - -```shell -service solr stop -``` - -Replace schema.xml - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/schema.xml -cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf -``` - -Start Solr (but if you use any custom metadata blocks or adding 3D Objects, perform the next step, 8a first). - -```shell -service solr start -``` -8a\. For installations with custom or experimental metadata blocks: - -Before starting Solr, update the schema to include all the extra metadata fields that your installation uses. We do this by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the names of the directories, if different): - -```shell - wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/update-fields.sh - chmod +x update-fields.sh - curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml -``` - -Now start Solr. - -9\. Reindex Solr diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 6a7f3bea9fc..5f193478dc0 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -12,6 +12,7 @@ Highlights for Dataverse 6.6 include: - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting - collections can be moved from the superuser dashboard +- new 3D Objects metadata block - optionally prevent publishing of datasets without files - infrastructure updates (Payara and Solr) @@ -65,6 +66,12 @@ Specifically, the following fields were updated: - timePeriodCoveredEnd - timePeriodCoveredStart +### New 3D Objects Metadata Block + +A new metadata block has been added for describing 3D object data. You can download it from the [guides](https://guides.dataverse.org/en/6.6/user/appendix.html). See also #11120 and #11167. + +All new Dataverse installations will receive this metadata block by default. We recommend adding it by following the upgrade instructions below. + ## Prevent Publishing of Datasets Without Files Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. @@ -217,6 +224,14 @@ wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/meta curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv ``` +The 3D Objects metadata block is included in all new installations of Dataverse so we recommend adding it. + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/3d_objects.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file 3d_objects.tsv +``` + 9\. Upgrade Solr Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature, the default `schema.xml` files has changed so you must upgrade. From 2c18a4066b0900cab01a3efd18edf9242352a6e7 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 09:54:51 -0400 Subject: [PATCH 17/71] delete files, citation --- ...0735-search-dataset-name-and-dataset-citation-different.md | 4 ---- doc/release-notes/11230-deleteFiles api call.md | 1 - doc/release-notes/6.6-release-notes.md | 3 +++ 3 files changed, 3 insertions(+), 5 deletions(-) delete mode 100644 doc/release-notes/10735-search-dataset-name-and-dataset-citation-different.md delete mode 100644 doc/release-notes/11230-deleteFiles api call.md diff --git a/doc/release-notes/10735-search-dataset-name-and-dataset-citation-different.md b/doc/release-notes/10735-search-dataset-name-and-dataset-citation-different.md deleted file mode 100644 index 6a6b2008772..00000000000 --- a/doc/release-notes/10735-search-dataset-name-and-dataset-citation-different.md +++ /dev/null @@ -1,4 +0,0 @@ - -### Search files Bug fix - -dataset-citation was displaying DRAFT version instead of latest released version diff --git a/doc/release-notes/11230-deleteFiles api call.md b/doc/release-notes/11230-deleteFiles api call.md deleted file mode 100644 index 98754f24e60..00000000000 --- a/doc/release-notes/11230-deleteFiles api call.md +++ /dev/null @@ -1 +0,0 @@ -A new /api/datasets/{id}/deleteFiles call has beed added to the API, allowing delete of multiple file from the latest version of a dataset in one operation. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 5f193478dc0..469113c24fd 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -96,9 +96,12 @@ Datasets without files can be optionally prevented from being published through - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. +- In the Search API, files were displaying DRAFT version instead of latest released version under `dataset_citation`. See #10735 and #11051. ## API Updates +- Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. + ## Settings Added ## Backward Incompatible Changes From 4402abdc2c244802dfdfb3c30428ff099279f33a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:02:05 -0400 Subject: [PATCH 18/71] search api fields for files --- ...7-extend-datasets-files-from-search-api.md | 22 ------------------ doc/release-notes/6.6-release-notes.md | 23 ++++++++++++++++++- 2 files changed, 22 insertions(+), 23 deletions(-) delete mode 100644 doc/release-notes/11027-extend-datasets-files-from-search-api.md diff --git a/doc/release-notes/11027-extend-datasets-files-from-search-api.md b/doc/release-notes/11027-extend-datasets-files-from-search-api.md deleted file mode 100644 index 7b20daeeb0f..00000000000 --- a/doc/release-notes/11027-extend-datasets-files-from-search-api.md +++ /dev/null @@ -1,22 +0,0 @@ -### Feature to extend Search API for SPA - -Added new fields to search results type=files - -For Files: -- restricted: boolean -- canDownloadFile: boolean ( from file user permission) -- categories: array of string "categories" would be similar to what it is in metadata api. -For tabular files: -- tabularTags: array of string for example,{"tabularTags" : ["Event", "Genomics", "Geospatial"]} -- variables: number/int shows how many variables we have for the tabular file -- observations: number/int shows how many observations for the tabular file - - - -New fields added to solr schema.xml (Note: upgrade instructions will need to include instructions for schema.xml): - - - - - -See https://github.com/IQSS/dataverse/issues/11027 diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 469113c24fd..36ea5c5d434 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -100,8 +100,29 @@ Datasets without files can be optionally prevented from being published through ## API Updates +### Search API Returns Additional Fields for Files + +Added new fields to search results type=files + +For Files: + +- restricted: boolean +- canDownloadFile: boolean ( from file user permission) +- categories: array of string "categories" would be similar to what it is in metadata api. + +For tabular files: + +- tabularTags: array of string for example, `{"tabularTags" : ["Event", "Genomics", "Geospatial"]}` +- variables: number/int shows how many variables we have for the tabular file +- observations: number/int shows how many observations for the tabular file + +See #11027 and #11097. + +### Other API Updates + - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. + ## Settings Added ## Backward Incompatible Changes @@ -237,7 +258,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta 9\. Upgrade Solr -Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature, the default `schema.xml` files has changed so you must upgrade. +Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature and the addition of fields (e.g. fileRestricted, canDownloadFile, variableCount, and observations), the default `schema.xml` files has changed so you must upgrade. Install Solr 9.8.0 following the [instructions](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr) from the Installation Guide. From 769f1b3eea5681bbc8a6e73120c1c5d4d343b970 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:05:37 -0400 Subject: [PATCH 19/71] signposting --- doc/release-notes/10542-signposting.md | 11 ----------- doc/release-notes/6.6-release-notes.md | 13 +++++++++++++ 2 files changed, 13 insertions(+), 11 deletions(-) delete mode 100644 doc/release-notes/10542-signposting.md diff --git a/doc/release-notes/10542-signposting.md b/doc/release-notes/10542-signposting.md deleted file mode 100644 index f847e6ba94a..00000000000 --- a/doc/release-notes/10542-signposting.md +++ /dev/null @@ -1,11 +0,0 @@ -# Signposting Output Now Contains Links to All Dataset Metadata Export Formats - -When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. - -The output of HEAD, GET, and the Signposting "linkset" API have all been updated to include links to all available dataset metadata export formats (including any external exporters, such as Croissant, that have been enabled). - -This provides a lightweight machine-readable way to first retrieve a list of links (via a HTTP HEAD request, for example) to each available metadata export format and then follow up with a request for the export format of interest. - -In addition, the content type for the `schema.org` dataset metadata export format has been corrected. It was `application/json` and now it is `application/ld+json`. - -See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#retrieve-signposting-information) and #10542. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 36ea5c5d434..315d39d9f98 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -14,6 +14,7 @@ Highlights for Dataverse 6.6 include: - collections can be moved from the superuser dashboard - new 3D Objects metadata block - optionally prevent publishing of datasets without files +- Signposting output now contains links to all dataset metadata export formats - infrastructure updates (Payara and Solr) ## Features Added @@ -76,6 +77,18 @@ All new Dataverse installations will receive this metadata block by default. We Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. +# Signposting Output Now Contains Links to All Dataset Metadata Export Formats + +When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. + +The output of HEAD, GET, and the Signposting "linkset" API have all been updated to include links to all available dataset metadata export formats (including any external exporters, such as Croissant, that have been enabled). + +This provides a lightweight machine-readable way to first retrieve a list of links (via a HTTP HEAD request, for example) to each available metadata export format and then follow up with a request for the export format of interest. + +In addition, the content type for the `schema.org` dataset metadata export format has been corrected. It was `application/json` and now it is `application/ld+json`. + +See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#retrieve-signposting-information), #10542 and #11045. + ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. From 56bc4a45f4ab44640bf8c9c20c8440026587896d Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:10:25 -0400 Subject: [PATCH 20/71] collection featured items --- doc/release-notes/10943-featured-items.md | 15 --------------- doc/release-notes/6.6-release-notes.md | 18 +++++++++++++++++- 2 files changed, 17 insertions(+), 16 deletions(-) delete mode 100644 doc/release-notes/10943-featured-items.md diff --git a/doc/release-notes/10943-featured-items.md b/doc/release-notes/10943-featured-items.md deleted file mode 100644 index fa61d4e4875..00000000000 --- a/doc/release-notes/10943-featured-items.md +++ /dev/null @@ -1,15 +0,0 @@ -CRUD endpoints for Collection Featured Items have been implemented. In particular, the following endpoints have been implemented: - -- Create a feature item (POST /api/dataverses//featuredItems) -- Update a feature item (PUT /api/dataverseFeaturedItems/) -- Delete a feature item (DELETE /api/dataverseFeaturedItems/) -- List all featured items in a collection (GET /api/dataverses//featuredItems) -- Delete all featured items in a collection (DELETE /api/dataverses//featuredItems) -- Update all featured items in a collection (PUT /api/dataverses//featuredItems) - -New settings: - -- dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. -- dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. - -See also #10943 and #11124. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 315d39d9f98..d454981f1d7 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -131,13 +131,29 @@ For tabular files: See #11027 and #11097. +### Backend Support for Collection Featured Items + +CRUD endpoints for Collection Featured Items have been implemented. In particular, the following endpoints have been implemented: + +- Create a feature item (POST `/api/dataverses//featuredItems`) +- Update a feature item (PUT `/api/dataverseFeaturedItems/`) +- Delete a feature item (DELETE `/api/dataverseFeaturedItems/`) +- List all featured items in a collection (GET `/api/dataverses//featuredItems`) +- Delete all featured items in a collection (DELETE `/api/dataverses//featuredItems`) +- Update all featured items in a collection (PUT `/api/dataverses//featuredItems`) + +See also the "Settings Added" section, #10943 and #11124. + ### Other API Updates - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. - ## Settings Added +- dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. +- dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. + + ## Backward Incompatible Changes Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. From 7fee2f872a72a9b58f69b46dbd12d05c6403d2c5 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:17:02 -0400 Subject: [PATCH 21/71] dataset types linked to metadata blocks --- doc/release-notes/10519-dataset-types.md | 12 ------------ doc/release-notes/6.6-release-notes.md | 13 +++++++++++++ 2 files changed, 13 insertions(+), 12 deletions(-) delete mode 100644 doc/release-notes/10519-dataset-types.md diff --git a/doc/release-notes/10519-dataset-types.md b/doc/release-notes/10519-dataset-types.md deleted file mode 100644 index 99cf79a796f..00000000000 --- a/doc/release-notes/10519-dataset-types.md +++ /dev/null @@ -1,12 +0,0 @@ -## Dataset Types can be linked to Metadata Blocks - -Metadata blocks (e.g. "CodeMeta") can now be linked to dataset types (e.g. "software") using new superuser APIs. - -This will have the following effects for the APIs used by the new Dataverse UI ( https://github.com/IQSS/dataverse-frontend ): - -- The list of fields shown when creating a dataset will include fields marked as "displayoncreate" (in the tsv/database) for metadata blocks (e.g. "CodeMeta") that are linked to the dataset type (e.g. "software") that is passed to the API. -- The metadata blocks shown when editing a dataset will include metadata blocks (e.g. "CodeMeta") that are linked to the dataset type (e.g. "software") that is passed to the API. - -Mostly in order to write automated tests for the above, a [displayOnCreate](https://dataverse-guide--11001.org.readthedocs.build/en/11001/api/native-api.html#set-displayoncreate-for-a-dataset-field) API endpoint has been added. - -For more information, see the guides ([overview](https://dataverse-guide--11001.org.readthedocs.build/en/11001/user/dataset-management.html#dataset-types), [new APIs](https://dataverse-guide--11001.org.readthedocs.build/en/11001/api/native-api.html#link-dataset-type-with-metadata-blocks)), #10519 and #11001. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index d454981f1d7..13d5eaa3f2b 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -89,6 +89,19 @@ In addition, the content type for the `schema.org` dataset metadata export forma See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#retrieve-signposting-information), #10542 and #11045. +### Dataset Types Can Be Linked to Metadata Blocks + +Metadata blocks (e.g. "CodeMeta") can now be linked to dataset types (e.g. "software") using new superuser APIs. + +This will have the following effects for the APIs used by the [new Dataverse UI](https://github.com/IQSS/dataverse-frontend): + +- The list of fields shown when creating a dataset will include fields marked as "displayoncreate" (in the tsv/database) for metadata blocks (e.g. "CodeMeta") that are linked to the dataset type (e.g. "software") that is passed to the API. +- The metadata blocks shown when editing a dataset will include metadata blocks (e.g. "CodeMeta") that are linked to the dataset type (e.g. "software") that is passed to the API. + +Mostly in order to write automated tests for the above, a [displayOnCreate](https://guides.dataverse.org/en/6.6/api/native-api.html#set-displayoncreate-for-a-dataset-field) API endpoint has been added. + +For more information, see the guides ([overview](https://guides.dataverse.org/en/6.6/user/dataset-management.html#dataset-types), [new APIs](https://guides.dataverse.org/en/6.6/api/native-api.html#link-dataset-type-with-metadata-blocks)), #10519 and #11001. + ### Other Features - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. From e8627a654dcc0aefa98ba717981f27ec4d03bd39 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:17:47 -0400 Subject: [PATCH 22/71] fix heading levels --- doc/release-notes/6.6-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 13d5eaa3f2b..3bc30bd6b5a 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -73,11 +73,11 @@ A new metadata block has been added for describing 3D object data. You can downl All new Dataverse installations will receive this metadata block by default. We recommend adding it by following the upgrade instructions below. -## Prevent Publishing of Datasets Without Files +### Prevent Publishing of Datasets Without Files Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. -# Signposting Output Now Contains Links to All Dataset Metadata Export Formats +### Signposting Output Now Contains Links to All Dataset Metadata Export Formats When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. From f253c7be5f0c11143865750024937048cd6d7700 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:46:22 -0400 Subject: [PATCH 23/71] harvesting --- .../11038-unconsidered-harvesting-granularity.md | 2 -- doc/release-notes/6.6-release-notes.md | 2 ++ ...mprovement-and-internationalization-of-harvest-status.md | 6 ------ 3 files changed, 2 insertions(+), 8 deletions(-) delete mode 100644 doc/release-notes/11038-unconsidered-harvesting-granularity.md delete mode 100644 doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md diff --git a/doc/release-notes/11038-unconsidered-harvesting-granularity.md b/doc/release-notes/11038-unconsidered-harvesting-granularity.md deleted file mode 100644 index 72ebd522831..00000000000 --- a/doc/release-notes/11038-unconsidered-harvesting-granularity.md +++ /dev/null @@ -1,2 +0,0 @@ -Bug Fix: -OAI Client harvesting now uses the correct granularity while re-run a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 3bc30bd6b5a..e8221c4193e 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -107,6 +107,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. - [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. +- A new harvest status differentiates between a complete harvest with errors (completed with failures) and without errors (completed). Also, harvest status labels are now internationalized. See #9294 and #11017. - The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. @@ -119,6 +120,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. +- Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. diff --git a/doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md b/doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md deleted file mode 100644 index f9fc465292c..00000000000 --- a/doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md +++ /dev/null @@ -1,6 +0,0 @@ -## Improvement and internationalization of harvest status - -Added a harvest status to differentiate a complete harvest with errors (Completed with failures) and without errors (Completed) -Harvest status labels are now internationalized - -For more information, see issue [#9294](https://github.com/IQSS/dataverse/issues/9294) \ No newline at end of file From 40368d620323aba7a315f80e722afd8f52e0ebe8 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 10:56:40 -0400 Subject: [PATCH 24/71] locks and permalink --- doc/release-notes/10516_legacy_permalink_config_fix.md | 1 - .../11264-slower refresh for long running locks.md | 1 - doc/release-notes/6.6-release-notes.md | 2 ++ 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 doc/release-notes/10516_legacy_permalink_config_fix.md delete mode 100644 doc/release-notes/11264-slower refresh for long running locks.md diff --git a/doc/release-notes/10516_legacy_permalink_config_fix.md b/doc/release-notes/10516_legacy_permalink_config_fix.md deleted file mode 100644 index d78395252d4..00000000000 --- a/doc/release-notes/10516_legacy_permalink_config_fix.md +++ /dev/null @@ -1 +0,0 @@ -Support for legacy configuration of a PermaLink PID provider, e.g. using the :Protocol,:Authority, and :Shoulder settings, is fixed. \ No newline at end of file diff --git a/doc/release-notes/11264-slower refresh for long running locks.md b/doc/release-notes/11264-slower refresh for long running locks.md deleted file mode 100644 index 96e0cec0ccb..00000000000 --- a/doc/release-notes/11264-slower refresh for long running locks.md +++ /dev/null @@ -1 +0,0 @@ -When a dataset has a long running lock, including when it is 'in review', Dataverse will now slow the page refresh rate over time. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index e8221c4193e..dabe344997e 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -110,6 +110,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - A new harvest status differentiates between a complete harvest with errors (completed with failures) and without errors (completed). Also, harvest status labels are now internationalized. See #9294 and #11017. - The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. +- Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. @@ -121,6 +122,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. - Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. +- When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. From a43bcb477ac8e195eb3182f32b60688fc8e954eb Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 11:28:15 -0400 Subject: [PATCH 25/71] 3 more --- .../ 11065-extend-search-api-to-include-type-counts.md | 1 - doc/release-notes/10809-oai-ore-nested-compound.md | 1 - doc/release-notes/11136-solr-nested.md | 1 - doc/release-notes/6.6-release-notes.md | 3 +++ 4 files changed, 3 insertions(+), 3 deletions(-) delete mode 100644 doc/release-notes/ 11065-extend-search-api-to-include-type-counts.md delete mode 100644 doc/release-notes/10809-oai-ore-nested-compound.md delete mode 100644 doc/release-notes/11136-solr-nested.md diff --git a/doc/release-notes/ 11065-extend-search-api-to-include-type-counts.md b/doc/release-notes/ 11065-extend-search-api-to-include-type-counts.md deleted file mode 100644 index 0ba188c8637..00000000000 --- a/doc/release-notes/ 11065-extend-search-api-to-include-type-counts.md +++ /dev/null @@ -1 +0,0 @@ -The JSON payload of the search endpoint has been extended to include total_count_per_object_type for types: dataverse, dataset, and files when the search parameter "&show_type_counts=true" is passed in. diff --git a/doc/release-notes/10809-oai-ore-nested-compound.md b/doc/release-notes/10809-oai-ore-nested-compound.md deleted file mode 100644 index d67f2b65592..00000000000 --- a/doc/release-notes/10809-oai-ore-nested-compound.md +++ /dev/null @@ -1 +0,0 @@ -The OAI-ORE exporter can now export metadata containing nested compound fields (i.e. compound fields within compound fields). See #10809 and #11190. diff --git a/doc/release-notes/11136-solr-nested.md b/doc/release-notes/11136-solr-nested.md deleted file mode 100644 index f85c02cde20..00000000000 --- a/doc/release-notes/11136-solr-nested.md +++ /dev/null @@ -1 +0,0 @@ -Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index dabe344997e..d7ccee3dadc 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -108,9 +108,11 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. - [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. - A new harvest status differentiates between a complete harvest with errors (completed with failures) and without errors (completed). Also, harvest status labels are now internationalized. See #9294 and #11017. +- The OAI-ORE exporter can now export metadata containing nested compound fields (i.e. compound fields within compound fields). See #10809 and #11190. - The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. +- Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. @@ -164,6 +166,7 @@ See also the "Settings Added" section, #10943 and #11124. ### Other API Updates - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. +- The JSON payload of the search endpoint has been extended to include `total_count_per_object_type` for types: dataverse, dataset, and files when the search parameter `show_type_counts=true` is passed in. See #11065 and #11082. ## Settings Added From dfc3a74cf463dad36ba70140d13b1d93ba5f13fb Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 12:18:14 -0400 Subject: [PATCH 26/71] remind those with text customizations to update properties files See https://dataverse.zulipchat.com/#narrow/channel/378866-troubleshooting/topic/upgrade.20from.206.2E3.20to.206.2E4/near/504689559 --- doc/release-notes/6.6-release-notes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index d7ccee3dadc..79464781b7e 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -274,10 +274,12 @@ sudo rm -rf $PAYARA/glassfish/domains/domain1/lib/databases sudo service payara start ``` -6\. For installations with internationalization: +6\. For installations with internationalization or text customizations: Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). +If you have text customizations you can get the latest English files from . + 7\. Restart Payara ```shell From 7fd9268a639a5f4f9185bcdcadca19f353cfb851 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 14:07:12 -0400 Subject: [PATCH 27/71] docker updates --- doc/release-notes/10541-root-alias-name2.md | 1 - doc/release-notes/11107-fake-to-perma-demo.md | 3 --- doc/release-notes/6.6-release-notes.md | 1 + 3 files changed, 1 insertion(+), 4 deletions(-) delete mode 100644 doc/release-notes/10541-root-alias-name2.md delete mode 100644 doc/release-notes/11107-fake-to-perma-demo.md diff --git a/doc/release-notes/10541-root-alias-name2.md b/doc/release-notes/10541-root-alias-name2.md deleted file mode 100644 index e699d40994b..00000000000 --- a/doc/release-notes/10541-root-alias-name2.md +++ /dev/null @@ -1 +0,0 @@ -The [tutorial](https://dataverse-guide--11201.org.readthedocs.build/en/11201/container/running/demo.html#root-collection-customization-alias-name-etc) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file. See also #10541 and #11201. diff --git a/doc/release-notes/11107-fake-to-perma-demo.md b/doc/release-notes/11107-fake-to-perma-demo.md deleted file mode 100644 index afb6b8a7917..00000000000 --- a/doc/release-notes/11107-fake-to-perma-demo.md +++ /dev/null @@ -1,3 +0,0 @@ -### Demo/Eval Container Tutorial - -The demo/eval container tutorial has been updated to use the Permalink PID provider instead of the FAKE DOI Provider. See also #11107. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 79464781b7e..9e4a8182991 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -113,6 +113,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. - Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. +- The [tutorial](https://guides.dataverse.org/en/6.6/container/running/demo.html) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file (#10541 and #11201) and now uses the Permalink PID provider instead of the FAKE DOI Provider (#11107 and #11108). - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. From 61d4f8647ef2f3533f58775c96205c9f17362a13 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 14:21:36 -0400 Subject: [PATCH 28/71] 3 more --- doc/release-notes/10975-fix-file-replace-via-api | 3 --- doc/release-notes/11228-summaryfields.md | 1 - doc/release-notes/6.6-release-notes.md | 7 ++++++- doc/release-notes/master_json_fix.md | 1 - 4 files changed, 6 insertions(+), 6 deletions(-) delete mode 100644 doc/release-notes/10975-fix-file-replace-via-api delete mode 100644 doc/release-notes/11228-summaryfields.md delete mode 100644 doc/release-notes/master_json_fix.md diff --git a/doc/release-notes/10975-fix-file-replace-via-api b/doc/release-notes/10975-fix-file-replace-via-api deleted file mode 100644 index 71646015a32..00000000000 --- a/doc/release-notes/10975-fix-file-replace-via-api +++ /dev/null @@ -1,3 +0,0 @@ -A bug that caused replacing files via API when file PIDs were enabled has been fixed. - - For testing purposes, the FAKE PID provider can now be used with file PIDs enabled. (The FAKE provider is not recommended for any production use.) \ No newline at end of file diff --git a/doc/release-notes/11228-summaryfields.md b/doc/release-notes/11228-summaryfields.md deleted file mode 100644 index 63a55637934..00000000000 --- a/doc/release-notes/11228-summaryfields.md +++ /dev/null @@ -1 +0,0 @@ -The :CustomDatasetSummaryFields setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 9e4a8182991..d7651e09581 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -116,6 +116,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - The [tutorial](https://guides.dataverse.org/en/6.6/container/running/demo.html) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file (#10541 and #11201) and now uses the Permalink PID provider instead of the FAKE DOI Provider (#11107 and #11108). - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. - Solr has been upgraded to version 9.8.0. See #10713. +- For testing purposes, the FAKE PID provider can now be used with [file PIDs enabled](https://guides.dataverse.org/en/6.6/installation/config.html#filepidsenabled). (The FAKE provider is not recommended for any production use.) See #10979. ## Bugs Fixed @@ -126,6 +127,8 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. - Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. +- A bug that caused replacing files via API when file PIDs were enabled has been fixed. See #10975 and #10979. +- The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. @@ -168,17 +171,19 @@ See also the "Settings Added" section, #10943 and #11124. - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. - The JSON payload of the search endpoint has been extended to include `total_count_per_object_type` for types: dataverse, dataset, and files when the search parameter `show_type_counts=true` is passed in. See #11065 and #11082. +- /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. ## Settings Added - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. - dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. - ## Backward Incompatible Changes Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. +- /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See #10764. + ## Complete List of Changes For the complete list of code changes in this release, see the [6.6 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.6+is%3Aclosed) in GitHub. diff --git a/doc/release-notes/master_json_fix.md b/doc/release-notes/master_json_fix.md deleted file mode 100644 index aa30b90c2cb..00000000000 --- a/doc/release-notes/master_json_fix.md +++ /dev/null @@ -1 +0,0 @@ -This pull request fixes an issue in the JsonPrinter class so that there are no duplicated entries in the JSON metadata or ommitted metadata properties. After the fix is applied the /api/metadatablocks/ endpoint should return correct JSON. \ No newline at end of file From 076810a8093ab3000f7c6965d5f5b481b337d414 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 14:22:49 -0400 Subject: [PATCH 29/71] typo --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index d7651e09581..f04cf7999d9 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -127,7 +127,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. - Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. -- A bug that caused replacing files via API when file PIDs were enabled has been fixed. See #10975 and #10979. +- A bug that caused replacing files via API when file PIDs were enabled to fail has been fixed. See #10975 and #10979. - The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. From 425c53fdcb2c7e54146d45460706b824f208c731 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 14:34:26 -0400 Subject: [PATCH 30/71] API updates --- .../11127-update-search-api-to-show-dvobject-type-counts.md | 5 ----- doc/release-notes/11188-dataset-version-api.md | 4 ---- .../11255-get-dataverse-api-return-child-count.md | 1 - doc/release-notes/6.6-release-notes.md | 4 +++- 4 files changed, 3 insertions(+), 11 deletions(-) delete mode 100644 doc/release-notes/11127-update-search-api-to-show-dvobject-type-counts.md delete mode 100644 doc/release-notes/11188-dataset-version-api.md delete mode 100644 doc/release-notes/11255-get-dataverse-api-return-child-count.md diff --git a/doc/release-notes/11127-update-search-api-to-show-dvobject-type-counts.md b/doc/release-notes/11127-update-search-api-to-show-dvobject-type-counts.md deleted file mode 100644 index c9b63340a1f..00000000000 --- a/doc/release-notes/11127-update-search-api-to-show-dvobject-type-counts.md +++ /dev/null @@ -1,5 +0,0 @@ -### show-type-counts Behavior changed in Search API - -In the Search API if you set show_type_counts=true the response will include all object types (Dataverses, Datasets, and Files) even if the search result for any given type is 0. - -See also the [guides](https://preview.guides.gdcc.io/en/develop/api/search.html#parameters), #11127 and #11138. diff --git a/doc/release-notes/11188-dataset-version-api.md b/doc/release-notes/11188-dataset-version-api.md deleted file mode 100644 index 2367286d05b..00000000000 --- a/doc/release-notes/11188-dataset-version-api.md +++ /dev/null @@ -1,4 +0,0 @@ -### Retreiving a list of Dataset Versions - -An API has been added to get dataset versions including a summary of diiferences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), and #10888. - diff --git a/doc/release-notes/11255-get-dataverse-api-return-child-count.md b/doc/release-notes/11255-get-dataverse-api-return-child-count.md deleted file mode 100644 index f91cc9ea9f9..00000000000 --- a/doc/release-notes/11255-get-dataverse-api-return-child-count.md +++ /dev/null @@ -1 +0,0 @@ -Added a new query param, ``returnChildCount``, to getDataverse endpoint(``/api/dataverses/``) for optionally retrieving the child count, which represents the number of dataverses, datasets, or files within the dataverse (direct children only). See also #11255 and #11259. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index f04cf7999d9..7cd84d2935a 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -170,8 +170,10 @@ See also the "Settings Added" section, #10943 and #11124. ### Other API Updates - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. -- The JSON payload of the search endpoint has been extended to include `total_count_per_object_type` for types: dataverse, dataset, and files when the search parameter `show_type_counts=true` is passed in. See #11065 and #11082. +- An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. +- The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. +- A new query param, `returnChildCount`, has been added to the getDataverse endpoint (`/api/dataverses/{id}`) for optionally retrieving the child count, which represents the number of collections, datasets, or files within the collection (direct children only). See also #11255 and #11259. ## Settings Added From 4cbf6530b1fe99b014c90d4efc6b52defff7d8b8 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 14:55:20 -0400 Subject: [PATCH 31/71] guides --- doc/release-notes/10320-cookie-consent.md | 3 --- doc/release-notes/6.6-release-notes.md | 2 ++ doc/release-notes/guides-toc-depth.md | 3 --- 3 files changed, 2 insertions(+), 6 deletions(-) delete mode 100644 doc/release-notes/10320-cookie-consent.md delete mode 100644 doc/release-notes/guides-toc-depth.md diff --git a/doc/release-notes/10320-cookie-consent.md b/doc/release-notes/10320-cookie-consent.md deleted file mode 100644 index a3fd7f5466b..00000000000 --- a/doc/release-notes/10320-cookie-consent.md +++ /dev/null @@ -1,3 +0,0 @@ -## Cookie Consent Popup (GDPR) - -For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://dataverse-guide--10320.org.readthedocs.build/en/10320/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 7cd84d2935a..dc9dbfcd78d 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -112,6 +112,8 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. +- On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. +- For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://guides.dataverse.org/en/6.6/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320. - Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. - The [tutorial](https://guides.dataverse.org/en/6.6/container/running/demo.html) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file (#10541 and #11201) and now uses the Permalink PID provider instead of the FAKE DOI Provider (#11107 and #11108). - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. diff --git a/doc/release-notes/guides-toc-depth.md b/doc/release-notes/guides-toc-depth.md deleted file mode 100644 index 0f5dd39186f..00000000000 --- a/doc/release-notes/guides-toc-depth.md +++ /dev/null @@ -1,3 +0,0 @@ -### Guides Table of Contents Depth - -On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. See #11166. From 6d4717e5909728ab2ac80a4b85190ecd752f3f01 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 15:45:33 -0400 Subject: [PATCH 32/71] 6 more --- .../10108-stata-version-detected-in-direct-upload.md | 1 - doc/release-notes/10384-link.md | 3 --- ...0714-access-requests-missing-since-upgrade-v6-0.md | 6 ------ doc/release-notes/11113-avoid-orphan-perm-docs.md | 5 ----- .../11272-fix-order-on-advancedsearchpage.md | 5 ----- doc/release-notes/6.6-release-notes.md | 7 +++++++ doc/release-notes/8808-10575-update-global-role.md | 11 ----------- 7 files changed, 7 insertions(+), 31 deletions(-) delete mode 100644 doc/release-notes/10108-stata-version-detected-in-direct-upload.md delete mode 100644 doc/release-notes/10384-link.md delete mode 100644 doc/release-notes/10714-access-requests-missing-since-upgrade-v6-0.md delete mode 100644 doc/release-notes/11113-avoid-orphan-perm-docs.md delete mode 100644 doc/release-notes/11272-fix-order-on-advancedsearchpage.md delete mode 100644 doc/release-notes/8808-10575-update-global-role.md diff --git a/doc/release-notes/10108-stata-version-detected-in-direct-upload.md b/doc/release-notes/10108-stata-version-detected-in-direct-upload.md deleted file mode 100644 index a69680e1c48..00000000000 --- a/doc/release-notes/10108-stata-version-detected-in-direct-upload.md +++ /dev/null @@ -1 +0,0 @@ -The version of Stata files is now detected during S3 direct upload (as it was for normal uploads), allowing ingest of Stata 14 and 15 files that have been uploaded directly. See [the guides](https://dataverse-guide--11054.org.readthedocs.build/en/11054/developers/big-data-support.html#features-that-are-disabled-if-s3-direct-upload-is-enabled), #10108, and #11054. diff --git a/doc/release-notes/10384-link.md b/doc/release-notes/10384-link.md deleted file mode 100644 index 7092241adf4..00000000000 --- a/doc/release-notes/10384-link.md +++ /dev/null @@ -1,3 +0,0 @@ -### Broken Link in Email When Users Request Access to Files - -When users request access to a files, the people who have permission to grant access receive an email with a link in it that didn't work due to a trailing period (full stop) right next to the link (e.g. `https://demo.dataverse.org/permissions-manage-files.xhtml?id=9.`) A space has been added to fix this. See #10384 and #11115. diff --git a/doc/release-notes/10714-access-requests-missing-since-upgrade-v6-0.md b/doc/release-notes/10714-access-requests-missing-since-upgrade-v6-0.md deleted file mode 100644 index a220c150791..00000000000 --- a/doc/release-notes/10714-access-requests-missing-since-upgrade-v6-0.md +++ /dev/null @@ -1,6 +0,0 @@ -### Flyway Script added to Fix File Access Requests when upgrading from Dataverse 6.0 - -Database update script added to prevent duplicate keys when upgrading from V6.0 -This script will delete access requests made after the initial request and will set the initial request to "Created" - -See: https://github.com/IQSS/dataverse/issues/10714 diff --git a/doc/release-notes/11113-avoid-orphan-perm-docs.md b/doc/release-notes/11113-avoid-orphan-perm-docs.md deleted file mode 100644 index 93c030d0afe..00000000000 --- a/doc/release-notes/11113-avoid-orphan-perm-docs.md +++ /dev/null @@ -1,5 +0,0 @@ -This release fixes a bug that caused Dataverse to generate unnecessary solr documents for files when a file is added/deleted from a draft dataset. These documents could accumulate and potentially impact performance. - -Assuming the upgrade to solr 9.8.0 also occurs in this release, there's nothing else needed for this PR. (Starting with a new solr insures the solr db is empty and that a reindex is already required.) - - diff --git a/doc/release-notes/11272-fix-order-on-advancedsearchpage.md b/doc/release-notes/11272-fix-order-on-advancedsearchpage.md deleted file mode 100644 index 9db098be3c4..00000000000 --- a/doc/release-notes/11272-fix-order-on-advancedsearchpage.md +++ /dev/null @@ -1,5 +0,0 @@ -### Fix order of metadata fields on the advanced search page - -The metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value. -The order of the fields on the AdvancedSearch page will then be more similar to that on the metadata forms. -Note that fields that are not defined in the TSV file, like the 'Persistent ID' and 'Publication Date', will be displayed at the end. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index dc9dbfcd78d..a78db64c320 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -109,7 +109,9 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. - A new harvest status differentiates between a complete harvest with errors (completed with failures) and without errors (completed). Also, harvest status labels are now internationalized. See #9294 and #11017. - The OAI-ORE exporter can now export metadata containing nested compound fields (i.e. compound fields within compound fields). See #10809 and #11190. +- It is now possible to edit a custom role with the same alias. See #8808 and #10612. - The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. +- The version of Stata files is now detected during S3 direct upload (as it was for normal uploads), allowing ingest of Stata 14 and 15 files that have been uploaded directly. See [the guides](https://guides.dataverse.org/en/6.6/developers/big-data-support.html#features-that-are-disabled-if-s3-direct-upload-is-enabled) #10108, and #11054. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. - On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. @@ -123,11 +125,14 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en ## Bugs Fixed - A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. +- On the Advanced Search page, the metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value, making the order the same as when you view or edit metadata. Note that fields that are not defined in the TSV file, like the "Persistent ID" and "Publication Date", will be displayed at the end. See #11272 and #11279. - The file page version table now shows whether a file has been replaced. See #11142 and #11145. - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. +- When users request access to files, the people who have permission to grant access received an email with a link in it that didn't work due to a trailing period (full stop) right next to the link (e.g. `https://demo.dataverse.org/permissions-manage-files.xhtml?id=9.`) A space has been added to fix this. See #10384 and #11115. - Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. +- Access requests were missing on the File Permission page after upgrading from Dataverse 6.0. This has been corrected with a database update script. See #10714 and #11061. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. - A bug that caused replacing files via API when file PIDs were enabled to fail has been fixed. See #10975 and #10979. - The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. @@ -135,6 +140,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. - In the Search API, files were displaying DRAFT version instead of latest released version under `dataset_citation`. See #10735 and #11051. +- Unnecessary Solr documents were being created when a file was added or deleted from a draft dataset. These documents could accumulate and potentially impact performance. There is no action to take because this release includes a new Solr version, which will start with an empty database. See #11113 and #11114. ## API Updates @@ -174,6 +180,7 @@ See also the "Settings Added" section, #10943 and #11124. - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. - An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. - The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). +- A new API endpoint has been added that allows a global role to be updated. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#update-global-role) and #10612. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. - A new query param, `returnChildCount`, has been added to the getDataverse endpoint (`/api/dataverses/{id}`) for optionally retrieving the child count, which represents the number of collections, datasets, or files within the collection (direct children only). See also #11255 and #11259. diff --git a/doc/release-notes/8808-10575-update-global-role.md b/doc/release-notes/8808-10575-update-global-role.md deleted file mode 100644 index 38142f972e8..00000000000 --- a/doc/release-notes/8808-10575-update-global-role.md +++ /dev/null @@ -1,11 +0,0 @@ -## Release Highlights - -### Update a Global Role - -A new API endpoint has been added that allows a global role to be updated. See [Native API Guide > Update Global Role](https://guides.dataverse.org/en/6.3/api/native-api.html#update-global-role) (#10612) - -## Bug fixes - -### Edition of custom role fixed - -It is now possible to edit a custom role with the same alias (reported in #8808) \ No newline at end of file From 12fb9d69771cab8c6c58e151ed35fff285e99f18 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 15:50:17 -0400 Subject: [PATCH 33/71] archival block, etalab --- doc/release-notes/10626-archival-tsv.md | 2 -- doc/release-notes/6.6-release-notes.md | 6 ++++++ doc/release-notes/9302-add-Etalab-License.md | 5 ----- 3 files changed, 6 insertions(+), 7 deletions(-) delete mode 100644 doc/release-notes/10626-archival-tsv.md delete mode 100644 doc/release-notes/9302-add-Etalab-License.md diff --git a/doc/release-notes/10626-archival-tsv.md b/doc/release-notes/10626-archival-tsv.md deleted file mode 100644 index 2cd7c4e60b1..00000000000 --- a/doc/release-notes/10626-archival-tsv.md +++ /dev/null @@ -1,2 +0,0 @@ -Release Highlights: -An experimental "Archival" metadata block has been added, [downloadable](https://dataverse-guide--10626.org.readthedocs.build/en/10626/user/appendix.html) from the User Guide. The purpose of the metadata block is to enable repositories to register metadata relating to the potential archiving of the dataset at a depositor archive, whether that being your own institutional archive or an external archive, i.e. a historical archive. See also #10626. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index a78db64c320..46f36bf64fe 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -13,6 +13,7 @@ Highlights for Dataverse 6.6 include: - more accurate search highlighting - collections can be moved from the superuser dashboard - new 3D Objects metadata block +- new Archival metadata block (experimental) - optionally prevent publishing of datasets without files - Signposting output now contains links to all dataset metadata export formats - infrastructure updates (Payara and Solr) @@ -73,6 +74,10 @@ A new metadata block has been added for describing 3D object data. You can downl All new Dataverse installations will receive this metadata block by default. We recommend adding it by following the upgrade instructions below. +### New Archival Metadata Block (Experimental) + +An experimental "Archival" metadata block has been added, [downloadable](https://guides.dataverse.org/en/6.6/user/appendix.html) from the User Guide. The purpose of the metadata block is to enable repositories to register metadata relating to the potential archiving of the dataset at a depositor archive, whether that being your own institutional archive or an external archive, i.e. a historical archive. See also #10626. + ### Prevent Publishing of Datasets Without Files Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. @@ -116,6 +121,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. - On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. - For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://guides.dataverse.org/en/6.6/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320. +- A new file has been added to import the French Open License to Dataverse: licenseEtalab-2.0.json. You can download it from [the guides](http://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses). This license, which is compatible with the Creative Commons license, is recommended by the French government for open documents. See #9301, #9302, and #11302. - Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. - The [tutorial](https://guides.dataverse.org/en/6.6/container/running/demo.html) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file (#10541 and #11201) and now uses the Permalink PID provider instead of the FAKE DOI Provider (#11107 and #11108). - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. diff --git a/doc/release-notes/9302-add-Etalab-License.md b/doc/release-notes/9302-add-Etalab-License.md deleted file mode 100644 index 8009f3c1b53..00000000000 --- a/doc/release-notes/9302-add-Etalab-License.md +++ /dev/null @@ -1,5 +0,0 @@ -A new file has been added to import the French Open License to Dataverse: licenseEtalab-2.0.json. You can download it from [the guides](https://dataverse-guide--9302.org.readthedocs.build/en/9302/installation/config.html#adding-licenses). - -This license, which is compatible with the Creative Commons license, is recommended by the French government for open documents. - -For more information, see #9301, #9302, and #11302. From aa86f7094365a43c9571d656acb530e1c9dcd0a5 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 16:31:52 -0400 Subject: [PATCH 34/71] reharvest, guestbook bugs --- doc/release-notes/11203-request-access-fixes.md | 2 -- doc/release-notes/6.6-release-notes.md | 8 ++++++++ doc/release-notes/8739-publisher-during-harvesting.md | 1 - 3 files changed, 8 insertions(+), 3 deletions(-) delete mode 100644 doc/release-notes/11203-request-access-fixes.md delete mode 100644 doc/release-notes/8739-publisher-during-harvesting.md diff --git a/doc/release-notes/11203-request-access-fixes.md b/doc/release-notes/11203-request-access-fixes.md deleted file mode 100644 index 7087588c916..00000000000 --- a/doc/release-notes/11203-request-access-fixes.md +++ /dev/null @@ -1,2 +0,0 @@ -Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. -Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 46f36bf64fe..8810e0c31f8 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -112,6 +112,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. - The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. - [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. +- The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. This improves the citation associated with these datasets, but the change only affects newly harvested datasets. See "Upgrade Instructions" below on how to re-harvest. For more information, see [the guides](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. - A new harvest status differentiates between a complete harvest with errors (completed with failures) and without errors (completed). Also, harvest status labels are now internationalized. See #9294 and #11017. - The OAI-ORE exporter can now export metadata containing nested compound fields (i.e. compound fields within compound fields). See #10809 and #11190. - It is now possible to edit a custom role with the same alias. See #8808 and #10612. @@ -132,6 +133,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. - On the Advanced Search page, the metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value, making the order the same as when you view or edit metadata. Note that fields that are not defined in the TSV file, like the "Persistent ID" and "Publication Date", will be displayed at the end. See #11272 and #11279. +- Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. See #11203. - The file page version table now shows whether a file has been replaced. See #11142 and #11145. - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. @@ -378,3 +380,9 @@ For this reason, we recommend reexporting all dataset metadata. For more advance ```shell curl http://localhost:8080/api/admin/metadata/reExportAll ``` + +12\. Re-harvest datasets + +The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. For more information, see [the guides](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. + +This improves the citation associated with these datasets, but the change only affects newly harvested datasets. All datasets should be re-harvested if you wish to pick up this change on already harvested datasets. diff --git a/doc/release-notes/8739-publisher-during-harvesting.md b/doc/release-notes/8739-publisher-during-harvesting.md deleted file mode 100644 index 3e1555396c0..00000000000 --- a/doc/release-notes/8739-publisher-during-harvesting.md +++ /dev/null @@ -1 +0,0 @@ -The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. This improves the citation associated with these datasets, but the change only affects newly harvested datasets. All datasets should be re-harvested if you wish to pick up this change on already harvested datasets. For more information, see [the guides](https://dataverse-guide--9013.org.readthedocs.build/en/9013/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. \ No newline at end of file From 3fb1c67248f882e87e236fec796c83cd40ccbde3 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 16:41:46 -0400 Subject: [PATCH 35/71] metrics and sorting fixes --- doc/release-notes/11178-bug-fix-sort-by-newest-first.md | 8 -------- doc/release-notes/6.6-release-notes.md | 2 ++ doc/release-notes/Metrics-FixFileAPIs.md | 2 -- 3 files changed, 2 insertions(+), 10 deletions(-) delete mode 100644 doc/release-notes/11178-bug-fix-sort-by-newest-first.md delete mode 100644 doc/release-notes/Metrics-FixFileAPIs.md diff --git a/doc/release-notes/11178-bug-fix-sort-by-newest-first.md b/doc/release-notes/11178-bug-fix-sort-by-newest-first.md deleted file mode 100644 index 8fb3f65b2f1..00000000000 --- a/doc/release-notes/11178-bug-fix-sort-by-newest-first.md +++ /dev/null @@ -1,8 +0,0 @@ -### Bug fix: Sorting by "newest first" - -Fixed an issue where draft versions of datasets were sorted using the release timestamp of their most recent major version. -This caused newer drafts to appear incorrectly alongside their corresponding major version, instead of at the top, when sorted by "newest first". -Sorting now uses the last update timestamp when sorting draft datasets. -The sorting behavior of published dataset versions (major and minor) is unchanged. - -**Upgrade instructions**: draft datasets must be reindexed for this fix to take effect. \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 8810e0c31f8..ff4c61f5413 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -135,6 +135,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - On the Advanced Search page, the metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value, making the order the same as when you view or edit metadata. Note that fields that are not defined in the TSV file, like the "Persistent ID" and "Publication Date", will be displayed at the end. See #11272 and #11279. - Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. See #11203. - The file page version table now shows whether a file has been replaced. See #11142 and #11145. +- We fixed an issue where draft versions of datasets were sorted using the release timestamp of their most recent major version. This caused newer drafts to appear incorrectly alongside their corresponding major version, instead of at the top, when sorted by "newest first". Sorting now uses the last update timestamp when sorting draft datasets. The sorting behavior of published dataset versions (major and minor) is unchanged. There is no need to reindex datasets because Solr is being upgraded (see "Upgrade Instructions"), which will result in an empty database that will be reindexed. See #11178. - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. @@ -142,6 +143,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. - Access requests were missing on the File Permission page after upgrading from Dataverse 6.0. This has been corrected with a database update script. See #10714 and #11061. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. +- The `/api/info/metrics/files/monthly` API call had a bug that resulted in files being counted each time they were published in a new version if those publication events occurred in different months. This resulted in an over-count. The `/api/info/metrics/files` and `/api/info/metrics/files/toMonth` API calls had a bug that resulted in files that were published but no longer in the latest published version as of the specified date (now, or the date entered in the `/toMonth` variant). This resulted in an under-count. See 11189. - A bug that caused replacing files via API when file PIDs were enabled to fail has been fixed. See #10975 and #10979. - The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. diff --git a/doc/release-notes/Metrics-FixFileAPIs.md b/doc/release-notes/Metrics-FixFileAPIs.md deleted file mode 100644 index 871f606f2f6..00000000000 --- a/doc/release-notes/Metrics-FixFileAPIs.md +++ /dev/null @@ -1,2 +0,0 @@ -The /api/info/metrics/files/monthly API call had a bug that resulted in files being counted each time they were published in a new version if those publication events occurred in different months. This resulted in an over-count. -The /api/info/metrics/files and /api/info/metrics/files/toMonth API calls had a bug that resulted in files that were published but no longer in the latest published version as of the specified date (now, or the date entered in the /toMonth variant). This resulted in an under-count. From 81d75318f27232503e29dbf930bbaef87ca29a6c Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 17:02:39 -0400 Subject: [PATCH 36/71] postgres --- doc/release-notes/11212-postgresql-16.md | 5 ----- doc/release-notes/6.6-release-notes.md | 10 ++++++++++ 2 files changed, 10 insertions(+), 5 deletions(-) delete mode 100644 doc/release-notes/11212-postgresql-16.md diff --git a/doc/release-notes/11212-postgresql-16.md b/doc/release-notes/11212-postgresql-16.md deleted file mode 100644 index b1a9c8691bf..00000000000 --- a/doc/release-notes/11212-postgresql-16.md +++ /dev/null @@ -1,5 +0,0 @@ -Per PostgreSQL 13 reaches EOL on 13 November 2025. Our first step toward moving off version 13 was to [switch](https://github.com/gdcc/dataverse-ansible/commit/8ebbd84ad2cf3903b8f995f0d34578250f4223ff) our testing to version 16, as we've [noted](https://dataverse-guide--11215.org.readthedocs.build/en/11215/installation/prerequisites.html#postgresql) in the guides. You are encouraged to start planning your upgrade and may want to review the [Dataverse 5.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v5.4) as the upgrade process (e.g. `pg_dumpall`, etc.) will likely be similar. If you notice any bumps along the way, please let us know! - -Dataverse developers [using Docker](https://guides.dataverse.org/en/6.5/container/dev-usage.html) have been using PostgreSQL 17 since Dataverse 6.5 (#10912). (Developers not using Docker who are still on PostgreSQL 13 are encouraged to upgrade.) Older or newer versions should work, within reason. - -See also #11212 and #11215. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index ff4c61f5413..4838f2cffc8 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -194,6 +194,16 @@ See also the "Settings Added" section, #10943 and #11124. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. - A new query param, `returnChildCount`, has been added to the getDataverse endpoint (`/api/dataverses/{id}`) for optionally retrieving the child count, which represents the number of collections, datasets, or files within the collection (direct children only). See also #11255 and #11259. +## End-Of-Life (EOL) Announcements + +### PostgreSQL 13 reaches EOL on 13 November 2025 + +Per PostgreSQL 13 reaches EOL on 13 November 2025. Our first step toward moving off version 13 was to [switch](https://github.com/gdcc/dataverse-ansible/commit/8ebbd84ad2cf3903b8f995f0d34578250f4223ff) our testing to version 16, as we've [noted](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#postgresql) in the guides. You are encouraged to start planning your upgrade and may want to review the [Dataverse 5.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v5.4) as the upgrade process (e.g. `pg_dumpall`, etc.) will likely be similar. If you notice any bumps along the way, please let us know! + +Dataverse developers [using Docker](https://guides.dataverse.org/en/6.6/container/dev-usage.html) have been using PostgreSQL 17 since Dataverse 6.5 (#10912). (Developers not using Docker who are still on PostgreSQL 13 are encouraged to upgrade.) Older or newer versions should work, within reason. + +See also #11212 and #11215. + ## Settings Added - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. From 3386882d6803aad796849958de5421b0242264bb Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 11 Mar 2025 17:14:51 -0400 Subject: [PATCH 37/71] source name harvesting --- .../11217-source-name-harvesting-client.md | 13 --------- doc/release-notes/6.6-release-notes.md | 28 ++++++++++++++----- 2 files changed, 21 insertions(+), 20 deletions(-) delete mode 100644 doc/release-notes/11217-source-name-harvesting-client.md diff --git a/doc/release-notes/11217-source-name-harvesting-client.md b/doc/release-notes/11217-source-name-harvesting-client.md deleted file mode 100644 index 53347de694c..00000000000 --- a/doc/release-notes/11217-source-name-harvesting-client.md +++ /dev/null @@ -1,13 +0,0 @@ -### Metadata Source Facet Can Now Differentiate Between Harvested Sources - -The behavior of the feature flag `index-harvested-metadata-source` and the "Metadata Source" facet, which were added and updated, respectively, in [Dataverse 6.3](https://github.com/IQSS/dataverse/releases/tag/v6.3) (through pull requests #10464 and #10651), have been updated. A new field called "Source Name" has been added to harvesting clients. - -Before Dataverse 6.3, all harvested content (datasets and files) appeared together under "Harvested" under the "Metadata Source" facet. This is still the behavior of Dataverse out of the box. Since Dataverse 6.3, enabling the `index-harvested-metadata-source` feature flag (and reindexing) resulted in harvested content appearing under the nickname for whatever harvesting client was used to bring in the content. This meant that instead of having all harvested content lumped together under "Harvested", content would appear under "client1", "client2", etc. - -Now, as this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to group harvested content from various harvesting clients under the same name if you wish (by reusing the same source name). - -Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://dataverse-guide--11217.org.readthedocs.build/en/11217/installation/config.html#feature-flags). See also #10217 and #11217. - -## Upgrade instructions - -If you have enabled the `dataverse.feature.index-harvested-metadata-source` feature flag and given some of your harvesting clients a source name, you should reindex to have those source names appear under the "Metadata Source" facet. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 4838f2cffc8..b20f2569ded 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -82,6 +82,16 @@ An experimental "Archival" metadata block has been added, [downloadable](https:/ Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. +### Metadata Source Facet Can Now Differentiate Between Harvested Sources + +The behavior of the feature flag `index-harvested-metadata-source` and the "Metadata Source" facet, which were added and updated, respectively, in [Dataverse 6.3](https://github.com/IQSS/dataverse/releases/tag/v6.3) (through pull requests #10464 and #10651), have been updated. A new field called "Source Name" has been added to harvesting clients. + +Before Dataverse 6.3, all harvested content (datasets and files) appeared together under "Harvested" under the "Metadata Source" facet. This is still the behavior of Dataverse out of the box. Since Dataverse 6.3, enabling the `index-harvested-metadata-source` feature flag (and reindexing) resulted in harvested content appearing under the nickname for whatever harvesting client was used to bring in the content. This meant that instead of having all harvested content lumped together under "Harvested", content would appear under "client1", "client2", etc. + +Now, as this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to group harvested content from various harvesting clients under the same name if you wish (by reusing the same source name). + +Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags). See also #10217 and #11217. + ### Signposting Output Now Contains Links to All Dataset Metadata Export Formats When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. @@ -317,14 +327,18 @@ Please remember to update translations via [Dataverse language packs](https://gi If you have text customizations you can get the latest English files from . -7\. Restart Payara +7\. Decide to enable (or not) the `index-harvested-metadata-source` feature flag + +Decide whether or not to enable the `dataverse.feature.index-harvested-metadata-source` feature flag described above, in [the guides](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags), #10217 and #11217. The reason to decide now is that reindexing is required and the next steps involve restarting Payara and upgrading Solr, which will result in a fresh index. + +8\. Restart Payara ```shell sudo service payara stop sudo service payara start ``` -8\. Update metadata blocks +9\. Update metadata blocks These changes reflect incremental improvements made to the handling of core metadata fields. @@ -344,7 +358,7 @@ wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/meta curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file 3d_objects.tsv ``` -9\. Upgrade Solr +10\. Upgrade Solr Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature and the addition of fields (e.g. fileRestricted, canDownloadFile, variableCount, and observations), the default `schema.xml` files has changed so you must upgrade. @@ -358,7 +372,7 @@ wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/schema.xml cp solrconfig.xml schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf ``` -9a\. For installations with custom or experimental metadata blocks, update fields +10a\. For installations with custom or experimental metadata blocks, update fields - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr-init-script)). @@ -372,13 +386,13 @@ curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /u - Start Solr instance (usually `service solr start` depending on Solr/OS). -10\. Reindex Solr +11\. Reindex Solr ```shell curl http://localhost:8080/api/admin/index ``` -11\. Run reExportAll to update dataset metadata exports +12\. Run reExportAll to update dataset metadata exports For existing published datasets, additional license metadata will not be available from DataCite or in metadata exports until @@ -393,7 +407,7 @@ For this reason, we recommend reexporting all dataset metadata. For more advance curl http://localhost:8080/api/admin/metadata/reExportAll ``` -12\. Re-harvest datasets +13\. Re-harvest datasets The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. For more information, see [the guides](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. From dc74a036e94e70759800111196a0771940cc03d8 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 09:40:14 -0400 Subject: [PATCH 38/71] samesite --- doc/release-notes/6.6-release-notes.md | 38 +++++++++++++++++++++++++ doc/release-notes/ds27-samesite-attr.md | 35 ----------------------- 2 files changed, 38 insertions(+), 35 deletions(-) delete mode 100644 doc/release-notes/ds27-samesite-attr.md diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index b20f2569ded..daafe5b430e 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -214,6 +214,20 @@ Dataverse developers [using Docker](https://guides.dataverse.org/en/6.6/containe See also #11212 and #11215. +## Security + +### SameSite Cookie Attribute + +The SameSite cookie attribute is defined in an upcoming revision to [RFC 6265](https://datatracker.ietf.org/doc/html/rfc6265) (HTTP State Management Mechanism) called [6265bis](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis-19>) ("bis" meaning "repeated"). The possible values are "None", "Lax", and "Strict". + +"If no SameSite attribute is set, the cookie is treated as Lax by default" by browsers according to [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#controlling_third-party_cookies_with_samesite). This was the previous behavior of Dataverse, to not set the SameSite attribute. + +New Dataverse installations now explicitly set to the SameSite cookie attribute to "Lax" out of the box through the installer (in the case of a "classic" installation) or through an updated base image (in the case of a Docker installation). Classic installations should follow the upgrade instructions below to bring their installation up to date with the behavior for new installations. Docker installations will automatically get the updated base image. + +While you are welcome to experiment with "Strict", which is intended to help prevent Cross-Site Request Forgery (CSRF) attacks, as described in the RFC proposal and an OWASP [cheetsheet](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#samesite-cookie-attribute), our testing so far indicates that some functionality, such as OIDC login, seems to be incompatible with "Strict". + +You should avoid the use of "None" as it is less secure that "Lax". See also [the guides](https://guides.dataverse.org/en/6.6/installation/config.html#samesite-cookie-attribute), https://github.com/IQSS/dataverse-security/issues/27, #11210, and the upgrade instructions below. + ## Settings Added - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. @@ -331,6 +345,30 @@ If you have text customizations you can get the latest English files from ) ("bis" meaning "repeated"). The possible values are "None", "Lax", and "Strict". - -"If no SameSite attribute is set, the cookie is treated as Lax by default" by browsers according to [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#controlling_third-party_cookies_with_samesite). This was the previous behavior of Dataverse, to not set the SameSite attribute. - -New Dataverse installations now explicitly set to the SameSite cookie attribute to "Lax" out of the box through the installer (in the case of a "classic" installation) or through an updated base image (in the case of a Docker installation). Classic installations should follow the upgrade instructions below to bring their installation up to date with the behavior for new installations. Docker installations will automatically get the updated base image. - -While you are welcome to experiment with "Strict", which is intended to help prevent Cross-Site Request Forgery (CSRF) attacks, as described in the RFC proposal and an OWASP [cheetsheet](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#samesite-cookie-attribute), our testing so far indicates that some functionality, such as OIDC login, seems to be incompatible with "Strict". - -You should avoid the use of "None" as it is less secure that "Lax". See also [the guides](https://dataverse-guide--11210.org.readthedocs.build/en/11210/installation/config.html#samesite-cookie-attribute), https://github.com/IQSS/dataverse-security/issues/27, #11210, and the upgrade instructions below. - -## Upgrade instructions - -To bring your Dataverse installation in line with new installations, as described in [the guides](https://dataverse-guide--11210.org.readthedocs.build/en/11210/installation/config.html#samesite-cookie-attribute), we recommend running the following commands: - -``` -./asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.cookie-same-site-value=Lax - -./asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.cookie-same-site-enabled=true -``` - -Please note that "None" is less secure than "Lax" and should be avoided. You can test the setting by inspecting headers with curl, looking at the JSESSIONID cookie for "SameSite=Lax" (yes, it's expected to be repeated, probably due to a bug in Payara) like this: - -``` -% curl -s -I http://localhost:8080 | grep JSESSIONID -Set-Cookie: JSESSIONID=6574324d75aebeb86dc96ecb3bb0; Path=/;SameSite=Lax;SameSite=Lax -``` - -Before making the changes above, SameSite attribute should be absent, like this: - -``` -% curl -s -I http://localhost:8080 | grep JSESSIONID -Set-Cookie: JSESSIONID=6574324d75aebeb86dc96ecb3bb0; Path=/ -``` From f2f3b93e0201d8279b19710b2758c662c363269f Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 09:42:31 -0400 Subject: [PATCH 39/71] show role --- doc/release-notes/10340-forbidden.md | 3 --- doc/release-notes/6.6-release-notes.md | 1 + 2 files changed, 1 insertion(+), 3 deletions(-) delete mode 100644 doc/release-notes/10340-forbidden.md diff --git a/doc/release-notes/10340-forbidden.md b/doc/release-notes/10340-forbidden.md deleted file mode 100644 index 5997f717d64..00000000000 --- a/doc/release-notes/10340-forbidden.md +++ /dev/null @@ -1,3 +0,0 @@ -### Backward Incompatible Changes - -The [Show Role](https://dataverse-guide--11116.org.readthedocs.build/en/11116/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://dataverse-guide--11116.org.readthedocs.build/en/11116/api/changelog.html), #10340, and #11116. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index daafe5b430e..cdc6f5496e8 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -238,6 +238,7 @@ You should avoid the use of "None" as it is less secure that "Lax". See also [th Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See #10764. +- The [Show Role](https://guides.dataverse.org/en/6.6/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://guides.dataverse.org/en/6.6/11116/api/changelog.html), #10340, and #11116. ## Complete List of Changes From 96ed7d4b3f986415839e3527beb01939ab1e4013 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 10:10:06 -0400 Subject: [PATCH 40/71] API updates --- doc/release-notes/10171-exlude-metadatablocks.md | 4 ---- ...11130-update-dataverse-api-remove-metadatablocks.md | 8 -------- doc/release-notes/11198-list-file-versions-tab | 10 ---------- doc/release-notes/6.6-release-notes.md | 3 +++ 4 files changed, 3 insertions(+), 22 deletions(-) delete mode 100644 doc/release-notes/10171-exlude-metadatablocks.md delete mode 100644 doc/release-notes/11130-update-dataverse-api-remove-metadatablocks.md delete mode 100644 doc/release-notes/11198-list-file-versions-tab diff --git a/doc/release-notes/10171-exlude-metadatablocks.md b/doc/release-notes/10171-exlude-metadatablocks.md deleted file mode 100644 index 7c0a9b030eb..00000000000 --- a/doc/release-notes/10171-exlude-metadatablocks.md +++ /dev/null @@ -1,4 +0,0 @@ -Extension of API `{id}/versions` and `{id}/versions/{versionId}` with an optional ``excludeMetadataBlocks`` parameter, -that specifies whether the metadataBlocks should be listed in the output. It defaults to ``false``, preserving backward -compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks -included can dramatically increase the volume of the output). See also [the guides](https://dataverse-guide--10778.org.readthedocs.build/en/10778/api/native-api.html#list-versions-of-a-dataset), #10778, and #10171. diff --git a/doc/release-notes/11130-update-dataverse-api-remove-metadatablocks.md b/doc/release-notes/11130-update-dataverse-api-remove-metadatablocks.md deleted file mode 100644 index e71a67142b3..00000000000 --- a/doc/release-notes/11130-update-dataverse-api-remove-metadatablocks.md +++ /dev/null @@ -1,8 +0,0 @@ -### Fixes consequences for not adding some optional fields in update dataverse API - -Omitting optional fields inputLevels, facetIds, or metadataBlockNames caused data to be deleted. -This fix no longer deletes data for these fields. Two new flags have been added to the ``metadataBlocks`` Json object to signal the deletion of the data. -- ``inheritMetadataBlocksFromParent: true`` will remove ``metadataBlockNames`` and ``inputLevels`` if the Json objects are omitted. -- ``inheritFacetsFromParent: true`` will remove ``facetIds`` if the Json object is omitted. - -For more information, see issue [#11130](https://github.com/IQSS/dataverse/issues/11130) diff --git a/doc/release-notes/11198-list-file-versions-tab b/doc/release-notes/11198-list-file-versions-tab deleted file mode 100644 index 83a4aa00e2c..00000000000 --- a/doc/release-notes/11198-list-file-versions-tab +++ /dev/null @@ -1,10 +0,0 @@ -Feature: New API for SPA to get the file versions metadata to populate the File Page Version Tab - - export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx - export SERVER_URL=https://demo.dataverse.org - export ID=19 - - ID can be either a DataFileID or ":persistentId" with a query parameter ?persistentId=doi:10.5072/FK2/ADMYJF - -``curl -H "X-Dataverse-key: $API_TOKEN" -X GET "$SERVER_URL/api/files/${ID}/versionDifferences"`` -``curl -H "X-Dataverse-key: $API_TOKEN" -X GET "$SERVER_URL/api/files/:persistentId/versionDifferences?persistentId=${ID}"`` diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index cdc6f5496e8..3003a6d32d8 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -133,6 +133,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. - For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://guides.dataverse.org/en/6.6/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320. - A new file has been added to import the French Open License to Dataverse: licenseEtalab-2.0.json. You can download it from [the guides](http://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses). This license, which is compatible with the Creative Commons license, is recommended by the French government for open documents. See #9301, #9302, and #11302. +- The API that lists versions of a dataset now features an optional `excludeMetadataBlocks` parameter, which defaults to "false" for backward compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks included can dramatically increase the volume of the output). See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#list-versions-of-a-dataset), #10171, and #10778. - Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. - The [tutorial](https://guides.dataverse.org/en/6.6/container/running/demo.html) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file (#10541 and #11201) and now uses the Permalink PID provider instead of the FAKE DOI Provider (#11107 and #11108). - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. @@ -161,6 +162,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. - In the Search API, files were displaying DRAFT version instead of latest released version under `dataset_citation`. See #10735 and #11051. - Unnecessary Solr documents were being created when a file was added or deleted from a draft dataset. These documents could accumulate and potentially impact performance. There is no action to take because this release includes a new Solr version, which will start with an empty database. See #11113 and #11114. +- When using the API to update a collection, omitting optional fields such as `inputLevels`, `facetIds`, or `metadataBlockNames` caused data to be deleted. The fix no longer deletes data for these fields. Two new flags have been added to the `metadataBlocks` JSON object to signal the deletion of the data: `inheritMetadataBlocksFromParent: true` and `inheritFacetsFromParent: true`. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#update-a-dataverse-collection), #11130, and #11144. ## API Updates @@ -199,6 +201,7 @@ See also the "Settings Added" section, #10943 and #11124. - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. - An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. +- An API has been added to list of versions of a data file showing any changes that affected the file with each version. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#get-json-representation-of-a-file-s-versions), #11198 and #11237. - The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). - A new API endpoint has been added that allows a global role to be updated. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#update-global-role) and #10612. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. From 601251182ac722b8b9e01375739db6bc020f751c Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 10:14:03 -0400 Subject: [PATCH 41/71] PID formatting --- ...ix pid handling in exporters and citations.md | 16 ---------------- doc/release-notes/6.6-release-notes.md | 2 ++ 2 files changed, 2 insertions(+), 16 deletions(-) delete mode 100644 doc/release-notes/10790-fix pid handling in exporters and citations.md diff --git a/doc/release-notes/10790-fix pid handling in exporters and citations.md b/doc/release-notes/10790-fix pid handling in exporters and citations.md deleted file mode 100644 index da430ed9671..00000000000 --- a/doc/release-notes/10790-fix pid handling in exporters and citations.md +++ /dev/null @@ -1,16 +0,0 @@ -### Improvements to PID formatting in exports and citations - -Multiple small issues with the formatting of PIDs in the -DDI exporters, and EndNote and BibTeX citation formats have -been addressed. These should improve the ability to import -Dataverse citations into reference managers and fix potential -issues harvesting datasets using PermaLinks. - -Backward Incompatibility - -Changes to PID formatting occur in the DDI/DDI Html export formats -and the EndNote and BibTex citation formats. These changes correct -errors and improve conformance with best practices but could break -parsing of these formats. - -For more information, see #10790. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 3003a6d32d8..a1b81188f84 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -143,6 +143,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en ## Bugs Fixed - A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. +- Multiple small issues with the formatting of PIDs in the DDI exporters, and EndNote and BibTeX citation formats have been addressed. These should improve the ability to import Dataverse citations into reference managers and fix potential issues harvesting datasets using PermaLinks. See #10768, #10769, #11165, and #10790. - On the Advanced Search page, the metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value, making the order the same as when you view or edit metadata. Note that fields that are not defined in the TSV file, like the "Persistent ID" and "Publication Date", will be displayed at the end. See #11272 and #11279. - Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. See #11203. - The file page version table now shows whether a file has been replaced. See #11142 and #11145. @@ -242,6 +243,7 @@ Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/late - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See #10764. - The [Show Role](https://guides.dataverse.org/en/6.6/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://guides.dataverse.org/en/6.6/11116/api/changelog.html), #10340, and #11116. +- Changes to PID formatting occur in the DDI/DDI Html export formats and the EndNote and BibTex citation formats. These changes correct errors and improve conformance with best practices but could break parsing of these formats. See #10768, #10769, #11165, and #10790. ## Complete List of Changes From c9d5aceb6ff83fce29b4e10d33bd1885dabb7000 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 10:25:50 -0400 Subject: [PATCH 42/71] external tools --- .../10930-marketplace-external-tools-apis.md | 14 -------------- doc/release-notes/6.6-release-notes.md | 1 + 2 files changed, 1 insertion(+), 14 deletions(-) delete mode 100644 doc/release-notes/10930-marketplace-external-tools-apis.md diff --git a/doc/release-notes/10930-marketplace-external-tools-apis.md b/doc/release-notes/10930-marketplace-external-tools-apis.md deleted file mode 100644 index e3350a8b2d2..00000000000 --- a/doc/release-notes/10930-marketplace-external-tools-apis.md +++ /dev/null @@ -1,14 +0,0 @@ -## New APIs for External Tools Registration for Marketplace - -New API base path /api/externalTools created that mimics the admin APIs /api/admin/externalTools. These new add and delete apis require an authenticated superuser token. - -Example: -``` - API_TOKEN='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - export TOOL_ID=1 - - curl http://localhost:8080/api/externalTools - curl http://localhost:8080/api/externalTools/$TOOL_ID - curl -s -H "X-Dataverse-key:$API_TOKEN" -X POST -H 'Content-type: application/json' http://localhost:8080/api/externalTools --upload-file fabulousFileTool.json - curl -s -H "X-Dataverse-key:$API_TOKEN" -X DELETE http://localhost:8080/api/externalTools/$TOOL_ID -``` diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index a1b81188f84..bbc1f380863 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -204,6 +204,7 @@ See also the "Settings Added" section, #10943 and #11124. - An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. - An API has been added to list of versions of a data file showing any changes that affected the file with each version. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#get-json-representation-of-a-file-s-versions), #11198 and #11237. - The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). +- CRUD operations for external tools are now available (for superusers) from non-localhost. See [the guides](https://guides.dataverse.org/en/6.6/admin/external-tools.html#managing-external-tools), #10930 and #11079. - A new API endpoint has been added that allows a global role to be updated. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#update-global-role) and #10612. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. - A new query param, `returnChildCount`, has been added to the getDataverse endpoint (`/api/dataverses/{id}`) for optionally retrieving the child count, which represents the number of collections, datasets, or files within the collection (direct children only). See also #11255 and #11259. From 1ea7d47cfae32f62a8e4ab91f1201a64878a3ae9 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 10:35:23 -0400 Subject: [PATCH 43/71] send feedback to contacts --- .../11129-send-feedback-to-contacts.md | 14 -------------- doc/release-notes/6.6-release-notes.md | 2 ++ 2 files changed, 2 insertions(+), 14 deletions(-) delete mode 100644 doc/release-notes/11129-send-feedback-to-contacts.md diff --git a/doc/release-notes/11129-send-feedback-to-contacts.md b/doc/release-notes/11129-send-feedback-to-contacts.md deleted file mode 100644 index 56eff133e0d..00000000000 --- a/doc/release-notes/11129-send-feedback-to-contacts.md +++ /dev/null @@ -1,14 +0,0 @@ -This feature adds a new API to send feedback to the Collection, Dataset, or DataFile's contacts. -Similar to the "admin/feedback" API the "sendfeedback" API sends an email to all the contacts listed for the Dataset. The main differences for this feature are: -1. This API is not limited to Admins -2. This API does not return the email addresses in the "toEmail" and "ccEmail" elements for privacy reasons -3. This API can be rate limited to avoid spamming -4. The body size limit can be configured -5. The body will be stripped of any html code to prevent malicious scripts or links -6. The fromEmail will be validated for correct format - -To set the Rate Limiting for guest users (See Rate Limiting Configuration for more details. This example allows 1 send per hour for any guest) -``curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{\"tier\": 0, \"limitPerHour\": 1, \"actions\": [\"CheckRateLimitForDatasetFeedbackCommand\"]}]'`` - -To set the message size limit (example limit of 1080 chars): -``curl -X PUT -d 1080 http://localhost:8080/api/admin/settings/:ContactFeedbackMessageSizeLimit`` diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index bbc1f380863..74e5f4cb6d1 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -206,6 +206,7 @@ See also the "Settings Added" section, #10943 and #11124. - The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). - CRUD operations for external tools are now available (for superusers) from non-localhost. See [the guides](https://guides.dataverse.org/en/6.6/admin/external-tools.html#managing-external-tools), #10930 and #11079. - A new API endpoint has been added that allows a global role to be updated. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#update-global-role) and #10612. +- An API has been added to send feedback to the collection, dataset, or data file's contacts. If necessary, you can [rate limit](https://guides.dataverse.org/en/6.6/installation/config.html#rate-limiting) the `CheckRateLimitForDatasetFeedbackCommand` and configure the new [:ContactFeedbackMessageSizeLimit](https://guides.dataverse.org/en/6.6/installation/config.html#contactfeedbackmessagesizelimit) database setting. See [the guides](http://guides.dataverse.org/en/6.6/api/native-api.html#send-feedback-to-contact-s), #11129, and #11162. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. - A new query param, `returnChildCount`, has been added to the getDataverse endpoint (`/api/dataverses/{id}`) for optionally retrieving the child count, which represents the number of collections, datasets, or files within the collection (direct children only). See also #11255 and #11259. @@ -237,6 +238,7 @@ You should avoid the use of "None" as it is less secure that "Lax". See also [th - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. - dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. +- :ContactFeedbackMessageSizeLimit ## Backward Incompatible Changes From 96dce81a7e72b503a88615ca9e902d44a38ecfdc Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 10:47:08 -0400 Subject: [PATCH 44/71] dft fix --- ...0472-review-modify-jsonprinter-for-datasetfieldtype.md | 8 -------- doc/release-notes/6.6-release-notes.md | 2 ++ 2 files changed, 2 insertions(+), 8 deletions(-) delete mode 100644 doc/release-notes/10472-review-modify-jsonprinter-for-datasetfieldtype.md diff --git a/doc/release-notes/10472-review-modify-jsonprinter-for-datasetfieldtype.md b/doc/release-notes/10472-review-modify-jsonprinter-for-datasetfieldtype.md deleted file mode 100644 index f0b9c30c9cd..00000000000 --- a/doc/release-notes/10472-review-modify-jsonprinter-for-datasetfieldtype.md +++ /dev/null @@ -1,8 +0,0 @@ -### Json Printer Bug fix - -DatasetFieldTypes in MetadataBlock response that are also a child of another DatasetFieldType were being returned twice. The child DatasetFieldType was included in the "fields" object as well as in the "childFields" of it's parent DatasetFieldType. This fix suppresses the standalone object so only one instance of the DatasetFieldType is returned (in the "childFields" of its parent). -This fix changes the Json output of the API `/api/dataverses/{dataverseAlias}/metadatablocks` - -## Backward Incompatible Changes - -The Json response of API call `/api/dataverses/{dataverseAlias}/metadatablocks` will no longer include the DatasetFieldTypes in "fields" if they are children of another DatasetFieldType. The child DatasetFieldType will only be included in the "childFields" of it's parent DatasetFieldType. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 74e5f4cb6d1..c4d84ec990c 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -156,6 +156,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Access requests were missing on the File Permission page after upgrading from Dataverse 6.0. This has been corrected with a database update script. See #10714 and #11061. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. - The `/api/info/metrics/files/monthly` API call had a bug that resulted in files being counted each time they were published in a new version if those publication events occurred in different months. This resulted in an over-count. The `/api/info/metrics/files` and `/api/info/metrics/files/toMonth` API calls had a bug that resulted in files that were published but no longer in the latest published version as of the specified date (now, or the date entered in the `/toMonth` variant). This resulted in an under-count. See 11189. +- DatasetFieldTypes in MetadataBlock response that are also a child of another DatasetFieldType were being returned twice. The child DatasetFieldType was included in the "fields" object as well as in the "childFields" of its parent DatasetFieldType. This fix suppresses the standalone object so only one instance of the DatasetFieldType is returned (in the "childFields" of its parent). This fix changes the JSON output of the API `/api/dataverses/{dataverseAlias}/metadatablocks` (see "Backward Incompatible Changes", below). See #10472 and #11066. - A bug that caused replacing files via API when file PIDs were enabled to fail has been fixed. See #10975 and #10979. - The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. @@ -245,6 +246,7 @@ You should avoid the use of "None" as it is less secure that "Lax". See also [th Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See #10764. +- The JSON response of API call `/api/dataverses/{dataverseAlias}/metadatablocks` will no longer include the DatasetFieldTypes in "fields" if they are children of another DatasetFieldType. The child DatasetFieldType will only be included in the "childFields" of its parent DatasetFieldType. See #10472 and #11066. - The [Show Role](https://guides.dataverse.org/en/6.6/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://guides.dataverse.org/en/6.6/11116/api/changelog.html), #10340, and #11116. - Changes to PID formatting occur in the DDI/DDI Html export formats and the EndNote and BibTex citation formats. These changes correct errors and improve conformance with best practices but could break parsing of these formats. See #10768, #10769, #11165, and #10790. From ba7db78ada41eca26a1d551eced230b0e5a464cc Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 10:51:42 -0400 Subject: [PATCH 45/71] perms --- doc/release-notes/6.6-release-notes.md | 1 + .../6467-optimize-permission-lookups-for-a-user.md | 9 --------- 2 files changed, 1 insertion(+), 9 deletions(-) delete mode 100644 doc/release-notes/6467-optimize-permission-lookups-for-a-user.md diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index c4d84ec990c..58649dc2f1f 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -202,6 +202,7 @@ See also the "Settings Added" section, #10943 and #11124. ### Other API Updates - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. +- An API has been added that lists the collections that the user has access to via the permission passed. See [the guides](http://guides.dataverse.org/en/6.6/api/native-api.html#list-dataverse-collections-a-user-can-act-on-based-on-their-permissions), #6467, and #10906. - An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. - An API has been added to list of versions of a data file showing any changes that affected the file with each version. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#get-json-representation-of-a-file-s-versions), #11198 and #11237. - The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). diff --git a/doc/release-notes/6467-optimize-permission-lookups-for-a-user.md b/doc/release-notes/6467-optimize-permission-lookups-for-a-user.md deleted file mode 100644 index 395c6b05c0a..00000000000 --- a/doc/release-notes/6467-optimize-permission-lookups-for-a-user.md +++ /dev/null @@ -1,9 +0,0 @@ -The following API have been added: - -/api/users/{identifier}/allowedCollections/{permission} - -This API lists the dataverses/collections that the user has access to via the permission passed. -By passing "any" as the permission the list will return all dataverse/collections that the user can access regardless of which permission is used. -This API can be executed only by the User requesting their own list of accessible collections or by an Administrator. -Valid Permissions are: AddDataverse, AddDataset, ViewUnpublishedDataverse, ViewUnpublishedDataset, DownloadFile, EditDataverse, EditDataset, ManageDataversePermissions, -ManageDatasetPermissions, ManageFilePermissions, PublishDataverse, PublishDataset, DeleteDataverse, DeleteDatasetDraft, and "any" as a wildcard option. From 679b27c94564582f0ff87206bee34e48c50f798e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 11:26:08 -0400 Subject: [PATCH 46/71] indexing cvoc --- doc/release-notes/11095-fix-extcvoc-indexing.md | 7 ------- doc/release-notes/6.6-release-notes.md | 3 ++- 2 files changed, 2 insertions(+), 8 deletions(-) delete mode 100644 doc/release-notes/11095-fix-extcvoc-indexing.md diff --git a/doc/release-notes/11095-fix-extcvoc-indexing.md b/doc/release-notes/11095-fix-extcvoc-indexing.md deleted file mode 100644 index a072491d04b..00000000000 --- a/doc/release-notes/11095-fix-extcvoc-indexing.md +++ /dev/null @@ -1,7 +0,0 @@ -Some External Controlled Vocabulary scripts/configurations, when used on a metadata field that is single-valued could result -in indexing failure for the dataset (e.g. when the script tried to index both the identifier and name of the identified entity for indexing). -Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to /api/admin/index/solr/schema. -Configuring the Solr schema and the update-fields.sh script as usually recommended when using custom metadata blocks will resolve the issue. - -The overall release notes should include a Solr update (which hopefully is required by an update to 9.8.0 anyway) and our standard instructions -should change to recommending use of the update-fields.sh script when using custom metadatablocks *and/or external vocabulary scripts*. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 58649dc2f1f..bcfc26db449 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -148,6 +148,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. See #11203. - The file page version table now shows whether a file has been replaced. See #11142 and #11145. - We fixed an issue where draft versions of datasets were sorted using the release timestamp of their most recent major version. This caused newer drafts to appear incorrectly alongside their corresponding major version, instead of at the top, when sorted by "newest first". Sorting now uses the last update timestamp when sorting draft datasets. The sorting behavior of published dataset versions (major and minor) is unchanged. There is no need to reindex datasets because Solr is being upgraded (see "Upgrade Instructions"), which will result in an empty database that will be reindexed. See #11178. +- Some external controlled vocabulary scripts/configurations, when used on a metadata field that is single-valued, could result in indexing failure for the dataset (e.g. when the script tried to index both the identifier and name of the identified entity for indexing). Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to `/api/admin/index/solr/schema`. Configuring the Solr schema and running the update-fields.sh script as usually recommended when using custom metadata blocks (see "Upgrade Instructions") will resolve the issue. See [the guides](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#using-external-vocabulary-services), #11095, and #11096. - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 - A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. @@ -422,7 +423,7 @@ wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/conf/solr/schema.xml cp solrconfig.xml schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf ``` -10a\. For installations with custom or experimental metadata blocks, update fields +10a\. For installations with additional metadata blocks or external controlled vocabulary scripts, update fields - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr-init-script)). From 32e2ca1a0a0d81b65e49bb566c7ea8e618bee25a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 11:31:01 -0400 Subject: [PATCH 47/71] API for classic download count --- .../11244-get-dataset-download-count-api.md | 12 ------------ doc/release-notes/6.6-release-notes.md | 1 + 2 files changed, 1 insertion(+), 12 deletions(-) delete mode 100644 doc/release-notes/11244-get-dataset-download-count-api.md diff --git a/doc/release-notes/11244-get-dataset-download-count-api.md b/doc/release-notes/11244-get-dataset-download-count-api.md deleted file mode 100644 index a03510cf1f2..00000000000 --- a/doc/release-notes/11244-get-dataset-download-count-api.md +++ /dev/null @@ -1,12 +0,0 @@ -### New API for the SPA, in order to replicate the "classic" download count. -- ``includeMDC`` parameter is optional. False or not included will return the total count of downloads if MDC is not running. Or, if MDCStartDate is set the count returned will be limited to the time prior to the MDCStartDate. Setting ``includeMDC`` to true will ignore any MDCStartDate and return the total download count for the dataset. - -Example: -``` - API_TOKEN='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - export DATASET_ID=1 - export includeMDC=False - - curl -s -H "X-Dataverse-key:$API_TOKEN" -X GET http://localhost:8080/api/datasets/$DATASET_ID/download/count - curl -s -H "X-Dataverse-key:$API_TOKEN" -X GET http://localhost:8080/api/datasets/$DATASET_ID/download/count?includeMDC=true -``` diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index bcfc26db449..163c465ce1b 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -203,6 +203,7 @@ See also the "Settings Added" section, #10943 and #11124. ### Other API Updates - Multiple files can be deleted from a dataset at once. See the [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#delete-files-from-a-dataset) and #11230. +- An API has been added to get the "classic" download count from a dataset with an optional `includeMDC` parameter (for Make Data Count). See [the guides](http://guides.dataverse.org/en/6.6/api/native-api.html#get-the-download-count-of-a-dataset), #11244 and #11282. - An API has been added that lists the collections that the user has access to via the permission passed. See [the guides](http://guides.dataverse.org/en/6.6/api/native-api.html#list-dataverse-collections-a-user-can-act-on-based-on-their-permissions), #6467, and #10906. - An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. - An API has been added to list of versions of a data file showing any changes that affected the file with each version. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#get-json-representation-of-a-file-s-versions), #11198 and #11237. From 45d85121665a3648f13fb58425a63d80e33e2563 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:05:52 -0400 Subject: [PATCH 48/71] optional version note --- doc/release-notes/6.6-release-notes.md | 15 ++++++++++++++- doc/release-notes/8431-versionNotes.md | 12 ------------ 2 files changed, 14 insertions(+), 13 deletions(-) delete mode 100644 doc/release-notes/8431-versionNotes.md diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 163c465ce1b..2456536fe2b 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,6 +8,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- version notes - license metadata enhancements - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting @@ -20,6 +21,16 @@ Highlights for Dataverse 6.6 include: ## Features Added +### Version Notes + +Dataverse now supports the option of adding a version note before or during the publication of a dataset. These notes can be used, for example, to indicate why a version was created or how it differs from the prior version. Whether this feature is enabled is controlled by the flag `dataverse.feature.enable-version-note`. Version notes are shown in the user interface (in the dataset page version table), indexed (as `versionNote`), available via the API, and have been added to the JSON, DDI, DataCite, and OAI-ORE exports. + +With the addition of this feature, work has been done to clean-up and rename fields that have been used for specifying the reason for deaccessioning a dataset and providing an optional link to a non-Dataverse location where the dataset still can be found. The former was listed in some JSON-based API calls and exports as "versionNote" and is now "deaccessionNote", while the latter was referred to as "archiveNote" and is now "deaccessionLink". These result in incompatibilities in the UI related to deaccessioned datasets. + +Further, some database consolidation has been done to combine the deaccessionlink and archivenote fields which appear to have both been used for the same purpose (the deaccessionlink database field is older and was not displayed in the current UI). Going forward, only the deaccessionlink column exists. + +See the [User Guide](https://guides.dataverse.org/en/6.6/user/dataset-management.html#data-provenance), [API Guide](https://guides.dataverse.org/en/6.6/api/native-api.html#dataset-version-notes) #8431, and #11068. + ### License Metadata Enhancements - Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode (see JSON files under [Adding Licenses](https://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses) in the guides) @@ -240,6 +251,7 @@ You should avoid the use of "None" as it is less secure that "Lax". See also [th ## Settings Added +- dataverse.feature.enable-version-note - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. - dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. - :ContactFeedbackMessageSizeLimit @@ -250,6 +262,7 @@ Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/late - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See #10764. - The JSON response of API call `/api/dataverses/{dataverseAlias}/metadatablocks` will no longer include the DatasetFieldTypes in "fields" if they are children of another DatasetFieldType. The child DatasetFieldType will only be included in the "childFields" of its parent DatasetFieldType. See #10472 and #11066. +- `versionNote` and been renamed to `deaccessionNote`. `archiveNote` have been renamed to `deaccessionLink`. See #11068. - The [Show Role](https://guides.dataverse.org/en/6.6/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://guides.dataverse.org/en/6.6/11116/api/changelog.html), #10340, and #11116. - Changes to PID formatting occur in the DDI/DDI Html export formats and the EndNote and BibTex citation formats. These changes correct errors and improve conformance with best practices but could break parsing of these formats. See #10768, #10769, #11165, and #10790. @@ -412,7 +425,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta 10\. Upgrade Solr -Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature and the addition of fields (e.g. fileRestricted, canDownloadFile, variableCount, and observations), the default `schema.xml` files has changed so you must upgrade. +Solr 9.8.0 is now the version recommended in our Installation Guide and used with automated testing. Additionally, due to the new range search support feature and the addition of fields (e.g. versionNote, fileRestricted, canDownloadFile, variableCount, and observations), the default `schema.xml` files has changed so you must upgrade. Install Solr 9.8.0 following the [instructions](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr) from the Installation Guide. diff --git a/doc/release-notes/8431-versionNotes.md b/doc/release-notes/8431-versionNotes.md deleted file mode 100644 index b9804833548..00000000000 --- a/doc/release-notes/8431-versionNotes.md +++ /dev/null @@ -1,12 +0,0 @@ -Dataverse now supports the option of adding a versionNote before/during publication of a dataset that can be used to indicate why a version was created and/or how it differs from the prior version. Whether this feature is enabled is controlled by a flag. Version notes are shown in the user interface (dataset page version table), indexed, available via the API, and have been added to the JSON, DDI, DataCite, and OAI-ORE exports. - -With the addition of this feature, work has been done to clean-up and rename fields that have been used for specifying the reason for deaccessioning a dataset and providing an optional link to a non-Dataverse location where the dataset still can be found. The former was listed in some JSON-based API calls and exports as "versionNote" and is now "deaccessionNote", while the latter was referred to as "archiveNote" and is now "deacccessionLink". These result in incompatibilities in the UI related to deaccessioned datasets. - -Further, some database consolidation has been done to combine the deaccessionlink and archivenote fields which appear to have both been used for the same purpose (the deaccessionlink db field is older and was not displayed in the current UI. Going forward, only the deaccessionlink column exists. - -New Feature Flags: - -VERSION_NOTE - false by default. - -Update of the solr schema (using the standard instructions) is needed. - From 2d96640ce93564d9b5ceb297dacd9b3a7bf3f896 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:08:16 -0400 Subject: [PATCH 49/71] remove scary line from optional version note --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 2456536fe2b..21a314e673e 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -25,7 +25,7 @@ Highlights for Dataverse 6.6 include: Dataverse now supports the option of adding a version note before or during the publication of a dataset. These notes can be used, for example, to indicate why a version was created or how it differs from the prior version. Whether this feature is enabled is controlled by the flag `dataverse.feature.enable-version-note`. Version notes are shown in the user interface (in the dataset page version table), indexed (as `versionNote`), available via the API, and have been added to the JSON, DDI, DataCite, and OAI-ORE exports. -With the addition of this feature, work has been done to clean-up and rename fields that have been used for specifying the reason for deaccessioning a dataset and providing an optional link to a non-Dataverse location where the dataset still can be found. The former was listed in some JSON-based API calls and exports as "versionNote" and is now "deaccessionNote", while the latter was referred to as "archiveNote" and is now "deaccessionLink". These result in incompatibilities in the UI related to deaccessioned datasets. +With the addition of this feature, work has been done to clean-up and rename fields that have been used for specifying the reason for deaccessioning a dataset and providing an optional link to a non-Dataverse location where the dataset still can be found. The former was listed in some JSON-based API calls and exports as "versionNote" and is now "deaccessionNote", while the latter was referred to as "archiveNote" and is now "deaccessionLink". Further, some database consolidation has been done to combine the deaccessionlink and archivenote fields which appear to have both been used for the same purpose (the deaccessionlink database field is older and was not displayed in the current UI). Going forward, only the deaccessionlink column exists. From b90abf1e5aa8176872cce1b962671d62c3f9a6ea Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:27:19 -0400 Subject: [PATCH 50/71] csl --- doc/release-notes/11163-CSL citation formats.md | 5 ----- doc/release-notes/6.6-release-notes.md | 12 ++++++++++++ 2 files changed, 12 insertions(+), 5 deletions(-) delete mode 100644 doc/release-notes/11163-CSL citation formats.md diff --git a/doc/release-notes/11163-CSL citation formats.md b/doc/release-notes/11163-CSL citation formats.md deleted file mode 100644 index d02a0ba4c3b..00000000000 --- a/doc/release-notes/11163-CSL citation formats.md +++ /dev/null @@ -1,5 +0,0 @@ -This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org/). -The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up the dataset page. -An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. (The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open-source libraries to generate all of the other CSL stypes citations.) - -Admins can use a new dataverse.csl.common-styles setting to highlight commonly used styles - common styles are listed in the pop-up, others can be found by type-ahead search in a list of 1000+ options. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 21a314e673e..7b5bc3c9b92 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -9,6 +9,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: - version notes +- citations using Citation Style Language (CSL) - license metadata enhancements - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting @@ -31,6 +32,16 @@ Further, some database consolidation has been done to combine the deaccessionlin See the [User Guide](https://guides.dataverse.org/en/6.6/user/dataset-management.html#data-provenance), [API Guide](https://guides.dataverse.org/en/6.6/api/native-api.html#dataset-version-notes) #8431, and #11068. +### Citations using Citation Style Language (CSL) + +This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org). + +The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. (The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL stypes citations.) + +Admins can use a new `dataverse.csl.common-styles` setting to highlight commonly used styles - common styles are listed in the pop-up, others can be found by type-ahead search in a list of 1000+ options. + +See the [User Guide](http://guides.dataverse.org/en/6.6/user/find-use-data.html#cite-data), [Settings](http://guides.dataverse.org/en/6.6/installation/config.html#dataverse-csl-common-styles), [API Guide](http://guides.dataverse.org/en/6.6/api/native-api.html#get-citation-in-other-formats), and #11163. + ### License Metadata Enhancements - Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode (see JSON files under [Adding Licenses](https://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses) in the guides) @@ -252,6 +263,7 @@ You should avoid the use of "None" as it is less secure that "Lax". See also [th ## Settings Added - dataverse.feature.enable-version-note +- dataverse.csl.common-styles - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. - dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. - :ContactFeedbackMessageSizeLimit From f7c841fc98caa31a3dbd62328f5d27dd12cdfd78 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:29:22 -0400 Subject: [PATCH 51/71] delete snippet without adding to notes, too in the weeds --- doc/release-notes/10241-new-solr-client.md | 9 --------- 1 file changed, 9 deletions(-) delete mode 100644 doc/release-notes/10241-new-solr-client.md diff --git a/doc/release-notes/10241-new-solr-client.md b/doc/release-notes/10241-new-solr-client.md deleted file mode 100644 index 67ccdd4f184..00000000000 --- a/doc/release-notes/10241-new-solr-client.md +++ /dev/null @@ -1,9 +0,0 @@ -[HttpSolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html) is deprecated as of Solr 9, and which will be removed in a future major release of Solr. It's recommended to use [Http2SolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/Http2SolrClient.html) instead. - -[Solr documentation](https://solr.apache.org/guide/solr/latest/deployment-guide/solrj.html#types-of-solrclients) describe it as a _async, non-blocking and general-purpose client that leverage HTTP/2 using the Jetty Http library_. - -With Solr 9.4.1, the Http2SolrClient is indicate as experimental. But since the 9.6 version of Solr, this mention is no longer maintained. - -The ConcurrentUpdateHttp2SolrClient is now also used in some cases, which is supposed to be more efficient for indexing. - -For more information, see issue [#10161](https://github.com/IQSS/dataverse/issues/10161) and pull request [#10241](https://github.com/IQSS/dataverse/pull/10241) From 5014082608b5afdc26d401c4733e2c74e22c73f1 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:40:33 -0400 Subject: [PATCH 52/71] harvest from datacite --- doc/release-notes/10909-datacite-oai-harvesting.md | 10 ---------- doc/release-notes/6.6-release-notes.md | 10 ++++++++++ 2 files changed, 10 insertions(+), 10 deletions(-) delete mode 100644 doc/release-notes/10909-datacite-oai-harvesting.md diff --git a/doc/release-notes/10909-datacite-oai-harvesting.md b/doc/release-notes/10909-datacite-oai-harvesting.md deleted file mode 100644 index be8265263fd..00000000000 --- a/doc/release-notes/10909-datacite-oai-harvesting.md +++ /dev/null @@ -1,10 +0,0 @@ -### OAI Harvesting from DataCite - -DataCite maintains an OAI server (https://oai.datacite.org/oai) that serves records for every DOI they have registered. There's been a lot of interest in the community in being able to harvest from them. This way, it will be possible to harvest metadata from institution X even if the institution X does not maintain an OAI server of their own, if they happen to register their DOIs with DataCite. One extra element of this harvesting model that makes it especially powerful and flexible is the DataCite's concept of a "dynamic OAI set": a harvester is not limited to harvesting the pre-defined set of ALL the records registered by the Institution X, but can instead harvest virtually any arbitrary subset thereof; any query that the DataCite search API understands can be used as an OAI set (!). The feature is already in use at IQSS, as a beta version patch. - -For various reasons, in order to take advantage of this feature harvesting clients must be created using the `/api/harvest/clients` API. Once configured however, harvests can be run from the Harvesting Clients control panel in the UI. - -DataCite-harvesting clients must be configured with 2 new feature flags, `useListRecords` and `useOaiIdentifiersAsPids` (added in v6.5). Note that these features may be of use when harvesting from other sources, not just from DataCite. - -See "Harvesting from DataCite" under https://guides.dataverse.org/en/latest/api/native-api.html#managing-harvesting-clients for more information. - diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 7b5bc3c9b92..e00c3d0a5fc 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -9,6 +9,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: - version notes +- harvesting from DataCite - citations using Citation Style Language (CSL) - license metadata enhancements - metadata fields now support range searches (dates, integers, etc.) @@ -32,6 +33,15 @@ Further, some database consolidation has been done to combine the deaccessionlin See the [User Guide](https://guides.dataverse.org/en/6.6/user/dataset-management.html#data-provenance), [API Guide](https://guides.dataverse.org/en/6.6/api/native-api.html#dataset-version-notes) #8431, and #11068. +### OAI-PMH Harvesting from DataCite + +DataCite maintains an OAI server () that serves records for every DOI they have registered. There's been a lot of interest in the community in being able to harvest from them. This way, it will be possible to harvest metadata from institution X even if the institution X does not maintain an OAI server of their own, if they happen to register their DOIs with DataCite. One extra element of this harvesting model that makes it especially powerful and flexible is the DataCite's concept of a "dynamic OAI set": a harvester is not limited to harvesting the pre-defined set of ALL the records registered by the institution X, but can instead harvest virtually any arbitrary subset thereof; any query that the DataCite search API understands can be used as an OAI set (!). The feature is already in use at IQSS, as a beta version patch. + +For various reasons, in order to take advantage of this feature harvesting clients must be created using the `/api/harvest/clients` API. Once configured however, harvests can be run from the Harvesting Clients control panel in the UI. + +DataCite-harvesting clients must be configured with 2 new feature flags, `useListRecords` and `useOaiIdentifiersAsPids` (added in Dataverse 6.5). Note that these features may be of use when harvesting from other sources, not just from DataCite. + +See the [Admin Guide](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-from-datacite), [API Guide](http://guides.dataverse.org/en/6.6/api/native-api.html#harvesting-from-datacite), #10909, and #11011. ### Citations using Citation Style Language (CSL) This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org). From 1bef954b9f9a64b74d02da7b8ee78ae3bea147dc Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:44:54 -0400 Subject: [PATCH 53/71] globus --- doc/release-notes/11057-globus-improvements.md | 11 ----------- doc/release-notes/6.6-release-notes.md | 10 ++++++++++ 2 files changed, 10 insertions(+), 11 deletions(-) delete mode 100644 doc/release-notes/11057-globus-improvements.md diff --git a/doc/release-notes/11057-globus-improvements.md b/doc/release-notes/11057-globus-improvements.md deleted file mode 100644 index d559d7afa82..00000000000 --- a/doc/release-notes/11057-globus-improvements.md +++ /dev/null @@ -1,11 +0,0 @@ -## Globus framework improvements - -The improvements and optimizations in this release build on top of the earlier work (such as PR #10781). They are based on the experience gained at IQSS as part of the production rollout of the Large Data Storage services that utilizes Globus. - -The changes in this PR (#11125) focus on improving Globus *downloads* (i.e., transfers from Dataverse-linked Globus volumes to users' Globus collections). Most importantly, th mechanism of "Asynchronous Task Monitoring", first introduced in #10781 for *uploads*, has been extended to handle downloads as well. This generally makes downloads more reliable (specifically, in how Dataverse manages temporary access rules granted to users, minimizing the risk of consequent downloads failing because of stale access rules left in place). - -See `globus-use-experimental-async-framework` under [Feature Flags](https://guides.dataverse.org/en/latest/installation/config.html#feature-flags) and [dataverse.files.globus-monitoring-server](https://guides.dataverse.org/en/latest/installation/config.html#dataverse-files-globus-monitoring-server) in the Installation Guide. - -Multiple other improvements have been made making the underlying Globus framework more reliable and robust. - - diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index e00c3d0a5fc..7d740bd4104 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -124,6 +124,16 @@ Now, as this release, enabling the `index-harvested-metadata-source` feature fla Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags). See also #10217 and #11217. +## Globus Framework Improvements + +The improvements and optimizations in this release build on top of the earlier work (such as #10781). They are based on the experience gained at IQSS as part of the production rollout of the Large Data Storage services that utilizes Globus. + +The changes in this release focus on improving Globus *downloads* (i.e., transfers from Dataverse-linked Globus volumes to users' Globus collections). Most importantly, the mechanism of "Asynchronous Task Monitoring", first introduced in #10781 for *uploads*, has been extended to handle downloads as well. This generally makes downloads more reliable (specifically, in how Dataverse manages temporary access rules granted to users, minimizing the risk of consequent downloads failing because of stale access rules left in place). + +Multiple other improvements have been made making the underlying Globus framework more reliable and robust. + +See `globus-use-experimental-async-framework` under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags) and [dataverse.files.globus-monitoring-server](https://guides.dataverse.org/en/6.6/installation/config.html#dataverse-files-globus-monitoring-server) in the Installation Guide, #11057, and #11125. + ### Signposting Output Now Contains Links to All Dataset Metadata Export Formats When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. From a2c1cfe116a8423787c25cf28537478491214dc0 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:50:17 -0400 Subject: [PATCH 54/71] bearer tokens --- doc/release-notes/10959-oidc-api-auth-ext.md | 14 -------------- doc/release-notes/6.6-release-notes.md | 12 ++++++++++++ 2 files changed, 12 insertions(+), 14 deletions(-) delete mode 100644 doc/release-notes/10959-oidc-api-auth-ext.md diff --git a/doc/release-notes/10959-oidc-api-auth-ext.md b/doc/release-notes/10959-oidc-api-auth-ext.md deleted file mode 100644 index 04ee2099f68..00000000000 --- a/doc/release-notes/10959-oidc-api-auth-ext.md +++ /dev/null @@ -1,14 +0,0 @@ -Extends the OIDC API auth mechanism (available through feature flag ``api-bearer-auth``) to properly handle cases -where ``BearerTokenAuthMechanism`` successfully validates the token but cannot identify any Dataverse user because there -is no account associated with the token. - -To register a new user who has authenticated via an OIDC provider, a new endpoint has been -implemented (``/users/register``). A feature flag named ``api-bearer-auth-provide-missing-claims`` has been implemented -to allow -sending missing user claims in the request JSON. This is useful when the identity provider does not supply the necessary -claims. However, this flag will only be considered if the ``api-bearer-auth`` feature flag is enabled. If the latter is -not enabled, the ``api-bearer-auth-provide-missing-claims`` flag will be ignored. - -A feature flag named ``api-bearer-auth-handle-tos-acceptance-in-idp`` has been implemented. When enabled, it specifies -that Terms of Service acceptance is managed by the identity provider, eliminating the need to explicitly include the -acceptance in the user registration request JSON. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 7d740bd4104..244461d72a0 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -134,6 +134,16 @@ Multiple other improvements have been made making the underlying Globus framewor See `globus-use-experimental-async-framework` under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags) and [dataverse.files.globus-monitoring-server](https://guides.dataverse.org/en/6.6/installation/config.html#dataverse-files-globus-monitoring-server) in the Installation Guide, #11057, and #11125. +### OIDC Bearer Tokens + +The release extends the OIDC API auth mechanism (available through feature flag `api-bearer-auth`) to properly handle cases where ``BearerTokenAuthMechanism`` successfully validates the token but cannot identify any Dataverse user because there is no account associated with the token. + +To register a new user who has authenticated via an OIDC provider, a new endpoint has been implemented (`/users/register`). A feature flag named `api-bearer-auth-provide-missing-claims` has been implemented to allow sending missing user claims in the request JSON. This is useful when the identity provider does not supply the necessary claims. However, this flag will only be considered if the `api-bearer-auth` feature flag is enabled. If the latter is not enabled, the `api-bearer-auth-provide-missing-claims` flag will be ignored. + +A feature flag named `api-bearer-auth-handle-tos-acceptance-in-idp` has been implemented. When enabled, it specifies that Terms of Service acceptance is managed by the identity provider, eliminating the need to explicitly include the acceptance in the user registration request JSON. + +See [the guides](https://guides.dataverse.org/en/6.5/api/auth.html#bearer-tokens), #10959, and #10972. + ### Signposting Output Now Contains Links to All Dataset Metadata Export Formats When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. @@ -286,6 +296,8 @@ You should avoid the use of "None" as it is less secure that "Lax". See also [th - dataverse.csl.common-styles - dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item. - dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory. +- dataverse.feature.api-bearer-auth-provide-missing-claims +- dataverse.feature.api-bearer-auth-handle-tos-acceptance-in-idp - :ContactFeedbackMessageSizeLimit ## Backward Incompatible Changes From 153291c39b24a6a16e38e9ac2b9232e5a2ea1331 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 12:50:47 -0400 Subject: [PATCH 55/71] fix heading level --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 244461d72a0..230b0d19c6a 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -124,7 +124,7 @@ Now, as this release, enabling the `index-harvested-metadata-source` feature fla Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags). See also #10217 and #11217. -## Globus Framework Improvements +### Globus Framework Improvements The improvements and optimizations in this release build on top of the earlier work (such as #10781). They are based on the experience gained at IQSS as part of the production rollout of the Large Data Storage services that utilizes Globus. From fed1541198b43c9cf9f5adeeb86afc14762275d6 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 13:06:26 -0400 Subject: [PATCH 56/71] ORCID stuff --- doc/release-notes/11222-ORCID in profile.md | 9 --------- doc/release-notes/11242-fix-oricid-recognition.md | 3 --- doc/release-notes/6.6-release-notes.md | 14 ++++++++++++++ 3 files changed, 14 insertions(+), 12 deletions(-) delete mode 100644 doc/release-notes/11222-ORCID in profile.md delete mode 100644 doc/release-notes/11242-fix-oricid-recognition.md diff --git a/doc/release-notes/11222-ORCID in profile.md b/doc/release-notes/11222-ORCID in profile.md deleted file mode 100644 index a4e0a5cbcc0..00000000000 --- a/doc/release-notes/11222-ORCID in profile.md +++ /dev/null @@ -1,9 +0,0 @@ -Dataverse now includes improved integration with ORCID, supported through a grant to GDCC from the ([ORCID Global Participation Fund](https://info.orcid.org/global-participation-fund-announces-fourth-round-of-awardees/).)[https://info.orcid.org/global-participation-fund-announces-fourth-round-of-awardees/]. - -Specifically, Dataverse users can now link their Dataverse account with their ORCID profile which then allows Dataverse to use automatically add their ORCID to their author metadata when they create a dataset. - -This functionality leverages Dataverse's existing support for login via ORCID, but can be turned on independently of it. If ORCID login is enabled, the user's ORCID will automatically be added to their profile. If the user has logged in via some other mechanism, they are able to click a button to initiate a similar authentication process in which the user must login to their ORCID account and approve the connection. - -Feedback from installations that enable this functionality is requested and we expect that updates can be made in the next Dataverse release. - -See also [the](https://dataverse-guide--11222.org.readthedocs.build/en/11222/user/account.html#linking-orcid-with-your-account-profile) [guides](https://dataverse-guide--11222.org.readthedocs.build/en/11222/installation/orcid.html), #7284, and #11222. diff --git a/doc/release-notes/11242-fix-oricid-recognition.md b/doc/release-notes/11242-fix-oricid-recognition.md deleted file mode 100644 index 268a63bb05e..00000000000 --- a/doc/release-notes/11242-fix-oricid-recognition.md +++ /dev/null @@ -1,3 +0,0 @@ -A bug that caused ORCIDs start with https://orcid.org/ entered as author identifier to be ignored when creating the DataCite metadata has been fixed. This primarily affected users of the ORCID external vocabulary script as the manual entry form recommends not using the URL form. - -The display of authorIdentifier, when not using any external vocabulary scripts, has been improved so that either the plain identifier (e.g. "0000-0002-1825-0097") or its URL form (e.g. "https://orcid.org/0000-0002-1825-0097") will result in valid links in the display (for identifier types that have a URL form). The URL form is now recommended when doing manual entry. diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 230b0d19c6a..1b5b8424346 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,6 +8,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- ORCIDS linked to accounts - version notes - harvesting from DataCite - citations using Citation Style Language (CSL) @@ -23,6 +24,18 @@ Highlights for Dataverse 6.6 include: ## Features Added +### ORCIDs Linked to Accounts + +Dataverse now includes improved integration with ORCID, supported through a grant to GDCC from the ([ORCID Global Participation Fund](https://info.orcid.org/global-participation-fund-announces-fourth-round-of-awardees/)). + +Specifically, Dataverse users can now link their Dataverse account with their ORCID profile. (Previously, this was only availabe to users who logged in with ORCID.) Once linked, Dataverse will automatically prepopulate their ORCID to their author metadata when they create a dataset. + +This functionality leverages Dataverse's existing support for login via ORCID, but can be turned on independently of it. If ORCID login is enabled, the user's ORCID will automatically be added to their profile. If the user has logged in via some other mechanism, they are able to click a button to initiate a similar authentication process in which the user must login to their ORCID account and approve the connection. + +Feedback from installations that enable this functionality is requested and we expect that updates can be made in the next Dataverse release. + +See the [User Guide](http://guides.dataverse.org/en/6.6/user/account.html#linking-orcid-with-your-account-profile), [Installation Guide](http://guides.dataverse.org/en/6.6/installation/orcid.html), #7284, and #11222. + ### Version Notes Dataverse now supports the option of adding a version note before or during the publication of a dataset. These notes can be used, for example, to indicate why a version was created or how it differs from the prior version. Whether this feature is enabled is controlled by the flag `dataverse.feature.enable-version-note`. Version notes are shown in the user interface (in the dataset page version table), indexed (as `versionNote`), available via the API, and have been added to the JSON, DDI, DataCite, and OAI-ORE exports. @@ -195,6 +208,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en ## Bugs Fixed - A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. +- A bug that caused ORCIDs starting with "https://orcid.org/" entered as author identifier to be ignored when creating the DataCite metadata has been fixed. This primarily affected users of the ORCID external vocabulary script as the manual entry form recommends not using the URL form. The display of authorIdentifier, when not using any external vocabulary scripts, has been improved so that either the plain identifier (e.g. "0000-0002-1825-0097") or its URL form (e.g. "https://orcid.org/0000-0002-1825-0097") will result in valid links in the display (for identifier types that have a URL form). The URL form is now [recommended](http://guides.dataverse.org/en/6.6/user/dataset-management.html#adding-a-new-dataset) when doing manual entry. See #11242 and #11242. - Multiple small issues with the formatting of PIDs in the DDI exporters, and EndNote and BibTeX citation formats have been addressed. These should improve the ability to import Dataverse citations into reference managers and fix potential issues harvesting datasets using PermaLinks. See #10768, #10769, #11165, and #10790. - On the Advanced Search page, the metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value, making the order the same as when you view or edit metadata. Note that fields that are not defined in the TSV file, like the "Persistent ID" and "Publication Date", will be displayed at the end. See #11272 and #11279. - Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. See #11203. From 4bbd25d8a0e9643be2b12a7cbb905eb0d439c523 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 13:12:44 -0400 Subject: [PATCH 57/71] how to re-harvest --- doc/release-notes/6.6-release-notes.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 1b5b8424346..2360ca0cfae 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -530,8 +530,10 @@ For this reason, we recommend reexporting all dataset metadata. For more advance curl http://localhost:8080/api/admin/metadata/reExportAll ``` -13\. Re-harvest datasets +13\. (Optional) Re-harvest datasets The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. For more information, see [the guides](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. -This improves the citation associated with these datasets, but the change only affects newly harvested datasets. All datasets should be re-harvested if you wish to pick up this change on already harvested datasets. +This improves the citation associated with these datasets, but the change only affects newly harvested datasets. + +If you would like to pick up this change for existing harvested datasets, you should re-harvest them. This can be accomplished by deleting and re-adding each harvesting client, followed by a harvesting run. You may want to use [harvesting client APIs](https://guides.dataverse.org/en/6.6/api/native-api.html#managing-harvesting-clients) to save (serialize), add, and remove clients. From 09bbce876ec4ac074a6c9685a2c17504cba7add2 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 13:13:49 -0400 Subject: [PATCH 58/71] fix link --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 2360ca0cfae..2ceae0edf78 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -155,7 +155,7 @@ To register a new user who has authenticated via an OIDC provider, a new endpoin A feature flag named `api-bearer-auth-handle-tos-acceptance-in-idp` has been implemented. When enabled, it specifies that Terms of Service acceptance is managed by the identity provider, eliminating the need to explicitly include the acceptance in the user registration request JSON. -See [the guides](https://guides.dataverse.org/en/6.5/api/auth.html#bearer-tokens), #10959, and #10972. +See [the guides](https://guides.dataverse.org/en/6.6/api/auth.html#bearer-tokens), #10959, and #10972. ### Signposting Output Now Contains Links to All Dataset Metadata Export Formats From eb4975edf84142fe54c66c220ca5c832f13f17ad Mon Sep 17 00:00:00 2001 From: Julian Gautier Date: Wed, 12 Mar 2025 14:39:26 -0400 Subject: [PATCH 59/71] Update 6.6-release-notes.md Fixed some typos and made some changes to improve readability --- doc/release-notes/6.6-release-notes.md | 66 +++++++++++++------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 2ceae0edf78..93b555e62ae 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -15,7 +15,7 @@ Highlights for Dataverse 6.6 include: - license metadata enhancements - metadata fields now support range searches (dates, integers, etc.) - more accurate search highlighting -- collections can be moved from the superuser dashboard +- collections can be moved by using the superuser dashboard - new 3D Objects metadata block - new Archival metadata block (experimental) - optionally prevent publishing of datasets without files @@ -28,7 +28,7 @@ Highlights for Dataverse 6.6 include: Dataverse now includes improved integration with ORCID, supported through a grant to GDCC from the ([ORCID Global Participation Fund](https://info.orcid.org/global-participation-fund-announces-fourth-round-of-awardees/)). -Specifically, Dataverse users can now link their Dataverse account with their ORCID profile. (Previously, this was only availabe to users who logged in with ORCID.) Once linked, Dataverse will automatically prepopulate their ORCID to their author metadata when they create a dataset. +Specifically, Dataverse users can now link their Dataverse account with their ORCID profile. Previously, this was only availabe to users who logged in with ORCID. Once linked, Dataverse will automatically prepopulate their ORCID to their author metadata when they create a dataset. This functionality leverages Dataverse's existing support for login via ORCID, but can be turned on independently of it. If ORCID login is enabled, the user's ORCID will automatically be added to their profile. If the user has logged in via some other mechanism, they are able to click a button to initiate a similar authentication process in which the user must login to their ORCID account and approve the connection. @@ -42,13 +42,13 @@ Dataverse now supports the option of adding a version note before or during the With the addition of this feature, work has been done to clean-up and rename fields that have been used for specifying the reason for deaccessioning a dataset and providing an optional link to a non-Dataverse location where the dataset still can be found. The former was listed in some JSON-based API calls and exports as "versionNote" and is now "deaccessionNote", while the latter was referred to as "archiveNote" and is now "deaccessionLink". -Further, some database consolidation has been done to combine the deaccessionlink and archivenote fields which appear to have both been used for the same purpose (the deaccessionlink database field is older and was not displayed in the current UI). Going forward, only the deaccessionlink column exists. +Further, some database consolidation has been done to combine the deaccessionlink and archivenote fields, which appear to have both been used for the same purpose. The deaccessionlink database field is older and also was not displayed in the current UI. Going forward, only the deaccessionlink column exists. See the [User Guide](https://guides.dataverse.org/en/6.6/user/dataset-management.html#data-provenance), [API Guide](https://guides.dataverse.org/en/6.6/api/native-api.html#dataset-version-notes) #8431, and #11068. ### OAI-PMH Harvesting from DataCite -DataCite maintains an OAI server () that serves records for every DOI they have registered. There's been a lot of interest in the community in being able to harvest from them. This way, it will be possible to harvest metadata from institution X even if the institution X does not maintain an OAI server of their own, if they happen to register their DOIs with DataCite. One extra element of this harvesting model that makes it especially powerful and flexible is the DataCite's concept of a "dynamic OAI set": a harvester is not limited to harvesting the pre-defined set of ALL the records registered by the institution X, but can instead harvest virtually any arbitrary subset thereof; any query that the DataCite search API understands can be used as an OAI set (!). The feature is already in use at IQSS, as a beta version patch. +DataCite maintains an OAI server () that serves records for every DOI they have registered. There's been a lot of interest in the community in being able to harvest from them. This way, it will be possible to harvest metadata from institution X even if the institution X does not maintain an OAI server of their own, if they happen to register their DOIs with DataCite. One extra element of this harvesting model that makes it especially powerful and flexible is the DataCite's concept of a "dynamic OAI set": a harvester is not limited to harvesting the pre-defined set of ALL the records registered by the institution X, but can instead harvest virtually any arbitrary subset thereof; any query that the DataCite search API understands can be used as an OAI set. The feature is already in use at Harvard Dataverse, as a beta version patch. For various reasons, in order to take advantage of this feature harvesting clients must be created using the `/api/harvest/clients` API. Once configured however, harvests can be run from the Harvesting Clients control panel in the UI. @@ -59,15 +59,15 @@ See the [Admin Guide](http://guides.dataverse.org/en/6.6/admin/harvestclients.ht This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org). -The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. (The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL stypes citations.) +The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up on the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL stypes citations. -Admins can use a new `dataverse.csl.common-styles` setting to highlight commonly used styles - common styles are listed in the pop-up, others can be found by type-ahead search in a list of 1000+ options. +Admins can use a new `dataverse.csl.common-styles` setting to highlight commonly used styles. Common styles are listed in the pop-up, while others can be found by type-ahead search in a list of 1000+ options. See the [User Guide](http://guides.dataverse.org/en/6.6/user/find-use-data.html#cite-data), [Settings](http://guides.dataverse.org/en/6.6/installation/config.html#dataverse-csl-common-styles), [API Guide](http://guides.dataverse.org/en/6.6/api/native-api.html#get-citation-in-other-formats), and #11163. ### License Metadata Enhancements -- Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode (see JSON files under [Adding Licenses](https://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses) in the guides) +- Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode. See JSON files under [Adding Licenses](https://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses) in the guides - Updated DataCite metadata export to include rightsIdentifier, rightsIdentifierScheme, and schemeUri consistent with the DataCite 4.5 schema and examples - Enhanced metadata exports to include all new license fields - Existing licenses from the example set included with Dataverse will be automatically updated with new fields @@ -125,7 +125,7 @@ An experimental "Archival" metadata block has been added, [downloadable](https:/ ### Prevent Publishing of Datasets Without Files -Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can only be set via API and only by a superuser (see [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes)). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. +Datasets without files can be optionally prevented from being published through a new "requireFilesToPublishDataset" boolean defined at the collection level. This boolean can be set only via API and only by a superuser. See [Change Collection Attributes](https://guides.dataverse.org/en/6.6/api/native-api.html#change-collection-attributes). If the boolean is not set, the parent collection is consulted. If you do not set the boolean, the existing behavior of datasets being able to be published without files will continue. Superusers can still publish datasets whether or not the boolean is set. See #10981 and #10994. ### Metadata Source Facet Can Now Differentiate Between Harvested Sources @@ -133,7 +133,7 @@ The behavior of the feature flag `index-harvested-metadata-source` and the "Meta Before Dataverse 6.3, all harvested content (datasets and files) appeared together under "Harvested" under the "Metadata Source" facet. This is still the behavior of Dataverse out of the box. Since Dataverse 6.3, enabling the `index-harvested-metadata-source` feature flag (and reindexing) resulted in harvested content appearing under the nickname for whatever harvesting client was used to bring in the content. This meant that instead of having all harvested content lumped together under "Harvested", content would appear under "client1", "client2", etc. -Now, as this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to group harvested content from various harvesting clients under the same name if you wish (by reusing the same source name). +With this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to reuse the same source name to group harvested content from various harvesting clients under the same name if you wish. Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags). See also #10217 and #11217. @@ -141,7 +141,7 @@ Previously, `index-harvested-metadata-source` was not documented in the guides, The improvements and optimizations in this release build on top of the earlier work (such as #10781). They are based on the experience gained at IQSS as part of the production rollout of the Large Data Storage services that utilizes Globus. -The changes in this release focus on improving Globus *downloads* (i.e., transfers from Dataverse-linked Globus volumes to users' Globus collections). Most importantly, the mechanism of "Asynchronous Task Monitoring", first introduced in #10781 for *uploads*, has been extended to handle downloads as well. This generally makes downloads more reliable (specifically, in how Dataverse manages temporary access rules granted to users, minimizing the risk of consequent downloads failing because of stale access rules left in place). +The changes in this release focus on improving Globus *downloads*, i.e., transfers from Dataverse-linked Globus volumes to users' Globus collections. Most importantly, the mechanism of "Asynchronous Task Monitoring", first introduced in #10781 for *uploads*, has been extended to handle downloads as well. This generally makes downloads more reliable, specifically in how Dataverse manages temporary access rules granted to users, minimizing the risk of consequent downloads failing because of stale access rules left in place. Multiple other improvements have been made making the underlying Globus framework more reliable and robust. @@ -149,7 +149,7 @@ See `globus-use-experimental-async-framework` under [Feature Flags](https://guid ### OIDC Bearer Tokens -The release extends the OIDC API auth mechanism (available through feature flag `api-bearer-auth`) to properly handle cases where ``BearerTokenAuthMechanism`` successfully validates the token but cannot identify any Dataverse user because there is no account associated with the token. +The release extends the OIDC API auth mechanism, available through feature flag `api-bearer-auth`, to properly handle cases where ``BearerTokenAuthMechanism`` successfully validates the token but cannot identify any Dataverse user because there is no account associated with the token. To register a new user who has authenticated via an OIDC provider, a new endpoint has been implemented (`/users/register`). A feature flag named `api-bearer-auth-provide-missing-claims` has been implemented to allow sending missing user claims in the request JSON. This is useful when the identity provider does not supply the necessary claims. However, this flag will only be considered if the `api-bearer-auth` feature flag is enabled. If the latter is not enabled, the `api-bearer-auth-provide-missing-claims` flag will be ignored. @@ -159,11 +159,11 @@ See [the guides](https://guides.dataverse.org/en/6.6/api/auth.html#bearer-tokens ### Signposting Output Now Contains Links to All Dataset Metadata Export Formats -When Signposting was added in Dataverse 5.14 (#8981), it only provided links for the `schema.org` metadata export format. +When Signposting was added in Dataverse 5.14 (#8981), it provided links only for the `schema.org` metadata export format. -The output of HEAD, GET, and the Signposting "linkset" API have all been updated to include links to all available dataset metadata export formats (including any external exporters, such as Croissant, that have been enabled). +The output of HEAD, GET, and the Signposting "linkset" API have all been updated to include links to all available dataset metadata export formats, including any external exporters, such as Croissant, that have been enabled. -This provides a lightweight machine-readable way to first retrieve a list of links (via a HTTP HEAD request, for example) to each available metadata export format and then follow up with a request for the export format of interest. +This provides a lightweight machine-readable way to first retrieve a list of links, such as via a HTTP HEAD request, to each available metadata export format and then follow up with a request for the export format of interest. In addition, the content type for the `schema.org` dataset metadata export format has been corrected. It was `application/json` and now it is `application/ld+json`. @@ -171,7 +171,7 @@ See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#re ### Dataset Types Can Be Linked to Metadata Blocks -Metadata blocks (e.g. "CodeMeta") can now be linked to dataset types (e.g. "software") using new superuser APIs. +Metadata blocks, such as (e.g. "CodeMeta") can now be linked to dataset types (e.g. "software") using new superuser APIs. This will have the following effects for the APIs used by the [new Dataverse UI](https://github.com/IQSS/dataverse-frontend): @@ -184,21 +184,21 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en ### Other Features -- In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard. See #10304 and #11150. -- The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. -- [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. -- The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. This improves the citation associated with these datasets, but the change only affects newly harvested datasets. See "Upgrade Instructions" below on how to re-harvest. For more information, see [the guides](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. -- A new harvest status differentiates between a complete harvest with errors (completed with failures) and without errors (completed). Also, harvest status labels are now internationalized. See #9294 and #11017. -- The OAI-ORE exporter can now export metadata containing nested compound fields (i.e. compound fields within compound fields). See #10809 and #11190. +- In addition to the API [Move a Dataverse Collection](https://guides.dataverse.org/en/6.6/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection using the Dataverse dashboard. See #10304 and #11150. +- The Preview URL popup and [related documentation](https://guides.dataverse.org/en/6.6/user/dataset-management.html#preview-url-to-review-unpublished-dataset) have been updated to give more information about anonymous access, including the names of the dataset fields that will be withheld from the Anonymous Preview URL user and to suggest how to review the URL before releasing it. See also #11159 and #11164. +- [ROR](https://ror.org) (Research Organization Registry) has been added as an Author Identifier Type for when the author is an organization rather than a person. Like ORCID, ROR will appear in the "Datacite" metadata export format. See #11075 and #11118. +- The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. This improves the citation associated with these datasets, but the change affects only newly harvested datasets. See "Upgrade Instructions" below on how to re-harvest. For more information, see [the guides](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013. +- A new harvest status differentiates between a complete harvest with errors ("completed with failures") and without errors ("completed"). Also, harvest status labels are now internationalized. See #9294 and #11017. +- The OAI-ORE exporter can now export metadata containing nested compound fields or compound fields within compound fields. See #10809 and #11190. - It is now possible to edit a custom role with the same alias. See #8808 and #10612. - The [Metadata Customization](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties) documentation has been updated to explain how to implement a boolean fieldtype (look for "boolean"). See #7961 and #11064. - The version of Stata files is now detected during S3 direct upload (as it was for normal uploads), allowing ingest of Stata 14 and 15 files that have been uploaded directly. See [the guides](https://guides.dataverse.org/en/6.6/developers/big-data-support.html#features-that-are-disabled-if-s3-direct-upload-is-enabled) #10108, and #11054. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. -- Support for legacy configuration of a PermaLink PID provider (e.g. using the :Protocol,:Authority, and :Shoulder settings) has been fixed. See #10516 and #10521. +- Support for legacy configuration of a PermaLink PID provider, such as using the :Protocol,:Authority, and :Shoulder settings, has been fixed. See #10516 and #10521. - On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. - For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://guides.dataverse.org/en/6.6/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320. - A new file has been added to import the French Open License to Dataverse: licenseEtalab-2.0.json. You can download it from [the guides](http://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses). This license, which is compatible with the Creative Commons license, is recommended by the French government for open documents. See #9301, #9302, and #11302. -- The API that lists versions of a dataset now features an optional `excludeMetadataBlocks` parameter, which defaults to "false" for backward compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks included can dramatically increase the volume of the output). See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#list-versions-of-a-dataset), #10171, and #10778. +- The API that lists versions of a dataset now features an optional `excludeMetadataBlocks` parameter, which defaults to "false" for backward compatibility. For a dataset with a large number of versions and/or metadataBlocks, having the metadata blocks included can dramatically increase the volume of the output. See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#list-versions-of-a-dataset), #10171, and #10778. - Deeply nested metadata fields are not supported but the code used to generate the Solr schema has been adjusted to support them. See #11136. - The [tutorial](https://guides.dataverse.org/en/6.6/container/running/demo.html) on running Dataverse in Docker has been updated to explain how to configure the root collection using a JSON file (#10541 and #11201) and now uses the Permalink PID provider instead of the FAKE DOI Provider (#11107 and #11108). - Payara application server has been upgraded to version 6.2025.2. See #11126 and #11128. @@ -208,18 +208,18 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en ## Bugs Fixed - A bug which causes users of the Anonymous Review URL to have some metadata of published datasets withheld has been fixed. See #11202 and #11164. -- A bug that caused ORCIDs starting with "https://orcid.org/" entered as author identifier to be ignored when creating the DataCite metadata has been fixed. This primarily affected users of the ORCID external vocabulary script as the manual entry form recommends not using the URL form. The display of authorIdentifier, when not using any external vocabulary scripts, has been improved so that either the plain identifier (e.g. "0000-0002-1825-0097") or its URL form (e.g. "https://orcid.org/0000-0002-1825-0097") will result in valid links in the display (for identifier types that have a URL form). The URL form is now [recommended](http://guides.dataverse.org/en/6.6/user/dataset-management.html#adding-a-new-dataset) when doing manual entry. See #11242 and #11242. +- A bug that caused ORCIDs starting with "https://orcid.org/" entered as author identifier to be ignored when creating the DataCite metadata has been fixed. This primarily affected users of the ORCID external vocabulary script; for the manual entry form, we used to recommend not using the URL form. The display of authorIdentifier, when not using any external vocabulary scripts, has been improved so that either the plain identifier (e.g. "0000-0002-1825-0097") or its URL form (e.g. "https://orcid.org/0000-0002-1825-0097") will result in valid links in the display (for identifier types that have a URL form). The URL form is now [recommended](http://guides.dataverse.org/en/6.6/user/dataset-management.html#adding-a-new-dataset) when doing manual entry. See #11242 and #11242. - Multiple small issues with the formatting of PIDs in the DDI exporters, and EndNote and BibTeX citation formats have been addressed. These should improve the ability to import Dataverse citations into reference managers and fix potential issues harvesting datasets using PermaLinks. See #10768, #10769, #11165, and #10790. - On the Advanced Search page, the metadata fields are now displayed in the correct order as defined in the TSV file via the displayOrder value, making the order the same as when you view or edit metadata. Note that fields that are not defined in the TSV file, like the "Persistent ID" and "Publication Date", will be displayed at the end. See #11272 and #11279. -- Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this only affects datasets where guestbooks have been configured. See #11203. +- Bugs that caused 1) guestbook questions to appear along with terms of use/terms of access in the request access dialog when no guestbook was configured, and 2) terms of access to not be shown when using the per-file request access/download menu items have been fixed. Text related to configuring the choice to have guestbooks appear when file access is requested or when files are downloaded has been updated to make it clearer that this affects only datasets where guestbooks have been configured. See #11203. - The file page version table now shows whether a file has been replaced. See #11142 and #11145. -- We fixed an issue where draft versions of datasets were sorted using the release timestamp of their most recent major version. This caused newer drafts to appear incorrectly alongside their corresponding major version, instead of at the top, when sorted by "newest first". Sorting now uses the last update timestamp when sorting draft datasets. The sorting behavior of published dataset versions (major and minor) is unchanged. There is no need to reindex datasets because Solr is being upgraded (see "Upgrade Instructions"), which will result in an empty database that will be reindexed. See #11178. -- Some external controlled vocabulary scripts/configurations, when used on a metadata field that is single-valued, could result in indexing failure for the dataset (e.g. when the script tried to index both the identifier and name of the identified entity for indexing). Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to `/api/admin/index/solr/schema`. Configuring the Solr schema and running the update-fields.sh script as usually recommended when using custom metadata blocks (see "Upgrade Instructions") will resolve the issue. See [the guides](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#using-external-vocabulary-services), #11095, and #11096. +- We fixed an issue where draft versions of datasets were sorted using the release timestamp of their most recent major version. This caused newer drafts to appear incorrectly alongside their corresponding major version, instead of at the top, when sorted by "newest first". Sorting now uses the last update timestamp when sorting draft datasets. The sorting behavior of published major and minor dataset versions is unchanged. There is no need to reindex datasets because Solr is being upgraded (see "Upgrade Instructions"), which will result in an empty database that will be reindexed. See #11178. +- Some external controlled vocabulary scripts/configurations, when used on a metadata field that is single-valued, could result in indexing failure for the dataset, e.g. when the script tried to index both the identifier and name of the identified entity for indexing. Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to `/api/admin/index/solr/schema`. Configuring the Solr schema and running the update-fields.sh script as usually recommended when using custom metadata blocks (see "Upgrade Instructions") will resolve the issue. See [the guides](https://guides.dataverse.org/en/6.6/admin/metadatacustomization.html#using-external-vocabulary-services), #11095, and #11096. - The OpenAIRE metadata export format can now correctly process one or multiple productionPlaces as geolocation. See #9546 and #11194 -- A bug that caused adding free-form provenance to a file to fail has been fixed. See #11145. +- We fixed a bug that caused adding free-form provenance to a file to fail. See #11145. - A bug has been fixed which could cause publication of datasets to fail in cases where they were not assigned a DOI at creation. See #11234 and #11236. -- When users request access to files, the people who have permission to grant access received an email with a link in it that didn't work due to a trailing period (full stop) right next to the link (e.g. `https://demo.dataverse.org/permissions-manage-files.xhtml?id=9.`) A space has been added to fix this. See #10384 and #11115. -- Harvesting clients now use the correct granularity while re-running a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request. See #11020 and #11038. +- When users request access to files, the people who have permission to grant access received an email with a link that didn't work due to a trailing period (full stop) right next to the link, e.g. `https://demo.dataverse.org/permissions-manage-files.xhtml?id=9.` A space has been added to fix this. See #10384 and #11115. +- Harvesting clients now use the correct granularity while re-running a partial harvest, using the `from` parameter. The correct granularity comes from the `Identify` verb request. See #11020 and #11038. - Access requests were missing on the File Permission page after upgrading from Dataverse 6.0. This has been corrected with a database update script. See #10714 and #11061. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. - The `/api/info/metrics/files/monthly` API call had a bug that resulted in files being counted each time they were published in a new version if those publication events occurred in different months. This resulted in an over-count. The `/api/info/metrics/files` and `/api/info/metrics/files/toMonth` API calls had a bug that resulted in files that were published but no longer in the latest published version as of the specified date (now, or the date entered in the `/toMonth` variant). This resulted in an under-count. See 11189. @@ -227,7 +227,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - A bug that caused replacing files via API when file PIDs were enabled to fail has been fixed. See #10975 and #10979. - The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. - Dataverse 6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the `AVOID_EXPENSIVE_SOLR_JOIN` feature flag is set to true. This release fixes the bug. See #11133 and #11134. -- Fix a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. +- We fixed a bug with My Data where listing collections for a user with only rights on harvested collections would result in a server error response. See #11083. - Minor styling fixes for the Related Publication field and fields using ORCID or ROR have been made. See #11053, #10964, and #11106. - In the Search API, files were displaying DRAFT version instead of latest released version under `dataset_citation`. See #10735 and #11051. - Unnecessary Solr documents were being created when a file was added or deleted from a draft dataset. These documents could accumulate and potentially impact performance. There is no action to take because this release includes a new Solr version, which will start with an empty database. See #11113 and #11114. @@ -242,7 +242,7 @@ Added new fields to search results type=files For Files: - restricted: boolean -- canDownloadFile: boolean ( from file user permission) +- canDownloadFile: boolean (from file user permission) - categories: array of string "categories" would be similar to what it is in metadata api. For tabular files: @@ -274,7 +274,7 @@ See also the "Settings Added" section, #10943 and #11124. - An API has been added to get dataset versions including a summary of differences between consecutive versions where available. See [the docs](https://guides.dataverse.org/en/6.6/api/native-api.html#get-versions-of-a-dataset-with-summary-of-changes ), #10888, and #10945. - An API has been added to list of versions of a data file showing any changes that affected the file with each version. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#get-json-representation-of-a-file-s-versions), #11198 and #11237. - The Search API has a new [parameter](https://guides.dataverse.org/en/6.6/api/search.html#parameters) called `show_type_counts`. If you set it to true, it will return `total_count_per_object_type` for the types dataverse, dataset, and files (#11065 and #11082) even if the search result for any given type is 0 (#11127 and #11138). -- CRUD operations for external tools are now available (for superusers) from non-localhost. See [the guides](https://guides.dataverse.org/en/6.6/admin/external-tools.html#managing-external-tools), #10930 and #11079. +- CRUD operations for external tools are now available for superusers from non-localhost. See [the guides](https://guides.dataverse.org/en/6.6/admin/external-tools.html#managing-external-tools), #10930 and #11079. - A new API endpoint has been added that allows a global role to be updated. See [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#update-global-role) and #10612. - An API has been added to send feedback to the collection, dataset, or data file's contacts. If necessary, you can [rate limit](https://guides.dataverse.org/en/6.6/installation/config.html#rate-limiting) the `CheckRateLimitForDatasetFeedbackCommand` and configure the new [:ContactFeedbackMessageSizeLimit](https://guides.dataverse.org/en/6.6/installation/config.html#contactfeedbackmessagesizelimit) database setting. See [the guides](http://guides.dataverse.org/en/6.6/api/native-api.html#send-feedback-to-contact-s), #11129, and #11162. - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See "Backward Incompatible Changes" below and #10764. From 42f2e3c966d131ad189df3fcd6a9f57f468c843e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 16:09:11 -0400 Subject: [PATCH 60/71] fix ORCIDs --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 93b555e62ae..f7243f9c889 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,7 +8,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: -- ORCIDS linked to accounts +- ORCIDs linked to accounts - version notes - harvesting from DataCite - citations using Citation Style Language (CSL) From 79744f5f64593b8e2c3b1accbcfefca609c25141 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 16:12:28 -0400 Subject: [PATCH 61/71] mention video talking about highlights --- doc/release-notes/6.6-release-notes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index f7243f9c889..2ea18182fcf 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -22,6 +22,8 @@ Highlights for Dataverse 6.6 include: - Signposting output now contains links to all dataset metadata export formats - infrastructure updates (Payara and Solr) +In a recent community call, we talked about many of these highlights if you'd like to watch the [video](https://harvard.zoom.us/rec/share/Ir5CkFHkzoya9b5Nk69rLFUpTyGics3-KGLl9WITSLMy4ezHRsB8CnY22cUNg2g.JPpxrjzHMeCii_zO) (around 22:30). + ## Features Added ### ORCIDs Linked to Accounts From 55c8e59fa8d9d64c57e8a5da5fa1283303ef5b6a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 12 Mar 2025 17:00:35 -0400 Subject: [PATCH 62/71] tweaks --- doc/release-notes/6.6-release-notes.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 2ea18182fcf..53a87c0dcff 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -57,11 +57,12 @@ For various reasons, in order to take advantage of this feature harvesting clien DataCite-harvesting clients must be configured with 2 new feature flags, `useListRecords` and `useOaiIdentifiersAsPids` (added in Dataverse 6.5). Note that these features may be of use when harvesting from other sources, not just from DataCite. See the [Admin Guide](http://guides.dataverse.org/en/6.6/admin/harvestclients.html#harvesting-from-datacite), [API Guide](http://guides.dataverse.org/en/6.6/api/native-api.html#harvesting-from-datacite), #10909, and #11011. -### Citations using Citation Style Language (CSL) + +### Citations Using Citation Style Language (CSL) This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org). -The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up on the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL stypes citations. +The CSL formats are available to copy/paste if you click "Cite Dataset" and then "View Styled Citations" on the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. The first three have been available as downloads from the UI (CSLJson is not) but have not been directly accessible via API until now. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL stypes citations. Admins can use a new `dataverse.csl.common-styles` setting to highlight commonly used styles. Common styles are listed in the pop-up, while others can be found by type-ahead search in a list of 1000+ options. @@ -123,7 +124,7 @@ All new Dataverse installations will receive this metadata block by default. We ### New Archival Metadata Block (Experimental) -An experimental "Archival" metadata block has been added, [downloadable](https://guides.dataverse.org/en/6.6/user/appendix.html) from the User Guide. The purpose of the metadata block is to enable repositories to register metadata relating to the potential archiving of the dataset at a depositor archive, whether that being your own institutional archive or an external archive, i.e. a historical archive. See also #10626. +An experimental "Archival" metadata block has been added, [downloadable](https://guides.dataverse.org/en/6.6/user/appendix.html) from the User Guide. The purpose of the metadata block is to enable repositories to register metadata relating to the potential archiving of the dataset at a depositor archive, whether that being your own institutional archive or an external archive, i.e. a historical archive. Feedback is welcome! See also #10626. ### Prevent Publishing of Datasets Without Files @@ -135,7 +136,7 @@ The behavior of the feature flag `index-harvested-metadata-source` and the "Meta Before Dataverse 6.3, all harvested content (datasets and files) appeared together under "Harvested" under the "Metadata Source" facet. This is still the behavior of Dataverse out of the box. Since Dataverse 6.3, enabling the `index-harvested-metadata-source` feature flag (and reindexing) resulted in harvested content appearing under the nickname for whatever harvesting client was used to bring in the content. This meant that instead of having all harvested content lumped together under "Harvested", content would appear under "client1", "client2", etc. -With this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to reuse the same source name to group harvested content from various harvesting clients under the same name if you wish. +With this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below) results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to reuse the same source name to group harvested content from various harvesting clients under the same name if you wish. Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://guides.dataverse.org/en/6.6/installation/config.html#feature-flags). See also #10217 and #11217. From df070b5c9176e73e547b21f72cc7620e93d92a11 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Thu, 13 Mar 2025 09:10:58 -0400 Subject: [PATCH 63/71] #11183 fix typo --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 53a87c0dcff..d5509313399 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -370,7 +370,7 @@ $PAYARA/bin/asadmin list-applications 2\. Undeploy the previous version (should match "list-applications" above) ```shell -$PAYARA/bin/asadmin undeploy dataverse-6.4 +$PAYARA/bin/asadmin undeploy dataverse-6.5 ``` 3\. Stop Payara From bf46b9f1706c682cc236254d1af001e717f6a514 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 13 Mar 2025 09:13:31 -0400 Subject: [PATCH 64/71] fix link to war file --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index d5509313399..31f9b0d33d2 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -409,7 +409,7 @@ mv payara6.6.2024.6/glassfish/domains/domain1 payara6/glassfish/domains/ 5\. Download and deploy this version ```shell -wget https://github.com/IQSS/dataverse/releases/download/v6.5/dataverse-6.6.war +wget https://github.com/IQSS/dataverse/releases/download/v6.6/dataverse-6.6.war $PAYARA/bin/asadmin deploy dataverse-6.6.war ``` From 388e3a9c19bcbaaf811c132382f023e741bffe36 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 13 Mar 2025 09:14:58 -0400 Subject: [PATCH 65/71] Update doc/release-notes/6.6-release-notes.md Co-authored-by: Omer Fahim --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 31f9b0d33d2..05499a0f870 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -323,7 +323,7 @@ Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/late - /api/metadatablocks is no longer returning duplicated metadata properties and does not omit metadata properties when called. See #10764. - The JSON response of API call `/api/dataverses/{dataverseAlias}/metadatablocks` will no longer include the DatasetFieldTypes in "fields" if they are children of another DatasetFieldType. The child DatasetFieldType will only be included in the "childFields" of its parent DatasetFieldType. See #10472 and #11066. -- `versionNote` and been renamed to `deaccessionNote`. `archiveNote` have been renamed to `deaccessionLink`. See #11068. +- `versionNote` has been renamed to `deaccessionNote`. `archiveNote` has been renamed to `deaccessionLink`. See #11068. - The [Show Role](https://guides.dataverse.org/en/6.6/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://guides.dataverse.org/en/6.6/11116/api/changelog.html), #10340, and #11116. - Changes to PID formatting occur in the DDI/DDI Html export formats and the EndNote and BibTex citation formats. These changes correct errors and improve conformance with best practices but could break parsing of these formats. See #10768, #10769, #11165, and #10790. From f5fbbe3122b59543aed7ecf2317bbafc6270e784 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 13 Mar 2025 09:15:18 -0400 Subject: [PATCH 66/71] Update doc/release-notes/6.6-release-notes.md Co-authored-by: Omer Fahim --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 05499a0f870..8e05dc53c72 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -305,7 +305,7 @@ New Dataverse installations now explicitly set to the SameSite cookie attribute While you are welcome to experiment with "Strict", which is intended to help prevent Cross-Site Request Forgery (CSRF) attacks, as described in the RFC proposal and an OWASP [cheetsheet](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#samesite-cookie-attribute), our testing so far indicates that some functionality, such as OIDC login, seems to be incompatible with "Strict". -You should avoid the use of "None" as it is less secure that "Lax". See also [the guides](https://guides.dataverse.org/en/6.6/installation/config.html#samesite-cookie-attribute), https://github.com/IQSS/dataverse-security/issues/27, #11210, and the upgrade instructions below. +You should avoid the use of "None" as it is less secure than "Lax". See also [the guides](https://guides.dataverse.org/en/6.6/installation/config.html#samesite-cookie-attribute), https://github.com/IQSS/dataverse-security/issues/27, #11210, and the upgrade instructions below. ## Settings Added From 84710b5dac5fbe1e36bafa5fcf8299f959070408 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 13 Mar 2025 09:15:41 -0400 Subject: [PATCH 67/71] typo: cheat Co-authored-by: Omer Fahim --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 8e05dc53c72..27c6727db26 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -303,7 +303,7 @@ The SameSite cookie attribute is defined in an upcoming revision to [RFC 6265](h New Dataverse installations now explicitly set to the SameSite cookie attribute to "Lax" out of the box through the installer (in the case of a "classic" installation) or through an updated base image (in the case of a Docker installation). Classic installations should follow the upgrade instructions below to bring their installation up to date with the behavior for new installations. Docker installations will automatically get the updated base image. -While you are welcome to experiment with "Strict", which is intended to help prevent Cross-Site Request Forgery (CSRF) attacks, as described in the RFC proposal and an OWASP [cheetsheet](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#samesite-cookie-attribute), our testing so far indicates that some functionality, such as OIDC login, seems to be incompatible with "Strict". +While you are welcome to experiment with "Strict", which is intended to help prevent Cross-Site Request Forgery (CSRF) attacks, as described in the RFC proposal and an OWASP [cheatsheet](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#samesite-cookie-attribute), our testing so far indicates that some functionality, such as OIDC login, seems to be incompatible with "Strict". You should avoid the use of "None" as it is less secure than "Lax". See also [the guides](https://guides.dataverse.org/en/6.6/installation/config.html#samesite-cookie-attribute), https://github.com/IQSS/dataverse-security/issues/27, #11210, and the upgrade instructions below. From 9453f1de6576e05bf3de55697d9fc3ab94b82fe4 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 13 Mar 2025 09:18:59 -0400 Subject: [PATCH 68/71] fix link to issue --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 27c6727db26..054cfab9ad3 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -225,7 +225,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - Harvesting clients now use the correct granularity while re-running a partial harvest, using the `from` parameter. The correct granularity comes from the `Identify` verb request. See #11020 and #11038. - Access requests were missing on the File Permission page after upgrading from Dataverse 6.0. This has been corrected with a database update script. See #10714 and #11061. - When a dataset has a long running lock, including when it is "in review", Dataverse will now slow the page refresh rate over time. See #11264 and #11269. -- The `/api/info/metrics/files/monthly` API call had a bug that resulted in files being counted each time they were published in a new version if those publication events occurred in different months. This resulted in an over-count. The `/api/info/metrics/files` and `/api/info/metrics/files/toMonth` API calls had a bug that resulted in files that were published but no longer in the latest published version as of the specified date (now, or the date entered in the `/toMonth` variant). This resulted in an under-count. See 11189. +- The `/api/info/metrics/files/monthly` API call had a bug that resulted in files being counted each time they were published in a new version if those publication events occurred in different months. This resulted in an over-count. The `/api/info/metrics/files` and `/api/info/metrics/files/toMonth` API calls had a bug that resulted in files that were published but no longer in the latest published version as of the specified date (now, or the date entered in the `/toMonth` variant). This resulted in an under-count. See #11189. - DatasetFieldTypes in MetadataBlock response that are also a child of another DatasetFieldType were being returned twice. The child DatasetFieldType was included in the "fields" object as well as in the "childFields" of its parent DatasetFieldType. This fix suppresses the standalone object so only one instance of the DatasetFieldType is returned (in the "childFields" of its parent). This fix changes the JSON output of the API `/api/dataverses/{dataverseAlias}/metadatablocks` (see "Backward Incompatible Changes", below). See #10472 and #11066. - A bug that caused replacing files via API when file PIDs were enabled to fail has been fixed. See #10975 and #10979. - The [:CustomDatasetSummaryFields](https://guides.dataverse.org/en/6.6/installation/config.html#customdatasetsummaryfields) setting now allows spaces along with a comma separating field names. In addition, a bug that caused license information to be hidden if there are no values for any of the custom fields specified has been fixed. See #11228 and #11229. From 6ce47eb58344d4149bc0151d97807afaa839167a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 13 Mar 2025 09:21:01 -0400 Subject: [PATCH 69/71] typos --- doc/release-notes/6.6-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index 054cfab9ad3..e0214a006e2 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -30,7 +30,7 @@ In a recent community call, we talked about many of these highlights if you'd li Dataverse now includes improved integration with ORCID, supported through a grant to GDCC from the ([ORCID Global Participation Fund](https://info.orcid.org/global-participation-fund-announces-fourth-round-of-awardees/)). -Specifically, Dataverse users can now link their Dataverse account with their ORCID profile. Previously, this was only availabe to users who logged in with ORCID. Once linked, Dataverse will automatically prepopulate their ORCID to their author metadata when they create a dataset. +Specifically, Dataverse users can now link their Dataverse account with their ORCID profile. Previously, this was only available to users who logged in with ORCID. Once linked, Dataverse will automatically prepopulate their ORCID to their author metadata when they create a dataset. This functionality leverages Dataverse's existing support for login via ORCID, but can be turned on independently of it. If ORCID login is enabled, the user's ORCID will automatically be added to their profile. If the user has logged in via some other mechanism, they are able to click a button to initiate a similar authentication process in which the user must login to their ORCID account and approve the connection. @@ -62,7 +62,7 @@ See the [Admin Guide](http://guides.dataverse.org/en/6.6/admin/harvestclients.ht This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org). -The CSL formats are available to copy/paste if you click "Cite Dataset" and then "View Styled Citations" on the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. The first three have been available as downloads from the UI (CSLJson is not) but have not been directly accessible via API until now. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL stypes citations. +The CSL formats are available to copy/paste if you click "Cite Dataset" and then "View Styled Citations" on the dataset page. An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. The first three have been available as downloads from the UI (CSLJson is not) but have not been directly accessible via API until now. The CSLJson format is new to Dataverse and can be used with open source libraries to generate all of the other CSL-style citations. Admins can use a new `dataverse.csl.common-styles` setting to highlight commonly used styles. Common styles are listed in the pop-up, while others can be found by type-ahead search in a list of 1000+ options. From 10b6532aacaf02f89ff68f37217176bb01059246 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 14 Mar 2025 07:56:54 -0400 Subject: [PATCH 70/71] display on create --- doc/release-notes/10476-display-on-create-field-option.md | 6 ------ doc/release-notes/6.6-release-notes.md | 7 +++++++ 2 files changed, 7 insertions(+), 6 deletions(-) delete mode 100644 doc/release-notes/10476-display-on-create-field-option.md diff --git a/doc/release-notes/10476-display-on-create-field-option.md b/doc/release-notes/10476-display-on-create-field-option.md deleted file mode 100644 index e4a38e181a2..00000000000 --- a/doc/release-notes/10476-display-on-create-field-option.md +++ /dev/null @@ -1,6 +0,0 @@ -New feature: Collection administrators can now configure which metadata fields appear during dataset creation through the `displayOnCreate` property, even when fields are not required. This provides greater control over metadata visibility and can help improve metadata completeness. - -- The feature is currently available through the API endpoint `/api/dataverses/{alias}/inputLevels` -- UI implementation will be available in a future release [#11221](https://github.com/IQSS/dataverse/issues/11221) - -For more information, see the [API Guide](https://guides.dataverse.org/en/latest/api/native-api.html#update-collection-input-levels) and issues [#10476](https://github.com/IQSS/dataverse/issues/10476) and [#11224](https://github.com/IQSS/dataverse/pull/11224). \ No newline at end of file diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index e0214a006e2..b5e3e1ee57a 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -8,6 +8,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.6 include: +- metadata fields can be "display on create" per collection - ORCIDs linked to accounts - version notes - harvesting from DataCite @@ -26,6 +27,12 @@ In a recent community call, we talked about many of these highlights if you'd li ## Features Added +### Metadata Fields Can Be "Display on Create" Per Collection + +Collection administrators can now configure which metadata fields appear during dataset creation through the `displayOnCreate` property, even when fields are not required. This provides greater control over metadata visibility and can help improve metadata completeness. + +Currently this feature can only be configured [via API](https://guides.dataverse.org/en/6.6/api/native-api.html#update-collection-input-levels), but a UI implementation is planned in #11221. See #10476, #11224, and #11312. + ### ORCIDs Linked to Accounts Dataverse now includes improved integration with ORCID, supported through a grant to GDCC from the ([ORCID Global Participation Fund](https://info.orcid.org/global-participation-fund-announces-fourth-round-of-awardees/)). From 85fe341dd871ea8917c89ff0faede850ecaa9f0a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 14 Mar 2025 09:34:00 -0400 Subject: [PATCH 71/71] typo Co-authored-by: Omer Fahim --- doc/release-notes/6.6-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.6-release-notes.md b/doc/release-notes/6.6-release-notes.md index b5e3e1ee57a..751c471a0a3 100644 --- a/doc/release-notes/6.6-release-notes.md +++ b/doc/release-notes/6.6-release-notes.md @@ -205,7 +205,7 @@ For more information, see the guides ([overview](https://guides.dataverse.org/en - The version of Stata files is now detected during S3 direct upload (as it was for normal uploads), allowing ingest of Stata 14 and 15 files that have been uploaded directly. See [the guides](https://guides.dataverse.org/en/6.6/developers/big-data-support.html#features-that-are-disabled-if-s3-direct-upload-is-enabled) #10108, and #11054. - It is now possible to populate the "Keyword" metadata field from an [OntoPortal](https://ontoportal.org) service. The code has been shared to the GDCC [dataverse-external-vocab-support](https://github.com/gdcc/dataverse-external-vocab-support#scripts-in-production) GitHub repository. See #11258. - Support for legacy configuration of a PermaLink PID provider, such as using the :Protocol,:Authority, and :Shoulder settings, has been fixed. See #10516 and #10521. -- On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested tabled of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. +- On the home page for each guide (User Guide, etc.) there was an overwhelming amount of information in the form of a deeply nested table of contents. The depth of the table of contents has been reduced to two levels, making the home page for each guide more readable. Compare the User Guide for [6.5](https://guides.dataverse.org/en/6.5/user/index.html) vs. [6.6](https://guides.dataverse.org/en/6.6/user/index.html) and see #11166. - For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://guides.dataverse.org/en/6.6/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320. - A new file has been added to import the French Open License to Dataverse: licenseEtalab-2.0.json. You can download it from [the guides](http://guides.dataverse.org/en/6.6/installation/config.html#adding-licenses). This license, which is compatible with the Creative Commons license, is recommended by the French government for open documents. See #9301, #9302, and #11302. - The API that lists versions of a dataset now features an optional `excludeMetadataBlocks` parameter, which defaults to "false" for backward compatibility. For a dataset with a large number of versions and/or metadataBlocks, having the metadata blocks included can dramatically increase the volume of the output. See also [the guides](https://guides.dataverse.org/en/6.6/api/native-api.html#list-versions-of-a-dataset), #10171, and #10778.