Pubmed api key support (#13899)

Siedlerchr · calixtus · koppor · web-flow · commit d714281cb11b · 2025-09-14T15:57:09.000Z
* fix git modules requires and uses

* add pubmed api url

* test

* Support Medline APi Key

* Support Medline APi Key

* Support Medline APi Key

* add default value and env variable

* checkstyle

* Typo

* remove

* add to fetchers

* remove binaries

* fix

* Fix typo in decision document title

* add a space

---------

Co-authored-by: Carl Christian Snethlage &lt;50491877+calixtus@users.noreply.github.com&gt;
Co-authored-by: Oliver Kopp &lt;kopp.dev@gmail.com&gt;
Co-authored-by: Subhramit Basu &lt;subhramit.bb@live.in&gt;
diff --git a/.github/workflows/tests-code-fetchers.yml b/.github/workflows/tests-code-fetchers.yml
@@ -32,7 +32,8 @@ env:
   SpringerNatureAPIKey: ${{ secrets.SPRINGERNATUREAPIKEY_FOR_TESTS }}
   AstrophysicsDataSystemAPIKey: ${{ secrets.AstrophysicsDataSystemAPIKey_FOR_TESTS }}
   IEEEAPIKey: ${{ secrets.IEEEAPIKey_FOR_TESTS }}
-  BiodiversityHeritageApiKey: ${{ secrets.BiodiversityHeritageApiKey_FOR_TESTS}}
+  BiodiversityHeritageApiKey: ${{ secrets.BiodiversityHeritageApiKey_FOR_TESTS }}
+  MedlineApiKey: ${{ secrets.MedlineApiKey_FOR_TESTS }}
 
 concurrency:
   group: "${{ github.workflow }}-${{ github.head_ref || github.ref }}"
diff --git a/.github/workflows/tests-code.yml b/.github/workflows/tests-code.yml
@@ -14,6 +14,7 @@ env:
   BiodiversityHeritageApiKey: ${{ secrets.BiodiversityHeritageApiKey_FOR_TESTS}}
   IEEEAPIKey: ${{ secrets.IEEEAPIKey_FOR_TESTS }}
   SpringerNatureAPIKey: ${{ secrets.SPRINGERNATUREAPIKEY_FOR_TESTS }}
+  MedlineAPiKey: ${{ secrets.MedlineApiKey_FOR_TESTS }}
   GRADLE_OPTS: -Xmx4g
   JAVA_OPTS: -Xmx4g
 
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -40,6 +40,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
 - We added BibLaTeX datamodel validation support in order to improve error message quality in entries' fields validation. [#13318](https://github.com/JabRef/jabref/issues/13318)
 - We added more supported formats of CAYW endpoint of HTTP server. [#13578](https://github.com/JabRef/jabref/issues/13578)
 - We added chronological navigation for entries in each library. [#6352](https://github.com/JabRef/jabref/issues/6352)
+- We added support for using Medline/Pubmed fetcher with an API key. [#11296](https://github.com/JabRef/jabref/issues/11296#issuecomment-3289005011)
 
 ### Changed
 
diff --git a/docs/code-howtos/fetchers.md b/docs/code-howtos/fetchers.md
@@ -7,14 +7,15 @@ Fetchers are the implementation of the [search using online services](https://do
 
 | Service                                                                                                                                           | Key Source                                                                                                     | Environment Variable           | Rate Limit                       |
 |:--------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|--------------------------------|----------------------------------|
+| [Biodiversity Heritage Library](https://www.biodiversitylibrary.org/)                                                                             | [Biodiversitylibrary](https://about.biodiversitylibrary.org/tools-and-services/developer-and-data-tools/#APIs) | `BiodiversityHeritageApiKey`   | -                                |
 | [IEEEXplore](https://docs.jabref.org/collect/import-using-online-bibliographic-database#ieeexplore)                                               | [IEEE Xplore API portal](https://developer.ieee.org)                                                           | `IEEEAPIKey`                   | 200 calls/day                    |
+| [Medline/Pubmed](https://pubmed.ncbi.nlm.nih.gov/)                                                                                                | [NCBI User account](https://account.ncbi.nlm.nih.gov/settings/)                                                | `medlineApiKey`                | 10 requests/seconds              |
 | [MathSciNet](http://www.ams.org/mathscinet)                                                                                                       | (none)                                                                                                         | (none)                         | Depending on the current network |
 | [SAO/NASA Astrophysics Data System](https://docs.jabref.org/collect/import-using-online-bibliographic-database#sao-nasa-astrophysics-data-system) | [ADS UI](https://ui.adsabs.harvard.edu/user/settings/token)                                                    | `AstrophysicsDataSystemAPIKey` | 5000 calls/day                   |
 | [ScienceDirect](https://www.sciencedirect.com)                                                                                                    |                                                                                                                | `ScienceDirectApiKey`          |                                  |
 | [SemanticScholar](https://www.semanticscholar.org/)                                                                                               | <https://www.semanticscholar.org/product/api#api-key-form>                                                     | `SemanticScholarApiKey`        |                                  |
-| [Springer Nature](https://docs.jabref.org/collect/import-using-online-bibliographic-database#springer)                                            | [Springer Nature API Portal](https://dev.springernature.com). Use the "Meta API" API key.                      | `SpringerNatureAPIKey`         | 5000 calls/day                   |
+| [Springer Nature](https://docs.jabref.org/collect/import-using-online-bibliographic-database#springer)                                            | [Springer Nature API portal](https://dev.springernature.com). Use the "Meta API" API key.                      | `SpringerNatureAPIKey`         | 5000 calls/day                   |
 | [Zentralblatt Math](https://www.zbmath.org)                                                                                                       | (none)                                                                                                         | (none)                         | Depending on the current network |
-| [Biodiversity Heritage Library](https://www.biodiversitylibrary.org/)                                                                             | [Biodiversitylibrary](https://about.biodiversitylibrary.org/tools-and-services/developer-and-data-tools/#APIs) | `BiodiversityHeritageApiKey`   | -                                |
 
 "Depending on the current network" means that it depends on whether your request is routed through a network having paid access. For instance, some universities have subscriptions to MathSciNet.
 
diff --git a/docs/decisions/0012-handle-different-bibEntry-formats-of-fetchers.md b/docs/decisions/0012-handle-different-bibEntry-formats-of-fetchers.md
@@ -10,7 +10,7 @@ All fetchers (except IDFetchers) in JabRef return BibEntries when fetching entri
 Some fetchers directly receive BibTeX entries from their API, the other fetchers receive their entries in some kind of exchange format such as JSON or XML and then parse this into BibEntries.
 Currently, all fetchers either return BibEntries in BibTeX or BibLaTeX format.
 This can lead to importing BibEntries of one format in a database of the other format.
-How can this inconsistency between fetchers, and their used formats be addressed?
+How can this inconsistency between fetchers and their used formats be addressed?
 
 ## Considered Options
 
diff --git a/jablib/build.gradle.kts b/jablib/build.gradle.kts
@@ -352,6 +352,7 @@ val ieeeAPIKey = providers.environmentVariable("IEEEAPIKey").orElse("")
 val scienceDirectApiKey = providers.environmentVariable("SCIENCEDIRECTAPIKEY").orElse("")
 val biodiversityHeritageApiKey = providers.environmentVariable("BiodiversityHeritageApiKey").orElse("")
 val semanticScholarApiKey = providers.environmentVariable("SemanticScholarApiKey").orElse("")
+val medlineApiKey = providers.environmentVariable("MedlineApiKey").orElse("")
 
 tasks.named<ProcessResources>("processResources") {
     dependsOn(extractMaintainers)
@@ -370,6 +371,7 @@ tasks.named<ProcessResources>("processResources") {
     inputs.property("scienceDirectApiKey", scienceDirectApiKey)
     inputs.property("biodiversityHeritageApiKey", biodiversityHeritageApiKey)
     inputs.property("semanticScholarApiKey", semanticScholarApiKey)
+    inputs.property("medlineApiKey", medlineApiKey)
 
     filesMatching("build.properties") {
         expand(
@@ -383,7 +385,8 @@ tasks.named<ProcessResources>("processResources") {
                 "ieeeAPIKey" to inputs.properties["ieeeAPIKey"],
                 "scienceDirectApiKey" to inputs.properties["scienceDirectApiKey"],
                 "biodiversityHeritageApiKey" to inputs.properties["biodiversityHeritageApiKey"],
-                "semanticScholarApiKey" to inputs.properties["semanticScholarApiKey"]
+                "semanticScholarApiKey" to inputs.properties["semanticScholarApiKey"],
+                "medlineApiKey" to inputs.properties["medlineApiKey"]
             )
         )
     }
diff --git a/jablib/src/main/java/org/jabref/logic/importer/WebFetchers.java b/jablib/src/main/java/org/jabref/logic/importer/WebFetchers.java
@@ -112,7 +112,7 @@ public static SortedSet<SearchBasedFetcher> getSearchBasedFetchers(ImportFormatP
         set.add(new INSPIREFetcher(importFormatPreferences));
         set.add(new GvkFetcher(importFormatPreferences));
         set.add(new BvbFetcher());
-        set.add(new MedlineFetcher());
+        set.add(new MedlineFetcher(importerPreferences));
         set.add(new AstrophysicsDataSystem(importFormatPreferences, importerPreferences));
         set.add(new MathSciNet(importFormatPreferences));
         set.add(new ZbMATH(importFormatPreferences));
@@ -150,7 +150,7 @@ public static SortedSet<IdBasedFetcher> getIdBasedFetchers(ImportFormatPreferenc
         set.add(new DiVA(importFormatPreferences));
         set.add(new DoiFetcher(importFormatPreferences));
         set.add(new EuropePmcFetcher());
-        set.add(new MedlineFetcher());
+        set.add(new MedlineFetcher(importerPreferences));
         set.add(new TitleFetcher(importFormatPreferences));
         set.add(new MathSciNet(importFormatPreferences));
         set.add(new ZbMATH(importFormatPreferences));
@@ -237,6 +237,7 @@ public static Set<CustomizableKeyFetcher> getCustomizableKeyFetchers(ImportForma
         fetchers.add(new ScienceDirect(importerPreferences));
         fetchers.add(new AstrophysicsDataSystem(importFormatPreferences, importerPreferences));
         fetchers.add(new BiodiversityLibrary(importerPreferences));
+        fetchers.add(new MedlineFetcher(importerPreferences));
         return fetchers;
     }
 }
diff --git a/jablib/src/main/java/org/jabref/logic/importer/fetcher/MedlineFetcher.java b/jablib/src/main/java/org/jabref/logic/importer/fetcher/MedlineFetcher.java
@@ -24,6 +24,7 @@
 import org.jabref.logic.help.HelpFile;
 import org.jabref.logic.importer.FetcherException;
 import org.jabref.logic.importer.IdBasedParserFetcher;
+import org.jabref.logic.importer.ImporterPreferences;
 import org.jabref.logic.importer.Parser;
 import org.jabref.logic.importer.ParserResult;
 import org.jabref.logic.importer.SearchBasedFetcher;
@@ -44,15 +45,23 @@
  * The MedlineFetcher fetches the entries from the PubMed database.
  * See <a href="https://docs.jabref.org/collect/import-using-online-bibliographic-database#medline-pubmed">docs.jabref.org</a> for a detailed documentation of the available fields.
  */
-public class MedlineFetcher implements IdBasedParserFetcher, SearchBasedFetcher {
+public class MedlineFetcher implements IdBasedParserFetcher, SearchBasedFetcher, CustomizableKeyFetcher {
+    public static final String FETCHER_NAME = "Medline/PubMed";
+
     private static final Logger LOGGER = LoggerFactory.getLogger(MedlineFetcher.class);
 
     private static final int NUMBER_TO_FETCH = 50;
     private static final String ID_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi";
     private static final String SEARCH_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi";
+    private static final String TEST_URL_WITHOUT_API_KEY = "https://www.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?db=pubmed&api_key=";
 
+    private final ImporterPreferences importerPreferences;
     private int numberOfResultsFound;
 
+    public MedlineFetcher(ImporterPreferences importerPreferences) {
+        this.importerPreferences = importerPreferences;
+    }
+
     /**
      * When using 'esearch.fcgi?db=&lt;database>&term=&lt;query>' we will get a list of IDs matching the query.
      * Input: Any text query (&term)
@@ -130,6 +139,7 @@ public URL getUrlForIdentifier(String identifier) throws URISyntaxException, Mal
         uriBuilder.addParameter("db", "pubmed");
         uriBuilder.addParameter("retmode", "xml");
         uriBuilder.addParameter("id", identifier);
+        importerPreferences.getApiKey(FETCHER_NAME).ifPresent(apiKey -> uriBuilder.addParameter("api_key", apiKey));
         return uriBuilder.build().toURL();
     }
 
@@ -154,6 +164,7 @@ private URL createSearchUrl(String query) throws URISyntaxException, MalformedUR
         uriBuilder.addParameter("sort", "relevance");
         uriBuilder.addParameter("retmax", String.valueOf(NUMBER_TO_FETCH));
         uriBuilder.addParameter("term", query); // already lucene query
+        importerPreferences.getApiKey(FETCHER_NAME).ifPresent(apiKey -> uriBuilder.addParameter("api_key", apiKey));
         return uriBuilder.build().toURL();
     }
 
@@ -212,4 +223,9 @@ public List<BibEntry> performSearch(BaseQueryNode queryNode) throws FetcherExcep
             return entryList;
         }
     }
+
+    @Override
+    public String getTestUrl() {
+        return TEST_URL_WITHOUT_API_KEY;
+    }
 }
diff --git a/jablib/src/main/java/org/jabref/logic/util/BuildInfo.java b/jablib/src/main/java/org/jabref/logic/util/BuildInfo.java
@@ -43,6 +43,7 @@ public final class BuildInfo {
     public final String scienceDirectApiKey;
     public final String semanticScholarApiKey;
     public final String springerNatureAPIKey;
+    public final String medlineApiKey;
 
     public BuildInfo() {
         this("/build.properties");
@@ -71,6 +72,7 @@ public BuildInfo(String path) {
         scienceDirectApiKey = BuildInfo.getValue(properties, "scienceDirectApiKey", "fb82f2e692b3c72dafe5f4f1fa0ac00b");
         semanticScholarApiKey = BuildInfo.getValue(properties, "semanticScholarApiKey", "");
         springerNatureAPIKey = BuildInfo.getValue(properties, "springerNatureAPIKey", "118d90a519d0fc2a01ee9715400054d4");
+        medlineApiKey = BuildInfo.getValue(properties, "medlineApiKey", "");
     }
 
     private static String getValue(Properties properties, String key, String defaultValue) {
diff --git a/jablib/src/main/resources/build.properties b/jablib/src/main/resources/build.properties
@@ -7,3 +7,4 @@ astrophysicsDataSystemAPIKey=${astrophysicsDataSystemAPIKey}
 ieeeAPIKey=${ieeeAPIKey}
 biodiversityHeritageApiKey=${biodiversityHeritageApiKey}
 semanticScholarApiKey=${semanticScholarApiKey}
+medlineApiKey=${medlineApiKey}
diff --git a/jablib/src/test/java/org/jabref/logic/importer/fetcher/MedlineFetcherTest.java b/jablib/src/test/java/org/jabref/logic/importer/fetcher/MedlineFetcherTest.java
@@ -3,8 +3,12 @@
 import java.util.List;
 import java.util.Optional;
 
+import javafx.collections.FXCollections;
+
 import org.jabref.logic.importer.FetcherClientException;
 import org.jabref.logic.importer.FetcherException;
+import org.jabref.logic.importer.ImporterPreferences;
+import org.jabref.logic.util.BuildInfo;
 import org.jabref.model.entry.BibEntry;
 import org.jabref.model.entry.field.StandardField;
 import org.jabref.model.entry.field.UnknownField;
@@ -17,9 +21,12 @@
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
 
 @FetcherTest
 class MedlineFetcherTest {
+    private static final Optional<String> API_KEY = Optional.of(new BuildInfo().medlineApiKey);
 
     private MedlineFetcher fetcher;
     private BibEntry entryWijedasa;
@@ -32,7 +39,11 @@ void setUp() throws InterruptedException {
         // pause between runs to avoid 403 and 429 at Medline
         Thread.sleep(1000);
 
-        fetcher = new MedlineFetcher();
+        ImporterPreferences importerPreferences = mock(ImporterPreferences.class);
+        when(importerPreferences.getApiKeys()).thenReturn(FXCollections.emptyObservableSet());
+        when(importerPreferences.getApiKey(MedlineFetcher.FETCHER_NAME)).thenReturn(API_KEY);
+
+        fetcher = new MedlineFetcher(importerPreferences);
 
         entryWijedasa = new BibEntry(StandardEntryType.Article)
                 .withField(StandardField.AUTHOR, "Wijedasa, Lahiru S. and Jauhiainen, Jyrki and Könönen, Mari and Lampela, Maija and Vasander, Harri and Leblanc, Marie-Claire and Evers, Stephanie and Smith, Thomas E. L. and Yule, Catherine M. and Varkkey, Helena and Lupascu, Massimo and Parish, Faizal and Singleton, Ian and Clements, Gopalasamy R. and Aziz, Sheema Abdul and Harrison, Mark E. and Cheyne, Susan and Anshari, Gusti Z. and Meijaard, Erik and Goldstein, Jenny E. and Waldron, Susan and Hergoualc'h, Kristell and Dommain, Rene and Frolking, Steve and Evans, Christopher D. and Posa, Mary Rose C. and Glaser, Paul H. and Suryadiputra, Nyoman and Lubis, Reza and Santika, Truly and Padfield, Rory and Kurnianto, Sofyan and Hadisiswoyo, Panut and Lim, Teck Wyn and Page, Susan E. and Gauci, Vincent and Van Der Meer, Peter J. and Buckland, Helen and Garnier, Fabien and Samuel, Marshall K. and Choo, Liza Nuriati Lim Kim and O'Reilly, Patrick and Warren, Matthew and Suksuwan, Surin and Sumarga, Elham and Jain, Anuj and Laurance, William F. and Couwenberg, John and Joosten, Hans and Vernimmen, Ronald and Hooijer, Aljosja and Malins, Chris and Cochrane, Mark A. and Perumal, Balu and Siegert, Florian and Peh, Kelvin S.-H. and Comeau, Louis-Pierre and Verchot, Louis and Harvey, Charles F. and Cobb, Alex and Jaafar, Zeehan and Wösten, Henk and Manuri, Solichin and Müller, Moritz and Giesen, Wim and Phelps, Jacob and Yong, Ding Li and Silvius, Marcel and Wedeux, Béatrice M. M. and Hoyt, Alison and Osaki, Mitsuru and Hirano, Takashi and Takahashi, Hidenori and Kohyama, Takashi S. and Haraguchi, Akira and Nugroho, Nunung P. and Coomes, David A. and Quoi, Le Phat and Dohong, Alue and Gunawan, Haris and Gaveau, David L. A. and Langner, Andreas and Lim, Felix K. S. and Edwards, David P. and Giam, Xingli and Van Der Werf, Guido and Carmenta, Rachel and Verwer, Caspar C. and Gibson, Luke and Gandois, Laure and Graham, Laura Linda Bozena and Regalino, Jhanson and Wich, Serge A. and Rieley, Jack and Kettridge, Nicholas and Brown, Chloe and Pirard, Romain and Moore, Sam and Capilla, B. Ripoll and Ballhorn, Uwe and Ho, Hua Chew and Hoscilo, Agata and Lohberger, Sandra and Evans, Theodore A. and Yulianti, Nina and Blackham, Grace and Onrizal and Husson, Simon and Murdiyarso, Daniel and Pangala, Sunita and Cole, Lydia E. S. and Tacconi, Luca and Segah, Hendrik and Tonoto, Prayoto and Lee, Janice S. H. and Schmilewski, Gerald and Wulffraat, Stephan and Putra, Erianto Indra and Cattau, Megan E. and Clymo, R. S. and Morrison, Ross and Mujahid, Aazani and Miettinen, Jukka and Liew, Soo Chin and Valpola, Samu and Wilson, David and D'Arcy, Laura and Gerding, Michiel and Sundari, Siti and Thornton, Sara A. and Kalisz, Barbara and Chapman, Stephen J. and Su, Ahmad Suhaizi Mat and Basuki, Imam and Itoh, Masayuki and Traeholt, Carl and Sloan, Sean and Sayok, Alexander K. and Andersen, Roxane")
