Feature/ paper discovery module chitchat

.github/workflows/pyright.yml

@@ -22,7 +22,7 @@ jobs:
         run: pipx install uv
 
       - name: Install base dependencies
-        run: uv venv .venv && source .venv/bin/activate && uv pip install -e ".[process,index,rag,api,tui,cpu,dev,websearch,privacy]"
+        run: uv venv .venv && source .venv/bin/activate && uv pip install -e ".[process,index,rag,api,tui,cpu,dev,websearch,privacy,paper_discovery]"
 
       - name: Run Pyright (base)
         continue-on-error: true

.github/workflows/tests.yml

@@ -33,7 +33,7 @@ jobs:
         run: |
           uv venv .venv
           source .venv/bin/activate
-          uv pip install -e ".[process,index,rag,api,cpu,dev,websearch,privacy]"
+          uv pip install -e ".[process,index,rag,api,cpu,dev,websearch,privacy,paper_discovery]"
 
       - name: Show installed cohere and langchain-cohere versions
         run: |

.gitignore

@@ -128,8 +128,16 @@ test*.sh
 .venv
 
 # Examples
-examples/outputs
+examples/outputs/
 outputs/
 
+# Paper Discovery local artifacts
+examples/paper_discovery/pdf_cache/
+examples/paper_discovery/papers.json
+examples/paper_discovery/papers.jsonl
+examples/paper_discovery/papers.samples.jsonl
+# ...but keep the small committed sample of the output format
+!examples/paper_discovery/papers.sample.jsonl
+
 paper/
 chitchat/

docs/source/core_features/paper_discovery.md

@@ -0,0 +1,267 @@
+# 📄 Paper Discovery
+
+## Overview
+
+The **Paper Discovery** module helps you build a targeted collection of academic papers on a topic you care about. You describe the topic once — as a list of keywords with synonyms — and the module searches several open academic repositories (OpenAlex, Europe PMC, arXiv, optionally Google Scholar) on your behalf, downloads whatever PDFs it can, and writes a single JSON file with the metadata and extracted text.
+
+The output is a plain `Paper[]` JSON list. What you do with it next is up to you — feed it into an indexer, hand it to a screening tool, or just read the abstracts. This page describes the standalone `paper_discovery` module. It is independent of the `rag` and `index` pipelines.
+
+## Installation
+
+```bash
+uv pip install "mmore[paper_discovery]"
+```
+
+For optional Google Scholar support (captcha-prone, best-effort):
+
+```bash
+uv pip install scholarly
+```
+
+`scholarly` is **not** in the `paper_discovery` extra by design — it is captcha-prone. Install only if needed.
+
+## Supported sources
+
+| Source | What it covers |
+|--------|----------------|
+| **OpenAlex** | Broadest general index of academic papers. Abstracts included by default. |
+| **Europe PMC** | Biomedical and life-sciences literature with links to full text where available. |
+| **arXiv** | Preprints in ML, physics, math, and CS. Slower than the others because arXiv enforces a 3-second gap between requests. |
+| **Google Scholar** | Widest overall coverage but captcha-prone. Opt-in — requires `scholarly`. |
+
+All four sources are anonymous — no API keys needed. Precise rate limits, retry back-off, and API-specific details live in each adapter's docstring under `src/mmore/paper_discovery/sources/`.
+
+## 🔁 Workflow
+
+```
+synonyms.jsonl + categories.yaml
+        │
+        ▼
+Stage 1: build boolean queries (pure, offline)
+        │
+        ▼
+Stage 2: fetch from each source, dedupe, optionally download PDFs
+        │
+        ▼
+   papers.jsonl
+```
+
+Stage 1 doesn't touch the network — it just turns your synonyms + categories into search queries. Stage 2 is where everything network-related happens: hitting each source, respecting their rate limits, downloading PDFs, retrying when things go wrong.
+
+## 💻 Minimal Example
+
+### 1. Prepare your synonym table
+
+A **JSONL** file with one `{"word": ..., "synonyms": [...]}` object per line. Easy to diff, append, and edit line-by-line:
+
+```jsonl
+{"word": "Foundation model", "synonyms": ["LLM", "large language model", "GPT"]}
+{"word": "Humanitarian & Crisis Response", "synonyms": ["humanitarian aid", "disaster response"]}
+```
+
+You don't have to worry about capitalization — `"Foundation model"`, `"foundation model"` and `"FOUNDATION MODEL"` are treated as the same word. Whitespace does need to match. If any of your terms happen to contain a `"` character, don't stress — it's silently stripped when the file is loaded.
+
+### 2. Define your categories
+
+Categories live in their own YAML file, loaded via a small `CategoriesFile` dataclass:
+
+```yaml
+# categories.yaml
+categories:
+  Broad Foundational Search:
+    - Foundation model
+    - Machine Learning
+  Humanitarian AI Search:
+    - Foundation model
+    - Humanitarian & Crisis Response
+```
+
+Each name under a category must match a `word` in your synonyms file. For every category, the module builds one search that finds papers mentioning **at least one term from each group of synonyms**. So the "Broad Foundational Search" example above will match a paper if it talks about *any* foundation-model synonym AND *any* machine-learning synonym.
+
+### 3. Create a config file
+
+See [`examples/paper_discovery/config.yaml`](https://github.com/EPFLiGHT/mmore/blob/master/examples/paper_discovery/config.yaml). It points at your `synonyms_path` and `categories_path`.
+
+### 4. Run the pipeline
+
+```bash
+python3 -m mmore paper-discovery --config-file examples/paper_discovery/config.yaml
+```
+
+Progress is shown live with a tqdm bar while PDFs are being downloaded:
+
+```
+PDFs:  42%|████▏     | 52/124 [01:15<01:43, 1.45s/paper, ok=42, cache=0, paywall=8, err=2]
+```
+
+Press **Ctrl+C** at any time — the pipeline catches the interrupt and writes whatever it has so far to `output_file` before exiting.
+
+## 📦 Output
+
+A JSONL file — one `Paper` record per line. Example line:
+
+```json
+{"title": "A foundation model for humanitarian response", "authors": ["Ada Lovelace", "Alan Turing"], "url": "https://arxiv.org/pdf/2401.00001.pdf", "abstract": "We introduce …", "year": 2024, "extracted_text": "<full PDF text>", "source": "arxiv", "search_category": "Humanitarian AI Search"}
+```
+
+Line-per-record makes the file streamable (read one paper at a time), diff-friendly, and easy to append to. Any JSONL-aware tool (`jq`, `pandas.read_json(lines=True)`, mmore's `MultimodalSample.from_jsonl`) can consume it directly.
+
+Fields are **nullable on purpose** — sources differ in what they return. `null` means "we don't know."
+
+## ⚙️ Configuration knobs
+
+| Knob | Default | Notes |
+|------|---------|-------|
+| `synonyms_path` | *(required)* | Path to a `.jsonl` synonyms file (one object per line) |
+| `categories_path` | *(required)* | Path to a `categories.yaml` file (see step 2) |
+| `sources` | `[openalex, europepmc, arxiv]` | Add `google_scholar` to opt in |
+| `download_pdfs` | `true` | Set `false` to skip the PDF stage entirely |
+| `max_pages` | `3` | Pages per source per query |
+| `max_results` | `50` | Hard cap per source per query |
+| `pdf_dir` | `./pdf_cache` | Reused across runs (see *PDF caching* below) |
+| `force_redownload` | `false` | Set `true` to ignore the on-disk cache and re-fetch every PDF |
+| `pdf_extractor` | `"fast"` | Which mmore PDF processor to use. `"fast"` = PyMuPDF-backed, no models loaded. `"full"` = marker + surya for better parsing (slow, downloads models) |
+| `multimodal_output_file` | `null` | If set, also write a JSONL of `MultimodalSample` records that mmore's post-process / index / RAG pipelines can consume directly (see [Feeding results into mmore's index / RAG](#-feeding-results-into-mmores-index--rag)) |
+| `pdf_proxy_prefix` | `null` | EZproxy host, only if your institution runs one. Leave unset for VPN-based access (see *Paywalled PDFs* below) |
+| `user_agent` | `mmore-paper-discovery/1.0 …` | HTTP `User-Agent` header sent on every outbound request — see below |
+| `arxiv_category_map` | `null` | Maps a substring of your category title to an arXiv code (e.g. `Foundational` → `cs.LG`) — adds `cat:<code>` to the arXiv query |
+| `arxiv_enable_pair_query` | `true` | Runs one extra arXiv search per category that requires the top two terms together (better precision). Turn off if you'd rather save a few seconds per category |
+
+### `user_agent`
+
+This is the "who's asking?" string sent with every network request the module makes. Sources use it to identify who's hitting their API, and OpenAlex specifically gives faster, more reliable responses to requests that include a contact address.
+
+You should set it to something that identifies your project so the source's team can reach you if you're making too many requests. A concrete example:
+
+```yaml
+user_agent: "my-lab-pipeline/1.0 (mailto:alice@example.com)"
+```
+
+The default just identifies mmore + the repo URL, which works but doesn't tell anyone who *you* are.
+
+## 💾 PDF caching
+
+`pdf_dir` is reused across runs. Before downloading a PDF, the pipeline checks whether a file with the same name already exists; if so, the HTTP fetch is skipped and text is extracted directly from the cached file.
+
+The summary line at the end of a run shows the split:
+
+```
+PDF download: 108/124 succeeded (45 cached, 63 fresh), 16 paywalled, 0 errors, 0 skipped
+```
+
+This makes interrupted runs cheap to resume — every PDF that landed on disk before Ctrl+C is reused, only the missing ones are fetched.
+
+To force a full re-download (e.g. after a publisher updates a paper), set `force_redownload: true` in your config.
+
+## 🔒 Paywalled PDFs
+
+Expect a chunk of your run to come back without full text. Some of that you can fix, some of it you cannot. Read this section before spending time on it.
+
+### There are two different reasons a PDF fails
+
+They look the same in the summary line but have completely different fixes.
+
+**1. You don't have access.** Your institution has no subscription to that journal. Nothing in this pipeline can fix that. Request the paper through your library instead.
+
+**2. You have access, but the publisher blocks automated tools.** This is the common one, and it surprises people. Publishers like Wiley, ACM, and Science return `403` to anything that doesn't look like a browser, *regardless of whether your institution subscribes*. You can click the same link in your browser and get the PDF, then watch the pipeline get refused for the identical URL.
+
+mmore does **not** work around this by pretending to be a browser. Spoofing the User-Agent violates most publishers' terms of service, and a spoofed default would get the project's identifier blocklisted for every user of the library. That's a deliberate choice, not an oversight.
+
+### How your institution grants access matters
+
+Two common models. Check which one yours uses before touching any config.
+
+**VPN and IP recognition.** You connect to your institution's VPN, the publisher sees an institutional IP, and access is granted automatically. **Leave `pdf_proxy_prefix` unset.** The direct URL already works. EPFL works this way.
+
+**EZproxy.** Your library gives you a hostname that rewrites URLs. If, and only if, your institution runs one, set:
+
+```yaml
+pdf_proxy_prefix: "https://ezproxy.example.edu"
+```
+
+Use the exact host your library publishes. Do not guess it. A wrong host either fails DNS or serves you a login page, and neither yields a PDF.
+
+Even with the right host, an EZproxy that needs an interactive sign-in will return its login page instead of the PDF. The pipeline detects this and warns you:
+
+```
+12 downloads returned a sign-in page instead of a PDF. This pipeline
+cannot log in for you.
+```
+
+There is no headless workaround for that today. The pipeline cannot complete a SAML or Shibboleth login.
+
+### Skip PDFs entirely
+
+If full text isn't essential, this is the cheapest path and it always works:
+
+```yaml
+download_pdfs: false
+```
+
+You still get every paper's metadata and abstract. Only `extracted_text` is left empty.
+
+## 📄 PDF text extraction
+
+Text extraction goes through the same PDF processor the rest of mmore uses, so you get consistent output whether a paper comes from Paper Discovery or from another `mmore process` run. There are two settings you can pick between with `pdf_extractor`:
+
+- **`fast` (default)** — Uses PyMuPDF under the hood. Nothing to download, works right out of the box, and it's good enough for most academic PDFs.
+- **`full`** — Uses mmore's fuller pipeline (with layout-aware parsing). Better on messy PDFs — multi-column layouts, scanned pages, complex figures — but it downloads model weights the first time it runs, and it's really only worth it if you have a GPU.
+
+Start with `fast`. Only switch to `full` if you notice extraction is losing structure on the papers you care about.
+
+### Why we don't spoof the User-Agent
+
+A common workaround for publisher 403s is to set the `User-Agent` to a browser string (Chrome, Firefox, …). mmore does **not** do that by default for two reasons:
+
+1. It violates most publishers' terms of service.
+2. A baked-in spoofed UA gets the **library's** default identifier blocklisted on first abuse — for every downstream user.
+
+If you have a specific arrangement with a publisher (e.g. a registered crawler agreement), you can set `user_agent` to whatever they require. That's an opt-in you take responsibility for — not a default the library ships.
+
+## 🔌 Feeding results into mmore's index / RAG
+
+If you plan to index the discovered papers or run RAG over them, you don't need to send them back through `mmore process`. Ask the pipeline to write an extra output file in mmore's canonical `MultimodalSample` shape:
+
+```yaml
+multimodal_output_file: examples/paper_discovery/papers.samples.jsonl
+```
+
+Every paper is converted to a `MultimodalSample`:
+
+- **`text`** — the extracted PDF body if we downloaded it, otherwise the abstract, otherwise the title.
+- **`metadata.file_path`** — points at the cached PDF when we have one.
+- **`metadata.processor_type`** — always `"paper_discovery"`, so downstream filters can recognise the source.
+- **`metadata.extra`** — carries the paper-specific fields (title, authors, year, source, url, search_category, abstract).
+
+The resulting JSONL is a drop-in input for the post-process, index, and RAG pipelines. The default `papers.jsonl` output is still written the same way alongside it.
+
+## 🐍 Programmatic use
+
+For embedding the pipeline in another script:
+
+```python
+from mmore.paper_discovery import PaperDiscoveryConfig, PaperDiscoveryPipeline
+from mmore.utils import load_config
+
+config = load_config("examples/paper_discovery/config.yaml", PaperDiscoveryConfig)
+papers = PaperDiscoveryPipeline(config).run()
+print(f"Got {len(papers)} papers")
+```
+
+Or compose Stage 1 alone (no network) for testing:
+
+```python
+from mmore.paper_discovery import build_boolean_queries
+from mmore.paper_discovery.boolean import load_synonyms
+
+synonyms = load_synonyms("examples/paper_discovery/synonyms.jsonl")
+queries = build_boolean_queries(synonyms, {"My Category": ["Foundation model"]})
+for q in queries:
+    print(q.combination_title, "->", q.boolean_combination)
+```
+
+## See also
+
+- [Indexing](../getting_started/indexing.md) — feed `extracted_text` into the indexer
+- [RAG](../getting_started/rag.md) — query the indexed papers
+- [Processing pipeline](../getting_started/process.md) — convert other document formats

docs/source/index.md

@@ -50,6 +50,7 @@ getting_started/windows
 
 core_features/colvision
 core_features/websearch
+core_features/paper_discovery
 core_features/evaluation
 core_features/llm_as_a_judge
 ```
@@ -85,6 +86,7 @@ Here is a quick overview of the main pages:
 - [RAG](getting_started/rag.md): structure retrieval-augmented generation workflows
 - [ColVision](core_features/colvision.md): multimodal retrieval-related documentation
 - [Websearch](core_features/websearch.md): web search integration and related workflows
+- [Paper Discovery](core_features/paper_discovery.md): fetch academic papers from OpenAlex, Europe PMC, arXiv, and Google Scholar
 - [Evaluation](core_features/evaluation.md): assess system performance
 - [LLM as a judge](core_features/llm_as_a_judge.md): corrective retrieval with an LLM judge
 - [Distributed processing](advanced_usage/distributed_processing.md): scale processing across larger workloads

examples/paper_discovery/categories.yaml

@@ -0,0 +1,10 @@
+# Categories drive the pipeline: each category produces one search per
+# source. The list under each name MUST reference canonical `word`
+# entries in your synonyms file (lookup is case-insensitive).
+categories:
+  Broad Foundational Search:
+    - Foundation model
+    - Machine Learning
+  Humanitarian AI Search:
+    - Foundation model
+    - Humanitarian & Crisis Response

examples/paper_discovery/config.yaml

@@ -0,0 +1,64 @@
+# Synonyms file - JSONL, one {"word": ..., "synonyms": [...]} object per line.
+synonyms_path: examples/paper_discovery/synonyms.jsonl
+
+# Categories live in their own YAML, loaded via the `CategoriesFile`
+# dataclass. Keeps PaperDiscoveryConfig flat and lets you swap category
+# sets without touching the rest of the pipeline config.
+categories_path: examples/paper_discovery/categories.yaml
+
+output_file: examples/paper_discovery/papers.jsonl
+pdf_dir: examples/paper_discovery/pdf_cache
+
+# Optional. If set, the pipeline ALSO writes results as JSONL of
+# MultimodalSample records (mmore's canonical processed-document shape).
+# That file plugs directly into `mmore process` post-processing,
+# `mmore index`, and the RAG pipeline - no re-processing needed.
+multimodal_output_file: examples/paper_discovery/papers.samples.jsonl
+
+sources:
+  - openalex
+  - europepmc
+  - arxiv
+  # - google_scholar  # opt-in, captcha-prone. Requires `pip install scholarly`.
+  #                   See docs/core_features/paper_discovery.md.
+
+download_pdfs: true
+max_pages: 2
+max_results: 25
+
+# Which mmore PDFProcessor path to use for text extraction.
+#   "fast" - PyMuPDF-backed process_fast(). No models loaded. Good default.
+#   "full" - marker + surya (mmore process). Better on complex layouts,
+#            but downloads model weights on first use and wants a GPU.
+pdf_extractor: fast
+
+# Reuse PDFs already in pdf_dir from previous runs. Set true to force
+# re-downloading everything (e.g. after a publisher updates a paper).
+force_redownload: false
+
+# Optional EZproxy prefix for institutional access to paywalled PDFs.
+# Most people should leave this unset. Only set it if your institution
+# actually runs EZproxy, and use the host your own library publishes.
+# If your institution grants access by VPN instead (EPFL works this way),
+# leave it unset and connect to the VPN. See the docs for details.
+# pdf_proxy_prefix: "https://ezproxy.example.edu"
+
+# HTTP User-Agent header sent on every outbound request. Identify your
+# caller honestly so rate-limiters / abuse desks can reach you. OpenAlex
+# routes UAs with a contact address into a faster pool.
+user_agent: "mmore-paper-discovery/1.0 (https://github.com/EPFLiGHT/mmore)"
+
+# Custom labels for arXiv category codes. Each key is a substring matched
+# (case-insensitive) against the human-readable category title; the value
+# is the arXiv category code injected into the arXiv query as
+# `cat:<code>`. Categories without a match here just don't get a `cat:`
+# clause - the search still runs.
+arxiv_category_map:
+  Foundational: cs.LG
+  Humanitarian: cs.CY
+
+# When true (default), the arXiv adapter adds one extra targeted query
+# that ANDs the top two simplified terms (`all:"X" AND all:"Y"`) on top
+# of the standalone per-term queries. Set false to skip - saves one
+# 3-second round-trip per category.
+arxiv_enable_pair_query: true