Skip to content

Feature/ paper discovery module chitchat - #319

Open
fiifidawson wants to merge 32 commits into
EPFLiGHT:mainfrom
fiifidawson:feature/-paper-discovery-module-chitchat
Open

Feature/ paper discovery module chitchat#319
fiifidawson wants to merge 32 commits into
EPFLiGHT:mainfrom
fiifidawson:feature/-paper-discovery-module-chitchat

Conversation

@fiifidawson

Copy link
Copy Markdown
Contributor

No description provided.

Introduce the mmore.paper_discovery package with initial components for a paper discovery pipeline:

- schema: dataclasses for CategoryQuery, Paper, and SynonymEntry to normalize data shapes.
- boolean: functions to load synonym tables and build category-level boolean queries (load_synonyms, _or_group, build_boolean_queries).
- logging_config: simple logger configuration for PaperDiscovery.
- sources/base: SourceAdapter protocol defining the search interface and guidance that adapters must not raise on network errors.
- Stubs added for other modules (config, pdf, pipeline, and individual source modules) to be implemented later.

These changes set up the core types and query-building logic used by downstream source adapters and pipeline stages.
- Introduce src/mmore/paper_discovery/sources/__init__.py which centralizes source adapter imports and registers them in REGISTRY (openalex, europepmc, arxiv; google_scholar is commented).
- Adds typed get_adapter(name, **kwargs) to instantiate a SourceAdapter or raise a ValueError for unknown sources. Provides a single entrypoint for resolving source adapters.
- Introduce a new `paper-discovery` subcommand and entrypoint to run the Paper Discovery pipeline.
- Adds src/mmore/run_paper_discovery.py which loads a PaperDiscoveryConfig, constructs and runs PaperDiscoveryPipeline (with timing, dotenv, and profiling support), and wires it into the CLI (src/mmore/cli.py).
- Also removes a stale tmp_pdf file.
  - boolean: skip categories with no resolved synonyms instead of emitting an empty boolean string
  - schema: include search_category in Paper.to_dict() output
  - openalex: widen exception catch to never raise from search()
  - tests/conftest: make langchain_milvus import optional so the suite is collectible on partial installs
Add robust PDF fetching with paywall detection and optional EZproxy proxying, plus progress/counting and partial-result handling.

- Add pdf_proxy_prefix config option and docs.
- Replace simple download return value with DownloadResult (path, paywalled, errored, status), detect common paywall statuses, and use a polite default User-Agent.
- Introduce _proxify helper to wrap PDF URLs via an EZproxy prefix and use it for initial/follow-up fetches.
- Improve logging and error handling for network failures; surface paywalled PDFs separately from errors.
- In the pipeline, add tqdm progress (with a lightweight fallback), track succeeded/paywalled/errored/skipped counts, log a summary and a tip when paywalled PDFs are encountered, and ensure partial results are written on KeyboardInterrupt.
- Add arXiv client improvements: request timeout, 429 backoff handling, and related logging.
- Update .gitignore to include local paper_discovery artifacts (pdf cache and papers.json) and normalize example output paths.

@fabnemEPFL fabnemEPFL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty cool PR with this new module. several things to change to integrate it properly into mmore

Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread src/mmore/paper_discovery/pipeline.py Outdated
Comment thread src/mmore/paper_discovery/sources/__init__.py
Comment thread src/mmore/paper_discovery/boolean.py Outdated
Comment thread src/mmore/paper_discovery/boolean.py Outdated
Comment thread src/mmore/paper_discovery/boolean.py Outdated

def build_boolean_queries(
synonyms: List[SynonymEntry],
categories: Dict[str, List[str]],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be more specific about the expected format for categories

…ass A)

  - Adapters now explicitly inherit `SourceAdapter` (EPFLiGHT#20, EPFLiGHT#22, EPFLiGHT#23, EPFLiGHT#24)
  - `SourceAdapter` Protocol clarified: `name` + `search()` only;
    ctor kwargs documented as `get_adapter()` surface (EPFLiGHT#11, EPFLiGHT#12, EPFLiGHT#27, EPFLiGHT#29)
  - Public docstrings filled in (`PaperDiscoveryPipeline`, `run`,
    `load_synonyms`, `build_boolean_queries`, `get_adapter`, `extract_text`)
    (EPFLiGHT#17, EPFLiGHT#30, EPFLiGHT#31, EPFLiGHT#33, EPFLiGHT#34, EPFLiGHT#35, EPFLiGHT#36)
  - `_enrich_with_pdf_text` signature `Iterable[Paper]` -> `List[Paper]` (EPFLiGHT#32)
  - `boolean.by_word` lookup made case-insensitive (EPFLiGHT#3)
  - `config.py` / docs: clarified what `user_agent` is and gave a
    concrete example (EPFLiGHT#4, EPFLiGHT#5, EPFLiGHT#13)
  - Docs: uncommented Google Scholar install + sources-table row,
    added `user_agent` subsection, rewrote "User-managed UA" ->
    "Why we don't spoof the UA" (EPFLiGHT#1, EPFLiGHT#2, EPFLiGHT#6)
  - `examples/config.yaml`: commented `google_scholar` source with
    doc pointer; clarified `arxiv_category_map` purpose (EPFLiGHT#7, EPFLiGHT#8)
  - Drive-by: fixed latent `list[str](REGISTRY)` typo in `get_adapter`
    error path
  - pdf.extract_text now routes through mmore.process.PDFProcessor (fast path, PyMuPDF-backed; no marker models loaded). The `paper_discovery` extra now depends on `mmore[process]` for the unified text-extraction surface (EPFLiGHT#16, EPFLiGHT#18).
  - arxiv: pair query (`all:"X" AND all:"Y"`) is now configurable via `arxiv_enable_pair_query` (default True). Hardcoded top_n=4 lifted into `MAX_SIMPLIFIED_TERMS` constant with rationale (EPFLiGHT#25, EPFLiGHT#26, EPFLiGHT#28).
  - boolean.load_synonyms: synonym files are now JSONL only (one {"word": ..., "synonyms": [...]} per line). The old JSON array format and example file are removed (EPFLiGHT#9).
  - boolean: embedded `"` in synonym terms now stripped at load time (`_sanitize_term`) so they can't break the quoted boolean query (EPFLiGHT#15).
  - Categories moved out of inline `PaperDiscoveryConfig.categories` into a separate `categories.yaml`, loaded via a `CategoriesFile` dataclass. Config now takes `categories_path: str` (EPFLiGHT#14).
  - sources/__init__: Google Scholar registered unconditionally; the scholarly import was already lazy inside the adapter's search() (EPFLiGHT#10).

@JCHAVEROT JCHAVEROT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @fiifidawson!

Just tested your feature, and with the example config it looks really great sofar ✨

I left a few comments, if you can take a look at them when you have the chance, let me know if you want to discuss things

python -m mmore paper-discovery --config-file examples/paper_discovery/config.yaml
[PaperDiscovery 📄 -- 2026-06-30 15:17:16] Running Paper Discovery pipeline...
[PaperDiscovery 📄 -- 2026-06-30 15:17:16] Built 2 category queries
[PaperDiscovery 📄 -- 2026-06-30 15:17:16] Searching openalex for 'Broad Foundational Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:19] openalex returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:19] Searching europepmc for 'Broad Foundational Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:20] europepmc returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:20] Searching arxiv for 'Broad Foundational Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:23] arxiv returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:23] Searching openalex for 'Humanitarian AI Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:25] openalex returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:25] Searching europepmc for 'Humanitarian AI Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:26] europepmc returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:26] Searching arxiv for 'Humanitarian AI Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:29] arxiv returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:29] After dedupe: 125 papers (from 150)
PDFs: 100%|█████████████████████████████████████████████████████| 125/125 [04:37<00:00,  2.22s/paper, cache=2, err=4, ok=70, paywall=24]
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] PDF download: 70/125 succeeded (2 cached, 68 fresh), 24 paywalled, 4 errors, 27 skipped
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] Tip: 24 PDFs were blocked by publisher paywalls. Set `pdf_proxy_prefix` in your config to use institutional access (e.g. EPFL: 'https://login.proxy.epfl.ch'), or set `download_pdfs: false` to skip PDFs entirely.
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] Wrote 125 papers to examples/paper_discovery/papers.json
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] Completed in 291.00s

Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread docs/source/core_features/paper_discovery.md Outdated
Comment thread tests/conftest.py
Comment thread docs/source/core_features/paper_discovery.md
  #3499552199 part 2 - MultimodalSample integration:

  - schema: Paper.to_multimodal_sample(pdf_path="") returns a mmore.type.MultimodalSample. Text falls back extracted_text -> abstract -> title -> "". Paper-specific fields (title, authors, year, source, url, search_category, abstract) land in metadata.extra; metadata.processor_type = "paper_discovery" so downstream filters can recognise the origin.
  - config: new `multimodal_output_file: Optional[str]` knob.
  - pipeline: when the knob is set, _write_multimodal_jsonl writes a JSONL of MultimodalSample records alongside papers.json. Existing papers.json output is unchanged - additive only.
  - docs: new "Feeding results into mmore's index / RAG" section and row in the knobs table.
  - tests: +4 covering the extracted->abstract->title fallback and metadata carriage.

#3499643602 - CI wiring:
  - tests/conftest.py: revert the try/except guard on angchain_milvus. CI's install list includes [index,rag] so the import is guaranteed.
  - .github/workflows/tests.yml: add paper_discovery to the extras installed on CI.
  - .github/workflows/pyright.yml: same, so type checks cover the new package.
@JCHAVEROT JCHAVEROT linked an issue Jul 2, 2026 that may be closed by this pull request
…SONL

  - pipeline._write_output writes one JSON object per line.
  - examples/paper_discovery/config.yaml points at papers.jsonl.
  - .gitignore updated for papers.jsonl + papers.samples.jsonl.
  - config.py docstring documents the .jsonl convention.
  - Docs (paper_discovery.md) show the JSONL shape, workflow diagram and "Feeding results" section updated.
- schema.py: authors: Optional[str] -> Optional[List[str]]
- sources/openalex.py: keep the list comprehension result unjoined.
- sources/arxiv.py: same, using a walrus-filtered comprehension.
- sources/google_scholar.py: same.
- sources/europepmc.py: new _parse_authors() helper - prefer the structured authorList.author[].fullName (available with resultType=core, which we already request); fall back to naive authorString.split(",") only when the structured shape is missing.
- Tests: updated OpenAlex assertion; added 3 tests for the Europe PMC parser covering the structured path, the fallback path, and the empty-input path.
- Docs: sample JSONL line uses ["Ada Lovelace", "Alan Turing"].
@fiifidawson
fiifidawson requested a review from fabnemEPFL July 14, 2026 06:02

@JCHAVEROT JCHAVEROT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general comment, please change all the old types from Typing to use the ones available by default with python's latest version. For example Dict --> dict, List --> list, Union --> |, Optional --> | None

Would be great if you make a pass on all docstrings to simplify and make them more dev-friendly for those who will maintain your code later, they are a bit too heavy/long sometimes and not high-level enough (also avoid em-dashes or semi-colons). However it's nice when you document the public functions arguments and return types

Comment on lines +160 to +170
### Institutional access via EZproxy (recommended)

Set `pdf_proxy_prefix` in your config to your institution's EZproxy URL. Every paywalled URL will be wrapped through the proxy automatically:

```yaml
pdf_proxy_prefix: "https://login.proxy.epfl.ch"
```

The proxy handles SAML/Shibboleth authentication and the publisher sees a valid institutional session.

**Caveat:** the first request through the proxy may redirect to your institution's login page, which a script cannot fill in. The simplest workaround for v1 is to sign in once in a browser to seed the session cookie, then run the pipeline.

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like its not working for me

When I uncomment the line in the config file I am never redirected to the EPFL login page 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for testing this.

EPFL doesn't support this, but other institutions do. I'll leave it as a placeholder for others. For EPFL, you simply need to use the network or be on the VPN.

But you can disable it so it's optional to use it.

Comment on lines +198 to +204
## 🔌 Feeding results into mmore's index / RAG

If you plan to index the discovered papers or run RAG over them, you don't need to send them back through `mmore process`. Ask the pipeline to write an extra output file in mmore's canonical `MultimodalSample` shape:

```yaml
multimodal_output_file: examples/paper_discovery/papers.samples.jsonl
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The post process then indexing are working like a charm 👍

Image

Comment thread README.md Outdated

mmore is an open-source, end-to-end pipeline to ingest, process, index, and retrieve knowledge from heterogeneous files: PDFs, Office docs, spreadsheets, emails, images, audio, video, and web pages. It standardizes content into a unified multimodal format, supports distributed CPU/GPU processing, and provides hybrid dense+sparse retrieval with an integrated RAG service (CLI, APIs).

https://github.com/EPFLiGHT/mmore/pull/319/conflict?name=.github%252Fworkflows%252Fpyright.yml&ancestor_oid=e880609d900c8d1ca113e404d0d60dc5604ddc65&base_oid=652683914c68996968fa4d333618d000927a80eb&head_oid=104f07055d58c2e6c1420d89649cd5de5e57f32e

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like a nasty left over from a merge conflict resolution, don't forget to remove it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Came in with the v2 merge. Removed.

Comment thread pyproject.toml Outdated
Comment on lines +178 to +181
# PDF text extraction goes through mmore.process.PDFProcessor for
# consistency with the rest of mmore. The fast path uses PyMuPDF
# (no marker models loaded), so runtime cost is small even though
# the install pulls the full `process` stack.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this much detail in the pyproject.toml

Suggested change
# PDF text extraction goes through mmore.process.PDFProcessor for
# consistency with the rest of mmore. The fast path uses PyMuPDF
# (no marker models loaded), so runtime cost is small even though
# the install pulls the full `process` stack.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trimmed.

Comment thread examples/paper_discovery/papers.json Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it wanted that you uploaded this file ?
Maybe a smaller one as an example is better, this one is 2.7MO big

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intentional, they slipped in before the ignore rules were right.

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this one, did you push it so that we have an example of the results format?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to have this standardized pattern with the search() function for the sources 👍

Comment thread src/mmore/paper_discovery/sources/europepmc.py Outdated
Comment on lines +21 to +22
user_agent: str = "mmore-paper-discovery/1.0",
max_pages: int = 1,

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both these arguments are unused, might be a careless mistake

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're accepted because get_adapter() passes the same kwargs to every adapter, but scholarly drives its own HTTP transport and pagination so neither is forwarded. Made that explicit rather than leaving them silently dropped.

Comment on lines +66 to +70
def _safe_year(value):
try:
return int(value) if value else None
except (TypeError, ValueError):
return None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You defined multiple time private safe_year() or coerce_year() or coerce_int() functions in the sources arXiv.py, europepmc.py, google_scholar.py, openalex.py, they are all doing intrinsically the same thing avoid redundancy by defining helper functions you could reuse

Comment thread src/mmore/paper_discovery/boolean.py Outdated
Comment on lines +54 to +55
word = row.get("word") or row.get("WORD")
raw = row.get("synonyms") or row.get("SYNONYMS AND NEAR SYNONYMS") or ""

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why having these or row.get("WORD") and or row.get("SYNONYMS AND NEAR SYNONYMS")? You set the synonyms.jsonl to have the format {"word": ..., "synonyms": [...]} in examples/paper_discovery/config.yaml, i guess no need to support other structures

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Comment thread src/mmore/paper_discovery/pipeline.py Outdated
Comment on lines +17 to +33
try:
from tqdm.auto import tqdm
except ImportError: # pragma: no cover
# tqdm isn't in paper_discovery's hard deps - if missing, run silently
# but still expose the methods we touch (set_postfix).
class _NoTqdm:
def __init__(self, iterable, **_kwargs):
self._iterable = iterable

def __iter__(self):
return iter(self._iterable)

def set_postfix(self, **_kwargs):
pass

def tqdm(iterable, **kwargs): # type: ignore[no-redef]
return _NoTqdm(iterable, **kwargs)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid the pragma and the type flags, this is anti-pattern because it could hide problems instead of solving them

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also tqdm should always be found, normally no need this check, or maybe it was missing and you had errors for other reasons?

Comment thread src/mmore/paper_discovery/schema.py Outdated
if TYPE_CHECKING:
from ..type import MultimodalSample

SourceName = Literal["arxiv", "openalex", "europepmc", "google_scholar"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a (Str, Enum) class would be cleaner here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread src/mmore/run_paper_discovery.py Outdated
from .paper_discovery.pipeline import PaperDiscoveryPipeline
from .utils import load_config

load_dotenv()

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you actually need a .env ?

In the repo its more common to export environment variables for the secrets

Comment thread pyproject.toml Outdated
# the install pulls the full `process` stack.
"mmore[process]",
"requests>=2.31",
"beautifulsoup4>=4.13",

@JCHAVEROT JCHAVEROT Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beautifulsoup4 is already in mmore[process], I think no need to redefine it here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

Flagging one related change so it doesn't look inconsistent: I'm adding tqdm to the extra in the same round. The difference is that bs4 is declared in process, whereas tqdm is declared nowhere and only reaches us transitively through transformers. Since we import it directly, it should be declared.

- README: remove a stray GitHub conflict-resolution URL left behind by the v2 merge (#3648952822).
- europepmc: fix `docementStyle` -> `documentStyle`. The misspelled key never matched, so the PDF-URL preference silently never fired and every Europe PMC result fell through to its landing page.
- boolean.load_synonyms: drop the `WORD` / `SYNONYMS AND NEAR SYNONYMS` fallbacks. We document exactly one input shape, the alternates were dead weight from a spreadsheet-import experiment.
- google_scholar: resolve the unused `user_agent` / `max_pages` ctor args explicitly instead of accepting and dropping them.
- pyproject: drop `beautifulsoup4`, already a direct dep of `mmore[process]` which `paper_discovery` depends on (#3649954641).
- examples: untrack `papers.json` (2.7 MB) and `papers.samples.jsonl` (6.4 MB). `papers.json` also predates the JSON -> JSONL switch. A 3-line `papers.sample.jsonl` is committed instead so the output format is still visible in-repo (#3649860650, 3649862748).
…red year helper

- pipeline: drop the tqdm try/except shim and switch to `mmore.ux.progress`, the shared Rich progress bar introduced in EPFLiGHT#333 and already used by the index, post-process, colvision, and websearch pipelines(#3649937401,  #3649940848). This removes the `# pragma: no cover` and the `# type: ignore[no-redef]`, which were the only two suppressions in the module, and needs no new dependency since `mmore.ux` is core. Counters now render through `set_postfix_str` as `ok=70 cache=2 paywall=24 err=4`.

- schema: `SourceName` is now a `(str, Enum)` instead of a `Literal` (#3649950643). The `str` mixin keeps JSON output unchanged, so `papers.jsonl` still carries `"source": "arxiv"` rather than `"SourceName.ARXIV"`. All four adapters use the members. Three tests cover the round-trip, string equality, and that the enum matches `REGISTRY` exactly.

- sources: collapse four near-identical private year/int coercion helpers  (`_coerce_year` x2, `_safe_year`, `_coerce_int`) into `coerce_year` in the new `sources/_utils.py` (#3649910611). They differed in input shape, not intent, so the shared helper handles ints, year strings, and ISO dates. Europe PMC's two-key lookup is now a `first_year` call rather than its own function.

- pdf: fix two pre-existing pyright errors in `_find_pdf_link` where bs4 types an href as `str | list[str]`.
- Remove the hostname from all seven locations. Keep `pdf_proxy_prefix` as a generic knob.

- Add `_looks_like_login_page`. A proxy needing interactive sign-in answers 200 OK with its login form, which the pipeline was filing as a silent skip. Runs now warn: "N downloads returned a sign-in page instead of a PDF. This pipeline cannot log in for you."

- Rewrite the paywall log line, which pointed at the bogus URL. It now explains that 403s are usually User-Agent blocking and that the VPN does not always help.

- Rewrite the Paywalled PDFs docs around the distinction that matters: no subscription versus have-access-but-blocked-as-a-bot. Covers both institutional access models and what to set for each.

- Add scripts/diagnose_pdf_access.py, which prints the status, redirect chain, content type, sign-in-page verdict and resulting bucket for a URL, either given directly or pulled from a previous run's failures. This is what surfaced the DNS failure.
…nused dotenv

- Docstrings trimmed to be shorter and more high-level, keeping the Args/Returns blocks on public functions. Removed all em-dashes and prose semicolons.

- removed `load_dotenv()` from run_paper_discovery (r3649953527). This pipeline reads no environment variables, every source it queries is anonymous, so the call was copied from entrypoints that do need API keys.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Paper Discovery Module / CHITCHAT

3 participants