Feature/ paper discovery module chitchat - #319
Conversation
Introduce the mmore.paper_discovery package with initial components for a paper discovery pipeline: - schema: dataclasses for CategoryQuery, Paper, and SynonymEntry to normalize data shapes. - boolean: functions to load synonym tables and build category-level boolean queries (load_synonyms, _or_group, build_boolean_queries). - logging_config: simple logger configuration for PaperDiscovery. - sources/base: SourceAdapter protocol defining the search interface and guidance that adapters must not raise on network errors. - Stubs added for other modules (config, pdf, pipeline, and individual source modules) to be implemented later. These changes set up the core types and query-building logic used by downstream source adapters and pipeline stages.
- currently not being used
- Introduce src/mmore/paper_discovery/sources/__init__.py which centralizes source adapter imports and registers them in REGISTRY (openalex, europepmc, arxiv; google_scholar is commented). - Adds typed get_adapter(name, **kwargs) to instantiate a SourceAdapter or raise a ValueError for unknown sources. Provides a single entrypoint for resolving source adapters.
- Introduce a new `paper-discovery` subcommand and entrypoint to run the Paper Discovery pipeline. - Adds src/mmore/run_paper_discovery.py which loads a PaperDiscoveryConfig, constructs and runs PaperDiscoveryPipeline (with timing, dotenv, and profiling support), and wires it into the CLI (src/mmore/cli.py). - Also removes a stale tmp_pdf file.
- boolean: skip categories with no resolved synonyms instead of emitting an empty boolean string - schema: include search_category in Paper.to_dict() output - openalex: widen exception catch to never raise from search() - tests/conftest: make langchain_milvus import optional so the suite is collectible on partial installs
Add robust PDF fetching with paywall detection and optional EZproxy proxying, plus progress/counting and partial-result handling. - Add pdf_proxy_prefix config option and docs. - Replace simple download return value with DownloadResult (path, paywalled, errored, status), detect common paywall statuses, and use a polite default User-Agent. - Introduce _proxify helper to wrap PDF URLs via an EZproxy prefix and use it for initial/follow-up fetches. - Improve logging and error handling for network failures; surface paywalled PDFs separately from errors. - In the pipeline, add tqdm progress (with a lightweight fallback), track succeeded/paywalled/errored/skipped counts, log a summary and a tip when paywalled PDFs are encountered, and ensure partial results are written on KeyboardInterrupt. - Add arXiv client improvements: request timeout, 429 backoff handling, and related logging. - Update .gitignore to include local paper_discovery artifacts (pdf cache and papers.json) and normalize example output paths.
fabnemEPFL
left a comment
There was a problem hiding this comment.
pretty cool PR with this new module. several things to change to integrate it properly into mmore
|
|
||
| def build_boolean_queries( | ||
| synonyms: List[SynonymEntry], | ||
| categories: Dict[str, List[str]], |
There was a problem hiding this comment.
be more specific about the expected format for categories
…ass A) - Adapters now explicitly inherit `SourceAdapter` (EPFLiGHT#20, EPFLiGHT#22, EPFLiGHT#23, EPFLiGHT#24) - `SourceAdapter` Protocol clarified: `name` + `search()` only; ctor kwargs documented as `get_adapter()` surface (EPFLiGHT#11, EPFLiGHT#12, EPFLiGHT#27, EPFLiGHT#29) - Public docstrings filled in (`PaperDiscoveryPipeline`, `run`, `load_synonyms`, `build_boolean_queries`, `get_adapter`, `extract_text`) (EPFLiGHT#17, EPFLiGHT#30, EPFLiGHT#31, EPFLiGHT#33, EPFLiGHT#34, EPFLiGHT#35, EPFLiGHT#36) - `_enrich_with_pdf_text` signature `Iterable[Paper]` -> `List[Paper]` (EPFLiGHT#32) - `boolean.by_word` lookup made case-insensitive (EPFLiGHT#3) - `config.py` / docs: clarified what `user_agent` is and gave a concrete example (EPFLiGHT#4, EPFLiGHT#5, EPFLiGHT#13) - Docs: uncommented Google Scholar install + sources-table row, added `user_agent` subsection, rewrote "User-managed UA" -> "Why we don't spoof the UA" (EPFLiGHT#1, EPFLiGHT#2, EPFLiGHT#6) - `examples/config.yaml`: commented `google_scholar` source with doc pointer; clarified `arxiv_category_map` purpose (EPFLiGHT#7, EPFLiGHT#8) - Drive-by: fixed latent `list[str](REGISTRY)` typo in `get_adapter` error path
…thub.com/fiifidawson/mmore into feature/-paper-discovery-module-chitchat
- pdf.extract_text now routes through mmore.process.PDFProcessor (fast path, PyMuPDF-backed; no marker models loaded). The `paper_discovery` extra now depends on `mmore[process]` for the unified text-extraction surface (EPFLiGHT#16, EPFLiGHT#18). - arxiv: pair query (`all:"X" AND all:"Y"`) is now configurable via `arxiv_enable_pair_query` (default True). Hardcoded top_n=4 lifted into `MAX_SIMPLIFIED_TERMS` constant with rationale (EPFLiGHT#25, EPFLiGHT#26, EPFLiGHT#28). - boolean.load_synonyms: synonym files are now JSONL only (one {"word": ..., "synonyms": [...]} per line). The old JSON array format and example file are removed (EPFLiGHT#9). - boolean: embedded `"` in synonym terms now stripped at load time (`_sanitize_term`) so they can't break the quoted boolean query (EPFLiGHT#15). - Categories moved out of inline `PaperDiscoveryConfig.categories` into a separate `categories.yaml`, loaded via a `CategoriesFile` dataclass. Config now takes `categories_path: str` (EPFLiGHT#14). - sources/__init__: Google Scholar registered unconditionally; the scholarly import was already lazy inside the adapter's search() (EPFLiGHT#10).
JCHAVEROT
left a comment
There was a problem hiding this comment.
Hi @fiifidawson!
Just tested your feature, and with the example config it looks really great sofar ✨
I left a few comments, if you can take a look at them when you have the chance, let me know if you want to discuss things
python -m mmore paper-discovery --config-file examples/paper_discovery/config.yaml
[PaperDiscovery 📄 -- 2026-06-30 15:17:16] Running Paper Discovery pipeline...
[PaperDiscovery 📄 -- 2026-06-30 15:17:16] Built 2 category queries
[PaperDiscovery 📄 -- 2026-06-30 15:17:16] Searching openalex for 'Broad Foundational Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:19] openalex returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:19] Searching europepmc for 'Broad Foundational Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:20] europepmc returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:20] Searching arxiv for 'Broad Foundational Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:23] arxiv returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:23] Searching openalex for 'Humanitarian AI Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:25] openalex returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:25] Searching europepmc for 'Humanitarian AI Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:26] europepmc returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:26] Searching arxiv for 'Humanitarian AI Search'
[PaperDiscovery 📄 -- 2026-06-30 15:17:29] arxiv returned 25 papers
[PaperDiscovery 📄 -- 2026-06-30 15:17:29] After dedupe: 125 papers (from 150)
PDFs: 100%|█████████████████████████████████████████████████████| 125/125 [04:37<00:00, 2.22s/paper, cache=2, err=4, ok=70, paywall=24]
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] PDF download: 70/125 succeeded (2 cached, 68 fresh), 24 paywalled, 4 errors, 27 skipped
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] Tip: 24 PDFs were blocked by publisher paywalls. Set `pdf_proxy_prefix` in your config to use institutional access (e.g. EPFL: 'https://login.proxy.epfl.ch'), or set `download_pdfs: false` to skip PDFs entirely.
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] Wrote 125 papers to examples/paper_discovery/papers.json
[PaperDiscovery 📄 -- 2026-06-30 15:22:07] Completed in 291.00s#3499552199 part 2 - MultimodalSample integration: - schema: Paper.to_multimodal_sample(pdf_path="") returns a mmore.type.MultimodalSample. Text falls back extracted_text -> abstract -> title -> "". Paper-specific fields (title, authors, year, source, url, search_category, abstract) land in metadata.extra; metadata.processor_type = "paper_discovery" so downstream filters can recognise the origin. - config: new `multimodal_output_file: Optional[str]` knob. - pipeline: when the knob is set, _write_multimodal_jsonl writes a JSONL of MultimodalSample records alongside papers.json. Existing papers.json output is unchanged - additive only. - docs: new "Feeding results into mmore's index / RAG" section and row in the knobs table. - tests: +4 covering the extracted->abstract->title fallback and metadata carriage. #3499643602 - CI wiring: - tests/conftest.py: revert the try/except guard on angchain_milvus. CI's install list includes [index,rag] so the import is guaranteed. - .github/workflows/tests.yml: add paper_discovery to the extras installed on CI. - .github/workflows/pyright.yml: same, so type checks cover the new package.
…SONL - pipeline._write_output writes one JSON object per line. - examples/paper_discovery/config.yaml points at papers.jsonl. - .gitignore updated for papers.jsonl + papers.samples.jsonl. - config.py docstring documents the .jsonl convention. - Docs (paper_discovery.md) show the JSONL shape, workflow diagram and "Feeding results" section updated.
- schema.py: authors: Optional[str] -> Optional[List[str]]
- sources/openalex.py: keep the list comprehension result unjoined.
- sources/arxiv.py: same, using a walrus-filtered comprehension.
- sources/google_scholar.py: same.
- sources/europepmc.py: new _parse_authors() helper - prefer the structured authorList.author[].fullName (available with resultType=core, which we already request); fall back to naive authorString.split(",") only when the structured shape is missing.
- Tests: updated OpenAlex assertion; added 3 tests for the Europe PMC parser covering the structured path, the fallback path, and the empty-input path.
- Docs: sample JSONL line uses ["Ada Lovelace", "Alan Turing"].
JCHAVEROT
left a comment
There was a problem hiding this comment.
As a general comment, please change all the old types from Typing to use the ones available by default with python's latest version. For example Dict --> dict, List --> list, Union --> |, Optional --> | None
Would be great if you make a pass on all docstrings to simplify and make them more dev-friendly for those who will maintain your code later, they are a bit too heavy/long sometimes and not high-level enough (also avoid em-dashes or semi-colons). However it's nice when you document the public functions arguments and return types
| ### Institutional access via EZproxy (recommended) | ||
|
|
||
| Set `pdf_proxy_prefix` in your config to your institution's EZproxy URL. Every paywalled URL will be wrapped through the proxy automatically: | ||
|
|
||
| ```yaml | ||
| pdf_proxy_prefix: "https://login.proxy.epfl.ch" | ||
| ``` | ||
|
|
||
| The proxy handles SAML/Shibboleth authentication and the publisher sees a valid institutional session. | ||
|
|
||
| **Caveat:** the first request through the proxy may redirect to your institution's login page, which a script cannot fill in. The simplest workaround for v1 is to sign in once in a browser to seed the session cookie, then run the pipeline. |
There was a problem hiding this comment.
Seems like its not working for me
When I uncomment the line in the config file I am never redirected to the EPFL login page 🤔
There was a problem hiding this comment.
Thanks for testing this.
EPFL doesn't support this, but other institutions do. I'll leave it as a placeholder for others. For EPFL, you simply need to use the network or be on the VPN.
But you can disable it so it's optional to use it.
| ## 🔌 Feeding results into mmore's index / RAG | ||
|
|
||
| If you plan to index the discovered papers or run RAG over them, you don't need to send them back through `mmore process`. Ask the pipeline to write an extra output file in mmore's canonical `MultimodalSample` shape: | ||
|
|
||
| ```yaml | ||
| multimodal_output_file: examples/paper_discovery/papers.samples.jsonl | ||
| ``` |
|
|
||
| mmore is an open-source, end-to-end pipeline to ingest, process, index, and retrieve knowledge from heterogeneous files: PDFs, Office docs, spreadsheets, emails, images, audio, video, and web pages. It standardizes content into a unified multimodal format, supports distributed CPU/GPU processing, and provides hybrid dense+sparse retrieval with an integrated RAG service (CLI, APIs). | ||
|
|
||
| https://github.com/EPFLiGHT/mmore/pull/319/conflict?name=.github%252Fworkflows%252Fpyright.yml&ancestor_oid=e880609d900c8d1ca113e404d0d60dc5604ddc65&base_oid=652683914c68996968fa4d333618d000927a80eb&head_oid=104f07055d58c2e6c1420d89649cd5de5e57f32e |
There was a problem hiding this comment.
seems like a nasty left over from a merge conflict resolution, don't forget to remove it
There was a problem hiding this comment.
Came in with the v2 merge. Removed.
| # PDF text extraction goes through mmore.process.PDFProcessor for | ||
| # consistency with the rest of mmore. The fast path uses PyMuPDF | ||
| # (no marker models loaded), so runtime cost is small even though | ||
| # the install pulls the full `process` stack. |
There was a problem hiding this comment.
We don't need this much detail in the pyproject.toml
| # PDF text extraction goes through mmore.process.PDFProcessor for | |
| # consistency with the rest of mmore. The fast path uses PyMuPDF | |
| # (no marker models loaded), so runtime cost is small even though | |
| # the install pulls the full `process` stack. |
There was a problem hiding this comment.
Is it wanted that you uploaded this file ?
Maybe a smaller one as an example is better, this one is 2.7MO big
There was a problem hiding this comment.
Not intentional, they slipped in before the ignore rules were right.
There was a problem hiding this comment.
Same for this one, did you push it so that we have an example of the results format?
There was a problem hiding this comment.
Nice to have this standardized pattern with the search() function for the sources 👍
| user_agent: str = "mmore-paper-discovery/1.0", | ||
| max_pages: int = 1, |
There was a problem hiding this comment.
both these arguments are unused, might be a careless mistake
There was a problem hiding this comment.
They're accepted because get_adapter() passes the same kwargs to every adapter, but scholarly drives its own HTTP transport and pagination so neither is forwarded. Made that explicit rather than leaving them silently dropped.
| def _safe_year(value): | ||
| try: | ||
| return int(value) if value else None | ||
| except (TypeError, ValueError): | ||
| return None |
There was a problem hiding this comment.
You defined multiple time private safe_year() or coerce_year() or coerce_int() functions in the sources arXiv.py, europepmc.py, google_scholar.py, openalex.py, they are all doing intrinsically the same thing avoid redundancy by defining helper functions you could reuse
| word = row.get("word") or row.get("WORD") | ||
| raw = row.get("synonyms") or row.get("SYNONYMS AND NEAR SYNONYMS") or "" |
There was a problem hiding this comment.
Why having these or row.get("WORD") and or row.get("SYNONYMS AND NEAR SYNONYMS")? You set the synonyms.jsonl to have the format {"word": ..., "synonyms": [...]} in examples/paper_discovery/config.yaml, i guess no need to support other structures
| try: | ||
| from tqdm.auto import tqdm | ||
| except ImportError: # pragma: no cover | ||
| # tqdm isn't in paper_discovery's hard deps - if missing, run silently | ||
| # but still expose the methods we touch (set_postfix). | ||
| class _NoTqdm: | ||
| def __init__(self, iterable, **_kwargs): | ||
| self._iterable = iterable | ||
|
|
||
| def __iter__(self): | ||
| return iter(self._iterable) | ||
|
|
||
| def set_postfix(self, **_kwargs): | ||
| pass | ||
|
|
||
| def tqdm(iterable, **kwargs): # type: ignore[no-redef] | ||
| return _NoTqdm(iterable, **kwargs) |
There was a problem hiding this comment.
Avoid the pragma and the type flags, this is anti-pattern because it could hide problems instead of solving them
There was a problem hiding this comment.
Also tqdm should always be found, normally no need this check, or maybe it was missing and you had errors for other reasons?
| if TYPE_CHECKING: | ||
| from ..type import MultimodalSample | ||
|
|
||
| SourceName = Literal["arxiv", "openalex", "europepmc", "google_scholar"] |
There was a problem hiding this comment.
I think a (Str, Enum) class would be cleaner here
| from .paper_discovery.pipeline import PaperDiscoveryPipeline | ||
| from .utils import load_config | ||
|
|
||
| load_dotenv() |
There was a problem hiding this comment.
do you actually need a .env ?
In the repo its more common to export environment variables for the secrets
| # the install pulls the full `process` stack. | ||
| "mmore[process]", | ||
| "requests>=2.31", | ||
| "beautifulsoup4>=4.13", |
There was a problem hiding this comment.
beautifulsoup4 is already in mmore[process], I think no need to redefine it here
There was a problem hiding this comment.
Removed.
Flagging one related change so it doesn't look inconsistent: I'm adding tqdm to the extra in the same round. The difference is that bs4 is declared in process, whereas tqdm is declared nowhere and only reaches us transitively through transformers. Since we import it directly, it should be declared.
- README: remove a stray GitHub conflict-resolution URL left behind by the v2 merge (#3648952822). - europepmc: fix `docementStyle` -> `documentStyle`. The misspelled key never matched, so the PDF-URL preference silently never fired and every Europe PMC result fell through to its landing page. - boolean.load_synonyms: drop the `WORD` / `SYNONYMS AND NEAR SYNONYMS` fallbacks. We document exactly one input shape, the alternates were dead weight from a spreadsheet-import experiment. - google_scholar: resolve the unused `user_agent` / `max_pages` ctor args explicitly instead of accepting and dropping them. - pyproject: drop `beautifulsoup4`, already a direct dep of `mmore[process]` which `paper_discovery` depends on (#3649954641). - examples: untrack `papers.json` (2.7 MB) and `papers.samples.jsonl` (6.4 MB). `papers.json` also predates the JSON -> JSONL switch. A 3-line `papers.sample.jsonl` is committed instead so the output format is still visible in-repo (#3649860650, 3649862748).
…red year helper - pipeline: drop the tqdm try/except shim and switch to `mmore.ux.progress`, the shared Rich progress bar introduced in EPFLiGHT#333 and already used by the index, post-process, colvision, and websearch pipelines(#3649937401, #3649940848). This removes the `# pragma: no cover` and the `# type: ignore[no-redef]`, which were the only two suppressions in the module, and needs no new dependency since `mmore.ux` is core. Counters now render through `set_postfix_str` as `ok=70 cache=2 paywall=24 err=4`. - schema: `SourceName` is now a `(str, Enum)` instead of a `Literal` (#3649950643). The `str` mixin keeps JSON output unchanged, so `papers.jsonl` still carries `"source": "arxiv"` rather than `"SourceName.ARXIV"`. All four adapters use the members. Three tests cover the round-trip, string equality, and that the enum matches `REGISTRY` exactly. - sources: collapse four near-identical private year/int coercion helpers (`_coerce_year` x2, `_safe_year`, `_coerce_int`) into `coerce_year` in the new `sources/_utils.py` (#3649910611). They differed in input shape, not intent, so the shared helper handles ints, year strings, and ISO dates. Europe PMC's two-key lookup is now a `first_year` call rather than its own function. - pdf: fix two pre-existing pyright errors in `_find_pdf_link` where bs4 types an href as `str | list[str]`.
- Remove the hostname from all seven locations. Keep `pdf_proxy_prefix` as a generic knob. - Add `_looks_like_login_page`. A proxy needing interactive sign-in answers 200 OK with its login form, which the pipeline was filing as a silent skip. Runs now warn: "N downloads returned a sign-in page instead of a PDF. This pipeline cannot log in for you." - Rewrite the paywall log line, which pointed at the bogus URL. It now explains that 403s are usually User-Agent blocking and that the VPN does not always help. - Rewrite the Paywalled PDFs docs around the distinction that matters: no subscription versus have-access-but-blocked-as-a-bot. Covers both institutional access models and what to set for each. - Add scripts/diagnose_pdf_access.py, which prints the status, redirect chain, content type, sign-in-page verdict and resulting bucket for a URL, either given directly or pulled from a previous run's failures. This is what surfaced the DNS failure.
…nused dotenv - Docstrings trimmed to be shorter and more high-level, keeping the Args/Returns blocks on public functions. Removed all em-dashes and prose semicolons. - removed `load_dotenv()` from run_paper_discovery (r3649953527). This pipeline reads no environment variables, every source it queries is anonymous, so the call was copied from entrypoints that do need API keys.

No description provided.