New request: medRxiv

[Popolechien]

• Website URL: https://www.medrxiv.org/
• License: **Creative Commons **
• Desired ZIM Title: medRxiv
• Desired ZIM Description: the preprint server for med science
• Desired ZIM Icon –png (URL or attach one): n/A
• Language (ISO 639-3): eng
• Is this a MediaWiki?: no

Note: the idea at this stage is to test doability. In particular, I'd be interested to know if we can easily-ish select for a particular content category (e.g. Addiction medicine), and what the requirements for a dedicated scraper would be.

API is available at https://api.biorxiv.org/
and dumps are generated monthl, see https://www.biorxiv.org/tdm

They're using Cloudflare, so let me know if we need to request whitelisting.

[benoit74]

I would like to see this happen, and it seems doable, custom scraper needed.

Some insight:

• this is not feasible with zimit, both for technical reasons (website seems a bit too complex for zimit to work well) and for "legal" reasons (medRxiv explicitly requests we do not scrape their website but use their APIs and dumps)
• the API / dumps of full texts are sufficient to create a "dedicated" scraper ; I would encourage we begin to build something which is capable to be reused for any "scientific source", in order to not reinvent the wheel every time ; maybe even gutenberg could make it into this scraper, we are always speaking about authors, abstracts, full text, ...
• filtering for a particular category and licensing is totally possible
• dump formats are a bit unclear at this stage, looks like they are mostly oriented towards "months" while we might want only a given set of articles for a given category / licensing ; tbc if it raises a concern or not (I can't access them for now, see point below)

Some remarks:

• to access full text we will have to pay for it ; fee is expected to be very moderate, but we will have to take care of that ; we will need to create an AWS account with billing activated
• I'm curious / concerned about the final size of the ZIMs ; even if we filter by category (like we have one ZIM per category), I would like to check final size is feasible / practical
• I'm concerned about the fact that we will rebuild (and pay again for) mostly the same data over and over again, because what has been published is not going to change ; to be checked if we need some sort of optimization / logic / splitting for that, again it is a matter of processing time, price, and final ZIM size ; tbc if we have any benefit of storing the dumps on our storage once retrieved a first time