Skip to content

Conversation

@breuleux
Copy link
Member

@breuleux breuleux commented Aug 1, 2025

We can now define rules to redirect certain URLs to various fetchers:

  • Normal requests
  • Cached requests (SQLite)
  • Requests through cloudscraper for CloudFlare-protected pages
  • A scraping proxy service (ScraperAPI)
  • A sequence of fetchers (e.g. try without proxy, fallback to the next)

Also: code for locating paper PDFs and downloading papers

@breuleux breuleux merged commit 4e8d06e into mila-iqia:v3 Aug 6, 2025
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants