Auto-fetch all SIR ZIP URLs from Frontex PRD

[aborruso]

Goal

Automate discovery and collection of all ZIP URLs from the Frontex Public Register of Documents (SIR section), instead of maintaining zip_urls.txt manually.

Implementation

Script: fetch_sir_zip_urls.py

How it works:

1. Iterates all listing pages at https://prd.frontex.europa.eu/?form-fields[document-tag][0]=409&form-fields[paged]=N
2. Extracts document-post-id from each entry
3. Fetches each dialog page (dialog.php?card-post-id=2722&document-post-id=<ID>) to extract ZIP URL from <select option value>
4. Appends only new URLs to zip_urls.txt (idempotent, safe for cron)

Usage:

# Dry run (no writes)
uv run python3 fetch_sir_zip_urls.py --dry-run

# Append new URLs to zip_urls.txt
uv run python3 fetch_sir_zip_urls.py

TODOs

• Run full scrape (all 20 pages, 96 docs) and update zip_urls.txt
• Add to periodic cron/schedule (e.g. weekly) to catch new publications
• Optionally: add --date-from filter to limit to recent documents

[aborruso]

Implementato: fetch_sir_zip_urls.py raccoglie tutti gli URL (ZIP+PDF) e i metadati da PRD Frontex, scrivendo zip_urls.txt e sir_documents.jsonl. 96 documenti, idempotente.