Scope the fetch-stage file-list planner for monthly-partitioned wikis (enwiki)

[schiste]

Summary

Scope the work needed to implement the fetch-stage file-list planner for monthly-partitioned mediawiki_history dumps (enwiki, wikidatawiki, commonswiki), which is currently an explicit, tested rejection rather than a real implementation gap analysis.

Context

build_file_list in src/fetch.rs only knows how to enumerate one dump file per year, for wikis in YEARLY_WIKIS (src/fetch.rs:64-69). For wikis in MONTHLY_WIKIS (src/fetch.rs:75, currently ["enwiki", "wikidatawiki", "commonswiki"]), it bails immediately with "Monthly-partitioned wikis (enwiki, etc.) are not yet supported. Use yearly wikis." (src/fetch.rs:658-662) before any network or disk work — there is no monthly-filename-enumeration branch at all. This is deliberate and tested (src/fetch.rs:1846-1849 build_file_list_rejects_monthly_wikis, tests/cli.rs:33-49 binary_entrypoint_reports_monthly_fetch_error), and docs/architecture.md:53 confirms: "Monthly-partitioned giant wikis are still rejected by the fetch planner. That is deliberate until the raw-file planning for those projects is implemented properly."

Downstream stages (ingest, compute, merge) have no enwiki/monthly-specific logic either way — they're wiki-agnostic once handed files, so this fetch-side planner is, as far as the codebase shows, the one concretely missing piece for monthly-partitioned wikis to work end-to-end.

Separately: deploy/toolforge/README.md:15 states "enwiki is out of scope for Toolforge entirely," and the production refresh job runs with 6Gi RAM / 1 CPU against a small NFS quota (deploy/toolforge/jobs.yaml:31-38), sized around nlwiki/frwiki. This ticket is about the fetch planner specifically — actually running enwiki end-to-end would additionally require a deployment-target decision that's out of scope here (see below).

Proposed scope (to be defined by this ticket)

• Determine the actual upstream filename pattern for monthly-partitioned mediawiki_history dumps (probe https://dumps.wikimedia.org/other/mediawiki_history/<version>/<wiki>/ the same way YEARLY_WIKIS was determined, per the comment at src/fetch.rs:59-63) and confirm it's consistent across all three MONTHLY_WIKIS entries.
• Design the month-enumeration branch in build_file_list (parallel to the existing per-year branch at src/fetch.rs:670-672), including how it interacts with WIKI_ECON_MAX_SNAPSHOT_LAG_MONTHS (src/fetch.rs:41-42) and the existing resume/range-capable download support.
• Assess whether FETCH_DISK_HEADROOM_MARGIN_BYTES (src/fetch.rs:44-49, currently sized off frwiki's ~31GB transient peak) and the disk-headroom preflight (check_disk_headroom, src/fetch.rs:1438) scale sensibly for a wiki with dozens-to-hundreds of monthly files instead of a handful of yearly ones.
• Assess whether the default serialized fetch concurrency (FETCH_MAX_PARALLELISM = 1, src/fetch.rs:20-24, deliberately forced down after 429s from WMF at 4 concurrent) is still the right default given monthly partitioning means many more, smaller files — more files to serialize could meaningfully change wall-clock fetch time even without touching the rate-limit tradeoff itself.
• Identify whether wikidatawiki and commonswiki — not Wikipedia language editions, currently only in MONTHLY_WIKIS so they hit the same rejection path (src/fetch.rs:71-74) — should be in scope for this planner work at all, or whether this should focus on enwiki only for now.
• Flag (not necessarily solve) what it would take to validate ingest/compute/merge actually hold up at monthly-wiki scale once real files exist, since those stages currently have zero test coverage or benchmarking at anything beyond nlwiki/frwiki-scale data.

Open questions

• Is there a target wiki to validate against once the planner exists (enwiki itself, or a smaller monthly-partitioned wiki as a proxy — though currently only enwiki/wikidatawiki/commonswiki are monthly)?
• Does this ticket's output block on, or run independently of, a deployment-target decision (Toolforge is explicitly out of scope for enwiki per deploy/toolforge/README.md:15)?
• Should disk-headroom and concurrency defaults become per-wiki-class config (yearly vs. monthly) rather than global constants, given they were tuned around yearly-wiki assumptions?

Out of scope

• Actually running/backfilling enwiki, wikidatawiki, or commonswiki.
• Choosing or provisioning a deployment target capable of hosting a monthly-partitioned giant wiki (Toolforge is explicitly excluded today).
• Ingest/compute/merge implementation changes, unless this scoping work uncovers a concrete blocker in those stages.