Write a PMID manifest on corpus runs so drops_since_previous can ever run

[gaurav]

Note

Two of the three bullets below are done on PR #9; the third is what keeps this open.

• Add --manifest to the documented validate invocation — done. 05-validate.sbatch always passes it, so it is no longer something to remember, and the root README.md example matches.
• Decide where the manifest lives — done: data/manifests/pmids-<date>.txt.gz, outside the export directory. The reason given below is not quite the real one: the export's stale sweep only globs pubmed_metadata_*.ndjson*, so a manifest under data/json/ survives the next export — and then looks current while describing the previous corpus, which is worse than being deleted. The sharper reason is the flags: with one shared name the obvious next invocation passes the same path as both --previous-manifest and --manifest, overwriting the file it just diffed against and making run N vs N+2 impossible. Dating it fixes both. The script also picks the newest earlier manifest automatically, skipping today's so a same-day re-run cannot diff against itself.
• The first run produces no comparison — still true, and it is why this stays open. The imminent rebuild writes the first real manifest and will report drops_since_previous as skip; the run after it is the first that can catch a silent drop. Nothing to do during this run beyond keeping data/manifests/ around.

Verified end to end on a small export: the manifest lands at the nested path, and a second run with one PMID removed reports 1 dropped (0 explained by a recorded deletion), 1 added.

---

validate's drops_since_previous check compares this export's PMID set against a previous export's manifest, and it is the only check that can catch two same-sized exports whose contents differ — a coverage count cannot. It has never run on real data, because no corpus-scale export has written a manifest for a later run to diff against.

The manifest is opt-in (--manifest PATH), written from the PMID set the structure check already holds, so it costs one sort and one write rather than another pass over the shards. The 2026-08-05 corpus run did not pass it, so the next run has nothing to compare against either — the check will report skip again, and will keep doing so until one run writes a manifest and a subsequent run reads it.

What to do:

• Add --manifest to the documented validate invocation in slurm/README.md (and the README's validate example), so it is written by default on a cluster run rather than remembered.
• Decide where the manifest lives. Writing it into the export directory (data/json/pmids.txt.gz) keeps it with the data it describes, but the export directory is republished in place by the next export — so the file needs copying out, or writing somewhere that survives.
• The first run after that produces no comparison; the run after it is the one that finally exercises the check.

Worth doing before the next full rebuild, since that is the run whose PMID set someone will want to diff later.