[evals] Add ASR and OCR-noisy text PPL evals

[dlwh]

🤖 Part of #5005.

Description

Add PPL/gap eval sets for noisy text produced by ASR, OCR, screen scraping, and vision-to-text tools. Agents increasingly consume imperfect transcriptions rather than clean web text.

Initial sources:

• HypR ASR hypotheses: https://www.isca-archive.org/interspeech_2024/wang24j_interspeech.html
• FLEURS transcripts: https://huggingface.co/datasets/google/fleurs
• Common Voice: https://www.mozillafoundation.org/common-voice/platform-and-dataset/
• OCR-D ground truth: https://ocr-d.de/gt_structure_text/
• RETAS OCR evaluation data: https://ciir.cs.umass.edu/downloads/ocr-evaluation/
• ReadingTimeMachine OCR post-correction: https://github.com/ReadingTimeMachine/ocr_post_correction

Definition of Done

• Add at least one ASR-noise source and one OCR-noise source.
• Preserve noisy text and clean reference text where available.
• Report clean-vs-noisy BPB deltas.
• Document licensing/access constraints, especially for Common Voice.

[claude]

🤖 Plan:

• Read AGENTS.md, fix-issue skill, pull-request skill
• Fetch issue [evals] Add ASR and OCR-noisy text PPL evals #5097 + parent [evals] Build pretraining checkpoint confidence portfolio #5005 + sibling [data] Source licensed ASR and OCR-noisy text corpora #5102
• Duplicate-work preflight (no overlapping PRs; [data] Source licensed ASR and OCR-noisy text corpora #5102 covers ingestion)
• Map existing PPL/BPB registry + gap reporting
• Post Research + Proposed Approach comment
• Await human review before implementation (scope > 100 LOC; multi-module)

Research

Marin already has a metadata-first diagnostic PPL registry (long_tail_ppl.py) and a two-model BPB gap pipeline, but it has no notion of paired clean/noisy references, which is the new signal #5097 asks for (same-text, same-model, noisy-minus-clean BPB). OCR surfaces exist as loose slices (TextOCR, OCR-VQA) with no reference pairing, and there is no ASR coverage at all.

Sibling #5102 owns data ingestion/licensing for the same sources, so #5097 should stay on the eval side — register slices, wire delta reporting, pull raw paths from whatever #5102 lands.

Relevant code

• long_tail_ppl.py#L29-L65 — LongTailPplFamily / LongTailPplSlice registry primitives
• long_tail_ppl.py#L126-L143 — existing unpaired OCR slices
• perplexity_gap.py#L37-L80 — RawTextEvaluationDataset + raw_text_dataset (supports HF or file path, single text_key)
• eval.py#L454-L468 — BPB formula used by the gap finder
• test_long_tail_ppl.py — registry test pattern to extend

Proposed Approach

The change has two moving parts. I want sign-off on the shape before implementation.

1. Registry: paired slices. Add a LongTailPplPair dataclass that groups two LongTailPplSlices sharing a doc-id key, plus a new NOISY_TRANSCRIPTION family. Each pair records which side is noisy and which is clean; single-sided sources (FLEURS transcripts with no paired clean, OCR-D gold with no paired dirty) stay as plain slices. Initial entries:

• ASR pairs: HypR (ASR hypothesis ↔ reference transcript), Common Voice (ASR output ↔ verified sentence) — gated behind HF_TOKEN / CV account per [data] Source licensed ASR and OCR-noisy text corpora #5102's license inventory.
• OCR pairs: RETAS (OCR output ↔ ground truth), ReadingTimeMachine (OCR output ↔ corrected). OCR-D contributes gold-only.
• Single-sided: FLEURS clean transcripts (clean-only reference for ASR surface form).

Pair key layout: long_tail_ppl/noisy_transcription/hypr/{noisy,clean} so both sides flow through the existing long_tail_raw_validation_sets() dict untouched.

2. Reporting: clean-vs-noisy delta. New executor step clean_noisy_bpb_delta that:

• runs the existing BPB eval over both sides of each pair with one model,
• asserts equal doc counts + same doc-id ordering (fail-fast per AGENTS.md error rules),
• emits delta_bpb = bpb_noisy - bpb_clean per pair to W&B and a per-family table.

This is orthogonal to default_model_perplexity_gap (which is model-vs-model). I'll put it in lib/marin/src/marin/evaluation/clean_noisy_gap.py alongside perplexity_gap.py.

Licensing handling. Keep the notes field of each LongTailPplSlice populated with the license string and whether the side is training-appropriate or eval-only (consistent with how #5102 will tag sources). Common Voice CC0-with-attribution + account-gated download is called out explicitly in the HypR/CV slice notes.

Tests. Extend tests/evals/test_long_tail_ppl.py: deterministic path assertions for the new slices, pair-key invariants (noisy/clean both resolve, share doc-id key), markdown renderer includes license strings. Add tests/evaluation/test_clean_noisy_gap.py that drives clean_noisy_bpb_delta against a fixture with two tiny JSONLs (one clean, one with simulated character substitutions) and asserts delta_bpb > 0 — integration-style, no mocks.

Scope estimate. ~250 LOC registry + reporting + tests. Defers actual data download to #5102. Does not modify training mixtures.

Open questions for the reviewer:

• New family NOISY_TRANSCRIPTION, or extend WEB_MARKUP_IMAGE_TEXT? I lean new family because ASR belongs nowhere in the current taxonomy.
• Is W&B the right reporting sink, or should this feed a JSON artifact consumed by the epic-5005 confidence report schema directly?
• Should single-sided slices (FLEURS, OCR-D gold) ship now or wait until a paired source for them exists?