[evals] Add paraphrase and translation robustness PPL evals

[dlwh]

🤖 Part of #5005.

Description

Add paired PPL/gap evals that test whether Marin models equivalent content across surface forms instead of only memorized text forms.

Initial sources:

• PAWS / PAWS-X: https://github.com/google-research-datasets/paws
• ParaNMT-50M: https://aclanthology.org/P18-1042/
• ParaSCI: https://arxiv.org/abs/2101.08382
• FLORES-200: https://huggingface.co/datasets/facebook/flores
• Project CodeNet for code translation-style pairs: https://github.com/IBM/Project_CodeNet

Score both unconditional BPB over variants and conditional likelihood of a target variant given the source variant.

Definition of Done

• Add at least one paraphrase source and one translation source.
• Define a stable text linearization for paired examples.
• Report variant-level deltas, not only aggregate BPB.
• Keep held-out splits separate from any data-mixture iteration loop.

[claude]

🤖 Plan:

• Fetch issue and related context ([evals] Build pretraining checkpoint confidence portfolio #5005, [levanter] Land raw perplexity gap reports #4934, [evals] Add base-model PPL eval sets for chat and agent data #4963)
• Research existing PPL / gap-finder infra
• Duplicate-work preflight — no parallel PRs/issues
• Post Research + Proposed approach (scope is a design change, not a <100 LOC bug fix — deferring implementation for human review)

Research

No paired-variant eval exists today. The gap-finder pipeline (perplexity_gap.py → GapReportBuilder) scores a single document per example and aggregates bpb per dataset; it does not tokenize pairs jointly, does not mask a prompt region, and does not emit per-pair deltas. GrugLmExample.from_prompt_and_completion supports conditional loss at the training-data layer but is not wired into the eval harness. The metadata-first long_tail_ppl.py registry is the closest pattern for how this eval should land (registry → raw_text_dataset() → default_model_perplexity_gap()), but its RawTextEvaluationDataset holds one source, not a pair.

Relevant code

• perplexity_gap.py#L38-L45 — RawTextEvaluationDataset (single-source)
• perplexity_gap.py#L185-L207 — dataset → DatasetComponent boundary
• levanter/analysis/perplexity_gap.py — GapReportBuilder, per-byte aggregation, pattern buckets
• levanter/data/text/examples.py — GrugLmExample.from_prompt_and_completion (prompt-mask primitive, unused in eval)
• long_tail_ppl.py#L41-L64 — metadata-first slice registry with tags field

Proposed approach (design)

Scope is a new eval family spanning levanter + marin + experiments + two dataset downloaders; ~600–900 LOC. Deferring implementation until a human signs off on the shape below.

Linearization. One stable text template per pair, chosen so unconditional and conditional scoring share the same tokenization. Paraphrase: "{source}\n{variant}". Translation: "{src_lang}: {source}\n{tgt_lang}: {variant}". source is the conditioning prefix; variant is the scored region. Exact template is versioned in the cache key so re-running is deterministic and regressions are attributable.

Data model. Add PairedRawTextEvaluationDataset(source_field, variant_field, pair_kind: PairKind, held_out: bool, tags) next to RawTextEvaluationDataset (not a union — new dataclass). PairKind is a StrEnum {paraphrase, translation}. The held_out: bool is asserted at registration time and propagated as a tag holdout:paired_ppl_v1; a top-level guard in experiments/defaults.py refuses to include any dataset with that tag in data-mixture steps.

Scoring. Extend _ModelRunner.score_texts to accept prompt_length per example and drop prompt-region losses from the accumulator before bpb normalization. For each pair we emit four numbers: uncond_bpb(source), uncond_bpb(variant), cond_bpb(variant | source), and their per-model delta. The existing byte-span machinery handles the prompt_length → byte-offset mapping; we reuse GrugLmExample.from_prompt_and_completion to build examples.

Report. GapReportBuilder gains per-pair rows (pair_id, variant_kind, uncond_delta, cond_delta, agree) alongside existing dataset-level rollups. Added aggregate metrics: mean signed delta per pair, pair-consistency rate (% pairs where model A beats model B on both source and variant), and worst-pair listings bucketed by PairKind. No change to existing single-doc code path.

Datasets (starter). PAWS (paraphrase, label=1 only, en) + FLORES-200 devtest (translation, en↔{de,zh,sw,hi}). Both small enough to live as HF sources via HfDatasetSourceConfig; no GCS mirroring needed on day one. ParaNMT / ParaSCI / CodeNet as follow-ups — they require staging and are out of scope for the first PR.

Registry. New experiments/evals/paired_ppl.py with a PairedPplSlice dataclass mirroring LongTailPplSlice (family ∈ {paraphrase, translation}, held-out flag, source URL, surface form, pair template version). paired_raw_validation_sets() returns dict[str, PairedRawTextEvaluationDataset] for plug-in to default_model_perplexity_gap().

Split discipline. Paired slices are held-out by construction: PAWS dev+test, FLORES devtest only. Registry rejects held_out=False on first pass. Enforcement is a lint-style check in the executor that scans any mixture config for datasets carrying the holdout:paired_ppl_v1 tag — same pattern as existing tag plumbing on LongTailPplSlice.

Tests. One levanter unit test that a two-document batch with prompt-mask recovers known losses against a tiny model; one marin integration test that paired_raw_validation_sets() registers both families, fails closed on held_out=False, and round-trips through _to_dataset_component. No mocks; both use the existing tiny-model/tiny-tokenizer fixtures.

Open questions for @dlwh

1. Tokenizer parity across variants in translation pairs — accept per-language bpb noise, or require byte-level normalization before comparison?
2. Is aligning paraphrase pairs by length (reject pairs with |len_a − len_b| > τ) in scope, or is variant-length a feature we want to measure?
3. Do we want conditional scoring required, or can v1 ship unconditional-only and add conditional in a follow-up? (Unconditional only is ~300 LOC; conditional adds the _ModelRunner change and roughly doubles the PR.)