[evals] Add diagnostic log-stream PPL eval sets

[dlwh]

🤖 Part of #5005.

Description

Add held-out PPL/gap eval sets for diagnostic streams that agents read during debugging: CI logs, compiler/linker output, pytest failures, stack traces, package install errors, and system/application logs.

Initial sources:

• GHALogs: https://zenodo.org/records/14796970
• LogChunks: https://papertalk.org/papertalks/24713
• LogHub: https://github.com/logpai/loghub
• Sanitized Marin GitHub Actions/Iris/Zephyr logs as eval-only heldout data.

Keep this separate from source-code evals. These are not code files; they are failure/diagnostic registers.

Definition of Done

• Add raw eval dataset builders for at least two public log sources.
• Add a small sanitized Marin-internal eval slice, or document why it is deferred.
• Register slices outside the default validation set list.
• Include worst-doc artifacts in gap reports.
• Document leakage/contamination handling for any Marin-owned logs.

[claude]

🤖 Plan:

• Fetch issue + epic context ([evals] Build pretraining checkpoint confidence portfolio #5005, [levanter] Land raw perplexity gap reports #4934, [data] Add a raw technical-text tranche beyond GitHub #4961, [evals] Add base-model PPL eval sets for chat and agent data #4963)
• Research existing long-tail PPL registry pattern
• Confirm no overlapping PR exists
• Add DIAGNOSTIC_LOGS family + slices to experiments/evals/long_tail_ppl.py
• Add tests in tests/evals/test_long_tail_ppl.py
• Run pre-commit and pytest (7/7 passing locally; 2 pre-existing GCS-auth failures in test_lm_eval.py unrelated)
• Open PR — [evals] Add diagnostic-log long-tail PPL slices #5104
• CI green

Research

Pattern is established by #5061 and #5057: metadata-first long-tail PPL slices in experiments/evals/long_tail_ppl.py under a new LongTailPplFamily value, registered outside the default validation set. No data is downloaded; entries record source URL, surface form, and a deterministic GCS-relative path for later ingestion. Materialization goes through raw_text_dataset() in lib/marin/src/marin/evaluation/perplexity_gap.py.

Relevant code

• long_tail_ppl.py#L29-L38 — family enum
• long_tail_ppl.py#L19-L26 — issue constants
• long_tail_ppl.py#L88-L447 — slice tuple
• perplexity_gap.py#L38-L80 — raw_text_dataset() materialization
• test_long_tail_ppl.py — registry tests

Proposed Fix

Add a DIAGNOSTIC_LOGS family + DIAGNOSTIC_LOGS_ISSUE = 5093, then register 12 metadata-only slices covering the seven failure-register surfaces in the DoD:

• GHALogs (Zenodo 14796970) and LogChunks (papertalk 24713) — public CI / failure log corpora.
• LogHub subsets apache, linux, hdfs, openssh, thunderbird — system / application registers.
• pytest_failures, compiler_linker_output, stack_traces, package_install_errors — surface-form slices with deterministic raw paths; ingestion deferred per epic phasing.
• marin_internal_logs_sanitized — held-out internal slice. Notes flag the leakage policy (mirror only after PII / secret / hostname / bucket / token scrub; never include in any training mixture).

Slices stay outside default_raw_validation_sets (registry-only). Worst-doc artifacts surface in gap reports when an experiment opts in via long_tail_raw_validation_sets(family=LongTailPplFamily.DIAGNOSTIC_LOGS).

PR: #5104 (+174 LOC, metadata + tests only).