[evals] Add GH Archive structured-output PPL evals

[dlwh]

🤖 Part of #5005.

Description

Add PPL/gap eval sets over real GitHub API event payloads from GH Archive. This covers structured output as consumed by tools: nested JSON, IDs, URLs, timestamps, actor/repo metadata, issue/PR payloads, and event schemas.

Primary source:

• GH Archive: https://www.gharchive.org/

Related structured-output sources for later comparison:

• APIs.guru OpenAPI directory: https://github.com/APIs-guru/openapi-directory
• SchemaStore: https://www.schemastore.org/

This should be separate from generic JSON evals because agent/tool workflows often consume GitHub-shaped payloads directly.

Definition of Done

• Build a small held-out GH Archive slice with date-based selection.
• Split metrics by event type, at least PushEvent, PullRequestEvent, IssuesEvent, IssueCommentEvent, and WorkflowRunEvent if present.
• Mask or bucket fields that should not dominate metrics, such as long IDs and hashes.
• Keep this out of default validation sets.

[claude]

🤖 Plan:

• Fetch issue context ([evals] Add GH Archive structured-output PPL evals #5098, epic [evals] Build pretraining checkpoint confidence portfolio #5005, sibling [data] Add GH Archive JSON events to pretraining data #5099)
• Map existing PPL-gap eval infrastructure
• Duplicate-work preflight (no prior PRs; see overlap note with [data] Add GH Archive JSON events to pretraining data #5099 below)
• Post Research + Proposed Approach (scope >100 lines → analysis, not PR)

Research

Marin already has the full scaffolding to run raw PPL/gap evals over a dict of RawTextEvaluationDataset, with hierarchical tag-based slicing that is a direct fit for per-event-type metrics. What's missing is (a) a GH Archive downloader/normalizer and (b) a registry declaring one dataset per event type. No Levanter changes required — UrlDataSource already infers .jsonl.gz and GapReportBuilder.register_dataset already expands slash-separated tags into hierarchical groups.

Relevant code

• perplexity_gap.py#L38-L80 — RawTextEvaluationDataset + raw_text_dataset() entry point
• perplexity_gap.py#L158-L181 — GapReportBuilder.register_dataset builds tag hierarchy
• exp1600_uncheatable_evals.py#L80-L86 — canonical "downloader step → raw validation sets" pattern
• uncheatable_eval.py#L228-L269 — JSON → normalized {id, text, source} writer into .jsonl.gz
• long_tail_ppl_runnable.py#L20-L98 — RunnableLongTailPplSlice registry-style pattern with hierarchical tags

Overlap with #5099: #5099 asks for a GH Archive downloader/normalizer for pretraining data with "train/dev/test date splits before ingest" and masking. The eval slice in #5098 needs the same ingest + masking; only the registration layer differs. A single downloader under lib/marin/src/marin/datakit/download/gh_archive.py, parameterized by date range and event types, can satisfy both. Recommend landing the downloader with #5099 and scoping #5098 to the eval registry + per-event-type splits on top of it.

Proposed Approach

Minimally disruptive, two-layer addition (est. ~350-450 LOC + tests; no Levanter changes):

1. Downloader — lib/marin/src/marin/datakit/download/gh_archive.py (shared with #5099)

• Fetches hourly dumps from https://data.gharchive.org/YYYY-MM-DD-H.json.gz over a configurable UTC date range. Gharchive publishes one gzipped JSONL per hour; rigging.filesystem.open_url(..., compression=\"gzip\") streams them, mirroring the uncheatable_eval flow.
• Shards output by event_type: gs://marin-.../raw/gh_archive/{date_range}/{EventType}/part-{NNN}.jsonl.gz.
• Normalizer emits one record per event: {\"id\": event.id, \"text\": <serialized payload>, \"event_type\": ..., \"created_at\": ..., \"source\": \"gh_archive:<date_range>\"}. text is the payload re-serialized as compact JSON (json.dumps(payload, sort_keys=True, ensure_ascii=False)) so we evaluate PPL on the exact surface form consumers see.
• Masking: replace id/sha/node_id/url ID suffixes with stable placeholders (<ID>, <SHA>) and bucket timestamps to <TS> before serialization. Keeps byte/pattern-bucket metrics (URL/number/hex) meaningful; per Definition of Done these fields shouldn't dominate. Configurable allow/deny list of JSON paths.
• License: GH Archive is CC-BY-4.0; governance gate belongs in [data] Add GH Archive JSON events to pretraining data #5099.

2. Eval registry — experiments/evals/exp_gh_archive_ppl.py

GH_ARCHIVE_EVENT_TYPES = (\"PushEvent\", \"PullRequestEvent\", \"IssuesEvent\",
                          \"IssueCommentEvent\", \"WorkflowRunEvent\")

def gh_archive_raw_validation_sets(
    *, gh_archive_raw: ExecutorStep = gh_archive_eval_slice,
) -> dict[str, RawTextEvaluationDataset]:
    return {
        f\"gh_archive/{event_type}\": raw_text_dataset(
            gh_archive_raw.cd(f\"{event_type}/*.jsonl.gz\"),
            text_key=\"text\",
            tags=(\"gh_archive\", \"gh_archive/structured_output\", event_type),
        )
        for event_type in GH_ARCHIVE_EVENT_TYPES
    }

Tags (\"gh_archive\", \"gh_archive/structured_output\", \"PushEvent\") produce three slices in the gap report: overall gh_archive, the gh_archive/structured_output group (for comparison with future JSON/OpenAPI/SchemaStore slices), and the per-event-type leaf. This satisfies "split metrics by event type" from the DoD without new report code.

3. Held-out split

• Eval slice = a single fixed date window (e.g., last 48h of a past month, chosen so the same window is excluded from the [data] Add GH Archive JSON events to pretraining data #5099 training slice). Declare the window as a top-level constant.
• Train/dev/test coordination lives in [data] Add GH Archive JSON events to pretraining data #5099; [evals] Add GH Archive structured-output PPL evals #5098 only consumes the eval window.

4. Kept out of default validation sets

• Not wired into experiments/evals/task_configs.py defaults. Exposed only via explicit import, matching the runnable_long_tail_raw_validation_sets() opt-in pattern.

5. Tests — tests/test_gh_archive_ppl.py

• Fixture with a hand-written 5-event .json.gz exercising each event type.
• Assert normalizer emits one record per event, masks sha/id/timestamps, routes to correct event-type shard.
• Assert the registry yields one RawTextEvaluationDataset per event type with the expected tag tuple. No network, no mocks for GCS.

Open questions for human review

1. Sequencing: land the downloader in [data] Add GH Archive JSON events to pretraining data #5099 first and depend on it here, or land both together in one PR? (Favor the former — it gives [data] Add GH Archive JSON events to pretraining data #5099 standalone value and lets masking get license review before we use it.)
2. text serialization choice: compact JSON of the full event vs. payload-only vs. a templated Type: X\\nActor: Y\\nPayload: {...} form. Compact-JSON is the most faithful to the tool-consumption surface form named in the issue body; confirm.
3. Eval window size: 256 docs per event type × 5 event types × 32KB cap is the current default (exp_model_perplexity_gap_long_tail_runnable.py). Confirm that's enough signal per event type.
4. Companion gap experiment: add a exp_model_perplexity_gap_gh_archive.py wiring marin-8b vs Llama-3.1-8B / Qwen3-8B now, or wait?

Happy to proceed on any of the above once the approach is blessed.