[data] Add GH Archive JSON events to pretraining data

[dlwh]

🤖 Standalone data-ingestion task.

Description

Source and prepare GH Archive event payloads as pretraining data. These records cover real GitHub-shaped JSON: event metadata, actor/repo fields, issue and PR payloads, timestamps, URLs, IDs, and nested tool/API objects.

Primary source:

• GH Archive: https://www.gharchive.org/

This should be license/governance reviewed before inclusion. It should also preserve a date-based held-out split for eval and avoid training on any GH Archive eval slice.

Definition of Done

• Document license/governance status and any redistribution constraints.
• Build a downloader/normalizer for selected date ranges and event types.
• Define train/dev/test date splits before ingest.
• Mask or bucket fields that would dominate without useful learning, such as long hashes when appropriate.
• Propose mixture weights and byte/token estimates.

[claude]

🤖 Plan:

• Fetch issue and duplicate-work preflight
• Map Marin dataset pipeline (downloader → normalize → tokenize → mixture)
• Post research + proposed design

Research

GH Archive publishes one gzipped JSONL file per hour at https://data.gharchive.org/YYYY-MM-DD-H.json.gz (~2009-02 → present). The issue asks to use the raw event payloads as a structured-JSON pretraining source — different from the existing common-pile/github_archive_filtered, which is license-filtered to permissive repos and reduced to code/diff text, not the full event stream.

Open questions block a direct implementation:

1. License / governance. GH Archive site is CC-BY-4.0 on the archive format, but event records embed user-generated content (issue/PR bodies, comments, review text) whose copyright belongs to authors and whose repo license may be non-permissive. GitHub ToS permits public API consumption but does not grant redistribution rights for ingested UGC. Need explicit signoff on: (a) whether to restrict to repos with permissive licenses via a repo→license join (common-pile's approach), (b) whether to drop body/comment text from private-repo-shaped events, (c) whether actor login / email needs hashing. Recommendation: restrict to public Public/Push/PullRequestEvent/IssuesEvent/IssueCommentEvent/WorkflowRunEvent and apply a permissive-license allowlist on repo.name.
2. Eval coordination with [evals] Add GH Archive structured-output PPL evals #5098. [evals] Add GH Archive structured-output PPL evals #5098 wants a held-out GH Archive slice for structured-output PPL evals. The date splits here must be chosen jointly so the pretrain set never overlaps the eval dates — this is a hard gate, not a nice-to-have.

Relevant code

• experiments/pretraining_datasets/nsf_awards.py - closest template: custom HTTP downloader → normalize → tokenize
• lib/marin/src/marin/datakit/download/nsf_awards.py - zephyr pipeline that fetches, filters, writes per-partition parquet
• lib/marin/src/marin/datakit/normalize.py - standard normalize step (text_field, id_field, dedup)
• experiments/common_pile/tokenize_common_pile.py#L109-L119 - existing license-filtered GH archive (different scope)
• experiments/pretraining_datasets/nemotron.py#L39-L47 - mixture weight pattern

Proposed Design

Scope: new module marin.datakit.download.gh_archive + experiments/pretraining_datasets/gh_archive.py + docs entry. Estimated ~400-600 LOC including tests — too large for auto-PR; needs human signoff on the governance items before code lands.

Downloader (lib/marin/src/marin/datakit/download/gh_archive.py):

• Config: start_date, end_date, event_types: set[str], license_allowlist_path (GCS path to a repo→spdx table), train_dates, dev_dates, test_dates (explicit; no silent defaults — per AGENTS.md).
• Fetch each hourly .json.gz via requests, stream-parse JSONL; filter by event type and repo-license allowlist; emit one parquet file per day per split (split=train/YYYY-MM-DD.parquet).
• Use zephyr's Dataset.from_list(tasks).map(...).write_parquet_file(...) pattern (mirror nsf_awards.download_nsf_awards). Resource cost: ~3-4 TB raw/year compressed; one year of PushEvent alone is O(100GB). Must run in US-CENTRAL (GH Archive is us-east but Google Storage egress is free cross-region within US; verify before running — cross-region bandwidth is an AGENTS.md cost-gate).
• Keep each record as a canonical JSON string in the text field (the whole point is to expose API-shaped JSON to the model), preserve id, type, created_at, repo.name as source-id columns.

Field masking / bucketing (issue calls this out explicitly):

• Replace SHA-1 commit hashes (^[0-9a-f]{40}$) with <SHA> token — they're uniformly random, dominate entropy, learn nothing.
• Replace GitHub numeric node-IDs >10 digits with bucketed <ID:small>/<ID:med>/<ID:large>.
• Keep URLs intact — the model should learn the api.github.com/repos/{owner}/{repo}/... shape.
• Truncate body fields at 16KB (long issue threads otherwise dominate by length).
• Drop payload.pages[].sha, payload.head, payload.before (pure hashes).
• Mask actor.email unconditionally.
• Add a --no-mask debug flag for evals ([evals] Add GH Archive structured-output PPL evals #5098 may want unmasked).

Date splits (must coordinate with #5098):

• Proposal: train = 2015-01-01 → 2024-06-30, dev = 2024-07, test = 2024-08 through 2024-12. Any date used in [evals] Add GH Archive structured-output PPL evals #5098's eval slice is excluded from train. Contiguous month-size holdouts (not random) so event distributions match — GH Archive has strong diurnal + weekday patterns.
• Before ingest, commit the exact split manifest to experiments/pretraining_datasets/gh_archive.py as a versioned constant so re-runs are reproducible.

Mixture weights / token estimate:

• Rough: one year of filtered GH Archive events ≈ 50-80B tokens after masking (llama3 tokenizer, permissive-license filter ≈ 10-20% of repos). Ten years ≈ 0.5-1T tokens raw → propose target weight 0.005-0.01 of pretraining mix, pattern matching common_pile/github_archive at 0.066. Exact numbers need empirical measurement after a 1-month pilot ingest.

Testing:

• Unit test on a single synthetic hour (3-5 events) validating: event-type filter, SHA masking, license allowlist filter, date→split assignment, parquet schema.
• No network in tests — fixture a gzipped JSONL string.

Proposed phasing to de-risk:

1. Pilot: 1 month of data, log token counts + masking-counter histograms. Human reviews.
2. Full ingest after license signoff.
3. Tokenize + register mixture weight. Land eval hookup from [evals] Add GH Archive structured-output PPL evals #5098.

Requesting @dlwh signoff on:

• (a) permissive-license allowlist vs. full public firehose,
• (b) the proposed train/dev/test date partition (and whether [evals] Add GH Archive structured-output PPL evals #5098 needs specific months reserved),
• (c) masking policy for actor.email and long hashes.

Once those are answered I can open the phase-1 pilot PR (~200 LOC, just the downloader + tests, no tokenization yet).