[data] Source public diagnostic logs for training data

[dlwh]

🤖 Standalone data-ingestion task.

Description

Identify and prepare large public diagnostic-log corpora suitable for pretraining data, separate from held-out log-stream eval construction.

Candidate sources:

• Public CI/build logs such as GHALogs: https://zenodo.org/records/14796970
• Public failure-log datasets such as LogChunks: https://papertalk.org/papertalks/24713
• System/application logs from LogHub: https://github.com/logpai/loghub
• Checked-in GitHub log fixtures, golden outputs, stack traces, and test snapshots from existing source corpora.

Do not train on held-out eval slices. Treat Marin-owned CI/Iris/Zephyr logs as eval-only until sanitization, governance, and leakage policy are agreed.

Definition of Done

• Produce a source inventory with license, size, format, and contamination risks.
• Define train/dev/test separation before ingest.
• Add sanitization rules for secrets, internal paths, tokens, and user identifiers.
• Propose a training-mixture tranche with byte/token estimates.

[claude]

🤖 Plan:

• Fetch issue info
• Duplicate-work preflight (open PRs/issues)
• Survey existing pretraining-dataset / sanitization infra
• Post research + proposed approach comment
• Await human sign-off on source set, split policy, and sanitization scope

Research

Scope is data curation + governance, not a bug: license audit, train/eval partition (must coordinate with #5093), sanitization rules, and byte/token estimates. No existing log-specific ingestion in the tree — datakit/download/ has nothing for LogHub/GHALogs/LogChunks; closest precedent is nemotron_terminal.py (CLI rollouts → {id,text,source} parquet). No open PR is touching this; #5093 is the sibling eval-side task and must share the split policy.

Relevant code

• datakit/download/nemotron_terminal.py — template: HF download + zephyr transform + hash_attrs
• datakit/download/huggingface.py — download_hf_step used for all HF sources
• experiments/pretraining_datasets/simple.py — where a new family registers its download + tokenize
• transform/ar5iv/transform_ar5iv.py#L78 — only prior PII-stripping precedent; no shared secret-scrubber lives in the repo
• datakit/normalize.py — canonical doc normalization to plug sanitization into

Proposed Approach (needs sign-off before implementation)

1. Source inventory (no download yet). Land this as a doc under .agents/projects/20260422_public_diag_logs.md and iterate. First pass:

Source	License	Hosting	Approx. size	Contamination risks
LogHub (logpai/loghub)	MIT / per-dataset-varies	GitHub + Zenodo mirrors	~77 GB raw across 19 subsets; many are 1 GB each	Known eval target in log-anomaly papers — must hold out, de-dup vs LLM-bench leakage
GHALogs (Zenodo 14796970)	CC-BY-4.0 (Zenodo default; confirm)	Zenodo	Millions of GH Actions logs, log-compressed ≈ tens of GB	GitHub usernames, repo URLs, secrets leaked into CI output, action run IDs
LogChunks (MSR'23)	Research, usage terms TBD	Paper artifact	MB-scale; too small for pretraining by itself	Curated failure chunks — may overlap SWE-bench-style evals
Starcoder/Dolma/Nemotron log-like artifacts	Already cleared upstream	Existing mirrors	unknown until grep’d	Already in PT mix — don't double-count; extract subset rather than re-ingest
Marin-owned CI/Iris/Zephyr logs	Internal	GCS	unknown	Secrets, SSH keys, tenant IDs — eval-only until sanitization + legal sign-off

Exact license lines and subset-level counts need a real get_hf_dataset_schema.py / Zenodo metadata pass; flagging this as the first implementation step once scope is approved.

2. Train/dev/test separation — coordinate with #5093. Single split policy per source, declared in one place and imported by both issues:

• If the source has native train/validation/test, reserve validation+test for evals, train on train.
• Otherwise, deterministic partition by sha256(record_id) mod 100: 0–1 → test (eval held-out, also used by [evals] Add diagnostic log-stream PPL eval sets #5093), 2 → dev, 3–99 → train. Salt per-source so a single doc can't migrate between sources.
• LogHub's anomaly-labeled splits go to eval only.
• Marin-owned logs: no training inclusion at v0.

3. Sanitization. Add lib/marin/src/marin/transform/logs/sanitize.py — a pure text → text function wired into the normalize step. v0 rule set (single pass, regex + allow-list):

• Credentials: (?i)(api[-_]?key|secret|token|password|authorization|bearer)\s*[:=]\s*\S+, AWS access keys (AKIA[0-9A-Z]{16}), GCP "private_key": "...", Slack/GitHub tokens (ghp_, xox[baprs]-), JWT-shaped tokens.
• Identity: emails, user:password@host URL auth, SSH private key PEM blocks.
• Paths: /home/<user>, /Users/<user>, Windows C:\\Users\\<user>.
• Network: private IPv4 ranges and internal hostnames only if on a configured allowlist (keep public IPs; logs about public services are valuable).
• Pipe through detect-secrets-style entropy check for remaining high-entropy strings → mask as <REDACTED:high-entropy>.
  Unit-test with fixed inputs/outputs — this is a text-in/text-out function, no mocks needed.

4. Training-mixture tranche. Register under experiments/pretraining_datasets/logs.py, mirroring nemotron_terminal.py:

• loghub_public_v0 (train split only, sanitized, deduped against existing mix).
• ghalogs_public_v0 (sanitized).
• Tokenized step tokenize_logs_public_v0 → single logs_public_v0 entry in the family registry.
• Byte/token estimates: compute during first implementation PR after schema inspection; don't guess now.
• Weight in the overall PT mix is a separate follow-up after a small-scale ablation; not part of this issue.

5. Delivery plan (each step is its own PR).

1. Inventory doc + split-policy module + sanitization function with tests (small, reviewable).
2. LogHub download + transform + tokenize step, train-split only.
3. GHALogs download + transform + tokenize step.
4. LogChunks + checked-in-source-log carve-out (optional).
5. Marin-owned log pipeline — blocked on governance, defer.

Deferring PR creation until maintainer approves (a) the source set, (b) the deterministic split policy (especially whether #5093 accepts the sha256 mod 100 approach), and (c) the sanitization rule scope. Reply with 👍 / deltas and I'll proceed with step 1.