Agent MoE Experiment: focus crawl (CC-SUPPLEMENTAL-2026-22) vs main crawl (CC-MAIN-2024-18)

[Helw150]

TL;DR

Token-matched data comparison: train the grug-MoE compute-optimal ladder
(d512/d768/d1024/d1280) on raw WET text from Common Crawl's focus crawl
CC-SUPPLEMENTAL-2026-22 ("top-10k science domains") vs the main crawl
CC-MAIN-2024-18 (the crawl used to build the focus seed list). Same model
configs, same per-rung token budgets, only the data differs. Primary metric:
eval/paloma/macro_loss per rung, focus vs main.

Description

Common Crawl ran a quality-steered "focus" crawl (CC-SUPPLEMENTAL-2026-22)
that biases fetching toward high-value science domains instead of the usual
search-engine-style ranking. We want to know whether pretraining on that focus
crawl's text yields lower loss than pretraining on a token-matched random sample
of the contemporaneous general main crawl, at every rung of the grug-MoE
compute-optimal ladder.

Both corpora are ingested from the crawls' WET (extracted plain-text) files over
the data.commoncrawl.org HTTPS gateway, sampled to token pools large enough
that the largest rung (d1280 = 1.24e10 tokens) never repeats data, tokenized
with the llama3 tokenizer, and fed unchanged through the documented baseline
ladder. Because each rung's (num_steps, batch_size) come from the shared
heuristic, every rung consumes an identical token count from whichever corpus —
the comparison is token-matched per rung by construction.

Initiating prompt (verbatim)

Cool - can you set up a side by side experiment comparing the focus crawl from
CC-SUPPLEMENTAL-2026-22 to CC-MAIN-2024-18? The goal should be getting the
same number of randomly sampled tokens for each and then training a ladder of
models from @experiments/grug/moe/agent.md on them

Hypothesis or Goal

H: At matched compute/tokens, the focus (science-steered) crawl yields lower
eval/paloma/macro_loss than a random sample of the main crawl, with the gap
widening (or holding) up the ladder.

Scope

• Primary metric: eval/paloma/macro_loss (final), per rung.
• Secondary: throughput/tokens_per_second, throughput/total_tokens.
• Axis under test: data source only. Model/optimizer/batch/steps/seq are the
  documented compute-optimal baselines; nothing else varies.

Method

• Ingestion: lib/marin/src/marin/datakit/download/common_crawl_wet.py
  (WET WARC parser → dolma JSONL; focus manifest from the crawl's index parquet,
  main manifest from wet.paths.gz; seeded random file sampling).
• Launch: experiments/grug/moe/launch_crawl_compare.py (STAGE=data ingest+tokenize;
  STAGE=train CRAWL=<focus|main> SCALE=<d512|d768|d1024|d1280>).
• Sample sizes (seed 0): focus 1200 WET files (~16.5B tok), main 230 files
  (~18.4B tok). Measured yield: ~13.8M llama3 tok per focus WET file.
• Hardware: one v5p-8 per training run (8 runs total), via Iris.

Repro

# data prep (once)
STAGE=data python -m experiments.grug.moe.launch_crawl_compare
# each rung
STAGE=train CRAWL=focus SCALE=d512 python -m experiments.grug.moe.launch_crawl_compare

Current baseline (Nemotron mix, from experiments/grug/moe/README.md)

Budget	Dim	Layers	Tokens	Paloma macro
2.19e17	d512	6	8.37e8	3.8104
1.70e18	d768	8	2.71e9	3.4339
9.00e18	d1024	11	6.63e9	3.1605
2.83e19	d1280	13	1.24e10	3.0065

(The Nemotron numbers are a curated-mix reference, not a direct arm of this
experiment; the head-to-head is focus vs main below.)

Results

Rung	focus macro	main macro	Δ (main−focus)
d512	pending	pending
d768	pending	pending
d1024	pending	pending
d1280	pending	pending

Links

• Branch: agent/grug-moe-crawl-compare
• Logbook: .agents/logbooks/grug-moe-crawl-compare.md
• W&B: group crawl-compare-focus-vs-main, project marin_moe (link once runs start)

[claude]

🤖 Status: not opening a PR — large experiment requiring human sign-off.

This is an experiment tracking issue, not a code defect. Its definition of done is running the 8-run ladder and filling the results table — ~80 v5p-8-hours plus a one-time ~120–180 GB open-internet pull from data.commoncrawl.org. Both the spend and the bulk egress need explicit authorization per AGENTS.md (egress/storage cost rule; no launching jobs without permission), so an agent should not kick this off. The scaffolding already lives on agent/grug-moe-crawl-compare (no PR yet); the code itself is reviewable and largely sound, with the caveats below.

Research

Two new files do the real work; the rest of the branch diff is just the branch trailing main.

Relevant code

• common_crawl_wet.py — WET WARC parser → dolma JSONL; manifest from wet.paths.gz (main) or index parquet (focus).
• launch_crawl_compare.py — STAGE=data ingest+tokenize; STAGE=train CRAWL=… SCALE=… per rung.

WARC parsing is correct: it frames bodies by Content-Length and self-syncs on the WARC/ line, so trailing \r\n\r\n between records is harmless. Token-matching holds by construction — each rung's (steps × batch × seq) fixes tokens consumed identically for both corpora, so the project-doc "truncate caches to min" step is unnecessary and is (correctly) not implemented.

Concerns to resolve before launch

1. Focus crawl provenance. The index parquet is under …/projects/cc-open-athena-test/CC-SUPPLEMENTAL-2026-22/… (_FOCUS_INDEX_PARQUET). The cc-open-athena-test prefix reads like a scratch/test bucket, not a published crawl artifact. Confirm this is the intended production focus crawl before spending ~80 TPU-hours on it.
2. Single-part universe assumption. focus_crawl_wet_paths reads one parquet part and assumes it enumerates all 4,573 WET files ("row-partitioned, one part = full universe"). Logbook says verified; if wrong, the sampling universe is silently incomplete (biased, not erroring). Cheap to assert: compare distinct warc_filename count across two parts.
3. Sample-size headroom from n=1. _FOCUS_NUM_FILES=1200 (~16.5B tok) and _MAIN_NUM_FILES=230 (~18.4B) are extrapolated from one measured WET file each (~13.8M / ~80M tok). d1280 needs 1.24e10; focus margin is ~33%. If per-file yield varies, the top rung could repeat data. Worth a quick yield check on the actual STAGE=data output before the long rungs.

Open decisions (from the branch project doc, still unanswered)

• Token budget per crawl (must stay ≥ d1280's 1.24e10).
• Run scope: full ladder now vs validate-smallest-first vs data-only.
• Egress authorization + target MARIN_PREFIX.

Recommendation

Split code from compute. Open a PR for just the two new files (~390 LOC, independently reviewable, with a small WET-parser test on a fixture); land it on its own. Then, with a human's go-ahead, run STAGE=data first, verify focus yield clears 1.24e10 with margin and resolve concern #1, and only then fan out the ladder — ideally d512 (~0.6h) as a smoke before committing the d1024/d1280 long runs (~37h combined). I'm not opening that PR unprompted since the runs (not the code) are the deliverable here and need authorization; happy to prep it on request.

[Helw150]

🤖 Update: pipeline built, data-prep launched

• Change: Added WET ingestion (marin.datakit.download.common_crawl_wet) and the
  ladder launcher (experiments/grug/moe/launch_crawl_compare.py). Validated the
  WET parser end-to-end on one real file per crawl, the manifest builders
  (main 90,000 files via wet.paths.gz; focus 4,573 files via the index parquet),
  seeded sampling, and the full ingest→tokenize→train + paloma-validation step graph.
• Data-prep job running on Iris (/held/grug-crawlcmp-data, us-central2, plain CPU
  coordinator). Sampling confirmed in-job: focus 1200/4573 WET files, main 230/90000
  (seed 0). Writing to gs://marin-us-central2/raw/cc_wet/{focus,main}..., then
  llama3 tokenize.
• Rung configs reproduce the README baseline exactly (only the data differs):
  d512 8.37e8 tok / d768 2.71e9 / d1024 6.63e9 / d1280 1.24e10.
• Confidence: exploratory (no training numbers yet).
• Links:
  • Branch agent/grug-moe-crawl-compare
  • Logbook .agents/logbooks/grug-moe-crawl-compare.md
• Next: verify both tokenized caches clear ~13B tokens, then submit 8 v5p-8 runs
  (focus/main × d512/d768/d1024/d1280) and fill the results table.

[Helw150]

🤖 Update: data-prep complete, 8 v5p-8 runs launched

• Data-prep done. Token pools (llama3, seed 0):
  • focus CC-SUPPLEMENTAL-2026-22: 1200 WET files → 11,328,978 docs ≈ ~23B tokens
  • main CC-MAIN-2024-18: 230 WET files → 6,963,215 docs ≈ ~16B tokens
  • Both exceed the largest rung (d1280 = 1.24e10), so every rung trains on an
    identical fixed token budget drawn from each corpus — token-matched per run.
• One recovery: the focus tokenize first failed because the source bucket is named
  cc-open-athena-test and the tokenizer guards against test/validation in train
  paths. Fixed by threading allow_test_in_train through default_tokenize; main
  tokenize + both downloads were cached and reused.
• Launched 8 runs on Iris (plain CPU coordinators; each dispatches its own v5p-8 child):
  /held/grug-crawlcmp-{focus,main}-{d512,d768,d1024,d1280}.
• Tracking: no W&B key in the runner env, so these log via the JSON-logger fallback
  (macro_loss still recorded). Group would be crawl-compare-focus-vs-main if W&B is added.
• Next: monitor to finished, pull final eval/paloma/macro_loss per rung, fill the
  results table (focus vs main).

[Helw150]

🤖 Update: two-bug recovery, all 8 v5p-8 runs relaunched and progressing

The first training launch failed fast on two infra issues, both fixed:

1. W&B key hard-required. The grug training dispatch (marin.training.training._check_for_wandb_key)
   raises unless WANDB_API_KEY is set or WANDB_MODE disables it — independent of the tracker.
   Resolved by submitting with WANDB_MODE=disabled (metrics go to the JSON-logger fallback).
2. Cross-region scheduling. The executor infers the data region (us-central2) for the training
   step, but v5p-8 lives in us-central1/us-east5, so the TPU child was unschedulable: no groups in region us-central2. Fixed by pinning ResourceConfig.with_tpu("v5p-8", regions=["us-central1","us-east5"]);
   the run reads tokenized data cross-region like the nemotron baseline.

Both verified on a single d512 canary (cleared the wandb guard, dispatched the Fray v5p-8 child with
no unschedulable error, running). All 8 runs (/held/grug-crawlcmp-{focus,main}-{d512,d768,d1024,d1280}-r3)
are now up. d512 ≈ 0.6h … d1280 ≈ 27h on a v5p-8.

Next: monitor to completion, pull final eval/paloma/macro_loss per rung, fill the results table.

[Helw150]

🤖 Update: d512 rung complete (both arms)

Final eval/paloma/macro_loss (v5p-8, json-logger):

Rung	focus (CC-SUPPLEMENTAL-2026-22)	main (CC-MAIN-2024-18)	Δ (main−focus)
d512	4.712	4.402	−0.310

• At d512 the main crawl has the lower paloma macro_loss (−0.31 vs focus).
  Plausible read: paloma is a broad general-web benchmark, so the science-skewed
  focus crawl generalizes worse to it than the broad main crawl. (overall
  eval/macro_loss incl. uncheatable: focus 4.627 vs main 4.420 — same direction.)
• Both arms sit well above the Nemotron-curated baseline (d512 = 3.81), as expected
  for raw WET crawl text vs a curated mix.
• Each d512 run took one TPU preemption and auto-resumed from checkpoint.
• Confidence: exploratory (single seed, smallest rung). d768/d1024/d1280 in flight.

[Helw150]

🤖 Update: preemption mitigation (d512 done; d768–d1280 in flight)

Overnight v5p capacity is heavily preempted. With the default 10-min checkpoint
interval the d768–d1280 runs were thrashing (e.g. main-d768 rolled 993→454 over an
hour) — preemptions arrived faster than progress could be banked, so net progress
went negative.

Mitigation: made the grug checkpointer interval env-configurable
(GRUG_CHECKPOINT_MINUTES, default 10; hash-safe) and relaunched the 6 remaining
rungs with it set to 3 min and --max-retries 10. Each preemption now costs
≤3 min instead of ≤10, so the runs make reliable net-positive progress under load.
They resume from their existing region-local GCS checkpoints (step hash unchanged).

Status: d512 done (focus 4.712 / main 4.402). d768/d1024/d1280 × {focus,main}
running and self-healing; ETAs depend on capacity. Will report each rung's
eval/paloma/macro_loss as it lands.

[Helw150]

🤖 Update: root-caused the d768+ failures — cross-region egress, now fixed by co-location

The larger rungs weren't preemption-limited (my earlier read was wrong). They crashed
deterministically at ~step 490 with TransferBudgetExceeded (rigging's cross-region
transfer-budget guard), raised during checkpoint save.

Root cause: I had pinned MARIN_PREFIX=gs://marin-us-central2, but v5p only exists in
us-central1/us-east5. So data reads + checkpoint writes all crossed regions and tripped
the budget. d512 (smaller/shorter) squeaked under it; d768+ blew it.

Fix (don't fight the inference): dropped the MARIN_PREFIX override and the TPU
regions= pin, added a STAGE=all mode, and resubmitted one job with --reserve v5p-8
(no --region, no prefix). The coordinator landed in us-east5 and marin_prefix()
inferred gs://marin-us-east5 — so ingest, tokenize, training, and checkpoints are now
all co-located in us-east5. Zero cross-region transfer; the budget guard can't trip.

Re-running the full ladder there now (/held/grug-crawlcmp-all), logging to W&B
(project marin_moe, group crawl-compare-focus-vs-main). d512 numbers already recorded
above stand; will fill d768/d1024/d1280 as they complete.

[Helw150]

🤖 Update: d768 pair complete (4/8) — science vs general-web contrast replicates at the 2nd scale

Now co-located in us-east5 (no cross-region egress) and W&B-tracked
(held/marin_moe, group crawl-compare-focus-vs-main). d512 and d768 pairs done.

Final eval loss (nats/token), focus vs main, Δ = focus−main (negative ⇒ focus better):

eval set	d512 Δ	d768 Δ	direction
arxiv_physics	−0.775	−0.732	focus (science)
arxiv_computer_science	−0.581	−0.565	focus (science)
m2d2_s2orc (papers)	−0.563	−0.513	focus (science)
mc4 (multilingual web)	+0.514	+0.485	main (general)
ptb (WSJ news)	+0.481	+0.315	main (general)
paloma macro	+0.310	+0.290	main (general)

The focus crawl's science advantage and general-web cost are stable across both
scales — a replicated effect, not a small-scale artifact. Magnitudes are large on
the scientific splits (arxiv_physics ≈ −0.7 nats).

Live shareable figure (D3, power-law-with-asymptote fits, forecast speedup/slowdown
per panel): the page fills in as d1024/d1280 land. Fits + speedup badges require ≥3
rungs per corpus, so they appear after d1024. d1024 is ~30% done; d1280 behind it.