[SWE-ZERO] Quality validation: midtrain Marin-8B on SWE-ZERO trajectories and evaluate on SWE-bench

[AlienKevin]

Objective

Validate the quality of the SWE-ZERO 140B trajectory dataset (#4719) by continue-training Marin-8B base on a representative subset and measuring before/after on SWE-bench Verified and SWE-bench Multilingual.

Dataset: AlienKevin/SWE-ZERO-12M-trajectories — 1.45M clean trajectories (20B checkpoint)

Deadline: Monday April 21 (results needed for go/no-go on scaling to 140B)

Experiment Design

Training

Parameter	Value	Rationale
Base model	Marin-8B base (or Qwen3-8B)	Matches existing SFT baselines (#3490, #3896, #4420)
Dataset	100K trajectory subset from SWE-ZERO-12M	Representative sample; full 1.45M would take ~10 days
Sequence length	32K	Matches SWE-ZERO generation config
Epochs	1	Speed — quality signal detectable in 1 epoch
Batch size	16-128 (depending on TPU)	Follow #3490/#3896 configs
Learning rate	2e-5 to 4e-5	Standard for continue pre-training on agentic data
TPU	v5p-32 or v5p-64	~1-2 days training time
Format	mini-swe-agent v1 chat format	Same format the data was generated in

Estimated training time: ~1-2 days on v5p-32 (100K trajectories × 1 epoch × 32K context)

Reference configs:

• [Agentic SFT] Reproduce NemotronTerminal-8B #3490 (NemotronTerminal-8B reproduction): exp3490b_sft_nemotron_terminal_corpus_qwen3_8b.py
• [Agentic SFT] Reproduce OT-Agent's best 32K context length SFT #3896 (OT-Agent 32K reproduction): exp3896_sft_ota_32k_qwen3_8b.py
• [Agentic SFT] Post-train Marin-8B Instruct on the TerminalCorpus #4420 (Marin-8B on Terminal-Corpus): exp4420_sft_marin_8b_instruct_terminal_corpus.py
• [Agentic SFT] Estimate Marin 8B and 32B post-training compute needs #4510 (compute estimates): v5p-64 = ~5 days for 366K examples

Evaluation

Run Harbor evaluation on both the base model and the fine-tuned model:

Benchmark	Dataset	Agent	Expected baseline
SWE-bench Verified	swebench-verified@1.0	mini-swe-agent v1 (terminus-2)	~0% (untrained base)
SWE-bench Multilingual	swebench-multilingual@1.0	mini-swe-agent v1 (terminus-2)	~0% (untrained base)

Reference eval configs:

• [Agentic SFT] Reproduce NemotronTerminal-32B #4307 (Harbor eval template): exp4307_eval_released_nemotron_terminal_32b_tb2.py
• [Harbor] Speed up agentic evals #3846 (Harbor speedup techniques): task sharding, vLLM tuning
• Experiment: Execution-based SWE-ZERO evaluation via ConTree #4683 (ConTree execution-based eval): alternative evaluation path

Estimated eval time: ~12-24 hours per benchmark on v5p-8 with task sharding

Success Criteria

1. Minimum: Fine-tuned model shows any positive resolve rate on SWE-bench Verified (base model at ~0%)
2. Good: resolve rate ≥ 5% on SWE-bench Verified (comparable to ConTree pass@1 of 6.0% from Experiment: SWE-ZERO scaling to 1B tokens (32k PRs × 3 rollouts) #4666)
3. Stretch: resolve rate ≥ 10% on SWE-bench Verified

A negative result (no improvement) would indicate the trajectories need quality filtering beyond error/dedup removal — e.g., filtering to submitted-only rollouts, or increasing MAX_TURNS back to 30.

Timeline

Day	Task
Friday Apr 18	File issue, write training config, launch training run
Sat-Sun Apr 19-20	Training completes (~1-2 days), launch Harbor evals
Monday Apr 21	Eval results, post analysis to this issue

Data Preparation

The 100K subset should be stratified:

• Sample proportionally across languages (matching SWE-rebench V2 distribution)
• Include both "Submitted" (8.9%) and "incomplete" (91.1%) rollouts
• Filter to MAX_TURNS=15 rollouts only (the ongoing production config) for consistency

Related Issues

• [SWE-ZERO] Scale to 140B tokens: 122K PRs x 100 rollouts from SWE-rebench V2-PRs #4719 — SWE-ZERO 140B scaling run (the dataset being validated)
• Experiment: SWE-ZERO scaling to 1B tokens (32k PRs × 3 rollouts) #4666 — SWE-ZERO 1B scaling (ConTree eval: pass@1=6.0%, pass@3=11.3%)
• Experiment: SWE-ZERO multi-language scaling (20 languages × 5 PRs × 3 runs) #4653 — SWE-ZERO multilang validation (300 rollouts, 20 languages)
• [Agentic SFT] Reproduce NemotronTerminal-8B #3490 — NemotronTerminal-8B reproduction (SFT workflow template)
• [Agentic SFT] Reproduce OT-Agent's best 32K context length SFT #3896 — OT-Agent 32K reproduction (32K context SFT)
• [Agentic SFT] Post-train Marin-8B Instruct on the TerminalCorpus #4420 — Marin-8B on Terminal-Corpus
• [Agentic SFT] Estimate Marin 8B and 32B post-training compute needs #4510 — Compute estimates for 8B/32B post-training
• [Agentic SFT] Reproduce NemotronTerminal-32B #4307 — Harbor SWE-bench evaluation template
• [Harbor] Speed up agentic evals #3846 — Harbor eval speedup techniques

[AlienKevin]

🤖 Training config ready: exp4898_sft_marin_8b_swe_zero.py

Changed to Marin-8B base (from Qwen3-8B) per request.

Key hyperparameters

Parameter	Value	Rationale
Base model	marin-community/marin-8b-base	Target model for quality validation
Learning rate	1e-4	5× higher than exp4420's 2e-5 which got only 1.1% TB2 on Marin-8B. See #4225 for weight decay / LR interaction analysis — Marin-8B's pretraining WD requires elevated post-training LR
Weight decay	0.0	No additional WD during SFT (model already has pretraining WD effects)
Context length	32K	Matches SWE-ZERO generation config
Epochs	1	Speed for quality signal
Schedule	Cosine, 10% warmup	Standard
AdamW	β1=0.9, β2=0.95	Standard

Scaling law runs

Subset	Steps	Est. time (v5p-32)
10K	625	~6 hours
50K	3,125	~30 hours
100K	6,250	~60 hours

For Monday deadline: 10K and 50K are feasible. 100K may need v5p-64 to finish in time.

Eval plan (parallel with training)

1. Baseline eval: Marin-8B base (no SFT) on SWE-bench Verified + Multilingual via Harbor — launches now
2. Post-training eval: each checkpoint (10K, 50K, 100K) on same benchmarks
3. Scaling plot: resolve rate vs training samples

Commit: experiment script at experiments/exp4898_sft_marin_8b_swe_zero.py

[AlienKevin]

🤖 Progress update (04:30 UTC Apr 19)

Baseline Eval

• Result: 0/25 resolved (0%) on SWE-bench Verified with Marin-8B base
• Pipeline verified end-to-end: Iris → vLLM (v6e-4) → Harbor → Daytona → mini-swe-agent v1 → verifier
• wandb: https://wandb.ai/marin-community/harbor/runs/wrcu6xho
• GCS: gs://marin-us-central2/evaluation/harbor/swebench-verified/marin-8b-base/mini-swe-agent/baseline-100/

Issue found & fixed: The InstalledAgent version of mini-swe-agent runs the mini CLI inside the Daytona sandbox, which can't reach the vLLM server on the Iris worker's localhost. Rewrote MiniSweAgentV1 as a BaseAgent subclass that directly reuses the official DefaultAgent from minisweagent.agents.default (SWE-agent/mini-SWE-agent@v1) with two thin adapters:

• LitellmModelAdapter → litellm.completion() on the host (reaches localhost vLLM)
• HarborEnvironmentAdapter → environment.exec() in the sandbox (runs bash in the repo)

SFT Training

Run	Status	Progress	Loss	ETA
50K	TRAINING	Step 413/3125 (13%)	0.475	~Sun afternoon
10K	Relaunched (v4)	Pending TPU	—	~Sat morning
100K	Relaunched (v4)	Pending TPU	—	~Mon

Loss trajectory (50K): 2.86 → 2.28 → 1.81 → 0.475 over 413 steps. The model is learning from SWE-ZERO trajectories.

Issues resolved:

1. Empty-message crash → Levanter now skips sequences with no assistant message
2. Cross-region data mismatch → using MARIN_PREFIX=gs://marin-us-central1
3. HF dataset cache → cache-busted tokenizer name (v3)
4. Daytona API key → MiniSweAgentV1 handles hosted_vllm natively

140B Pipeline

• High-water: 1,958k rollouts / 22.3B tokens (15.9% of target)
• Pipeline running autonomously with 10-13 batches

[AlienKevin]

🤖 Baseline eval complete — eval pipeline fully validated

Baseline: Marin-8B base on SWE-bench Verified (25 tasks)

Metric	Value
Resolve rate	0/16 (0%)
Total trials	25
With verifier	16
Failed (vLLM errors)	9

Pipeline verified end-to-end with real agent interactions:

1. Iris → vLLM (v6e-4, --tokenizer marin-community/marin-tokenizer) ✓
2. MiniSweAgentV1 (official DefaultAgent from mini-SWE-agent@v1) ✓
3. LLM calls via litellm → localhost vLLM (in-process, not in sandbox) ✓
4. Bash execution via Harbor environment.exec() → Daytona sandbox ✓
5. Verifier → reward=0 ✓
6. ATIF trajectory saved ✓

GCS: gs://marin-us-central2/evaluation/harbor/swebench-verified/marin-8b-base/mini-swe-agent-v1/baseline-100/

SFT Training Progress

Run	Status	Step	Loss	ETA
50K	TRAINING	1,210/3,125 (39%)	0.286	~Sun 20:00 UTC
10K	Pending TPU (region issue)	—	—	~3h after start
100K	Pending TPU	—	—	~Mon

Loss curve: 2.86 → 1.81 → 0.475 → 0.376 → 0.286

140B Pipeline

• 2,040k rollouts / 23.3B tokens (16.6% of target)
• Crossed 2M milestone at 07:44 UTC today

[AlienKevin]

🤖 SWE-ZERO Quality Validation — Final Results (3 runs per model)

Summary

SFT on Marin-8B base model using SWE-ZERO trajectories, evaluated on SWE-bench Verified (100 random tasks from DCAgent2/swebench-verified-random-100-folders) using mini-swe-agent v1 harness with Harbor. Each model evaluated 3 times with different random seeds (temperature=1.0) to measure variance.

Results

Model	Run 1	Run 2	Run 3	Mean ± Std
Baseline (untrained)	0%	—	—	0.0%
10K SWE-ZERO SFT	3%	4%	3%	3.3% ± 0.6%
50K SWE-ZERO SFT	6%	4%	2%	4.0% ± 2.0%
100K SWE-ZERO SFT	4%	7%	5%	5.3% ± 1.5%

Scaling Curve (with error bars)

Key Findings

1. Clear improvement from SWE-ZERO training: 0% → 3-5% resolve rate on SWE-bench Verified with an 8B model trained purely on synthetic execution-free trajectories.

2. No significant difference between 50K and 100K: Error bars overlap heavily (4.0% ± 2.0% vs 5.3% ± 1.5%). The earlier claim that "100K regresses" was driven by single-run variance. With 3 runs, 100K is actually slightly higher on average.

3. High stochastic variance: At temperature=1.0 on 100 tasks, resolve rate varies by 2-4 percentage points between runs. This highlights the need for multiple evaluation runs when reporting SWE-bench results on small models.

4. Monotonic scaling trend: 10K (3.3%) → 50K (4.0%) → 100K (5.3%) shows a positive trend, though the 50K→100K gain is within error bars.

Per-Run Resolved Tasks

Task	10K R1	10K R2	10K R3	50K R1	50K R2	50K R3	100K R1	100K R2	100K R3
django-13109		✅					✅	✅	✅
django-13363	✅	✅		✅		✅			✅
django-13410								✅
django-13449							✅
django-14855				✅				✅
django-15277			✅
django-15368			✅		✅
django-15467				✅			✅	✅
django-16485				✅
django-16527					✅		✅	✅	✅
xarray-4629	✅	✅	✅	✅	✅	✅		✅	✅
pytest-8399	✅	✅
pylint-6903				✅				✅
astropy-13236					✅

Most reliably solved: xarray-4629 (resolved in 8/9 runs across all models), django-13363 (5/9), django-16527 (4/9), django-13109 (4/9).

Training Data Composition

Model	Rows	Unique PRs	Mean rollouts/PR	Training steps
10K	10,000	104	96.2	625
50K	50,000	490	102.0	3,125
100K	100,000	989	101.1	6,250

Training data is dominated by non-Python repos (redis, go-zero, rust-analyzer). Almost none of the SWE-bench eval repos appear in training data — the model generalizes from diverse multilingual code-editing trajectories to Python-specific SWE-bench tasks.

Degeneration Analysis

Code blocks in user prompts trigger !!! degeneration at different rates per model:

Prompt has code blocks?	10K Degen	50K Degen	100K Degen
No code blocks	8%	17%	15%
Has code blocks	36%	17%	77%

100K shows U-shaped sensitivity: the model over-associates ``` with its output format after extended training, causing degeneration on code-heavy prompts. Despite this, 100K's mean resolve rate is still highest due to stronger performance on non-degenerate trials.

Artifacts

Models

• AlienKevin/marin-8b-swe-zero-10k
• AlienKevin/marin-8b-swe-zero-50k
• AlienKevin/marin-8b-swe-zero-100k

Eval Traces

• AlienKevin/SWE-ZERO-eval-traces
• GCS: gs://marin-us-central2/evaluation/harbor/swebench-verified/marin-8b-swe-zero-{10k,50k,100k}/mini-swe-agent-v1/run-{1,2,3}/

Training & Eval Config

• Training data: AlienKevin/SWE-ZERO-12M-trajectories@ab6295c3 — 1,613,271 rollouts (18.4 B tokens, "20B-clean" snapshot). Note: this pre-cap snapshot mixes max_turns=15 (~73 %) and max_turns=30 (~18 %) rollouts; current HF main is purified to 15-turn only.
• Base model: marin-community/marin-8b-base, LR: 1e-4, 1 epoch, batch_size=16, 32K context
• Agent: mini-swe-agent v1, max_turns=15, temperature=1.0, max_tokens=4096, stop_token_ids=[128009]
• Environment: Daytona sandboxes, vLLM native on v6e-4

Conclusion

SWE-ZERO synthetic trajectories produce real SWE-bench capability:

• 0% → 3.3-5.3% resolve rate on Marin-8B (100 SWE-bench Verified tasks)
• Positive scaling trend from 10K to 100K training samples
• High variance (±1-2%) necessitates multiple eval runs
• 500K SFT training in progress to test further scaling

---

Reproducibility (added 2026-05-09):

• Eval script: experiments/exp4898_eval_marin_8b_swebench.py at commit 486d2692e5 (branch agent/20260402-fix-4307).
• Eval invocation:

  uv run iris --config lib/iris/examples/marin.yaml job run \
      --cpu 0.5 --memory 4GB --disk 10GB \
      -e DAYTONA_API_KEY ${DAYTONA_API_KEY} \
      -e WANDB_API_KEY ${WANDB_API_KEY} \
      -e WANDB_ENTITY marin-community \
      -e WANDB_PROJECT harbor \
      -e HF_TOKEN ${HF_TOKEN} \
      -e MARIN_PREFIX gs://marin-us-central2 \
      -e ENV_TYPE daytona \
      --no-wait \
      -- python experiments/exp4898_eval_marin_8b_swebench.py

  Set MODEL_PATH / MODEL_NAME / HARBOR_RUN_ID env vars to vary the checkpoint and run-id; set HARBOR_TASK_NAMES_JSON for sharded eval. Defaults pinned to marin-community/marin-8b-base if not overridden.
• Harbor dataset: laude-institute/harbor-datasets @ 86723674f04e4209ac479d0fb75d9d9f44b4377e (extracted from config.json of an eval trial).
• Agent: mini-SWE-agent v1 branch — uv tool install "git+https://github.com/SWE-agent/mini-SWE-agent.git@v1" (pinned in lib/harbor/src/harbor/agents/installed/install-mini-swe-agent.sh.j2 at the same commit).
• Engine config (extracted from .executor_info of a run-2 shard): max_model_len=40960, max_num_seqs=4, tensor_parallel_size=4, n_concurrent=4, env=daytona, temperature=1.0, max_input_tokens=40960, max_output_tokens=4096.
• Resource per shard: cpu=32, ram=128 GB, disk=50 GB, device=v6e-4 (preemptible).
• Model checkpoints evaluated:
  • 10K: gs://marin-us-central1/checkpoints/exp4898_sft_marin_8b_swe_zero_10k_32768tokens_v5p32-8fefe2/hf/step-624
  • 50K: gs://marin-us-central1/checkpoints/exp4898_sft_marin_8b_swe_zero_50k_32768tokens_v5p32-6add41/hf/step-3124
  • 100K: gs://marin-us-central1/checkpoints/exp4898_sft_marin_8b_swe_zero_100k_32768tokens_v5p32-4153d9/hf/step-6249

[AlienKevin]

🤖 Detailed Cross-Task Analysis & Degeneration Breakdown

Cross-Task Comparison: What 100K fails on that 50K/10K solved

Task	10K	50K	100K	100K Failure Mode
django-13363	✅	✅	❌	Clean traj, correct concept but subtly wrong implementation
django-14855	❌	✅	❌	Clean traj, 3 edit attempts, fix didn't pass tests
django-16485	❌	✅	❌	Clean traj, struggled with sed, ran out of turns
xarray-4629	✅	✅	❌	Immediate !!! degeneration from turn 1
pylint-6903	❌	✅	❌	Immediate !!! degeneration from turn 1
pytest-8399	✅	❌	❌	Immediate !!! degeneration from turn 1

Two distinct failure modes for 100K:

1. Degeneration (3/6): Model produces !!!... gibberish from the first token, never generates a coherent response. Affects OOD repos (xarray, pylint, pytest).
2. Close but wrong (3/6): Model navigates correctly but makes subtly different edits than 50K — overfitting shifted its code generation distribution.

Degeneration Issue

Degeneration is present across ALL models (base + all SFTs) but at very different rates:

Model	Degenerate trials	Rate	Pattern
Base (untrained)	all	100%	Spams COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT (doesn't know format)
10K SFT	21/100	21%	Immediate !!! on ~1/5 of tasks
50K SFT	22/114	19%	Immediate !!! on ~1/5 of tasks
100K SFT	65/108	60%	Immediate !!! on majority of tasks

Key finding: Degeneration is almost always immediate (turn 1), not gradual. The model either works on a task or fails from the very first token. Only 1-3 trials per model show late-onset degeneration.

Root Cause

The <|eot_id|> stop token (token ID 128009) is what the model learns during SFT to signal "end of my response." When the model fails to generate this token, vLLM fills up to max_tokens=4096 with degenerate repetition.

The degeneration is prompt-conditional: certain task descriptions trigger a mode where the model never enters its learned "THOUGHT + bash" generation pattern. Instead it outputs ! repetition, which is a low-entropy attractor state for small LMs.

Illustrative Example

Clean trial (50K on django-13363 — resolved ✅):

Turn 1 (84 tokens):
THOUGHT: First, I need to understand the structure of the Django codebase...

```bash
find . -name "datetime.py" -path "*/db/*"

Turn 2 (62 tokens):
THOUGHT: I can see there's a datetime.py file in db/models/functions...

cat ./django/db/models/functions/datetime.py | head -50

[...continues productively for 15 turns, makes correct edit...]

**Degenerate trial** (100K on xarray-4629 — failed ❌):

Turn 1 (4096 tokens):
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[...4096 tokens of ! characters, hits max_tokens cap...]

Turn 2 (4096 tokens):
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[...same pattern repeats for all 15 turns...]

The model never produces a single coherent token on degenerate trials. This is fundamentally a **generalization failure** of the 8B model — it learned the mini-swe-agent format on training-distribution prompts but cannot reliably activate this behavior on all inputs.

### Why 100K is worse

The 100K model trained for 6250 steps (vs 3125 for 50K). The additional training:
- Sharpened the model's behavior on in-distribution prompts (Django, Astropy)
- But narrowed its activation region — prompts from xarray, pylint, pytest, etc. fall outside the narrower distribution and trigger the degenerate attractor
- This is classic **catastrophic narrowing** from overfitting on a narrow-domain dataset

[AlienKevin]

🤖 Root Cause Analysis: Degeneration in SWE-ZERO SFT Models

Update: The 100K "regression" reported earlier was based on a single eval run (4%). With 3 runs, 100K averages 5.3% ± 1.5% — actually the highest of all models. The degeneration analysis below still holds as a real phenomenon, but it doesn't cause a net performance regression because 100K compensates with stronger performance on non-degenerate trials.

Finding: Code blocks in prompts trigger degeneration

The !!! degeneration is strongly correlated with markdown code blocks (```) in the user prompt:

Prompt has code blocks?	10K Degen	50K Degen	100K Degen
No code blocks	8% (4/52)	17% (9/52)	15% (9/59)
Has code blocks	36% (16/45)	17% (10/58)	77% (53/69)

U-shaped sensitivity curve

The code-block sensitivity follows a U-shape across training:

• 10K (625 steps): Moderate sensitivity (36% degen on code-block prompts). Model hasn't fully learned the format yet — code fences in prompts partially confuse it.
• 50K (3125 steps): Sensitivity resolved (17% degen regardless of code blocks). Model learned to distinguish code blocks in user input from its own ```bash output blocks.
• 100K (6250 steps): Sensitivity amplified (77% degen on code-block prompts). Extra training eroded the input/output distinction.

Despite 100K's higher degeneration rate on code-block prompts, its overall resolve rate is still the highest (5.3% mean vs 4.0% for 50K) because it performs better on the ~40% of tasks without code blocks. This was obscured in a single trial but became clear with 3 runs.

How it works mechanically

1. The SWE-ZERO training format is: THOUGHT: reasoning + ```bash\ncommand\n```
2. The model learns that ``` signals "I should generate code"
3. When a user prompt contains ``` (e.g., a bug report with a reproducer snippet), the 100K model gets confused about whether it's reading input or generating output
4. This confusion causes it to skip the THOUGHT: prefix and go straight to generating tokens — but without the proper generation context, it falls into the ! repetition attractor
5. The very first token is ! instead of T (for THOUGHT), and the model never recovers

Implications

1. 100K is NOT worse than 50K overall — it just has a different failure profile (more degeneration on code-heavy prompts, but stronger performance elsewhere)
2. The degeneration is addressable: lower LR for longer training, or data augmentation with code-block-heavy system prompts could reduce the 100K degeneration rate without sacrificing its gains
3. For evaluation: multiple runs are essential — single-run results can be misleading due to high stochastic variance (±2% at temperature=1.0 on 100 tasks)

[AlienKevin]

🤖 Status check (2026-04-28 20:32 UTC)

• Pipeline (140B): 12 running, 1 pending v6e-4 jobs in us-east5 writing to gs://marin-us-east5 — no cross-region transfer. Sample shard sizes: 000=176MB, 050=326MB, 100=598MB, 125=140MB (post-restart, growing).
• 500K SFT (v5p-16): relaunching as v6 (v5 failed due to wrong working dir); v5p-16 in us-central1, output to gs://marin-us-central1 — no cross-region transfer.
• All workloads pinned to single US regions.

[AlienKevin]

🤖 Status check (2026-04-28 20:45 UTC)

• Pipeline (140B): 12 running + 1 pending v6e-4 jobs in us-east5. Output gs://marin-us-east5 totals 52.9GB; sample shards 000=177MB, 050=344MB, 100=598MB, 125=140MB. Single-region ✓
• 500K SFT: v6 failed — Levanter region check rejected v5p-16 landing in us-east5 with us-central1 cache. Patched script to pin regions=["us-central1"]; v7 just submitted. No checkpoints yet.
• All workloads in single US regions, no cross-region transfers.

[AlienKevin]

🤖 Status check (2026-04-28 21:15 UTC)

• Pipeline (140B): 11 running + 1 pending v6e-4 in us-east5 → gs://marin-us-east5 (53.3GB total). Sample shards: 000=177MB, 050=344MB, 100=598MB, 125=140MB. Single-region ✓
• 500K SFT: v7 failed — orchestrator VM landed in us-east5 while data is in us-central1. Relaunched as v8 with --region us-central1 on the iris entrypoint. No checkpoints yet; investigating if v5p-16 capacity available.
• All workloads single-region (no cross-region transfers).

[AlienKevin]

🤖 Status check (2026-04-28 21:38 UTC)

• Pipeline (140B): 11 running + 1 pending v6e-4 in us-east5 → gs://marin-us-east5 (53.9GB total, +0.6GB since last check). Sample shards: 000=177MB, 050=351MB, 100=598MB, 125=140MB. Single-region ✓
• 500K SFT (v8): orchestrator running on us-central1 entrypoint; train_lm child job pending v5p-16 capacity ("No workers match constraints"). executor_status=RUNNING, no checkpoints yet. Single-region ✓
• All workloads single-region (no cross-region transfers).