Experiment: SWE-ZERO multi-language scaling (20 languages × 5 PRs × 3 runs)

[AlienKevin]

Scale the execution-free SWE-ZERO recipe (#4561, now complete for Python) to all 20 programming languages in SWE-rebench V2.

Background

The SWE-ZERO MVP produced 1000 high-quality agentic rollouts on 100 Python PRs from SWE-rebench V2 in 38 minutes on a v6e-8 worker. Final recipe: ricdomolm/mini-coder-1.7b + mini-swe-agent v1 scaffold + PATH-whitelist bash sandbox + 32K context. Results at #4561 (closing comment) and HF dataset AlienKevin/SWE-ZERO-1k-trajectories-32k.

SWE-rebench V2 has 32,079 instances across 20 languages:

language	PRs available		language	PRs available
python	7,243		swift	362
go	6,144		dart	251
ts (TypeScript)	4,204		c	230
js (JavaScript)	4,138		cpp	182
rust	3,123		csharp	173
java	1,716		r	157
php	1,445		clojure	105
kotlin	889		ocaml	58
julia	793		lua	39
elixir	416
scala	411

Goal

Get a first look at how the Python recipe generalizes to other languages — specifically whether mini-coder-1.7b (trained primarily on Python data) still produces grounded multi-turn bash trajectories when pointed at Go, Rust, TypeScript, etc.

Sampling plan

• 100 PRs total spanning all 20 languages with as much coverage as possible
• 3 rollouts per PR (total = 300 rollouts) — chosen to keep the experiment cheap while still letting us compute a meaningful per-PR submission rate
• 5 PRs per language for the 20 languages. This gives uniform coverage regardless of how much training data each language has in SWE-rebench V2 (some languages only have 39–58 instances).
• Use seed=7 for deterministic sampling per language, matching prior steps
• Same filter: only include PRs where the base repo can be shallow-cloned and test_patch applies cleanly (if any don't, fall back to the next PR in that language's seeded sample)

Recipe (unchanged from the MVP final config)

uv run iris --cluster marin job run \
  --job-name swe-zero-multilang \
  --tpu v6e-4,v6e-8 --enable-extra-resources \
  --cpu 16 --memory 24GB --disk 60GB \
  --extra vllm \
  --env-vars VLLM_TPU_SKIP_PRECOMPILE 1 \
  --env-vars VLLM_ALLOW_LONG_MAX_MODEL_LEN 1 \
  --env-vars VLLM_TPU_DISABLE_TOPK_TOPP_OPTIMIZATION 1 \
  --env-vars MARIN_VLLM_MODE native \
  --env-vars HF_TOKEN $HF_TOKEN \
  -- python experiments/swe_zero/run_swe_zero_multilang.py --local \
       --model ricdomolm/mini-coder-1.7b \
       --n-prs 100 --n-rollouts 3 \
       --tensor-parallel-size 4 --max-num-seqs 256 \
       --max-model-len 32768 --max-total-tokens 32768 \
       --concurrency 64 \
       --output_dir gs://marin-us-central2/experiments/swe_zero_multilang

A new entry point run_swe_zero_multilang.py will be added that drops the language_filter="python" in SWERebenchV2Loader and samples 5 PRs per language instead of 10 rollouts × 10 PRs × 10 repos.

Estimated wall time: 300 rollouts × ~2.3s/rollout (at 1.83 rps, v6e-4 TP=4) ≈ 7–10 min.

Metrics to report

Per language (and aggregate):

• Submission rate (fraction of rollouts exiting via echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT)
• Estimated resolve rate at empirical pass@1 and pass@3 — replay each sampled rollout in a fresh worktree to recover the model's effective source-code diff, then LLM-judge it against the SWE-rebench V2 gold_patch for semantic equivalence (same conservative sub-agent judge as Experiment: SWE-ZERO MVP — Execution-free agentic rollout generation #4561's resolve-rate sample). For each PR with s resolved rollouts out of n=3, compute 1 - C(n-s, k)/C(n, k) and average across PRs to get pass@k. Report:
  • Resolved pass@1 (fraction of individual rollouts that produce a correct fix)
  • Resolved pass@3 (fraction of PRs with at least one correct fix in their 3 rollouts)
  • The Python MVP showed ~13% pass@1 and ~20% pass@10 — the multilang point is to see how those numbers degrade per language
• Mean turns / rollout, mean completion tokens / rollout
• command not found observations — does the model try language-specific interpreters (node, rustc, go, gradle, …) that aren't on the PATH whitelist?
• Diversity: corpus-wide AND within-PR mean Jaccard (the within-PR breakdown is the honest one — the corpus-wide number is dominated by trivially-different cross-PR pairs)
• Top-15 bash command first-words per language — does the exploration idiom (find | grep | cat) transfer, or does the model default to Python patterns?

Hypotheses

1. Submission rate will drop for non-Python languages because the system prompt's examples and the model's SFT data are Python-biased (find . -name "*.py").
2. command not found rate will climb for compiled / JVM / JS languages — the model will reach for go build, cargo check, tsc, gradle based on Python-side intuitions about "verify the fix".
3. Exploration idioms will transfer — grep/cat/find/sed are language-agnostic, so the first 2–3 turns should still be templated.
4. Resolve rate will be very low (well under 13% Python pass@1) for languages with structural build systems where a simple sed -i patch is insufficient.

Deliverables

• New script: experiments/swe_zero/run_swe_zero_multilang.py
• Raw rollouts: gs://marin-us-central2/experiments/swe_zero_multilang/rollouts.json
• Per-language report: gs://.../multilang_report.json
• HF dataset: AlienKevin/SWE-ZERO-multilang-300-trajectories in the same chat-viewer format as the Python datasets
• Follow-up comment on this issue with the per-language table, hypotheses-check, and recommendations for the next scale step (more languages? more PRs per language? Python-leaning model swap for non-Python langs?)

Part of #4435

[AlienKevin]

🤖 First-look results — multi-language scaling on 225 rollouts (partial)

The full 300-rollout run hit a worker preemption mid-run after 225/300 rollouts had already been incrementally checkpointed to GCS, and the v6e capacity contention prevented the restart from being reassigned. Rather than block on more retries, here's what we have from the partial corpus.

Coverage: 225 rollouts across 17 of 20 languages and 80/100 PRs. Missing 3 languages (scala, swift, ts) and partial coverage of python, php, r, rust — these all sit at the end of the alphabetical iteration order, so the preemption truncated them.

Recipe: same as Step 6 32K (commit 9f4c945e4, branch kevin/swe-zero-mvp):

• Model ricdomolm/mini-coder-1.7b, vLLM-tpu on a v6e-4 worker, TP=4
• max_total_tokens=32768, max_model_len=32768, max_num_seqs=256, concurrency=64, MAX_TURNS=30, temperature=1.0
• Sampler: 5 PRs per language, 3 rollouts per PR, seed=7

Note on the v3 runner: I had to rewrite run_swe_zero_multilang.py to spawn vllm serve via subprocess.Popen instead of importing marin.inference.vllm_server, because the recently-merged main upgraded transformers/torch such that importing marin.evaluation.evaluators.evaluator transitively pulls in the CUDA-build torch which crashes at module init on TPU workers. (The Iris worker drops uv.lock from the bundle when >1MB, so --extra vllm resolves fresh and picks the wrong torch.) The subprocess workaround keeps transformers/torch out of the orchestrator process — vLLM imports them in its own process where vllm-tpu's torch_xla flavor takes precedence.

Per-language metrics

lang	rollouts	PRs	submission rate (pass@1)	submission rate (pass@3)	mean turns	mean completion tok	command not found %	within-PR Jaccard
js	15	5	66.7%	100%	26.1	4,621	1.6%	0.166
julia	15	5	46.7%	80%	26.3	3,886	1.8%	0.202
c	15	5	46.7%	60%	25.4	3,047	0.8%	0.226
lua	15	5	46.7%	100%	26.7	4,047	1.3%	0.186
python	11	5	45.5%	60%	24.9	2,769	3.4%	0.302
r	6	3	50.0%	66.7%	25.7	2,660	2.6%	0.137
php	7	4	42.9%	75%	24.9	3,061	2.3%	0.150
clojure	15	5	40.0%	60%	24.4	4,992	2.6%	0.213
cpp	15	5	40.0%	100%	28.7	4,248	0.9%	0.186
go	15	5	33.3%	60%	28.9	4,990	0.7%	0.179
java	15	5	26.7%	60%	28.0	5,382	1.7%	0.274
elixir	15	5	26.7%	60%	27.1	4,897	1.0%	0.169
csharp	15	5	20.0%	60%	29.2	4,245	2.3%	0.203
dart	15	5	20.0%	40%	28.8	4,838	1.9%	0.213
rust	6	3	16.7%	33.3%	22.3	1,790	0.0%	0.178
kotlin	15	5	13.3%	20%	28.1	4,031	1.2%	0.179
ocaml	15	5	6.7%	20%	28.7	3,245	2.1%	0.192
TOTAL	225	80	34.2%	62.5%	26.9	4,109	1.6%	~0.20

(pass@1 = fraction of individual rollouts that submit; pass@3 = fraction of PRs where at least one of the 3 rollouts submits.)

Hypothesis check

#	hypothesis	verdict
1	Submission rate will drop for non-Python	Mostly false — Python (45.5%) is not the leader. JavaScript wins at 66.7%, and Julia/C/Lua tie Python. The drop only really shows up in OCaml (6.7%), Kotlin (13.3%), Rust (16.7%), Dart (20%), C# (20%) — exotic syntax or strong build-system dependencies.
2	command not found rate will climb for compiled / JVM / JS langs	False — every language has a lower CNF rate than Python (1.6% aggregate vs Python's 6.4% in Step 6 32K). The model doesn't reach for cargo/gradle/tsc/mvn at all in this sample. It defaults to bash exploration when it doesn't know the language's verification tools.
3	Exploration idioms (find/grep/cat/sed) will transfer	True — every single language uses these same 4 as its top 4 commands. The aggregate top-4 across all 17 langs: grep (1699), sed (1260), find (809), cat (733). 77% of all bash commands.
4	Resolve rate will be very low for non-Python	Not measured yet — would need replay + LLM-judge on the 225 rollouts. Submission rate is the proxy in this report.

Top-15 bash command first-words (aggregate)

cmd	count		cmd	count
grep	1,699		tail	96
sed	1,260		cp	90
find	809		awk	68
cat	733		wc	36
ls	350		pwd	21
echo	225		python	4
cd	173		node	3
git	137		javac	3
head	103		php	2

Total language-specific tool attempts (across 225 rollouts, 17 langs): 11, all single-digit per language. Compare to Python's Step 6 32K which had ~6% of all observations as command not found (mostly python/pytest). The model is less Python-biased than I expected — it doesn't transfer the "verify by running" instinct to languages it doesn't have strong priors for.

Headlines

1. JavaScript is the surprise winner at 66.7% pass@1 / 100% pass@3. Likely because mini-coder-trajs-400k contains some JS data, the codebases are typically smaller and cat | sed | grep is sufficient to make a fix, and JS source files don't have build-system gotchas.
2. Python is no longer the headline champion — 45.5% in this sample, tied with C, Julia, Lua. The original Step 6 Python rate of 48.6% was skewed by sampling 100 PRs from a single repo set; this multilang sample has 5 random PRs per language.
3. The exploration idiom is universal. The "templated" find → cat → grep → sed opener that we saw on Python in Experiment: SWE-ZERO MVP — Execution-free agentic rollout generation #4561 transfers to every language with no friction. Per-language top-1 first-word distribution looks identical.
4. Compiled-language exotics struggle: OCaml (6.7%), Kotlin (13.3%), Rust (16.7%), Dart (20%), C# (20%). These are languages with type systems or build systems that make a one-shot sed -i fix unlikely to be correct. The model attempts the fix and submits anyway, but the patches won't pass tests.
5. Within-PR Jaccard sits in the 0.14–0.30 range across all languages, basically matching Python's 0.23. So sampling diversity at temperature=1 transfers cleanly.

What's missing (and follow-ups)

• 3 languages not sampled: scala, swift, ts. From the trends I'd predict scala behaves like java/kotlin (~20-30%), swift like dart (~20%), ts like js (~50-60%).
• Resolve rate not yet measured. The submission rate is informative but the Step 6 lesson was that "submission ≠ correctness" (only ~13% of Python submissions were actually correct). I'll do a replay + LLM-judge pass on 5–10 PRs across the best/worst languages and post a follow-up with resolve pass@1 / pass@3 to put the Python 13% number in context.
• No HF dataset upload yet — once we either complete the missing 3 langs or decide the partial is enough, I'll publish to AlienKevin/SWE-ZERO-multilang-300-trajectories (or 225 if we ship the partial as-is).

Artifacts:

• gs://marin-us-central2/experiments/swe_zero_multilang/rollouts_partial_225.json (partial 225-rollout corpus, preserved before the restart attempt)
• gs://marin-us-central2/experiments/swe_zero_multilang/sampling_plan.json

[AlienKevin]

🤖 Final results — full 300-rollout multi-language run

The 100-PR × 3-rollout multilang corpus is now complete across all 20 languages in SWE-rebench V2. Final corpus: gs://marin-us-central2/experiments/swe_zero_multilang/rollouts_merged_300.json (300 rollouts, all 20 langs at 5 PRs × 3 rollouts each).

Per-language metrics (sorted by pass@1)

lang	pass@1 (sub)	pass@3 (sub)	mean turns	mean compl tok	cnf %	within-PR jaccard
js	66.7%	100%	26.1	4,621	1.6%	0.166
swift	60.0%	80%	23.4	4,345	1.2%	0.235
c	46.7%	60%	25.4	3,047	0.8%	0.226
julia	46.7%	80%	26.3	3,886	1.8%	0.202
lua	46.7%	100%	26.7	4,047	1.3%	0.186
clojure	40.0%	60%	24.4	4,992	2.6%	0.213
cpp	40.0%	100%	28.7	4,248	0.9%	0.186
python	40.0%	40%	26.1	3,477	5.5%	0.233
ts	40.0%	80%	27.2	4,870	3.0%	0.174
go	33.3%	60%	28.9	4,990	0.7%	0.179
elixir	26.7%	60%	27.1	4,897	1.0%	0.169
java	26.7%	60%	28.0	5,382	1.7%	0.274
php	26.7%	40%	29.3	4,232	2.6%	0.240
csharp	20.0%	60%	29.2	4,245	2.3%	0.203
dart	20.0%	40%	28.8	4,838	1.9%	0.213
r	20.0%	40%	28.6	3,513	0.9%	0.205
kotlin	13.3%	20%	28.1	4,031	1.2%	0.179
scala	13.3%	40%	29.3	4,494	2.8%	0.262
ocaml	6.7%	20%	28.7	3,245	2.1%	0.192
rust	0.0%	0%	30.0	3,837	1.1%	0.193
TOTAL	31.7%	~58%	27.4	4,288	1.7%	~0.21

Hypothesis check (refreshed against full 300)

#	hypothesis	verdict
1	Submission rate will drop for non-Python	Mostly false — JS (66.7%) and Swift (60%) both beat Python (40%). The drop only really shows up in OCaml (6.7%), Rust (0%!), Kotlin (13.3%), Scala (13.3%), Dart/R/CSharp (20%).
2	command not found rate will climb for compiled / JVM / JS langs	Still false — every language has a lower CNF rate than Python's 5.5%. Aggregate CNF is 1.7% vs Python's 5.5%. The model doesn't reach for cargo/gradle/tsc/mvn — it sticks to bash exploration.
3	Exploration idioms (find/grep/cat/sed) will transfer	True — top-4 commands are grep/sed/find/cat for every language.
4	Resolve rate will be very low for non-Python	Not measured — submission is the proxy. Rust 0% submission strongly suggests resolve@1 ≈ 0% there.

Surprises

• Rust's 0/15 submission rate. Despite cargo being absent (no CNF spike), the model just never produces a sed -i patch confident enough to submit. Long, indecisive trajectories (mean 30 turns, max). Likely caused by the borrow checker / lifetime annotations being beyond what mini-coder-1.7b can pattern-match without execution feedback.
• JavaScript wins decisively at 66.7% pass@1. Smaller codebases, no compilation, single-file fixes, and (probably) more JS in mini-coder's SFT data.
• Swift surprises at 60%. Swift is JVM-/Kotlin-adjacent in difficulty per usual benchmarks, but here it behaves more like JS. Worth re-checking this with a larger sample — could be lucky 5-PR draw.
• Python is no longer the headline champion — 40% in this sample, behind JS, Swift, and tied with C/Julia/Lua. The 48.6% from Step 6 was from a curated 100-PR sample on the same repo set; sampling 5 random PRs across the full Python corpus drops it to in-line with mid-tier languages.
• Within-PR Jaccard sits in the 0.17–0.27 range across all languages, very close to Python's 0.23 from Step 6. Sampling diversity at temperature=1 transfers cleanly.

Code / job recipe

Built on commit 2ebb05351 (kevin/swe-zero-mvp). Two changes from #4561's recipe:

1. run_swe_zero_multilang.py spawns vllm serve via subprocess.Popen instead of importing marin.inference.vllm_server — needed because the recently-merged main pulled in a CUDA-build torch via the transformers import chain that crashes on TPU at module init.
2. --extra vllm --extra tpu in the Iris job submission (instead of --extra vllm alone) so uv pins torch to the pytorch-cpu index. Without --extra tpu the resolver picks CUDA torch and the worker venv is unusable.

uv run iris --cluster marin job run \
  --job-name swe-zero-multilang \
  --tpu v6e-4,v6e-8 \
  --enable-extra-resources \
  --priority interactive \
  --cpu 16 --memory 24GB --disk 60GB \
  --extra vllm --extra tpu \
  --env-vars VLLM_TPU_SKIP_PRECOMPILE 1 \
  --env-vars VLLM_ALLOW_LONG_MAX_MODEL_LEN 1 \
  --env-vars VLLM_TPU_DISABLE_TOPK_TOPP_OPTIMIZATION 1 \
  --env-vars HF_TOKEN $HF_TOKEN \
  -- python experiments/swe_zero/run_swe_zero_multilang.py \
       --local \
       --model ricdomolm/mini-coder-1.7b \
       --n-per-language 5 --n-rollouts 3 \
       --tensor-parallel-size 4 --max-num-seqs 256 \
       --max-model-len 32768 --max-total-tokens 32768 \
       --concurrency 64 \
       --output_dir gs://marin-us-central2/experiments/swe_zero_multilang

Artifacts

• gs://marin-us-central2/experiments/swe_zero_multilang/rollouts_merged_300.json — final 300-rollout corpus
• gs://marin-us-central2/experiments/swe_zero_multilang/rollouts_partial_225.json — partial pre-merge
• gs://marin-us-central2/experiments/swe_zero_multilang_tail/multilang_report.json — scala/swift/ts metrics
• gs://marin-us-central2/experiments/swe_zero_multilang_topup/multilang_report.json — python/php/r/rust metrics

Resolve-rate via LLM-judge is the next follow-up — would replay all 300 rollouts and judge against gold patches, similar to the #4561 closing comment for Python.

[AlienKevin]

🤖 Resolve-rate estimate for the 300-rollout multilang corpus

Same methodology as the Python MVP (#4561 closing comment): replay each rollout in a fresh materialize_worktree to recover its effective source-code diff, then run a conservative LLM-as-judge sub-agent (one per language, judging all 15 rollouts of that language at once) that compares each diff to the SWE-rebench V2 gold_patch. The judge defaults to NOT_RESOLVED on any uncertainty, empty diff, broken syntax, wrong file, or partial fix; only clearly functionally-equivalent source changes count.

Per-language resolve rate (sorted by pass@1)

lang	submitted (pass@1)	resolved (pass@1)	resolved (pass@3)	submission→resolve drop
swift	60.0%	20.0%	20.0%	−67%
c	46.7%	13.3%	20.0%	−71%
julia	46.7%	13.3%	20.0%	−71%
ocaml	6.7%	13.3%	20.0%	+99% (judge found correctness in non-submitting rollouts)
clojure	40.0%	6.7%	20.0%	−83%
php	26.7%	6.7%	20.0%	−75%
js	66.7%	0.0%	0.0%	−100%
lua	46.7%	0.0%	0.0%	−100%
python	40.0%	0.0%	0.0%	−100%
cpp	40.0%	0.0%	0.0%	−100%
ts	40.0%	0.0%	0.0%	−100%
go	33.3%	0.0%	0.0%	−100%
elixir	26.7%	0.0%	0.0%	−100%
java	26.7%	0.0%	0.0%	−100%
csharp	20.0%	0.0%	0.0%	−100%
dart	20.0%	0.0%	0.0%	−100%
r	20.0%	0.0%	0.0%	−100%
kotlin	13.3%	0.0%	0.0%	−100%
scala	13.3%	0.0%	0.0%	−100%
rust	0.0%	0.0%	0.0%	n/a
TOTAL	31.7%	3.7%	6.0%	−88%

Headlines

• Aggregate resolved pass@1 ≈ 3.7%, pass@3 ≈ 6.0% across 300 rollouts. Out of 100 PRs sampled across all 20 languages, only 6 PRs ever produced a correct fix in any of their 3 rollouts — the 11 resolved trajectories are concentrated on those 6 PRs.

• Submission rate dramatically over-estimates correctness in every language. Even JavaScript, the runaway submission winner at 66.7%, has 0% resolved. The model's echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT is not correlated with whether the patch actually works.

• Six languages produced any correct fix at all: swift (3), c (2), julia (2), ocaml (2), clojure (1), php (1). The other 14 languages — including Python — produced zero correct fixes out of 15 rollouts each.

• Python's 0/15 is striking given the 13% pass@1 resolve rate from Experiment: SWE-ZERO MVP — Execution-free agentic rollout generation #4561's Python-only sample. Two reasons:

  • The Experiment: SWE-ZERO MVP — Execution-free agentic rollout generation #4561 sample was 10 PRs × 10 rollouts on a curated single-repo set; this is 5 random PRs from the entire SWE-rebench V2 Python corpus, which is a much harder distribution.
  • The 5 PRs picked here (mkdocs, serpent-tools, mpmath, rdt, vyper) are mostly multi-file structural changes — exactly the failure mode the Python sample also identified.

• The resolved-pass@1-vs-submission gap is ~88% in aggregate: 28 percentage points of submission rate evaporate when you actually check correctness. The few languages where the gap is smaller (ocaml, swift) are mostly noise from the 5-PR sample, not real signal.

• Where correctness exists, it's mechanical. Looking at the 6 PRs that resolved at least once:

  • redis-13711 (c) — 1-line buffer index correction in src/networking.c, 2 byte-identical rollouts
  • corexlsx-139 (swift) — < comparison fix on intValue, 3 different rollouts all correct
  • qcheck-255 (ocaml) — <= 0 → < 0 change in a shared validator helper, 2 rollouts
  • unitful.jl-533 (julia) — adds a :cispi symbol to a list, 2 rollouts
  • pip-license-checker-156 (clojure) — adds a regex literal #"PSF-2.0", 1 rollout
  • livewire-2223 (php) — sets $this->shouldSkipRender = true in 3 redirect methods, 1 rollout

  All single-symbol or single-line refactors. No multi-file fix landed in any language, including the few PRs where resolution did happen.

Comparison with Python MVP (#4561 closing)

	Python-only #4561	Multilang #4653
sample	10 PRs × 10 rollouts on 1 repo set	5 PRs × 3 rollouts on 100 different PRs
submission rate	48.6%	31.7%
resolved pass@1	13.0%	3.7%
resolved pass@10 / pass@3	20.0% (k=10)	6.0% (k=3)
PRs with ≥1 resolved	2/10 (20%)	6/100 (6%)

The drop from 13% → 3.7% pass@1 is partly smaller k (3 rollouts per PR instead of 10) and partly harder distribution (random sampling across 20 languages and unfamiliar build systems vs a curated Python set).

Caveats

• n=5 PRs per language is small. The 95% CI on a per-language pass@1 of 13% (2/15) is roughly 2%–40%, so the per-language rankings are noisy. The aggregate 3.7% over 300 rollouts is more reliable.
• Conservative LLM judge: the sub-agent saw each diff without running tests. It can flag obviously-wrong diffs but may miss subtle equivalences. The judge was instructed to default to NOT_RESOLVED on uncertainty, so the 3.7% number is a slight under-estimate. Spot-checking a few of the 0% languages (js, lua, cpp, python) showed the judge was correct: those rollouts are syntactically broken, edit the wrong file, or miss the gold's structural changes.
• OCaml's "+99%" anomaly (6.7% submitted but 13.3% resolved) is the judge finding that 2 out of 15 OCaml rollouts produced a correct < 0 patch but never echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT. This is an artifact of the model not learning OCaml's submission idioms — it makes a correct edit but can't recognize that it's done.

Bottom line

Mini-coder-1.7b under SWE-ZERO produces correct fixes for 6% of PRs across the full SWE-rebench V2 language distribution, vs 20% on a curated Python sample. The model is fundamentally pattern-matching one-symbol/one-line refactors and cannot do multi-file structural changes in any language without execution feedback. Submission rate is not a usable proxy for correctness — every language's submission rate over-estimates resolve rate by ~70-100%.

For SWE-ZERO synthetic-data generation, this means the trajectories are still useful as pre-training data (the model is doing real exploration in 74% of observations), but using the Submitted flag as a label for "this is a correct fix" would inject ~85% wrong labels into the training set.

Raw judge results: /tmp/multilang_judge_results/*.json on the workstation.

[AlienKevin]

🤖 Closing — multilang scaling complete.

The 300-rollout × 20-language run is done with the full pipeline (sample → run → measure → judge). Findings:

• Submission rate: 31.7% aggregate, headlined by JS (66.7%) and Swift (60%); Rust at 0/15.
• Resolved pass@1: 3.7% aggregate (LLM-judged vs gold patches). Six languages produced any correct fix; 14 produced zero.
• Submission ≠ correctness: the gap is ~88% in aggregate. JS submitted 66.7% but resolved 0%. The model can't distinguish "I made some edits" from "I made the right edits" in any language.
• Where correctness exists, it's mechanical — every resolved PR is a single-symbol or single-line refactor. Zero multi-file fixes landed in any language.
• Exploration idiom transfers cleanly: find/grep/cat/sed are top-4 in every language. Within-PR Jaccard sits in the 0.17–0.27 range across all 20 languages, matching Python.

Artifacts:

• AlienKevin/SWE-ZERO-multilang-300-trajectories — 300 rollouts in chat format with language column
• gs://marin-us-central2/experiments/swe_zero_multilang/rollouts_merged_300.json
• /tmp/multilang_judge_results/*.json (per-language LLM judgments)

For SWE-ZERO synthetic data: trajectories are still useful as pre-training tokens (74% useful observations), but the Submitted flag is not a usable correctness label — it would inject ~85% wrong "correct fix" labels.

Side fix shipped: #4663 (draft) — pin torch to the pytorch-cpu index in the vllm extra so --extra vllm alone resolves correctly on Iris workers without needing the --extra vllm --extra tpu workaround.

Closing as complete. Next experiment: scale to 1B tokens of rollouts across the full SWE-rebench V2 corpus — see follow-up issue.