Expand to full 500-task SWE-bench Verified

[nicolalessi]

Summary

The current benchmark evaluates on a 100-task stratified subset of the full 500-task SWE-bench Verified dataset. This issue tracks expanding to the complete 500-task evaluation.

Why we started with 100 tasks

• Statistical validity. 100 instances with stratified sampling across all 12 repositories provide a ±8.7% margin of error at 95% confidence. This is sufficient for directional comparisons and identifying significant performance gaps.
• Cost. A full 500-task run at $3/task budget costs up to $1,500 per agent. With 4+ agents to compare, a single round of benchmarking could exceed $6,000. The 100-task subset keeps the initial evaluation under $300 total.
• Reproducibility. A lower barrier to reproduction means more people can independently verify the results. The setup script handles everything and a full run completes in a few hours, not days.
• Comparison fairness. External agent resolution data from swe-bench/experiments is available per-instance, so we compare on the exact same 100 tasks — no extrapolation needed.

What the 100-task subset covers

• All 12 SWE-bench Verified repositories represented proportionally
• Subset median complexity (22) matches full dataset median (23)
• Outlier filtering: complexity ceiling at 250 removes extreme outliers (~1% of instances)
• Full methodology in docs/TASK_SELECTION.md

What expanding to 500 would add

• Tighter confidence intervals. Margin of error drops from ±8.7% to ±3.9%.
• Per-repository depth. Some repos only have 3-5 instances in the 100-task subset, making repo-level comparisons noisy. With 500 tasks, every repo has enough instances for meaningful repo-level analysis.
• Edge case coverage. The 100-task subset filters complexity outliers (>250). A full run would include these extreme cases, which may reveal different performance patterns.
• Leaderboard comparability. Most published SWE-bench results evaluate on the full 500-task Verified set (or 484 in Epoch's case). Running the full set makes direct comparison with published leaderboard numbers possible.

Estimated cost

Agent	Estimated cost (500 tasks at ≤$3/task)
vexp + Claude Code	~$335 (based on $0.67/task average)
Without vexp baseline	~$500-1,000 (based on comparable agents)

Total for a two-configuration run: approximately $800-$1,300.

How to contribute

If you want to help with this:

1. Run the existing 100-task benchmark on a new agent and submit results (see issues Add Cursor agent adapter #1, Add Codex CLI agent adapter #2, Add Gemini CLI agent adapter #3)
2. Sponsor a full 500-task run — if you have API credits to spare and want to run the full set, the harness supports it out of the box: just pass --data with the full dataset JSONL
3. Analyze the existing 100-task results — help identify which additional tasks would be most informative for the expanded run

Running on the full dataset

The harness already supports this:

# Download the full 500-task dataset (setup.sh only downloads the 100-task subset by default)
python scripts/download_full_dataset.py
 
# Run on all 500 tasks
node dist/cli.js run --data data/swebench-verified-full.jsonl

We'll update the benchmark page and comparison charts once a full 500-task run is completed and validated.