Benchmark: SWE-bench Verified — close gap from 10% to competitive

[0bserver07]

What This Benchmark Measures

SWE-bench Verified is the gold-standard benchmark for evaluating AI coding agents on real-world software engineering tasks. It contains 500 human-verified GitHub issues from 12 popular Python repositories. Each task requires the agent to read a natural-language issue description, navigate a real codebase, localize the bug or feature request, and produce a code patch that passes the repository's test suite. The "Verified" variant filters out ambiguous or unsolvable instances from the original SWE-bench, making scores more reliable.

Current Status

• Our score: 10% (2/20 on a small sample)
• Reference leaders: Claude Opus 4.5 80.9%, GLM-5 w/OpenHands 77.8%
• Gap: ~70 percentage points

Root Causes Identified

1. Max iterations: 30 vs OpenHands' 500 — agent gives up too early on complex repos
2. Action space: bash-only vs bash+IPython+Jupyter — limits exploration patterns
3. Condensation: window truncation vs LLM summarization — loses context on long runs
4. Action queueing: single action vs multiple per LLM call — slower iteration

Dataset

• Source: princeton-nlp/SWE-bench_Verified on HuggingFace
• Size: 500 verified instances
• Format: HuggingFace dataset (auto-downloads as JSONL)
• Download:

  from datasets import load_dataset
  ds = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")

How to Run

Quick smoke test (5 instances, CodingAgent)

cd /path/to/chimera
export ANTHROPIC_BASE_URL="https://api.z.ai/api/anthropic"
export ANTHROPIC_AUTH_TOKEN="..."
export ANTHROPIC_MODEL="glm-5.1"

python examples/swe_bench_coding_agent.py --count 5 --output data/swebench-smoke.jsonl

Full 500-instance run

python examples/swe_bench_coding_agent.py --count 500 --output data/swebench-full.jsonl

Using the harness API directly

from chimera.eval.benchmarks.swe_bench import SWEBench
from chimera.eval.harness import Harness

bench = SWEBench(dataset_path="data/swe-bench-verified.jsonl", limit=20)
harness = Harness(benchmark=bench, agent=my_agent, env_factory=docker_env_factory)
result = harness.run()
print(f"{result.passed}/{result.total} = {result.pass_rate:.1%}")

Adapter

• File: chimera/eval/benchmarks/swe_bench.py (already exists)
• Class: SWEBench(Benchmark)
• Key methods: tasks() returns task dicts with id, prompt, repo, base_commit, test_patch; evaluate() applies test patch and runs pytest in the environment
• Runner: examples/swe_bench_coding_agent.py — full CodingAgent runner with trace capture

Trace System

The trace system (chimera/eval/trace.py) captures comprehensive diagnostics per instance:

• TraceCollector records every LoopEvent from AgentLoop
• RunTrace computes: diagnosis (EXPLORE_ONLY, NO_EDITS, EDIT_FAILURES, MAX_TURNS, WRONG_PATCH), tools_used, files_read, files_edited, failed_edits
• Traces are saved as JSON to data/traces/{instance_id}.json

Integration with CodingAgent

• Preset: swebench (defined in chimera/assembly/presets.py) — permissions off, hooks off, streaming off, max_turns=30
• Tools needed: bash, read_file, edit_file, write_file, search, glob, git
• Environment: Local clone per instance (git clone + checkout base_commit into tmpdir)

Reference Scores

Model / Agent	Score	Source
Claude Opus 4.5 (best scaffold)	80.9%	SWE-bench leaderboard Mar 2026
Claude Opus 4.6	80.8%	SWE-bench leaderboard Mar 2026
Gemini 3.1 Pro	80.6%	SWE-bench leaderboard Mar 2026
GLM-5 w/ OpenHands	77.8%	SWE-bench leaderboard
Claude Sonnet 4.5	77.2%	SWE-bench leaderboard
Chimera (current)	10%	Internal 20-instance run

Target: 20%+ with strong model (first milestone), 40%+ with iteration improvements.

Action Items

• Increase max_turns from 30 to at least 100 in swebench preset
• Implement LLM-based context condensation (replace window truncation)
• Add IPython action space for richer exploration
• Support multiple actions per LLM call
• Verify OpenHands' 77.8% by running it directly with GLM-5

Test Plan

• Download SWE-bench Verified dataset (500 instances)
• Run 5 instances as smoke test with CodingAgent
• Analyze traces for failure patterns using RunTrace.diagnosis
• Run 20 instances, target 20%+ resolve rate
• Run full 500-instance suite
• Write report to docs/benchmarks/swe_bench.md