Skip to content

summarize the current state of AI harnesses #95

Description

@dem-extra1

The State of AI Agent Harnesses (August 2026): Coding Agents and Beyond

TL;DR

  • The model race has flattened at the top, so the harness now decides your experience. As of August 2026, the best-regarded coding harnesses are Claude Code (deepest programmable terminal harness, top on SWE-bench Verified), OpenAI Codex (broadest cross-surface reach, 5M+ weekly users), and Cursor (strongest in-editor agent), with Cline, OpenCode, Amp, OpenHands, and Devin rounding out the field. Which is "best" depends on the job — terminal depth, cross-surface autonomy, in-editor speed, or hands-off delegation.
  • Open-source has become a first-class tier, not a fallback. Codex CLI (Apache 2.0), OpenCode (MIT), Cline (Apache 2.0), OpenHands (MIT), Goose (now Linux Foundation-governed), and Aider give full model freedom and BYOK economics; the frontier of quality still tilts slightly to proprietary Claude Code/Cursor/Amp, but the gap is narrow.
  • The landscape has consolidated and reshuffled hard in 2026: Windsurf became Devin Desktop, SpaceX agreed to buy Cursor's parent Anysphere for ~$60B, Google is folding Gemini CLI into Antigravity CLI, Roo Code shut down, Aider went into maintenance mode, and a new minimalist-harness movement (led by "Pi") is challenging the "deep harness" orthodoxy.

Key Findings

1. "Agent = Model + Harness" is the organizing idea of 2026. Across nearly every serious comparison, the consensus is that frontier models have largely converged on benchmarks, and the scaffolding around the model — the loop, tools, memory, permissions, and context management — now does most of the differentiating work. A widely cited community framing: "People often over-index on the model and under-index on the execution loop." Researcher Jack Morris' viral question — why the same Claude model is far more effective inside Claude Code than inside OpenCode — is repeatedly answered with a single word: the harness.

2. The clear front-runners are Claude Code, Codex, and Cursor, with a strong second tier of open-source and specialist tools. Claude Code leads on raw harness depth and code-quality benchmarks; Codex leads on reach, cost, and terminal tasks; Cursor leads in-editor.

3. Benchmarks are saturating and should be read skeptically. Top models now cluster within ~1 point on SWE-bench Verified, which is widely regarded as near-saturated and contamination-exposed. OpenAI itself has moved to retire SWE-bench Verified as a frontier signal. Harder, contamination-resistant benchmarks (SWE-bench Pro, Terminal-Bench 2.x) show much lower absolute scores and remain the more useful discriminators — but they are vendor-reported and version-specific, so treat cross-tool comparisons as approximate.

4. Consolidation is real. 2026 saw the Windsurf→Devin Desktop rebrand, the SpaceX/Anysphere deal, Sourcegraph spinning Amp out as its own company, Google sunsetting Gemini CLI's free individual tier, Roo Code's shutdown, and Aider's slide into maintenance-only releases.

5. A counter-movement toward minimalism emerged. The "Pi" harness (four tools, sub-1,000-token system prompt) placed second on Terminal-Bench 2.0 and sparked a genuine industry debate about whether heavy harnesses (Claude Code's hooks, MCP, sub-agents, Dynamic Workflows) help or hurt.

Details

CLI-based harnesses (terminal-first)

Claude Code (Anthropic) — proprietary; the depth leader. Runs in the terminal plus VS Code, JetBrains, the web (claude.ai/code), and mobile. Its default model has moved up Anthropic's fast release cadence — Opus 4.8 (May 28, 2026), then Claude Opus 5 (July 24, 2026) and Sonnet 5 became the current flagships. Its harness is the deepest in the field: a hooks system exposing roughly two-to-three dozen lifecycle events you can script (commonly cited as ~30, though the exact count is unverified against Anthropic's docs), plus Skills, plugins, sub-agents, MCP, and "Dynamic Workflows" that orchestrate tens-to-hundreds of parallel sub-agents. Claude Opus reports the highest published SWE-bench Verified numbers (around 88.6% for Opus 4.8; frontier Claude models are quoted at 95–96% on some trackers, reflecting benchmark saturation). Community sentiment is strongly positive on quality but consistently flags two things: it is the heaviest token spender (community head-to-heads put it at ~3–4× Codex on the same task), and its $20 Pro tier hits usage limits fast, with real volume starting on the $100+ Max plan. r/ClaudeCode is the most active dedicated community (reportedly ~4,200 weekly contributors, roughly triple r/Codex). A March 2026 usage-limit controversy drew an Anthropic apology.

OpenAI Codex — Apache 2.0 CLI + proprietary cloud; the reach leader. A Rust binary that also runs as a cloud service, IDE extension, ChatGPT app, mobile, and Chrome extension. Per OpenAI's June 2, 2026 report Codex is becoming a productivity tool for everyone: "Codex now has more than 5 million weekly active users, up more than 6x since the launch of the desktop app in February… knowledge workers now represent about 20 percent of users and are growing." The default model has advanced to the GPT-5.5/GPT-5.6 "Codex" family (GPT-5.6 Sol became the default in July 2026). Codex leads terminal-native benchmarks (Terminal-Bench 2.x), uses OS/kernel-level sandboxing (Seatbelt, Landlock, seccomp, bubblewrap), and is markedly more token-efficient than Claude Code. It closed most of the "bare-bones" gap in 2026 by adding Skills, a plugin marketplace, sub-agents, and a (small) hook system. The dominant complaint on r/codex is mid-session usage-limit cuts. A notable Reddit take: "With the right skills, Codex is honestly better than Claude Code for me" (468 upvotes).

OpenCode (Anomaly, formerly SST) — MIT; the open-source favorite. Model-agnostic across 75+ providers including local models, with ~171–172k GitHub stars and a claimed 7.5M monthly developers — the most-starred open harness. Client-server architecture (opencode serve runs a headless OpenAPI server), custom agents/sub-agents, MCP, AGENTS.md, LSP support, and cross-compatible SKILL.md skills. Caveat: Anthropic prohibits Claude Pro/Max subscription use in OpenCode, so Claude works only via API key. A cult following praises its TUI.

Aider — Apache 2.0; the git-native pioneer, now fading. The original git-first terminal pair programmer (maps the repo, auto-commits each edit, 100+ providers via LiteLLM). Multiple 2026 sources report it is now in maintenance mode — the last stable release, v0.86.2, shipped February 12, 2026, with no new feature releases since. Still respected for clean auditability and local-model flexibility, but no longer the tool practitioners reach for for long unattended runs. ~41–46k GitHub stars, 5–6.8M installs.

Goose (Block → Linux Foundation) — Apache 2.0. A Rust-based terminal-first agent runtime donated by Block to the Linux Foundation's new Agentic AI Foundation (AAIF) on April 7, 2026, giving it vendor-neutral governance. 25+ model providers, 70+ MCP extensions, sub-agents (up to 10 parallel workers), and Agent Client Protocol (ACP) support to reuse Claude Code/Codex subscriptions. ~51–53k stars. Local-first with no hosted option.

Gemini CLI (Google) — Apache 2.0; being sunset. Google's open-source terminal agent with a generous free tier (1,000 requests/day) is being folded into Antigravity CLI, with free/consumer access for individuals ending June 18, 2026 (enterprise Code Assist license holders keep access).

Amp (Sourcegraph, now spun out as Amp, Inc.) — proprietary. A "frontier" CLI + VS Code agent that deliberately optimizes for best outcomes with "unfettered access to tokens and tools" rather than low cost. Multi-model routing, parallel sub-agents, an "Oracle Mode" for deep reasoning, and a free ad-supported tier plus a paid zero-data-sharing "Smart Mode." Reported ~40,000 teams adopting it in early 2026. Grew out of the deprecated Sourcegraph Cody (new Cody signups ended July 2025).

The minimalist-harness movement — "Pi" (badlogic/pi-mono) — MIT. Built by Mario Zechner (creator of libGDX) and championed by Armin Ronacher (creator of Flask/Jinja, at Sentry), Pi is the flagship of a 2026 counter-trend. Its design is radically bare: four tools only (read, bash, edit, write) and a system prompt plus tool definitions "below 1000 tokens." Zechner's thesis (Nov 30, 2025): frontier models are "RL-trained up the wazoo, so they inherently understand what a coding agent is. There does not appear to be a need for 10,000 tokens of system prompt." It deliberately ships no MCP, no plan mode, no built-in to-dos, no sub-agents, and is "YOLO by default" (relying on external sandboxing rather than permission prompts). Extension happens by asking the agent to write its own TypeScript extensions ("agents built for agents building agents"), with hot-reloading and tree-structured sessions. In Zechner's own Terminal-Bench 2.0 run (Dec 2, 2025, Pi on Claude Opus 4.5), Pi placed 2nd on the leaderboard, behind only Terminus 2 (the benchmark's own ultra-minimal reference agent) — "more evidence that a minimal approach can do just as well." MIT-licensed, TypeScript, ~30k+ GitHub stars (higher figures circulate but are unverified). Ronacher's June 23, 2026 essay "The Coming Loop" is the movement's key text, distinguishing the "agent loop" from the "harness loop" and cautioning that "present-day hands-off harnesses like Claude Code with ultracode produce worse code than what we were producing last autumn." Notably, even Ronacher concedes that "task queues for coding tasks, orchestration of agents, subagents, durable sessions will matter more and more" — a sign the minimalist and maximalist camps may converge. A representative Hacker News counterpoint defends heavier harnesses: "most of the extra tokens encode product features and rational design choices: a memory system, scheduled tasks, sub-agents, plan mode, worktree support."

IDE-integrated harnesses

Cursor (Anysphere) — proprietary; the in-editor leader. A VS Code fork with a deep agent loop, its in-house Composer model (Composer 2.5, May 18, 2026) tuned for fast agentic editing. Artificial Analysis scored Composer 2.5 at 62 on its Coding Agent Index (third overall, behind only higher-effort Claude Opus variants) and the cheapest agent above 60, at ~$0.07/task. It ships Rules, MCP, Hooks, Skills, Plugins, and Sub-agents, and reads .cursor, .claude/agents, and .codex/agents configs. The big corporate story: after a $2.3B Series D at $29.3B (Nov 2025), SpaceX agreed to acquire Anysphere for ~$60 billion in all-stock on June 16, 2026 — confirmed by SpaceX's SEC Form 8-K filed that day (via merger subsidiary X67 Inc., at $60.0B implied equity value), widely described as the largest acquisition of a venture-backed startup ever, expected to close Q3 2026 on reported annualized revenue of roughly $4B (~15× revenue). Cursor's July 2025 pricing controversy (which prompted a public apology) still colors newcomer sentiment.

GitHub Copilot (Microsoft/GitHub) — proprietary; the reach champion. The most widely distributed option (VS Code + github.com), multi-model across Anthropic/OpenAI/Google. Its standout is a cloud agent that turns a GitHub issue into a PR inside an ephemeral Actions environment (sessions cap ~59 minutes, one repo/branch, firewall on by default). Framed by GitHub as best for "low-to-medium complexity tasks in well-tested codebases"; it publishes no SWE-bench score. Per GitHub's Octoverse 2025 report, "nearly 80% of new developers on GitHub use GitHub Copilot within their first week." Pricing moved to usage-based "AI Credits" in 2026.

Cline — Apache 2.0; the open-source VS Code standard. The most widely adopted open-source coding agent by installs (8M+ installs, ~62–64k GitHub stars). A sidebar agent for VS Code (plus JetBrains, Cursor, Zed, Neovim, and a preview CLI/SDK) built on a Plan/Act two-phase workflow with human approval at each step — the right choice for regulated/audit-sensitive environments. BYOK across 12+ providers including local Ollama. Added an embeddable SDK (May 2026) and "Cline Kanban" for orchestrating parallel agents across Cline, Claude Code, and Codex. Samsung's DX division adopted it in 2025.

Google Antigravity — proprietary; the agent-first IDE. Launched November 20, 2025 as an "agentic development platform" built around an Agent Manager surface, running Gemini 3.5 Flash by default (also offers Claude and gpt-oss). It is the new center of Google's coding strategy (Antigravity 2.0, CLI, SDK, and Managed Agents shipped at I/O 2026). Early reception was rough — r/Bard threads call its rate limits "a slap in the face" to paying subscribers.

Devin Desktop (Cognition, formerly Windsurf) — proprietary. On June 2, 2026, Cognition rebranded the Windsurf IDE as Devin Desktop via an over-the-air update, completing its acquisition of Windsurf (agreed July 14, 2025). The built-in Cascade agent was replaced by "Devin Local" (rewritten in Rust, ~30% more token-efficient, with sub-agents); Cascade was retired July 1, 2026. The default surface is now an "Agent Command Center" (a Kanban board for local and cloud agents) plus "Spaces," and it ships open Agent Client Protocol (ACP) support. Cognition raised over $1B at a ~$26B valuation in May 2026.

Continue, Tabnine, Kilo Code, Zed — specialist IDE options. Continue (Apache 2.0) adds an agent to existing VS Code/JetBrains with BYO/local models. Tabnine is the on-prem/air-gapped enterprise pick (SOC 2, zero retention, IP indemnification). Kilo Code (fork of Roo Code, which forked Cline) preserves the Architect/Code/Debug/Ask modes after Roo's shutdown. Zed is an open-source AI-native editor.

Autonomous / cloud-based harnesses

Devin (Cognition) — proprietary; the autonomous archetype. Delegate a task and parallel "Managed Devins" each run in isolated cloud VMs, opening PRs when done. Cognition reports a 67% PR merge rate (up from 34% a year earlier). Devin earned heavy early skepticism (an Answer.AI trial logged "14 failures and just 3 successes" across 20 tasks; its original unassisted SWE-bench score was 13.86%), but Devin 2.0 dropped entry pricing to $20 and the product has matured.

OpenHands (All Hands AI) — open-source (MIT); the open autonomous leader. The best-performing open-source autonomous agent on SWE-bench Verified (widely cited at ~72%+), built on the CodeAct framework; runs headless in CI, browses, edits, tests, and retries. Raised an $18.8M Series A in June 2026 and shipped a composable Software Agent SDK plus the "OpenHands Index" benchmark (Jan 2026). ~75–76k GitHub stars.

GitHub Copilot cloud agent, Codex Cloud, Cursor Cloud Agents — the async tier of otherwise-interactive tools, all running sandboxed cloud tasks that open PRs. The shared design assumption: the bottleneck is no longer what an agent can do, but how many you can direct and review at once.

Open-source vs proprietary — the current split

  • Free and open source: OpenCode (MIT), Codex CLI (Apache 2.0, though cloud/models are paid), Cline (Apache 2.0), Aider (Apache 2.0), OpenHands (MIT), Goose (Apache 2.0, LF-governed), Continue (Apache 2.0), Gemini CLI (Apache 2.0, being sunset), Pi (MIT), Zed. You pay only your model provider (BYOK) or run local models.
  • Proprietary: Claude Code, Cursor, GitHub Copilot, Devin/Devin Desktop, Amp, Google Antigravity, Replit Agent, Tabnine.
  • Practical verdict: For pure capability, proprietary Claude Code/Cursor/Amp still edge ahead, but open-source options (especially Cline for VS Code, OpenCode for terminal/self-host, OpenHands for autonomous CI work) are now genuinely competitive and win outright on cost, privacy, auditability, and model freedom. The 2026 open-source signature is BYOK — bring your own key. Note two friction points: Anthropic restricted third-party harnesses' use of flat-rate Claude subscriptions (an April 2026 policy change pushed some third-party-harness users onto per-token billing), and OpenCode lost Claude Pro/Max login access after a dispute with Anthropic.

Local-model coding (r/LocalLLaMA)

Local coding has matured: the best open-weight models now score ~80% on SWE-bench Verified. The community favorites for consumer hardware are the Qwen 3.6 family (27B dense for tool-using agents; 35B-A3B MoE for tighter VRAM), Devstral Small 2 (Mistral, 24B, agentic), Gemma 4, and DeepSeek variants; server-class open weights like GLM-5.2 (SWE-bench Pro ~62.1) and Kimi K2.7-Code are used as API options. The typical local stack pairs OpenCode or Aider with a local OpenAI-compatible endpoint via Ollama/llama.cpp/vLLM.

General-purpose (non-coding) agent harnesses and frameworks

Distinct from coding harnesses, the general agent-framework layer settled into three tiers in 2026:

  • Graph-based orchestration: LangGraph (the production default for stateful, auditable, human-in-the-loop workflows; ~34.5M monthly downloads, 87% task success in one comparative test) and Mastra (TypeScript).
  • Role-based multi-agent: CrewAI ("agents as employees," ~45,900 GitHub stars) and AutoGen — the latter now merged with Semantic Kernel into Microsoft Agent Framework (v1.0 GA, April 2026), putting AutoGen into maintenance mode.
  • Lab SDK-native: OpenAI Agents SDK, Anthropic's Claude Agent SDK (minimal tool-use loop), Google ADK.
  • Protocol standardization: MCP and Agent-to-Agent (A2A) moved to Linux Foundation stewardship; MCP is now near-universal across frameworks and harnesses, and AGENTS.md / SKILL.md have become de facto cross-tool standards. An emerging practitioner warning: production-agent surveys increasingly find that most deployed agents are short-horizon and single-agent — one industry study of production/pilot systems found 68% execute at most 10 steps before human intervention — and that multi-agent designs add substantial cost and debugging complexity. The prevailing advice is "start single-agent."

Browser-use agents are a fast-growing adjacent category: the open-source Browser Use library, Skyvern (vision-first, 21.5k stars, 10M+ workflows), Stagehand (Playwright-based), plus consumer agentic browsers (Perplexity Comet, ChatGPT Atlas, Google's Chrome "auto browse" powered by Gemini 3). A key caution: browser agents are structurally vulnerable to indirect prompt injection — per Anthropic's Claude Opus 4.8 system card (May 28, 2026), the browser agent was hijacked 31.5% of the time per attempt without safeguards, dropping to 0.5% with safeguards engaged. The Perplexity Comet prompt-injection incident was a public wake-up call. Well-funded new entrants (e.g., Hark, which raised $700M Series A) keep arriving.

Community sentiment (HN, Reddit, X)

  • Hacker News has "mostly moved past" model-vs-model arguments toward harness/workflow questions: context preservation over long sessions, codebase inspection efficiency, composability with shell/git/browser, and supervisability. "Harness engineering" is now a discussed discipline, with practitioners noting that "lots of cool-sounding ideas can have a negative impact on performance due to emergent and confounding effects."
  • Reddit sentiment splits by tool: r/ClaudeAI (~747k members) and r/ClaudeCode broadly hold that "Claude wins on code, ChatGPT wins on ecosystem," and that Claude Code is "the best coding agent in the market, and it's not close" — tempered by cost/usage-limit complaints. r/codex users counter that Codex, properly configured, matches or beats it at lower token cost; one sentiment analysis of 500+ comments found ~80% of upvote-weighted Reddit sentiment preferred Codex on token efficiency/usability, even as Claude Code has ~4× the discussion volume. r/LocalLLaMA is focused on Qwen/GLM/DeepSeek local stacks.
  • X/Twitter: Andrej Karpathy's late-January 2026 post capturing the shift — "I rapidly went from about 80% manual+autocomplete coding and 20% agents in November to 80% agent coding and 20% edits+touchups" — became a widely shared marker of the moment. Predictions that "2026 will be the year of Agent Harnesses" circulated widely.

Recommendations

Start by matching the harness to the job, then pilot two on a repo you actually ship — a single day of real work reveals more than any benchmark.

  1. If you want maximum code quality and programmable depth (and can absorb the token cost): Use Claude Code on a Max plan. Trigger to change this: if your token bill exceeds budget or you mostly do well-scoped tasks, drop to Codex or Cursor.
  2. If you want cross-surface continuity, terminal tasks, and lower cost: Use OpenAI Codex. It is the safest default for most teams on price/performance, and its 5M-user scale means fast iteration and a deep ecosystem.
  3. If you live in an editor: Use Cursor (fastest, cheapest-per-task in-editor agent) or GitHub Copilot (if you're GitHub-native and want issue-to-PR automation). Monitor the SpaceX/Anysphere integration for any disruption to Cursor's roadmap or neutrality — that's the trigger to re-evaluate.
  4. If you need open-source, self-hosting, local models, or auditability: Use Cline (VS Code, approval-gated, regulated-friendly), OpenCode (terminal/headless, model-agnostic), or OpenHands (autonomous CI issue-fixing). For air-gapped/regulated deployments, use Tabnine or a local model via OpenCode/Ollama.
  5. If you want hands-off delegation of well-scoped backlog work: Use Devin or the cloud agents in Codex/Cursor/Copilot — but keep humans reviewing PRs; results remain uneven on open-ended work.
  6. For general (non-coding) agents: Default to LangGraph when stakes and state matter; CrewAI or Microsoft Agent Framework for fast prototyping; and start single-agent — only add multi-agent orchestration for a genuine compliance boundary, parallelizable workload, or team-ownership split.
  7. Watch the minimalist trend. If you value control, observability, and cheap parallel runs in a sandbox, trial Pi or Goose. If your workflow needs long unattended runs, stay with a heavier harness for now.

Thresholds that should change your choice: token/usage-limit pain → switch toward Codex or an open BYOK harness; need for audit trails → Cline or LangGraph; need for local/private inference → OpenCode/Aider + Qwen 3.6/Devstral; benchmark saturation means you should weight your own task performance and cost over headline SWE-bench numbers.

Caveats

  • Fast-moving and version-churny. Both Claude Code and Codex ship roughly weekly; model defaults changed multiple times in 2026 (Opus 4.6→4.7→4.8→Opus 5; GPT-5.4→5.5→5.6). Any specific version or price here has a short shelf life.
  • Model naming is genuinely confusing right now. Anthropic introduced additional model lines in 2026 (the "Mythos"-class Fable 5 and Mythos 5, alongside Opus/Sonnet/Haiku), some of which were briefly suspended under a June 2026 US government export-control directive and later restored. Several third-party trackers disagree on exact model names and version numbers; where sources conflicted, this report favored primary vendor disclosures and flagged uncertainty.
  • Benchmarks are vendor-reported, version-specific, and partly saturated. SWE-bench Verified is near its ceiling and contamination-exposed; SWE-bench Pro's "best score" varies by which split you cite (standardized public vs. vendor aggregate vs. private). Read all scores as directional, not definitive.
  • Adoption/star counts vary by source and are snapshots; some SEO/AI-generated comparison sites inflate figures. Where possible this report used primary disclosures (OpenAI's 5M Codex users, GitHub's Octoverse, Anthropic/Cognition/SpaceX announcements) and flagged secondary claims (e.g., Pi's star count, unverified Pi benchmarks attributed to Databricks/Shopify).
  • A few specifics could not be fully verified against primary sources, including Claude Code's exact hook-event count ("~30"), some Amp adoption numbers, the highest circulating Pi star/download figures, and the precise "47 deployments / 3.7× debugging penalty" multi-agent statistic (the closest verifiable figure is that ~68% of surveyed production agents run ≤10 steps before human intervention). These are labeled as such in the text.
  • Prompt-injection and autonomy risk remain unsolved for browser and fully autonomous agents; do not give agents credentials or financial access without sandboxing and human checkpoints.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions