diff --git a/chapters/ai-tools/ai-harness-landscape.qmd b/chapters/ai-tools/ai-harness-landscape.qmd new file mode 100644 index 00000000..0eef7ba5 --- /dev/null +++ b/chapters/ai-tools/ai-harness-landscape.qmd @@ -0,0 +1,182 @@ +@sec-ai-harnesses explains what a harness is, +and @sec-ai-coding-agent-platforms lists common platforms as a starting point. +This section is the wider map: +which harnesses exist as of August 2026, +how they group, +and how to choose among them. + +::: {.callout-warning} +#### A fast-moving, opinionated snapshot + +This section condenses a research summary compiled in August 2026 +([issue #95](https://github.com/Morrison-Lab/wai/issues/95)), +which drew on vendor announcements, project repositories, and community discussion. +Harness releases arrive weekly, +default models changed several times during 2026, +and the corporate landscape reshuffled more than once, +so treat every named version, figure, and ownership claim as dated to that month. +Check the linked project pages before acting on any of it. +::: + +#### The organizing idea: agent = model + harness + +The consensus across the comparisons surveyed is that the frontier models have converged on the standard coding benchmarks, +so the harness layers that @sec-ai-harnesses describes +now do most of the differentiating. +The commonly cited puzzle is that the same Claude model performs noticeably better inside Claude Code than inside a model-agnostic harness, +and the answer offered is repeatedly the same word: the harness. + +Two consequences follow for reading the rest of this section: + +- **Benchmarks are saturating.** + Leading models cluster within about a point on SWE-bench Verified, + which is widely regarded as near its ceiling and exposed to training-data contamination. + Harder successors (SWE-bench Pro, Terminal-Bench 2.x) show much lower absolute scores and separate tools better, + but their figures are vendor-reported and version-specific. +- **Your own tasks are the benchmark that matters.** + A day of real work on a repository you ship reveals more than any leaderboard, + which is why @tbl-harness-landscape-choice ends in a pilot rather than a ranking. + +#### Terminal-first harnesses + +| Harness | License | Character (August 2026) | +|---|---|---| +| [Claude Code](https://www.anthropic.com/claude-code) (Anthropic) | Proprietary | The deepest programmable harness: hooks, skills, plugins, subagents, MCP, and multi-agent workflows. Widely rated first on code quality, and consistently the heaviest token spender, with real volume starting on the higher subscription tiers. | +| [OpenAI Codex](https://github.com/openai/codex) | Apache 2.0 CLI; proprietary cloud | The reach leader: terminal, cloud, IDE, ChatGPT app, and browser surfaces. Strong on terminal benchmarks, sandboxed at the OS level, markedly more token-efficient than Claude Code, and now carrying skills, plugins, subagents, and a small hook system. | +| [OpenCode](https://opencode.ai) | MIT | The most-starred open harness: model-agnostic across many providers including local models, client-server architecture with a headless server mode, custom agents, MCP, and `AGENTS.md`. Claude works only through an API key, not a subscription login. | +| [Aider](https://aider.chat/) | Apache 2.0 | The git-native pioneer: maps the repository and commits each edit. Reported to be in maintenance mode since early 2026, still respected for auditability and local-model flexibility. | +| [Goose](https://github.com/aaif-goose/goose) (Block, now Linux Foundation) | Apache 2.0 | Rust terminal agent with vendor-neutral governance, many model providers and MCP extensions, parallel subagents, and Agent Client Protocol support. Local-first with no hosted option. | +| [Gemini CLI](https://github.com/google-gemini/gemini-cli) (Google) | Apache 2.0 | Being folded into Antigravity CLI; issue #95 records free individual access ending on 2026-06-18, with enterprise Code Assist licenses unaffected. | +| [Amp](https://ampcode.com) | Proprietary | Deliberately optimizes for the best outcome regardless of token cost: multi-model routing, parallel subagents, and an "Oracle" deep-reasoning mode. Spun out of Sourcegraph. | + +: Terminal-first coding harnesses {#tbl-harness-landscape-terminal} + +#### Editor-integrated harnesses + +| Harness | License | Character (August 2026) | +|---|---|---| +| [Cursor](https://cursor.com/) | Proprietary | The in-editor leader: a VS Code fork with a deep agent loop, an in-house model tuned for fast agentic editing, and rules, MCP, hooks, skills, plugins, and subagents. Reads `.cursor`, `.claude/agents`, and `.codex/agents` configuration, per issue #95. | +| [GitHub Copilot](https://github.com/features/copilot) | Proprietary | The most widely distributed option, multi-model, with a cloud agent that turns an issue into a pull request in an ephemeral Actions environment; framed by GitHub for low-to-medium complexity tasks in well-tested codebases. | +| [Cline](https://github.com/cline/cline) | Apache 2.0 | The open-source VS Code standard: a plan-then-act workflow with human approval at each step, bring-your-own-key across many providers including local Ollama, and a Kanban surface for orchestrating parallel agents. The usual pick for audit-sensitive environments. | +| [Google Antigravity](https://antigravity.google/) | Proprietary | Google's agent-first development platform, built around an agent manager surface and now the center of its coding strategy; early reception focused on rate limits. | +| [Devin Desktop](https://devin.ai/) (Cognition) | Proprietary | The former Windsurf editor, rebranded in June 2026 with a local Devin agent, an agent command center, and Agent Client Protocol support. | +| [Continue](https://continue.dev/), [Tabnine](https://www.tabnine.com/), [Kilo Code](https://kilo.ai/), [Zed](https://zed.dev/) | Mixed | Specialists: Continue adds an open-source agent to an existing editor; Tabnine is the on-premises, air-gapped enterprise pick; Kilo Code carries on the Roo Code lineage after that project's 2026 shutdown; Zed is an open-source AI-native editor. | + +: Editor-integrated coding harnesses {#tbl-harness-landscape-editor} + +#### Autonomous and cloud harnesses + +- **Devin** (Cognition) is the archetype: + delegate a task, and parallel managed instances each run in an isolated cloud machine and open pull requests. + Early trials were poor, and the product has matured since. +- **[OpenHands](https://github.com/OpenHands/OpenHands)** (MIT) is the open-source autonomous leader, + running headless in CI to browse, edit, test, and retry. +- **The cloud tiers of interactive tools** + (Copilot's cloud agent, Codex Cloud, Cursor's cloud agents) + share one design assumption: + the bottleneck is no longer what one agent can do, + but how many you can direct and review at once. + [When orchestration helps](agent-orchestration.qmd#sec-orch-when) covers when that parallelism is worth having. + +#### The minimalist counter-trend + +Against the deep-harness orthodoxy, +a 2026 movement argues that frontier models already know what a coding agent is, +so a harness should get out of the way. +Its flagship is [Pi](https://github.com/earendil-works/pi) (MIT), +built by Mario Zechner and championed by Armin Ronacher: +four tools (read, bash, edit, write), +a system prompt plus tool definitions under a thousand tokens, +no MCP, plan mode, to-do list, or subagents, +and extension by asking the agent to write its own TypeScript extensions. +In its author's own Terminal-Bench 2.0 run +([post dated 2025-11-30, on Claude Opus 4.5](https://mariozechner.at/posts/2025-11-30-pi-coding-agent/)) +it placed eighth, +which its author read as evidence that a minimal harness competes with the heavy ones. + +The maximalist reply is that the extra tokens in a heavy harness encode product features +(memory, scheduled tasks, subagents, plan mode, worktree support) +rather than waste. +Even Ronacher concedes that task queues, orchestration, and durable sessions will matter more over time, +so the two camps may converge. + +#### Open source versus proprietary + +Open source, where you pay only your model provider or run local models: + +- OpenCode +- Codex CLI +- Cline +- Aider +- OpenHands +- Goose +- Continue +- Gemini CLI +- Pi +- Zed + +Proprietary: + +- Claude Code +- Cursor +- GitHub Copilot +- Devin +- Amp +- Antigravity +- Replit Agent +- Tabnine + +For raw capability the proprietary leaders still edge ahead, +but the open options are competitive and win outright on cost, privacy, auditability, and model freedom. +The 2026 signature of open-source use is bring-your-own-key, +with two friction points worth knowing. +Anthropic restricted third-party harnesses' use of flat-rate Claude subscriptions in April 2026, +which pushed some of those users onto extra-usage or per-token billing +(@sec-ai-claude-code-other-models covers the same rule from the Claude Code side), +and OpenCode lost Claude Pro and Max login access after a dispute with Anthropic. + +#### Local-model coding + +Local coding has matured to the point where the strongest open-weight models compete on the standard benchmarks. +The community favorites for consumer hardware are the Qwen, Devstral, Gemma, and DeepSeek families, +and the typical stack pairs OpenCode or Aider with a local OpenAI-compatible endpoint. +@sec-ai-offline and @sec-ai-small-local-models cover the mechanics and the model choice. + +#### Beyond coding: general agent frameworks + +The general agent-framework layer settled into three tiers: + +- **Graph-based orchestration:** [LangGraph](https://github.com/langchain-ai/langgraph), + the production default for stateful, auditable, human-in-the-loop workflows, and Mastra for TypeScript. +- **Role-based multi-agent:** [CrewAI](https://github.com/crewAIInc/crewAI), + and AutoGen, now merged with Semantic Kernel into the + [Microsoft Agent Framework](https://github.com/microsoft/agent-framework). +- **Lab SDKs:** the OpenAI Agents SDK, Anthropic's Claude Agent SDK, and Google's ADK. + +MCP and the Agent-to-Agent protocol moved to Linux Foundation stewardship, +and `AGENTS.md` and `SKILL.md` have become cross-tool standards in practice. +The practitioner warning that recurs in production surveys is that most deployed agents are short-horizon and single-agent, +so the prevailing advice is to start single-agent; +[when orchestration helps](agent-orchestration.qmd#sec-orch-when) makes the same case for this lab's own use. + +Browser-use agents are a fast-growing adjacent category, +and they remain structurally vulnerable to indirect prompt injection; +do not give one credentials or financial access without sandboxing and human checkpoints. + +#### Choosing + +| If you want | Reach for | +|---|---| +| Maximum code quality and programmable depth, and can absorb the token cost | Claude Code on a high-volume plan; drop to Codex or Cursor if the bill or the task mix does not justify it | +| Cross-surface continuity, terminal tasks, and lower cost | Codex, the safest default on price for most teams | +| To stay in the editor | Cursor, or Copilot if you are GitHub-native and want issue-to-pull-request automation | +| Open source, self-hosting, local models, or auditability | Cline (editor, approval-gated), OpenCode (terminal, headless), or OpenHands (autonomous CI); Tabnine for air-gapped deployments | +| Hands-off delegation of well-scoped backlog work | Devin or the cloud agents in Codex, Cursor, and Copilot, with humans still reviewing the pull requests | +| Control, observability, and cheap parallel runs in a sandbox | Pi or Goose; stay with a heavier harness for long unattended runs | + +: Matching a harness to the job {#tbl-harness-landscape-choice} + +Then pilot two of them on a repository you actually ship. +Token or usage-limit pain points toward Codex or an open bring-your-own-key harness; +a need for audit trails points toward Cline or LangGraph; +a need for private inference points toward OpenCode or Aider with a local model. diff --git a/chapters/coding-agents.qmd b/chapters/coding-agents.qmd index 378a63f2..198e4935 100644 --- a/chapters/coding-agents.qmd +++ b/chapters/coding-agents.qmd @@ -45,6 +45,10 @@ to [help you code](https://en.wikipedia.org/wiki/AI-assisted_software_developmen {{< include ai-tools/harness-agent-relationship.qmd >}} +# The Harness Landscape in 2026 {#sec-ai-harness-landscape} + +{{< include ai-tools/ai-harness-landscape.qmd >}} + # AI Agents and the Technological Singularity {{< include ai-tools/technological-singularity.qmd >}} diff --git a/inst/WORDLIST b/inst/WORDLIST index 78537f51..898b583b 100644 --- a/inst/WORDLIST +++ b/inst/WORDLIST @@ -1,9 +1,12 @@ +ADK Anthropic Antigravity Apache Apikey +Armin Asimov Atlas +AutoGen BYOK Badham Belcak @@ -21,6 +24,7 @@ Composio Contextify Conchords Cowork +CrewAI DOCX Databricks Daytona @@ -46,17 +50,20 @@ Homebrew Inflexa Integromat Jules +Kanban Karpathy Kepler Kimi Kong Kubrick LLMs +LangGraph LeCun LiteLLM LoRA MCP Maia +Mastra MattFlower Metis MiniMax @@ -90,14 +97,21 @@ QLoRA Qwen Rakazo REPL +Replit RevealJS +Ronacher +Roo +SDK +SDKs SERG SWE SemBr Sisyphus +Sourcegraph Stanford SuperGrok TORQCLAW +Tabnine Treg TUI TinyTeX @@ -112,7 +126,16 @@ YAGNI YAML Z.ai Zapier +Zechner Zsh +auditability +auditable +gapped +leaderboard +maximalist +observability +rebranded +stateful xAI agentic aider diff --git a/lychee.toml b/lychee.toml index d4241b31..7419068f 100644 --- a/lychee.toml +++ b/lychee.toml @@ -49,4 +49,8 @@ exclude = [ # GitHub fallback cannot see it with this repo's token, so it always # reads as broken. Anchored to that one URL so nothing else is masked. "^https://github\\.com/ucdavis/matt\\.contracts/pull/2$", + # Amp's site answers lychee's HTTP/2 probe with a protocol error + # ("Server may not support HTTP/2 properly") while loading normally in a + # browser; linked from chapters/ai-tools/ai-harness-landscape.qmd. + "^https://ampcode\\.com/?$", ]