AgentPatchCheck

Coding Agent Runtime & Evaluation Harness for controlled, repository-level software repair.

HAL SWE-bench Verified Mini fixed-50

Metric	Result
Official resolved	35 / 50
Resolution rate	70.0%
Valid executions	50 / 50
Harness-invalid	0
Grading-invalid	0

This is the end-to-end result of deepseek-v4-flash + APC Runtime/tooling + frozen execution policy, graded by the official SWE-bench evaluator. It is not a full SWE-bench Verified score, a model-only score, or an independently measured APC causal uplift.

Agent Loop · Tool Calling · Workspace Isolation · Budget & Termination · Verification · Trace · Official Evaluation

Why AgentPatchCheck

A completed agent turn or emitted diff is not proof of a correct repair. APC bounds the runtime, controls repository tools and workspaces, records the execution trace, verifies the patch, and keeps application behind a separate approval gate. It is a practical harness for building, testing, and evaluating coding agents without conflating completion with correctness.

🏗️ Architecture

flowchart LR
    task[TaskSpec] --> policy[Policy validation]
    policy --> workspace[Isolated Git workspace]
    workspace --> runtime[Agent Runtime]
    runtime --> loop[Agent loop]
    runtime --> broker[Tool broker]
    runtime --> state[Attempts, budget, termination]
    broker --> tools[File, search, shell, Git]
    tools --> verify[Public verification]
    verify --> evidence[Evidence, trace, patch]
    evidence --> console[APC Console: local read-only viewer]
    evidence --> assess[Assessment and approval]
    assess --> apply[Guarded apply]

Loading

Frozen SWE-bench execution remains a distinct upstream-evaluator path:

flowchart LR
    manifest[Frozen manifest] --> runtime[APC Runtime and tooling]
    runtime --> prediction[Prediction JSONL]
    prediction --> evaluator[Official SWE-bench evaluator]
    evaluator --> outcome[Resolved or unresolved]

Loading

Core Capabilities

• Agent loop and tool calling: bounded iterations, calls, output, deadlines, cancellation, and recovery.
• Workspace isolation: per-run Git worktrees plus constrained file, search, shell, and Git tools.
• Verification and assessment: public verification, hidden Oracle evaluation, risk checks, approval, and guarded apply.
• Evidence and trace: structured trajectories, execution results, patches, structured evidence artifacts, assessment reports, and benchmark reports with sensitive-text redaction.
• Evaluation orchestration: sequential benchmark execution, prediction generation, evaluator bridging, and independent execution/grading validity tracking.

APC Console

APC Console is a local, read-only Runtime & Evaluation viewer for imported Agent runs, verification results, artifacts, and benchmark analysis. It uses browser-authorized folder import and recursive discovery to consume existing artifacts only.

cd apc-console
npm ci
npm run dev

It supports:

• Open Local Workspace with recursive artifact discovery.
• Runs browsing and Run Detail with verification facts.
• An Artifact explorer and preview for imported runtime and evaluator outputs.
• Benchmark Evaluation with the canonical failure taxonomy and termination × correctness analysis.

The Console is the observation and evaluation layer after controlled execution and independent evaluation; it is not an execution control plane. It never writes back to the workspace, launches a Candidate, executes a Provider or benchmark, or exposes Apply, Retry, or mutation controls.

Run inspection

Run Detail exposes the imported verification facts for an individual real run.

Benchmark evaluation

The frozen aggregate view makes the official HAL result, execution validity, failure taxonomy, and termination × correctness analysis directly inspectable.

Benchmark Analysis

Observed non-resolved categories

Observed non-resolved category	Count
Budget-bound termination	11
Incorrect / incomplete fix	3
Provider failure	1

Budget-bound termination is the dominant observed endpoint pattern among the 15 non-resolved runs (11/15), indicating substantial execution-budget pressure under the frozen policy. This does not establish that increasing the budget would necessarily resolve those tasks.

Termination and correctness

Observed pairing	Count
iteration-limit → officially resolved	7
tool-limit → officially resolved	1
model-failed → officially resolved	1
finished → officially unresolved	3

Termination status should not be used as a substitute for evaluator-derived resolution. Termination reasons remain useful operational and diagnostic signals.

Research Note

Auditing Runtime Termination as a Resolution Proxy: A Frozen Coding-Agent Case Study

A configuration-specific empirical audit of typed runtime termination as a resolution proxy under one frozen APC configuration. The mechanical finished → resolved / non-finished → unresolved rule disagreed with the official evaluator on 12/45 binary-outcome runs (≈26.7%), with errors in both directions.

🚀 Quick Start

Prerequisites

• Node.js 22 or newer
• npm 10 or newer
• Git
• An installed agent/provider required by the TaskSpec

Headless development path

Install root dependencies and run the existing development CLI with a TaskSpec that points to your own checkout and target repository:

npm install
npm run agentpatchcheck:run -- --task-spec ./path/to/task-spec.json

The script invokes tsx src/agentpatchcheck/cli.ts run. Read the Headless Core guide before preparing a TaskSpec or applying a patch.

Repository build

For the repository build:

npm run install:all
npm run build
node dist/agentpatchcheck.js --help

How It Works

Benchmark evaluation

Frozen Manifest → APC Runtime / Tooling → Prediction → Official SWE-bench Evaluator → BenchmarkReport → APC Console

1. A frozen manifest drives APC Runtime and tooling to produce predictions.
2. The official SWE-bench evaluator is used only for this benchmark path and produces the BenchmarkReport.
3. APC Console reads existing runtime artifacts and benchmark outputs locally; it does not participate in execution.

See Headless Core for the headless repair lifecycle, CLI, evidence, retention, and guarded-apply contracts.

Implementation Boundaries

Area	APC implements	Reused / integrated
Agent execution	Agent runtime/loop, tool broker, attempts, budgets, termination, evidence, assessment	Provider model reasoning
Repository operations	Policy-controlled workspace lifecycle and repair orchestration	Git / worktree primitives
Benchmarking	Manifest orchestration, prediction bridge, validity tracking, reports	Official SWE-bench evaluator and upstream datasets
Code Mode	APC integration and tool execution boundary	Selected DeepSeek Harness mechanisms

Third-party components and retained upstream attribution are documented in THIRD_PARTY_NOTICES.md. APC does not claim authorship of the official evaluator or complete benchmark datasets.

Reproducibility

The fixed-50 benchmark is backed by a frozen dataset revision, selected manifest, evaluator revision, model profile, and execution policy.

View the frozen benchmark reproduction contract

Documentation

• Headless Core — TaskSpec, CLI, evidence, assessment, approval, cleanup, and apply.
• Benchmark reproduction contract — frozen fixed-50 dataset, evaluator, model, environment, and invocation details.
• Harness-native Benchmark Suite — deterministic suite boundaries and reports.
• Third-party notices — retained attribution and license notices.

Limitations

• Fixed-50 is not a complete SWE-bench Verified score.
• This result has no alternative-runtime ablation and does not isolate APC Runtime causal uplift.
• Budget-bound termination is not proof that additional budget would resolve a task.
• Provider access, evaluator setup, Docker images, upstream repositories, and benchmark data are external dependencies.

Project Status

AgentPatchCheck is an open-source engineering project focused on auditable, bounded software-repair execution and truthful benchmark reporting. Interfaces and operational requirements may change before a stable release.

Licensed under Apache-2.0. See THIRD_PARTY_NOTICES.md for retained upstream copyright and attribution.