Expand harness docs, README, and offline verify script

.github/workflows/ci.yml

@@ -0,0 +1,25 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  verify-harness:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install test deps
+        run: pip install pytest
+
+      - name: pytest verify_harness
+        run: pytest tests/test_verify_harness.py -v
+
+      - name: Mock harness script
+        run: python scripts/verify_harness.py --mock --json

.gitignore

@@ -0,0 +1,2 @@
+tests/__pycache__/
+.venv/

README.md

@@ -1,4 +1,78 @@
-# swebench-results
-Cognition's results and methodology on SWE-bench
+# Cognition SWE-bench Results
 
-Find the blog post here: https://cognition-labs.com/post/swe-bench-technical-report
+Cognition's public SWE-bench evaluation harness, agent prompts, and patch outputs.
+
+**Methodology:** [SWE-bench technical report](https://cognition.ai/blog/swe-bench-technical-report)  
+**Detailed harness docs:** [docs/HARNESS.md](docs/HARNESS.md)
+
+This repository documents how Cognition adapted SWE-bench for **autonomous agents** (not assisted LLM file lists). It does **not** reproduce Devin scores — it publishes harness mechanics and result artifacts.
+
+---
+
+## Repository layout
+
+| Path | Purpose |
+|------|---------|
+| `harness/` | Python package that generates agent setup scripts, eval scripts, and prompts per SWE-bench instance |
+| `output_diffs/pass/` | Agent patches that passed grading |
+| `output_diffs/fail/` | Agent patches that failed grading |
+| `docs/HARNESS.md` | Step-by-step eval pipeline (git hygiene, patch ordering, grading) |
+| `scripts/verify_harness.py` | Mock validator for harness invariants (no dataset download) |
+
+---
+
+## Quickstart
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+
+# Validate harness invariants on bundled mock instance (offline)
+python scripts/verify_harness.py --mock
+
+# Inspect generated scripts for a real instance (requires `datasets` + network)
+python - <<'PY'
+from harness.dataset import get_dataset
+from harness.scripts import make_test_spec
+
+spec = make_test_spec(get_dataset()[0])
+print(spec.instance_id)
+print(spec.setup_script[:500])
+print("---")
+print(spec.eval_script[:500])
+PY
+```
+
+---
+
+## Eval pipeline (summary)
+
+Cognition's agent eval differs from the original SWE-bench LLM setting:
+
+1. **Setup** — clone repo at `base_commit`, strip `git remote`, pre-create conda env  
+2. **Agent run** — agent receives issue text + repo; may edit any files (45 min ceiling in original report)  
+3. **Grading prep** — reset files touched by `test_patch` to pre-test state  
+4. **Patch extraction** — collect agent diff from filesystem (excluding test files)  
+5. **Apply order** — apply **agent patch**, then **test_patch**  
+6. **Verify** — run fail-to-pass tests; all must pass  
+
+See [docs/HARNESS.md](docs/HARNESS.md) for the full sequence with code references.
+
+---
+
+## Harness verification script
+
+`scripts/verify_harness.py` checks ordering and hygiene invariants without running SWE-bench:
+
+```bash
+python scripts/verify_harness.py --mock --json
+```
+
+Checks include: git remote removed, test files reset before grading patch apply, agent patch before test patch, non-empty fail-to-pass manifest.
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE). SWE-bench dataset terms apply separately ([LICENSE.swebench](LICENSE.swebench)).

docs/HARNESS.md

@@ -0,0 +1,200 @@
+# SWE-bench Agent Eval Harness
+
+This document describes Cognition's **agent-adapted** SWE-bench evaluation pipeline as implemented in this repository and described in the [SWE-bench technical report](https://cognition.ai/blog/swe-bench-technical-report).
+
+It covers **harness mechanics only** — not Devin model weights, proprietary prompts, or score reproduction.
+
+---
+
+## How agent eval differs from original SWE-bench
+
+| Aspect | Original SWE-bench (LLM) | Cognition agent harness |
+|--------|--------------------------|-------------------------|
+| Input | Issue + file list (assisted) or retrieval (unassisted) | Issue + full repo checkout |
+| Environment | Pre-built per instance | Agent may install deps; conda env pre-created in setup script |
+| Patch source | Model output | Filesystem diff after agent session |
+| Time limit | N/A (single shot) | 45 minutes (original Devin report) |
+| Git hygiene | Standard checkout | **Remote removed** to prevent leakage |
+| Test files | Hidden from model | Agent may edit; **reset before grading** |
+
+---
+
+## Phase 1 — Environment setup (`setup_script`)
+
+Generated by `harness/scripts.py::make_test_spec()`.
+
+### 1.1 Clone and pin commit
+
+```bash
+git clone -o origin --no-tags https://github.com/{owner}/{repo} {repo_directory}
+cd {repo_directory}
+git reset --hard {base_commit}
+```
+
+### 1.2 Strip git remote (leakage prevention)
+
+```bash
+git remote remove origin
+```
+
+> During test setup, we remove the Github remote and all future commits from the repository so agents can't access those directly.
+> — [SWE-bench technical report](https://cognition.ai/blog/swe-bench-technical-report)
+
+This prevents `git pull` from fetching the solution commit or test patches from upstream.
+
+### 1.3 Conda environment
+
+The harness selects install instructions from `harness/constants.py` (`MAP_VERSION_TO_INSTALL`, `MAP_REPO_TO_INSTALL`) based on repo + version. Setup may use:
+
+- `requirements.txt` fetched at the instance's commit via `harness/utils.py::get_requirements()`
+- `environment.yml` via `get_environment_yml()`
+- Inline conda package lists
+
+### 1.4 Agent prompt
+
+The standardized agent prompt (`SWEBENCH_PROMPT` in `harness/scripts.py`) includes:
+
+- Repository directory name
+- Conda env name
+- GitHub issue text (`problem_statement`)
+
+No file hints are provided — the agent navigates the repo freely.
+
+---
+
+## Phase 2 — Agent execution
+
+Not implemented in this repo (Devin is closed source). The agent:
+
+- Reads the issue
+- Explores the codebase
+- Edits source files
+- May run tests / install tools
+- Terminates when done or timeout reached
+
+**Important:** The agent may modify test files during exploration. That is expected and corrected in Phase 3.
+
+---
+
+## Phase 3 — Grading prep (`eval_script`)
+
+After the agent exits, grading runs the generated `eval_script`.
+
+### 3.1 Capture agent state (informational)
+
+```bash
+git status
+git show
+git diff {base_commit}
+```
+
+These commands record what changed; they do not affect pass/fail.
+
+### 3.2 Reset test files
+
+Files modified in `test_patch` are checked out to `base_commit`:
+
+```python
+test_files = re.findall(r"--- a/(.*)", test_patch)
+git checkout {base_commit} {test_files...}
+```
+
+> Once the agent's run exits, we reset all of the test files to the original state, in case the agent modified the tests.
+> — [SWE-bench technical report](https://cognition.ai/blog/swe-bench-technical-report)
+
+### 3.3 Apply test patch
+
+```bash
+git apply -v - <<'EOF'
+{test_patch contents}
+EOF
+```
+
+This restores the **fail-to-pass** tests that define success for the instance.
+
+### 3.4 Apply agent patch (ordering)
+
+In the published blog flow, the agent patch is extracted from the filesystem **before** grading. The eval script applies:
+
+1. Agent-produced source changes (already in working tree after reset + test_patch apply order in script)
+
+Looking at `harness/scripts.py`, the eval script order is:
+
+1. `git checkout {base_commit} {test_files}` — reset tests  
+2. `git apply test_patch` — install fail-to-pass tests  
+3. Run test command  
+
+The **agent patch** must already be present in the working tree from the agent session, **except** for test file edits which were reset. Equivalently:
+
+> We apply the agent's patch to the repo, followed by the test patch.
+> — [SWE-bench technical report](https://cognition.ai/blog/swe-bench-technical-report)
+
+The harness assumes agent changes to non-test files persist through test reset.
+
+### 3.5 Run tests
+
+```bash
+{MAP_REPO_TO_TEST_FRAMEWORK[repo]} {directives from test_patch}
+```
+
+Success = all fail-to-pass tests pass.
+
+Directives are parsed by `harness/utils.py::get_test_directives()`.
+
+---
+
+## Phase 4 — Patch extraction (conceptual)
+
+When extracting an agent patch for external harnesses:
+
+1. Identify test files from `test_patch` (`--- a/` paths)
+2. Reset those files to `base_commit`
+3. `git diff` remaining changes → **agent patch**
+4. Apply agent patch, then `test_patch`, then run tests
+
+Getting this order wrong silently invalidates scores — a common source of eval fragility:
+
+> dependency issues in agent and grading environment setup, inconsistent handling of timeouts across harnesses, edge cases in patch collection and application
+> — [SWE-1.6 research update](https://cognition.ai/blog/swe-1-6-preview)
+
+---
+
+## Timeout policy
+
+Original Devin SWE-bench report: **45 minute** agent runtime ceiling.
+
+Other harnesses (Terminal-Bench, SWE-Bench Pro) may use different limits. Always record the configured ceiling alongside scores.
+
+---
+
+## Using this repo
+
+### Generate scripts for an instance
+
+```python
+from harness.dataset import get_dataset
+from harness.scripts import make_test_spec
+
+instances = get_dataset()
+spec = make_test_spec(instances[0])
+
+print(spec.instance_id)
+print(spec.setup_script)
+print(spec.eval_script)
+```
+
+### Offline harness invariant check
+
+```bash
+python scripts/verify_harness.py --mock
+python scripts/verify_harness.py --mock --json
+```
+
+---
+
+## References
+
+- [SWE-bench technical report](https://cognition.ai/blog/swe-bench-technical-report)
+- [Evaluating coding agents (cognition-golden)](https://cognition.ai/blog/evaluating-coding-agents)
+- [SWE-1.6 research update (eval reproducibility)](https://cognition.ai/blog/swe-1-6-preview)
+- Upstream SWE-bench: https://github.com/swe-bench/SWE-bench