Skip to content

Expand harness docs, README, and offline verify script - #3

Open
Abhishek21g wants to merge 1 commit into
CognitionAI:mainfrom
Abhishek21g:docs/swebench-harness-repro
Open

Expand harness docs, README, and offline verify script#3
Abhishek21g wants to merge 1 commit into
CognitionAI:mainfrom
Abhishek21g:docs/swebench-harness-repro

Conversation

@Abhishek21g

@Abhishek21g Abhishek21g commented Jul 11, 2026

Copy link
Copy Markdown

Problem

Issue #1 — README is minimal and does not explain how to run the harness or reproduce the eval pipeline described in Cognition's SWE-bench technical report.

Fix

  • Expand README.md with repo layout, quickstart, and eval summary
  • Add docs/HARNESS.md documenting the agent eval pipeline:
    • git remote strip (leakage prevention)
    • test-file reset before grading
    • test_patch apply order
    • fail-to-pass test execution
  • Add scripts/verify_harness.py — offline mock validator for harness invariants (no SWE-bench download)
  • Add 7 pytest tests + GitHub Actions CI

Verification

python -m venv .venv && source .venv/bin/activate
pip install pytest
pytest tests/test_verify_harness.py -v
python scripts/verify_harness.py --mock --json
python scripts/verify_harness.py --fail-demo   # expect exit 1 (missing test reset)

Closes #1


Devin Review

Status Commit
⚪ Not started

Run Devin Review

💡 Connect your GitHub account to enable automatic code reviews.

Open in Devin Review (Staging)

- Add docs/HARNESS.md mirroring SWE-bench technical report eval pipeline
- Expand README with layout, quickstart, and verification commands
- Add scripts/verify_harness.py for mock harness invariant checks
- Add pytest suite (7 tests) and GitHub Actions CI

Closes CognitionAI#1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand the readme

1 participant