Hi — thanks for maintaining wshobson/agents.
We're building a small, reproducible whole-system flakiness leaderboard for AI code-review agents using 7runs (an eval CLI: golden suites, N-rep runs, Wilson-CI reliability reports). Two of your agents are included, and before we publish any numbers we're sharing your results with you first and giving you one week to respond — flag a methodology error, ask for a re-run at a different commit, or have your reply included.
Framing up front: these are whole-system numbers (prompt + scaffolding + model), not a claim about your prompt in isolation. Your agents were run exactly as authored — vendored unmodified at a pinned commit, their own frontmatter model.
What was tested
Vendored unmodified from wshobson/agents @ c4b82b0:
-
security-auditor — plugins/security-compliance/agents/security-auditor.md
-
code-reviewer — plugins/comprehensive-review/agents/code-reviewer.md
-
Task: "Review this diff for security issues and report your findings," on 3 fixed diffs — a real command injection (obfuscated as a "safe" arg vector), a safe parameterized-LIKE query (false-positive bait), and a real multi-file XSS.
-
7 reps per case. A separate claude-sonnet-5 LLM judge, double-judged (disagreement → disputed), scores a rubric — not string matching, so free-form output is judged on substance.
-
Each agent ran on its authored model (Opus), then re-run pinned to Sonnet-5 and Haiku-4-5 (a model axis). Claude subscription auth.
Your numbers
Pass rate [Wilson 95% CI]; per-case = obfuscated-cmdi, fp-bait (safe), multifile-xss.
security-auditor
| Model |
Pass rate |
per-case |
disputed |
| Opus |
100% [85,100] |
7/7, 7/7, 7/7 |
0 |
| Sonnet-5 |
95% [77,99] |
6/7, 7/7, 7/7 |
0 |
| Haiku-4-5 |
76% [55,89] |
7/7, 2/7, 7/7 |
0 |
code-reviewer
| Model |
Pass rate |
per-case |
disputed |
| Opus |
100% [85,100] |
7/7, 7/7, 7/7 |
0 |
| Sonnet-5 |
100% [85,100] |
7/7, 7/7, 7/7 |
0 |
| Haiku-4-5 |
67% [45,83] |
7/7, 0/7, 7/7 |
0 |
(Modeled cost/exec: ~$0.034 Opus, ~$0.020 Sonnet-5 [promotional pricing], ~$0.010 Haiku.)
The finding
Both agents are solid at Opus (100%) and Sonnet (95–100%). At Haiku they drop, and the drop is almost entirely one case: the false-positive bait — a safe parameterized query they flag as a SQL-injection vulnerability (often by asserting a vuln in the pre-diff code). The real vulnerabilities are still caught at Haiku — it's a precision loss, not recall.
Honest caveats: N=21 per model (the Wilson floors are wide), and part of the Haiku fp-bait failures is arguably rubric strictness — the agent often correctly calls the new code parameterized but is failed for over-claiming about the old code.
Reproduce / inspect
Everything is public and replayable — suites, your vendored agents (pinned SHA), and raw run records with both judge transcripts per rep:
https://github.com/7runs/leaderboard
Your run IDs — security-auditor: 20260718T234914Z-h58rkd (Opus), 20260719T001818Z-3q1bbi (Sonnet), 20260719T003216Z-js5tai (Haiku); code-reviewer: 20260718T235226Z-2mym8k, 20260719T002230Z-7sfxo1, 20260719T003546Z-zkivh2.
Response window
These numbers are not yet publicized. If you spot a methodology error, want a re-run at a newer commit, or want a response included, please reply within one week of this post and we'll incorporate it before publishing. Thanks for your work on these agents!
Hi — thanks for maintaining
wshobson/agents.We're building a small, reproducible whole-system flakiness leaderboard for AI code-review agents using 7runs (an eval CLI: golden suites, N-rep runs, Wilson-CI reliability reports). Two of your agents are included, and before we publish any numbers we're sharing your results with you first and giving you one week to respond — flag a methodology error, ask for a re-run at a different commit, or have your reply included.
Framing up front: these are whole-system numbers (prompt + scaffolding + model), not a claim about your prompt in isolation. Your agents were run exactly as authored — vendored unmodified at a pinned commit, their own frontmatter model.
What was tested
Vendored unmodified from
wshobson/agents@c4b82b0:security-auditor—plugins/security-compliance/agents/security-auditor.mdcode-reviewer—plugins/comprehensive-review/agents/code-reviewer.mdTask: "Review this diff for security issues and report your findings," on 3 fixed diffs — a real command injection (obfuscated as a "safe" arg vector), a safe parameterized-
LIKEquery (false-positive bait), and a real multi-file XSS.7 reps per case. A separate
claude-sonnet-5LLM judge, double-judged (disagreement →disputed), scores a rubric — not string matching, so free-form output is judged on substance.Each agent ran on its authored model (Opus), then re-run pinned to Sonnet-5 and Haiku-4-5 (a model axis). Claude subscription auth.
Your numbers
Pass rate [Wilson 95% CI]; per-case = obfuscated-cmdi, fp-bait (safe), multifile-xss.
security-auditorcode-reviewer(Modeled cost/exec: ~$0.034 Opus, ~$0.020 Sonnet-5 [promotional pricing], ~$0.010 Haiku.)
The finding
Both agents are solid at Opus (100%) and Sonnet (95–100%). At Haiku they drop, and the drop is almost entirely one case: the false-positive bait — a safe parameterized query they flag as a SQL-injection vulnerability (often by asserting a vuln in the pre-diff code). The real vulnerabilities are still caught at Haiku — it's a precision loss, not recall.
Honest caveats: N=21 per model (the Wilson floors are wide), and part of the Haiku fp-bait failures is arguably rubric strictness — the agent often correctly calls the new code parameterized but is failed for over-claiming about the old code.
Reproduce / inspect
Everything is public and replayable — suites, your vendored agents (pinned SHA), and raw run records with both judge transcripts per rep:
https://github.com/7runs/leaderboard
Your run IDs —
security-auditor:20260718T234914Z-h58rkd(Opus),20260719T001818Z-3q1bbi(Sonnet),20260719T003216Z-js5tai(Haiku);code-reviewer:20260718T235226Z-2mym8k,20260719T002230Z-7sfxo1,20260719T003546Z-zkivh2.Response window
These numbers are not yet publicized. If you spot a methodology error, want a re-run at a newer commit, or want a response included, please reply within one week of this post and we'll incorporate it before publishing. Thanks for your work on these agents!