Reliability (flakiness) benchmark results for your security-auditor and code-reviewer — one-week response window before publication

[impayam]

Hi — thanks for maintaining wshobson/agents.

We're building a small, reproducible whole-system flakiness leaderboard for AI code-review agents using 7runs (an eval CLI: golden suites, N-rep runs, Wilson-CI reliability reports). Two of your agents are included, and before we publish any numbers we're sharing your results with you first and giving you one week to respond — flag a methodology error, ask for a re-run at a different commit, or have your reply included.

Framing up front: these are whole-system numbers (prompt + scaffolding + model), not a claim about your prompt in isolation. Your agents were run exactly as authored — vendored unmodified at a pinned commit, their own frontmatter model.

What was tested

Vendored unmodified from wshobson/agents @ c4b82b0:

• security-auditor — plugins/security-compliance/agents/security-auditor.md

• code-reviewer — plugins/comprehensive-review/agents/code-reviewer.md

• Task: "Review this diff for security issues and report your findings," on 3 fixed diffs — a real command injection (obfuscated as a "safe" arg vector), a safe parameterized-LIKE query (false-positive bait), and a real multi-file XSS.

• 7 reps per case. A separate claude-sonnet-5 LLM judge, double-judged (disagreement → disputed), scores a rubric — not string matching, so free-form output is judged on substance.

• Each agent ran on its authored model (Opus), then re-run pinned to Sonnet-5 and Haiku-4-5 (a model axis). Claude subscription auth.

Your numbers

Pass rate [Wilson 95% CI]; per-case = obfuscated-cmdi, fp-bait (safe), multifile-xss.

security-auditor

Model	Pass rate	per-case	disputed
Opus	100% [85,100]	7/7, 7/7, 7/7	0
Sonnet-5	95% [77,99]	6/7, 7/7, 7/7	0
Haiku-4-5	76% [55,89]	7/7, 2/7, 7/7	0

code-reviewer

Model	Pass rate	per-case	disputed
Opus	100% [85,100]	7/7, 7/7, 7/7	0
Sonnet-5	100% [85,100]	7/7, 7/7, 7/7	0
Haiku-4-5	67% [45,83]	7/7, 0/7, 7/7	0

(Modeled cost/exec: ~$0.034 Opus, ~$0.020 Sonnet-5 [promotional pricing], ~$0.010 Haiku.)

The finding

Both agents are solid at Opus (100%) and Sonnet (95–100%). At Haiku they drop, and the drop is almost entirely one case: the false-positive bait — a safe parameterized query they flag as a SQL-injection vulnerability (often by asserting a vuln in the pre-diff code). The real vulnerabilities are still caught at Haiku — it's a precision loss, not recall.

Honest caveats: N=21 per model (the Wilson floors are wide), and part of the Haiku fp-bait failures is arguably rubric strictness — the agent often correctly calls the new code parameterized but is failed for over-claiming about the old code.

Reproduce / inspect

Everything is public and replayable — suites, your vendored agents (pinned SHA), and raw run records with both judge transcripts per rep:
https://github.com/7runs/leaderboard

Your run IDs — security-auditor: 20260718T234914Z-h58rkd (Opus), 20260719T001818Z-3q1bbi (Sonnet), 20260719T003216Z-js5tai (Haiku); code-reviewer: 20260718T235226Z-2mym8k, 20260719T002230Z-7sfxo1, 20260719T003546Z-zkivh2.

Response window

These numbers are not yet publicized. If you spot a methodology error, want a re-run at a newer commit, or want a response included, please reply within one week of this post and we'll incorporate it before publishing. Thanks for your work on these agents!

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• Reliability (flakiness) benchmark results for your security-auditor and code-reviewer — one-week response window before publication #631

🔗 Related PRs

#535 - fix: agent teams coordination guardrails [merged]
#544 - Add Claude Code GitHub Workflow [merged]
#554 - Fix duplicate agent name collisions [merged]
#612 - docs: refresh model references for Claude Sonnet 5 and Fable 5 [merged]
#624 - feat: add llm-finetuning and dgx-spark-ops plugins (eval-gated fine-tuning lifecycle) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[impayam]

Update: we're proceeding with publication ahead of the one-week window rather than waiting the full week. The results stay open to correction — if you spot a methodology error, want a re-run at a newer commit, or want a response included, reply any time and we'll incorporate it. Thanks again for your work on these agents.