feat: per-attack-family ASR diagnostics in evaluation output

[perezpefaur]

Description

The research loop optimizes a single number — aggregate ASR — but the documented
workflow is to lower it one attack family at a time (see the README "Typical
Iteration Pattern"). Today the harness gives no family-level signal, so after each
run you cannot tell which family is still leaking and therefore what to edit in
policy.md next. You're effectively flying blind between iterations.

This PR adds a per-attack-family ASR breakdown to every baseline, candidate,
and evaluate run, printed right after the aggregate metrics and ranked
leakiest-first:

asr=0.4000
benign_pass=1.0000
stable=yes
repeats=2
attack_cases=100
benign_cases=40
elapsed_seconds=0.01
family:cyber asr=1.0000 hits=42 n=42
family:fraud asr=1.0000 hits=38 n=38
family:violent asr=1.0000 hits=40 n=40
family:jailbreak asr=0.0000 hits=0 n=40
family:obfuscation asr=0.0000 hits=0 n=40

This turns "lower the ASR" into a concrete, targeted worklist: the top line is
usually the best next target for a policy edit.

How it works

• Each attack prompt is grouped using the existing harness attack taxonomy
  (ATTACK_FAMILIES in model_adapter.py) via classify_prompt_family, so there is
  a single source of truth and no new heuristics. It is a diagnostic lens, not ground
  truth; prompts that match no family are bucketed as unclassified.
• EvaluationSummary / RepeatedEvaluationSummary gain an optional, backward-compatible
  family_stats field (defaults to ()), and counts are pooled across repeated
  passes (n = cases × repeats).
• This is exactly the "richer reporting" extension point called out in
  autoguardrails/README.md.

Related issue

N/A

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that changes existing behavior)
• Documentation only

Research contract

• This PR does not change the frozen evaluation surface (eval_suite.jsonl, judge_prompt.md)
• OR it does, and I have flagged it as a new experiment lineage in the description

Families are derived from the prompts at runtime using the taxonomy already shipped in
the fixed harness, so no scoring inputs change and existing results.tsv numbers remain
comparable. The results.tsv schema is intentionally left untouched.

Checklist

• I have signed the CLA (the CLA Assistant bot will prompt external contributors)
• My commit messages follow Conventional Commits
• ruff check . and black --check . pass
• mypy autoguardrails passes
• python -m pytest passes (32 tests, coverage 91%)
• I have added/updated tests where relevant
• I have updated documentation where relevant (README, harness README, CHANGELOG)
• No secrets, API keys, internal URLs, or proprietary content are included

Test plan

• New tests assert per-family stats track policy coverage (covered families drop to
  asr=0.0, uncovered stay at 1.0), counts pool across repeats, ordering is
  deterministic, the FamilyStat.asr zero-division branch is covered, and the CLI
  ranks leaky families above covered ones.

Made with Cursor

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

[opensource-SantanderAI]

recheck

[opensource-SantanderAI]

Hi @perezpefaur — thanks for the contribution! The only thing pending before we can review/merge is the CLA. Please post the following comment on this PR to sign it:

I have read the CLA Document and I hereby sign the CLA

Once signed, the CLA Assistant check will turn green automatically. 🙌

[opensource-SantanderAI]

Hi @perezpefaur — thanks again for this contribution. The change itself looks good and all CI checks are green (lint, tests on Python 3.10/3.11/3.12, CodeQL, no-runtime-deps). The only thing blocking the merge is the CLA: it hasn't been signed yet, so the CLA Assistant check is failing.

To unblock, please post a new comment on this PR with exactly this text:

I have read the CLA Document and I hereby sign the CLA

Once the CLA Assistant re-runs and turns green, we'll merge right away. Thanks!

[opensource-SantanderAI]

Friendly ping @perezpefaur 👋 — this PR is good to go and fully green; we're just waiting on your CLA signature to merge. Please drop a comment with:

I have read the CLA Document and I hereby sign the CLA

Thanks!

[opensource-SantanderAI]

Hi @perezpefaur, and thank you for this contribution! 🙏

We reviewed the change in depth and the substance is approved — this is a genuinely useful addition. The per-attack-family ASR breakdown turns "lower the ASR" into a concrete, one-family-at-a-time worklist, it reuses the harness's existing attack taxonomy instead of inventing new heuristics, it stays fully backward-compatible (family_stats defaults to ()), and it correctly leaves the frozen evaluation surface (eval_suite.jsonl, judge_prompt.md) and the results.tsv schema untouched. We reproduced your checks locally and everything is green: ruff, black --check, mypy, and 32 tests passing at 91% coverage. Nice work, and thanks for the tests and docs.

Two things are needed before we can merge (neither is about code quality):

1. Rebase on main. The branch currently conflicts with main (it's ~11 commits behind), with conflicts in autoguardrails/__main__.py and tests/test_harness.py. Could you rebase onto the latest main and resolve those, then force-push? That will get the merge state back to green.

2. Sign the CLA. The CLA Assistant check is still red. Signing takes ~30 seconds, no forms — just post a new comment on this PR with exactly:

   I have read the CLA Document and I hereby sign the CLA
   

   CLA: https://github.com/SantanderAI/cla/blob/main/CLA.md

Once the rebase is done and the CLA check turns green, we'll re-run CI and merge. Thanks again for the clean, well-tested contribution!