feat: report unguarded ASR policy delta - #16
Conversation
Closes SantanderAI#7. Compute an empty-policy ASR alongside each evaluated policy and report asr_unguarded, asr_with_policy, and policy_delta in CLI output and results.tsv. Keep legacy 5-column results.tsv rows readable and migrate the results header before appending new 7-column rows.
|
All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
|
Thanks for picking this up and turning it into actual code, @sergiotc19. Much appreciated. The three-column output (asr_unguarded, asr_with_policy, policy_delta) is exactly what the issue was after, and handling the legacy 5-column results.tsv migration cleanly is a nice touch I hadn't even flagged. Happy to help test once it's merged. I have baseline and policy runs across gpt-4.1-nano, gpt-4.1-mini and gpt-5.4-mini that I can re-run against this branch to sanity check the delta column matches what I measured by hand. Just say the word if that's useful for review. |
|
Thanks @wakeupneo1978, that would be great. Appreciate it. |
opensource-SantanderAI
left a comment
There was a problem hiding this comment.
Approved: adds unguarded ASR baseline and policy_delta. Verified locally (ruff/black/mypy, 73 tests, 90.6% coverage); respects the frozen evaluation surface. No security concerns.
Description
Compute an empty-policy ASR alongside each evaluated policy and report
asr_unguarded,asr_with_policy, andpolicy_deltain CLI output andresults.tsv.Keep legacy 5-column
results.tsvrows readable and migrate the results header before appending new 7-column rows.Related issue
Closes #7.
Type of change
Research contract
eval_suite.jsonl,judge_prompt.md)Checklist
ruff check .andblack --check .passmypy autoguardrailspassespython -m pytestpasses