Skip to content

feat: report unguarded ASR policy delta - #16

Merged
opensource-SantanderAI merged 2 commits into
SantanderAI:mainfrom
sergiotc19:feature/report-unguarded-asr-delta
Jul 16, 2026
Merged

feat: report unguarded ASR policy delta#16
opensource-SantanderAI merged 2 commits into
SantanderAI:mainfrom
sergiotc19:feature/report-unguarded-asr-delta

Conversation

@sergiotc19

@sergiotc19 sergiotc19 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Description

Compute an empty-policy ASR alongside each evaluated policy and report asr_unguarded, asr_with_policy, and policy_delta in CLI output and results.tsv.

Keep legacy 5-column results.tsv rows readable and migrate the results header before appending new 7-column rows.

Related issue

Closes #7.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Documentation only

Research contract

  • This PR does not change the frozen evaluation surface (eval_suite.jsonl, judge_prompt.md)
  • OR it does, and I have flagged it as a new experiment lineage in the description

Checklist

  • I have signed the CLA (the CLA Assistant bot will prompt external contributors)
  • My commit messages follow Conventional Commits
  • ruff check . and black --check . pass
  • mypy autoguardrails passes
  • python -m pytest passes
  • I have added/updated tests where relevant
  • I have updated documentation where relevant
  • No secrets, API keys, internal URLs, or proprietary content are included

Closes SantanderAI#7.

Compute an empty-policy ASR alongside each evaluated policy and report asr_unguarded, asr_with_policy, and policy_delta in CLI output and results.tsv.

Keep legacy 5-column results.tsv rows readable and migrate the results header before appending new 7-column rows.
@sergiotc19
sergiotc19 requested review from a team as code owners July 9, 2026 17:57
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@sergiotc19

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Jul 9, 2026
@wakeupneo1978

Copy link
Copy Markdown

Thanks for picking this up and turning it into actual code, @sergiotc19. Much appreciated. The three-column output (asr_unguarded, asr_with_policy, policy_delta) is exactly what the issue was after, and handling the legacy 5-column results.tsv migration cleanly is a nice touch I hadn't even flagged.

Happy to help test once it's merged. I have baseline and policy runs across gpt-4.1-nano, gpt-4.1-mini and gpt-5.4-mini that I can re-run against this branch to sanity check the delta column matches what I measured by hand. Just say the word if that's useful for review.

@sergiotc19

Copy link
Copy Markdown
Contributor Author

Thanks @wakeupneo1978, that would be great. Appreciate it.

@opensource-SantanderAI opensource-SantanderAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: adds unguarded ASR baseline and policy_delta. Verified locally (ruff/black/mypy, 73 tests, 90.6% coverage); respects the frozen evaluation surface. No security concerns.

@opensource-SantanderAI
opensource-SantanderAI merged commit 1ca0c9b into SantanderAI:main Jul 16, 2026
11 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report the delta over the unguarded baseline, not only the absolute ASR

3 participants