Declarative judge response_schema + helsedir/binary_abstention judges

[kelkalot]

Summary

• Framework refactor: judges may now declare an optional response_schema in their config; ModelAuditor reads it and threads it through to the judge LLM call. Lets judges emit non-default output shapes (binary classifiers, score+category, etc.) without the json_format=False workaround. Backwards-compatible — judges without response_schema continue using the default severity schema.
• Three new judges:
  • helsedir_sexhealth_no and helsedir_sexhealth_no_rag — Norwegian sexual-health evaluation, six criteria from a Helsedirektoratet domain-expert review. The sixth (BEVARING AV USIKKERHET OG NYANSER) is reviewer-added beyond SimpleAudit's standard five. Two variants share criteria; the RAG one references «dokumentene» verbatim, the generic one softens to «etablerte faglige kilder».
  • binary_abstention — language-agnostic substance-vs-words classifier on whether the model delivered the substantive content requested.
• Built-in abstention judge fixed: gains its own response_schema declaration. Was previously broken under json_format=True against Anthropic — the hardcoded severity schema rejected its score/abstained/category fields.
• 13 new unit tests for the response_schema mechanism (default schema preservation, registry behaviour, threading, backwards compat, malformed-response fallback).
• Docs: README lists the new judges and the mechanism; judge-config guidelines doc updated; .env added to .gitignore.

Experimental validation

Two experiments in examples/helsedir_sexhealth_no/ against 47 stored qwen3.5:9b conversations on the hei_refusal pack, judged by claude-opus-4-7.

Experiment 1 — helsedir judges vs safety baseline.

Judge	Score
safety (baseline)	75.0
helsedir_sexhealth_no	50.5
helsedir_sexhealth_no_rag	39.9

The new criteria (missing-referral, age-appropriate framing, nuance preservation) catch signal the safety judge passes. The RAG variant is uniformly harsher on non-RAG targets — appropriate when applied to actual RAG bots, less discriminative otherwise.

Experiment 2 — abstention judges.

Judge	F1
binary_abstention	0.52
abstention (AbstentionBench reference)	0.45

98% inter-judge agreement on abstained. Required one prompt revision: initial binary_abstention F1 was 0.11 because the rule checked refusal-words instead of substantive-content-delivery; the corrected rule generalises correctly.

Test plan

• All existing tests pass — 136 passed, 3 skipped (require API keys), 0 failures
• 13 new unit tests for response_schema mechanism pass
• All 8 registered judges load via list_judge_configs()
• Backwards-compatibility: judges without response_schema use the default severity schema (verified in tests)
• Two end-to-end experiments executed against Anthropic; results and analyses checked in
• No API keys or personal identifiers in committed files

[avalyset]

Nice generalization of the json_schema mechanism — having judges declare their own response_schema is cleaner than the per-provider workaround in PR #16. One question: does the new built-in abstention judge's schema work cleanly against both Anthropic and OpenAI providers, or does it still hit the any-llm provider-specific path that motivated PR #16?