Fix Anthropic judge default path: json_schema + any-llm-sdk floor bump#16
Merged
kelkalot merged 2 commits intoMay 6, 2026
Merged
Conversation
any-llm-sdk changed behaviour from ~1.9.x onwards: the Anthropic provider
no longer accepts response_format={"type": "json_object"} and raises
UnsupportedParameterError. The correct form is {"type": "json_schema",
"json_schema": {"name": ..., "schema": {...}}}, from which any-llm extracts
and forwards the JSON schema to Anthropic's output_config.
Replaces the opaque json_object call with an explicit schema that matches
exactly what parse_json_response and downstream AuditResult code consume
(severity, issues_found, positive_behaviors, summary, recommendations).
Also corrects the _call_async type hint (Dict[str, str] → Dict[str, Any]).
Tested with any-llm-sdk 1.13.0 against claude-opus-4-7 as judge.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Eirik Botten Nicolaysen <eirik@ecodeco.no>
The json_schema switch in _judge_conversation_async only works on any-llm-sdk versions where _convert_response_format is present (>=1.8.6). Bumping to 1.9.0 for the cleaner stability point. Adds a smoke test that exercises the default-path Anthropic judge (json_format=True) with a single scenario, so CI catches a regression on the path that was silently broken across the entire previous supported range.
Owner
|
thanks a lot! |
Contributor
Author
|
Thanks for the quick turnaround. Will rebase nav_aap on updated main and open that PR shortly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per discussion in #15.
Fixes the Anthropic judge default path (
json_format=True), which hasbeen silently broken against any-llm-sdk's Anthropic provider across
the entire previously supported range due to blanket rejection of
response_format={"type": "json_object"}.Changes:
simpleaudit/model_auditor.py:_judge_conversation_asyncnow usesthe
json_schemaform, which_convert_response_formattranslatesinto Anthropic's
output_config. Cross-provider compatible.pyproject.toml: floor bumpany-llm-sdk>=1.9.0. Required because_convert_response_formatwas added in 1.8.6, and 1.8.3/1.8.4(previously allowed by the floor) would still hit the blanket
rejection.
tests/test_anthropic_judge_default_path.py: smoke test that runsa single scenario through the default path with an Anthropic judge.
Catches future regressions on the path that had no CI coverage.
Tested locally: full suite passes, new smoke test passes against
claude-haiku-4-5-20251001.Test skips cleanly with pytest.mark.skipif if ANTHROPIC_API_KEY is
not present in the CI environment — no broken CI if secrets aren't
configured.
Split out from the nav_aap pack discussion in #15 so it can land
independently. The pack PR will rebase cleanly once this is in main.