fix(qa): tolerate non-dict JSON from QA LLM instead of crashing#408
Open
Mubashirrrr wants to merge 1 commit into
Open
fix(qa): tolerate non-dict JSON from QA LLM instead of crashing#408Mubashirrrr wants to merge 1 commit into
Mubashirrrr wants to merge 1 commit into
Conversation
parse_llm_json is explicitly designed to return a list when the model emits a
top-level JSON array (it has a dedicated test for that). The QA analyzers then
call parsed.get("tags", ...) directly on the result. When parsed is a list,
that raises AttributeError, which is NOT caught by the surrounding
except (json.JSONDecodeError, ValueError) — so a single stray array response
from the QA model crashed the entire QA analysis run instead of degrading to
empty results.
The live variable-extraction path already guards this exact case with an
isinstance(..., dict) check; mirror it in both QA analysis call sites
(_run_qa_analysis per-node and _run_whole_call_qa_analysis fallback) so a
non-dict parse result coerces to {} and the run produces empty defaults.
Adds a regression test that drives the whole-call analyzer with an array
response and asserts empty results rather than a crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
parse_llm_json(api/services/gen_ai/json_parser.py) is explicitly designed to return a list when the model emits a top-level JSON array — it has a dedicated unit test for"[1, 2, 3]", and_extract_json_arrayis one of its parse strategies.The QA analyzers in
api/services/workflow/qa/analysis.py(both_run_qa_analysisper-node and the_run_whole_call_qa_analysisfallback) call.get()directly on the result:When
parsedis a list,parsed.get(...)raisesAttributeError, which is not caught by the surroundingexcept (json.JSONDecodeError, ValueError). So a single stray array response from the QA model crashes the entire QA analysis run instead of degrading to empty results.Notably, the live variable-extraction path already guards this exact case (
api/services/workflow/pipecat_engine.py:if not isinstance(extracted_data, dict): ... skip), so this is an inconsistency — the same defensive check is present in one consumer of LLM-parsed JSON but missing in the QA consumers.Reproduce
Fix
Mirror the existing engine guard in both QA call sites: coerce a non-dict parse result to
{}so the.get()lookups produce empty defaults instead of crashing.Regression test
api/tests/test_qa_analysis_non_dict_response.pydrives_run_whole_call_qa_analysiswith an array LLM response (mocking the LLM seams) and asserts empty results (tags == [],summary == "",score is None) rather than an exception. The core parse-block behavior was verified directly: it raisesAttributeErroron the old code and degrades to empty defaults on the new code, while the normal dict path still maps correctly.