Skip to content

fix(llm): surface the response body when a completion has no choices#36

Open
wmeddie wants to merge 2 commits into
mainfrom
fix/no-choices-error
Open

fix(llm): surface the response body when a completion has no choices#36
wmeddie wants to merge 2 commits into
mainfrom
fix/no-choices-error

Conversation

@wmeddie

@wmeddie wmeddie commented Jun 12, 2026

Copy link
Copy Markdown
Member

Two resilience fixes in the OpenAI adapter's response handling, both observed with OpenAI-compatible gateways in production use.

1. A completion without choices now raises a useful error

Some OpenAI-compatible gateways return upstream error bodies (rate limits, provider failures) with HTTP 200. The SDK constructs those leniently, so choices ends up None/absent and response.choices[0] raises a bare 'NoneType' object is not subscriptable — the actual upstream error (sitting right there in the body) never reaches the caller or the logs.

OpenAILLM.generate now checks before subscripting and raises a RuntimeError carrying the response body (truncated to 500 chars):

RuntimeError: LLM API returned a completion without choices: {"code": "rate_limit_daily", "message": "Error code: 429 - ..."}

The streaming path already guards (if chunk.choices and ...), so only generate needed it.

2. Tool-call argument parsing tolerates recoverable malformations

The strict json.loads in the tool-call path dropped two recoverable cases (skipping the call with only a log line):

  • raw control characters (literal newlines) inside JSON string values — models emit these routinely, and strict=False parses them fine;
  • empty arguments from parameterless tool calls — json.loads("") raises, but the right reading is {}, not dropping the call.

Truly irrecoverable arguments (garbage, non-dict JSON like a bare list) keep the existing skip-with-warning behavior from #33; a bare non-dict additionally can't crash LLMFunctionCall validation anymore.

Tests

test_openai_no_choices.py (incl. the SDK's real lenient construct() path) and test_openai_tool_arguments.py.

Note: commit 70f52e6 on the already-merged #35 branch is a stray push of the first fix made before noticing the merge — ignore that branch head; this PR supersedes it, rebased onto current main (composes with the finish_reason handling from #33).

wmeddie added 2 commits June 13, 2026 03:02
Some OpenAI-compatible gateways return upstream error bodies (rate
limits, provider failures) with HTTP 200. The SDK constructs those
leniently, leaving choices None/absent, and response.choices[0] then
raised a bare "'NoneType' object is not subscriptable" that hid the
actual upstream error. Raise a RuntimeError carrying the body instead,
so callers and logs see the real reason.
json.loads in the tool-call path was strict, so two recoverable cases
dropped tool calls with only a log line to show for it:

- raw control characters (literal newlines) inside JSON string values —
  models emit these routinely and strict=False parses them fine
- empty arguments from parameterless tool calls, which must become {}
  instead of being skipped

Truly irrecoverable arguments (garbage, non-dict JSON) keep the existing
skip-with-warning behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants