IndexError: list index out of range in response_adapter.py:164 when streaming API sends final chunk without choices

[jingbarry]

Bug Description

response_adapter.py line 164 crashes with IndexError: list index out of range when the streaming API sends a final SSE chunk that does not contain a choices field (e.g., a usage-only statistics chunk).

# Line 164 — no guard for missing or empty choices
if "message" not in done_message["choices"][0]:

Steps to Reproduce

1. Configure Agently (v4.1.3.5) with an OpenAI-compatible API that sends a usage-only chunk as the final SSE event before [DONE], such as MiMo:

agent.set_settings(
    "OpenAICompatible",
    {
        "base_url": os.environ.get("MIMO_BASE_URL"),
        "model": os.environ.get("MIMO_DEFAULT_MODEL"),
        "auth": os.environ.get("MIMO_API_KEY"),
    },
)

2. Make a streaming request:

result = agent.prompt("Hello").get_delta()

3. The stream completes successfully (content is accumulated in content_buffer), but the final [DONE] handler crashes.

Error Traceback

IndexError: list index out of range

  File ".../agently/core/model/ModelResponse.py", line 321, in _get_response_generator
    async for event, data in broadcast_generator:
  File ".../agently/builtins/plugins/ModelRequester/OpenAICompatible/modules/response_adapter.py", line 164, in broadcast_response
    if "message" not in done_message["choices"][0]:

Expected Behavior

The adapter should guard against missing or empty choices before accessing done_message["choices"][0]. For example:

if "choices" in done_message and len(done_message["choices"]) > 0:
    if "message" not in done_message["choices"][0]:
        done_message["choices"][0].update({"message": {}})
    done_message["choices"][0]["message"].update(...)
else:
    done_message["choices"] = [{"message": {"role": "assistant", "content": done_content or content_buffer}}]

Actual Behavior

The code assumes every SSE chunk has a non-empty choices array. Some non-standard OpenAI-compatible APIs (e.g., MiMo) send a final chunk like:

{"id":"...","usage":{"prompt_tokens":...,"completion_tokens":...}}

This overwrites message_record (line 66), and when [DONE] is processed, done_message["choices"] is a KeyError or done_message["choices"] is [], causing IndexError.

Workaround

Set "stream": False in plugin settings to bypass streaming.

Environment

• Agently version: 4.1.3.5
• Python: 3.10
• Model provider: MiMo (OpenAI-compatible API)
• OS: macOS

[Maplemx]

Confirmed and fixed. Thanks for the detailed report and the precise root-cause analysis — it was exactly right.

Root cause
OpenAICompatible.broadcast_response overwrites message_record with every streamed chunk, including a usage-only final chunk that carries an empty choices array (e.g. {"choices":[],"usage":{...}}) right before [DONE]. When [DONE] was then processed, the done handler unconditionally accessed done_message["choices"][0], which raises IndexError (or KeyError when choices is absent).

Fix
The [DONE] handler now guards choices: when it is empty or missing, it synthesizes choices[0].message from the accumulated streamed content instead of indexing into an empty list, so the response text is preserved rather than lost.

Validation
Reproduced end-to-end against a gateway whose Chat Completions stream sends a usage-only final chunk before [DONE] (same shape as MiMo). With the fix, streaming returns the full content with no crash. Added regression test test_broadcast_response_handles_usage_only_final_chunk_without_choices covering content deltas followed by a {"choices":[],"usage":{...}} final chunk.

Related fix bundled in the same change
While investigating the same model-request stream-end path, a second defect was fixed: the core AttemptRunner misclassified GeneratorExit (raised when a response adapter closes the stream early after the terminal event) as a model error and emitted a spurious, empty-message model.requester.error after an otherwise successful response. GeneratorExit is now treated as a control-flow signal and re-raised, alongside CancelledError / KeyboardInterrupt / SystemExit.

Release line: fixed on the dev branch (commit 05535bcd), targeting the 4.1.3.6 development line.

---

已确认并修复。 感谢非常详细的报告与精准的根因分析——和实际原因完全一致。

根因
OpenAICompatible.broadcast_response 会用每一个流式 chunk 覆盖 message_record，其中包括 [DONE] 之前那个只带用量、choices 为空数组的收尾 chunk（例如 {"choices":[],"usage":{...}}）。随后处理 [DONE] 时，done 处理逻辑无条件访问 done_message["choices"][0]，于是抛出 IndexError（当缺少 choices 字段时则是 KeyError）。

修复
[DONE] 处理逻辑现在会对 choices 做保护：当其为空或缺失时，用已累积的流式内容合成 choices[0].message，而不是对空列表取下标，从而保留响应文本而不会丢失。

验证
针对会在 [DONE] 前发送只带用量收尾 chunk 的网关（与 MiMo 形态一致）做了端到端复现。修复后流式可正常返回完整内容、不再崩溃。新增回归测试 test_broadcast_response_handles_usage_only_final_chunk_without_choices，覆盖「内容增量 + {"choices":[],"usage":{...}} 收尾 chunk」场景。

同一改动中一并修复的相关问题
在排查同一条「模型请求流结束」路径时，修复了第二个缺陷：核心 AttemptRunner 把 GeneratorExit（当响应适配器在收到终止事件后提前关闭流时抛出）误判为模型错误，从而在一次本应成功的响应之后发出一条空消息的 model.requester.error。现在 GeneratorExit 与 CancelledError / KeyboardInterrupt / SystemExit 一样被视为控制流信号并直接重新抛出。

发布线： 已在 dev 分支修复（提交 05535bcd），目标为 4.1.3.6 开发线。

[Maplemx]

Thanks for the report. This is fixed in Agently 4.1.3.6, which has now been released.

What changed:

• The OpenAICompatible streaming adapter now treats final chunks without choices, including usage-only chunks, as a valid stream end instead of indexing into choices[0].
• Regression coverage was added for final chunks with empty or missing choices.

Validation:

• pyright agently tests examples
• python -m pytest
• Targeted OpenAICompatible stream-end regression coverage
• The Publish on version change workflow completed successfully for the 4.1.3.6 release.

I have marked this as solved. Leaving it open for reporter or maintainer confirmation unless you prefer it closed now.

[richardchen874-sys]

I like the way this issue frames the usage-only final streaming chunk that can arrive without choices. In real workloads, the hard part is usually not sending one request; it is keeping cost, latency, and response shape predictable once users switch providers.

I would treat usage-only chunks as a normal terminal event and test providers that send empty choices, no choices, or final metadata separately. Streaming parsers need to be defensive for OpenAI-compatible APIs.

I am testing an OpenAI-compatible multi-model API layer around official Chinese models, so this kind of routing, usage, and compatibility boundary is very relevant. Should Agently normalize these stream events internally, or expose provider quirks to advanced users?

[Maplemx]

I like the way this issue frames the usage-only final streaming chunk that can arrive without choices. In real workloads, the hard part is usually not sending one request; it is keeping cost, latency, and response shape predictable once users switch providers.

I would treat usage-only chunks as a normal terminal event and test providers that send empty choices, no choices, or final metadata separately. Streaming parsers need to be defensive for OpenAI-compatible APIs.

I am testing an OpenAI-compatible multi-model API layer around official Chinese models, so this kind of routing, usage, and compatibility boundary is very relevant. Should Agently normalize these stream events internally, or expose provider quirks to advanced users?

Advanced users(developers) can modify how meta data is collected and broadcasted in: https://github.com/AgentEra/Agently/blob/main/agently/builtins/plugins/ModelRequester/OpenAICompatible/modules/response_adapter.py

It can work with settings to maintain same handling logic but different location of meta data in response data from different providers.

[richardchen874-sys]

@Maplemx Great to see the provider abstraction land through #166. If the next integration phase needs real OpenAI-compatible endpoints beyond the usual providers, I am helping build a private-beta API layer for Chinese models with normalized usage and routing metadata. Happy to provide a small compatibility endpoint when that becomes useful.