[Question]: Inconsistent thinking streaming pattern between Ollama and Anthropic integrations

[RakeshReddyKondeti]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hi

While testing stream_chat, I noticed thinking is exposed very differently across providers.

# Ollama
resp_gen = ollama_llm.stream_chat(messages)
for r in resp_gen:
    thinking_delta = r.additional_kwargs["thinking_delta"]  # incremental
    print(thinking_delta)

# Anthropic
from anthropic.types import ThinkingDelta

resp_gen = anthropic_llm.stream_chat(messages)
for r in resp_gen:
    delta = r.raw.get("delta")
    if delta and isinstance(delta, ThinkingDelta):
        print(delta.thinking)  # incremental
    # r.message.additional_kwargs["thinking"] -> full accumulated thinking

Key differences:

• Ollama: incremental thinking is in r.additional_kwargs["thinking_delta"].
• Anthropic: one must inspect r.raw["delta"] for a ThinkingDelta;
  r.message.additional_kwargs["thinking"] holds accumulated reasoning text.

Could the API be made consistent — or could you clarify the intended pattern for streaming both text and thinking?
That would simplify unified handling across LLMs.

---

llamaindex version - 0.14.4

[dosubot]

You're right—there's currently an inconsistency in how incremental and accumulated "thinking" is exposed in streaming responses between Ollama and Anthropic integrations. Ollama provides incremental thinking via r.additional_kwargs["thinking_delta"], while Anthropic requires inspecting r.raw["delta"] for a ThinkingDelta, with the accumulated reasoning in r.message.additional_kwargs["thinking"].

This inconsistency is known and has been acknowledged by maintainers, who agree that a unified data structure for streaming reasoning and tool calls would be ideal, possibly by streaming ContentBlocks instead of plain text deltas. However, this hasn't been implemented yet, so each integration exposes "thinking" in its own way, and users need to handle these differences in their code for now. If you need unified handling, you'll have to write a compatibility layer that normalizes these fields across providers until a core abstraction is introduced in LlamaIndex itself. For more details, see the discussion here.

Explicit streaming of reasoning content also depends on the model and prompt design—LlamaIndex doesn't currently structure reasoning steps separately from normal output unless the integration provides them explicitly. For advanced use cases, extending the agent or LLM interface to emit structured reasoning events would be required, but that's not built-in yet (reference).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[logan-markewich]

Working on making it consistent, I know.

Plan is to add streaming content blocks and stop abusing kwargs