feat: add LiteLLM as chat model provider for 100+ LLM backends#456
feat: add LiteLLM as chat model provider for 100+ LLM backends#456RheagalFire wants to merge 4 commits into
Conversation
|
cc @yiboyasss |
There was a problem hiding this comment.
Code Review
This pull request introduces a new LiteLLM provider to the xagent core model, enabling access to numerous LLM providers through a unified interface. The implementation includes the LiteLLM class, updates to the model adapter factory, and a suite of unit tests. Feedback highlights several critical issues in the new implementation: the chat method fails to include the mandatory raw field in tool call responses, the stream_chat method uses incorrect chunk types and lacks support for tool calls, and there is a risk of an IndexError when accessing response choices without prior validation.
| }, | ||
| } | ||
| ) | ||
| return {"type": "tool_call", "tool_calls": tool_calls} |
There was a problem hiding this comment.
The tool call response dictionary is missing the raw field. According to the BaseLLM.chat interface documentation (see base.py line 163), the returned dictionary for tool calls must include the full response JSON in a raw field to allow for debugging and advanced processing.
return {
"type": "tool_call",
"tool_calls": tool_calls,
"raw": response.model_dump() if hasattr(response, "model_dump") else dict(response),
}| async for chunk in response: | ||
| delta = chunk.choices[0].delta if chunk.choices else None | ||
| if delta is None: | ||
| continue | ||
| content = getattr(delta, "content", None) | ||
| if content: | ||
| yield StreamChunk(type=ChunkType.TEXT, content=content) |
There was a problem hiding this comment.
The stream_chat implementation has several issues that break consistency and functionality:
- It uses
ChunkType.TEXTinstead of the standardChunkType.TOKENused in the rest of the codebase (e.g., inBaseLLM.stream_chat). - It omits the
deltafield inStreamChunk, which is expected by consumers. - It completely ignores tool calls in the stream, which will cause them to be dropped silently during streaming.
Updating the loop to handle these cases ensures compatibility with the expected streaming behavior and supports tool calling in streaming mode.
async for chunk in response:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
# Handle text content
content = getattr(delta, "content", None)
if content:
yield StreamChunk(type=ChunkType.TOKEN, content=content, delta=content)
# Handle tool calls
if hasattr(delta, "tool_calls") and delta.tool_calls:
tool_calls = []
for tc in delta.tool_calls:
tool_calls.append({
"index": getattr(tc, "index", 0),
"id": getattr(tc, "id", None),
"type": "function",
"function": {
"name": getattr(tc.function, "name", None),
"arguments": getattr(tc.function, "arguments", ""),
},
})
yield StreamChunk(
type=ChunkType.TOOL_CALL,
tool_calls=tool_calls,
raw=chunk.model_dump() if hasattr(chunk, "model_dump") else dict(chunk)
)| ) as e: | ||
| raise LLMRetryableError(str(e)) from e | ||
|
|
||
| choice = response.choices[0] |
There was a problem hiding this comment.
The code accesses response.choices[0] without verifying that choices is not empty. While LiteLLM typically returns at least one choice on success, certain edge cases (like content filtering or provider-specific errors) could result in an empty list, leading to an IndexError.
| choice = response.choices[0] | |
| if not response.choices: | |
| raise LLMRetryableError("LiteLLM returned an empty response (no choices).") | |
| choice = response.choices[0] |
|
Could you fix the lint, and resolve the gemini comments? |
Summary
Adds
LiteLLMas a new chat model provider alongside OpenAI, Claude, DeepSeek, Gemini, and Zhipu, enabling access to 100+ LLM backends through LiteLLM's unified interface.Changes
src/xagent/core/model/chat/basic/litellm.pyLiteLLM(BaseLLM)with asyncchat()andstream_chat(). Useslitellm.acompletion()withdrop_params=True. Supports tool calling, response format, token usage tracking. Maps litellm exceptions toLLMRetryableError/LLMTimeoutError.src/xagent/core/model/chat/basic/__init__.pyLiteLLM.src/xagent/core/model/chat/basic/adapter.py"litellm"provider branch increate_base_llm()factory.tests/core/model/chat/basic/test_litellm.pyTests
Unit tests -- 20 passed:
Live e2e via Anthropic:
Lint:
Example usage
Any LiteLLM model string works. See LiteLLM docs for full list.