System Info / 系統信息
Python 3.10
vllm 0.18.1
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
Version info / 版本信息
v2.8.0
The command used to start Xinference / 用以启动 xinference 的命令
...
Reproduction / 复现过程
Yesterday a user reported that he can't use LLM with zread_cli. After investigation, I found that xinference does not keep history tool_call messages at all!
After fixing this issue, I found another bug that in chat_template of Qwen3.5, function arguments should be a dict object instead of string (which is OpenAI standard). I don't know how to fix it completely, so I currently add a from_json filter and change the Qwen3.5's chat_template.
Here is the current patch, it is not perfect and I don't want to create another PR now:
diff --git a/xinference/model/llm/utils.py b/xinference/model/llm/utils.py
index 9d47d399..dde957ea 100644
--- a/xinference/model/llm/utils.py
+++ b/xinference/model/llm/utils.py
@@ -135,12 +135,23 @@ class ChatModelMixin:
def raise_exception(message):
raise TemplateError(message)
+ def _from_json(value):
+ if isinstance(value, (dict, list)):
+ return value
+ if isinstance(value, str):
+ try:
+ return json.loads(value)
+ except (json.JSONDecodeError, TypeError):
+ return value
+ return value
+
jinja_env = ImmutableSandboxedEnvironment(
trim_blocks=True,
lstrip_blocks=True,
extensions=["jinja2.ext.loopcontrols"],
)
jinja_env.globals["raise_exception"] = raise_exception
+ jinja_env.filters["from_json"] = _from_json
return jinja_env.from_string(chat_template)
def _build_from_raw_template(
@@ -785,8 +796,7 @@ class ChatModelMixin:
transformed_messages = []
for msg in messages:
new_content = []
- role = msg["role"]
- content = msg["content"]
+ content = msg.get("content")
if isinstance(content, str):
new_content.append({"type": "text", "text": content})
elif isinstance(content, List):
@@ -810,7 +820,8 @@ class ChatModelMixin:
"Unknown message type, message: %s, this message may be ignored",
messages,
)
- new_message = {"role": role, "content": new_content}
+ new_message = dict(msg)
+ new_message["content"] = new_content if new_content else None
transformed_messages.append(new_message)
return transformed_messages
Expected behavior / 期待表现
Handle tool_call messages correctly.
System Info / 系統信息
Python 3.10
vllm 0.18.1
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
Version info / 版本信息
v2.8.0
The command used to start Xinference / 用以启动 xinference 的命令
...
Reproduction / 复现过程
Yesterday a user reported that he can't use LLM with zread_cli. After investigation, I found that xinference does not keep history tool_call messages at all!
After fixing this issue, I found another bug that in chat_template of Qwen3.5, function arguments should be a dict object instead of string (which is OpenAI standard). I don't know how to fix it completely, so I currently add a
from_jsonfilter and change the Qwen3.5's chat_template.Here is the current patch, it is not perfect and I don't want to create another PR now:
Expected behavior / 期待表现
Handle tool_call messages correctly.