[Bug]: GLM-5.3-Flash with forced tool_choice (named function) fails to converge — runs to max_tokens and returns ▎ truncated tool_call.arguments

[jarvisz3]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

Your current environment

Environment

vLLM version : v0.1.dev20051+g487ecf187
Container    : docker.io/vllm/vllm-openai:glm53-flash-x86_64-cu130
Model        : zai-org/GLM-5.3-Flash  (native FP8, Glm5NextForConditionalGeneration)
GPU          : 4 x NVIDIA H200 (SXM, NVSwitch), TP=4, EP off
Driver       : 580.105.08, CUDA 13.0
Host OS      : RHEL 10.0, kernel 6.12.0
Runtime      : podman 5.x (rootful)

Server args:

python3 -m vllm.entrypoints.openai.api_server \
  --model zai-org/GLM-5.3-Flash \
  --served-model-name zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype auto \
  --gpu-memory-utilization 0.92 \
  --max-model-len 524288 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 16 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --host 0.0.0.0 --port 8000

Relevant engine config from the startup log:

structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False,
    disable_additional_properties=False, reasoning_parser='glm45', ...)
kv_cache_dtype=auto, quantization=fp8, enforce_eager=False
compilation_config={'mode': <CompilationMode.NONE: 0>, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, ...}

Describe the bug

When serving zai-org/GLM-5.3-Flash with a forced tool_choice
({"type": "function", "function": {"name": "<fn>"}}) and a demanding JSON schema,
a fraction of requests never converge: generation runs all the way to max_tokens
and the response comes back with finish_reason: "length" and
tool_calls[0].function.arguments truncated to 133–144 characters, which is not
valid JSON.

The same schema and the same prompt with tool_choice: "auto" never fails.

This is not a speculative-decoding bug — see the control experiment below.

Observed failure rate

Identical 54-case matrix (3 scenarios x 3 temperatures x streaming/non-streaming x 3 repetitions)
run against two otherwise-identical instances on the same host:

Configuration	Failures
No speculative decoding	4 / 54
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'	3 / 54

All failures were in the forced-tool_choice scenario. Zero failures in the
tool_choice: "auto" scenarios (18 runs each), including with the same nested schema
and outputs of 5,000–12,700 characters.

Failure signature

Truncation always lands at a nested-object boundary. Two representative failures
(max_tokens=6144, generation took 15–38 s, so the budget really was consumed):

finish_reason = "length",  len(arguments) = 133
arguments = {"title": "Senior Backend Engineer", "company": "Bossjob", "location":
             {"city": "Manila", "country": "Philippines", "remote": false}}

finish_reason = "length",  len(arguments) = 144
arguments = {"title": "Senior Backend Engineer", "company": "Bossjob", "location":
             {"city": "Manila", "country": "Philippines", "remote": false}, "salary": 

The second one shows the request stopping right after the "salary": key, with the
nested object never opened.

One request also returned HTTP 500 Internal Server Error under the same conditions
(forced tool_choice, temperature=0.0, non-streaming).

Independent of

• Speculative decoding — reproduces with MTP off (4/54) and on (3/54).
• Temperature — reproduces at 0.0, 0.7 and 1.0.
• Streaming — reproduces with stream: true and stream: false.

Correlates with

• Forced tool_choice (named function). "auto" never reproduced it.
• Expected output size. A reduced schema (6 required fields, max_tokens=4096,
  producing ~3,000-character arguments) passed 18/18 across nested-forced,
  flat-forced and nested-auto. The failing configuration used 8 required fields with
  a prompt asking for "at least 10 requirements, 10 responsibilities and a 250-word
  description", i.e. expected arguments of 6,000–12,000 characters.

Nesting alone is not the trigger — a flat schema with forced tool_choice also
passed 6/6 at the smaller size. The combination of forced tool_choice and a large
expected output is what derails.

Steps to reproduce

import json, urllib.request

URL = "http://<host>:8000/v1/chat/completions"
HDRS = {"Content-Type": "application/json", "Authorization": "Bearer <key>"}

TOOL = [{
    "type": "function",
    "function": {
        "name": "create_job_posting",
        "description": "Create a detailed job posting record",
        "parameters": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "company": {"type": "string"},
                "location": {"type": "object", "properties": {
                    "city": {"type": "string"}, "country": {"type": "string"},
                    "remote": {"type": "boolean"}}},
                "salary": {"type": "object", "properties": {
                    "min": {"type": "integer"}, "max": {"type": "integer"},
                    "currency": {"type": "string"}}},
                "requirements": {"type": "array", "items": {"type": "string"},
                                 "description": "At least 10 detailed requirement strings"},
                "responsibilities": {"type": "array", "items": {"type": "string"},
                                     "description": "At least 10 detailed responsibility strings"},
                "benefits": {"type": "array", "items": {"type": "string"}},
                "description": {"type": "string", "description": "A long free-text description, 250+ words"},
            },
            "required": ["title", "company", "location", "salary", "requirements",
                         "responsibilities", "benefits", "description"],
        },
    },
}]

PROMPT = ("Create a job posting for a Senior Backend Engineer at Bossjob in Manila, "
          "Philippines (hybrid). Salary 120000-180000 PHP. Fill in EVERY field very "
          "thoroughly with realistic detail - at least 10 requirements, 10 "
          "responsibilities, and a 250-word description.")

for i in range(10):
    payload = {
        "model": "zai-org/GLM-5.3-Flash",
        "messages": [{"role": "user", "content": PROMPT}],
        "tools": TOOL,
        # the trigger: forcing the function by name
        "tool_choice": {"type": "function", "function": {"name": "create_job_posting"}},
        "max_tokens": 6144,
        "temperature": 0.7,
        "stream": False,
    }
    req = urllib.request.Request(URL, data=json.dumps(payload).encode(), headers=HDRS)
    d = json.load(urllib.request.urlopen(req, timeout=300))
    ch = d["choices"][0]
    args = ch["message"]["tool_calls"][0]["function"]["arguments"]
    ok = True
    try:
        json.loads(args)
    except Exception:
        ok = False
    print(f"#{i+1} finish={ch['finish_reason']:>10}  len={len(args):>6}  valid_json={ok}")

Expected: every run produces valid JSON with finish_reason="tool_calls" or "stop".

Actual: roughly 1 in 8–15 runs produces finish_reason="length" with len(args) around
133–144 and invalid JSON. Switching tool_choice to "auto" makes it pass every time.

Additional context

• --tool-call-parser glm47 and --reasoning-parser glm45 are the values the model card
  and the vLLM recipe for this model recommend. The same parsers behave correctly for
  tool_choice: "auto", including for 12,000-character argument payloads, so the incremental
  streaming parser itself does not look implicated.
• Because forced tool_choice routes through the structured-outputs backend
  (backend='auto'), the suspicion is a grammar/FSM issue rather than a tool-parser issue,
  but we have not isolated which backend was selected.
• Happy to run additional experiments on this hardware (4x H200, TP=4) if that helps —
  e.g. pinning --structured-outputs-config.backend to xgrammar vs guidance.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.