[None][perf] Chat frontend fastpath: single template render + suffix token-id cache + to_thread offload

[lingjiew]

Problem

The OpenAI chat frontend renders the chat template twice and tokenizes the
prompt twice per request (stable-prefix pass for reusable_prompt_len + full-
prompt pass), synchronously on the asyncio event loop. For ~100k-token AgentX
chats this is ~0.3-0.5s CPU per request; at C32 closed-loop the single-threaded
frontend saturates and the stage inflates to 3.82s mean / 10.9s p99 of
queueing — 45% of TTFT (perf_metrics 8-stage breakdown; ~14 of 32 lanes
resident in front of the tokenizer thread by Little's law, while only ~7 wait
at the executor).

Fix

New _render_and_tokenize_chat(), called via asyncio.to_thread:

• render the full template once, tokenize once;
• cache generation-suffix token ids per (template hash, tools, documents, add_special_tokens, kwargs) — constant per template;
• one-time boundary probe per cache key verifies
  encode(stable) + encode(suffix) == encode(full);
• submit token ids to generate_async (skips the internal re-tokenize).

Safety

• probe mismatch or any exception → permanent per-key fallback to the legacy path;
• multimodal / no-generation-prompt / truncate_prompt_tokens requests always take the legacy path.

Measured (Qwen3.5-397B-A17B-NVFP4 disagg ctx, GB300, AgentX 256k traces, C32 overlay A/B)

metric	before	after
frontend+response software segments	6.37s	0.50s
ctx throughput	4.33 req/s	6.60 req/s (+52%)
TTFT p50	—	-65%

The committed file is byte-identical to the overlay validated on lyris GB300
(deployed per-cell via container mount during the AgentX benchmark campaign).

🤖 Generated with Claude Code

[jiangjb01]

Hi @nv-guomingz,
I noticed you have submitted a large number of commits related to Qwen3.5 model adaptations in the TensorRT-LLM repository. Could you please share any official deployment guides or practical materials for Qwen3.5-35B and Qwen3.5-27B models?
My personal email is jcxz125@outlook.com. It would be greatly appreciated if you could send the relevant documents to my mailbox. Looking forward to your reply, thank you very much!

[nv-guomingz]

Hi @nv-guomingz, I noticed you have submitted a large number of commits related to Qwen3.5 model adaptations in the TensorRT-LLM repository. Could you please share any official deployment guides or practical materials for Qwen3.5-35B and Qwen3.5-27B models? My personal email is jcxz125@outlook.com. It would be greatly appreciated if you could send the relevant documents to my mailbox. Looking forward to your reply, thank you very much!

Thanks @jiangjb01 , could u plz take a look at this link ? https://nvidia.github.io/TensorRT-LLM/deployment-guide/deployment-guide-for-qwen3.5-on-trtllm.html

Although this document is specifically for Qwen 3.5 397B, I believe it will still be a highly relevant reference for you.

[jiangjb01]

Hi @nv-guomingz, I noticed you have submitted a large number of commits related to Qwen3.5 model adaptations in the TensorRT-LLM repository. Could you please share any official deployment guides or practical materials for Qwen3.5-35B and Qwen3.5-27B models? My personal email is jcxz125@outlook.com. It would be greatly appreciated if you could send the relevant documents to my mailbox. Looking forward to your reply, thank you very much!

Thanks @jiangjb01 , could u plz take a look at this link ? https://nvidia.github.io/TensorRT-LLM/deployment-guide/deployment-guide-for-qwen3.5-on-trtllm.html

Although this document is specifically for Qwen 3.5 397B, I believe it will still be a highly relevant reference for you.

Hi @nv-guomingz! Thank you very much for your guidance. I will follow the steps in the shared link to resolve this issue. Much appreciated!