[None][perf] Offload chat template rendering to the input-processor pool

[lingjiew]

Description

Offload OpenAI chat-template rendering (apply_chat_template) from the asyncio event loop to the existing _input_proc_executor thread pool in the chat completions path.

This is the main-branch port of an AgentX serving fastpath originally developed and validated on the qwen3.5_agentx fork (fork commit 346c3d6bc8, fork PR nv-guomingz#7). Most of the original patch has since been absorbed into main in other forms — the prompt is rendered once, tokenization runs on the dedicated input-processor pool via the generator preprocess path, and orchestrator-relayed requests bypass text entirely via prompt_token_ids_b64. The one piece still missing on main is the template render itself, which remains synchronous on the event loop.

Motivation / measurements (fork stack, GB300, Qwen3.5-397B, 256k-context agentic traces)

• The render is pure-CPU jinja + python work that grows with conversation length: hundreds of ms per request at ~100k-token chats.
• Under closed-loop load at concurrency 32, the single-threaded frontend saturates (rho ~= 1) and the stage inflates to 3.82 s mean / 10.9 s p99 of queueing (~45% of TTFT) — measured via /perf_metrics five-stage timestamps.
• Moving render+tokenize off the loop recovered +57% requests/s (3.77 -> 5.93 req/s) and cut TTFT p50 by 65% on the fork stack. On main only the render share of that applies (tokenization is already offloaded), so the expected win is smaller but the failure mode it removes — long-context bursts serializing behind the event loop — is the same.

What changed

• tensorrt_llm/serve/openai_server.py: the apply_chat_template(...) call in the chat completions handler is dispatched via loop.run_in_executor(self._input_proc_executor, ...), mirroring the preprocess offload a few lines below. No behavior change otherwise.

Deliberately NOT ported from the fork patch

• Suffix token-id cache + double-render elimination: main renders once and has no reusable_prompt_len plumbing, so these are moot here.
• Frontend tokenize + token-id submission: already covered on main by the preprocess -> PreprocessedInputs path.
• The multimodal encoder endpoint (openai_mm_encoder) keeps its synchronous render — cold path.

Test Coverage

Draft until re-validated on main: the fork-stack A/B was measured with aiperf closed-loop AgentX traces; I plan to rerun a like-for-like A/B on a main build. Thread-safety of apply_chat_template under concurrent rendering was exercised on the fork at concurrency 32 for 3600 s runs without incident.

PR Checklist

• PR title follows the [ticket][type] convention
• Signed-off (DCO)
• Re-validation on main (draft gate)

🤖 Generated with Claude Code