perf(sidecar): Qwen3-TTS ONNX runs at only ~1.2x realtime on GPU (launch-bound AR loop)

[jiangzhuo]

Problem

Qwen3-TTS 0.6B through the ONNX talker runtime (sidecar/sokuji_sidecar/qwen3_tts/) generates at only ~1.2x realtime on a capable GPU — barely faster than playback. Since the card is non-streaming (STREAMING=False), a 5s sentence stalls the pipeline for ~4.2s before audio starts, which is very noticeable in a live translation session.

Measured on a DGX Spark (GB10, CUDA via NVIDIA's sbsa onnxruntime-gpu 1.24.0), qwen3-tts-0.6b fp32, ICL preset voice:

Device	Throughput
CUDA (GB10)	1.15x realtime (app-reported rtf: 0.839)
CPU (20-core Grace)	0.38x realtime (unusable)

For contrast, MOSS-TTS-Nano (100M, streaming) does 3-4x realtime on the same box.

Model assets: jiangzhuo9357/qwen3-tts-0.6b-onnx / jiangzhuo9357/qwen3-tts-1.7b-onnx (fp32 ONNX, repacked from zukky/Qwen3-TTS-ONNX-DLL, exported from QwenLM/Qwen3-TTS).

Why it is slow (analysis, verified on-device)

The bottleneck is per-step host/device round-trip overhead, not GPU compute — the GPU is mostly idle:

1. 16 sequential session.run calls per 80ms audio frame. The talker is a 12.5Hz AR model: each frame runs 1x talker_decode + 15x code_predictor (16 codebooks, each sub-step depends on the previous one's embedding). Every call is a full Python -> ORT -> GPU round trip with a tiny batch — pure kernel-launch latency.
2. CPU<->GPU copies every step. ORT inserts 12-36 Memcpy nodes into the graphs on CUDA (logged at session build), and the AR loop's sampling / embedding concat runs Python/numpy-side, so tensors bounce between host and device each step.
3. fp32-only export — double the memory bandwidth of an fp16 path, no KV quantization.

Possible directions (unverified ideas only — none validated, other paths not excluded)

• IO binding / keep tensors resident on device between steps to eliminate the per-step Memcpy traffic. Moderate effort; plausibly the best effort/reward ratio.
• fp16 re-export of the talker/code-predictor graphs (halves bandwidth; needs new model assets).
• TensorRT EP (available in the sbsa onnxruntime-gpu build): graph fusion could help the most, but engine build time and op coverage are unknowns.
• Fuse the 15 code_predictor sub-steps into a single graph (loop inside the ONNX graph); biggest change, requires re-export.
• Streaming support for the qwen3 lane would not raise throughput but would hide latency (start playback at the first decoded chunk), similar to how MOSS feels fast.

These are starting points for investigation, not conclusions — the right fix may well be something else entirely.

Context

• Runtime: sidecar/sokuji_sidecar/qwen3_tts/ (pure-onnxruntime port; reference was zukky/Qwen3-TTS-ONNX-DLL's run_pipeline.py)
• The macOS lane uses MLX instead and is unaffected.
• Related recent fix: the tokenizer added_tokens bug (babble output) — fixed in 132ebd5; this issue is purely about throughput/latency.

🤖 Generated with Claude Code

[jiangzhuo]

Results (PR #294): on GB10 CUDA, 0.6B goes from 0.94x to 2.17x realtime (short sentences 0.70x → 2.08x, 38-43 ms/frame) and 1.7B from 0.48-0.76x to 1.71x. Correctness held to the ASR-loopback red line: 19/20 and 9/10 sense-voice transcriptions exact; every failure is the pre-existing temp-0.9 leading-word drop that the unmodified fp32 baseline also produces on the same sentence.

Where the time actually went (measured, per generated frame, 0.6B fp32 baseline ~84ms)

component	share	root cause
code_predictor (15 serial calls)	45%	per-call dispatch of a ~565-node graph + 440MB weight re-read per call
codec decode	26%	the export pads every input to a fixed 1024 frames (82 seconds) and decodes all of it — 1.3s per call, length-independent
talker_decode	23%	16385-node fragmented export → dispatch-bound; KV round-trips host<->device as numpy every step
python/sampling	2%	negligible — the original launch-bound thesis was right, but the overhead lives inside session.run

What shipped

• IOBinding device-resident KV cache (ping-pong double binding; ORT 1.24's get_outputs() values are invalidated by clear_binding_outputs() — single-binding reuse segfaults), embed-table extraction, per-length CUDA-graph replays for code_predictor (bit-identical), 12-frame ref-decode cap (the ICL prefix is pure vocoder warm-up), seed/greedy reproducibility hooks.
• Dynamized codec (biggest single win): the 29 frozen affine pad-target constants now derive from Shape(audio_codes) rounded to a multiple of 64 — 1308ms → 82ms at 32 frames, max|diff| ~1e-5. Already uploaded into both HF repos in place (safe for all lanes).
• bf16 talker + code_predictor under onnx-bf16/ in both repos, picked up only on device=cuda. The working recipe is ORT-optimize-first-then-convert (the fused contrib ops have bf16 CUDA kernels); bf16 keeps fp32's exponent range.

Dead ends (measured, so nobody re-walks them)

• fp16: Qwen3 activation-outlier channels exceed 65504 inside the residual stream by layer 3 on real inputs (all-NaN logits); blocking SwiGLU products and the RoPE chain is not enough. fp16 is unusable for these graphs.
• int4/int8 MatMulNBits: 4.1 vs 2.7 ms/call on GB10 — dequant overhead beats the bandwidth saving at [1,2..16,1024] GEMV shapes.
• Padded-static-KV CUDA graph for the talker: the new token's RoPE position is derived from the past tensor's shape, not the attention mask, so padded KV shifts positions and wrecks the logits.

Remaining headroom

talker_decode (~20 ms/frame) is now the dominant cost and is dispatch-bound; ORT-level options are exhausted. TensorRT EP (whole-graph fusion, engine cache, optional trt_cuda_graph_enable) is the credible next step; it requires installing the TensorRT 10 sbsa runtime, which is absent on the dev box today. Note the accel bench cache persists rtf per (machine, model, backend, device, compute) — clear it after upgrading or the old numbers keep being reported.

[jiangzhuo]

Update — merged to main and shipped (2026-07-12)

The perf stack in the results comment above is now on main, delivered as part of the new Local Native provider:

• PR perf(sidecar): 2x realtime Qwen3-TTS on CUDA (issue #293) #294 (perf stack) → PR Merge main (provider-descriptor registry) into native-sidecar #296 (main's provider-descriptor refactor merged in) → PR feat: Local Native provider — fully offline ASR / translation / TTS via a python sidecar #297 (→ main, merge 322cfeec). sidecar v0.1.5 tagged (all-SKU bundles).
• Final GB10 CUDA numbers stand: 0.6B 0.94× → 2.17× (short sentences 2.08×), 1.7B 0.48× → 1.71×. The dynamized codec (fp32, all lanes) and the onnx-bf16/ cuda-only assets are uploaded in place to both HF repos.

TensorRT EP — evaluated and declined. The "credible next step" flagged in the previous comment is not being pursued: bundling/installing the TensorRT 10 sbsa runtime plus a per-machine engine build is too much distribution cost for how this ships. That makes talker_decode (~20 ms/frame, dispatch-bound) the endpoint of the pure-ORT route — any further gain would have to come from an upstream re-export that fuses the AR sub-steps into a single graph, or a different runtime, not from more EP/graph tuning here.

Net: the ≥2× realtime goal is met and shipped. This can be closed unless we want to keep it open to track the upstream re-export idea.