Qwen3.5-0.8B WebGPU translation is too slow for real-time (~0.7 tok/s); investigate acceleration / text-only path

[jiangzhuo]

Summary

qwen3.5-0.8b-translation (onnx-community/Qwen3.5-0.8B-ONNX, VLM arch, qwen35-translation.worker.ts) decodes far too slowly to be usable for real-time translation. Measured on an RTX 4070: a single short sentence took ~21 s — and this is not a thinking-mode problem.

Measurement (live, RTX 4070, en→zh)

Input: "It's an automatic speech recognition model." → output "这是一个自动语音识别模型。" (correct) in ~21,000 ms.

Instrumenting the worker showed:

• Thinking is already OFF. The packaged chat template renders a closed empty block: …assistant\n<think>\n\n</think>\n\n. Raw model output before the <think> strip was len=13, hasThink=false — i.e. it generated only the ~15-token answer, no reasoning.
• So ~15 tokens in ~21 s ≈ ~0.7 tok/s. For a 0.8B model on a 4070, expected is 30–60 tok/s — this is ~50× too slow.

What this rules out

• Not thinking — confirmed off (template default + raw output).
• Not the vision encoder / VLM packaging — per the analysis in Evaluate Qwen3.5-0.8B-Text-ONNX as text-only alternative to VLM variant #144, the VLM and Text-ONNX variants share identical decoder weights and the vision encoder is dead code for text-only, so they decode at the same speed. Switching to Qwen3.5-0.8B-Text-ONNX would only save ~36 MB of download, not fix speed.

Hypothesis

Qwen3.5 is a brand-new architecture (Qwen3_5ForConditionalGeneration); transformers.js WebGPU likely lacks kernels for some of its ops, so they silently fall back to CPU/WASM, collapsing decode throughput. (Secondary: q4 vs q4f16 variant selection on shader-f16 devices.)

Investigation TODO

• Confirm CPU fallback: watch nvidia-smi -l 1 during a translation — low GPU utilization over the ~21 s ⇒ CPU-bound.
• Compare against a known-good WebGPU model on the same input (e.g. qwen2.5-0.5b-translation, or another WebGPU translation worker). If Qwen2.5 is fast and Qwen3.5 is slow, it isolates the Qwen3.5 arch.
• Verify which variant is loaded (q4 vs q4f16) on a shader-f16 device; ensure the faster variant is selected.
• Evaluate driving the worker with AutoTokenizer + string ChatML instead of AutoProcessor (parallels the native/sidecar fix for the same model — video processor is heavy/gated there). Likely an init/cleanup win, not the decode fix, but worth confirming it doesn't pull in unused multimodal preprocessing.
• Decide the outcome: wait for transformers.js Qwen3.5 WebGPU kernels, or drop Qwen3.5-0.8B from the real-time translation options in favor of a faster model.

Practical note

Until this is resolved, Qwen3.5-0.8B is not viable for live translation. Faster local options: Opus-MT (per-pair, WASM/CPU, typically sub-second) or a lighter WebGPU model.

Related: #144 (Text-ONNX variant evaluation — download-size only), #148 (Qwen3.5 ASR).