Pocket TTS (Kyutai zero-shot voice cloning): technical investigation — runtime & performance characteristics

[jiangzhuo]

Summary

Technical investigation into running Pocket TTS (Kyutai CALM, zero-shot voice cloning) on sokuji's local-inference stack. The same int8 ONNX models were measured across three runtimes:

• Browser WASM (onnxruntime-web): ~0.6× realtime — not viable for real-time.
• Native CPU (Python sherpa-onnx ~4×, Node onnxruntime-node ~2–2.4×): real-time with headroom.
• The model is English-only.

PoC branches: feat/pocket-tts-playground-poc (browser WASM), feat/pocket-tts-electron-native-poc (native Node in Electron). Both dev-gated, not wired into the production pipeline.

What is Pocket TTS

Kyutai CALM (Continuous Audio LM), zero-shot voice cloning: a reference clip → voice embedding → conditioned autoregressive generation. Pipeline = mimi_encoder (reference → latents) → flow_lm_main (~100M AR backbone, per-frame) + flow_lm_flow (consistency/flow head) → mimi_decoder (latents → 24 kHz PCM), plus text_conditioner. int8 ONNX bundle ≈ 140 MB (5 onnx + tokenizer + bundle.json + BOS npy). BOS-prepend is required for faithful timbre.

What was tested

Same int8 graphs, three runtimes: (1) onnxruntime-web WASM in a dev playground; (2) sherpa-onnx native via Python; (3) onnxruntime-node native inside Electron (our TS pocketInferenceCore).

Performance

runtime	host	RTF (cached)
onnxruntime-web WASM (1 thread)	browser	~0.6×
sherpa-onnx WASM (4 threads)	browser	~0.65×
onnxruntime-node	plain Node	~3.2×
onnxruntime-node	Electron child_process	~2.0–2.4× headless · ~1.75–2.35× full app
sherpa-onnx	plain Python	~4× (1–8 threads)

Per-frame, per-stage (same int8 graphs):

stage	WASM	native node	ratio
flow_lm_main (AR backbone)	~56 ms	~22 ms	~2.5×
mimi_decoder (vocoder)	~38 ms	~6 ms	~6×

Why WASM ≪ native

int8 kernel quality dominates:

1. SIMD / VNNI — native MLAS uses AVX2 / AVX-512 / AVX-VNNI (single-instruction int8 dot-products); WASM SIMD is fixed 128-bit, no VNNI.
2. Kernels — hand-tuned assembly (native) vs generic Emscripten C++ (WASM).
3. Threading — native multi-threads the encoder/decoder; WASM is forced single-thread (threaded WASM OOMs on Pocket's KV-caches).
4. WASM overhead — linear-memory bounds-checks, growable memory, JS boundary.

The conv-heavy mimi_decoder is hit hardest (~6×). This is an intrinsic WASM↔native gap for int8 NN inference — not fixable in WASM; the only browser path to native-class speed is WebGPU (no Pocket/Mimi WebGPU port exists today).

Native host options (for future use)

A. Python — sherpa-onnx. Fastest (~4×), simplest API (OfflineTts + GenerationConfig), ships its own C++ engine + threading + model packaging. Would run as a sidecar (Python runtime to bundle).

B. Node — onnxruntime-node (this PoC). ~2–2.4×, reuses our TS pocketInferenceCore (one core; web + node hosts via Tensor dependency injection). Integrates into Electron with no extra runtime via child_process.fork + ELECTRON_RUN_AS_NODE=1 (Electron's own Node). Hard-won learnings:

• Use child_process.fork, not utilityProcess — the latter intermittently SIGTRAP-crashed while hosting the native addon.
• serialization: 'advanced' is required to pass Float32Array over IPC.
• Cap intraOpNumThreads — the default (= all logical cores) oversubscribes the tiny per-frame matmuls and ~halves throughput.
• onnxruntime-node ships napi-v6 → ABI-stable across Node & Electron; @electron/rebuild is not needed.

Platform / runtime viability

• Electron desktop: ✅ native (Python or Node), real-time.
• Browser extension: ❌ WASM ceiling ~0.6×; no WebGPU port.
• Constraint: English-only model.

Not covered by this investigation

Production packaging (Forge/asar native .node unpack, ~140 MB model distribution), LocalInferenceClient pipeline wiring, reference-voice UX, GPU execution providers, non-English, streaming, and model licensing (Kyutai/CALM weights + the KevinAHM ONNX port — to verify before any distribution).

References

• Spec: docs/superpowers/specs/2026-06-18-pocket-tts-electron-native-poc-design.md
• Plan: docs/superpowers/plans/2026-06-18-pocket-tts-electron-native-poc.md
• Branches: feat/pocket-tts-playground-poc (browser WASM), feat/pocket-tts-electron-native-poc (native Node)
• Key commits: cfa07803 (Tensor DI), f46ad7e9 (node bench), 4adc1a0d (Electron host), 6008dc02 (child_process fix)

[jiangzhuo]

Re-verification (latest code) — the web↔native gap is ~2.2×, not ~5×; browser WASM is already real-time

Re-ran the same shared pocketInferenceCore on both runtimes (identical int8 models, text, reference) to isolate the runtime. Machine: i7-14700F (AVX2 + AVX-VNNI, no AVX-512) + RTX 4070 SUPER, onnxruntime-node 1.26.0, onnxruntime-web 1.26.0-dev, Chrome 147.

Verified RTF (steady-state generate, best-of-3, cached voice)

runtime / config	RTF	per-frame
node onnxruntime-node, 1 thread	3.0×	main 18.7ms · decode 6.3ms
node, 4 / 8 threads	2.5× / 3.1×	(threads regress the seqlen-1 backbone)
web WASM, 1 thread	1.38×	main 38.7ms · decode 16.2ms
web WASM via the real pocket-tts.worker.ts	1.38×	identical per-stage → no harness bias

The documented "browser WASM ~0.6× — not viable" does not reproduce on current code. Node reproduces (~3.0× ≈ the old ~3.2×), but the web path is 1.38× — comfortably real-time here, confirmed via both a controlled bench and the real production worker. So the real gap is ~2.2×, not ~5×. Per-stage WASM tax is 2.07× (AR backbone) / 2.57× (decoder) — not the old 2.5× / 6×; the old "6× decoder" was an outlier (its native decoder 6 ms matches mine 6.3 ms, but its WASM decoder 38 ms vs my 16.2 ms).

The old 0.64× was recorded on this same machine. I ruled out SIMD (simd=0 gives the identical 1.38× — ORT-web 1.26 ships only the always-SIMD build, so the flag is a no-op) and code changes (web path unchanged). Most likely cause: CPU throttling in that earlier session (governor is powersave; cores boost to 5.3 GHz now, but an earlier run on E-cores / without boost would be ~2× slower). The hardware-stable metric is the ~2.07–2.57× ratio; absolute RTF tracks single-core speed, so a slow laptop can still dip below real-time.

Root cause (verified in ONNX Runtime source AND the shipped binary)

npm ort-wasm-simd-threaded.jsep.wasm runs MatMulInteger via a fixed 128-bit wasm_i32x4_dot_i16x8 (widen+pairwise-add, 2 MACs/lane); node dispatches AVX-VNNI VPDPBUSD (256-bit, 4 MACs/lane fused). ORT has a relaxed-SIMD VNNI-mapping kernel (qgemm_kernel_wasmrelaxedsimd.cpp, HasUSDot()-gated) but it is compiled only with --enable_wasm_relaxed_simd, which the official npm build does not set — binary check: 19× the fixed-dot opcode, zero relaxed int-dot opcodes. The gap is the missing single-instruction int8 dot plus WASM's 128-bit vs native's 256-bit vector width.

Levers (all tested)

• Custom relaxed-SIMD ORT-web build — the only real lever. Projected ~1.6–1.9× web RTF (from measured per-stage). Closes ~⅓ of the gap; cannot reach node 3.0× (128-bit cap). (Building this now; will follow up with measured numbers.)
• Multi-thread WASM — InferenceSession.create hangs at init across 3 environments, and node shows threads regress this AR-dominated workload → low value.
• WebGPU EP on the int8 graphs — measured 0.18× and all-NaN output (no WebGPU kernel for MatMulInteger/DynamicQuantizeLinear; per-node CPU fallback + GPU↔CPU copies). Dead end as-is. The only WebGPU fast path is re-export to MatMulNBits-4bit/fp16 (new model files + audio re-validation), which could exceed node on the GPU but is a larger effort.

Honest ceiling

Browser WASM cannot reach native 3.0× on this model class (128-bit vs 256-bit AVX2+VNNI is structural). Best realistic WASM web ≈ 1.6–1.9× with a relaxed-SIMD build (still real-time). The only path to beat node is a WebGPU re-export to MatMulNBits/fp16.

Harness + full report on branch investigate/pocket-perf-263 (POCKET_TTS_263_VERIFICATION.md).

[jiangzhuo]

Follow-up: custom relaxed-SIMD ORT-web build — built and measured

Built onnxruntime-web from the exact npm commit b7804b056c with --build_wasm --enable_wasm_simd --enable_wasm_threads --use_jsep --enable_wasm_relaxed_simd (emsdk 4.0.23, cmake 3.31). The resulting ort-wasm-relaxedsimd-threaded.jsep.wasm contains 44× the relaxed int-dot opcode fd 93 02 (wasm_i32x4_relaxed_dot_i8x16_i7x16_add → AVX-VNNI VPDPBUSD) vs the npm bundle's zero. Self-hosted it and re-ran the bench (same machine, threads=1, same-session A/B):

build	RTF	best gen	tMain	tDecode
npm (fixed 128-bit i16x8 dot)	1.40×	8995 ms	6013 ms	2507 ms
custom relaxed-SIMD	1.51×	8208 ms	5658 ms	2172 ms
delta	+8%	−8.7%	−5.9% (backbone)	−13.4% (decoder)

The VNNI kernel engages and helps — but only ~8%, not the ~1.6–1.9× I projected earlier. Output stays valid (no NaN). The shape matches expectation (decoder benefits most, GEMV backbone least), but the magnitude lands at the conservative end (≈ ORT #22533's ~1.15× whole-model calibration). Why so modest: at sequence-length 1 the AR backbone is memory-bandwidth-bound, so a faster int8 dot barely moves it; only the 12-frame-batched decoder is compute-dense enough to gain.

Bottom line (empirically bounded now)

• Web is real-time today (~1.4×) on a fast desktop; a relaxed-SIMD build nudges it to ~1.5×.
• It cannot approach node's 3.0× — WASM is 128-bit vs native 256-bit AVX2+VNNI, and the dominant stage is memory-bound. The relaxed-SIMD build is a real but small win and adds a custom-build/maintenance burden for ~8%.
• The only path that could exceed node in-browser is a WebGPU re-export to MatMulNBits-4bit/fp16 (new models + audio re-validation) — untested; the int8 graphs as-is are a WebGPU dead end (0.18× + NaN).

Recommendation: ship the default npm WASM for the browser (already real-time on capable hardware); reserve native (onnxruntime-node, optionally GPU EP) for the desktop fast path. Only invest in the relaxed-SIMD build if the ~8% matters for a specific target. Full report + reproducible harness on branch investigate/pocket-perf-263.

[jiangzhuo]

Follow-up: the old 0.64× vs now-1.40× — root-caused to CPU core placement (measured, not inferred)

My first comment said the old 0.64× (recorded on this same i7-14700F, same code) was probably CPU throttling. I went back and measured it instead of guessing.

This is a hybrid CPU (8 P-cores @ ~5.3 GHz, high IPC + 12 E-cores @ ~4.2 GHz, low IPC) under the powersave governor. The Pocket int8 path is pure integer-SIMD compute (no VNNI in WASM), so its RTF tracks the core's effective speed almost linearly — maximally sensitive to which core the compute thread lands on. Pinning the exact same workload with taskset:

pinned to	node (native)	web (WASM)
P-core (CPU 0, 5.3 GHz)	2.77× (22.4 ms/frame)	1.27×
E-core (CPU 16, 4.2 GHz)	1.27× (46.5 ms/frame)	0.76×
P/E ratio	2.18×	1.67×

Findings:

• Pinning the browser to E-cores reproduces the sub-real-time regime: web drops to 0.76×. That's the "0.6几" — it was the WASM compute thread running on an E-core (Intel Thread Director / scheduler often parks background/worker threads on E-cores).
• The node P/E ratio (2.18×) matches the historical 0.64↔1.40 ratio (2.19×) almost exactly.
• Telling cross-check: node-on-E-core (1.27×) ≈ web-on-P-core (1.27–1.40×) — the "WASM-vs-native ~2× tax" and the "E-core-vs-P-core ~2× slowdown" are two independent ~2× factors; the old 0.64× was both stacked (WASM and E-core).

(I reproduced 0.76×, not exactly 0.64× — I pinned to all 12 E-cores, which leaves scheduling headroom; a single E-core or more background load would land closer to 0.64×. Mechanism and magnitude both confirmed.)

Takeaway: absolute RTF for this model is dominated by core placement / frequency, so single-number RTF claims are fragile. The stable, hardware-independent metric is the ~2.07–2.57× web/native per-stage ratio. Practical implication: on capable hardware with the compute on a P-core, web is real-time (~1.4×); if it's parked on an E-core (or a slow laptop), it dips below real-time — exactly the band the +8% relaxed-SIMD build helps rescue.

[jiangzhuo]

Follow-up: Electron renderer (the desktop app's web environment) — measured

Ran the same WASM bench inside a real Electron BrowserWindow (Chromium 144 / Electron 40.8.5). It behaves identically to standalone Chrome — same V8/WASM engine:

environment	npm WASM	relaxed-SIMD	E-core pinned
standalone Chrome 147	1.27–1.40×	1.51×	0.76×
Electron 40 renderer	1.29×	1.37×	0.75×

• Foreground (visible/focused) window → P-cores → ~1.3× (real-time); the relaxed-SIMD build loads in Electron and gives the same ~+8%; pinned to E-cores it reproduces the sub-real-time ~0.75× regime.
• Electron-specific risk: Chromium deprioritizes backgrounded/hidden renderer processes, and on this hybrid CPU a deprioritized renderer can be scheduled onto E-cores → ~0.75× (below real-time). I measured the foreground best case (backgroundThrottling:false, focused) — a minimized/hidden window doing WASM could silently drop below real-time.

Implication: the Electron web environment offers no speedup over the browser — it hits the same WASM ceiling. The desktop advantage comes entirely from the native onnxruntime-node path (3.0× plain / ~2.0–2.4× via child_process), which is ~2× faster than WASM-in-renderer and immune to renderer scheduling. So: use native node for desktop; reserve WASM for the browser extension. Driver: scripts/electron-bench.cjs on branch investigate/pocket-perf-263.

[jiangzhuo]

Follow-up: backgrounding STALLS the WASM inference — fixed on main

While discussing P-core scheduling I found (and measured) a more urgent, concrete problem for the Electron desktop WASM path, and shipped a fix.

Problem: local inference runs in a renderer Web Worker. When the Sokuji window is minimized / hidden / occluded — the common case while the user is in a video call with Sokuji translating in the background — Chromium backgrounds the renderer and throttles its timers. The generate loop yields with setTimeout(0) every 16 frames, and those get clamped to a crawl, so the loop nearly halts.

Measured (Electron 40 renderer, same machine):

window state	RTF
visible	1.30×
hidden/minimized — default settings	stalls — 1 inference did NOT finish in 185s (vs ~10s visible)
hidden/minimized — with the fix	1.35× (full speed, = foreground)

So with default settings, translation effectively dies whenever the window isn't in front — exactly when users need it.

Fix (shipped to main, ea57cdbf) — electron/main.js, cross-platform, no native code:

app.commandLine.appendSwitch('disable-renderer-backgrounding');
app.commandLine.appendSwitch('disable-background-timer-throttling');
app.commandLine.appendSwitch('disable-backgrounding-occluded-windows');
// + webPreferences.backgroundThrottling: false on the main window

Tradeoff: when the window is hidden and idle it won't down-throttle timers (slightly more background CPU), but inference is event-driven, so the cost only appears while actively translating — which is when you want full speed.

On the broader P-core question: explicit P-core affinity / QoS pinning is deferred to the future native-inference (Python sidecar) task — it needs a real process to pin (a Chromium renderer can't be durably pinned; on macOS hard-pinning is impossible, QoS-only). Until that lands, this anti-backgrounding fix is the one actionable lever for the shipping WASM path, and it addresses the actual user-visible failure (background stall), which matters more than P-vs-E core placement.

[jiangzhuo]

Follow-up: shipping the relaxed-SIMD build to production — what it takes + verdict

Analyzed what it would take to productionize the custom relaxed-SIMD ORT-web build (the +8% one), and the compatibility risks.

Verdict: don't ship relaxed-SIMD now

+8% was measured on one VNNI CPU; fleet-wide it's likely less (non-VNNI CPUs fall back to the fixed kernel). It carries a permanent custom-build + dual-artifact + numeric-validation burden across two distribution targets (Electron + extension), with no upstream contract — not worth it while web RTF is already >1.4× (real-time) and the real user-visible problem (background stall) is already fixed on main. Higher-ROI lever: multi-threading (the threaded wasm is already built, upstream-supported; sherpa native hit ~4.5× at 4 threads) — only cost is COOP/COEP. Revisit relaxed-SIMD Electron-first only if profiling proves ORT inference is the fleet-wide bottleneck and threading is exhausted/blocked.

If shipped, must handle

• Ship BOTH relaxed + fixed artifacts + a JS-layer fallback. A relaxed .wasm hard-fails to instantiate (CompileError) on engines without relaxed-SIMD, and ORT throws with no auto-fallback (isRelaxedSimdSupported() → throw). Feature-detect, set env.wasm.simd='relaxed' only when true, catch + re-init on the fixed artifact otherwise.
• Real rebuild, not a byte-swap. The simd/relaxedsimd filename is baked into the JS glue at build time → rebuild js/web at the pinned commit. (The dev hack of serving relaxed bytes under the simd filename is not shippable.)
• Set env.wasm.simd='relaxed' in the workers — it's never auto-selected; default true loads the fixed build, so without this you ship the artifact and get zero gain, silently.
• Distribute to Electron (ASAR) + extension (web_accessible_resources); drop --use_jsep (WebGPU irrelevant, ~halves size); CI to rebuild + re-validate on every ORT bump.

Compatibility risks

dimension	finding	severity
Browser support	relaxed default-on: Chrome/Edge 114 (sokuji floors 116 / Chromium 144 are above it). But Firefox(older), all Safari/WKWebView ≤18.x, locked-down embedded Chromium are unsupported → ORT hard-throws → must ship the fixed fallback.	HIGH
CPU coverage	No regression — MLAS HasUSDot() runtime probe falls back to the fixed kernel where the relaxed dot misbehaves. Gain only on AVX-VNNI x86 + ARM-dotprod; others unchanged.	LOW
Numeric determinism	relaxed dot is spec-nondeterministic (2 impl-defined axes; WebAssembly/relaxed-simd#129 still open) and MLAS hits the impl-defined path on ~every inference (u8 activations >127). Safe (HasUSDot-gated, no NaN) but not bit-identical across builds/CPUs → breaks golden/snapshot audio tests; perceptually irrelevant for TTS/ASR.	MEDIUM

Validation required before shipping

Same-machine bit-diff (relaxed vs fixed); cross-vendor A/B (AVX-VNNI Intel, pre-VNNI x86, ARM/Apple) asserting no-NaN + TTS perceptual parity + ASR WER≈0; confirm the fixed fallback runs on a relaxed-unsupported engine; field telemetry of which kernel was selected (to catch silent fallback that loses the +8%).

Full write-up on branch investigate/pocket-perf-263 (POCKET_TTS_263_VERIFICATION.md).