Research: Windows AI Speech Recognition API (Microsoft.Windows.AI.Speech) as an on-device ASR engine for Windows

[jiangzhuo]

Summary

Feasibility research (2026-07-24) on integrating the new Windows AI Speech Recognition API (Microsoft.Windows.AI.Speech) as an on-device ASR engine for Sokuji on Windows.

Verdict: technically feasible — Microsoft even provides an official Electron integration path — but too early to integrate today. This issue records the research so we can act quickly when the blockers clear.

What the API offers

• Fully on-device speech-to-text (no cloud, no network, free), part of Windows AI Foundry / Windows App SDK.
• StreamingRecognition with both Recognizing (partial) and Recognized (final) events — matches the delta/final model our existing providers use for live captions.
• BatchRecognition for whole-file transcription.
• Audio input: AudioConfiguration.FromAudioDevice / FromFile / FromStream, or caller-pushed PCM via SpeechAudioProvider (16 kHz, 16-bit, mono) — the push path would let us keep our existing ModernAudioRecorder pipeline (echo cancellation, device management) and feed PCM over IPC instead of letting the API grab the microphone.
• Hardware: NPU on Copilot+ PCs (model preinstalled — zero download) or a capable CPU (model downloaded on demand via Windows Update, user-removable). GPU not supported. Hardware selection is automatic.
• Requires Windows 11 24H2 (build 26100)+.

Current status (verified against release notes, 2026-07-24)

Aspect	Status
API stability	Experimental / public preview. Introduced in Windows App SDK 2.2 Experimental 9 (2026-06-09); NOT in any stable release through 2.3.1 (2026-07-16). Classes carry the [Experimental] metadata attribute. Note: the WinAppSDK 2.x line itself IS stable (2.0 stable since 2026-04-29) — only the Speech namespace is experimental. The how-to doc's "WinAppSDK 1.7.1+" prerequisite appears to be a doc error.
Language support	English only at public preview; Microsoft says more languages will roll out gradually.
Package requirements	Calling process needs package identity + systemAIModels capability in the appxmanifest (MSIX or sparse package / packaged-with-external-location).
Official Electron path	@microsoft/windows-ai-electron (Public Preview) projects Windows AI Foundry APIs to JS, with @microsoft/winappcli handling manifest/identity. Speech is not yet projected (only Text / Imaging / AI core), and no one has filed a request for it yet.

Blockers (why not now)

1. English-only recognition. For a translation app this covers only the EN→X direction; our WASM and sidecar local ASR paths are already multilingual. This is the biggest long-term blocker.
2. Package identity vs. our distribution. Sokuji ships Windows via Squirrel (forge.config.js); production identity means adding a signed sparse package or moving to MSIX — a distribution-chain change. There is also a known (fixed-but-still-rolling-out) Windows bug where sparse-packaged Electron apps crash on start.
3. Experimental API — surface may change; needs the 2.2-experimental9+ package.
4. No JS projection for Speech yet in the official Electron addon. Alternatives (own C++/WinRT NAPI addon, or a small C# helper process sharing package identity, similar to our Python sidecar pattern) are possible but not worth building against an experimental surface.

Why we still want this

On Copilot+ PCs the model is preinstalled and runs on the NPU — zero model download and zero CPU/GPU load. Neither of our current local providers (WASM, native sidecar) can offer that. Free, offline, low-latency, with partial results.

Intended integration shape (when we do it)

• A Windows-only ASR engine, added as a peer provider (per our rule: no unified abstraction layered over the local providers).
• Feed renderer PCM to SpeechAudioProvider via IPC; do not let the API open the microphone directly (would bypass our echo cancellation / device switching).
• ASR only — translation and TTS keep using existing engines.

Action items

• File a feature request on microsoft/windows-ai-electron asking for Microsoft.Windows.AI.Speech projection — filed as microsoft/windows-ai-electron#44 (2026-07-24)
• Watch for: Speech APIs promoted to a stable WinAppSDK release · multilingual support (zh/ja/ko/etc.) · Speech landing in the official Electron addon. Note: WinAppSDK stable majors are shipping roughly monthly (2.0 Apr → 2.3 Jul 2026), so stabilization may come quickly; language coverage is the real long pole.
• Optional early spike: C# minihost + winapp node add-electron-debug-identity on a Copilot+ PC to measure streaming latency/quality without touching the main app

References

• Speech Recognition with Windows AI APIs
• Microsoft.Windows.AI.Speech API reference
• WinAppSDK release channels · 2.x release notes
• microsoft/windows-ai-electron · winapp CLI Electron guides
• Use Windows on-device AI in your Electron app (devblog)
• Grant package identity by packaging with external location
• Build 2026 Windows developer blog

[jiangzhuo]

Filed the upstream feature request for JS projection of Microsoft.Windows.AI.Speech: microsoft/windows-ai-electron#44 — covers streaming events (Recognizing/Recognized), caller-pushed PCM via SpeechAudioProvider, and the experimental-channel question. Remaining watch signals unchanged (stable WinAppSDK release, multilingual support, addon support).