feat: Real-time streaming transcription

[jatinkrmalik]

Show transcription results in real-time as the user speaks, rather than waiting for recording to complete:

• Display partial transcription in the menu bar popover during recording
• Use WhisperKit's streaming capabilities
• Allow user to see and cancel mid-transcription

[m13v]

We implemented real-time streaming transcription with WhisperKit in our macOS app and here are the key things that made it work well.

The core pattern: use a ring buffer that accumulates audio, and run WhisperKit inference on overlapping windows. We use a 5-second sliding window with 1-second stride - meaning every second we run inference on the last 5 seconds of audio. The overlap is what makes partial results feel smooth rather than jumpy.

For the "show partial results in the popover" part, the trick is separating confirmed vs tentative segments. When a new inference result comes back, compare it to the previous one. If the first N tokens match, those are confirmed and you can commit them to the display. The trailing tokens that differ between runs are tentative - show them in a different style (lighter color or italic) so the user knows they might change.

WhisperKit specific tips:

• Set DecodingOptions.usePrefillPrompt = true and pass the last confirmed text as the prompt. This dramatically improves continuity between chunks.
• Use clipTimestamps to tell WhisperKit where in the audio buffer to focus. Without this, it tries to transcribe silence and hallucinates.
• Call transcribe() on a background queue but update the UI on main. The inference takes 100-300ms on Apple Silicon depending on model size, which is fast enough for real-time feel with a 1s stride.

For cancellation mid-transcription, keep a reference to the Task running inference and call cancel() on it. WhisperKit checks for cancellation between decode steps so it stops reasonably quickly.

[m13v]

Our streaming transcription implementation with WhisperKit on macOS: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

This handles the ring buffer, partial results, and confirmed segment tracking patterns described above.

[jatinkrmalik]

@m13v Are you an AI bot? Are you trying to farm backlinks to your project? I'm not sure.

[m13v]

nope, real person. I build fazm.ai which does on-device transcription with WhisperKit so streaming/chunking is literally my day job right now. saw your issue and the sliding window stuff we figured out seemed directly relevant. happy to chat more about it if you want

[kacperpaczos]

Not a request — a pointer that might shorten this.

The SDK VocaMac already depends on ships a streaming model. Package.swift pulls FluidAudio 0.15.5, and its catalogue splits transcription in two: batch, which is Parakeet TDT v3 and which its own docs call "near real-time", and streaming, which is Parakeet EOU 120M with end-of-utterance detection. The EOU model is English-only and much smaller than the batch ones. That is the same path VoiceInk uses for its live mode.

Context from the other direction: on the ggml side, streaming with Parakeet was attempted in whisper.cpp and closed (ggml-org/whisper.cpp#3900). The author of Parakeet support there said TDT's token durations do not work for streaming, and pointed at the dedicated EOU and nemotron models. So the split between "batch model" and "streaming model" is not a runtime detail — it is the model architecture, and FluidAudio already has both sides of it here.

I do not have a Mac to test on. This came out of comparing engine stacks across the Voca clients.