Live (resumable) streaming transcription#147
Open
AlexCherrypi wants to merge 16 commits into
Open
Conversation
Point the whisper.cpp submodule at the fork branch that adds the resumable streaming API (whisper_append_audio / whisper_full_resumable / whisper_resumable_reset + configurable mel normalization). - WhisperFullParams: expose mel_norm_mode / half_life / max_drop - Whis.swift: bind resumableReset / appendAudio / fullResumable - StreamingWhisperEngine: AVAudioEngine tap -> append -> resumable loop, emitting committed segments live; finalize() flushes the trailing window on stop. Mel norm mode configurable (defaults to WINDOW for the live mic path).
Behind a new "Live Streaming (Whisper)" setting (default off, whisper engine only). When enabled, StreamingWhisperEngine captures in parallel to the file recorder: the WAV is still written (playback / re-transcribe intact), and on stop the live (resumable) text is used instead of a fresh file-based pass -> near-instant result for long recordings. - AppPreferences: liveStreamingEnabled flag - Settings: SettingsViewModel mirror + toggle in Output Options - IndicatorViewModel: start/stop/cancel the streaming engine, expose liveText, persist() saves the WAV + inserts the streamed text; preload the streaming model so the lead-in is not dropped
Recording must start immediately; the model load must not drop the lead-in (the streamed text is what gets saved). StreamingWhisperEngine now installs the tap and starts the AVAudioEngine right away, and queues the model load as the first item on its serial whisper queue. Captured audio appended by the tap is processed only after the load completes, so nothing is lost. stop() still flushes only the trailing window.
Allow starting a new recording while a previous one is still finalizing/transcribing (live mode only; the file path still blocks since it shares one context). The shared StreamingWhisperEngine's single serial queue guarantees recordings are finalized — and therefore inserted — strictly in the order they were made. - StreamingWhisperEngine is now a shared singleton (model loaded once); per-recording state resets run on the serial queue, and stop() captures the final text inside the sync block so the next recording's reset can't clear it first. Removed the unused live-text callback. - IndicatorViewModel uses the shared engine; live mode skips the busy block; finalize/insert run via the engine queue (ordered). Insertion honors the existing autoPaste/autoCopy prefs (cursor vs clipboard). - didFinishDecoding(for:) is VM-aware: a stale session finishing in the background no longer hides a newer recording's indicator (overshadow); the stale VM is just cleaned up.
When recording #2 starts while #1 is still finalizing on the shared engine, setting self.settings on the caller thread raced with #1's finalize reading it via makeParams() on the whisper queue. Move the per-recording settings assignment into the serial queue block so #1 reads its own settings before #2 overwrites them.
If the streaming engine never actually ran for a recording (no model selected, mic/AVAudioEngine start error, or model load failure), the WAV was saved but no text was produced or inserted -> silent data loss. stop() now returns a FinalizeResult distinguishing a real transcription (even an empty "no speech" one, which must NOT trigger a fallback) from .unavailable (infrastructure failure). On .unavailable the indicator falls back to transcriptionService.transcribeAudio(url:) so the recording is never lost.
The streaming AVAudioEngine tap did not react to the mic being unplugged or the input device/format changing, so audio would silently stop. Now the engine observes AVAudioEngineConfigurationChange and rebuilds the tap + converter for the new input format (keeping the resumable state so the stream continues), restarting the engine if it stopped. The observer is removed on stop()/cancel(). Tap setup is extracted into configureCapture().
isRunning / isCancelled are touched from the caller, the audio render thread and main (config-change handler). Guard them with an NSLock (matching WhisperEngine's pattern) for well-defined cross-thread access. Also verified (no change needed): stream segment timestamps are absolute over the whole stream (t1 = seek + ..., seek is the absolute resumable position), so showTimestamps is correct in live mode.
Fixes the two verified high-severity concurrency bugs in the live path: - Guard every AVAudioEngine graph mutation (start/stop/tap/converter/observer) with a new engineLock so an overlapping start (recording B) and stop (recording A) can't mutate the shared engine concurrently — CoreAudio is not safe for concurrent start/stop/tap changes and could crash or wedge capture. - Order A's finalize strictly before B's reset by enqueuing both under engineLock; the lock is released before the (long) decode, which now runs via async + semaphore, so B's capture is not delayed by A's finalize. Prevents a recording's transcription from being lost when recordings overlap. - Roll back the tap/observer if start() throws after partial setup. - Correct misleading comments asserting a "separate context" / guaranteed FIFO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FluidAudio 0.11.0 fails to compile under Xcode 26.5 (Swift 6 "sending" data-race errors inside FluidAudio itself). Bump to 0.15.2 (within the project's existing <1.0.0 constraint) and adapt to its API changes: - AsrManager.initialize(models:) -> loadModels(_:) (FluidAudioEngine, Settings, OnboardingView) - transcribe(url) now requires an explicit decoder state: pass a fresh TdtDecoderState (correct for one-shot file transcription). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the libwhisper/whisper.cpp submodule from 51f67d4 to 109efaae so the app links the hardened resumable API (finalize guard, doc/edge-case fixes). API is unchanged, so the app builds identically; this just folds in the cpp review fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gging - ContentViewModel (main-window record button) now streams transcription during recording via the shared StreamingWhisperEngine, like the hotkey indicator, and falls back to a file pass if streaming can't run. Re-transcribing existing recordings stays file-based (no live capture to stream). - Gate whisper realtime segment logging in the streaming path on the app's debug setting, so the live path is as verbose as the file path when logging is on, and quiet otherwise. - Log the live transcription result (parity with the file path's result log). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The realtime segment log printed a [t0 --> t1] header even with timestamps disabled (no_timestamps), producing a bogus constant-negative time because the header was built from an unset timestamp token. Now, when debug logging is on, timestamps are computed so the log shows valid times; otherwise the header is printed only when the user enabled "show timestamps". The stored transcription text is unchanged (its formatting still follows settings.showTimestamps). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sh fix - Line-buffer stdout at app launch (setvbuf _IOLBF) so log output is flushed per line when redirected to a file; block buffering otherwise withheld output until the buffer filled (and abort() on quit dropped it), making live/long logs look stalled or truncated even though transcription was complete. - Bump libwhisper to bf2c1277, which downgrades the ggml-metal residency-set teardown GGML_ASSERT to a warning so the app no longer crashes on quit (macOS 15+). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The resumable/streaming whisper API (plus the ggml-metal quit-crash workaround) is now merged into AlexCherrypi/whisper.cpp master, so track `master` instead of the feature branch. The pinned commit is unchanged (bf2c1277, == master tip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AlexCherrypi
added a commit
to AlexCherrypi/OpenSuperWhisper
that referenced
this pull request
Jun 9, 2026
Integrating the live-streaming and custom-dictionary features surfaced a composition gap: StreamingWhisperEngine never applied the custom dictionary, so with both enabled live dictation got neither prompt-biasing nor word replacements (the file path via WhisperEngine/FluidAudioEngine did). Mirror the file path, gated by settings.shouldApplyCustomDictionary: - weave CustomDictionary.promptBoost into the live initialPrompt - run CustomDictionary.apply on the final live text only (in cleaned(), never on partial segments), after Asian autocorrect Fork-master integration only; the upstream chunking branch (PR Starmel#147) stays free of the dictionary dependency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root-cause fix (app side) for the macOS 15+ quit-time abort: the whisper contexts live in singletons (StreamingWhisperEngine.shared and the engine held by TranscriptionService.shared), which are never deinit'd at process exit, so whisper_free never runs — leaving the model's Metal buffers and their residency sets registered when ggml's device static destructor tears the GPU down. Add a graceful applicationShouldTerminate that returns .terminateLater and: - waits for the in-flight transcription queue to drain (TranscriptionQueue.waitUntilIdle), - frees the live engine's context on its serial whisper queue (StreamingWhisperEngine.shutdown), - releases the file engine so its context deallocates (TranscriptionService.releaseEngine), then replies to allow termination. Freeing the contexts runs whisper_free, which drains the residency sets before device teardown — so OSW shuts down cleanly even against an unpatched ggml. (libwhisper still carries the defensive ggml-metal fix.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Live, "pipelined" transcription: audio is transcribed while recording via whisper's resumable API, so when you stop only the trailing <30s window remains to decode (near-instant), at full large-model quality.
StreamingWhisperEngine: ownAVAudioEnginetap → 16k mono → serial whisper queue drivingwhisper_append_audio+whisper_full_resumable.whisper.cpp dependency — works today via my fork
This uses a resumable streaming API I added to whisper.cpp, submitted upstream as ggml-org/whisper.cpp#3869. In the meantime it works as-is: the
libwhisper/whisper.cppsubmodule points at my fork (AlexCherrypi/whisper.cpp, branchmaster), which already has the API, so this branch builds and runs right now. Once #3869 lands upstream, the submodule can simply be repointed to the canonical repo — no code changes needed.Also included
sendingchecks); includes the smallAsrManager.loadModels/transcribe(_:decoderState:)API migration.applicationShouldTerminate→ release the live + file engines). That runswhisper_free, which releases the Metal backend's buffers and drains their macOS 15+ residency sets before the GPU device is torn down — avoiding an abort-on-quit. Complements the ggml-metal fix submitted upstream as metal : wind down leftover residency sets at teardown instead of aborting ggml-org/whisper.cpp#3870 and metal : wind down leftover residency sets at teardown instead of aborting ggml-org/llama.cpp#24368, so OSW shuts down cleanly even against an unpatched ggml.Happy to adjust — feedback welcome.