Add inputDeviceID passthrough to AudioStreamTranscriber - #503
Conversation
Backport of upstream PR argmaxinc#503 onto the v1.0.0 release, so OpenWhisp can route streaming/live transcription to a selected input device (e.g. a virtual device like BlackHole) instead of always the system default. Single-file change: threads an optional inputDeviceID through the init to AudioProcessor.startRecordingLive(inputDeviceID:), which already supports it. Kept isolated on top of v1.0.0 to avoid the macOS 26 CoreML/ANE changes on upstream main.
…op default-swap Replaces the system-default capture-and-restore workaround for the WhisperKit streaming path with the clean upstream mechanism: pin WhisperKit to OUR fork (initcore0/argmax-oss-swift) at v1.0.0 + a single-file backport of upstream PR argmaxinc/argmax-oss-swift#503, which threads an optional `inputDeviceID` through `AudioStreamTranscriber` to `AudioProcessor.startRecordingLive(inputDeviceID:)`. Why fork instead of pinning the upstream PR branch: that branch is diverged from v1.0.0 (8 ahead / 1 behind, ~60 files incl. the CoreML AudioEncoder/TextDecoder/ FeatureExtractor paths the 1.0.0 pin + GPU-encoder choice deliberately hold stable on macOS 26) and lives on a personal fork we don't control. Our fork branch (openwhisp/v1.0.0-input-device) is v1.0.0 + ONLY the 4-line device patch, pinned by immutable commit. When #503 lands in a tagged upstream release we bump and delete the fork. Engine changes: - WhisperKitBridge.makeStreamHandle takes `inputDeviceID: AudioDeviceID?` and passes it to AudioStreamTranscriber(inputDeviceID:). - WhisperKitStreamingEngine resolves the selected UID to an AudioDeviceID and passes it through — deleting the DefaultInputOverride engage/poll/restore dance and the isCapturing() helper. Unresolved (incl. a resolve→nil TOCTOU) stays a hard error, never a silent default capture. - DefaultInputOverride remains in AudioInputRouter, now used ONLY by the legacy AVAudioRecorder path (which genuinely can only capture the system default). Verified on live hardware against the actual forked WhisperKit lib (linked as the app links it): AudioProcessor.startRecordingLive(inputDeviceID:) started the engine bound to a non-default device (id 243) while the system default (id 110) stayed UNCHANGED — direct per-device capture, no swap. Fresh-clone resolve pulls the fork at the pinned commit and builds; ./build.sh green; swift test 435/435. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…op default-swap Replaces the system-default capture-and-restore workaround for the WhisperKit streaming path with the clean upstream mechanism: pin WhisperKit to OUR fork (initcore0/argmax-oss-swift) at v1.0.0 + a single-file backport of upstream PR argmaxinc/argmax-oss-swift#503, which threads an optional `inputDeviceID` through `AudioStreamTranscriber` to `AudioProcessor.startRecordingLive(inputDeviceID:)`. Why fork instead of pinning the upstream PR branch: that branch is diverged from v1.0.0 (8 ahead / 1 behind, ~60 files incl. the CoreML AudioEncoder/TextDecoder/ FeatureExtractor paths the 1.0.0 pin + GPU-encoder choice deliberately hold stable on macOS 26) and lives on a personal fork we don't control. Our fork branch (openwhisp/v1.0.0-input-device) is v1.0.0 + ONLY the 4-line device patch, pinned by immutable commit. When #503 lands in a tagged upstream release we bump and delete the fork. Engine changes: - WhisperKitBridge.makeStreamHandle takes `inputDeviceID: AudioDeviceID?` and passes it to AudioStreamTranscriber(inputDeviceID:). - WhisperKitStreamingEngine resolves the selected UID to an AudioDeviceID and passes it through — deleting the DefaultInputOverride engage/poll/restore dance and the isCapturing() helper. Unresolved (incl. a resolve→nil TOCTOU) stays a hard error, never a silent default capture. - DefaultInputOverride remains in AudioInputRouter, now used ONLY by the legacy AVAudioRecorder path (which genuinely can only capture the system default). Verified on live hardware against the actual forked WhisperKit lib (linked as the app links it): AudioProcessor.startRecordingLive(inputDeviceID:) started the engine bound to a non-default device (id 243) while the system default (id 110) stayed UNCHANGED — direct per-device capture, no swap. Fresh-clone resolve pulls the fork at the pinned commit and builds; ./build.sh green; swift test 435/435. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(audio): route dictation capture to the SELECTED input device Selecting a non-default microphone in Settings → Dictation was silently ignored: capture fell back to the system default input (the built-in mic). Confirmed via the E2E BlackHole smoke test, which only worked once it forced the SYSTEM default input to BlackHole. Root cause: the DEFAULT dictation path is the streaming seam (StreamingTranscriptionEngine — WhisperKit when a live preview is on, Apple Speech always), and NEITHER engine applied `microphoneID` at all. Only the legacy whisper.cpp `AudioRecorder` path applied it — and even that silently returned early when the device didn't resolve. Fix — route to the exact selected device on BOTH paths, and never silently fall back: - StreamingTranscriptionEngine gains `selectDevice(_:)`; AppState pins the session's `microphoneID` on the engine before start(). - AppleSpeechEngine retargets its own AVAudioEngine input node via setDeviceID (clean, no global mutation). - WhisperKitStreamingEngine: WhisperKit 1.0.0's AudioStreamTranscriber owns the mic and exposes NO per-engine device seam (and it's a pinned remote dep we can't patch), so it swaps the system default input around stream start and restores it once WhisperKit's engine has bound the device — a sub-second window, restored on every teardown/error path. - New shared AudioInputRouter centralizes CoreAudio resolution + default capture/restore + per-engine setDeviceID (removing the duplicated helpers in AudioRecorder). Pure decision logic lives in AudioInputRoutingPolicy (OpenWhispCore, unit-tested). - No silent fallback: a pinned device that can't be resolved surfaces a user-facing error on all paths (streaming + legacy). Empty `microphoneID` still means "follow the system default". Verified: `swift test` green (435 tests, incl. new AudioInputRoutingPolicy + StreamingTranscriptionEngine selectDevice tests); full app builds with the default WhisperKit backend (`./build.sh`); and a live-hardware harness exercising the real AudioInputRouter proved both mechanisms — per-engine setDeviceID binds the chosen device without touching the global default, and the default capture/restore switches to and back from a non-default device (BlackHole enumerated and resolvable). Follow-up (after PR #107 lands): update scripts/e2e-smoke.sh to set ONLY OpenWhisp's microphoneID to BlackHole's UID (leaving the SYSTEM default input as the built-in mic) — that is the regression test proving this fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(audio): route WhisperKit capture via inputDeviceID (our fork), drop default-swap Replaces the system-default capture-and-restore workaround for the WhisperKit streaming path with the clean upstream mechanism: pin WhisperKit to OUR fork (initcore0/argmax-oss-swift) at v1.0.0 + a single-file backport of upstream PR argmaxinc/argmax-oss-swift#503, which threads an optional `inputDeviceID` through `AudioStreamTranscriber` to `AudioProcessor.startRecordingLive(inputDeviceID:)`. Why fork instead of pinning the upstream PR branch: that branch is diverged from v1.0.0 (8 ahead / 1 behind, ~60 files incl. the CoreML AudioEncoder/TextDecoder/ FeatureExtractor paths the 1.0.0 pin + GPU-encoder choice deliberately hold stable on macOS 26) and lives on a personal fork we don't control. Our fork branch (openwhisp/v1.0.0-input-device) is v1.0.0 + ONLY the 4-line device patch, pinned by immutable commit. When #503 lands in a tagged upstream release we bump and delete the fork. Engine changes: - WhisperKitBridge.makeStreamHandle takes `inputDeviceID: AudioDeviceID?` and passes it to AudioStreamTranscriber(inputDeviceID:). - WhisperKitStreamingEngine resolves the selected UID to an AudioDeviceID and passes it through — deleting the DefaultInputOverride engage/poll/restore dance and the isCapturing() helper. Unresolved (incl. a resolve→nil TOCTOU) stays a hard error, never a silent default capture. - DefaultInputOverride remains in AudioInputRouter, now used ONLY by the legacy AVAudioRecorder path (which genuinely can only capture the system default). Verified on live hardware against the actual forked WhisperKit lib (linked as the app links it): AudioProcessor.startRecordingLive(inputDeviceID:) started the engine bound to a non-default device (id 243) while the system default (id 110) stayed UNCHANGED — direct per-device capture, no swap. Fresh-clone resolve pulls the fork at the pinned commit and builds; ./build.sh green; swift test 435/435. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(audio): error on failed legacy default-switch; fix stale swap-era comments The legacy AVAudioRecorder path could still silently record the system default when the resolve raced a disconnect or CoreAudio refused the default-input switch — contradicting the PR's no-silent-fallback rule. DefaultInputOverride.engage now distinguishes switched/alreadyDefault/failed and the caller surfaces an error on failure. Also update comments in AppState/AudioInputRouter that still described the WhisperKit path as a system-default swap; it now passes inputDeviceID into AudioStreamTranscriber via our fork (backport of upstream #503). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): prove mic-selection fix — pin microphoneID, leave system default input alone Now that #107 (E2E infra) is on main and the mic-routing fix captures the device named by `microphoneID` directly, update the BlackHole smoke test to be the regression proof: - Set OpenWhisp's `microphoneID` to BlackHole's UID (was: cleared to ""). - Point ONLY the default OUTPUT at BlackHole (for fixture playback). - Leave the system default INPUT untouched (the built-in mic). Under the old bug — capture following the system default — OpenWhisp would hear the room, not the fixture, and the assertion would fail. Passing now means capture followed the selected device. Also: drop the default-input save/restore (we no longer change it), and add a guard that warns when the running app's in-memory `microphoneID` is stale (it reads UserDefaults at launch, so a shell `defaults write` only lands on the next launch) — with clear relaunch guidance instead of silently testing a stale pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
Hi @ZachNagengast, just a gentle follow-up on this PR. I confirmed that the issue is still present in v1.0.0 and the current main branch. We have also validated this patch in a downstream macOS app: streaming transcription correctly captures from BlackHole while the system default input remains the MacBook microphone. Would you be open to reviewing this? I’m happy to add tests or adjust the API if needed. I also have a separate mitigation for the input-format/installTap crash, which I intentionally kept out of this focused PR. |
|
Hi @DTM-7-RX Could you merge main in here and validate against that with latest changes? Looks like a small contained change, if tests pass will be ok to merge as is from what I can tell. |
…ix-streaming-input-device
|
Thanks @ZachNagengast! Merged Validated on the merged branch (Swift 6.3.2, arm64 macOS):
No CI checks have run on the new commit yet — could you kick those off? |
There was a problem hiding this comment.
Pull request overview
Adds an optional inputDeviceID parameter to AudioStreamTranscriber so streaming transcription can record from a caller-selected input device (instead of always using the system default), by forwarding through to the existing AudioProcessor.startRecordingLive(inputDeviceID:) API.
Changes:
- Added
inputDeviceID: DeviceID? = niltoAudioStreamTranscriberinitializer and stored it as a private property. - Passed
inputDeviceIDthrough when starting live recording instartStreamTranscription().
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A failed start (e.g. an unavailable inputDeviceID) left state.isRecording set to true, so the guard at the top of startStreamTranscription() would silently return on every subsequent call and the transcriber could never be restarted. Reset the flag before rethrowing. Addresses Copilot review feedback on argmaxinc#503.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:45
- The new
inputDeviceIDcomment reads as if device selection is universally supported. In this codebaseDeviceIDis only meaningful on macOS (elsewhere it’s aStringand recording always uses the default input), so this comment may mislead future maintainers/callers reading the source.
/// Optional input device to capture from. nil = system default input.
/// Lets callers stream from a specific device (e.g. a virtual device) instead
/// of always using the system default. Passed through to startRecordingLive.
private let inputDeviceID: DeviceID?
|
Nice improvement, thanks for the contribution @DTM-7-RX! |
Summary
Adds an optional
inputDeviceIDparameter toAudioStreamTranscriberand forwards it to the existingAudioProcessor.startRecordingLive(inputDeviceID:)API.Today, streaming transcription always records from the system default input because
AudioStreamTranscribercallsstartRecordingLive()without passing a device. Lower-level recording already supports a specific input device, but the streaming wrapper does not expose it.This is additive and keeps existing behavior unchanged when
inputDeviceIDisnil.Why
This unblocks macOS apps that let users select an input device for streaming transcription, such as virtual or aggregate devices. For example, routing meeting audio into BlackHole works for non-streaming recording, but streaming transcription currently ignores the selected BlackHole device and captures the system default input instead.
I verified this fix in a local downstream app by routing audio into BlackHole and passing its device ID through
AudioStreamTranscriber; streaming transcription then captured the selected device as expected.Fixes #493.
Testing
git diff --check HEAD~1..HEADmake buildNote:
make buildrequired running outside the Codex sandbox because SwiftPM's ownsandbox-execfails withsandbox_apply: Operation not permittedinside the Codex sandbox. The build completed successfully outside that sandbox.