Skip to content

Add inputDeviceID passthrough to AudioStreamTranscriber - #503

Merged
ZachNagengast merged 3 commits into
argmaxinc:mainfrom
DTM-7-RX:fix-streaming-input-device
Aug 13, 2026
Merged

Add inputDeviceID passthrough to AudioStreamTranscriber#503
ZachNagengast merged 3 commits into
argmaxinc:mainfrom
DTM-7-RX:fix-streaming-input-device

Conversation

@DTM-7-RX

@DTM-7-RX DTM-7-RX commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an optional inputDeviceID parameter to AudioStreamTranscriber and forwards it to the existing AudioProcessor.startRecordingLive(inputDeviceID:) API.

Today, streaming transcription always records from the system default input because AudioStreamTranscriber calls startRecordingLive() without passing a device. Lower-level recording already supports a specific input device, but the streaming wrapper does not expose it.

This is additive and keeps existing behavior unchanged when inputDeviceID is nil.

Why

This unblocks macOS apps that let users select an input device for streaming transcription, such as virtual or aggregate devices. For example, routing meeting audio into BlackHole works for non-streaming recording, but streaming transcription currently ignores the selected BlackHole device and captures the system default input instead.

I verified this fix in a local downstream app by routing audio into BlackHole and passing its device ID through AudioStreamTranscriber; streaming transcription then captured the selected device as expected.

Fixes #493.

Testing

  • git diff --check HEAD~1..HEAD
  • make build

Note: make build required running outside the Codex sandbox because SwiftPM's own sandbox-exec fails with sandbox_apply: Operation not permitted inside the Codex sandbox. The build completed successfully outside that sandbox.

initcore0 added a commit to initcore0/argmax-oss-swift that referenced this pull request Jul 9, 2026
Backport of upstream PR argmaxinc#503 onto the v1.0.0 release,
so OpenWhisp can route streaming/live transcription to a selected input device
(e.g. a virtual device like BlackHole) instead of always the system default.

Single-file change: threads an optional inputDeviceID through the init to
AudioProcessor.startRecordingLive(inputDeviceID:), which already supports it.
Kept isolated on top of v1.0.0 to avoid the macOS 26 CoreML/ANE changes on
upstream main.
initcore0 added a commit to initcore0/openwhisp that referenced this pull request Jul 9, 2026
…op default-swap

Replaces the system-default capture-and-restore workaround for the WhisperKit
streaming path with the clean upstream mechanism: pin WhisperKit to OUR fork
(initcore0/argmax-oss-swift) at v1.0.0 + a single-file backport of upstream PR
argmaxinc/argmax-oss-swift#503, which threads an optional `inputDeviceID` through
`AudioStreamTranscriber` to `AudioProcessor.startRecordingLive(inputDeviceID:)`.

Why fork instead of pinning the upstream PR branch: that branch is diverged from
v1.0.0 (8 ahead / 1 behind, ~60 files incl. the CoreML AudioEncoder/TextDecoder/
FeatureExtractor paths the 1.0.0 pin + GPU-encoder choice deliberately hold
stable on macOS 26) and lives on a personal fork we don't control. Our fork
branch (openwhisp/v1.0.0-input-device) is v1.0.0 + ONLY the 4-line device patch,
pinned by immutable commit. When #503 lands in a tagged upstream release we bump
and delete the fork.

Engine changes:
- WhisperKitBridge.makeStreamHandle takes `inputDeviceID: AudioDeviceID?` and
  passes it to AudioStreamTranscriber(inputDeviceID:).
- WhisperKitStreamingEngine resolves the selected UID to an AudioDeviceID and
  passes it through — deleting the DefaultInputOverride engage/poll/restore dance
  and the isCapturing() helper. Unresolved (incl. a resolve→nil TOCTOU) stays a
  hard error, never a silent default capture.
- DefaultInputOverride remains in AudioInputRouter, now used ONLY by the legacy
  AVAudioRecorder path (which genuinely can only capture the system default).

Verified on live hardware against the actual forked WhisperKit lib (linked as the
app links it): AudioProcessor.startRecordingLive(inputDeviceID:) started the
engine bound to a non-default device (id 243) while the system default (id 110)
stayed UNCHANGED — direct per-device capture, no swap. Fresh-clone resolve pulls
the fork at the pinned commit and builds; ./build.sh green; swift test 435/435.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
initcore0 added a commit to initcore0/openwhisp that referenced this pull request Jul 9, 2026
…op default-swap

Replaces the system-default capture-and-restore workaround for the WhisperKit
streaming path with the clean upstream mechanism: pin WhisperKit to OUR fork
(initcore0/argmax-oss-swift) at v1.0.0 + a single-file backport of upstream PR
argmaxinc/argmax-oss-swift#503, which threads an optional `inputDeviceID` through
`AudioStreamTranscriber` to `AudioProcessor.startRecordingLive(inputDeviceID:)`.

Why fork instead of pinning the upstream PR branch: that branch is diverged from
v1.0.0 (8 ahead / 1 behind, ~60 files incl. the CoreML AudioEncoder/TextDecoder/
FeatureExtractor paths the 1.0.0 pin + GPU-encoder choice deliberately hold
stable on macOS 26) and lives on a personal fork we don't control. Our fork
branch (openwhisp/v1.0.0-input-device) is v1.0.0 + ONLY the 4-line device patch,
pinned by immutable commit. When #503 lands in a tagged upstream release we bump
and delete the fork.

Engine changes:
- WhisperKitBridge.makeStreamHandle takes `inputDeviceID: AudioDeviceID?` and
  passes it to AudioStreamTranscriber(inputDeviceID:).
- WhisperKitStreamingEngine resolves the selected UID to an AudioDeviceID and
  passes it through — deleting the DefaultInputOverride engage/poll/restore dance
  and the isCapturing() helper. Unresolved (incl. a resolve→nil TOCTOU) stays a
  hard error, never a silent default capture.
- DefaultInputOverride remains in AudioInputRouter, now used ONLY by the legacy
  AVAudioRecorder path (which genuinely can only capture the system default).

Verified on live hardware against the actual forked WhisperKit lib (linked as the
app links it): AudioProcessor.startRecordingLive(inputDeviceID:) started the
engine bound to a non-default device (id 243) while the system default (id 110)
stayed UNCHANGED — direct per-device capture, no swap. Fresh-clone resolve pulls
the fork at the pinned commit and builds; ./build.sh green; swift test 435/435.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
initcore0 added a commit to initcore0/openwhisp that referenced this pull request Jul 9, 2026
* fix(audio): route dictation capture to the SELECTED input device

Selecting a non-default microphone in Settings → Dictation was silently
ignored: capture fell back to the system default input (the built-in mic).
Confirmed via the E2E BlackHole smoke test, which only worked once it forced
the SYSTEM default input to BlackHole.

Root cause: the DEFAULT dictation path is the streaming seam
(StreamingTranscriptionEngine — WhisperKit when a live preview is on, Apple
Speech always), and NEITHER engine applied `microphoneID` at all. Only the
legacy whisper.cpp `AudioRecorder` path applied it — and even that silently
returned early when the device didn't resolve.

Fix — route to the exact selected device on BOTH paths, and never silently
fall back:

- StreamingTranscriptionEngine gains `selectDevice(_:)`; AppState pins the
  session's `microphoneID` on the engine before start().
- AppleSpeechEngine retargets its own AVAudioEngine input node via setDeviceID
  (clean, no global mutation).
- WhisperKitStreamingEngine: WhisperKit 1.0.0's AudioStreamTranscriber owns the
  mic and exposes NO per-engine device seam (and it's a pinned remote dep we
  can't patch), so it swaps the system default input around stream start and
  restores it once WhisperKit's engine has bound the device — a sub-second
  window, restored on every teardown/error path.
- New shared AudioInputRouter centralizes CoreAudio resolution + default
  capture/restore + per-engine setDeviceID (removing the duplicated helpers in
  AudioRecorder). Pure decision logic lives in AudioInputRoutingPolicy
  (OpenWhispCore, unit-tested).
- No silent fallback: a pinned device that can't be resolved surfaces a
  user-facing error on all paths (streaming + legacy). Empty `microphoneID`
  still means "follow the system default".

Verified: `swift test` green (435 tests, incl. new AudioInputRoutingPolicy +
StreamingTranscriptionEngine selectDevice tests); full app builds with the
default WhisperKit backend (`./build.sh`); and a live-hardware harness exercising
the real AudioInputRouter proved both mechanisms — per-engine setDeviceID binds
the chosen device without touching the global default, and the default
capture/restore switches to and back from a non-default device (BlackHole
enumerated and resolvable).

Follow-up (after PR #107 lands): update scripts/e2e-smoke.sh to set ONLY
OpenWhisp's microphoneID to BlackHole's UID (leaving the SYSTEM default input as
the built-in mic) — that is the regression test proving this fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(audio): route WhisperKit capture via inputDeviceID (our fork), drop default-swap

Replaces the system-default capture-and-restore workaround for the WhisperKit
streaming path with the clean upstream mechanism: pin WhisperKit to OUR fork
(initcore0/argmax-oss-swift) at v1.0.0 + a single-file backport of upstream PR
argmaxinc/argmax-oss-swift#503, which threads an optional `inputDeviceID` through
`AudioStreamTranscriber` to `AudioProcessor.startRecordingLive(inputDeviceID:)`.

Why fork instead of pinning the upstream PR branch: that branch is diverged from
v1.0.0 (8 ahead / 1 behind, ~60 files incl. the CoreML AudioEncoder/TextDecoder/
FeatureExtractor paths the 1.0.0 pin + GPU-encoder choice deliberately hold
stable on macOS 26) and lives on a personal fork we don't control. Our fork
branch (openwhisp/v1.0.0-input-device) is v1.0.0 + ONLY the 4-line device patch,
pinned by immutable commit. When #503 lands in a tagged upstream release we bump
and delete the fork.

Engine changes:
- WhisperKitBridge.makeStreamHandle takes `inputDeviceID: AudioDeviceID?` and
  passes it to AudioStreamTranscriber(inputDeviceID:).
- WhisperKitStreamingEngine resolves the selected UID to an AudioDeviceID and
  passes it through — deleting the DefaultInputOverride engage/poll/restore dance
  and the isCapturing() helper. Unresolved (incl. a resolve→nil TOCTOU) stays a
  hard error, never a silent default capture.
- DefaultInputOverride remains in AudioInputRouter, now used ONLY by the legacy
  AVAudioRecorder path (which genuinely can only capture the system default).

Verified on live hardware against the actual forked WhisperKit lib (linked as the
app links it): AudioProcessor.startRecordingLive(inputDeviceID:) started the
engine bound to a non-default device (id 243) while the system default (id 110)
stayed UNCHANGED — direct per-device capture, no swap. Fresh-clone resolve pulls
the fork at the pinned commit and builds; ./build.sh green; swift test 435/435.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(audio): error on failed legacy default-switch; fix stale swap-era comments

The legacy AVAudioRecorder path could still silently record the system
default when the resolve raced a disconnect or CoreAudio refused the
default-input switch — contradicting the PR's no-silent-fallback rule.
DefaultInputOverride.engage now distinguishes switched/alreadyDefault/failed
and the caller surfaces an error on failure.

Also update comments in AppState/AudioInputRouter that still described the
WhisperKit path as a system-default swap; it now passes inputDeviceID into
AudioStreamTranscriber via our fork (backport of upstream #503).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): prove mic-selection fix — pin microphoneID, leave system default input alone

Now that #107 (E2E infra) is on main and the mic-routing fix captures the device
named by `microphoneID` directly, update the BlackHole smoke test to be the
regression proof:

- Set OpenWhisp's `microphoneID` to BlackHole's UID (was: cleared to "").
- Point ONLY the default OUTPUT at BlackHole (for fixture playback).
- Leave the system default INPUT untouched (the built-in mic). Under the old bug
  — capture following the system default — OpenWhisp would hear the room, not the
  fixture, and the assertion would fail. Passing now means capture followed the
  selected device.

Also: drop the default-input save/restore (we no longer change it), and add a
guard that warns when the running app's in-memory `microphoneID` is stale (it
reads UserDefaults at launch, so a shell `defaults write` only lands on the next
launch) — with clear relaunch guidance instead of silently testing a stale pin.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@DTM-7-RX

Copy link
Copy Markdown
Contributor Author

Hi @ZachNagengast, just a gentle follow-up on this PR.

I confirmed that the issue is still present in v1.0.0 and the current main branch. We have also validated this patch in a downstream macOS app: streaming transcription correctly captures from BlackHole while the system default input remains the MacBook microphone.

Would you be open to reviewing this? I’m happy to add tests or adjust the API if needed. I also have a separate mitigation for the input-format/installTap crash, which I intentionally kept out of this focused PR.

@ZachNagengast

Copy link
Copy Markdown
Contributor

Hi @DTM-7-RX Could you merge main in here and validate against that with latest changes? Looks like a small contained change, if tests pass will be ok to merge as is from what I can tell.

@DTM-7-RX

Copy link
Copy Markdown
Contributor Author

Thanks @ZachNagengast! Merged main in — no conflicts.

Validated on the merged branch (Swift 6.3.2, arm64 macOS):

  • swift build — clean
  • AudioProcessorTests — 12 tests, 0 failures
  • WhisperKitTests.UnitTests — 131 tests, 1 skipped, 0 failures

No CI checks have run on the new commit yet — could you kick those off?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an optional inputDeviceID parameter to AudioStreamTranscriber so streaming transcription can record from a caller-selected input device (instead of always using the system default), by forwarding through to the existing AudioProcessor.startRecordingLive(inputDeviceID:) API.

Changes:

  • Added inputDeviceID: DeviceID? = nil to AudioStreamTranscriber initializer and stored it as a private property.
  • Passed inputDeviceID through when starting live recording in startStreamTranscription().

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift
A failed start (e.g. an unavailable inputDeviceID) left state.isRecording
set to true, so the guard at the top of startStreamTranscription() would
silently return on every subsequent call and the transcriber could never
be restarted. Reset the flag before rethrowing.

Addresses Copilot review feedback on argmaxinc#503.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:45

  • The new inputDeviceID comment reads as if device selection is universally supported. In this codebase DeviceID is only meaningful on macOS (elsewhere it’s a String and recording always uses the default input), so this comment may mislead future maintainers/callers reading the source.
    /// Optional input device to capture from. nil = system default input.
    /// Lets callers stream from a specific device (e.g. a virtual device) instead
    /// of always using the system default. Passed through to startRecordingLive.
    private let inputDeviceID: DeviceID?

@ZachNagengast
ZachNagengast merged commit ea872ff into argmaxinc:main Aug 13, 2026
21 of 25 checks passed
@ZachNagengast

Copy link
Copy Markdown
Contributor

Nice improvement, thanks for the contribution @DTM-7-RX!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming transcription can't select an input device — AudioStreamTranscriber doesn't forward inputDeviceID (macOS)

3 participants