Add on-device live captions for remote call participants#157
Open
rexbron wants to merge 1 commit into
Open
Conversation
Pipes each subscribed remote audio track through Apple's SpeechAnalyzer
+ SpeechTranscriber (macOS 26+) and renders the transcribed text over
each participant's video. All recognition is on-device; the locale
model is downloaded via AssetInventory on first use.
Pipeline:
- New CaptionTranscriber implements LiveKit's AudioRenderer protocol.
It owns one SpeechTranscriber configured with the
.progressiveTranscription preset plus .volatileResults reporting for
a low-latency partial-result stream. Audio buffers are converted to
the analyzer's preferred format with a cached AVAudioConverter and
yielded into the analyzer's AsyncStream<AnalyzerInput>.
- CallViewModel.setCaptionsEnabled(_:) walks every currently-subscribed
remote audio track, attaches a transcriber, and starts pulling
results. Participants who join later are picked up via the
didSubscribeTrack delegate path; tracks gone via didUnsubscribeTrack
and departing participants via participantDidDisconnect tear their
transcriber down. All transcribers stop on disconnect.
- Per-participant caption state is a rolling buffer:
history (committed text) + volatile (in-progress utterance)
Volatile results replace volatile in place; final results append to
history with a space separator and clear volatile. History is
clamped from the head at 168 chars.
- Idle fade scales with reading rate: hold = clamp(chars/17, 5⁄6s, 7s)
per Netflix's English Timed Text Style Guide.
- Volatile partials are debounced into a single visual update inside a
180 ms window so the recognizer's rapid mid-utterance revisions don't
make on-screen words visibly "jump." Final results bypass the
debounce — they're authoritative and apply immediately.
UI:
- New control-bar toggle (`captions.bubble.fill`) on CallView's bottom
pill, between the camera button and end-call button.
- Caption strip is white .title3-semibold with a tight glyph shadow
on a near-opaque black rounded rect with a hairline edge — readable
against any video frame, sized to its widest line (capped at 540pt).
- Wrapping follows Netflix style: 42 chars/line, max 2 lines visible.
Word-aware wrapping; hard-break for tokens longer than 42.
- Renders on every ParticipantTile in the group grid AND below the
primary video in the 1:1 layout.
Speech content is never logged — diagnostics emit metadata only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds opt-in live captions to MatrixRTC calls. Each remote participant's
audio is transcribed on-device with Apple's
SpeechAnalyzer/SpeechTranscriber(macOS 26+) and the text is rendered over their video.Nothing leaves the device — the locale model is fetched via
AssetInventoryon first use.
How it works
CaptionTranscriber(new) implements LiveKit'sAudioRenderer. Itdrives one
SpeechTranscriber(.progressiveTranscription+.volatileResults) and feeds it audio converted to the analyzer'spreferred format via a cached
AVAudioConverter.CallViewModelattaches a transcriber per remote audio track, keyed by(identity, track sid). The full lifecycle is covered: already-present
participants on toggle (
setCaptionsEnabled), late joiners viadidSubscribeTrack, and teardown viadidUnsubscribeTrack,participantDidDisconnect, and calldisconnect(). Sid-keying makes aleave→rejoin that reuses an identity attach a fresh transcriber regardless
of the order subscribe/unsubscribe events arrive.
history+volatilebuffer. Volatilepartials are debounced (180 ms) so the recognizer's rapid mid-utterance
revisions don't make on-screen words jump; finals apply immediately. Idle
fade scales with reading rate (
clamp(chars/17, 5⁄6s, 7s)).UI
captions.bubble.fill) on the call control pill, between thecamera and end-call buttons.
.title3-semibold on a near-opaque rounded rect,≤2 lines, Netflix-style 42-char word-aware wrapping. Renders on each tile
in the group grid and under the primary video in 1:1.
Privacy
Speech content is never logged — diagnostics emit metadata only.
Requirements
macOS 26 (Tahoe)+ —
SpeechAnalyzer/SpeechTranscriberare 26-only.Testing
and leave→rejoin (captions return on the second join).
🤖 Generated with Claude Code