Realtime transcription - #705
Open
mxyyz wants to merge 7 commits into
Open
Conversation
Add a `StreamingTranscriptionProvider` abstraction alongside the existing one-shot `TranscriptionProvider`, so Meetily can transcribe against a self-hosted realtime ASR websocket (e.g. vLLM serving Voxtral Realtime) instead of one request per VAD-segmented chunk. - Add a persistent session worker fed by a continuous pre-VAD tap on the mixed audio pipeline, resampled 48k -> 16k mono - Add the Voxtral realtime websocket client (session.update handshake, base64 PCM16-LE frames, incremental deltas, final commit on stop) - Segment the provider's cumulative text into sentences and emit them on the existing `transcript-update` event, so history persistence and the UI are unchanged; a sentence keeps one `sequence_id` while it grows and finalizes under that id - Reconnect with backoff when the socket drops mid-recording, and report an actionable `transcription-error` once attempts are exhausted, so a dead endpoint can never leave a meeting silently untranscribed - Persist endpoint/model/API key via a `customTranscriptionConfig` column on `transcript_settings`, mirroring the customOpenAI summary provider - Add `api_test_custom_transcription_connection` plus start-of-recording validation so an unreachable endpoint fails loudly Dialects dispatch on a `protocol` discriminator, so another streaming backend is a new module plus a match arm rather than pipeline changes. Refs: Zackriya-Solutions#657 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Expose the streaming websocket backend in the UI as a "Custom Realtime (WebSocket)" transcription provider. - Add endpoint, model and optional API key fields with a Test Connection action; saving persists the config and activates the provider in one step - Add configService wrappers over the api_*_custom_transcription_* commands - Treat the provider as auto-detect-only in LanguageSelection, since the server detects language itself (mirrors the Parakeet handling) - Extract transcript merging into `lib/transcript-merge` and upsert live segments by `sequence_id` instead of dropping repeats, so a segment refines in place while it streams; providers that emit fresh ids per segment keep the previous append-only behavior - Throttle the transcript buffer flush rather than debouncing it, so rapid partials render continuously instead of stalling until a speech pause Refs: Zackriya-Solutions#657 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cover configuring the provider, running a vLLM Voxtral Realtime endpoint, what the pipeline does during a recording, and reconnect behavior. Document the provider contract separately from the voxtral-realtime dialect, so the wire protocol reads as one implementation rather than the abstraction. Refs: Zackriya-Solutions#657 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fills A realtime ASR server holds an entire session in one bounded context window. Once it fills, the session stops emitting transcripts for the rest of the recording — so a long meeting silently loses transcription partway through. Feed each server session a bounded budget of audio and, when it is spent, finalize that session and continue the recording on a fresh one. The rollover is invisible in the transcript: the outgoing session's final text is emitted before the new one takes over. - Add `maxSessionSeconds` to the persisted config (`None` = 300s default, `0` = never roll over) plus `session_limit_seconds()` - Add `api_detect_custom_transcription_limits`, which reads `max_model_len` from the endpoint's `/v1/models` and converts it to a session length using the audio/text token rates, keeping 15% headroom. Servers that publish no limit are reported as such rather than as an error - Reject a configured length under 30 seconds Refs: Zackriya-Solutions#657 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Surface the streaming session budget so users on a smaller context window can match it to their backend. - Add a "Max session length" field with a Detect button that reads the endpoint's context size and fills in a fitting value - Validate before saving: a whole number of seconds, at least 30, or 0 to never split - Explain in the field's help text why the setting exists, since the failure it prevents (transcription stopping mid-meeting) is otherwise mysterious Refs: Zackriya-Solutions#657 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cover why a realtime session is bounded at all — the server holds the whole session in one context window, and a meeting long enough to fill it stops being transcribed with no error — and how Max session length, Detect, and 0 behave. Note the token-rate assumptions behind Detect, since they are Voxtral-specific and a different encoder needs the value set by hand, and distinguish a planned rollover from a reconnect: one keeps the transcript intact, the other reacts to a dead socket and drops the outage window. Refs: Zackriya-Solutions#657 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Not a maintainer, but your code seems to not realtime, after 34 seconds, the text does not appear here. |
Author
|
Could you provide some more information?
|
|
macos, |
Author
|
This PR does not enable non-realtime models to be used for realtime transcription. As far as I know, Whisper Large v3 is not a realtime capable model |
|
ok, I don't know about llm. Sorry about it. |
Author
Speach-to-text / ASR models aren't LLMs either actually! llms are written text predictors, asr's analyze audio signals and turn them into text! :) (superficial knowledge here) |
|
how about this ? This was coding by codex, split speech into 8s and tranform into text. |
Author
Well, yeah, that makes transcription quicker but it is not realtime and not using a realtime model via a websocket, like THIS PR is. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a streaming transcription provider so Meetily can transcribe against a self-hosted realtime ASR websocket, with vLLM serving Voxtral Realtime as the worked example.
The existing
TranscriptionProvidertrait is request/response per VAD-segmented chunk and cannot express a persistent socket, so this adds a siblingStreamingTranscriptionProvidertrait driven by its own session worker and fed by a continuous pre-VAD tap on the audio pipeline. Results are emitted on the existingtranscript-updateevent, leaving transcript history, persistence and the UI unchanged. Dialects dispatch on aprotocolfield, so another streaming backend is a new module rather than pipeline changes.A dropped socket reconnects with backoff and reports an actionable error once attempts are exhausted. Since a realtime server holds an entire session in one bounded context, sessions also roll over on an audio budget so long meetings keep transcribing.
Related Issue
Fixes #657
Type of Change
Testing
Tested with:
cargo test --features vulkan --lib audio::transcription— 21 passedcargo test --features vulkan --lib— 207 passed, plus one pre-existing unrelated failure (device_detection::test_calculate_buffer_timeout_bluetooth, a float-precision assertion in a file this PR does not touch)bun test tests/— 18 passednpx tsc --noEmit— clean apart from the repo's existingbun:testtype-resolution errors undertests/lib/Also validated end-to-end against a self-hosted vLLM Voxtral-Realtime endpoint, including killing the server mid-recording to exercise reconnect, and confirming Whisper and Parakeet are unaffected.
Documentation
Checklist
Screenshots (if applicable)
Additional Notes
Commits are split backend / frontend / docs. The API key is stored as plaintext JSON in SQLite, consistent with the existing customOpenAI summary config. Developed on Linux and untested on macOS and Windows, though no platform-specific audio code is touched. Exercising system audio on Linux additionally needs the PulseAudio capture backend, submitted separately.