fix(inference): report Ink-2 speech onset when the provider detects it - #2182
fix(inference): report Ink-2 speech onset when the provider detects it#2182rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: e0d9ced The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| private processStartOfSpeech(): void { | ||
| if (this.speaking) return; | ||
| this.speaking = true; | ||
| this.queue.put({ type: SpeechEventType.START_OF_SPEECH }); | ||
| } |
There was a problem hiding this comment.
🟡 Speech-start notification can throw an error when the transcription stream is shutting down
The new speech-onset notification is pushed onto the closed output queue (this.queue.put(...) at agents/src/inference/stt.ts:1016) without first checking whether the stream has already been shut down, so an error can be raised during teardown instead of the event being quietly dropped.
Impact: If a provider onset message arrives while the transcription stream is closing, the receive loop can fail with an error instead of shutting down cleanly.
Missing closed-queue guard compared to the transcript path
processTranscript deliberately guards against this race: it returns early when this.queue.closed (agents/src/inference/stt.ts:1007, comment "Check if queue is closed to avoid race condition during disconnect") and wraps all queue.put calls in a try/catch that specifically tolerates Queue is closed (agents/src/inference/stt.ts:1082-1092). The new processStartOfSpeech is invoked directly from recv()'s switch (agents/src/inference/stt.ts:924-926) with neither guard, and AsyncIterableQueue.put throws Error('Queue is closed') when closed (agents/src/utils.ts:327-331). The queue is closed by close() (agents/src/stt/stt.ts:547) and after mainTask() completes (agents/src/stt/stt.ts:339), which can happen between recv()'s loop condition check and the put. The thrown error propagates out of the recv task and out of run(); unless the abort flag is already set, mainTask treats it as a non-API error and emits a non-recoverable error event (agents/src/stt/stt.ts:392-395). Additionally this.speaking is set to true before the put, so a failed put leaves the flag inconsistent.
| private processStartOfSpeech(): void { | |
| if (this.speaking) return; | |
| this.speaking = true; | |
| this.queue.put({ type: SpeechEventType.START_OF_SPEECH }); | |
| } | |
| private processStartOfSpeech(): void { | |
| // Check if queue is closed to avoid race condition during disconnect | |
| if (this.queue.closed) return; | |
| if (this.speaking) return; | |
| this.speaking = true; | |
| this.queue.put({ type: SpeechEventType.START_OF_SPEECH }); | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6630.
Summary
start_of_speechevent and emit STT onset immediatelyValidation
pnpm test agents(1515 passed, 5 skipped)pnpm build(40 packages)pnpm lint(40 packages; existing warnings only)pnpm format:checkpnpm --filter @livekit/agents typecheckcue-clivoice-mode run with livecartesia/ink-2: observed user-speaking before final transcription; assertion resolved in 2.257sSource diff coverage
livekit-agents/livekit/agents/inference/stt.py: adapted toagents/src/inference/stt.tsfor equivalent event dispatch, deduplicated onset handling, and transcript fallback. Also adapted toagents/src/inference/api_protos.tsbecause agents-js validates gateway messages with a typed Zod union; addingstart_of_speechthere is required to prevent the event from being discarded as unknown.tests/test_inference_stt_start_of_speech.py: adapted toagents/src/inference/stt.test.ts, preserving all six source test cases and assertions in Vitest form.Ported from livekit/agents#6630
Original PR description
Companion to livekit/agent-gateway#1134, which makes the gateway send the message; this one handles it.
Independent of, and complementary to, #6629: that PR stops
inference.STTclaiming word alignment for Ink-2 (which was falsely enabling adaptive interruption and disabling the fast VAD path). Together the three PRs cover both causes of the slow Ink-2 barge-in. No file overlap — the hunks are in different regions ofstt.py.The bug
Customers reported that Ink-2 barge-in through LiveKit Inference is "consistently over 1 sec late versus Silero VAD", while the same model via
cartesia.STTis fine. That reproduces, but the delay is in speech onset, not end-of-turn.Cartesia's Turns API emits
turn.startabout 130ms after speech begins. The gateway forwarded it as aninterim_transcriptwhose transcript is the empty string, sinceturn.startcarries no text._process_transcriptdrops exactly that:So the onset signal vanished, and
START_OF_SPEECHwas instead synthesized from the first interim that carried words — roughly a second in, once a word had been decoded. Interruption keys off onset, so barge-in inherited the full delay.cartesia.STTbypasses this translation and mapsturn.startstraight toSTART_OF_SPEECH, which is why the plugin looked fine and inference didn't.The fix
Handle a
start_of_speechmessage and report onset from it. The transcript-based path stays as a fallback for providers that send no onset event, so their behaviour is unchanged.Fixing this SDK-side alone isn't possible: inferring onset from an empty interim would fire for any provider that emits empty interims for unrelated reasons, which is why the gateway needs to name the event explicitly.
Measurements
Same 5-turn fixture, real Cartesia, locally built gateway, pre-fix binary vs both halves of the fix. Numbers are ms after Silero VAD's onset:
cartesia.STTMedian onset: 1004ms to 162ms, within 6ms of the plugin instead of ~880ms behind. Three of five turns previously exceeded a second, matching the report.
Unchanged, as expected for an onset-only fix:
end_of_speechlateness (368ms vs the plugin's 363ms) and preemptive-generation reuse (5/5 turns).Rollout
The two halves are independent. The gateway side can land first: this dispatch is an
if/elifchain with noelse, so an SDK without this change ignoresstart_of_speechsilently and keeps today's behaviour, having already discarded the empty interim it replaces.Test plan
tests/test_inference_stt_start_of_speech.py— 6 tests: onset without waiting for a transcript, no duplicate onset, the fallback for providers with no onset event, that an empty interim alone never reports onset, and that onset resets across turns