fix(inference): don't claim word alignment for models that send none - #2181
fix(inference): don't claim word alignment for models that send none#2181rosetta-livekit-bot[bot] wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 0af2199 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const WORD_ALIGNED_MODELS = new Set([ | ||
| 'deepgram/nova-3', | ||
| 'deepgram/nova-3-medical', | ||
| 'deepgram/nova-2', | ||
| 'deepgram/nova-2-medical', | ||
| 'deepgram/nova-2-conversationalai', | ||
| 'deepgram/nova-2-phonecall', | ||
| 'deepgram/flux-general', | ||
| 'deepgram/flux-general-en', | ||
| 'deepgram/flux-general-multi', | ||
| 'cartesia/ink-whisper', | ||
| 'assemblyai/universal-streaming', | ||
| 'assemblyai/universal-streaming-multilingual', | ||
| 'assemblyai/u3-rt-pro', | ||
| 'assemblyai/universal-3-5-pro', | ||
| 'elevenlabs/scribe_v2_realtime', | ||
| 'xai/stt-1', | ||
| 'speechmatics/enhanced', | ||
| 'speechmatics/standard', | ||
| ]); | ||
|
|
||
| /** | ||
| * Word-level alignment, which adaptive interruption relies on to gatekeep transcripts. | ||
| * | ||
| * Returns false when the model sends no word timings, and for `auto`, where the provider is | ||
| * picked server-side per language and cannot be known here. Claiming alignment we do not have | ||
| * is the costlier error because it disables the fast VAD barge-in path. | ||
| */ | ||
| function alignedTranscriptForModel(model: string | undefined): 'word' | false { | ||
| return model && WORD_ALIGNED_MODELS.has(model) ? 'word' : false; | ||
| } |
There was a problem hiding this comment.
🟡 Word timing support is dropped for provider shorthand model names that do supply word timings
Speech models named by provider shorthand such as deepgram, assemblyai or cartesia are treated as unverified (WORD_ALIGNED_MODELS lookup at agents/src/inference/stt.ts:314-316) even though they resolve to providers that do send word timings, so those setups silently lose the faster, transcript-aware interruption behaviour.
Impact: Agents configured with the common short model names keep working but fall back to the less precise interruption path, a behaviour regression the change did not intend.
Allowlist only contains fully-qualified provider/model identifiers
The new allowlist (agents/src/inference/stt.ts:286-305) enumerates only fully-qualified names (deepgram/nova-3, assemblyai/universal-streaming, cartesia/ink-whisper, ...). Provider-only strings are valid model values in this module — parseSTTModelString tests cover 'assemblyai:es', 'cartesia:zh' and the shared test helper makeStt defaults to model: 'deepgram' (agents/src/inference/stt.test.ts:27), and connectWs forwards them verbatim as params.model. Because alignedTranscriptForModel requires an exact set membership, 'deepgram' yields false, whereas the upstream Python change only denies known word-less providers (cartesia/ink-2, inworld/, google/) and auto, leaving deepgram claiming 'word'. The same gap applies to any fully-qualified Deepgram/AssemblyAI/ElevenLabs model variant that exists in the model unions but was not listed.
Prompt for agents
In agents/src/inference/stt.ts the new WORD_ALIGNED_MODELS allowlist only contains fully-qualified provider/model identifiers, but provider-shorthand model strings ('deepgram', 'assemblyai', 'cartesia', 'elevenlabs', 'speechmatics', 'xai') are accepted elsewhere in this file (parseSTTModelString tests, makeStt default, connectWs forwards params.model as-is). alignedTranscriptForModel therefore reports false for shorthand names whose underlying providers do forward word timings, which disables adaptive interruption for a very common configuration and diverges from the upstream Python behaviour (which only denies cartesia/ink-2, inworld/, google/, and auto). Consider either adding the provider shorthand names known to send words to the allowlist, or matching by provider prefix with an explicit denylist for the word-less providers, and make sure every fully-qualified model in the Deepgram/AssemblyAI/ElevenLabs/xai/Speechmatics unions is covered consistently.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6629 and livekit/agents#6633.
Summary
updateOptionschanges the primary modelSource diff coverage
livekit/agents#6629
livekit-agents/livekit/agents/inference/stt.py: ported toagents/src/inference/stt.ts. The verified word-aligned model allowlist, conservative unknown/autofallback, constructor capability, and model-update recomputation are preserved.tests/fake_turn_stt.py: adapted using the target's existing co-located STT and voice test internals rather than adding an unused scripted scheduler. The source tests only inspect construction-time interruption state, so existing target fakes/internal test objects provide the same coverage without new production infrastructure.tests/test_inference_stt_aligned_transcript_claim.py: adapted acrossagents/src/inference/stt.test.ts,agents/src/voice/audio_recognition_interruption.test.ts,agents/src/voice/agent.test.ts, andplugins/cartesia/src/stt.test.ts. This covers plugin parity, verified/unknown models, model updates, gateway payload shape, transcript gate behavior before/after reconnect, adaptive selection, plain production behavior, explicit VAD mode, and session VAD retention.livekit/agents#6633
livekit-agents/livekit/agents/inference/stt.py: ported toagents/src/inference/stt.ts. Fallbacks are normalized before capability construction; the primary and every fallback must be verified as word-aligned, including after primary-model updates.tests/test_inference_stt_aligned_transcript_claim.py: adapted intoagents/src/inference/stt.test.ts. The aligned, unaligned, and unknown fallback cases and the model-update-with-unaligned-fallback case are ported directly to Vitest.Validation
pnpm buildpnpm test agents(1,526 passed, 5 skipped)pnpm test plugins/cartesia(7 passed, 2 skipped)pnpm lint(passes with existing warnings)pnpm format:checkpnpm --filter @livekit/agents typecheckcue-clivoice-mode run for Ink-2 primary: assertedalignedTranscript=false,adaptiveEnabled=false, andvadAvailable=truecue-clivoice-mode run for Deepgram primary with Ink-2 fallback: assertedalignedTranscript=false,adaptiveEnabled=false, andvadAvailable=true