feat(voice): auto-disable realtime server-side turn detection - #2158
feat(voice): auto-disable realtime server-side turn detection#2158rosetta-livekit-bot[bot] wants to merge 2 commits into
Conversation
- restore withAzure's destructured signature; the pre-existing code already reassigned its params, so only the default needed moving out - name the session options type instead of repeating the object literal - drop xAI's session() override, a no-op the canDisableTurnDetection gate already covers - use a template literal for the runtime-change warning
🦋 Changeset detectedLatest commit: 6d36985 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (hasTurnDetection) { | ||
| recognitionOptions.turnDetection = turnDetection; | ||
| recognitionOptions.turnDetection = this._resolvedTurnDetection; | ||
| } |
There was a problem hiding this comment.
🔴 Turning off a running session's turn-detection setting no longer takes effect in the speech recognizer
When turn detection is cleared at runtime, the recognizer is handed an "unchanged" value instead of an explicit reset (recognitionOptions.turnDetection = this._resolvedTurnDetection at agents/src/voice/agent_activity.ts:1123), so the previously configured mode (e.g. manual) keeps driving speech recognition.
Impact: After switching turn detection back to automatic, the agent can keep behaving as if the old mode (such as manual push-to-talk) were still active and stop replying on its own.
Null vs undefined semantics in AudioRecognition.updateOptions
AudioRecognition.updateOptions treats undefined as "leave unchanged" and only resets when a value (including null) is supplied: if (options.turnDetection !== undefined) { this.turnDetectionMode = options.turnDetection ?? undefined; } (agents/src/voice/audio_recognition.ts:543-545).
Previously the raw parameter was forwarded, so AgentSession.updateOptions({ turnDetection: null }) propagated null and cleared the recognizer's mode. Now _resolvedTurnDetection is forwarded, and _resolveTurnDetection(null ?? undefined) returns undefined, so the reset is silently dropped while this.turnDetectionMode on the activity is cleared — the two diverge. turnDetectionMode === 'manual' gates auto-commit of user turns in the recognizer (agents/src/voice/audio_recognition.ts:1204, 1377, 1396).
| if (hasTurnDetection) { | |
| recognitionOptions.turnDetection = turnDetection; | |
| recognitionOptions.turnDetection = this._resolvedTurnDetection; | |
| } | |
| if (hasTurnDetection) { | |
| // pass null (not undefined) so AudioRecognition clears its mode instead of keeping it | |
| recognitionOptions.turnDetection = this._resolvedTurnDetection ?? null; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Ports livekit/agents PR #6495.
Summary
canDisableTurnDetectioncapability and per-sessionturnDetectionDisabledcreation option.Source diff coverage
livekit-agents/livekit/agents/llm/realtime.pyagents/src/llm/realtime.ts: addedcanDisableTurnDetectioncapability andsession({ turnDetectionDisabled })option.livekit-agents/livekit/agents/llm/realtime_fallback_adapter.pylivekit-agents/livekit/agents/voice/agent_activity.pyagents/src/voice/agent_activity.ts: ported session-scoped realtime turn-detection resolution, validation/warnings, session creation flag, reuse guard, VAD wiring, interruption checks, commit behavior, and gatekeeping checks using TS naming and existing target flow.livekit-agents/livekit/agents/voice/agent_session.pyagents/src/voice/agent_session.ts: track whetherturnDetectionwas explicitly supplied and update that state on runtime changes.livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realtime/realtime_model.pylivekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.pyplugins/google/src/realtime/realtime_api.ts: accept the new session option while leaving disabling unsupported.livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/experimental/realtime/realtime_model.pylivekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.pyplugins/openai/src/realtime/realtime_model.ts: mark default server VAD as disable-capable, preserve Azure intent, and apply per-session disabled turn detection. Python's model-levelupdate_optionspropagation test has no JS counterpart because this target model has no model-level realtimeupdateOptionsAPI.livekit-plugins/livekit-plugins-phonic/livekit/plugins/phonic/realtime/realtime_model.pyplugins/phonic/src/realtime/realtime_model.ts: accept the new session option while leaving disabling unsupported.livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/realtime/realtime_model.pylivekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.pyplugins/xai/src/realtime/realtime_model.ts: keep OpenAI-derived disabling unsupported for xAI and ignore the new session option.tests/fake_realtime.pyagents/src/voice/realtime_auto_disable_turn_detection.test.ts: JS fake realtime model/session recordsturnDetectionDisabledfor ported tests.tests/test_realtime/test_openai_realtime_model.pyplugins/openai/src/realtime/realtime_model.test.ts: added OpenAI realtime default/disabled/Azure/session assertions.tests/test_realtime_auto_disable_turn_detection.pyagents/src/voice/realtime_auto_disable_turn_detection.test.ts: added equivalent JS coverage for resolution matrix, runtime warnings, default warning suppression, session creation flag, and handoff reuse behavior. Fallback adapter cases are not applicable because the target lacks that adapter.Verification
pnpm buildOPENAI_API_KEY= pnpm test agents plugins/openai plugins/google plugins/phonic plugins/xaipnpm lintcue-cliruntime validation with a temporary fake realtime JS agent: assertedconversation_item_added(.item.message.content[0].text =~ "rtTurnDetectionEnabled.*false.*turnDetectionDisabled.*true")Notes
OPENAI_API_KEYenabled optional OpenAI integration tests; those failed on local Silero/model setup unrelated to this port. The rerun above unsetsOPENAI_API_KEYso optional external tests are skipped.pnpm api:checkis blocked by an unrelated package (@livekit/agents-plugin-liveavatar) withoutapi-extractor.json; filtered API checks for touched packages are currently blocked in@livekit/agentsby API Extractor's existingexport * aslimitation againstagents/dist/index.d.ts.Ported from livekit/agents#6495
Original PR description
Auto-disables a realtime model's server-side turn detection for the session when the user configured client-side turn-taking (a TurnDetector,
vad,manual, or an explicit interruption mode) and the model supports it, so client-side EoT/barge-in works without also passingturn_detection=Noneto the RealtimeModel.Adds
RealtimeCapabilities.can_disable_turn_detection(OpenAI sets it from the ctorturn_detectiongiven-ness) and asession(turn_detection_disabled=...)seam applied on the initialsession.updatevia the per-session opts copy, so the model is never mutated and stays reusable.AgentActivityresolves the effective state once (_rt_turn_detection_enabled) and routes every server-side turn-detection check through it.Other realtime plugins accept-and-ignore the flag (
can_disable_turn_detectionstays False): xAI (client turn-taking not stable in testing) and Gemini/others (nocommit_audio/clear_audio; turns are activity-signal based).Stacked on
longc/realtime-barge-in; this PR targets that branch, not main.