[BidiGenerateContent] Multiple issues with gemini-3.1-flash-live-preview in production voice calls

[Hprg]

Summary

We operate a production voice AI system (real estate listing coordinator) that uses gemini-3.1-flash-live-preview via the raw WebSocket v1beta BidiGenerateContent endpoint, bridged to Twilio MediaStreams for live phone calls. We've encountered four issues that affect every call.

Environment

• Model: gemini-3.1-flash-live-preview
• Voice: Kore
• Transport: Raw WebSocket v1beta — wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=...
• Audio pipeline: Twilio MediaStreams (mulaw 8kHz) → PCM16 16kHz conversion → Gemini
• VAD config: automaticActivityDetection with startOfSpeechSensitivity: START_SENSITIVITY_HIGH, endOfSpeechSensitivity: END_SENSITIVITY_HIGH, prefixPaddingMs: 20, silenceDurationMs: 500
• System instruction size: ~14K-33K characters
• Platform: Node.js (ws library, no SDK), production system handling real calls daily

---

Issue 1: Stuttering/repeating on first turn (VAD self-interruption)

Behavior: When Gemini starts speaking on an outbound call, the caller's "Hello?" triggers VAD which interrupts Gemini mid-greeting. Gemini then restarts the greeting from the beginning, causing audible stuttering/repeating.

Reproduction: Occurs consistently on outbound calls when the caller picks up and says "Hello?" while Gemini is delivering its opening greeting. Correlates with large system instructions (~14K-33K chars).

Workaround: 2.5-second audio suppression shield after kickoff + recovery cue injection via realtimeInput.text.

Expected: First turn should have barge-in protection or the model should continue rather than restart when interrupted by brief speech.

---

Issue 2: False language switching despite explicit system instruction

Behavior: Despite the system instruction explicitly stating "Start in English" and "ONLY switch to Spanish if the caller explicitly asks," the model switches to Spanish when it hears a Spanish-sounding name (e.g., "Jose" pronounced "Ho-ZAY") — even though the caller responded in English ("this is he").

Reproduction: Call a contact named "Jose" or similar Spanish-origin name. When they answer in English, Gemini switches to Spanish.

Workaround: Extremely strict language rules: "NEVER guess the caller's language from their name, accent, or tone."

Expected: The model should respect explicit language instructions and not infer language from names or accents.

---

Issue 3: outputTranscription word splitting across serverContent messages

Behavior: When the caller interrupts Gemini mid-sentence, outputTranscription splits words across multiple serverContent messages.

Example: Gemini saying "Take care." gets split into:

• Message 1: outputTranscription: "car"
• Message 2: outputTranscription: "e."

Also, inputTranscription and outputTranscription arrive in the same serverContent message (~10ms apart) during interruptions.

Workaround: Post-call re-transcription using Gemini 2.5 Flash on the call recording.

Expected: outputTranscription should deliver complete words. Input/output transcriptions should not arrive in the same message, or should include timestamps for speaker diarization.

---

Issue 4: ~3 second cold-start latency on first turn

Behavior: First outbound turn takes ~3 seconds from kickoff to TURN_COMPLETE. Subsequent turns are fast (~500ms). Scales with system instruction size.

Impact: 3-second dead air on outbound calls. Callers say "Hello?" multiple times or hang up. Creates the window that triggers Issue 1.

Missing capability: No pre-warming mechanism for the WebSocket. Cannot send system instruction ahead of time before the call connects.

Expected: First-turn latency comparable to subsequent turns, or a pre-warming mechanism.

---

Production Impact

This system handles real estate seller leads via live phone calls daily. These issues affect real customer interactions:

• Issue 1 (stuttering) makes the AI sound broken on first impression
• Issue 2 (language switching) confuses English-speaking callers
• Issue 3 (transcript splitting) degrades call records and analytics
• Issue 4 (cold-start latency) causes awkward silences that lose caller engagement

We have workarounds for all four, but native fixes would substantially improve the developer experience for voice applications.

Happy to provide audio samples, transcript logs, or WebSocket message dumps if helpful.

[kkorpal]

Hi @Hprg, thanks for bringing this to us! I'll take this directly to the engineering team for investigation.

[Hprg]

Follow-up: Additional issues discovered in production (April 21, 2026)

Continuing to build on this same production system (real estate voice AI, Twilio MediaStreams ↔ Gemini Live gemini-3.1-flash-live-preview, voice Kore). Three new issues found:

---

Issue 5: System instruction non-compliance on first turn — hard negative rules ignored

Behavior: The system instruction explicitly states "You must UNMISTAKABLY never say 'may I speak with'" and provides an alternative opener ("Hey [name], this is Jasmine with..."). On the first outbound turn, Gemini ignores this and opens with "Hi, may I speak with [name]?" — the exact phrase that is banned.

Reproduction: Consistent on outbound calls. The banned phrase appears on the very first audio turn despite being explicitly prohibited in the system instruction. Subsequent turns generally respect the rules.

Impact: "May I speak with" is a known telemarketer signal — callers hang up within seconds. One wrong first impression loses the lead permanently.

Expected: Hard negative rules (phrases banned via system instruction) should be enforced from the first turn, not just after warm-up. A bannedPhrases field in the setup config would guarantee compliance at the model level rather than relying on prompt adherence.

---

Issue 6: Cannot distinguish call screener from human — no audio scene understanding

Behavior: When a call screening system answers (e.g., Google Call Screen, carrier screening), it plays a recorded/robotic message: "Record your name and reason for calling." Gemini treats this as a human greeting and delivers its full opener to the robot instead of following screener-specific instructions in the system prompt.

Reproduction: Call any number with Google Call Screen or carrier screening enabled. Gemini responds to the automated system as if it were the real person.

Context: The system instruction has a dedicated screener handling section that says "If a screening system asks you to record your name and reason, say [specific response]." Gemini ignores this because it doesn't recognize the audio as a screener — it processes it as regular speech.

Expected: Given that Gemini Live has native audio understanding, it should be able to detect the distinct quality of recorded/automated screening messages vs. live human speech. An audioSceneDetection capability or callback would allow voice agents to adapt behavior based on what they're hearing (screener, IVR, voicemail greeting, human).

---

Issue 7: Garbled speech output under turn pressure

Behavior: When Gemini is interrupted or responds quickly after a caller speaks, the generated audio occasionally produces phonetically similar but incorrect words. Example: Jasmine said "Sure thing. Mike and hold" — the intended output was "Sure thing. I'll hold."

Reproduction: Intermittent. Occurs more frequently when:

• The model is interrupted mid-sentence and needs to start a new response
• The caller's speech overlaps with Gemini's output
• Turn transitions happen rapidly

Expected: Audio generation should produce coherent speech even under turn pressure. If the model needs more processing time, a brief natural pause is preferable to garbled output.

---

Missing config: thinkingConfig not documented for Live API setup

We noticed thinkingConfig: { thinkingLevel: 'minimal' } is documented in the 3.1 migration notes but not in the Live API setup examples. We were not setting it at all. Is minimal the default for Live API, or does omitting it use a higher thinking level (which would explain the cold-start latency in Issue 4)?

---

All issues observed on gemini-3.1-flash-live-preview with voice Kore, Node.js ws library, Twilio MediaStreams bridge. Happy to provide call recordings or WebSocket message logs for any of these.

[Hprg]

Follow-up: Issues 8–10 + thinkingConfig findings (April 24, 2026)

Same production system — real estate voice AI, Twilio MediaStreams ↔ Gemini Live gemini-3.1-flash-live-preview, voice Kore.

---

thinkingConfig update (follow-up to our earlier question)

We tried adding thinkingConfig: { thinkingLevel: 'minimal' } to the setup payload as documented in the 3.1 migration guide. Result: WebSocket close code 1007 — setup rejected. The gemini-3.1-flash-live-preview model does not accept thinkingConfig in the BidiGenerateContent setup, even though the docs list it as a valid field for 3.1 Flash. We had to revert.

Is thinkingConfig not supported on the Live API, or is this a bug? If minimal is already the implicit default, it would help to know that explicitly in the docs so developers don't waste time debugging silent setup failures.

---

Issue 8: No "listen-only" activation mode — model always speaks when nudged

Problem: There is no way to signal "the call is connected, start listening to audio" without the model immediately generating a spoken response.

Context: On IVR/phone tree calls (automated 888 numbers), the AI needs to stay completely silent while listening to hold music, automated messages, and transfer tones — sometimes for 30+ seconds — and only speak when a real human connects. We use realtimeInput.text with a kickoff word (begin) to signal the start of outbound calls. The model treats ANY text input as a prompt to respond, so it immediately starts speaking its greeting over the automated IVR system, even when the system instruction explicitly says to stay silent.

Our workaround: We now skip the text kickoff entirely for IVR calls and rely on native VAD to eventually trigger a response when a human speaks. This works but is fragile — the model has no explicit signal that the session is "active" and should be listening.

Expected: A realtimeInput signal type that means "session is live, audio is flowing, but do not speak until you hear human speech." Something like { realtimeInput: { listenOnly: true } } or an activityHandling mode that suppresses model output until explicit speech is detected. This would be invaluable for any telephony use case where AI must navigate automated phone systems before reaching a person.

---

Issue 9: inputAudioTranscription inconsistent on short calls

Behavior: inputAudioTranscription: {} is enabled in setup. On some calls, caller-side audio is transcribed correctly (e.g., IVR saying "Please stay on the line" appears as an inputTranscription event). On other calls of similar length (10-12 seconds), zero inputTranscription events are received — only outputTranscription arrives.

Reproduction: Two back-to-back outbound calls to the same 888 number, same model config:

• Call 1 (11 sec): 567 audio chunks received from caller, 0 inputTranscription events
• Call 2 (12 sec): 598 audio chunks received from caller, 1 inputTranscription event captured

Both calls received substantial caller-side audio (IVR messages, hold tones). The audio was forwarded to Gemini identically in both cases. No errors in the WebSocket stream.

Expected: inputAudioTranscription should consistently transcribe all detected speech from the caller's audio stream. If the feature has minimum duration or audio quality thresholds, those should be documented.

---

Issue 10: No audio scene classification

This extends Issue 6 (screener detection). In production telephony, the AI encounters multiple distinct audio environments that require different behavior:

• IVR/phone tree: Automated menus, hold music, transfer tones → AI must stay silent
• Call screener: "Record your name and reason for calling" → AI must state name and wait
• Voicemail greeting: "Leave a message after the beep" → AI must wait for beep, then leave message
• Human greeting: "Hello?" → AI must respond naturally

Currently, the model cannot distinguish between these. It treats all audio as human speech and responds to everything. We've built server-side text detection (checking inputTranscription for known phrases) as a workaround, but this is slow and error-prone.

Expected: Given that Gemini Live has native audio understanding, an audioSceneClassification callback or event would be transformative for telephony use cases. Something like: { audioScene: { type: 'ivr' | 'screener' | 'voicemail' | 'human' | 'hold_music' | 'ring_tone', confidence: 0.95 } }. The model clearly has the audio understanding capability — it just needs an API surface to expose it.

---

All issues observed on gemini-3.1-flash-live-preview with voice Kore, Node.js ws library, Twilio MediaStreams bridge. Happy to provide WebSocket message logs or call recordings for any of these.

[Hprg]

Follow-up: Issues 11-13 — Session resumption, VAD interruptions, and listen mode update

Issue 11: Session resumption broken with system instructions >200 tokens

Confirmed bug. sessionResumption: { handle: previousHandle } stops working when systemInstruction exceeds ~200 tokens. Our production system instruction is ~17K chars (~4K tokens). We confirmed this matches the report at https://discuss.ai.google.dev/t/session-resumption-for-gemini-3-1-flash-live-preview-does-not-seem-to-work-with-long-systeminstructions/136654.

Impact: Every WebSocket reconnect (which happens regularly on 10-min session limits) loses all Gemini-side conversation context. Our workaround is injecting a transcript summary into the reconnect system instruction, but this is a lossy approximation.

Expected: Session resumption should work regardless of system instruction length. The 200-token threshold is far too low for any real-world voice agent.

---

Issue 12: sc.interrupted false-triggers on carrier voicemail menu audio

Behavior: During voicemail delivery on telephony calls, the carrier plays post-greeting menu audio ("Press 1 to replay your message, press 2 to continue recording..."). Gemini's VAD treats this as user speech and fires serverContent.interrupted. Our mid-voicemail pickup detection then incorrectly classifies the call as a live human pickup.

Root cause: In telephony, Gemini receives a single audio stream containing BOTH the caller's audio AND the carrier's system audio (hold music, VM menus, etc.). Gemini's VAD cannot distinguish between human speech and automated carrier audio on the same stream.

Our workaround: Server-side check — before acting on interrupted during VM delivery, we scan the latest inputTranscription text against known VM menu phrases ("press 1", "to replay", etc.). If it matches menu audio, we ignore the interruption.

Expected: This relates to Issue 10 (audio scene classification). If Gemini could distinguish automated system audio from human speech, false interruptions during VM delivery would be eliminated.

---

Issue 13: Listen mode update — outbound kickoff removal working well

Resolution for Issue 8 (listen-only mode): We removed the realtimeInput.text: 'begin' kickoff from ALL outbound calls (not just IVR). Gemini now listens via native VAD on every outbound call, detects when the person says "Hello?", and then delivers the opener from the system instruction.

Results after 1 day of production calls:

• Screener handling improved — Jasmine no longer talks over Google Call Assist
• IVR calls — Jasmine stays silent through automated audio (correct)
• Normal calls — Jasmine waits for "Hello?" then responds naturally
• Pre-warming the Gemini WebSocket at TwiML fetch time eliminates cold-start penalty

This is working better than the listenOnly API feature we requested in Issue 8. The only remaining gap is IVR rejection detection ("already claimed" phrases), which we handle server-side via inputTranscription text matching.

---

All issues on gemini-3.1-flash-live-preview, voice Kore, Node.js ws library, Twilio MediaStreams.

[thorwebdev]

Thank you @Hprg for providing the detailed feedback log, really appreciate it, and thank you also for providing the updates in Issue 13! I'm investigating these with the team and will revert back.