Skip to content

fix(react): End the assistant turn when a message is injected - #249

Open
jptaylor wants to merge 1 commit into
mainfrom
jpt/fix-text-interrupt-turn-merge
Open

fix(react): End the assistant turn when a message is injected#249
jptaylor wants to merge 1 commit into
mainfrom
jpt/fix-text-interrupt-turn-merge

Conversation

@jptaylor

Copy link
Copy Markdown
Collaborator

Fix: injectMessage now ends an in-flight assistant turn, so a text message opens a new bot bubble instead of appending to the previous one.

On the RTVI 2.x path a turn is ended only by UserStartedSpeaking or the 2500 ms BotStoppedSpeaking timer. Text input goes through sendText, so neither fires — no VAD event, and the timer is cancelled while the bot is speaking. The next BotOutput reopens the still-open message, and every following turn accumulates in one bubble.

Regressed in 1.8.2 (#246), which dropped the per-sentence final flag on the v2 path. That flag was incidentally the only thing closing the bubble on the text path.

System messages are exempt — they're deliberately backdated behind an in-flight message. The speech cursor is left where it stopped, matching the UserStartedSpeaking path.

4 tests added to eventWiring.test.tsx; 2 fail without the fix.

Text input reaches the bot through `sendText`, which produces no
UserStartedSpeaking, and the bot is usually mid-utterance, so the
BotStoppedSpeaking finalize timer is not armed either. On the RTVI 2.x
path those are the only two things that end an assistant turn, so the
open message stayed open and every following turn was appended to it.

`injectMessage` now closes an in-flight assistant turn first, mirroring
UserStartedSpeaking — including leaving the speech cursor where it
stopped. System messages are exempt: they are deliberately backdated
behind an in-flight message so they don't split it.
@jptaylor
jptaylor requested a review from Regaddi August 28, 2026 08:02

@Regaddi Regaddi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through this against 1.8.2 (90f8f42) by running the real wiring through usePipecatConversation and diffing the rendered output. The core fix does what it says on the tin, and I confirmed the legacy 1.4.x path, genuine mid-utterance interrupts, system-message backdating, and type-then-speak all come out identical before and after.

Two things I think are worth a look before merge, both inline below.

// leaves the speech cursor where it stopped — the turn was interrupted,
// so unspoken text must stay unspoken.
if (message.role !== "system") {
finalizeLastAssistantMessageIfPending();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finalizeLastAssistantMessageIfPending opens with an unconditional cancelFinalizeTimer(), and that timer is the only caller of snapSpeechCursorToEnd. Once it is cancelled the karaoke cursor freezes where it stopped, permanently, because the message is finalized in the same breath.

That is right when the bot is genuinely mid-utterance. It is wrong once BotStoppedSpeaking has already fired: the bot finished its turn, the text really was spoken, and the snap is what marks it so.

Repro: bot speaks a sentence whose last spoken_progress leaves the cursor mid-sentence (the mismatch case snapSpeechCursorToEnd's own docstring exists for), BotStoppedSpeaking fires, user types 500ms later.

1.8.2 this branch
spoken "Let me know if you need anything else." "Let me know if you"
unspoken "" " need anything else."

And it stays that way. Same outcome with a trailing will_be_spoken: false segment, which never receives a progress event at all, so only the snap can ever mark it spoken.

Worth flagging: the UserStartedSpeaking path this mirrors already has the same bug on 1.8.2. Speaking inside the 2500ms window produces the identical stuck cursor today. So this is inheriting the flaw rather than inventing it, but it does turn a currently-correct text path into an incorrect one.

The discriminator is already to hand, namely whether the timer was armed:

const botFinishedSpeaking = botStoppedSpeakingTimeoutRef.current !== undefined;
cancelFinalizeTimer();
// ...
if (lastAssistant && !lastAssistant.final) {
  if (botFinishedSpeaking) snapSpeechCursorToEnd(get, set);
  finalizeLastMessage(get, set, "assistant");
}

I tried that locally: it fixes the text path and the pre-existing voice path, still leaves a real mid-utterance interrupt unspoken (your cursor test passes unchanged), and all 269 tests stay green.

// Finalizing here mirrors the UserStartedSpeaking path, which likewise
// leaves the speech cursor where it stopped — the turn was interrupted,
// so unspoken text must stay unspoken.
if (message.role !== "system") {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

role !== "system" also catches injected assistant messages, which changes their layout. mergeMessages only merges into a non-final predecessor, so finalizing first blocks the merge:

Bot mid-turn, then injectMessage({ role: "assistant", ... }):

  • 1.8.2: one bubble, parts ["Working on it.", "(tool result attached)"]
  • this branch: two bubbles

injectMessage is public API via usePipecatConversation, so anyone using assistant injection to append to the bot's live message (a citation, a tool-result note) gets a different layout. The reasoning in the comment above is all about text input, and the assistant case is not covered by the new tests. If the intent is really "the user typed something", then role === "user" says that directly. If the split is deliberate, probably worth a line in the comment and a test pinning it.

Minor and not blocking, while you are in here: every non-system inject now fires an extra onMessageUpdated for the assistant message. Harmless for rendering, but consumers persisting on that callback will see one more event per typed message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants