fix(react): End the assistant turn when a message is injected - #249
fix(react): End the assistant turn when a message is injected#249jptaylor wants to merge 1 commit into
Conversation
Text input reaches the bot through `sendText`, which produces no UserStartedSpeaking, and the bot is usually mid-utterance, so the BotStoppedSpeaking finalize timer is not armed either. On the RTVI 2.x path those are the only two things that end an assistant turn, so the open message stayed open and every following turn was appended to it. `injectMessage` now closes an in-flight assistant turn first, mirroring UserStartedSpeaking — including leaving the speech cursor where it stopped. System messages are exempt: they are deliberately backdated behind an in-flight message so they don't split it.
Regaddi
left a comment
There was a problem hiding this comment.
Went through this against 1.8.2 (90f8f42) by running the real wiring through usePipecatConversation and diffing the rendered output. The core fix does what it says on the tin, and I confirmed the legacy 1.4.x path, genuine mid-utterance interrupts, system-message backdating, and type-then-speak all come out identical before and after.
Two things I think are worth a look before merge, both inline below.
| // leaves the speech cursor where it stopped — the turn was interrupted, | ||
| // so unspoken text must stay unspoken. | ||
| if (message.role !== "system") { | ||
| finalizeLastAssistantMessageIfPending(); |
There was a problem hiding this comment.
finalizeLastAssistantMessageIfPending opens with an unconditional cancelFinalizeTimer(), and that timer is the only caller of snapSpeechCursorToEnd. Once it is cancelled the karaoke cursor freezes where it stopped, permanently, because the message is finalized in the same breath.
That is right when the bot is genuinely mid-utterance. It is wrong once BotStoppedSpeaking has already fired: the bot finished its turn, the text really was spoken, and the snap is what marks it so.
Repro: bot speaks a sentence whose last spoken_progress leaves the cursor mid-sentence (the mismatch case snapSpeechCursorToEnd's own docstring exists for), BotStoppedSpeaking fires, user types 500ms later.
| 1.8.2 | this branch | |
|---|---|---|
| spoken | "Let me know if you need anything else." |
"Let me know if you" |
| unspoken | "" |
" need anything else." |
And it stays that way. Same outcome with a trailing will_be_spoken: false segment, which never receives a progress event at all, so only the snap can ever mark it spoken.
Worth flagging: the UserStartedSpeaking path this mirrors already has the same bug on 1.8.2. Speaking inside the 2500ms window produces the identical stuck cursor today. So this is inheriting the flaw rather than inventing it, but it does turn a currently-correct text path into an incorrect one.
The discriminator is already to hand, namely whether the timer was armed:
const botFinishedSpeaking = botStoppedSpeakingTimeoutRef.current !== undefined;
cancelFinalizeTimer();
// ...
if (lastAssistant && !lastAssistant.final) {
if (botFinishedSpeaking) snapSpeechCursorToEnd(get, set);
finalizeLastMessage(get, set, "assistant");
}I tried that locally: it fixes the text path and the pre-existing voice path, still leaves a real mid-utterance interrupt unspoken (your cursor test passes unchanged), and all 269 tests stay green.
| // Finalizing here mirrors the UserStartedSpeaking path, which likewise | ||
| // leaves the speech cursor where it stopped — the turn was interrupted, | ||
| // so unspoken text must stay unspoken. | ||
| if (message.role !== "system") { |
There was a problem hiding this comment.
role !== "system" also catches injected assistant messages, which changes their layout. mergeMessages only merges into a non-final predecessor, so finalizing first blocks the merge:
Bot mid-turn, then injectMessage({ role: "assistant", ... }):
- 1.8.2: one bubble, parts
["Working on it.", "(tool result attached)"] - this branch: two bubbles
injectMessage is public API via usePipecatConversation, so anyone using assistant injection to append to the bot's live message (a citation, a tool-result note) gets a different layout. The reasoning in the comment above is all about text input, and the assistant case is not covered by the new tests. If the intent is really "the user typed something", then role === "user" says that directly. If the split is deliberate, probably worth a line in the comment and a test pinning it.
Minor and not blocking, while you are in here: every non-system inject now fires an extra onMessageUpdated for the assistant message. Harmless for rendering, but consumers persisting on that callback will see one more event per typed message.
Fix:
injectMessagenow ends an in-flight assistant turn, so a text message opens a new bot bubble instead of appending to the previous one.On the RTVI 2.x path a turn is ended only by
UserStartedSpeakingor the 2500 msBotStoppedSpeakingtimer. Text input goes throughsendText, so neither fires — no VAD event, and the timer is cancelled while the bot is speaking. The nextBotOutputreopens the still-open message, and every following turn accumulates in one bubble.Regressed in 1.8.2 (#246), which dropped the per-sentence
finalflag on the v2 path. That flag was incidentally the only thing closing the bubble on the text path.System messages are exempt — they're deliberately backdated behind an in-flight message. The speech cursor is left where it stopped, matching the
UserStartedSpeakingpath.4 tests added to
eventWiring.test.tsx; 2 fail without the fix.