Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions client-react/src/conversation/PipecatConversationProvider.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,32 @@ export const ConversationContext =
export const PipecatConversationProvider: React.FC<React.PropsWithChildren> = ({
children,
}) => {
useConversationEventWiring();
const { finalizeLastAssistantMessageIfPending } =
useConversationEventWiring();

const injectMessage = useAtomCallback(
useCallback((get, set, message: {
role: "user" | "assistant" | "system";
parts: ConversationMessagePart[];
}) => {
// An injected message is a turn boundary the RTVI events never report:
// text input reaches the bot through `sendText`, so there is no
// UserStartedSpeaking to close the assistant's turn, and the bot was
// likely mid-utterance, so the BotStoppedSpeaking finalize timer is not
// armed either. Without this the next BotOutput reopens the still-open
// message and the following turn is appended to the previous one.
//
// System messages are excluded: `injectMessage` deliberately backdates
// them behind an in-flight assistant message so they don't split it.
//
// Finalizing here mirrors the UserStartedSpeaking path, which likewise
// leaves the speech cursor where it stopped — the turn was interrupted,
// so unspoken text must stay unspoken.
if (message.role !== "system") {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

role !== "system" also catches injected assistant messages, which changes their layout. mergeMessages only merges into a non-final predecessor, so finalizing first blocks the merge:

Bot mid-turn, then injectMessage({ role: "assistant", ... }):

  • 1.8.2: one bubble, parts ["Working on it.", "(tool result attached)"]
  • this branch: two bubbles

injectMessage is public API via usePipecatConversation, so anyone using assistant injection to append to the bot's live message (a citation, a tool-result note) gets a different layout. The reasoning in the comment above is all about text input, and the assistant case is not covered by the new tests. If the intent is really "the user typed something", then role === "user" says that directly. If the split is deliberate, probably worth a line in the comment and a test pinning it.

Minor and not blocking, while you are in here: every non-system inject now fires an extra onMessageUpdated for the assistant message. Harmless for rendering, but consumers persisting on that callback will see one more event per typed message.

finalizeLastAssistantMessageIfPending();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finalizeLastAssistantMessageIfPending opens with an unconditional cancelFinalizeTimer(), and that timer is the only caller of snapSpeechCursorToEnd. Once it is cancelled the karaoke cursor freezes where it stopped, permanently, because the message is finalized in the same breath.

That is right when the bot is genuinely mid-utterance. It is wrong once BotStoppedSpeaking has already fired: the bot finished its turn, the text really was spoken, and the snap is what marks it so.

Repro: bot speaks a sentence whose last spoken_progress leaves the cursor mid-sentence (the mismatch case snapSpeechCursorToEnd's own docstring exists for), BotStoppedSpeaking fires, user types 500ms later.

1.8.2 this branch
spoken "Let me know if you need anything else." "Let me know if you"
unspoken "" " need anything else."

And it stays that way. Same outcome with a trailing will_be_spoken: false segment, which never receives a progress event at all, so only the snap can ever mark it spoken.

Worth flagging: the UserStartedSpeaking path this mirrors already has the same bug on 1.8.2. Speaking inside the 2500ms window produces the identical stuck cursor today. So this is inheriting the flaw rather than inventing it, but it does turn a currently-correct text path into an incorrect one.

The discriminator is already to hand, namely whether the timer was armed:

const botFinishedSpeaking = botStoppedSpeakingTimeoutRef.current !== undefined;
cancelFinalizeTimer();
// ...
if (lastAssistant && !lastAssistant.final) {
  if (botFinishedSpeaking) snapSpeechCursorToEnd(get, set);
  finalizeLastMessage(get, set, "assistant");
}

I tried that locally: it fixes the text path and the pre-existing voice path, still leaves a real mid-utterance interrupt unspoken (your cursor test passes unchanged), and all 269 tests stay green.

}
injectMessageAction(get, set, message);
}, [])
}, [finalizeLastAssistantMessageIfPending])
);

const botOutputSupported = useAtomValue(botOutputSupportedAtom);
Expand Down
5 changes: 5 additions & 0 deletions client-react/src/conversation/useConversationEventWiring.ts
Original file line number Diff line number Diff line change
Expand Up @@ -437,4 +437,9 @@ export function useConversationEventWiring() {
}, [])
)
);

// Exposed so a caller-driven turn boundary — a message injected into the
// conversation, which no RTVI event announces — can end the assistant turn
// the same way UserStartedSpeaking does.
return { finalizeLastAssistantMessageIfPending };
}
133 changes: 125 additions & 8 deletions client-react/tests/conversation/integration/eventWiring.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,18 @@ import { RTVIEvent } from "@pipecat-ai/client-js";
import { act, render } from "@testing-library/react";
import { createStore, Provider } from "jotai";

import { messagesAtom } from "@/conversation/conversationAtoms";
import { PipecatConversationProvider } from "@/conversation/PipecatConversationProvider";
import type { ConversationMessage } from "@/conversation/types";
import {
botOutputMessageStateAtom,
messagesAtom,
} from "@/conversation/conversationAtoms";
import {
PipecatConversationProvider,
useConversationContext,
} from "@/conversation/PipecatConversationProvider";
import type {
ConversationMessage,
ConversationMessagePart,
} from "@/conversation/types";
import { RTVIEventContext } from "@/RTVIEventContext";

/**
Expand All @@ -41,6 +50,15 @@ function renderWiring() {
handlers.get(event)?.delete(handler);
};

let injectMessage: ReturnType<
typeof useConversationContext
>["injectMessage"];

const CaptureContext = () => {
injectMessage = useConversationContext().injectMessage;
return null;
};

render(
<Provider store={store}>
<RTVIEventContext.Provider
Expand All @@ -51,7 +69,9 @@ function renderWiring() {
off: off as any,
}}
>
<PipecatConversationProvider>{null}</PipecatConversationProvider>
<PipecatConversationProvider>
<CaptureContext />
</PipecatConversationProvider>
</RTVIEventContext.Provider>
</Provider>
);
Expand All @@ -69,14 +89,31 @@ function renderWiring() {
});
};

const getMessages = () => store.get(messagesAtom);

const inject = (
role: "user" | "assistant" | "system",
parts: ConversationMessagePart[]
) => {
act(() => {
injectMessage({ role, parts });
});
};

return {
emit,
advance,
getMessages: () => store.get(messagesAtom),
inject,
getMessages,
getAssistantMessages: () =>
store
.get(messagesAtom)
.filter((m: ConversationMessage) => m.role === "assistant"),
getMessages().filter((m: ConversationMessage) => m.role === "assistant"),
getLastAssistantCursor: () => {
const lastAssistant = [...getMessages()]
.reverse()
.find((m: ConversationMessage) => m.role === "assistant");
if (!lastAssistant) return undefined;
return store.get(botOutputMessageStateAtom).get(lastAssistant.createdAt);
},
};
}

Expand Down Expand Up @@ -208,6 +245,86 @@ describe("useConversationEventWiring", () => {
});
});

describe("RTVI 2.0.0+ text-input turn boundaries", () => {
const userText = (text: string): ConversationMessagePart[] => [
{ text, final: true, createdAt: new Date().toISOString() },
];

/**
* Text input reaches the bot through `sendText`, which produces no
* UserStartedSpeaking, and the bot is mid-utterance, so no finalize timer
* is armed. Injecting the user's message is the only turn boundary the
* conversation ever sees.
*/
function startInterruptedV2Turn() {
const w = renderWiring();
w.emit(RTVIEvent.BotReady, { version: "2.1.0" });
w.emit(RTVIEvent.BotStartedSpeaking);
w.emit(
RTVIEvent.BotOutput,
sentence("Hi there, how can I help you today?", 1, {
spoken_status: "new",
})
);
w.emit(
RTVIEvent.BotOutput,
sentence("Hi there, how can I help you today?", 1, {
spoken_status: "in-progress",
spoken_progress: {
accumulated_text: "Hi there,",
remaining_text: " how can I help you today?",
},
})
);
return w;
}

it("finalizes the open turn when a user message is injected", () => {
const w = startInterruptedV2Turn();
expect(w.getAssistantMessages()[0].final).toBeFalsy();

w.inject("user", userText("actually, never mind"));

expect(w.getAssistantMessages()[0].final).toBe(true);
});

it("opens a new message for the reply to injected text", () => {
const w = startInterruptedV2Turn();
w.inject("user", userText("actually, never mind"));

w.emit(RTVIEvent.BotStartedSpeaking);
w.emit(
RTVIEvent.BotOutput,
sentence("No problem.", 2, {
spoken_status: "new",
})
);

const assistant = w.getAssistantMessages();
expect(assistant).toHaveLength(2);
expect(assistant[1].parts.map((p) => p.text)).toEqual(["No problem."]);
});

it("leaves the speech cursor where the interruption stopped it", () => {
const w = startInterruptedV2Turn();
w.inject("user", userText("actually, never mind"));

// Finalizing must not snap the cursor to the end: the turn was cut off,
// so the unspoken tail stays unspoken.
expect(w.getLastAssistantCursor()!.currentCharIndex).toBe(
"Hi there,".length
);
});

it("does not finalize the turn for an injected system message", () => {
const w = startInterruptedV2Turn();
w.inject("system", userText("connection is unstable"));

expect(w.getAssistantMessages()[0].final).toBeFalsy();
expect(w.getAssistantMessages()).toHaveLength(1);
});
});

describe("legacy 1.4.x path", () => {
it("still finalizes per sentence", () => {
const w = renderWiring();
Expand Down
Loading