proposal: add OpenAI Realtime WebSocket voice-agent support

[liuzengh]

Summary

This is a proposal to add OpenAI Realtime API / WebSocket voice-agent support to trpc-agent-go as a staged, parallel session layer rather than forcing Realtime semantics into the existing chat-completions flow.

trpc-agent-go currently has a mature turn-based stack: Runner.Run -> agent / llmflow -> model.Model.GenerateContent -> event.Event -> session persistence -> SSE / AG-UI / OpenAI-chat servers. Realtime voice agents differ because they use a long-lived bidirectional WebSocket session, continuous audio chunks, audio/text deltas, VAD events, cancel/truncate events, and function-call argument deltas.

Current anchors

Relevant existing surfaces:

• runner/runner.go: Runner.Run, ManagedRunner.Cancel, and SteerableRunner.EnqueueUserMessage.
• event/event.go: shared event envelope and Extensions for protocol metadata.
• server/agui/service/service.go: service abstraction already allows transports beyond SSE.
• server/agui/service/sse/sse.go: useful reference for long-lived streaming, though server-to-client only.
• model/model.go: current Model.GenerateContent interface is request/response oriented, not session oriented.
• model/request.go: supports audio as batch content, not live duplex audio.

openai-go constraint

The repository currently depends on github.com/openai/openai-go v1.12.0 and uses chat-completions types such as ChatCompletionNewParams, ChatCompletionChunk, and ChatCompletionAccumulator.

Newer openai-go sources expose generated Realtime schema/config types, such as session config, audio config, tool config, client secrets, calls, and model constants. However, OpenAI Realtime WebSocket support does not appear to be available as a native Connect / Dial / Send / Receive runtime API in openai-go today. Public issue openai/openai-go#600 also tracks this gap.

Implication: this feature should not assume openai-go can own the Realtime WebSocket connection. We can optionally reuse generated schema/config types later, but the WebSocket runtime, read/write loops, event codec, audio buffering, function-call lifecycle, and cleanup logic should live behind a trpc-agent-go abstraction.

Design principles

• Keep existing chat-completions and SSE paths unchanged in the first phase.
• Treat Realtime as a separate session layer, not another GenerateContent request.
• Persist transcripts and tool results, not raw audio deltas.
• Keep OpenAI Realtime protocol concerns separate from AG-UI protocol concerns.
• Make WebSocket lifecycle and cancellation deterministic from the start.
• Reuse tool declarations, callbacks, runner/session primitives, and telemetry where practical.

Options considered

Option A: Realtime proxy

Expose a server/openai/realtime WebSocket endpoint that forwards client events to OpenAI Realtime and forwards OpenAI events back.

Pros:

• Lowest integration risk.
• Fastest way to validate WebSocket lifecycle and OpenAI Realtime protocol compatibility.
• Does not touch runner, llmflow, tools, or session storage.

Cons:

• Bypasses trpc-agent-go agents, tools, memory, graph, and session history.
• Mostly a transport proxy, not a framework-native voice agent.
• Tool calls are OpenAI-side only unless manually bridged later.

Option B: Realtime bridge to Runner

Manage the OpenAI Realtime WebSocket session, but bridge finalized transcripts and function calls into existing Runner and tool execution.

Pros:

• Reuses existing agents, tools, memory, sessions, and telemetry.
• Keeps llmflow mostly unchanged.
• Good balance between product value and contained risk.
• Gives a clear path for barge-in via ManagedRunner.Cancel and SteerableRunner.EnqueueUserMessage.

Cons:

• Realtime is continuous while Runner.Run is turn-based.
• Function-call argument deltas must be buffered before tool execution.
• Audio output still belongs to OpenAI Realtime or a separate TTS layer, not current event.Event text deltas.
• Safe-boundary steering is not truly instant barge-in.

Option C: Native RealtimeAgent and realtimeflow

Create agent/realtimeagent and internal/flow/realtimeflow as first-class Realtime abstractions parallel to LLMAgent and llmflow.

Pros:

• Cleanest long-term architecture.
• Avoids forcing continuous Realtime semantics through turn-based llmflow.
• Easier to model interruption, session update, response cancel, and tool lifecycle correctly.

Cons:

• Highest implementation cost.
• Requires new abstractions and a larger test surface.
• More likely to create parallel behavior that must stay consistent with LLMAgent.

Option D: AG-UI WebSocket transport

Implement server/agui/service/websocket using the existing AG-UI runner and translator.

Pros:

• Fits the existing AG-UI service abstraction.
• Useful for UI clients that need bidirectional control messages.
• Lower risk than native OpenAI Realtime compatibility.

Cons:

• AG-UI WebSocket is not OpenAI Realtime API compatibility.
• Does not solve native audio output or Realtime event semantics.
• Still needs a separate audio/realtime mapping layer for voice agents.

Option E: Third-party Realtime library wrapper

Use a community Go Realtime client behind a small internal interface.

Pros:

• Faster than building all event structs and WebSocket plumbing by hand.
• May already cover many OpenAI Realtime event types.

Cons:

• Adds dependency risk for a core server feature.
• Type semantics and lifecycle behavior may not match trpc-agent-go expectations.
• Must still design runner/tool/session integration.

Option F: Upgrade openai-go and reuse generated types only

Upgrade github.com/openai/openai-go and reuse generated Realtime schema/config types while implementing WebSocket runtime ourselves.

Pros:

• Avoids manually maintaining some schema structs.
• Stays closer to official generated API shapes.
• Useful for session config, tool config, audio config, and model constants.

Cons:

• Upgrading from openai-go v1.12.0 may touch unrelated chat-completions code and tests.
• Generated types may be verbose or not ideal for internal streaming.
• Does not remove the need for WebSocket connection management.

Recommended direction

Use a staged version of Option B, with Option A as the first milestone and Option C as the long-term target.

Recommended path:

1. Add server/openai/realtime and model/openai/realtime as new packages.
2. Define internal Realtime connector interfaces so the implementation can use hand-written WebSocket code, a wrapped third-party client, or future official openai-go support.
3. First deliver proxy mode to validate lifecycle and protocol.
4. Then add bridge mode to reuse Runner, tools, sessions, and cancellation.
5. Defer first-class RealtimeAgent / realtimeflow until bridge mode exposes enough concrete requirements.

Proposed internal interfaces

type RealtimeConnector interface {
    Connect(ctx context.Context, cfg RealtimeConfig) (RealtimeConn, error)
}

type RealtimeConn interface {
    Send(ctx context.Context, event RealtimeClientEvent) error
    Events() <-chan RealtimeServerEvent
    Close() error
}

Transport-facing bridge:

type SessionBridge interface {
    HandleClientEvent(ctx context.Context, event RealtimeClientEvent) error
    ServerEvents() <-chan RealtimeServerEvent
    Close(ctx context.Context) error
}

Event and session mapping

Use event.Event for durable framework events and Realtime-specific WebSocket events for transient audio/session events.

Persist by default:

• User transcript finalized event.
• Assistant transcript finalized event.
• Tool call requested/completed events.
• Run completion, cancellation, and error events.

Do not persist by default:

• input_audio_buffer.append chunks.
• response.audio.delta chunks.
• High-frequency VAD markers unless sampled or summarized.

Use event.Extensions for metadata such as:

• openai_realtime.session_id
• openai_realtime.response_id
• openai_realtime.item_id
• openai_realtime.audio_format
• openai_realtime.close_reason
• openai_realtime.interrupted

Tool integration

Tool declarations can be reused because Realtime tools still need names, descriptions, and JSON schemas.

Bridge requirements:

• Accumulate Realtime function-call argument deltas until complete.
• Validate/repair JSON arguments using existing JSON repair utilities when enabled.
• Execute the matching tool.Tool through existing callbacks and permission policy where possible.
• Send function-call output back to the Realtime connection.
• Emit normal tool result events for session/trace consumers.

If FunctionCallResponseProcessor is too coupled to chat responses, extract a shared internal tool executor used by both llmflow and the Realtime bridge.

Cancellation and barge-in

Map Realtime interruption concepts to existing runner APIs:

• WebSocket close: configurable behavior, either detach or cancel active run.
• response.cancel: call ManagedRunner.Cancel for active request.
• User speech during assistant output: send Realtime cancel/truncate upstream, then use SteerableRunner.EnqueueUserMessage if a run is active.
• Queued messages should only be appended at safe boundaries unless a future native realtimeflow supports more immediate interruption.

Phased delivery

1. Design and codec tests

   • Define minimal Realtime event/config structs.
   • Add JSON encode/decode tests for session update, audio append, transcript done, response done, function call arguments done, and function call output.

2. Proxy mode WebSocket

   • Add WebSocket handler and upstream Realtime connector.
   • Implement deterministic read/write loops, ping/pong, close handling, auth headers, and cleanup.
   • Add a text-only example before microphone/audio examples.

3. Bridge mode transcripts

   • Convert finalized user transcript events into model.Message and call Runner.Run.
   • Convert runner text events into Realtime-compatible text response events.
   • Persist transcript events in session.

4. Tool bridge

   • Map Realtime function calls to existing tools.
   • Send tool outputs back to OpenAI Realtime.
   • Emit framework tool events for trace/session consumers.

5. Voice polish

   • Handle barge-in, response cancel, truncation, and backpressure.
   • Add telemetry: session duration, time-to-first-text, time-to-first-audio, interruption count, tool latency, close reason.

6. Native RealtimeAgent evaluation

   • Decide whether bridge mode is enough.
   • If not, introduce agent/realtimeagent and internal/flow/realtimeflow based on observed requirements.

Testing strategy

• Unit tests for Realtime JSON event codecs.
• Unit tests for function-call argument delta accumulation.
• Unit tests for close/cancel behavior.
• Fake WebSocket upstream for proxy mode.
• Fake Runner for bridge mode transcript and tool mapping tests.
• Integration example with text-only Realtime events.
• Optional manual browser voice example behind docs/examples.

Main risks

• Treating Realtime as just another chat stream would blur lifecycle semantics and make cancellation/tool calls fragile.
• Persisting raw audio would create large session records and hurt summary/token behavior.
• Depending directly on unofficial third-party client types may leak unstable API shapes into public APIs.
• Upgrading openai-go only for generated Realtime types may cause unrelated chat-completion churn.
• Barge-in cannot be fully solved with current safe-boundary steering; native realtimeflow may eventually be needed.

Final recommendation

Adopt an internal Realtime connector boundary and implement server/openai/realtime as a new package. Start with proxy mode to validate WebSocket lifecycle, then bridge transcripts and tool calls into the existing runner. Do not wait for openai-go to provide WebSocket support, but keep the design flexible enough to swap in future official support or reuse generated schema types later.