|
1 | 1 | # Ably AI Transport SDK |
2 | 2 |
|
3 | | -A durable transport layer between AI agents and users. Streams AI responses over [Ably](https://ably.com/) channels — responses resume after disconnections, conversations persist across page reloads and devices — with cancellation, branching conversations, and multi-user sync. |
4 | | - |
5 | | -`@ably/ai-transport` ships as a single package with four entry points: core primitives, React hooks, Vercel AI SDK integration, and Vercel + React hooks. |
| 3 | +A durable transport layer between AI agents and users. Streams AI responses over [Ably](https://ably.com/) channels - responses resume after disconnections, conversations persist across page reloads and devices, with support for cancellation, branching conversations, and multi-user sync. |
6 | 4 |
|
7 | 5 | > **Status:** Pre-release (`0.x`). The API is evolving. Feedback and contributions are welcome. |
8 | 6 |
|
9 | | ---- |
| 7 | +## The problem |
| 8 | + |
| 9 | +Most AI frameworks stream tokens over HTTP response bodies or SSE. That works until it doesn't: connections drop through corporate proxies, responses vanish on page refresh, and sessions are stuck on a single device or tab. Once an agent starts a long-running task, the user has no way to interrupt it, check if it's still running, or continue the conversation from another device. If a human needs to take over from the agent, the session context is lost. |
| 10 | + |
| 11 | +Ably AI Transport replaces the HTTP stream with an Ably channel. The server publishes tokens to the channel as they arrive from the LLM; the response accumulates on the channel and persists, so partial responses survive disconnection. Any client can subscribe to the same channel from any device. Cancel signals, turn lifecycle events, and conversation history all flow through the channel rather than depending on a single HTTP connection. |
10 | 12 |
|
11 | | -## Why |
| 13 | +It is not an agent framework or orchestration layer - it works alongside tools like the Vercel AI SDK, Temporal, and AG-UI. |
| 14 | + |
| 15 | +```mermaid |
| 16 | +sequenceDiagram |
| 17 | + participant U as User |
| 18 | + participant CT as Client Transport |
| 19 | + participant AC as Ably Channel |
| 20 | + participant ST as Server Transport |
| 21 | + participant LLM |
| 22 | +
|
| 23 | + U->>CT: type message |
| 24 | + CT->>ST: HTTP POST (messages) |
| 25 | + ST->>LLM: prompt |
| 26 | + LLM-->>ST: token stream |
| 27 | + ST->>AC: publish chunks |
| 28 | + AC->>CT: subscribe (decode) |
| 29 | + CT->>U: render tokens |
| 30 | +``` |
12 | 31 |
|
13 | | -The default AI SDK transport streams tokens over an HTTP response body. This works for simple single-tab chat, but breaks down when you need: |
| 32 | +## What this gives you |
14 | 33 |
|
15 | | -- **Resumable streaming** — If the user's connection drops mid-response, the HTTP stream is lost. With Ably, the client reconnects and picks up where it left off. No token loss, no restart. |
16 | | -- **Multi-device and multi-tab sync** — Two browser tabs, a phone and a laptop, or multiple users on the same conversation. All devices subscribe to the same Ably channel and see the same stream in real time. |
17 | | -- **Reliable cancellation** — Cancel signals travel over the Ably channel, not the HTTP connection. Cancellation works across devices and survives network interruptions. |
18 | | -- **Concurrent turns** — Multiple request-response cycles can run in parallel on the same channel. Each turn has its own stream and abort signal. The transport multiplexes them automatically. |
19 | | -- **Conversation history from the channel** — The Ably channel *is* the conversation history. Clients can hydrate their UI from channel history on load or reconnection — no separate database query needed. |
20 | | -- **Serverless compatibility** — The AI response streams over Ably, not the HTTP response body. The HTTP request returns immediately, and Next.js `after()` keeps the function alive until streaming completes. No timeout risk. |
21 | | -- **Branching conversations** — Regenerate or edit messages to create forks in the conversation tree. The SDK tracks parent/child relationships and exposes a navigable tree. |
22 | | -- **Barge-in** — Users send new messages while the AI is still responding, with composable primitives for cancel-and-resend or queue-until-complete patterns. |
| 34 | +- **Resumable streaming** - If a connection drops mid-response, client reconnects and picks up where it left off. The response persists on the channel, so nothing is lost. |
| 35 | +- **Session continuity across surfaces** - The session belongs to the channel, not the connection. A user can change tab or device and pick up at the same point. |
| 36 | +- **Multi-client sync** - Multiple users, agents, or operators subscribe to the same channel. Human-AI handover is a channel operation, not a session migration. |
| 37 | +- **Cancellation** - Cancel signals travel over the Ably channel, not the HTTP connection, and the server turn's `abortSignal` fires automatically. |
| 38 | +- **Interruption** - Users send new messages while the AI is still responding, with composable primitives for cancel-and-resend or queue-until-complete. |
| 39 | +- **Concurrent turns** - Multiple request-response cycles run in parallel on the same channel. Each turn has its own stream and abort signal. |
| 40 | +- **History** - The Ably channel is the conversation record. Clients hydrate from channel history on load - no separate database query needed. |
| 41 | +- **Branching** - Regenerate or edit messages to fork the conversation. The SDK tracks parent/child relationships and exposes a navigable tree. |
| 42 | +- **Framework-agnostic** - A codec interface decouples transport from the AI framework. Ships with a Vercel AI SDK codec; bring your own for any other stack. |
23 | 43 |
|
24 | | -The SDK is codec-agnostic. A `Codec` translates between your AI framework's types and the Ably wire format. A Vercel AI SDK codec ships built-in. |
| 44 | +### When you need this |
| 45 | + |
| 46 | +- AI products where connection reliability and session durability are non-negotiable |
| 47 | +- Multi-surface experiences where a user needs to see the session in multiple tabs or devices |
| 48 | +- Collaborative AI where multiple users or agents interact in the same conversation |
| 49 | +- Customer support products where AI conversations are handed to human agents |
25 | 50 |
|
26 | 51 | --- |
27 | 52 |
|
@@ -53,9 +78,9 @@ npm install @ably/ai-transport ably ai |
53 | 78 |
|
54 | 79 | ## Usage with Vercel AI SDK |
55 | 80 |
|
56 | | -Use the Vercel entry points with `useChat` for the shortest integration path. |
| 81 | +AI Transport is complementary to the Vercel AI SDK, not a replacement. The Vercel AI SDK handles model calls, message formatting, and React hooks. AI Transport replaces the transport layer underneath, so tokens stream over Ably instead of an HTTP response body. You keep `useChat`, `streamText`, and everything else you're used to. |
57 | 82 |
|
58 | | -### Server — Next.js API route |
| 83 | +### Server - Next.js API route |
59 | 84 |
|
60 | 85 | ```typescript |
61 | 86 | import { after } from 'next/server'; |
@@ -112,7 +137,7 @@ export async function POST(req: Request) { |
112 | 137 | } |
113 | 138 | ``` |
114 | 139 |
|
115 | | -### Client — React with `useChat` |
| 140 | +### Client - React with `useChat` |
116 | 141 |
|
117 | 142 | ```tsx |
118 | 143 | 'use client'; |
@@ -197,7 +222,7 @@ const ably = new Ably.Realtime({ |
197 | 222 |
|
198 | 223 | --- |
199 | 224 |
|
200 | | -## Usage without Vercel AI SDK |
| 225 | +## Core usage with a custom codec |
201 | 226 |
|
202 | 227 | The core entry point is framework-agnostic. Bring your own `Codec` to map between your AI framework's event/message types and the Ably wire format. |
203 | 228 |
|
@@ -277,8 +302,8 @@ transport.close(); |
277 | 302 |
|
278 | 303 | Two mechanisms cover different failure modes: |
279 | 304 |
|
280 | | -- **Network blips** — Ably's connection protocol automatically reconnects and delivers any messages published while the client was disconnected. No application code required. |
281 | | -- **Resumable streams** — A client that joins or rejoins a channel mid-response (after a page refresh, on a new device, or as a second participant) receives the in-progress stream immediately on subscribing. Load previous conversation history from the channel via `history()`, or from your own database. |
| 305 | +- **Network blips** - Ably's connection protocol automatically reconnects and delivers any messages published while the client was disconnected. No application code required. |
| 306 | +- **Resumable streams** - A client that joins or rejoins a channel mid-response (after a page refresh, on a new device, or as a second participant) receives the in-progress stream immediately on subscribing. Load previous conversation history from the channel via `history()`, or from your own database. |
282 | 307 |
|
283 | 308 | ### Cancellation |
284 | 309 |
|
@@ -345,12 +370,25 @@ transport.on('error', (error) => { |
345 | 370 |
|
346 | 371 | --- |
347 | 372 |
|
| 373 | +## Documentation |
| 374 | + |
| 375 | +Detailed documentation lives in the [`docs/`](./docs/) directory: |
| 376 | + |
| 377 | +- **[Concepts](./docs/concepts/)** - [Transport architecture](./docs/concepts/transport.md), [Turns](./docs/concepts/turns.md) |
| 378 | +- **[Get started](./docs/get-started/)** - [Vercel AI SDK with useChat](./docs/get-started/vercel-use-chat.md), [Vercel AI SDK with useClientTransport](./docs/get-started/vercel-use-client-transport.md) |
| 379 | +- **[Frameworks](./docs/frameworks/)** - [Vercel AI SDK](./docs/frameworks/vercel-ai-sdk.md) |
| 380 | +- **[Features](./docs/features/)** - [Streaming](./docs/features/streaming.md), [Cancellation](./docs/features/cancel.md), [Barge-in](./docs/features/barge-in.md), [Optimistic updates](./docs/features/optimistic-updates.md), [History](./docs/features/history.md), [Branching](./docs/features/branching.md), [Multi-client sync](./docs/features/multi-client.md), [Tool calls](./docs/features/tool-calls.md), [Concurrent turns](./docs/features/concurrent-turns.md) |
| 381 | +- **[Reference](./docs/reference/)** - [React hooks](./docs/reference/react-hooks.md), [Error codes](./docs/reference/error-codes.md) |
| 382 | +- **[Internals](./docs/internals/)** - Architecture details for contributors |
| 383 | + |
| 384 | +--- |
| 385 | + |
348 | 386 | ## Demo apps |
349 | 387 |
|
350 | 388 | Working demo applications live in the [`demo/`](./demo/) directory: |
351 | 389 |
|
352 | | -- **[`demo/vercel/react/use-chat/`](./demo/vercel/react/use-chat/)** — Vercel AI SDK with `useChat` integration |
353 | | -- **[`demo/vercel/react/use-client-transport/`](./demo/vercel/react/use-client-transport/)** — Vercel AI SDK with direct `useClientTransport` hooks |
| 390 | +- **[`demo/vercel/react/use-chat/`](./demo/vercel/react/use-chat/)** - Vercel AI SDK with `useChat` integration |
| 391 | +- **[`demo/vercel/react/use-client-transport/`](./demo/vercel/react/use-client-transport/)** - Vercel AI SDK with direct `useClientTransport` hooks |
354 | 392 |
|
355 | 393 | --- |
356 | 394 |
|
|
384 | 422 |
|
385 | 423 | ## Contributing |
386 | 424 |
|
387 | | -[Open an issue](https://github.com/ably/ably-ai-transport-js/issues) to share feedback or request a feature. |
| 425 | +See [CONTRIBUTING.md](./CONTRIBUTING.md). [Open an issue](https://github.com/ably/ably-ai-transport-js/issues) to share feedback or request a feature. |
0 commit comments