RFC: AI Plugins

[AlemTuzlak]

RFC: The Plugin API

Summary

Add a plugin API to TanStack AI: a way to expose several named AI capabilities (chat and one-shot generation) behind one server handler and one typed client hook. definePlugin on the server, usePlugin / createPlugin on the client, and typed factories (chatPlugin, generationPlugin, plus media wrappers) for each capability. Every plugin is also runnable in-process via plugin.run(). Ships in a new @tanstack/ai-plugin-toolkit package.

Background & Motivation

TanStack AI today gives you the individual activities (chat(), generateImage(), generateSpeech(), ...) and framework hooks for a single chat. An app that offers several capabilities, say a studio that drafts a post, illustrates it, and narrates it, has to hand-wire each one: a separate route per activity, a separate client fetch or hook per call, and its own glue to keep request and response types in sync across the server/client boundary.

There is no first-class way to declare a set of named capabilities once and get, from that single declaration, one server endpoint and one fully typed client surface. As apps combine chat with media generation, that glue is repeated in every project and drifts out of type-sync easily.

Beyond a single app, this shape is a distribution primitive. An open-source project can ship a specialized agent as a plugin: a chat capability wired to its own tools and MCP servers, plus any one-shot generators, published so users install it and drop it into their app with usePlugin, and any client that speaks the endpoint can consume it. Inside a company, a team can build a reusable agent once and ship it across the org like a microservice: the same definition value hands every consumer a typed client, a mountable server handler, and an in-process .run(), so the agent is consumed the same way whether it runs in-process, behind an internal endpoint, or as a package on the registry. Because the definition is inert and holds no secrets, sharing it is safe; the packaged unit is "a set of typed capabilities," not a bespoke SDK per agent.

Because the plugin factories are ordinary functions, a platform can wrap them into its own branded factory that pre-wires its services while leaving the agent's behavior fully in the user's hands. A platform offering, say, self-improving agents publishes acmePlugin(config); the user configures system prompts, tools, and MCP servers, and the platform's models, telemetry, and improvement loop run under the hood:

// @acme/ai (the platform) — its services baked in, behavior left to the caller
import { chatPlugin } from '@tanstack/ai-plugin-toolkit'
import { chat } from '@tanstack/ai'

export function acmePlugin(config: AcmeChatConfig) {
  return chatPlugin((req) =>
    chat({
      adapter: acmeModel(),                              // Acme models + self-improvement loop, preconfigured
      messages: req.messages,
      systemPrompts: config.systemPrompts,               // user-owned
      tools: [...acmeTools, ...(config.tools ?? [])],    // Acme's tools + the user's (incl. their MCP servers)
    }),
  )
}

// the user — never sees Acme's wiring, consumes it through the same usePlugin surface
const support = acmePlugin({ systemPrompts: ['You are a billing assistant'], tools: [lookupInvoice] })
export const helpdesk = definePlugin({ support })

acmePlugin returns a normal chatPlugin, so it drops into definePlugin beside any other plugin and reaches the client as a fully typed surface. The platform ships one function; the user owns the configuration.

This proposes that missing layer. It is additive: it builds on the existing activities and the AG-UI wire protocol, and changes neither.

Goals

• One declaration yields both halves: definePlugin produces a server handler and the typed shape the client hook consumes.
• Public surface is two entry points: definePlugin (server) and usePlugin / createPlugin (client). No third declaration; no generics at the call site.
• Each capability is a typed factory with input and result contracts pre-bound end to end: chatPlugin, generationPlugin, and media wrappers (imagePlugin, videoPlugin, audioPlugin, speechPlugin, transcriptionPlugin, summarizePlugin).
• The client binds off the definePlugin value directly, with no server code or credentials leaking into the browser bundle.
• Plugins run in-process (plugin.run()), not only over HTTP, so they compose in app code and in future primitives.
• The authoring API lives in one importable package.
• Core stays schema-library-agnostic: no runtime zod (or any concrete schema library) dependency.

Non-Goals

• Server-side composition. No cross-plugin orchestration on the server (a plugin calling another plugin mid-request). That is a future workflowPlugin; plugin.run() is its building block. Multi-step flows today are orchestrated by the app (see Proposed Design).
• Moving the client runtime (PluginClient) or the framework hooks into the toolkit package; they stay in @tanstack/ai-client and the framework packages, importing plugin types from the toolkit.
• Changing the underlying activities or the AG-UI wire protocol.

Proposed Design

Server: name capabilities with factories, expose one handler.

// my-plugins.ts (shared module, safe to import from the client too)
import { definePlugin, chatPlugin, imagePlugin, speechPlugin } from '@tanstack/ai-plugin-toolkit'
import { chat, generateImage, generateSpeech } from '@tanstack/ai'
import { openaiText, openaiImage, openaiSpeech } from '@tanstack/ai-openai'

const drafting = chatPlugin((req) =>
  chat({ adapter: openaiText('gpt-5.5'), messages: req.messages, outputSchema: BlogPostSchema, stream: true }),
)
const heroImage = imagePlugin((req) => generateImage({ adapter: openaiImage('gpt-image-2'), prompt: req.input.prompt }))
const narration = speechPlugin((req) => generateSpeech({ adapter: openaiSpeech('tts-1'), text: req.input.text }))

export const blogPlugin = definePlugin({ drafting, heroImage, narration })
// route:  POST: ({ request }) => blogPlugin.handler(request)

Client: import the value, flat per-plugin options, no generics.

import { usePlugin } from '@tanstack/ai-react/plugin'
import { blogPlugin, heroPromptFor, forNarration } from '../lib/my-plugins'

const p = usePlugin(blogPlugin, {
  connection,
  drafting:  { forwardedProps: { tone: 'punchy' } },
  heroImage: { onResult: (img) => save(img) },
})

const draft = await p.drafting.sendMessage(topic)   // typed structured chat result
await Promise.all([                                   // multi-step orchestration is the app's job
  p.heroImage.run({ prompt: heroPromptFor(draft) }),
  p.narration.run({ text: forNarration(draft.body) }),
])

Direct execution: every plugin has .run(), a sibling to .handler, returning the typed result in-process (no HTTP). It accepts raw input, a Request, or an already-parsed body.

const img = await heroImage.run({ prompt: 'a cat' })       // → ImageGenerationResult
const { text, structured } = await drafting.run(messages)  // chat → collected result
export const GET = async ({ request }) => Response.json(await heroImage.run(request))

How it works:

• definePlugin(config) returns { plugins, pluginKinds, handler }. The definition is inert: no adapters or connections are constructed until a request reaches handler, so it is safe to import into an isomorphic module. pluginKinds (a name → 'chat' | 'one-shot' map carried in the value) lets the client build the correct sub-client per name with no second declaration.
• usePlugin(def, options) reads reserved keys (connection / id / threadId) off the flat options and treats every other key as per-plugin options, building one chat client per chat plugin and one generation client per one-shot plugin.
• Media factories are thin generationPlugin wrappers whose input schema is a hand-rolled Standard Schema, so no concrete schema library is pulled into the runtime bundle. Apps wanting richer validation pass their own schema to generationPlugin directly.
• handler and .run() share one request-build and validation core: handler emits the AG-UI SSE stream, .run() collects the terminal result instead.

flowchart LR
  subgraph toolkit["@tanstack/ai-plugin-toolkit"]
    cp["chatPlugin / generationPlugin / media factories"] --> dp["definePlugin({...})"]
  end
  dp -->|".handler(request)"| sse["AG-UI SSE"]
  sse --> pc["PluginClient (@tanstack/ai-client)"]
  pc --> uc["usePlugin / createPlugin (framework hooks)"]
  dp -.->|".run(input)"| res["typed result (in-process)"]

Loading

Alternatives Considered

A. Fixed capability set instead of an arbitrary named registry. Restrict definePlugin to a closed enum of known keys.

definePlugin({ chat: chatPlugin(...), image: imagePlugin(...), video: videoPlugin(...) }) // keys constrained

Rejected: loses custom, app-named generations and the ability to declare several chat surfaces (primaryChat, summaryChat). The chosen design keeps the media set as convenience factories over a generic generationPlugin, covering both the common and the custom case.

B. A separate type-only client declaration. Declare the plugin kinds a second time on the client instead of binding off the definition value.

const clientDef = clientPlugin<typeof blogPlugin>({ drafting: 'chat', heroImage: 'one-shot' })
const p = usePlugin(clientDef, { ... })

Rejected: a redundant second source of truth that drifts. The definePlugin value already carries names and kinds at runtime and is import-safe (inert, no secrets), so the client binds off it directly. Cost: the value's referenced adapter code is bundled inert into the client (see Risks).

C. .run() returns a Response. Make each plugin a one-line HTTP endpoint and handler a router over .run().

run(request: Request): Promise<Response>   // SSE
export const POST = ({ request }) => heroImage.run(request)

Rejected: collapses streaming for chat and conflates "serve over HTTP" with "execute and get a result." The chosen direct-result form is what app-side orchestration and a future workflowPlugin need; serving one plugin is still one line: Response.json(await plugin.run(request)).

D. Ship inside @tanstack/ai (a /plugin subpath) instead of a dedicated package.

Rejected: the authoring API is a distinct concern with its own dependency surface. A dedicated @tanstack/ai-plugin-toolkit keeps @tanstack/ai focused on activities and gives the plugin API a single home. Cost: the toolkit imports a few internals from @tanstack/ai (schema validation, SSE response, request parsing), which must be part of that package's public surface.

E. Build server-side composition into the registry now. Let a plugin receive a context and call sibling plugins server-side, streaming sub-results within one request.

Rejected for the first version: it is a large protocol (sub-run streaming, client demultiplexing, per-sub-run surfaces) that most apps do not need, and it belongs in a dedicated orchestration primitive. Client-side orchestration covers the common case now; .run() is the building block for a later workflowPlugin.

F. Distribute agents as a network service only (HTTP / MCP), no shared package. For the distribution use case, ship each agent as an endpoint plus a hand-written or generated client; consumers call it over the wire.

// consumer in another codebase: no shared types, contract re-described by hand
const res = await fetch('https://agents.acme.internal/blog/heroImage', {
  method: 'POST',
  body: JSON.stringify({ prompt }),
})

Rejected as the primary consumption path: it drops compile-time types across the boundary and the in-process .run() composability, and forces every consumer to re-describe the contract. Because a plugin ships as a definition value (types plus an inert handler), a consumer imports it for a fully typed client and a mountable server, and can still expose it over HTTP or MCP on top. Network-only consumption stays possible; it is not the default, and it is not either/or with shipping the typed definition.

Risks & Tradeoffs

• Client bundle carries inert adapter code. Importing the definePlugin value pulls referenced adapters into the client graph. Mitigation: callbacks never run client-side and hold no secrets (keys come from server env at request time); the definition is inert. Accepted; first-class lazy loading is a possible later optimization, not required.
• Chat .run() structured collection. Collecting { text, structured } from the stream must not diverge from how structured output is produced. Mitigation: it reads the terminal structured-output completion event's object directly (the same source the stream processor uses), not a separate parser.
• New package, workspace consistency. A new workspace member must keep dependency ranges aligned. Mitigation: ranges match sibling packages; test:sherif / test:knip gate it.
• .run() synthesizes a request envelope in direct mode (generated ids, a synthetic Request). Accepted; PluginRunOptions lets callers override threadId / runId / signal / forwardedProps.

Rollout Plan

• Ships as a PR targeting main: the new @tanstack/ai-plugin-toolkit package, plus plugin entry points on @tanstack/ai-client and the four framework packages.
• Changeset: minor for @tanstack/ai-plugin-toolkit (new), @tanstack/ai, @tanstack/ai-client, and the framework packages.
• Coverage: unit tests in the toolkit (registry, media factories, .run() across all three input forms including the abort signal); an E2E route exercising a chat plugin, a one-shot plugin, a media plugin, and a direct .run() call; documentation samples typechecked in CI.
• Docs and the agent skill land with the feature.
• Follow-up: workflowPlugin adds server-side composition on top of plugin.run().

Open Questions

• workflowPlugin shape. Whether composition is a new plugin kind that receives the other plugins and calls their .run(), or a separate defineWorkflow, is out of scope here but should stay compatible with .run()'s options surface.
• True lazy client loading. Whether to offer a first-class way to keep referenced adapter code out of the client bundle, or leave it to app-level code splitting.

Approvers

TanStack AI maintainers.

[jherr]

Broad questions:

• Plugins here are statically defined and imported, so this isn't about runtime extensibility. Given that, what does definePlugin buy over exporting plain functions and adding explicit routes? The types are already shared via imports either way.
• What are the ideal use cases? Specifically: is the target first-party organization of your own app, or third-party distribution? If third-party, what's the configuration story — a published plugin hardcodes prompts/model choices but the provider keys and config belong to the host app.
• What's the deployment model you're designing for? Some of these choices (single multiplexed endpoint, connection-configurable clients, secret-free definitions safe for the client bundle) look like they're anticipating a hosted proxy/gateway rather than an app's own backend. To be clear on where I stand: it's not TanStack's intention to provide a live service layer. If a hosted gateway/proxy service exists, it should come from a separate entity, and the integration should be a library that entity publishes against our public API — not gateway assumptions baked into TanStack AI's core surface. Designing our API around a service that doesn't yet exist means when its real requirements show up (auth, abuse, billing, multi-tenancy), either the service contorts to fit us or we eat breaking changes for it.
• Why does this need to be in a library at all? Everything shown here — grouping handlers, sharing types, one route — is achievable in userland with the existing public API. If there are technical reasons it can't be, the RFC should state them concretely, because right now it's shipping maintenance surface and semver obligations for something apps can compose themselves. Why not start with an example app, work out the use cases and the API against real usage, and then extract it into a library once it's earned it?

Technical questions/requests:

• More detail on the client → server API. Is the intention a single /api/ai endpoint discriminating by payload? If so: auth, rate limiting, and abuse mitigation differ a lot between chat and image/video generation — per-route middleware handles that today, where does it live under a multiplexed handler? And what's the wire protocol for non-chat capabilities — is an image result also an AG-UI SSE stream?
• This is locked into request handlers — plugin.handler(request). What's the format of that request? There are lots of kinds of requests out there. Would this work over websockets? Server functions? Mocked drivers for testing? The current client has a customizable transport layer — why does the plugin system bypass that and hard-bind to Request/Response specifically?
• For chat plugins, is the entire request (history, system prompt, tools?) in the payload with the server acting as a proxy?
• How do we handle streaming responses (chat) vs single/multiple results (images) vs long-running jobs (video)? Is there a job/polling model, or does a video generation hold an SSE connection open?
• The RFC notes client bundles carry inert adapter code and defers lazy loading. "Callbacks never execute client-side" is intent, not enforcement — importing server modules into isomorphic files is exactly how secrets/env leak into bundles. I'd want this solved, not deferred.

[AlemTuzlak]

1. what do you mean exactly with plain functions?
2. both as stated in the rfc, up to the plugin author to decide how extensible/restrictive his plugins are and how he wants to ship them, either a raw plugin or a function that returns a plugin and takes in additional config to extend it
3. none, the implementation doesn't care where its deployed as it lives in the users codebase as API endpoints
4. better ergonomics and ease of use and a bit more stability over the whole thing than using the original primitives as it offers a stable contract and gives you the raw request, i've already done the demo app and shown it to you so not sure what the remainder of the questions are, feels AI generated.

---

technical questions:

1. every handler gets a raw request to implement the behavior it needs so it's fully self contained, up to you if you call it inside the api route as a part of the routes logic or on the root of it
2. plugins take in request/body/raw args, I think the PR implementation we had was maybe bound to SSE but there's no reason it cna't be a headless transport as it is currently, the idea is that it is so yeah.
3. i guess? the handler forwards it to the plugin so it is a proxy in a sense
4. works the same way as it does with the primitives, no abstractions on the transport layer at all
5. sure

[jherr]

1. i mean import { chat } from "@tanstack/ai" vs defining a plugin on my client and using that.

TBH, I feel like this idea is going back to the original import { ai } from "@tanstack/ai"; idea that we abandoned and it has similar issues. The primary we figured out that we didn't want to do that was to give the LLMs a blind taste test; here are two APIs, which would you prefer and more predictably write (no extra context given). And the consensus was that a generateImage was easier/clearer than a ai({ type: "image", ... }) polymorphic deal. And perhaps we should do the same thing here; again, totally blind, no additional context.

2. I mean use case in the sense of practical customer facing applications that would benefit from this. The demo app was a blog studio that created a blog with associated images as an atomic unit, but there was no inherent advantage of plumbing both the chat and the image generation through a single API. We live in agentic age, the overhead of creating an endpoint or server function per generation target is exceedingly low. It's also easier to trace, is less susceptible to abuse, etc.

3. a couple of things, who are the plugin authors here? and what are the new plugins that this projects? if the idea is that vendors will take over writing the vendor libs and instead write plugins then we are actually racing back towards bespoke interfaces for the same request types since each plugin is unique and there is no standardized interface for say, image generation. Which effectively defeats the purpose of an ecosystem of libraries whose primary purpose is cross platform portability.

4. but don't we have that already? we have a contract for how to generate an image, or complete a chat. in a sense the plugins are the vendor adapter libraries we already have. and you just slot those libraries into the appropriate generation function and.. AI!

"i've already done the demo app and shown it to you", so this RFC doesn't stand alone?

on the technical points, I could continue on there but I have other things I need to do this morning.

[AlemTuzlak]

1, that would be fair for the initial assistant proposal with fixed things like chat, image etc, this would be named like draft, coverImage, I odn't think telling an llm hey generate a cover image using this api would result in it not knowing to call the coverImage one, but this is all conjecture, this would have to be tested and confirmed rather than guessed
2. Creating an endpoint is not the benefit of this, it's currated hand-crafted agents that are exeedingly good at doing something they are meant to do, I can come up with more examples for sure that showcase this.
3. i didn't have vendors in my mind for the primary use-case, more for people who wnat to create tailored plugins and provide it to their users, eg i'm running an ecommerce cms and I have an agent that hooks up to my ecommerce solution and helps the end-users buy stuff, i can distribute it to every customer through a plugin instead of copy pasting it per project, or maybe even a plugin marketplace for agents, vendors i don't know, i would still expect them to do things like write mcps, apis etc
4. as i've said many times, this does not target the same audience, we do have those

"i've already done the demo app and shown it to you", so this RFC doesn't stand alone?
you specifically asked for a demo of this RFC, i think it stands alone without a demo.