Middleware support

[aneequrrehman]

Would love to see a middleware or plugin system that allows intercepting and transforming the chat flow.

So I'm building a memory integration package for Tanstack AI (already have for Vercel AI SDK using wrapLanguageModel). The goal is to:

1. Query relevant memories before each chat request
2. Inject them into the system prompt
3. Trigger memory extraction after the response completes

Without middleware, developers would need to manually orchestrate multiple steps but with middleware support, it could be as simple as:

const stream = chat({
  adapter: openai(),
  messages: modelMessages,
  middleware: [recallMiddleware()],
});

This would enable building integrations for cross-cutting concerns like memory injection, logging, caching, rate limiting, etc.

[AlemTuzlak]

What exactly would you want to transform and when? Would like to add this, but make it right and flexible for all use-cases

[aneequrrehman]

My use-case at the moment is RAG-based where when the user sends a new message, I need to extract the latest message, run a similarity search on that and include the top results in the system prompt (so transforming system prompt).

But middlewares broadly speaking would ideally need to support scenarios like:

• Guardrails: so injecting safety instructions into system prompt
• PII redaction: transforming messages to scrub sensitive data
• Supporting tool calls on the AI models that don't have native support for those etc.

[MyPrototypeWhat]

Hey, just wanted to share that I've built a middleware that addresses several of the use cases discussed here — specifically context compression, tool result truncation, and dynamic state injection.

It's called @context-chef/tanstack-ai and it plugs directly into the middleware array of chat():

import { contextChefMiddleware } from '@context-chef/tanstack-ai';
import { chat } from '@tanstack/ai';
import { openaiText } from '@tanstack/ai-openai';

const stream = chat({
  adapter: openaiText('gpt-4o'),
  messages,
  middleware: [
    contextChefMiddleware({
      contextWindow: 128_000,
      compress: { adapter: openaiText('gpt-4o-mini') },
      truncate: { threshold: 5000 },
    }),
  ],
});

What it handles:

• History compression — summarizes old messages when over token budget
• Mechanical compaction — zero-LLM-cost pruning of tool call pairs
• Tool result truncation — auto-truncate large outputs, optionally persist originals via VFS
• Dynamic state injection — runtime state as XML into prompts
• Token tracking — feeds onUsage back to the compression engine

I wrote a more detailed post in a separate discussion — just wanted to drop a note here since this thread is directly relevant.

Full source: context-chef/packages/tanstack-ai