JS LLM Tool Selector Middleware: internal structuredModel.invoke() is streamed to UI

[michaelcolon19]

Checked other resources

• This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import { z } from "zod";
import { ChatOpenAI } from "@langchain/openai";
import { createAgent, llmToolSelectorMiddleware } from "langchain";
import { MemorySaver, MessagesZodState } from "@langchain/langgraph";
 
// --- Main agent model (intentional streaming for UI) ---
const mainModel = new ChatOpenAI({
  model: "gpt-4o-mini",
  streaming: true,
  temperature: 0,
});
 
// --- Internal tool selector model (should NOT stream) ---
const toolSelectorModel = new ChatOpenAI({
  model: "gpt-4o-mini",
  streaming: false, // <-- disabled
  callbacks: [],    // <-- none
  temperature: 0,
});

// --- LLM Tool Selector middleware  ---
const toolSelectorMiddleware = llmToolSelectorMiddleware({
  model: toolSelectorModel,        // <-- internal model
  alwaysInclude: ["transfer_to_date_expert"],
  systemPrompt:
      "Your goal is to select the most relevant tools for answering the user's query. If no tools are relevant then don't use any tools."
});

// --- Minimal agent state ---
const agentState = z.object({
  messages: MessagesZodState.shape.messages,
});

// --- Agent with only tool selector middleware ---
const agent = createAgent({
  model: mainModel,
  tools: [],
  middleware: [toolSelectorMiddleware],
  stateSchema: agentState,
  checkpointer: new MemorySaver(),
});

async function main() {
  console.log("Starting stream… Watch for a leaked summary BEFORE final answer.\n");

  const stream = await agent.stream(
    {
      messages: [{ role: "user", content: "what is today?"}],
    },
    {
      streamMode: ["messages"], // <-- required to observe the bug
      configurable: { thread_id: "toolSelector-reproduction" },
    }
  );
 
  for await (const [mode, chunk] of stream) {
    if (mode !== "messages") continue;

    const [msg] = chunk;
    const role = (msg as any).role ?? "unknown";
    const name = (msg as any).name ?? "";
    const content =
      typeof msg.content === "string"
        ? msg.content
        : JSON.stringify(msg.content);

    console.log(`\n[STREAMED MESSAGE] role=${role} name=${name}`);
    console.log(content.slice(0, 200) + (content.length > 200 ? "..." : ""));

    // Bug behavior:
    //
    // 1. You will see an object with a tools key appear FIRST ({"tools":[]}.  In this example no tools were selected.
    //    (coming from structuredModel.invoke inside llToolSelectorMiddleware)
    //
    // 2. Then you will see the real agent reply
    //
    // The selected tools should NEVER appear in the UI because it is an internal
    // model call
  }

  console.log("\nStream complete.");
}
 
main().catch((err) => console.error("Repro error:", err));

Error Message and Stack Trace (if applicable)

There is no thrown exception.

The bug is incorrect behavior: unwanted streamed middleware output.

Description

What I am doing

Using the official JS llmToolSelectorMiddleware with an agent that streams events (stream() or stream_mode="messages").

What I expect

Internal LLM Tool Selector calls should be:

• completely isolated
• not streamed to front end
• invisible to the user

What happens instead

The internal line:

await structuredModel?.invoke([
        { role: "system", content: selectionRequest.systemMessage },
        selectionRequest.lastUserMessage,
      ])

inside the middleware is always streamed to the UI as if it were a normal assistant response.

This causes:

• an extra message to appear in the UI

Related Issues

I believe this behavior is similar to issue #9455 which related to the same behavior but in the Summarization Middleware

System Info

Platform: macOs
Package Manager: npm (10.8.2)

langchain-js: 1.2.10
langgraph-js: 1.1.2
Node: 20.19.5