💻 feat: deeper MCP UI integration in the chat UI #9669

samuelpath · 2025-09-17T13:25:54Z

Note

Most lines of code changed in this PR are tests. Advice for reviewer: first toggle out all test files to focus on the main changes in the logic, since there are not that many of them. And only then can you review the tests.

Summary

The past weeks, we shipped the following 3 PRs to integrate MCP UI with LibreChat:

🎨 feat: MCP UI basic integration #9299
🚦 feat: Simplify MCP UI integration and add unit tests #9418
✂️ refactor: use artifacts and callbacks to pass UI resources #9472 (replaced by this one ✂️ refactor: Artifacts and Tool Callbacks to Pass UI Resources #9581)

These PRs made it such that whenever a tool response would send back a UI Resource, it would be displayed in the tool response detail section, as such:

This was only a first step.

The goal of the current PR is to allow the LLM to integrate UI Resource elements directly as part of the chat response. Here is what we would have now:

Or this:

Or when the chat is able to aggregate UI Resources from multiple tool responses:

You can now interact with UI Resources. When you click on an action for which the client should be aware, a message is posted to the client and can be picked up by the LLM.

For instance here I clicked on "Add to cart" in one of the product cards:

We see that LibreChat gets the message from the UI Resource, interprets it. For now, it's as if the UI Resource is impersonating the user. I will explain below how we intend to improve it in a follow-up PR.

The LLM then calls another tool call that flows from the user action (click on add to cart on a given product, product added to the cart, cart displayed):

Key changes made

1. Plugin-Based Architecture for MCP UI Resources

New Plugin System: Created mcpUIResourcePlugin that processes markdown text and converts UI resource markers (like ui0, ui1,2,3) into proper React components
Inline UI Rendering: UI resources now render directly inline within markdown content instead of being displayed separately

2. Data Structure Improvements

Array Format: Changed UI resources from object format {0: resource, 1: resource} to array format [resource1, resource2] for better indexing and iteration
Simplified Processing: This makes it easier to handle multiple resources and carousel displays

3. New Components Created

MCPUIResource: Renders individual UI resources based on markdown markers and message context
MCPUIResourceCarousel: Handles multiple UI resources in carousel format when multiple indices are specified
Improved UIResourceCarousel: Enhanced with proper action handling using handleUIAction

4. Enhanced Integration Points

Markdown Processing: Integrated the plugin into the main Markdown.tsx component to process UI resource markers
Tool Context: Added helpful context in handleTools.js:346 that instructs AI models how to properly format UI resource markers for inline display
Action Handling: Unified UI action handling through handleUIAction utility that properly submits messages when users interact with UI components

5. Better User Experience

Inline Display: UI resources now appear exactly where the AI places them in the text (e.g., "Here's the product details ui0")
Carousel Support: Multiple resources can be displayed together using comma-separated indices (e.g., ui0,1,2)
Interactive Elements: Users can now interact with UI components and their actions properly trigger new messages

6. Comprehensive Testing

Added extensive test coverage for the new components and plugin functionality
Tests cover edge cases, error handling, and proper integration with message context

The overall improvement transforms MCP UI from a separate attachment display to a seamless, inline experience where AI models can precisely control where and how UI resources appear within their responses.

Samples of improvement follow-ups to come

Have a special persona in the chat flow for messages sent by a UI resource user action to the LLM
Add a user setting to show or hide the messages sent by a UI resource user action to the LLM
Handle hints sent by the MCP server response in the metadata about how to best display the UI Resource

Change Type

Please delete any irrelevant options.

New feature (non-breaking change which adds functionality)
This change requires a documentation update (we'll do the documentation at the end of the iterations)

Testing

Here are the MCP servers I use for my testing:

mcpServers:
  mcp-ui-allbirds-tool-call:
    type: 'streamable-http'
    url: https://mcpstorefront.com/?storedomain=allbirds.com

  mcp-ui-allbirds-intent:
    type: 'streamable-http'
    url: https://mcpstorefront.com/?store=allbirds.com&style=default

  mcp-ui-allbirds-prompt:
    type: 'streamable-http'
    url: https://mcpstorefront.com/?store_domain=allbirds.com

  mcp-aharvard:
    type: 'streamable-http'
    url: https://mcp-aharvard.netlify.app/mcp

For the storefront servers, you can enter a prompt like: "looking for men sneakers", and then try to play with the UI elements displayed.

For the mcp-aharvard server, you can ask what the weather is like in some city.

I've shown above what kind of result you should be seeing.

Test Configuration:

Checklist

Please delete any irrelevant options.

My code adheres to this project's style guidelines
I have performed a self-review of my own code
I have commented in any complex areas of my code
My changes do not introduce new warnings
I have written tests demonstrating that my changes are effective or that my feature works
Local unit tests pass with my changes

danny-avila · 2025-09-17T18:31:02Z

client/src/components/Chat/Messages/Content/ToolCall.tsx


+  // Store UI resources when output is available
+  useEffect(() => {
+    if (output && attachments && !isSubmitting) {


instead of adding this useEffect, can you leverage the existing useAttachments hook?

client/src/hooks/Messages/useAttachments.ts

danny-avila · 2025-09-17T18:38:06Z

packages/api/src/mcp/utils.ts

  }
 }
+
+export const uiResourcesInstructions = `The tool response contains UI resources (i.e. Resource URI starting with "ui://"),


I think it's unnecessary to couple this with artifacts. We can easily show the UI resources adjacent to the tool call, or even in place of the tool call, without needing the LLM to "display" it

it's also possible to create a separate plugin for MCP UI resources if LLM input for rendering is what's desired, similar to the one for web search:
client/src/components/Web/plugin.ts

this would also make it "easier" for the LLM so the "artifacts" boilerplate is not necessary. In my experience, it's best to reduce the amount the LLM has to generate and even key the specific "attachment" to render, so it doesn't need to generate a long string to render the expected output.

For example, in web search, we can reference specific links from the search like this:
\ueturn0search3

api/app/clients/tools/util/handleTools.js

danny-avila · 2025-09-24T10:38:56Z

client/src/components/MCPUIResource/plugin.ts

+import type { Node } from 'unist';
+import type { UIResourceNode } from './types';
+
+export const UI_RESOURCE_MARKER = '\ue205';


I would not recommend using this as a marker, and instead use \ui or something similar, especially since the marker does not vary from \ue205.

As long as the plugin correctly falls back to \ui or the chosen marker, if it has no corresponding resource, to avoid edge cases where the AI writes the marker for purposes not related to UI resources.

Using "unicode-like" markers was a convention introduced by ChatGPT for their web search, which was a large inspiration for LC's UI/LLM interaction

Done! The marker is now \ui0 (or any other index) to show the resources. It works just as well!

Co-authored-by: Samuel Path [email protected] Co-authored-by: Pierre-Luc Godin [email protected]

danny-avila · 2025-09-25T12:31:36Z

api/app/clients/tools/util/handleTools.js

      };
      continue;
    } else if (tool && mcpToolPattern.test(tool)) {
+      toolContextMap[tool] = `# MCP Tool \`${tool}\`:


I take issue with this being added to all MCP tools. These instructions can be part of the response that we format in formatToolContent i.e. packages/api/src/mcp/parsers.ts to ensure it's only applied once per "MCP UI" output.

Even better, would be to make use of serverInstructions for this to be on the MCP server to provide:
https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/mcp_servers#serverinstructions

Some MCP Servers have 50+ tools and are loaded individually at this level, and we would have repeated instructions for them all.

I agree that we don't need to add this to all MCP tools and I think formatToolContent is the right place for this.

These instructions aren't about how to use the tools on the server, but rather what to do with the tool response, i.e. how to render them in the client using markers. This is specific to LibreChat's implementation of MCP-UI, and other MCP clients could choose to use a different marker or syntax to render UI resources. This makes no difference for the MCP server, so I don't think the serverInstructions are the right place. What do you think?

danny-avila · 2025-09-25T12:38:25Z

client/src/components/Chat/Messages/Content/UIResourceCarousel.tsx

  const [showRightArrow, setShowRightArrow] = useState(true);
  const [isContainerHovered, setIsContainerHovered] = useState(false);
  const scrollContainerRef = React.useRef<HTMLDivElement>(null);
+  const { submitMessage } = useSubmitMessage();


We should not utilize this hook within the message tree as it can cause unnecessary re-renders across all messages and cause significant client latency.

See ~/components/Chat/Messages/Content/Parts/EditTextPart and client/src/components/Chat/Messages/Content/EditMessage.tsx for how this is prevented and submissions are still allowed.

danny-avila · 2025-09-25T12:40:04Z

client/src/components/Chat/Messages/Content/Markdown.tsx

 import type { Pluggable } from 'unified';
 import { Citation, CompositeCitation, HighlightedText } from '~/components/Web/Citation';
+import {
+  mcpUIResourcePlugin,


imports order

danny-avila · 2025-09-25T12:40:28Z

client/src/components/MCPUIResource/MCPUIResource.tsx

+  const localize = useLocalize();
+  const { submitMessage } = useSubmitMessage();
+  const { messageId } = useMessageContext();
+  const { conversation } = useChatContext();


useChatContext and useSubmitMessage should not be used here as they can cause unnecessary re-renders across all messages and cause significant client latency.

See how other adjacent components access the same things (EditTextPart and EditMessage for submissions)

danny-avila · 2025-09-25T12:40:45Z

client/src/components/MCPUIResource/MCPUIResourceCarousel.tsx

+
+export function MCPUIResourceCarousel(props: MCPUIResourceCarouselProps) {
+  const { messageId } = useMessageContext();
+  const { conversation } = useChatContext();


No use of useChatContext allowed here (will cause lag within the message tree)

danny-avila · 2025-09-25T12:44:07Z

I was able to resolve significant client rendering delays in this PR by addressing similar concerns based on my latest review:

https://github.com/danny-avila/LibreChat/pull/9678/files

SebastianAwaze · 2025-10-07T14:02:42Z

Excited for this to get in!

mhempstock · 2025-10-13T14:09:51Z

Looking forward to this, saw the great presentation at the MCP London conference, which pushed me to add an MCP-UI to one of our current servers. Found it a little limited when it comes to UX right now. Once this is in place, it will enable make so many things possible!

samuelpath force-pushed the mcp-ui-next branch from 9933de0 to 2fc4441 Compare September 17, 2025 18:11

samuelpath marked this pull request as ready for review September 17, 2025 18:18

danny-avila requested changes Sep 17, 2025

View reviewed changes

danny-avila marked this pull request as draft September 18, 2025 12:47

samuelpath force-pushed the mcp-ui-next branch from 2fc4441 to 9d8cb66 Compare September 23, 2025 08:00

github-advanced-security bot found potential problems Sep 23, 2025

View reviewed changes

api/app/clients/tools/util/handleTools.js Fixed Show fixed Hide fixed

samuelpath force-pushed the mcp-ui-next branch 2 times, most recently from 9225fbd to 2d7b88e Compare September 23, 2025 13:38

samuelpath marked this pull request as ready for review September 23, 2025 13:38

samuelpath force-pushed the mcp-ui-next branch from 2d7b88e to e21eab2 Compare September 23, 2025 13:42

danny-avila requested changes Sep 24, 2025

View reviewed changes

plgodin force-pushed the mcp-ui-next branch from e21eab2 to fcf97ac Compare September 24, 2025 19:44

💻 feat: deeper MCP UI integration in the chat UI using plugins

42ee75f

Co-authored-by: Samuel Path [email protected] Co-authored-by: Pierre-Luc Godin [email protected]

samuelpath force-pushed the mcp-ui-next branch from fcf97ac to 42ee75f Compare September 25, 2025 09:25

danny-avila requested changes Sep 25, 2025

View reviewed changes

danny-avila marked this pull request as draft September 25, 2025 12:44

Uh oh!

💻 feat: deeper MCP UI integration in the chat UI #9669

Are you sure you want to change the base?

💻 feat: deeper MCP UI integration in the chat UI #9669

Conversation

samuelpath commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key changes made

Samples of improvement follow-ups to come

Change Type

Testing

Test Configuration:

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny-avila Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny-avila commented Sep 25, 2025

Uh oh!

SebastianAwaze commented Oct 7, 2025

Uh oh!

mhempstock commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

samuelpath commented Sep 17, 2025 •

edited

Loading

danny-avila Sep 25, 2025 •

edited

Loading