Skip to content

Commit 68df4f5

Browse files
RaananWRaananW
authored
feat(mcp): Add MCP servers for Babylon.js graph editors (BabylonJS#18344)
## Summary Adds a full Model Context Protocol (MCP) authoring layer for Babylon.js graph-based tools, plus a shared live-session architecture that lets MCP clients create or modify graphs and hand them directly to the matching web editor. The branch now covers the MCP servers themselves, shared server utilities, live editor session infrastructure, editor-side session panels/connectors, packaging for public consumption, coverage/registry maintenance, and focused validation. ## MCP servers This PR adds private tool workspaces for the Babylon.js authoring surfaces that can be driven through MCP-compatible clients such as VS Code Copilot, Claude Desktop, or other MCP hosts. | Workspace | Purpose | | --- | --- | | `@tools/mcp-server-core` | Shared MCP utilities: validation, JSON/text responses, scene attachment schemas, tool schemas, and the shared editor session server/controller | | `@tools/nme-mcp-server` | Node Material Editor material graph authoring | | `@tools/nge-mcp-server` | Node Geometry Editor geometry graph authoring | | `@tools/npe-mcp-server` | Node Particle Editor particle graph authoring | | `@tools/nrge-mcp-server` | Node Render Graph Editor render graph authoring | | `@tools/flow-graph-mcp-server` | Flow Graph visual scripting authoring | | `@tools/gui-mcp-server` | Babylon.js GUI layout authoring | | `@tools/smart-filters-mcp-server` | Smart Filters graph authoring | Each server includes a block/control registry or catalog, graph manager, MCP tools, README coverage, examples, and unit tests. The servers build through a shared Rollup MCP config and emit executable ESM bundles with shebangs. ## Shared live editor sessions Adds a reusable live-session server in `@tools/mcp-server-core` so every graph MCP server uses the same MCP-to-editor workflow instead of maintaining per-server session code. The shared session architecture provides: - A local HTTP/SSE editor session server with health, diagnostics, session listing, document retrieval, and event streaming endpoints. - A generic `McpEditorSessionController` used by all graph/editor MCP servers. - Conflict and session metadata for editor/agent handoff decisions. - Server identity metadata and discovery helpers so editors can locate the active MCP session server. - CORS restrictions for local editor use. - Keepalive and graceful idle shutdown support; by default the session server exits after 15 minutes without MCP/editor activity. - Unit coverage for session creation, reuse decisions, diagnostics, discovery, CORS, activity refresh, and idle timeout behavior. ## Editor integrations Adds MCP session UI integration to the graph editors so a user can connect an editor to a session started by an MCP server and load the agent-generated graph. Integrated editor surfaces: - Node Material Editor - Node Geometry Editor - Node Particle Editor - Node Render Graph Editor - Flow Graph Editor - GUI Editor - Smart Filters Editor Control The editor side shares a browser connector in `packages/dev/sharedUiComponents/src/mcp/mcpEditorSessionConnection.ts` and each tool adds an MCP session component wired into its property tab/global state flow. The earlier Playground MCP session UI and local `.vscode/mcp.json` test config were removed from the PR so the final branch stays focused on the graph tools and MCP packages. ## Public package Adds a public package under `packages/public/@babylonjs/mcp-servers` that bundles all supported MCP servers into one installable npm package. The package provides: - Direct binaries for each server, such as `babylonjs-nme-mcp-server`, `babylonjs-nge-mcp-server`, `babylonjs-nrge-mcp-server`, `babylonjs-npe-mcp-server`, `babylonjs-flow-graph-mcp-server`, `babylonjs-gui-mcp-server`, and `babylonjs-smart-filters-mcp-server`. - A dispatcher binary, `babylonjs-mcp-servers`, with aliases such as `nme`, `nge`, `npe`, `nrge`, `flow-graph`, `gui`, and `smart-filters`. - A build pipeline that builds the private server workspaces, copies the bundled outputs, preserves source maps, and makes the copied binaries executable. - README instructions for `npx` usage and MCP client configuration. `build:es6:tools` now includes `@babylonjs/mcp-servers` so the public package participates in the ES6/public-package build flow. ## Registry/catalog coverage The branch updates MCP registries and catalogs so the servers expose the current set of relevant authoring blocks and controls. Recent coverage work includes: - Flow Graph keyboard blocks: `FlowGraphKeyDownEventBlock`, `FlowGraphKeyUpEventBlock`, and `FlowGraphIsKeyPressedBlock`. - Node Geometry `BevelBlock`. - GUI serialized controls such as `Scrollbar` and `MultiLine`. - NRGE, NPE, NME, NGE, Smart Filters, and GUI audit notes for intentional omissions such as abstract/base classes, helper classes, gradients, and non-creatable types. - A new instruction file requiring future graph block or GUI control additions/removals/renames to update the matching MCP registry/catalog in the same change. ## Cleanup and maintenance Also included: - Removed the glTF MCP server package because it is no longer needed. - Removed redundant per-server session wrappers in favor of the shared session controller. - Reduced MCP server build warnings. - Fixed block registry gaps and class-name casing issues found during CI/self-review. - Kept unrelated lint/barrel-import churn out of the branch through revert commits. - Added documentation and examples for the new MCP server packages. ## Validation Focused validation performed on the latest branch state: - MCP server unit tests: `249 passed` across the graph/server test files. - Shared editor session server unit tests, including idle timeout and reuse behavior, were added and run during the session-server work. - `npm run build -w @babylonjs/mcp-servers` passes and rebuilds/copies all bundled MCP servers. - Affected private MCP server builds pass through Rollup/TypeScript. - Prettier checks pass for the changed MCP registry/catalog/instruction files. - `git diff --check` passes. - VS Code diagnostics reported no errors on the changed MCP registry/catalog/instruction files.
1 parent abaafbd commit 68df4f5

170 files changed

Lines changed: 71735 additions & 514 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/instructions/mcp-server-coverage.instructions.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
applyTo: "packages/dev/core/src/{FlowGraph,Materials/Node,Meshes/Node,FrameGraph,Particles/Node}/**/*.{ts,tsx},packages/dev/smartFilterBlocks/src/**/*.ts,packages/dev/gui/src/2D/controls/**/*.ts,packages/tools/{flow-graph-mcp-server,nme-mcp-server,nge-mcp-server,nrge-mcp-server,npe-mcp-server,smart-filters-mcp-server,gui-mcp-server}/**/*.{ts,tsx}"
3+
---
4+
5+
# MCP Server Coverage
6+
7+
When adding, removing, or renaming a graph block or GUI control, update the matching MCP server registry or catalog in the same change:
8+
9+
- Flow Graph blocks: `packages/tools/flow-graph-mcp-server/src/blockRegistry.ts`
10+
- Node Material blocks: `packages/tools/nme-mcp-server/src/blockRegistry.ts`
11+
- Node Geometry blocks: `packages/tools/nge-mcp-server/src/blockRegistry.ts`
12+
- Node Render Graph blocks: `packages/tools/nrge-mcp-server/src/blockRegistry.ts`
13+
- Node Particle blocks: `packages/tools/npe-mcp-server/src/blockRegistry.ts`
14+
- Smart Filters blocks: `packages/tools/smart-filters-mcp-server/src/blockRegistry.ts`
15+
- GUI controls: `packages/tools/gui-mcp-server/src/catalog.ts`
16+
17+
Keep the MCP metadata aligned with the runtime block/control class name, serialized class name, public inputs, public outputs, configurable properties, and default serialized properties. If a block/control is intentionally omitted because it is abstract, a base class, editor-internal, or non-creatable, leave the omission clear in nearby registry comments or tests.
18+
19+
After changing coverage, run the affected MCP server tests or build and also rebuild `@babylonjs/mcp-servers` so the public package stays current.

‎package-lock.json‎

Lines changed: 22 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@
6161
"build:umd:tools": "nx run-many --outputStyle=static --target=build --parallel --maxParallel=2 --projects=babylonjs-inspector-legacy,babylonjs-node-editor,babylonjs-node-geometry-editor,babylonjs-node-render-graph-editor,babylonjs-node-particle-editor,babylonjs-gui-editor,babylonjs-inspector,create-babylonjs",
6262
"build:es6": "npm run build:assets:smart-filters && npm run build:es6:libs && npm run build:es6:tools && npm run check:treeshaking-all",
6363
"build:es6:libs": "nx run-many --outputStyle=static --target=build --parallel --maxParallel=6 --projects=@babylonjs/core,@babylonjs/gui,@babylonjs/loaders,@babylonjs/materials,@babylonjs/serializers,@babylonjs/post-processes,@babylonjs/procedural-textures,@babylonjs/viewer,@babylonjs/shared-ui-components,@babylonjs/addons,@babylonjs/accessibility,@babylonjs/ktx2decoder,@babylonjs/smart-filters,@babylonjs/smart-filters-blocks,@babylonjs/lottie-player",
64-
"build:es6:tools": "nx run-many --outputStyle=static --target=build --parallel --maxParallel=2 --projects=@babylonjs/node-editor,@babylonjs/node-geometry-editor,@babylonjs/node-render-graph-editor,@babylonjs/node-particle-editor,@babylonjs/inspector-legacy,@babylonjs/gui-editor,@tools/smart-filters-editor-control,@tools/smart-filters-debugger,@tools/smart-filters-editor,@babylonjs/inspector",
64+
"build:es6:tools": "nx run-many --outputStyle=static --target=build --parallel --maxParallel=2 --projects=@babylonjs/node-editor,@babylonjs/node-geometry-editor,@babylonjs/node-render-graph-editor,@babylonjs/node-particle-editor,@babylonjs/inspector-legacy,@babylonjs/gui-editor,@tools/smart-filters-editor-control,@tools/smart-filters-debugger,@tools/smart-filters-editor,@babylonjs/inspector,@babylonjs/mcp-servers",
6565
"watch:shaders": "build-tools -c build-shaders --global --watch",
6666
"watch:assets": "build-tools -c pa --global --watch",
6767
"watch:assets:smart-filters": "npm run build:assets:smart-filters:prerequisites && node ./packages/dev/smartFilters/dist/utils/buildTools/watchShaders.js ./packages/dev/smartFilterBlocks/src/blocks smart-filters core",

‎packages/dev/sharedUiComponents/src/mcp/mcpEditorSessionConnection.ts‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
const DocumentRoute = "document";
2+
3+
/**
4+
* Options used to open an MCP editor session event stream.
5+
*/
6+
export interface IMcpEditorSessionEventSourceOptions {
7+
/** Base session URL, such as `http://localhost:3001/session/<id>`. */
8+
sessionUrl: string;
9+
/** Called whenever the server sends a document update. */
10+
onDocument: (document: unknown) => void;
11+
/** Called when the server explicitly closes the session. */
12+
onSessionClosed: (reason: string) => void;
13+
/** Called when the EventSource reports a connection error. */
14+
onConnectionError: () => void;
15+
}
16+
17+
/**
18+
* Normalize a user-provided MCP editor session URL.
19+
* @param sessionUrl - The URL entered by the user or returned by an MCP tool.
20+
* @returns The URL without trailing slash characters.
21+
*/
22+
export function NormalizeMcpEditorSessionUrl(sessionUrl: string): string {
23+
return sessionUrl.replace(/\/+$/, "");
24+
}
25+
26+
/**
27+
* Open an EventSource for server-to-editor MCP session updates.
28+
* @param options - Event stream options and callbacks.
29+
* @returns The opened EventSource. Call `CloseMcpEditorSessionEventSource` to disconnect it.
30+
*/
31+
export function OpenMcpEditorSessionEventSource(options: IMcpEditorSessionEventSourceOptions): EventSource {
32+
const normalizedUrl = NormalizeMcpEditorSessionUrl(options.sessionUrl);
33+
const eventSource = new EventSource(`${normalizedUrl}/events`);
34+
35+
eventSource.onmessage = (event) => {
36+
try {
37+
options.onDocument(JSON.parse(event.data));
38+
} catch {
39+
// Ignore malformed document updates and keep the live session open.
40+
}
41+
};
42+
43+
eventSource.addEventListener("session-closed", (event) => {
44+
options.onSessionClosed(ReadSessionClosedReason(event));
45+
eventSource.close();
46+
});
47+
48+
eventSource.onerror = () => {
49+
options.onConnectionError();
50+
eventSource.close();
51+
};
52+
53+
return eventSource;
54+
}
55+
56+
/**
57+
* Close an MCP editor session EventSource if one is active.
58+
* @param eventSource - EventSource to close.
59+
*/
60+
export function CloseMcpEditorSessionEventSource(eventSource: EventSource | null | undefined): void {
61+
eventSource?.close();
62+
}
63+
64+
/**
65+
* Post a document JSON payload to an MCP editor session.
66+
* @param sessionUrl - Base session URL, such as `http://localhost:3001/session/<id>`.
67+
* @param document - Serialized JSON document to send to the MCP server.
68+
* @param legacyDocumentRoute - Optional compatibility route to try when `/document` is unavailable.
69+
* @returns The final fetch response from the standard or compatibility route.
70+
*/
71+
export async function PostMcpEditorSessionDocumentAsync(sessionUrl: string, document: string, legacyDocumentRoute?: string): Promise<Response> {
72+
const normalizedUrl = NormalizeMcpEditorSessionUrl(sessionUrl);
73+
const response = await PostDocumentToRouteAsync(normalizedUrl, DocumentRoute, document);
74+
if (response.ok || !legacyDocumentRoute || (response.status !== 404 && response.status !== 405)) {
75+
return response;
76+
}
77+
78+
return await PostDocumentToRouteAsync(normalizedUrl, legacyDocumentRoute.replace(/^\//, ""), document);
79+
}
80+
81+
async function PostDocumentToRouteAsync(sessionUrl: string, route: string, document: string): Promise<Response> {
82+
const headers = new Headers();
83+
headers.set("Content-Type", "application/json");
84+
85+
return await fetch(`${sessionUrl}/${route}`, {
86+
method: "POST",
87+
headers,
88+
body: document,
89+
});
90+
}
91+
92+
function ReadSessionClosedReason(event: Event): string {
93+
try {
94+
const data = JSON.parse((event as MessageEvent).data);
95+
return typeof data.reason === "string" ? data.reason : "Session closed by MCP server";
96+
} catch {
97+
return "Session closed by MCP server";
98+
}
99+
}

‎packages/public/@babylonjs/mcp-servers/package.json‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{
2+
"name": "@babylonjs/mcp-servers",
3+
"version": "9.8.0",
4+
"description": "Bundled Model Context Protocol servers for Babylon.js authoring workflows",
5+
"type": "module",
6+
"bin": {
7+
"mcp-servers": "dist/babylonjs-mcp-servers.js",
8+
"babylonjs-mcp-servers": "dist/babylonjs-mcp-servers.js",
9+
"babylonjs-flow-graph-mcp-server": "dist/flow-graph-mcp-server.js",
10+
"babylonjs-gui-mcp-server": "dist/gui-mcp-server.js",
11+
"babylonjs-nge-mcp-server": "dist/nge-mcp-server.js",
12+
"babylonjs-nme-mcp-server": "dist/nme-mcp-server.js",
13+
"babylonjs-npe-mcp-server": "dist/npe-mcp-server.js",
14+
"babylonjs-nrge-mcp-server": "dist/nrge-mcp-server.js",
15+
"babylonjs-smart-filters-mcp-server": "dist/smart-filters-mcp-server.js"
16+
},
17+
"files": [
18+
"dist/**/*.*",
19+
"readme.md"
20+
],
21+
"scripts": {
22+
"build": "npm run clean && npm run build:servers && npm run copy:servers",
23+
"build:servers": "npm run build -w @tools/mcp-server-core && npm run build -w @tools/nme-mcp-server && npm run build -w @tools/npe-mcp-server && npm run build -w @tools/nge-mcp-server && npm run build -w @tools/nrge-mcp-server && npm run build -w @tools/gui-mcp-server && npm run build -w @tools/flow-graph-mcp-server && npm run build -w @tools/smart-filters-mcp-server",
24+
"clean": "rimraf dist",
25+
"copy:servers": "node scripts/copyMcpServers.mjs"
26+
},
27+
"engines": {
28+
"node": "^20.19.0 || >=22.13.0 <23.0.0"
29+
},
30+
"keywords": [
31+
"3D",
32+
"javascript",
33+
"html5",
34+
"webgl",
35+
"babylon.js",
36+
"mcp",
37+
"model-context-protocol"
38+
],
39+
"license": "Apache-2.0",
40+
"homepage": "https://www.babylonjs.com",
41+
"repository": {
42+
"type": "git",
43+
"url": "https://github.com/BabylonJS/Babylon.js.git"
44+
},
45+
"bugs": {
46+
"url": "https://github.com/BabylonJS/Babylon.js/issues"
47+
}
48+
}

‎packages/public/@babylonjs/mcp-servers/readme.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Babylon.js MCP Servers
2+
3+
`@babylonjs/mcp-servers` packages the Babylon.js Model Context Protocol servers as executable Node.js binaries. MCP-compatible clients can use these servers to create, edit, validate, import, export, and live-sync Babylon.js authoring graphs.
4+
5+
## Included Servers
6+
7+
| Server | Dispatcher name | Direct binary | Purpose |
8+
| ------------------------ | --------------- | ------------------------------------ | ------------------------------------------------------------------------ |
9+
| Node Material Editor | `nme` | `babylonjs-nme-mcp-server` | Node Material graph authoring and import/export workflows. |
10+
| Node Geometry Editor | `nge` | `babylonjs-nge-mcp-server` | Node Geometry graph authoring and export/import workflows. |
11+
| Node Render Graph Editor | `nrge` | `babylonjs-nrge-mcp-server` | Node Render Graph authoring and render-pipeline export/import workflows. |
12+
| Node Particle Editor | `npe` | `babylonjs-npe-mcp-server` | Node Particle graph authoring and export/import workflows. |
13+
| GUI Editor | `gui` | `babylonjs-gui-mcp-server` | Babylon.js GUI authoring, layout, export/import, and snippet flows. |
14+
| Flow Graph Editor | `flow-graph` | `babylonjs-flow-graph-mcp-server` | Flow Graph authoring and coordinator JSON export/import workflows. |
15+
| Smart Filters Editor | `smart-filters` | `babylonjs-smart-filters-mcp-server` | Smart Filters graph authoring and export/import workflows. |
16+
17+
## Run With npx
18+
19+
Use the dispatcher when you want a compact command:
20+
21+
```sh
22+
npx -y @babylonjs/mcp-servers nme
23+
```
24+
25+
Use direct binaries when your MCP client expects a command name:
26+
27+
```sh
28+
npx -y -p @babylonjs/mcp-servers babylonjs-nme-mcp-server
29+
```
30+
31+
## MCP Client Configuration
32+
33+
Most MCP clients accept a command plus arguments. This example starts the Node Material Editor MCP server through the dispatcher:
34+
35+
```json
36+
{
37+
"mcpServers": {
38+
"babylonjs-node-material": {
39+
"command": "npx",
40+
"args": ["-y", "@babylonjs/mcp-servers", "nme"]
41+
}
42+
}
43+
}
44+
```
45+
46+
This equivalent form uses the direct binary:
47+
48+
```json
49+
{
50+
"mcpServers": {
51+
"babylonjs-node-material": {
52+
"command": "npx",
53+
"args": ["-y", "-p", "@babylonjs/mcp-servers", "babylonjs-nme-mcp-server"]
54+
}
55+
}
56+
}
57+
```
58+
59+
## Live Editor Sessions
60+
61+
The graph MCP servers can start a local editor session server and return a session URL. Paste that URL into the matching Babylon.js editor's MCP session panel to see live updates from the MCP server and to push editor changes back to the MCP server.
62+
63+
The local editor session server binds to `127.0.0.1` by default. It stops when the MCP process exits, when the `stop_session_server` MCP tool is called, or after 15 minutes without MCP/editor activity.
64+
65+
## Local Development
66+
67+
From the Babylon.js repository, build the package with:
68+
69+
```sh
70+
npm run build -w @babylonjs/mcp-servers
71+
```
72+
73+
The build compiles the private MCP server workspaces and copies their bundled `dist/index.js` outputs into this package's `dist/` directory.

‎packages/public/@babylonjs/mcp-servers/scripts/babylonjs-mcp-servers.mjs‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#!/usr/bin/env node
2+
3+
import { spawn } from "node:child_process";
4+
import { dirname, join } from "node:path";
5+
import { fileURLToPath } from "node:url";
6+
7+
const serverAliases = new Map([
8+
["flow-graph", "flow-graph-mcp-server.js"],
9+
["flowgraph", "flow-graph-mcp-server.js"],
10+
["gui", "gui-mcp-server.js"],
11+
["nge", "nge-mcp-server.js"],
12+
["node-geometry", "nge-mcp-server.js"],
13+
["nme", "nme-mcp-server.js"],
14+
["node-material", "nme-mcp-server.js"],
15+
["npe", "npe-mcp-server.js"],
16+
["node-particle", "npe-mcp-server.js"],
17+
["nrge", "nrge-mcp-server.js"],
18+
["node-render-graph", "nrge-mcp-server.js"],
19+
["smart-filters", "smart-filters-mcp-server.js"],
20+
["sfe", "smart-filters-mcp-server.js"],
21+
]);
22+
23+
const [, , serverName, ...serverArguments] = process.argv;
24+
25+
if (!serverName || serverName === "--help" || serverName === "-h") {
26+
printUsage();
27+
process.exit(serverName ? 0 : 1);
28+
}
29+
30+
const serverFile = serverAliases.get(serverName.toLowerCase());
31+
if (!serverFile) {
32+
console.error(`Unknown Babylon.js MCP server "${serverName}".`);
33+
printUsage();
34+
process.exit(1);
35+
}
36+
37+
const distDirectory = dirname(fileURLToPath(import.meta.url));
38+
const child = spawn(process.execPath, [join(distDirectory, serverFile), ...serverArguments], { stdio: "inherit" });
39+
40+
child.on("exit", (code, signal) => {
41+
if (signal) {
42+
process.kill(process.pid, signal);
43+
return;
44+
}
45+
46+
process.exit(code ?? 0);
47+
});
48+
49+
child.on("error", (error) => {
50+
console.error(error.message);
51+
process.exit(1);
52+
});
53+
54+
function printUsage() {
55+
console.error(`Usage: babylonjs-mcp-servers <server>
56+
57+
Servers:
58+
nme Node Material Editor
59+
nge Node Geometry Editor
60+
nrge Node Render Graph Editor
61+
npe Node Particle Editor
62+
gui GUI Editor
63+
flow-graph Flow Graph Editor
64+
smart-filters Smart Filters Editor
65+
`);
66+
}

0 commit comments

Comments
 (0)