Add a command API for Phoenix (natural-language and AI-agent control)

[rx18-eng]

What this is about

Phoenix already has a rich programmatic API. EventDisplay exposes around 56 public methods (loadEvent, nextEvent / previousEvent, highlightObject(uuid), lookAtObject(uuid), getCollection / getCollections, zoomTo, animatePreset, the on / emit event bus, and more).

What it doesn't have is a single, described command surface that something outside Phoenix (a script, another application, or an AI agent) can inspect and say "these are the actions I can perform and how to call them."

This issue is about adding that command layer, then putting a natural-language interface on top of it so users can simply ask Phoenix to do things:

• "show the two muons and rotate to the transverse view"
• "hide the calorimeter"
• "next event"
• "filter tracks above 20 GeV"

instead of navigating the toolbar.

It is also a direct step toward #826 (making Phoenix easier to embed and integrate with other applications), in what I think is the most future-proof form.

Proposal

1. Command / action registry

Wrap the existing public API as named actions, each with a machine-readable description:

• name
• description
• parameters
• return value (where applicable)

This is the durable piece. Even without AI it immediately enables:

• scripting
• embedding
• automation
• testing

2. In-browser natural language control (zero install)

Run a small local model (WebLLM or Transformers.js on WebGPU) entirely in the browser.

The model simply maps natural language onto the existing command registry.

No server.
No API key.
Nothing leaves the browser.

Phoenix stays a static site that can run from GitHub Pages.

3. External agent bridge

Because the actions already exist in a structured schema, expose the same registry through the postMessage bridge discussed in #826.

That would allow external agents (ChatGPT, Claude, etc.) to inspect the available commands and drive a running Phoenix instance.

This is the part that makes Phoenix an event display AI agents can operate.

Why this fits Phoenix

Physics-safe

The AI never performs physics itself.

It only invokes deterministic Phoenix functions and reports reconstructed quantities already produced by the framework. It cannot invent detector data or "guess" physics.

Accessibility and teaching

Students who don't yet know the UI can simply ask for an action instead of hunting through menus.

This also improves accessibility for keyboard and screen-reader users.

Zero install

The in-browser model is lazy-loaded and fully client-side.

Users who never touch the feature pay no startup cost, and Phoenix remains deployable as static files.

Builds toward #826

The command registry is useful even without AI because it gives Phoenix a stable automation surface for embedding, scripting, and integration.

New capability

Current event displays (REve, iSpy, Firebird, HYPATIA, Belle II, etc.) are fundamentally click-driven.

A browser-based event display that can both understand natural-language commands and expose a structured action API for AI agents would be a genuinely new capability.

Rough plan

Phase 1: Command registry

• Introduce a command/action registry wrapping the existing public API.
• Add a command palette that executes registered commands.
• No AI model.

This alone advances #826 and provides immediate value for scripting and embedding.

Phase 2: Natural-language layer

• Add an optional in-browser model (WebLLM or Transformers.js).
• Map free text onto the command registry.
• Use constrained JSON/schema output so generated actions must match the registered command definitions.

Phase 3: External agent bridge

• Expose the command registry over the existing postMessage bridge.
• Allow external AI agents to inspect available actions and invoke them.

Why this is safe

• No existing APIs need to change; the registry wraps the current public methods.
• The AI layer only invokes existing deterministic commands.
• Everything remains optional and lazy-loaded.
• Phoenix continues to work completely offline and without any external service.

Out of scope

• AI that classifies events or identifies particles.
• Server-side inference.
• External API dependencies.
• Changes to the data model, loaders, or detector physics.

References

• WebLLM (browser-side LLM inference): https://github.com/mlc-ai/web-llm
• Transformers.js: https://huggingface.co/docs/transformers.js
• Model Context Protocol (MCP): https://en.wikipedia.org/wiki/Model_Context_Protocol
• Related issue: Allow easier integration of phoenix in other applications #826

[rx18-eng]

@EdwardMoyse @sponce , not totally sure if this is something you guys would want for Phoenix, and i might be missing something here, but the idea has been on my mind so i thought i would just raise it and hear what you think. completely fine to drop it if it does not fit, just wanted to put it out there :)

[sponce]

Very nice proposal I must say. I like a lot the phases 1 and 2, and the fact that it lowers the barrier to use phoenix. This would be great in the context of master classes for example. Would be definitely a huge improvement to phoenix

[rx18-eng]

Thank you so much @sponce again . I am glad that you liked the proposal and the fact that this is gonna be a huge enhancement , I also think this seems to be a good project as well for maybe the upcoming year . Though that may be too early for me to say and I may be wrong here :) . Though in any case I am just hugely motivated and excited to see this feature working in phoenix :) . waiting for @EdwardMoyse thoughts ...

[EdwardMoyse]

Agreed again - a very nice proposal.

[rx18-eng]

@EdwardMoyse @sponce I'm glad that both of you liked the idea ! wdyt shall we keep this maybe as a project for the next year or start working now , though anyways I am highly motivated for this idea to work or whoever works this :)

[sponce]

I would say it's up to you as you will be the one working on it. It clearly makes a nice project but there is no promise we will be more lucky next year than we were this year of course.

[EdwardMoyse]

I think if you want to do this, then go ahead. It's certainly a very interesting topic and I would happy to see it in the next major release of Phoenix.

[rx18-eng]

Thanks @EdwardMoyse @sponce for your sugesstions and yeah I have already started working on both the features and specifically I will try to implemet this one first since its surely a huge enhancemnet to our masterclass features and for our next release

[sponce]

Great. Thanks a lot !

[rx18-eng]

@EdwardMoyse @sponce so I have finally managed to complete about 60 percent of this and a lot of ui improvements and logical implementations and testing are left but just though of of sharing to you guys on how its looking now :) :

Recording.2026-07-12.014644.mp4

[sponce]

Very nice ! looks super cool