Add Jupyternaut adapter for JupyterLite integration #21463

guerler · 2025-12-15T16:48:44Z

This PR adds a minimal, self contained OpenAI Chat Completions compatibility endpoint to support JupyterLite via the Jupyternaut client.

The endpoint is intentionally narrow, stateless, and marked unstable. It exists solely to provide protocol compatibility for Galaxy managed integrations that already ship with an OpenAI compatible client. It is not a generic OpenAI proxy and is not intended for arbitrary external consumers.

Galaxy retains full control over scope and policy by injecting the system prompt server side and explicitly ignoring system prompts supplied in the request payload. Requests are forwarded to the configured AI backend using existing Galaxy configuration, authentication, and rate limiting.

Resolves galaxyproject/galaxy-visualizations#123

How to test the changes?

(Select all options that apply)

I've included appropriate automated tests.
This is a refactoring of components with existing test coverage.
Instructions for manual testing are as follows:
1. Make sure pip install openai is installed in your .venv
2. Add the following to your Galaxy configuration: ai_api_key, ai_api_base_url and ai_model
3. Navigate to JupyterLite and access Jupyternaut by clicking on the chat icon labeled Chat with AI assistant

License

I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

lib/galaxy/webapps/galaxy/api/compat.py

bgruening · 2025-12-20T17:08:34Z

Tested it with our University LLM and it works great!

mvdbeek

This is cool, we will likely need something like this!

lib/galaxy/webapps/galaxy/api/compat.py

mvdbeek · 2025-12-22T14:37:05Z

lib/galaxy/webapps/galaxy/api/compat.py

+"""
+
+# Jupyternaut default system prompt from: https://github.com/jupyterlite/ai
+SYSTEM_PROMPT = "You are Jupyternaut, an AI coding assistant built specifically for the JupyterLab environment.\n\n## Your Core Mission\nYou're designed to be a capable partner for data science, research, and development work in Jupyter notebooks. You can help with everything from quick code snippets to complex multi-notebook projects.\n\n## Your Capabilities\n**📁 File & Project Management:**\n- Create, read, edit, and organize Python files and notebooks\n- Manage project structure and navigate file systems\n- Help with version control and project organization\n\n**📊 Notebook Operations:**\n- Create new notebooks and manage existing ones\n- Add, edit, delete, and run cells (both code and markdown)\n- Help with notebook structure and organization\n- Retrieve and analyze cell outputs and execution results\n\n**🧠 Coding & Development:**\n- Write, debug, and optimize Python code\n- Explain complex algorithms and data structures\n- Help with data analysis, visualization, and machine learning\n- Support for scientific computing libraries (numpy, pandas, matplotlib, etc.)\n- Code reviews and best practices recommendations\n\n**💡 Adaptive Assistance:**\n- Understand context from your current work environment\n- Provide suggestions tailored to your specific use case\n- Help with both quick fixes and long-term project planning\n\n## How I Work\nI can actively interact with your JupyterLab environment using specialized tools. When you ask me to perform actions, I can:\n- Execute operations directly in your notebooks\n- Create and modify files as needed\n- Run code and analyze results\n- Make systematic changes across multiple files\n\n## My Approach\n- **Context-aware**: I understand you're working in a data science/research environment\n- **Practical**: I focus on actionable solutions that work in your current setup\n- **Educational**: I explain my reasoning and teach best practices along the way\n- **Collaborative**: Think of me as a pair programming partner, not just a code generator\n\n## Communication Style & Agent Behavior\n- **Conversational**: I maintain a friendly, natural conversation flow throughout our interaction\n- **Progress Updates**: I write brief progress messages between tool uses that appear directly in our conversation\n- **No Filler**: I avoid empty acknowledgments like \"Sounds good!\" or \"Okay, I will...\" - I get straight to work\n- **Purposeful Communication**: I start with what I'm doing, use tools, then share what I found and what's next\n- **Active Narration**: I actively write progress updates like \"Looking at the current code structure...\" or \"Found the issue in the notebook...\" between tool calls\n- **Checkpoint Updates**: After several operations, I summarize what I've accomplished and what remains\n- **Natural Flow**: My explanations and progress reports appear as normal conversation text, not just in tool blocks\n\n## IMPORTANT: Always write progress messages between tools that explain what you're doing and what you found. These should be conversational updates that help the user follow along with your work.\n\n## Technical Communication\n- Code is formatted in proper markdown blocks with syntax highlighting\n- Mathematical notation uses LaTeX formatting: \\\\(equations\\\\) and \\\\[display math\\\\]\n- I provide context for my actions and explain my reasoning as I work\n- When creating or modifying multiple files, I give brief summaries of changes\n- I keep users informed of progress while staying focused on the task\n\n## Multi-Step Task Handling\nWhen users request complex tasks that require multiple steps (like \"create a notebook with example cells\"), I use tools in sequence to accomplish the complete task. For example:\n- First use create_notebook to create the notebook\n- Then use add_code_cell or add_markdown_cell to add cells\n- Use set_cell_content to add content to cells as needed\n- Use run_cell to execute code when appropriate\n\nAlways think through multi-step tasks and use tools to fully complete the user's request rather than stopping after just one action.\n\nReady to help you build something great! What are you working on?\n\nIMPORTANT: Follow this message flow pattern for better user experience:\n\n1. FIRST: Explain what you're going to do and your approach\n2. THEN: Execute tools (these will show automatically with step numbers)\n3. FINALLY: Provide a concise summary of what was accomplished\n\nExample flow:\n- \"I'll help you create a notebook with example cells. Let me first create the file structure, then add Python and Markdown cells.\"\n- [Tool executions happen with automatic step display]\n- \"Successfully created your notebook with 3 cells: a title, code example, and visualization cell.\"\n\nGuidelines:\n- Start responses with your plan/approach before tool execution\n- Let the system handle tool execution display (don't duplicate details)\n- End with a brief summary of accomplishments\n- Use natural, conversational tone throughout\n\nCOMMAND DISCOVERY:\n- When you want to execute JupyterLab commands, ALWAYS use the 'discover_commands' tool first to find available commands and their metadata, with the optional query parameter.\n- The query should typically be a single word, e.g., 'terminal', 'notebook', 'cell', 'file', 'edit', 'view', 'run', etc, to find relevant commands.\n- If searching with a query does not yield the desired command, try again with a different query or use an empty query to list all commands.\n- This ensures you have complete information about command IDs, descriptions, and required arguments before attempting to execute them. Only after discovering the available commands should you use the 'execute_command' tool with the correct command ID and arguments.\n\nTOOL SELECTION GUIDELINES:\n- For file operations (create, read, write, modify files and directories): Use dedicated file manipulation tools\n- For general JupyterLab UI interactions (opening panels, running commands, navigating interface): Use the general command tool (execute_command)\n- Examples of file operations: Creating notebooks, editing code files, managing project structure\n- Examples of UI interactions: Opening terminal, switching tabs, running notebook cells, accessing menus\n"


Don't we want a simple proxy here ? Let jupyternaut send this ?

This is intentionally not a generic OpenAI proxy. Injecting the system prompt here ensures the endpoint cannot be reused as a general purpose chat endpoint and keeps its scope limited to the intended Jupyternaut use case.

But you can alway inject a prompt (https://github.com/swisskyrepo/PayloadsAllTheThings/blob/master/Prompt%20Injection/README.md#system-prompt). Surely we're not adding one route per consumer plugin ?

If the risk is resource abuse ... add a rate limiter.

I want to Galaxy to remain in control of the full system prompt and scope of the endpoint. Jupyternaut is a Galaxy-managed integration, not a generic client. Injecting the prompt server side and ignoring system prompts provided in the payload makes this explicit and prevents the endpoint from becoming a general-purpose chat surface. The adapter itself is reusable, but both the route and the prompt are intentionally integration owned.

I removed the hardcoded Jupyternaut prompt from the endpoint and now load the prompt from the JupyterLite visualization XML instead. The behavior is unchanged, but this moves prompt ownership to the visualization definition. Galaxy still remains in control of the full system prompt.

lib/galaxy/webapps/galaxy/api/compat.py

lib/galaxy/tool_util/verify/interactor.py

test/integration/test_plugins.py

test/integration/test_compat.py

mvdbeek · 2025-12-22T19:03:51Z

lib/galaxy/webapps/galaxy/fast_app.py

 from galaxy.webapps.base.webapp import config_allows_origin
 from galaxy.webapps.openapi.utils import get_openapi

+limiter = Limiter(key_func=get_remote_address)


you'll have to limit per user

Like this: e4ba8b8?

guerler · 2025-12-24T23:27:00Z

I believe all issues have been addressed. The remaining failing tests are unrelated.

guerler · 2026-01-02T06:12:41Z

@mvdbeek Thanks for the detailed review. All comments have been addressed. Please let me know if there is anything else to adjust.

jmchilton · 2026-01-05T01:47:37Z

client/src/api/schema/schema.ts

        patch?: never;
        trace?: never;
    };
+    "/api/ai/plugins/{plugin_name}/chat/completions": {


Can we do api/plugins/{plugin_id}/chat/completions instead? I think that properly scopes my mental model of what is going on here way better.

Can you also move the implementation into api/plugins.py? jobs.py is an example of us mixing older and FastAPI style endpoints in the same file (basically just put the implementations in two separate classes).

Yes, that makes sense. I switched the endpoint to /api/plugins/{plugin_id}/chat/completions, which better reflects the plugin scoped model, and I also moved the implementation into api/plugins.py. Thank you!

guerler · 2026-01-05T22:39:38Z

Failing tests are unrelated.

- Add test visualization plugin at test/integration/test_visualization_plugins/ - Configure visualization_plugins_directory in test config - Remove get_plugin patches from all tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>

jmchilton

I improved the tests with https://github.com/jmchilton/claude-commands/blob/main/py-challenge-patches.md. I'd obviously like to see some convergence between this and the other AI code but I think it is early days for both approaches and they both have their pros and cons - I'm happy to let them both evolve naturally for a while and learn from each other. I think stuff like the rate limiting here and the validation of various pratical payload issues is really great and should be features of the other APIs and tests I think.

guerler added this to the 26.0 milestone Dec 15, 2025

guerler added kind/feature area/visualizations area/client labels Dec 15, 2025

guerler force-pushed the jupyterlite_adapter branch 3 times, most recently from cd89a43 to a6716fb Compare December 18, 2025 11:14

github-advanced-security bot found potential problems Dec 18, 2025

View reviewed changes

lib/galaxy/webapps/galaxy/api/compat.py Fixed Show fixed Hide fixed

guerler force-pushed the jupyterlite_adapter branch 2 times, most recently from 20b1d6a to b0fbefa Compare December 19, 2025 13:04

github-advanced-security bot found potential problems Dec 19, 2025

View reviewed changes

lib/galaxy/webapps/galaxy/api/compat.py Fixed Show fixed Hide fixed

guerler force-pushed the jupyterlite_adapter branch 13 times, most recently from ddb186a to 438fdda Compare December 20, 2025 10:05

guerler marked this pull request as ready for review December 20, 2025 12:20

guerler force-pushed the jupyterlite_adapter branch from 438fdda to 377c897 Compare December 20, 2025 16:49

guerler requested a review from jmchilton December 20, 2025 20:37

guerler marked this pull request as draft December 21, 2025 12:31

guerler force-pushed the jupyterlite_adapter branch from 377c897 to dd20320 Compare December 21, 2025 14:49

guerler force-pushed the jupyterlite_adapter branch 3 times, most recently from a8e6098 to 7a53e3e Compare December 22, 2025 14:26

mvdbeek reviewed Dec 22, 2025

View reviewed changes

guerler force-pushed the jupyterlite_adapter branch 13 times, most recently from 4462313 to 8c1bcec Compare December 24, 2025 18:57

Add jupyterlite ai adapter, collect prompt form visualization xml

0f68492

guerler force-pushed the jupyterlite_adapter branch from 8c1bcec to 0f68492 Compare December 25, 2025 18:16

jmchilton reviewed Jan 5, 2026

View reviewed changes

guerler added 2 commits January 5, 2026 08:59

Move adapater to plugins api

0ed2914

Remove unused import

abc1312

jmchilton approved these changes Jan 6, 2026

View reviewed changes

guerler merged commit 962fe0a into galaxyproject:dev Jan 6, 2026
57 of 66 checks passed

guerler deleted the jupyterlite_adapter branch January 6, 2026 17:26

Add Jupyternaut adapter for JupyterLite integration #21463

Add Jupyternaut adapter for JupyterLite integration #21463

Uh oh!

Conversation

guerler commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test the changes?

License

Uh oh!

Uh oh!

Uh oh!

bgruening commented Dec 20, 2025

Uh oh!

mvdbeek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mvdbeek Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guerler Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guerler commented Dec 24, 2025

Uh oh!

guerler commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guerler commented Jan 5, 2026

Uh oh!

jmchilton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

guerler commented Dec 15, 2025 •

edited

Loading

mvdbeek Dec 22, 2025 •

edited

Loading

guerler Dec 23, 2025 •

edited

Loading

guerler commented Jan 2, 2026 •

edited

Loading