feat(gpt-apps): add tools and widget descriptors #375

jakcinmarina · 2025-12-22T15:47:09Z

Summary

This PR introduces the initial GPT Apps backend/tools layer, including new and updated tools for running actors, along with their widget descriptors used by the GPT Apps UI.

Changes

UI mode:

added support for UI widgets via ui query parameter
if value is set to ?ui=openai it will show UI in following tools:
- new async call-actor-widget tool with UI (actor-widget)
- new tool fetch-actor-details-widget to fetch Actor details with UI
- modified search-actors tool to return UI if uiMode is set to openai, if param doesn't exist it will return previously existing output

NOTE
Widget descriptors are included for early review, even though the UI that renders them is not yet part of this PR

Next steps

Add the web scaffold and shared UI components (separate PR incoming)
Introduce the actual widget/page implementations (separate PR incoming)
Address sync vs async execution UX once the full stack is in place

MQ37

Thank you for the PR 👍

I would minimize introduction of the new tools to bare minimum, there is already plenty of tools and new tools with duplicated logic are not great for agents.

So I would remove the fetch-actor-details-widget tools as per my comment and merge the logic into the existing one. Then for the call-actor-async (I would renamed it from call-actor-widget and make it generalized, I mean the description so it is not that widget centric and possibly usable outside ?uiMode) and the related get-actor-run-status I would add the logic to the tool loader that it enforces this async tools pair for the ?uiMode instead of the original call-actor tool so we do not confuse the agent/LLM with complicated descriptions and instructions. We are already working on the new asynchronous tool calls / tasks that were introduces into the MCP protocol but I think it makes sense to keep these tools as the ChatGPT MCP client will probably not support that feature immediately (see #360 and https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/tasks).

BTW is there any way to test this at chatgpt.com?

MQ37 · 2025-12-29T08:47:09Z

tests/integration/suite.ts

+        it.runIf(options.transport === 'stdio')('should use UI_MODE env var when CLI arg is not provided', async () => {
+            client = await createClientFn({ useEnv: true, uiMode: 'openai' });
+            const tools = await client.listTools();
+            expect(tools.tools.length).toBeGreaterThan(0);
+            await client.close();
+        });


question: does this test case test anything related to the ui mode? It just list tools and check if the tool list is not empty. I think it should at least check some meta fields related to openai ui mode.

MQ37 · 2025-12-29T08:57:38Z

src/const.ts

    GET_HTML_SKELETON = 'get-html-skeleton',
+    CALL_ACTOR_WIDGET = 'call-actor-widget',
+    GET_ACTOR_RUN_STATUS = 'get-actor-run-status',
+    FETCH_ACTOR_DETAILS_WIDGET = 'fetch-actor-details-widget',


change: I would use the already existing fetch-actor-details tool and branch the logic based on the uiMode option provided instead of a new tool.

MQ37 · 2025-12-29T09:01:05Z

src/const.ts

+- **Async vs sync Actor tools (${HelperTools.ACTOR_CALL} vs ${HelperTools.CALL_ACTOR_WIDGET}):**
+  Default to \`${HelperTools.ACTOR_CALL}\` (synchronous, no widget) when the user asks to “run/call” and does not request background/progress/UI. Use \`${HelperTools.CALL_ACTOR_WIDGET}\` only when the user wants background/progress/UI. After starting an async run and obtaining runId, do NOT start another async run—only poll with \`${HelperTools.GET_ACTOR_RUN_STATUS}\` using that runId.


question: can the agent event decide what tool to use and when? Does the chatgpt agent/LLM have this information in the context that in runs with the ui mode or not?

MQ37 · 2025-12-29T09:18:09Z

src/const.ts

    DOCS_SEARCH = 'search-apify-docs',
    DOCS_FETCH = 'fetch-apify-docs',
    GET_HTML_SKELETON = 'get-html-skeleton',
+    CALL_ACTOR_WIDGET = 'call-actor-widget',


change: I would rename this tool to call-actor-async and I would make it more general than strictly widget / ui oriented meaning I would change the tool description so It can be used generally. Then I would also change the tool loading logic for ?uiMode=openai (can be done in tool-loader.ts) so that the call-actor tool is automatically swapped in all cases for call-actor-async (this tool) when in ui mode so we don't need to do all the complicated instructions and every Actor call in the ui mode is async by default which I think is ok and simplifies the solution. What do you think? So this means that with this tool the get-actor-run-status one would have to also be present otherwise it would not work so the logic would need to reflect this.

MQ37 · 2025-12-29T09:23:28Z

src/tools/actor.ts

+**SYNCHRONOUS / NO WIDGET**: Waits for the Actor to finish and returns results in the response.
+
+**WHEN TO USE THIS TOOL:**
+- User wants immediate results (e.g., "get results", "fetch data", "scrape now")
+- User needs the output right away
+- Quick-running Actors where waiting is acceptable
+- User doesn't mention "start", "run in background", "monitor progress", "widget", or "UI"
+
+**WHEN NOT TO USE THIS TOOL:**
+- User explicitly wants to "start" a run or mentions "background" or "async"
+- Long-running Actors where waiting would timeout
+- User wants to monitor progress in a UI widget
+→ In these cases, use ${HelperTools.CALL_ACTOR_WIDGET} instead
+
+This tool stays synchronous: it waits for completion and returns the results in the response. Do not pair it with ${HelperTools.CALL_ACTOR_WIDGET} for the same task.


change: If we decide to go with the tool swapping logic for the call-actor-async tool I would remove these changes since the description is already too long and sometimes the agent/LLM does not respect that because of the length.

MQ37 · 2025-12-29T09:23:47Z

src/tools/actor.ts

+    _meta: {
+        'openai/toolInvocation/invoking': 'Calling Actor synchronously...',
+        'openai/toolInvocation/invoked': 'Actor run finished (sync)',
+        'openai/widgetAccessible': false,
+        'openai/resultCanProduceWidget': false,
+        // TODO: replace with real CSP domains
+        'openai/widgetCSP': {
+            connect_domains: ['https://api.example.com'],
+            resource_domains: ['https://persistent.oaistatic.com'],
+        },
+        'openai/widgetDomain': 'https://chatgpt.com',
+    },


change: if we decide for the tool swapping logic we should remove this.

MQ37 · 2025-12-29T09:23:56Z

src/tools/actor.ts

 USAGE:
 - Always use dedicated tools when available (e.g., ${actorNameToToolName('apify/rag-web-browser')})
 - Use the generic call-actor tool only if a dedicated tool does not exist for your Actor.
+- PREFER this tool when user wants immediate results


change: same here

MQ37 · 2025-12-29T09:25:23Z

src/mcp/server.ts

+                        // TODO: replace with real CSP domains
+                        'openai/widgetCSP': {
+                            connect_domains: ['https://api.example.com'],
+                            resource_domains: ['https://persistent.oaistatic.com'],


it will be hosted at https://mcp.apify.com where the remote version of the Apify MCP server is.

jirispilka

Thank you, I can see you have put a lot of work into this 💪🏻 . I appreciate it!

I reviewed it only partially, the main concern is about the new tools.
Do we really need them? :)

getActorRunStatus – we don't need IMO, we have getActorRun already. We can:

Always return structured content (not raw JSON)
Conditionally include widget metadata when uiMode === 'openai'`

Similarly for call-actor vs call-actor-widget: I’ would prefer a single call-actor tool that:

supports both sync and async modes (e.g. mode: "sync" | "async", we can over-ride defaults based on uiMode)
note that two-step (info, call) flow is gonna be removed in the next PR by @MQ37

That way we don't multiply tools just for UI concerns.
Ideally, I would like to leverage the MCP tasks (all the Actor calls are async). Client starts task and polls for results.

But I'm afraid this won't be supported by OpenAI anytime soon.

Let's discuss it on a call, I'll discuss with @drobnikj to move this forward.

jirispilka · 2026-01-07T09:23:34Z

src/mcp/server.ts


    private setupResourceHandlers(): void {
        this.server.setRequestHandler(ListResourcesRequestSchema, async () => {
+            const resources = [];


Just a note, no action needed. The mcp/server.ts is super loong and we'll need to refactor it anyway. We'll add a new dir with resources.ts to make it manageable.

feat(gpt-apps): add tools and widget descriptors

94ed1b7

jakcinmarina requested review from MQ37 and jirispilka December 22, 2025 15:47

jakcinmarina self-assigned this Dec 22, 2025

jakcinmarina requested a review from drobnikj December 22, 2025 18:17

MQ37 requested changes Dec 29, 2025

View reviewed changes

jirispilka reviewed Jan 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(gpt-apps): add tools and widget descriptors #375

feat(gpt-apps): add tools and widget descriptors #375

jakcinmarina commented Dec 22, 2025

Uh oh!

MQ37 left a comment •

edited

Loading

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

MQ37 Dec 29, 2025

Uh oh!

jirispilka left a comment •

edited

Loading

Uh oh!

jirispilka Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		- Async vs sync Actor tools (${HelperTools.ACTOR_CALL} vs ${HelperTools.CALL_ACTOR_WIDGET}):
		Default to \`${HelperTools.ACTOR_CALL}\` (synchronous, no widget) when the user asks to “run/call” and does not request background/progress/UI. Use \`${HelperTools.CALL_ACTOR_WIDGET}\` only when the user wants background/progress/UI. After starting an async run and obtaining runId, do NOT start another async run—only poll with \`${HelperTools.GET_ACTOR_RUN_STATUS}\` using that runId.

feat(gpt-apps): add tools and widget descriptors #375

Are you sure you want to change the base?

feat(gpt-apps): add tools and widget descriptors #375

Conversation

jakcinmarina commented Dec 22, 2025

Uh oh!

MQ37 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jirispilka left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MQ37 left a comment •

edited

Loading

jirispilka left a comment •

edited

Loading