Skip to content

Commit 24eec46

Browse files
ulivzulivz
committed
feat(workspace): add embed-frame support for navItems
- Add behavior and autoActive options to WorkspaceNavItem interface - Implement embed-frame rendering with iframe in workspace - Add state management for active embed frames per session - Support auto-activation of embed frames on load - Handle embed frame dismissal when tool content is shown - Update navbar to show active state and handle toggle behavior fix(agent-ui): resolve embed frame state flickering issue - Add proper conditions to prevent unnecessary embed frame clearing - Only clear embed frame for real tool call results, not loading states - Improve auto-activation logic to avoid repeated triggers - Ensure embed frame state stability during tool call processing refactor(agent-ui): redesign workspace state management - Create unified WorkspaceDisplayState with explicit modes (idle, embed-frame, tool-content) - Replace separate embed frame and panel content atoms with single workspace state - Add convenience action atoms (showEmbedFrame, hideEmbedFrame, showToolContent) - Simplify WorkspacePanel rendering with switch-based content display - Update Navbar and ToolHandler to use new state management - Eliminate state conflicts and flickering issues through unified state model debug: add console logs to track embed frame toggle behavior fix(navbar): prevent auto-activation after manual embed frame hide - Add hasAutoActivated flag to track auto-activation state - Prevent re-activation when user manually hides embed frame - Reset flag when session changes - Ensure user intent takes precedence over auto-activation debug: add console logs to track embed frame loading behavior fix(embed-frame): prevent event listener recreation causing loading stuck - Separate src change handling from event listener setup - Set up event listeners only once on mount - Reset loading state when src changes - Prevent race conditions between listener cleanup and setup refactor(embed-frame): simplify component to basic iframe - Remove complex loading state management and event listeners - Let browser handle iframe loading naturally - Reduce bundle size by removing unnecessary complexity - Keep only essential error handling for missing URLs feat(embed-frame): set fixed 1280x958 dimensions - Use fixed dimensions instead of responsive scaling - Prevent contain/cover scaling behavior - Maintain consistent iframe size across different screens feat(embed-frame): add responsive scaling with contain behavior - Center iframe in container with flex layout - Use maxWidth/maxHeight for responsive scaling - Apply contain-style scaling to fit within container - Maintain aspect ratio while ensuring full visibility refactor(embed-frame): use fully responsive iframe - Remove fixed width/height constraints - Let iframe fill container completely - Simplify to basic responsive behavior feat(embed-frame): use smaller 1000x750 default size - Reduce iframe dimensions to prevent VNC content overflow - Use more compact size that fits typical screen layouts - Maintain aspect ratio while reducing footprint feat(embed-frame): add dynamic sizing based on container - Calculate iframe dimensions from container size - Use 80% of container space with 1280x958 max limit - Maintain 4:3 aspect ratio to prevent distortion - Handle window resize events responsively feat(embed-frame): use 100% of container space - Remove 80% limitation for maximum utilization - Use full container width and height - Maintain aspect ratio and size caps feat(embed-frame): add controls and top alignment - Change layout to align iframe at top instead of center - Add Open button in top-right corner to open in new tab - Add fullscreen mode with Close button - Improve UI with proper button styling and icons fix: prevent embed-frame from being overridden by tool results - Add check in ToolHandler to respect active embed-frame state - Update Message components to use new unified state management - Ensure embed-frame stays active until manually dismissed - Allow tool blocks to override embed-frame when clicked fix(agent-ui): scale embed iframe to fit container using transform fix(agent-ui): improve embed frame responsiveness and ui placement - add ResizeObserver for container width change detection - align frame to top center for better positioning - move open-in-new-tab button to title area feat(agent-ui): enhance embed frame ui and navbar styling - redesign EmbedFrameView header with gradient and modern styling - add elegant title with animated pulse indicator - improve open-in-new-tab button with hover effects - implement HDR-style active state for navbar items - add gradient overlays, shadows, and animations to active buttons - enhance mobile menu active states with visual indicators chore: code format chore: nav config feat(agent-ui): support runtime-settings-based autoActive for embed frames - Add function support for WorkspaceNavItem.autoActive - Fetch runtime settings dynamically in Navbar - Maintain backward compatibility with boolean autoActive - Add comprehensive documentation and examples fix(agent-ui): use string expressions for autoActive instead of functions - Change autoActive type from function to string expression - Use new Function() to evaluate expressions safely - Support JSON-compatible configuration format - Add comprehensive expression syntax documentation chore: enhance autoActive expression evaluation docs: update runtime-settings-autoactive documentation - Update design section to reflect string expression approach - Add comprehensive expression syntax examples - Include JSON configuration examples for production use - Add security and troubleshooting sections - Document expression execution mechanism and safety measures docs: update runtime-settings-autoactive documentation chore: tweaks chore: clean unused examples docs: clean chore: tweaks
1 parent 1aa41c8 commit 24eec46

17 files changed

Lines changed: 808 additions & 131 deletions

File tree

‎multimodal/omni-tars/omni-agent/src/index.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,11 +205,14 @@ export default class OmniTARSAgent extends ComposableAgent {
205205
title: 'Code Server',
206206
link: sandboxBaseUrl + '/code-server/',
207207
icon: 'code',
208+
behavior: 'new-page',
208209
},
209210
{
210211
title: 'VNC',
211212
link: sandboxBaseUrl + '/vnc/index.html?autoconnect=true',
212213
icon: 'monitor',
214+
behavior: 'embed-frame',
215+
autoActive: true,
213216
},
214217
],
215218
},

‎multimodal/tarko/agent-ui/docs/runtime-settings-autoactive.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Runtime Settings-based AutoActive Feature
2+
3+
## Overview
4+
5+
The `autoActive` property in `workspace.navItems` supports dynamic evaluation based on `api/v1/runtime-settings` values, enabling conditional auto-activation of embedded frames.
6+
7+
## How It Works
8+
9+
### autoActive Design Logic
10+
11+
1. **Backward Compatibility**: `autoActive?: boolean` works as before
12+
2. **String Expressions**: `autoActive?: string` accepts JavaScript expressions
13+
3. **Dynamic Evaluation**: Navbar fetches runtime settings and evaluates expressions in real-time
14+
4. **Safe Execution**: Uses `new Function()` with limited scope for security
15+
16+
### Runtime Settings Consumption
17+
18+
1. **API Endpoint**: `apiService.getSessionRuntimeSettings(sessionId)` returns current settings
19+
2. **Data Structure**: `{ currentValues: Record<string, any> }` contains user settings
20+
3. **Real-time Updates**: Refetched when session changes
21+
4. **Expression Context**: Runtime settings passed as `runtimeSettings` parameter
22+
23+
## Usage Examples
24+
25+
### Basic Boolean (Backward Compatible)
26+
```json
27+
{
28+
"workspace": {
29+
"navItems": [
30+
{
31+
"title": "Code Server",
32+
"link": "{prefix}/code-server/",
33+
"icon": "code",
34+
"behavior": "embed-frame",
35+
"autoActive": true
36+
}
37+
]
38+
}
39+
}
40+
```
41+
42+
### Dynamic Expression
43+
```json
44+
{
45+
"workspace": {
46+
"navItems": [
47+
{
48+
"title": "VNC",
49+
"link": "{prefix}/vnc/index.html?autoconnect=true",
50+
"icon": "monitor",
51+
"behavior": "embed-frame",
52+
"autoActive": "runtimeSettings.agentMode === 'game'"
53+
}
54+
]
55+
}
56+
}
57+
```
58+
59+
## Real-world Example
60+
61+
From `examples/webui-config.json`:
62+
```json
63+
{
64+
"workspace": {
65+
"navItems": [
66+
{
67+
"title": "Code Server",
68+
"link": "{prefix}/code-server/",
69+
"icon": "code",
70+
"behavior": "embed-frame"
71+
},
72+
{
73+
"title": "VNC",
74+
"link": "{prefix}/vnc/index.html?autoconnect=true",
75+
"icon": "monitor",
76+
"behavior": "embed-frame",
77+
"autoActive": "debug(runtimeSettings) && runtimeSettings.agentMode === 'game'"
78+
}
79+
]
80+
}
81+
}
82+
```

‎multimodal/tarko/agent-ui/examples/webui-config.json‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -158,8 +158,19 @@
158158
],
159159
"workspace": {
160160
"navItems": [
161-
{ "title": "Code Server", "link": "{prefix}/code-server/", "icon": "code" },
162-
{ "title": "VNC", "link": "{prefix}/vnc/index.html?autoconnect=true", "icon": "monitor" }
161+
{
162+
"title": "Code Server",
163+
"link": "{prefix}/code-server/",
164+
"icon": "code",
165+
"behavior": "embed-frame"
166+
},
167+
{
168+
"title": "VNC",
169+
"link": "{prefix}/vnc/index.html?autoconnect=true",
170+
"icon": "monitor",
171+
"behavior": "embed-frame",
172+
"autoActive": "runtimeSettings.agentMode === 'game'"
173+
}
163174
]
164175
},
165176
"guiAgent": {

‎multimodal/tarko/agent-ui/src/common/hooks/useSession.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,13 @@ import { messagesAtom, groupedMessagesAtom } from '../state/atoms/message';
44
import { toolResultsAtom } from '../state/atoms/tool';
55

66
import { sessionFilesAtom } from '../state/atoms/files';
7-
import { isProcessingAtom, activePanelContentAtom, connectionStatusAtom } from '../state/atoms/ui';
7+
import {
8+
isProcessingAtom,
9+
activePanelContentAtom,
10+
connectionStatusAtom,
11+
activeEmbedFrameAtom,
12+
workspaceDisplayStateAtom,
13+
} from '../state/atoms/ui';
814
import { replayStateAtom } from '../state/atoms/replay';
915
import {
1016
loadSessionsAction,
@@ -35,6 +41,8 @@ export function useSession() {
3541
const [isProcessing, setIsProcessing] = useAtom(isProcessingAtom);
3642
const [activePanelContent, setActivePanelContent] = useAtom(activePanelContentAtom);
3743
const [connectionStatus, setConnectionStatus] = useAtom(connectionStatusAtom);
44+
const [activeEmbedFrame, setActiveEmbedFrame] = useAtom(activeEmbedFrameAtom);
45+
const [workspaceDisplayState, setWorkspaceDisplayState] = useAtom(workspaceDisplayStateAtom);
3846

3947
const [replayState, setReplayState] = useAtom(replayStateAtom);
4048

@@ -91,6 +99,8 @@ export function useSession() {
9199
isProcessing,
92100
activePanelContent,
93101
connectionStatus,
102+
activeEmbedFrame,
103+
workspaceDisplayState,
94104

95105
replayState,
96106
sessionMetadata,
@@ -106,6 +116,8 @@ export function useSession() {
106116
abortQuery,
107117

108118
setActivePanelContent,
119+
setActiveEmbedFrame,
120+
setWorkspaceDisplayState,
109121

110122
initConnectionMonitoring,
111123
checkServerStatus,
@@ -122,6 +134,8 @@ export function useSession() {
122134
isProcessing,
123135
activePanelContent,
124136
connectionStatus,
137+
activeEmbedFrame,
138+
workspaceDisplayState,
125139
replayState,
126140
sessionMetadata,
127141
loadSessions,
@@ -133,6 +147,8 @@ export function useSession() {
133147
sendMessage,
134148
abortQuery,
135149
setActivePanelContent,
150+
setActiveEmbedFrame,
151+
setWorkspaceDisplayState,
136152
initConnectionMonitoring,
137153
checkServerStatus,
138154
checkSessionStatus,

‎multimodal/tarko/agent-ui/src/common/state/actions/eventProcessors/handlers/SystemHandler.ts‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { messagesAtom } from '@/common/state/atoms/message';
55
import { sessionPanelContentAtom } from '@/common/state/atoms/ui';
66
import { shouldUpdatePanelContent } from '../utils/panelContentUpdater';
77
import { ChatCompletionContentPartImage } from '@tarko/agent-interface';
8+
import { StandardPanelContent } from '@/standalone/workspace/types/index';
89

910
export class SystemMessageHandler implements EventHandler<AgentEventStream.SystemEvent> {
1011
canHandle(event: AgentEventStream.Event): event is AgentEventStream.SystemEvent {
@@ -91,21 +92,20 @@ export class EnvironmentInputHandler
9192
const currentSessionPanel = currentPanelContent[sessionId];
9293

9394
// Common panel properties
94-
const basePanelContent = {
95+
const basePanelContent: Partial<StandardPanelContent> = {
9596
title: event.description || 'Environment Screenshot',
9697
timestamp: event.timestamp,
97-
originalContent: event.content,
9898
environmentId: event.id,
9999
};
100100

101-
let panelContent = null;
101+
let panelContent: StandardPanelContent | null = null;
102102

103103
if (isFirstEnvironmentInput) {
104104
// First environment input: always show as simple image
105105
panelContent = {
106106
...basePanelContent,
107107
type: 'image',
108-
source: imageContent.image_url.url,
108+
source: imageContent.image_url.url as string,
109109
};
110110
} else if (
111111
currentSessionPanel?.type === 'browser_vision_control' ||
@@ -115,7 +115,7 @@ export class EnvironmentInputHandler
115115
panelContent = {
116116
...basePanelContent,
117117
type: 'browser_vision_control',
118-
source: null,
118+
source: undefined,
119119
title: event.description || 'Browser Screenshot',
120120
};
121121
}

‎multimodal/tarko/agent-ui/src/common/state/actions/eventProcessors/handlers/ToolHandler.ts‎

Lines changed: 77 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,17 @@ import { AgentEventStream, ToolResult, Message } from '@/common/types';
55
import { determineToolRendererType } from '@/common/utils/tool-renderers';
66
import { messagesAtom } from '@/common/state/atoms/message';
77
import { toolResultsAtom, toolCallResultMap } from '@/common/state/atoms/tool';
8-
import { sessionPanelContentAtom } from '@/common/state/atoms/ui';
8+
import {
9+
sessionPanelContentAtom,
10+
showToolContentAtom,
11+
workspaceDisplayStateAtom,
12+
} from '@/common/state/atoms/ui';
913
import { rawToolMappingAtom } from '@/common/state/atoms/rawEvents';
1014
import { toolCallArgumentsCache, streamingToolCallCache } from '../utils/cacheManager';
1115
import { collectFileInfo } from '../utils/fileCollector';
1216
import { normalizeSearchResult } from '../utils/searchNormalizer';
1317
import { shouldUpdatePanelContent } from '../utils/panelContentUpdater';
18+
import { StandardPanelContent } from '@/standalone/workspace/types/panelContent';
1419

1520
export class ToolCallHandler implements EventHandler<AgentEventStream.ToolCallEvent> {
1621
canHandle(event: AgentEventStream.Event): event is AgentEventStream.ToolCallEvent {
@@ -126,58 +131,63 @@ export class ToolResultHandler implements EventHandler<AgentEventStream.ToolResu
126131

127132
// Update panel content only for active session
128133
if (shouldUpdatePanelContent(get, sessionId)) {
134+
const panelContent: StandardPanelContent = {
135+
type: result.type,
136+
source: result.content,
137+
title: result.name,
138+
timestamp: result.timestamp,
139+
toolCallId: result.toolCallId,
140+
error: result.error,
141+
arguments: args,
142+
_extra: result._extra,
143+
};
144+
145+
// Check if embed frame is currently active - if so, don't override it
146+
const workspaceState = get(workspaceDisplayStateAtom);
147+
const isEmbedFrameActive = workspaceState.mode === 'embed-frame';
148+
129149
// Special handling for browser vision control to preserve environment context
130150
if (result.type === 'browser_vision_control') {
131-
set(sessionPanelContentAtom, (prev) => {
132-
const currentContent = prev[sessionId];
133-
if (currentContent && currentContent.type === 'image' && currentContent.environmentId) {
134-
const environmentId = currentContent.environmentId;
135-
136-
return {
137-
...prev,
138-
[sessionId]: {
139-
...currentContent,
140-
type: 'browser_vision_control',
141-
source: event.content,
142-
title: currentContent.title,
143-
timestamp: event.timestamp,
144-
toolCallId: event.toolCallId,
145-
error: event.error,
146-
arguments: args,
147-
originalContent: currentContent.source,
148-
environmentId: environmentId,
149-
processedEnvironmentIds: [environmentId], // Track processed environment IDs
150-
},
151-
};
152-
} else {
153-
return {
154-
...prev,
155-
[sessionId]: {
156-
type: result.type,
157-
source: result.content,
158-
title: result.name,
159-
timestamp: result.timestamp,
160-
toolCallId: result.toolCallId,
161-
error: result.error,
162-
arguments: args,
163-
},
164-
};
151+
const currentContent = get(sessionPanelContentAtom)[sessionId];
152+
if (currentContent && currentContent.type === 'image' && currentContent.environmentId) {
153+
const environmentId = currentContent.environmentId;
154+
155+
const enhancedPanelContent: StandardPanelContent = {
156+
...panelContent,
157+
type: 'browser_vision_control' as const,
158+
environmentId: environmentId,
159+
};
160+
161+
set(sessionPanelContentAtom, (prev) => ({
162+
...prev,
163+
[sessionId]: enhancedPanelContent,
164+
}));
165+
166+
// Only update workspace display state if embed frame is not active
167+
if (!isEmbedFrameActive) {
168+
set(showToolContentAtom, enhancedPanelContent);
169+
}
170+
} else {
171+
set(sessionPanelContentAtom, (prev) => ({
172+
...prev,
173+
[sessionId]: panelContent,
174+
}));
175+
176+
// Only update workspace display state if embed frame is not active
177+
if (!isEmbedFrameActive) {
178+
set(showToolContentAtom, panelContent);
165179
}
166-
});
180+
}
167181
} else {
168182
set(sessionPanelContentAtom, (prev) => ({
169183
...prev,
170-
[sessionId]: {
171-
type: result.type,
172-
source: result.content,
173-
title: result.name,
174-
timestamp: result.timestamp,
175-
toolCallId: result.toolCallId,
176-
error: result.error,
177-
arguments: args,
178-
_extra: result._extra,
179-
},
184+
[sessionId]: panelContent,
180185
}));
186+
187+
// Only update workspace display state if embed frame is not active
188+
if (!isEmbedFrameActive) {
189+
set(showToolContentAtom, panelContent);
190+
}
181191
}
182192
}
183193

@@ -227,7 +237,7 @@ export class StreamingToolCallHandler
227237
streamingToolCallCache.set(toolCallId, newArgs);
228238

229239
// Safe JSON parsing with repair fallback
230-
let parsedArgs: unknown = {};
240+
let parsedArgs: Record<string, unknown> = {};
231241
try {
232242
if (newArgs) {
233243
const repairedJson = jsonrepair(newArgs);
@@ -344,18 +354,29 @@ export class StreamingToolCallHandler
344354
const content = 'content' in parsedArgs ? parsedArgs.content : '';
345355

346356
if (typeof path === 'string') {
357+
const panelContent: StandardPanelContent = {
358+
type: 'file' as const,
359+
source: typeof content === 'string' ? content : '',
360+
title: `Writing: ${path.split('/').pop()}`,
361+
timestamp: event.timestamp,
362+
toolCallId,
363+
arguments: parsedArgs as StandardPanelContent['arguments'],
364+
isStreaming: !isComplete,
365+
};
366+
347367
set(sessionPanelContentAtom, (prev) => ({
348368
...prev,
349-
[sessionId]: {
350-
type: 'file',
351-
source: typeof content === 'string' ? content : '',
352-
title: `Writing: ${path.split('/').pop()}`,
353-
timestamp: event.timestamp,
354-
toolCallId,
355-
arguments: parsedArgs,
356-
isStreaming: !isComplete,
357-
},
369+
[sessionId]: panelContent,
358370
}));
371+
372+
// Check if embed frame is currently active - if so, don't override it
373+
const workspaceState = get(workspaceDisplayStateAtom);
374+
const isEmbedFrameActive = workspaceState.mode === 'embed-frame';
375+
376+
// Only update workspace display state if embed frame is not active
377+
if (!isEmbedFrameActive) {
378+
set(showToolContentAtom, panelContent);
379+
}
359380
}
360381
}
361382

0 commit comments

Comments
 (0)