Interactive Agent Interface — Chat & Command Input

[kovtcharov]

T8: Interactive Agent Interface — Chat & Command Input

Priority: p1

Interactive interface for communicating with running agents — send commands, ask questions, receive responses. Bridges standalone agent experience into the tray app.

New Files

cpp/dotnet/GaiaTray/
├── Forms/
│   ├── AgentInteractForm.cs
│   └── AgentInteractForm.Designer.cs
├── Controls/
│   ├── ChatBubbleControl.cs         # Chat message bubbles
│   └── ToolCallCard.cs              # Inline tool call display
├── Services/
│   ├── AgentInterop.cs              # JSON-RPC agent/chat messaging
│   └── MessageHistory.cs            # Persist conversation history

Interactive UI

┌─────────────────────────────────────────────────────────┐
│  Process Intelligence — Interact                 [─][□][×]│
├─────────────────────────────────────────────────────────┤
│  🤖 I'm monitoring 142 processes. No anomalies.         │
│                                                         │
│  👤 What's using the most memory right now?              │
│                                                         │
│  🤖 Top memory consumers:                               │
│     🔧 list_processes(sort_by="memory") → 142 results  │
│     1. chrome.exe — 1.2 GB                              │
│     2. teams.exe — 890 MB                               │
│     3. code.exe — 654 MB                                │
├─────────────────────────────────────────────────────────┤
│ [Type a message or command...]              [Send] [⚙]  │
│ Quick: [Status] [Top Processes] [Security Scan] [Help]  │
└─────────────────────────────────────────────────────────┘

Features

• Chat interface: Natural language messages via agent/chat JSON-RPC
• Tool call visibility: Inline display of tool calls with expandable results
• Quick action buttons: Per-agent configurable one-click actions
• Command mode: / prefix for direct tool invocation (e.g., /list_processes sort_by=memory)
• Message history: Per-agent conversation persistence
• Multi-agent: Separate windows for each agent
• Agents must run in standalone mode (with LLM) for chat; MCP-only supports /command mode

Protocol (extends MCP)

Uses agent/chat and agent/chat_response JSON-RPC methods (see MCP milestone: "GAIA MCP Protocol Extensions" and "Standalone Mode & LLM Backend").

Blocked by: #415 (T1), #420 (T6), #421 (T7)

See full spec

[m13v]

for the interactive agent interface, the chat input is straightforward but the command mode is where it gets interesting. we have a floating control bar that lets the user switch between voice input and text, and shows the agent's current action in real-time (which tool it is calling, what it sees on screen). the key UX insight was making the agent's perception visible - showing the accessibility tree or screenshot that informed its decision reduces the trust gap significantly.

[m13v]

here's the floating control bar that shows agent perception in real-time: https://github.com/m13v/fazm/blob/main/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift - and the MCP server powering the accessibility tree view: https://github.com/mediar-ai/mcp-server-macos-use/blob/main/Sources/MCPServer/main.swift