M1: Browser use in the platform-hosted virtual desktop

ARCHITECTURE.md

@@ -25,7 +25,7 @@ This file is the cross-system architecture index. Detailed designs live in domai
 | Workflow orchestration engine               | [Workflow Orchestration Engine](#workflow-orchestration-engine) (this file)                        |
 | Watch sessions                              | [Watch Sessions](#watch-sessions) (this file)                                                      |
 | Screen annotation                           | [Screen Annotation](#screen-annotation) (this file)                                                |
-| Notification sender avatars                  | [Notification Sender Avatars](#notification-sender-avatars) (this file)                            |
+| Notification sender avatars                 | [Notification Sender Avatars](#notification-sender-avatars) (this file)                            |
 | Workflow authoring guide                    | [`assistant/docs/workflows.md`](assistant/docs/workflows.md)                                       |
 | Workflow manual testing runbook             | [`assistant/docs/workflows-testing.md`](assistant/docs/workflows-testing.md)                       |
 | Service communication matrix                | [`docs/service-communication-matrix.md`](docs/service-communication-matrix.md)                     |
@@ -805,6 +805,30 @@ graph LR
     WAKE -->|"assistant report only"| CONV
 ```
 
+## Assistant Desktop Stream
+
+A containerized assistant can serve an interactive desktop on demand. The setup-capable modal installs desktop-only system packages and Google Chrome after the guardian clicks **Install desktop**. Authenticated direct stream requests also start or join the same background setup for clients without the setup UI. A client that times out during installation can reconnect after setup finishes; a disconnected viewer does not start a desktop process tree. `GET /v1/desktop/setup` checks readiness without installing anything; `POST` starts one shared background installation. Both flat and assistant-scoped gateway paths require guardian authentication and proxy to a gateway-service-only runtime route. `assistant:self:desktop` sync invalidations refresh setup status as installation progresses, and a reopened modal or reconnected event stream refetches it. Older assistants returning 404 retain the direct streaming flow.
+
+`desktop-dependencies.ts` installs the X server, window manager, dock, compositor, clipboard bridge, terminal, wallpaper setter (`feh`), fonts and Chrome libraries through image-root `/usr/bin/apt-get` with `--no-upgrade` and `--no-remove`. The installer uses only system command paths, bypassing Kata persistent-apt wrappers so binaries, X assets and shared libraries live in the same filesystem. Packages must be installed again when a Kata save or container recreation discards that root; Chrome and its profile remain in persistent storage. Google Chrome is an exact-version, SHA-256-verified download for Linux x64 or ARM64, extracted under the assistant's internal external-dependency directory. Extraction does not run Chrome package scripts, register its repository or change the system's default browser. Successful setup is recorded only after Chrome runs; readiness also checks the desktop binaries and X fonts so a recreated container offers setup again. Configured `NODE_EXTRA_CA_CERTS` are combined with the system CA bundle for apt and Chrome downloads without changing system trust. Installation errors remain retryable. The base image carries no desktop-only packages or baked browser; existing browser-tool and PDF installations retain their own Playwright behavior.
+
+`DesktopSessionManager` owns `Xtigervnc` on display `:99` with VNC on `localhost:5999`, `openbox`, `xcompmgr`, `plank`, `tigervncconfig` and Google Chrome. Chrome starts directly with a loopback-only CDP port, using the existing `data/desktop-profile` directory. The assistant verifies that the listener belongs to its managed Chrome process before connecting. The dock configuration and launcher paths remain under `data/desktop-panel`. Children receive only an allowlisted environment. One viewer holds the slot at a time; the tree lingers five minutes after disconnect. Required child failures and browser crash loops tear down the tree, while cosmetic dock/compositor failures only log. Shutdown uses SIGTERM followed by SIGKILL after a two-second grace, and a subsequent start waits for teardown. The `assistant-desktop` flag and `IS_CONTAINERIZED` gate both setup and streaming. The companion platform PR adds authenticated desktop routing through velay; it does not change pod memory or shared-memory provisioning.
+
+Plank runs under `dbus-run-session` with private XDG configuration/data paths and the keyfile settings backend; its BAMF matcher shares that session bus and process group. The manager retains that group after a cosmetic dock exit and clears surviving applications at teardown. Chrome and Terminal launchers use their real X11 identities, with Chrome's official packaged icon and profile-specific identity. Pinned launchers represent running applications, with Plank providing focus, minimize/restore, window selection, and explicit new-window gestures. Default pins and preferences are published atomically on first use; subsequent starts refresh managed launcher paths while preserving user customization. Migration 154 removes generated tint2 files; an existing installation missing Plank or BAMF requests the same on-demand setup as a fresh one.
+
+Before launching Chrome or exposing its dock launcher, the Linux container session writes `CommandLineFlagSecurityWarningsEnabled=false` to `/etc/opt/chrome/policies/managed/vellum-desktop.json`. The extracted Google Chrome binary reads this system policy directory independently of its install location. This idempotent startup step covers fresh and previously installed desktops, preserves other policy files and unrelated values, and logs policy write failures without blocking the desktop. The policy hides command-line security warnings after Chrome restarts; it does not re-enable the sandbox or change launch flags. Host Chrome policies are untouched.
+
+`desktop-wallpaper.ts` reads the current avatar manifest and reuses the notification avatar renderer for character and uploaded images. It composites the avatar over a dark, accent-tinted background with subtle rings and raised lettering reading `[assistant name] OS`. The wordmark reads the existing identity name, falls back to `Vellum OS` for unset identities, escapes XML, and measures text to fit long names using the desktop setup fonts. The session manager refreshes `data/desktop-panel/wallpaper.png` on desktop start and viewer reconnect, then runs `feh --no-fehbg --bg-fill` on the existing display. Rendering and application are cosmetic and do not delay Chrome or fail the stream. A missing or unreadable avatar leaves the gradient and rings; unavailable native rendering leaves the X background unchanged. Reconnects during a render queue one fresh render of the latest avatar and discard the superseded result. Generation checks discard renders and queued refreshes after teardown. Wallpaper installation remains part of the on-demand desktop setup under the existing flag.
+
+**Transport.** `/v1/desktop/stream` is a pure RFB byte pipe: after the upgrade, every frame in both directions is binary and `DesktopStreamBridge` (`desktop-stream-bridge.ts`) pumps it to and from the VNC port, buffering client bytes that arrive before that socket is up. Nothing is signaled in-band; outcomes are close codes in the application range so they can neither collide with velay's own `1013` nor be remapped by the gateway's velay bridge. The manager decides them and the bridge only relays (`DesktopLoss`, through the viewer-slot result, `onDesktopLost`, or the `DesktopStartError` a start rejects with): `4008` desktop disabled or unsupported on this daemon, `4013` another viewer holds the slot, `4011` the desktop failed to start, died under the viewer, or the viewer fell too far behind (a dropped `ws.send`), and the standard `1001` when the runtime is shutting down, whether the socket arrived after shutdown began or a live viewer is cut off by it. On the managed path velay's bridge carries the runtime's `1001` as `4001` and the gateway's `1011` as `4011`, both of which the panel treats as retryable endings. The daemon upgrade is gated exactly as `/v1/watch/stream` (private-network peer and origin, gateway service token, one shared `upgradeRuntimeStream` path); the feature gate runs after the upgrade because the gateway relays close codes, not HTTP statuses, to the browser. VNC needs no password: only same-pod processes can reach the loopback port, and the authenticated upgrade is the only bridge to it.
+
+### Assistant desktop computer use
+
+The bundled `assistant-desktop` skill runs a container-local executor against the same X11 display and Chrome profile the viewer sees. The default-off `assistant-desktop` flag gates the viewer, skill discovery, and control execution. Control also requires an identified guardian conversation; no connected host desktop client is needed. Existing host computer-use tools keep their connected-client routing.
+
+`DesktopControl` serializes browser CLI and screenshot/input operations, binds ownership to one conversation and actor, and consumes an observation ID for each X11 action. The X11 executor uses `xwd` (MIT) with in-process PNG encoding for screenshots and `xdotool` (BSD-3-Clause) for input; both are installed by desktop setup. Commands run with a fixed display, system binary paths, a restricted environment, bounded output and timeouts. Text travels over stdin. `assistant browser --desktop` borrows a scoped direct CDP client inside the same lease and dispatches through the existing browser operation handlers. Snapshot references use a separate desktop namespace and are invalidated on navigation, tab changes, release and native desktop input. A page overlay animates the CDP pointer in the stream; X11 input remains available for browser chrome and native apps. See [desktop browser CLI](assistant/docs/desktop-browser-cli.md).
+
+An automation slot keeps the desktop alive independently of the viewer. While the assistant owns input, TigerVNC rejects viewer keyboard, pointer, clipboard and resize requests. The modal also becomes view-only. Guardian-only gateway routes proxy `GET/POST /v1/desktop/control` to the runtime. **Take control** cancels pending automation, releases keys/buttons and restores viewer input; **Allow assistant** permits a fresh observation to acquire another session. Cancellation, desktop loss, inactivity and `done` release the automation slot. State changes use the existing `assistant:self:desktop` invalidation tag. No control state or screenshots are persisted outside normal tool history.
+
 ## Screen Annotation
 
 The assistant points at things on the screen the user is sharing with a call, so they can go and do the thing themselves. It is the opposite errand from computer use and shares none of its actions: nothing here clicks, types or takes the mouse. The bundled `screen-annotation` skill (`assistant/src/config/bundled-skills/screen-annotation/`) offers two tools, `screen_point_at` and `screen_clear_marks`, and a request replaces whatever is currently drawn. Clearing is its own tool because it is a thing the model decides to do rather than an argument shape it has to remember; on the wire it is the same request carrying no marks.

assistant/docs/desktop-browser-cli.md

@@ -0,0 +1,30 @@
+# Streamed desktop browser CLI
+
+`assistant browser --desktop` uses the browser CLI's shared operation handlers against the Chrome process owned by `DesktopSessionManager`. It requires the existing `assistant-desktop` flag, completed desktop setup and an identified guardian conversation. The persistent profile remains `data/desktop-profile`; no data migration or extension installation is required.
+
+```mermaid
+flowchart LR
+  CLI[assistant browser --desktop] --> IPC[Existing browser IPC routes]
+  IPC --> Lease[DesktopControl conversation and actor lease]
+  Lease --> Shared[Shared browser operation handlers]
+  Shared --> CDP[Scoped desktop CDP client]
+  CDP --> Chrome[Managed Chrome on display :99]
+  Chrome --> Stream[Existing desktop stream]
+  X11[desktop_control] --> Lease
+  Lease --> Native[X11 screenshots and native input]
+  Native --> Stream
+```
+
+Chrome and its dock launcher share the managed profile and loopback debug port. Discovery checks `/proc` socket ownership against the managed browser PID, refuses redirects and validates the returned browser WebSocket endpoint. The connection stays within the container. No CDP endpoint or token is exposed to the renderer.
+
+The desktop client bypasses personal-browser discovery, extension reconnect waits and backend fallback. It reuses the existing AX snapshot, DOM element resolution, mouse, keyboard, extraction and credential-fill implementations. Operation-scoped clients borrow the lease's connection; disposing one does not release the lease. `detach`, `close`, takeover, cancellation, errors and idle expiry release control. Only `tabs close` closes a Chrome tab.
+
+Tab IDs are ephemeral numeric aliases for this managed browser's CDP target IDs. They are never personal extension tab IDs. Initial attachment selects an existing HTTP(S) page or opens a blank tab. Use `tabs list` and `tabs select --tab-id <id>` to choose explicitly. Navigation, tab selection, document replacement and release clear the desktop snapshot map. Personal-browser snapshots use a separate namespace. Switching between CLI browser actions and native desktop input invalidates the other interface's observations.
+
+CDP mouse events use page viewport CSS coordinates. Before dispatching them, the client animates a purple arrow overlay to the same point. The overlay is excluded from accessibility and hit testing. It appears in the existing desktop stream and is removed on release. It does not move the OS pointer, appear in browser toolbar UI or visualize every programmatic DOM operation. Native dialogs and other applications use X11 input through `desktop_control`.
+
+The client records key and mouse presses before dispatch. On release it opens a fresh bounded cleanup connection, attaches to the same live targets, releases uncertain held input and removes overlays. A failed cleanup preserves state and the automation slot for retry. Dispatched actions are never automatically retried. Closed targets need no input cleanup. Browser-process loss disposes the client, and later requests discover the replacement process.
+
+`--use-active-tab` and personal browser targeting are rejected with `--desktop`. Download waiting is unsupported. Browser operations are bounded to two minutes and share the desktop lease's action budget and idle expiry.
+
+Validation: focused client tests exercise shared snapshot/click behavior, namespace isolation, stale references, target changes, cancellation and uncertain-input cleanup. Desktop-control tests cover ownership and takeover across both interfaces. The Linux smoke script exercises real Chrome, the CLI and visible pointer feedback.

assistant/openapi.yaml

@@ -4101,6 +4101,8 @@ paths:
                 conversationId:
                   type: string
                   minLength: 1
+                desktop:
+                  type: boolean
               required:
                 - operation
       responses:
@@ -4136,7 +4138,7 @@ paths:
     post:
       operationId: browser_tabs_post
       summary: Manage browser tabs
-      description: List, create, select, or close browser tabs via the Chrome extension backend.
+      description: List, create, select, or close browser tabs in the Chrome extension or streamed desktop browser.
       tags:
         - browser
       requestBody:
@@ -4160,6 +4162,8 @@ paths:
                 conversationId:
                   type: string
                   minLength: 1
+                desktop:
+                  type: boolean
                 tabId:
                   type: number
                 url:
@@ -11932,6 +11936,126 @@ paths:
                 required:
                   - defers
                 additionalProperties: false
+  /v1/desktop/control:
+    get:
+      operationId: desktop_control_get
+      summary: Get desktop control status
+      tags:
+        - desktop
+      responses:
+        "200":
+          description: Successful response
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  state:
+                    type: string
+                    enum:
+                      - idle
+                      - assistant
+                      - human
+                required:
+                  - state
+                additionalProperties: false
+    post:
+      operationId: desktop_control_post
+      summary: Hand desktop control between the user and assistant
+      tags:
+        - desktop
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                action:
+                  type: string
+                  enum:
+                    - take
+                    - allow
+              required:
+                - action
+      responses:
+        "200":
+          description: Successful response
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  state:
+                    type: string
+                    enum:
+                      - idle
+                      - assistant
+                      - human
+                required:
+                  - state
+                additionalProperties: false
+  /v1/desktop/setup:
+    get:
+      operationId: desktop_setup_get
+      summary: Get desktop setup status
+      tags:
+        - desktop
+      responses:
+        "200":
+          description: Successful response
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  state:
+                    type: string
+                    enum:
+                      - required
+                      - installing
+                      - ready
+                      - failed
+                      - unsupported
+                  stage:
+                    type: string
+                    enum:
+                      - packages
+                      - chrome
+                      - checking
+                required:
+                  - state
+                additionalProperties: false
+    post:
+      operationId: desktop_setup_post
+      summary: Install desktop components
+      tags:
+        - desktop
+      responses:
+        "200":
+          description: Successful response
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  state:
+                    type: string
+                    enum:
+                      - required
+                      - installing
+                      - ready
+                      - failed
+                      - unsupported
+                  stage:
+                    type: string
+                    enum:
+                      - packages
+                      - chrome
+                      - checking
+                required:
+                  - state
+                additionalProperties: false
   /v1/diagnostics/env-vars:
     get:
       operationId: diagnostics_envvars_get