fix(live-voice,computer-use): narration budget, AX tree size, and the scripting story (#42806)

* fix(live-voice): give progress narration a budget it can actually meet

A live-voice turn that runs tools speaks short progress updates into the
dead air between them. The budget for generating that text was 1500ms,
which sits at the edge of the narrator model's real roundtrip: across one
three-minute computer-use turn, 9 of 22 attempts (41%) aborted at the
deadline, every one of them at totalMs 1500-1503 against timeoutMs 1500.

A budget that only sometimes covers the roundtrip does not make narration
faster, it thins it out - and it thins out precisely where the updates are
worth most, because the beats that take longest to generate are the ones
with the longest silence behind them. This is the failure mode the sibling
endpointDecisionTimeoutMs already documents. The budget is explicitly not
latency-critical (it speaks into dead air), so raise it to 5000ms, still
under the 6000ms minGapMs so an update that spends the whole budget lands
before the next one is due.

Two things hid this. The timeout aborted with kind "voice_session_aborted",
which reads in the logs as the session dying once per update rather than
one beat going unspoken, so give it its own kind. And every drop logged the
same line whatever the cause, so name the outcome (timed_out /
session_ended / error) as one greppable field. Neither the module nor the
log says anything on the success path - the spoken count lives in the
live-voice-metrics `progressUpdatesSpoken` field - so a drop rate was only
ever legible by pairing the two, which is how a 41% rate went unnoticed.

Also reflows one pre-existing over-long assertion in conversation-error
test so the touched file passes the formatting hook.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(live-voice): state the spoken-update count in the log even at zero

The turn log carried progressUpdatesSpoken only when it was above zero, so
the one value that distinguishes "this turn was asked for fewer updates and
correctly spoke none" from "narration failed every attempt" was the one
value the log never showed. Both render as an absent field.

That is not hypothetical: reading these logs, an absent field was twice read
as "narration never ran" on a turn that had in fact suppressed updates
exactly as the caller asked out loud.

Only the log changes. The telemetry payload keeps the omission, because
there the absence means something different and deliberate - a turn that
never engaged the feature is unchanged on the wire (see optionalTurnFields,
whose contract the existing "omits endpoint and progress fields" test
pins).

No new test: the log line is one field on a module-level logger, and
mocking that module here risks the cross-file mock leakage this repo
already has trouble with. The wire-shape guard that could actually regress
is the existing test, and it still passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(computer-use): leave scrolled-away rows out of the AX tree

A scroll view shows part of its contents and hides the rest, but the
accessibility tree reports a scrolled-away row at the frame it would have
if it were on screen. The CU tree walked the raw element tree and listed
every one of them, so a Finder window of ~700 files came back as ~735
interactive rows spanning y=224 to y=15460 on an 1117pt screen - roughly 45
of them actually drawn, the other ~94% reported at real-looking positions
with nothing there. At ~20 tokens a line that is ~15k tokens per step, and
since 42762 every action returns one.

The fix is already in this file: flattenClipped carries each ancestor's
crop down the tree and its doc comment states this exact problem, but it
had one caller (the coachmark measurement path in main.swift) and the tree
the model reads was not it. Thread the same AXClip.narrowed through
collectFormatted and drop what the crop leaves nothing of, reusing
AXDisplayMatch.frame for the geometric test so "any overlap counts" has one
definition rather than two - a row half out of the pane stays listed.

The count is reported rather than silently dropped. A list quietly reduced
to its visible rows reads as a short list, and the model needs to know the
rest are reachable by scrolling.

A tree with nothing cropping in it keeps a nil clip the whole way down and
is reported exactly as before, so apps that scroll nothing are unaffected.

Not unit-tested: formatAXTree lives in MacHelperExecutable, which has a
top-level main.swift and no test target. The geometric rule it now leans on
is covered in MacHelperCoreTests (AXClipTests already asserts a
scrolled-away row fails standsOn its pane's clip). Verify in the dev app
against a long Finder list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(computer-use): teach the scripting dictionary before the menu click

The scripting section lists three things AppleScript is for - "a menu item
(<worked example>), an app's own scripting dictionary, or window management"
- and then spends the rest of the paragraph on the menu item. The dictionary
gets four words and no example. Models copy the example.

They did. Asked to open Finder, go to the Desktop and sort by name, a voice
turn reached for computer_use_run_applescript exactly as told and used the
one pattern it had been shown: System Events clicking Go > Desktop. It then
fell back to two raw clicks on the column header to sort. Finder has a
scripting dictionary in which that whole task is four lines, and the run
cost an app open, a 7,497-element accessibility walk and two clicks instead.

So lead with the dictionary and give it the worked example, and demote
System Events to what it actually is: the fallback for apps with no
dictionary entry for what you need, and UI automation in a script's clothes
- still dependent on the menu sitting where you expect and the app being
frontmost. The iMovie menu example stays, one rung down.

The tool description carried the same ordering ("menu items, an app's
scripting dictionary, windows"), so both copies of it move too - the
manifest-regression test pins them to each other.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* perf(computer-use): stop walking rows a scroll view is not showing

Clipping the serialized tree cut what the model reads; it did not cut what
the helper does to produce it. The walk still visited every element: the
Finder observation that printed 735 rows enumerated 7,497 elements and took
28.5 seconds, against 39-653 elements for every other step in the same
session. The cost scales with how many files are in the folder rather than
with how many are on screen, and each skipped element is ~9 AX attribute
reads over Mach IPC plus its whole subtree.

So carry the same AXClip down the walk and skip children their scrolling
ancestor leaves nothing of, paying one frame read to save the rest.

Off by default, and behind a flag rather than applied to the enumerator at
large, because the other two callers need the opposite. `ax.locate` (the
coachmark path) exists to answer "where is this element", and a scrolled-away
element has to be in the tree for `flattenClipped` to answer "it exists and
is not on screen" - dropped during the walk, that answer becomes "not
found", which puts a coachmark somewhere else entirely. Only the CU
observation path, whose tree is a list of things to act on right now, opts
in.

Two deliberate asymmetries with the formatter's clipping:

- What the walk skips never reaches the formatter to be counted, so the
  count is passed across and summed. Otherwise the "scrolled out of view"
  note - the thing that tells the model to scroll rather than conclude the
  list is short - would silently go to zero.
- The walk does not reuse AXDisplayMatch's test, where an element with no
  area is on nothing. That is right for one being listed and wrong for one
  being walked, since skipping here discards the whole subtree under it, and
  a wrapper reporting no frame (web content does) must be descended into
  rather than judged.

Not unit-tested, same reason as the formatter change: MacHelperExecutable
has a top-level main.swift and no test target. Worth QAing against both a
long Finder list and a scrolled web page, and worth re-checking a coachmark
lands correctly since that path deliberately did not change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: drop the em dashes this branch added

AGENTS.md bans them outright, and the accessibility-tree note was the strict
case: a string the model reads, written in punctuation the assistant's own
prompt forbids. Replaced with a colon there and with periods or a colon in
the four comments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: alex-nork

assistant/src/__tests__/config-schema.test.ts

@@ -1021,7 +1021,7 @@ describe("AssistantConfigSchema", () => {
           maxSilenceMs: 35000,
           longOpMs: 15000,
           minGapMs: 6000,
-          generationTimeoutMs: 1500,
+          generationTimeoutMs: 5000,
         },
       },
     });

assistant/src/__tests__/conversation-error.test.ts

@@ -1490,6 +1490,7 @@ describe("classifyConversationError", () => {
       "subagent_aborted",
       "signal_cancel",
       "voice_session_aborted",
+      "voice_progress_narration_timeout",
     ];
 
     for (const kind of taggedKinds) {
@@ -1727,7 +1728,9 @@ describe("ConnectionResolutionError classification", () => {
     expect(result.userMessage).toContain("qwen/qwen3-8b");
     expect(result.userMessage).toContain("Vellum GPU route");
     expect(result.userMessage).toContain('profile "steer"');
-    expect(result.userMessage).toContain("was not sent through another provider");
+    expect(result.userMessage).toContain(
+      "was not sent through another provider",
+    );
   });
 
   it("classifies missing_credential naming the connection and fix", () => {

assistant/src/calls/__tests__/progress-narration.test.ts

@@ -185,6 +185,29 @@ describe("createVoiceProgressNarrator", () => {
     expect(Date.now() - startedAt).toBeLessThan(1000);
   });
 
+  test("an update that runs out of budget aborts as a narration timeout, not a session abort", async () => {
+    // The budget lapsing means this beat goes unspoken; it does not mean the
+    // call ended. Sharing `voice_session_aborted` made a narrator whose budget
+    // sat below the model's roundtrip read in the logs as a session dying once
+    // per update, which is what hid a turn that had gone silent for minutes.
+    let seen: AbortSignal | undefined;
+    const narrator = createVoiceProgressNarrator({
+      config: VoiceProgressConfigSchema.parse({ generationTimeoutMs: 20 }),
+      getProvider: async () =>
+        stubProvider((_messages, options) => {
+          seen = options?.signal;
+          return new Promise<ProviderResponse>(() => {});
+        }),
+    });
+
+    expect(await narrator.generateProgressText(progressInput)).toBeNull();
+    expect(seen?.aborted).toBe(true);
+    expect(seen?.reason).toMatchObject({
+      kind: "voice_progress_narration_timeout",
+      source: "voice-progress-narration",
+    });
+  });
+
   test("a caller abort settles promptly", async () => {
     const narrator = createVoiceProgressNarrator({
       config: VoiceProgressConfigSchema.parse({

assistant/src/calls/progress-narration.ts

@@ -16,7 +16,7 @@ import type {
   ProviderResponse,
   ToolDefinition,
 } from "../providers/types.js";
-import { createAbortReason } from "../util/abort-reasons.js";
+import { createAbortReason, isAbortReason } from "../util/abort-reasons.js";
 import { getLogger } from "../util/logger.js";
 
 const log = getLogger("voice-progress-narration");
@@ -164,7 +164,10 @@ async function requestBoundedResponse(args: {
   const timeoutTimer = setTimeout(
     () =>
       timeoutController.abort(
-        createAbortReason("voice_session_aborted", "voice-progress-narration"),
+        createAbortReason(
+          "voice_progress_narration_timeout",
+          "voice-progress-narration",
+        ),
       ),
     args.timeoutMs,
   );
@@ -260,9 +263,19 @@ export function createVoiceProgressNarrator(options: {
         }
         return trimmed;
       } catch (error) {
+        // Why the beat went unspoken, as one greppable field. A narrator whose
+        // budget is below the model's real roundtrip drops every update at the
+        // deadline and reads in the logs as a run of unrelated failures; naming
+        // the outcome is what makes that run legible as "the feature is off".
+        const outcome = isAbortReason(error)
+          ? error.kind === "voice_progress_narration_timeout"
+            ? "timed_out"
+            : "session_ended"
+          : "error";
         log.info(
           {
             error,
+            outcome,
             providerResolveMs,
             totalMs: Math.round(performance.now() - startedAt),
             timeoutMs: options.config.generationTimeoutMs,

assistant/src/config/bundled-skills/computer-use/SKILL.md

@@ -40,13 +40,32 @@ cut off. If the element you need is not in it, call `computer_use_observe` with
 ## Scripting apps (macOS)
 
 Reach for `computer_use_run_applescript` first when an app can be driven by
-script: a menu item (`click menu item "Split Clip" of menu "Modify" of menu bar
-1` inside `tell application "System Events" to tell process "iMovie"`), an
-app's own scripting dictionary, or window management. It does not take the
-cursor. A menu item that needs a selection or a playhead position does nothing
-when that context is missing, so set it up first, and read
-`enabled of menu item` when unsure. Click and type for everything a script
-cannot reach. `host_bash` is for shell commands, not for driving apps.
+script. It does not take the cursor.
+
+**Try the app's own scripting dictionary before anything else.** Ask it for the
+state you want, not for the clicks that would produce that state. Finder, Mail,
+Music, Notes, Safari, Terminal and many third-party apps have one, and where
+there is a dictionary the whole task is usually a sentence with no window to
+open, no tree to read and nothing to click:
+
+```applescript
+tell application "Finder"
+  set target of front window to desktop
+  set current view of front window to list view
+  set sort column of list view options of front window to name column
+end tell
+```
+
+**System Events menu clicking is the fallback**, for apps with no dictionary
+entry for what you need: `click menu item "Split Clip" of menu "Modify" of menu
+bar 1` inside `tell application "System Events" to tell process "iMovie"`. It is
+UI automation in a script's clothes: it still depends on the menu sitting where
+you expect and on the app being frontmost. A menu item that needs a selection or
+a playhead position does nothing when that context is missing, so set it up
+first, and read `enabled of menu item` when unsure.
+
+Click and type for everything a script cannot reach. `host_bash` is for shell
+commands, not for driving apps.
 
 ## Typing is not sending
 

assistant/src/config/bundled-skills/computer-use/TOOLS.json

@@ -269,7 +269,7 @@
     },
     {
       "name": "computer_use_run_applescript",
-      "description": "Run an AppleScript on the Mac: menu items, an app's scripting dictionary, windows. Prefer this over click and type when the app supports it; it does not move the cursor. The result is the script's return value, and the accessibility tree may not change even when the script worked. Never use 'do shell script' inside AppleScript (blocked for security).",
+      "description": "Run an AppleScript on the Mac. Try the target app's own scripting dictionary first, asking it for the state you want rather than for the clicks that would produce it; fall back to System Events menu clicking for apps with no dictionary entry for what you need. Prefer this over click and type when the app supports it; it does not move the cursor. The result is the script's return value, and the accessibility tree may not change even when the script worked. Never use 'do shell script' inside AppleScript (blocked for security).",
       "category": "computer-use",
       "risk": "medium",
       "input_schema": {

assistant/src/config/schemas/__tests__/voice.test.ts

@@ -9,7 +9,7 @@ const PROGRESS_DEFAULTS = {
   maxSilenceMs: 35_000,
   longOpMs: 15_000,
   minGapMs: 6_000,
-  generationTimeoutMs: 1_500,
+  generationTimeoutMs: 5_000,
 };
 
 const FRONT_MODEL_DEFAULTS = {
@@ -75,7 +75,7 @@ describe("VoiceFrontModelConfigSchema", () => {
     expect(parsed.progress.maxSilenceMs).toBe(35_000);
     expect(parsed.progress.longOpMs).toBe(15_000);
     expect(parsed.progress.minGapMs).toBe(6_000);
-    expect(parsed.progress.generationTimeoutMs).toBe(1_500);
+    expect(parsed.progress.generationTimeoutMs).toBe(5_000);
   });
 
   test("a stale maxPerTurn key is stripped, not rejected", () => {

assistant/src/config/schemas/voice.ts

@@ -74,9 +74,9 @@ export const VoiceProgressConfigSchema = z
       .positive(
         "voice.frontModel.progress.generationTimeoutMs must be a positive integer",
       )
-      .default(1_500)
+      .default(5_000)
       .describe(
-        "Budget (ms) for LLM-generated progress text. Not latency-critical: it speaks into dead air",
+        "Budget (ms) for LLM-generated progress text. Not latency-critical: it speaks into dead air, so the budget is set to cover the narrator's real roundtrip rather than to be tight. A budget that only sometimes covers that roundtrip thins narration out rather than speeding it up, and it thins out exactly where the updates matter most, since the slow beats are the ones with the longest silence behind them (see endpointDecisionTimeoutMs for the same failure mode). Kept under progress.minGapMs so an update that spends the whole budget still lands before the next one is due",
       ),
   })
   // The heartbeat is checked when the idle tick finds the turn silent, so a

assistant/src/live-voice/live-voice-metrics.ts

@@ -511,6 +511,15 @@ export class LiveVoiceMetricsCollector {
         turnId: turn.turnId,
         finishReason,
         ...aggregateFieldsForTurn(snapshotTurn(turn)),
+        // Stated even at zero, unlike the telemetry field this overrides.
+        // Zero is the single most informative value this count takes: it is
+        // what a turn asked for fewer updates looks like, and equally what a
+        // narrator that is failing every attempt looks like. Left absent, the
+        // two read identically to anyone reading the log, and "the feature is
+        // off" becomes indistinguishable from "the feature did its job".
+        // The wire payload keeps the omission (see optionalTurnFields), so
+        // turns that never engage narration are unchanged for telemetry.
+        progressUpdatesSpoken: turn.progressUpdatesSpoken,
       },
       "Live voice turn latency",
     );

assistant/src/tools/computer-use/definitions.ts

@@ -352,7 +352,7 @@ export const computerUseOpenAppTool = {
 export const computerUseRunAppleScriptTool = {
   name: "computer_use_run_applescript",
   description:
-    "Run an AppleScript on the Mac: menu items, an app's scripting dictionary, windows. Prefer this over click and type when the app supports it; it does not move the cursor. The result is the script's return value, and the accessibility tree may not change even when the script worked. Never use 'do shell script' inside AppleScript (blocked for security).",
+    "Run an AppleScript on the Mac. Try the target app's own scripting dictionary first, asking it for the state you want rather than for the clicks that would produce it; fall back to System Events menu clicking for apps with no dictionary entry for what you need. Prefer this over click and type when the app supports it; it does not move the cursor. The result is the script's return value, and the accessibility tree may not change even when the script worked. Never use 'do shell script' inside AppleScript (blocked for security).",
   category: "computer-use",
   defaultRiskLevel: RiskLevel.Low,
   executionTarget: "host",