Skip to content

Show browser screenshots in the chat transcript - #303

Open
frenchie4111 wants to merge 2 commits into
mainfrom
browser-tool-thumbnails-in-chat
Open

Show browser screenshots in the chat transcript#303
frenchie4111 wants to merge 2 commits into
mainfrom
browser-tool-thumbnails-in-chat

Conversation

@frenchie4111

Copy link
Copy Markdown
Collaborator

Summary

When the agent takes a screenshot with the ness-control browser tools in a Chat tab, the transcript now shows what it saw. Previously the tool_result extractor kept only text parts, so the image was dropped on the floor — you could see that a screenshot happened, but never what was in it.

How the image gets there. Tool-result images are spilled to the existing on-disk attachment store and only the path travels in slice state. A PNG screenshot is megabytes of base64 and every state event fans out to every connected client, so putting the bytes in the reducer wasn't an option. The renderer reads them back lazily through the same jsonClaude:readAttachmentImage path that pasted attachments already use.

Filenames are content-addressed (sha256), because resuming a session replays the entire transcript back through the extractor. With uuid names that would leak a fresh copy of every screenshot in the history on every resume.

Two expansion fixes, so the screenshot isn't buried: the tool card auto-expands when its result carries an image, and so does the enclosing ToolGroup. Both track the user's click as its own tri-state (boolean | null) rather than reverting via an effect — the tool_result arrives after the card mounts, so a useState(autoExpand) initial value would capture false; and since hasImages never goes back to false, an effect-driven revert would make a screenshot group impossible to collapse.

The second commit removes an unrelated wart it exposed: every expanded generic tool card was rendering its input three times — the collapsed subtitle, a raw JSON.stringify <pre>, and then ArgsBlock, which shows the same values parsed and already has a show raw toggle for the identical JSON. The <pre> was redundant with both. Barely noticeable while these cards were collapsed by default; auto-expanding them put it on screen.

Not covered

Only screenshots produce an image. click_tab / type_tab return plain text, so nothing visual renders for them — seeing where a click landed would need Ness to auto-capture after each interaction, which is a bigger and more invasive change.

Test plan

  • npm run typecheck, npx electron-vite build, npm run build:web — all clean
  • npx vitest run src/renderer — 40 files / 504 tests pass
  • New reducer test: toolResultAttached carries image refs onto the block
  • New extractor test on the seed/resume path — asserts the image lands on disk, the text content is preserved alongside it, the base64 does not appear in JSON.stringify(block), bytes round-trip, and extracting the same image twice returns the same path
  • Verified live in npm run dev: opened a Chat tab, had the agent screenshot example.com, confirmed the thumbnail renders, the card and group auto-expand while sibling cards stay collapsed, and click-to-zoom + Escape work
  • Verified the raw-JSON block is gone and show raw still reaches the same JSON

🤖 Generated with Claude Code

frenchie4111 and others added 2 commits August 27, 2026 10:11
Screenshots the agent takes with ness-control browser tools were being
dropped on the floor in Chat tabs — the tool_result extractor only kept
text parts, so you could see that a screenshot happened but never what
it showed.

Images are spilled to the existing on-disk attachment store and only the
path travels in slice state, since every state event fans out to every
connected client and a PNG is megabytes of base64. Filenames are content
-addressed so resuming a session (which replays the whole transcript
through the extractor) doesn't leak a fresh copy of every screenshot.

The tool card and its enclosing group both auto-expand when a result
carries an image, so the screenshot isn't buried behind two chevrons.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every expanded generic tool card rendered its input three times: the
collapsed subtitle, a raw JSON.stringify pre, and then ArgsBlock — which
shows the same values parsed and already has a "show raw" toggle for the
identical JSON. The pre was the ugliest of the three and the only one
with no reason to exist.

Barely noticeable before, since these cards were collapsed by default.
Auto-expanding them for screenshots put it on screen.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant