Skip to content

computeruse.mdx

Latest commit

 

History

History
64 lines (50 loc) · 2.98 KB

computeruse.mdx

File metadata and controls

64 lines (50 loc) · 2.98 KB
Raw
title Computer-use record/replay
description Record browser & desktop agents at the action-event level and replay them deterministically — keyed on the screen, flagging any UI state it never saw.

Computer-use agents act on a UI — they click, type, scroll, navigate. That makes them the hardest kind to test: the same action does different things on different screens, and you can't re-drive a live browser deterministically. volo-computeruse records the actions and replays them against the simulator, keyed on the screen so replay is exact and honest.

The idea

An ActionEvent is (kind, target, value, screen), where screen is a screenshot hash of the UI before the action. So an action's identity includes the state it ran in — "click #buy on the cart screen" is a different event from "click #buy on the home screen". Events map onto the ordinary Volo recording as cu.<kind> tool calls, so everything else (diff, fuzz, scoring) works on them too.

from volo_computeruse import ActionEvent, ComputerUseRecorder, screenshot_hash

rec = ComputerUseRecorder(session_name="checkout")
home = screenshot_hash(page.screenshot())          # or a DOM serialization
rec.record(ActionEvent(kind="click", target="#add-to-cart", screen=home),
           result={"ok": True}, screen_after=screenshot_hash(page.screenshot()))
rec.save(".volo/recordings/checkout.json")

Replay — exact, or flagged

uv run volo cu inspect .volo/recordings/checkout.json     # print the action trajectory

# serve recorded UI outcomes for action events read from stdin (one JSON per line)
echo '{"kind":"click","target":"#add-to-cart","screen":"<home-hash>"}' \
  | uv run volo cu replay .volo/recordings/checkout.json
# {"result": {"ok": true}, "screen_after": "..."}

A seen (action, screen) replays its recorded outcome. An action on a screen the recording never saw returns {"__flagged__": "..."} — the simulator never fabricates a UI state it didn't observe (the same flag-on-unknown invariant as the rest of Volo). Fidelity is exactly as strict as the screen fingerprint: identical DOM/screenshot → a hit; any difference → a flag.

Live driver (Playwright)

PlaywrightDriver is the live transport — it wraps a Playwright Page, performs each action on the real browser, hashes the screen before/after, and records an ActionEvent. The recording then replays deterministically offline. It's duck-typed over the Page (never imports playwright at the core), so it's browser-free unless you opt in with volo-computeruse[playwright].

from playwright.sync_api import sync_playwright
from volo_computeruse import PlaywrightDriver

with sync_playwright() as pw:
    page = pw.chromium.launch().new_page()
    driver = PlaywrightDriver(page, session_name="checkout")
    driver.navigate("https://shop.example")
    driver.click("#add-to-cart")
    driver.type("#coupon", "SAVE10")
    driver.save(".volo/recordings/checkout.json")   # now replay it in CI, offline, at $0