Skip to content

feat(runner): answer an action with a screenshot via act --screenshot - #1579

Draft
Goran Gajic (gorangajic) wants to merge 8 commits into
goran/wiz-11974-qawolf-runner-screenshot-can-write-to-stdoutfrom
goran/wiz-11973-qawolf-runner-act-answers-with-a-screenshot
Draft

feat(runner): answer an action with a screenshot via act --screenshot#1579
Goran Gajic (gorangajic) wants to merge 8 commits into
goran/wiz-11974-qawolf-runner-screenshot-can-write-to-stdoutfrom
goran/wiz-11973-qawolf-runner-act-answers-with-a-screenshot

Conversation

@gorangajic

@gorangajic Goran Gajic (gorangajic) commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Relates to WIZ-11973. Stacked on #1578 (WIZ-11974), which it reuses for writing image bytes.

Draft until the contract lands. The pinned @qawolf/api-contracts 0.39.0 has no option on runner.performAction for asking for a screenshot with the answer, and no platform branch I could find adds one yet. This PR is built against a placeholder in one file, performActionContract.ts: the published schemas extended with input screenshot: true and an optional imageJpegBase64 on the success. It is a schema rather than a type cast so the flag can be tried against a platform preview that has the change. Today, against a platform without it, the action is performed and the CLI answers "Performed click, but the runner answered without the screen it was asked for" with exit 4. Please point me at the real shape when part 1 is published and I will replace the placeholder, pin the release, handle any failure reasons it adds, and give the runner SDK's act verb the same option (its types may only import the published contracts, so it waits for the pin).

Overview of Changes

A Tester computer-use step is two CLI calls, act then screenshot, with a fixed 500 ms wait between them because nothing says when the screen has caught up. About 2.3 s per step against 1.4 s over WebRTC on the preview.

qawolf runner act ... --screenshot <path> asks the runner for its screen with the answer and writes it the way runner screenshot does: to the file, or to stdout with -. One call per step and the fixed wait goes away. When the action was performed but the image did not arrive or could not be written, the message says the action took effect and points at a plain screenshot, and every such case exits 4, the code the runner guide already reads as "screenshot before repeating", so nobody clicks twice to get a picture. A terminal on stdout is refused before a runner is resolved or anything is sent. The runner guide and the skill now tell agents to prefer the one call.

exec moved to its own registration file only to keep the interact file under the line limit.

Testing

bun run typecheck
bun run lint
bun run format:check
bun run knip
bun run test
bun run build

Added tests: the option is sent only when the flag is given; the image is written to the file or to stdout with the confirmation kept off stdout in json and agent mode; a forwarded stdin action works with --screenshot -; a terminal on stdout is refused with no runner resolved; an answer without an image, a non-JPEG, and an unwritable destination each exit 4, say the action happened and do not invite a repeat; refusals, lost answers and timed-out requests read as before.

Checklist

  • Changes follow the code style of this project
  • Self-review completed
  • Tests added/updated (or not applicable)
  • No breaking changes (or described below)

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

A Tester computer-use step is two CLI calls, act and then screenshot,
with a fixed 500 ms wait between them because nothing says when the
screen has caught up. `act --screenshot <path>` asks the runner for its
screen with the answer, settled after the action, and writes it the way
`runner screenshot` does: to the file, or to stdout with `-`.

Every failure after a performed action says the action took effect and
points at a plain screenshot rather than a repeat, so a caller does not
click twice to get its picture.

The pinned @qawolf/api-contracts (0.39.0) does not carry the option yet.
performActionContract.ts widens the published contract to the expected
shape and is deleted when the release that has it is pinned; until then
the wire parser strips the image off the answer.
Same rule as screenshot --out -, checked before the action is sent so
the caller is not left with a click that happened and a screen it cannot
get. The message now names the flag it is about.
@gorangajic
Goran Gajic (gorangajic) force-pushed the goran/wiz-11973-qawolf-runner-act-answers-with-a-screenshot branch from 692e86e to 003f922 Compare September 7, 2026 14:43
The check ran after resolveRunner, which launches a runner when none is
configured, so a human typing act --screenshot - at a terminal paid for
a launch and then got exit 2. It now runs right after the action is read.
An unwritable screenshot destination exited 2, and the runner guide reads
a 2 from act as an argument to fix and re-send, which is the double click
these messages exist to prevent. All three after-success failures now
exit 4, the code the guide already reads as "screenshot before
repeating", and the guide says so in one sentence.

The stdout and file cases get their own wording, since "give --out a
writable path" is no help for a pipe that closed, and the missing-image
message names the version skew that is its only cause.
A type cast said the image was on the answer while the pinned output
schema stripped it on the wire. Extending the published schemas instead
keeps the types derived from the parser and lets the flag be tried
against a platform that has the change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant