feat(runner): answer an action with a screenshot via act --screenshot - #1579
Draft
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
A Tester computer-use step is two CLI calls, act and then screenshot, with a fixed 500 ms wait between them because nothing says when the screen has caught up. `act --screenshot <path>` asks the runner for its screen with the answer, settled after the action, and writes it the way `runner screenshot` does: to the file, or to stdout with `-`. Every failure after a performed action says the action took effect and points at a plain screenshot rather than a repeat, so a caller does not click twice to get its picture. The pinned @qawolf/api-contracts (0.39.0) does not carry the option yet. performActionContract.ts widens the published contract to the expected shape and is deleted when the release that has it is pinned; until then the wire parser strips the image off the answer.
Same rule as screenshot --out -, checked before the action is sent so the caller is not left with a click that happened and a screen it cannot get. The message now names the flag it is about.
Goran Gajic (gorangajic)
force-pushed
the
goran/wiz-11973-qawolf-runner-act-answers-with-a-screenshot
branch
from
September 7, 2026 14:43
692e86e to
003f922
Compare
The check ran after resolveRunner, which launches a runner when none is configured, so a human typing act --screenshot - at a terminal paid for a launch and then got exit 2. It now runs right after the action is read.
An unwritable screenshot destination exited 2, and the runner guide reads a 2 from act as an argument to fix and re-send, which is the double click these messages exist to prevent. All three after-success failures now exit 4, the code the guide already reads as "screenshot before repeating", and the guide says so in one sentence. The stdout and file cases get their own wording, since "give --out a writable path" is no help for a pipe that closed, and the missing-image message names the version skew that is its only cause.
A type cast said the image was on the answer while the pinned output schema stripped it on the wire. Extending the published schemas instead keeps the types derived from the parser and lets the flag be tried against a platform that has the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to WIZ-11973. Stacked on #1578 (WIZ-11974), which it reuses for writing image bytes.
Draft until the contract lands. The pinned
@qawolf/api-contracts0.39.0 has no option onrunner.performActionfor asking for a screenshot with the answer, and no platform branch I could find adds one yet. This PR is built against a placeholder in one file,performActionContract.ts: the published schemas extended with inputscreenshot: trueand an optionalimageJpegBase64on the success. It is a schema rather than a type cast so the flag can be tried against a platform preview that has the change. Today, against a platform without it, the action is performed and the CLI answers "Performed click, but the runner answered without the screen it was asked for" with exit 4. Please point me at the real shape when part 1 is published and I will replace the placeholder, pin the release, handle any failure reasons it adds, and give the runner SDK'sactverb the same option (its types may only import the published contracts, so it waits for the pin).Overview of Changes
A Tester computer-use step is two CLI calls,
actthenscreenshot, with a fixed 500 ms wait between them because nothing says when the screen has caught up. About 2.3 s per step against 1.4 s over WebRTC on the preview.qawolf runner act ... --screenshot <path>asks the runner for its screen with the answer and writes it the wayrunner screenshotdoes: to the file, or to stdout with-. One call per step and the fixed wait goes away. When the action was performed but the image did not arrive or could not be written, the message says the action took effect and points at a plain screenshot, and every such case exits 4, the code the runner guide already reads as "screenshot before repeating", so nobody clicks twice to get a picture. A terminal on stdout is refused before a runner is resolved or anything is sent. The runner guide and the skill now tell agents to prefer the one call.execmoved to its own registration file only to keep the interact file under the line limit.Testing
bun run typecheck bun run lint bun run format:check bun run knip bun run test bun run buildAdded tests: the option is sent only when the flag is given; the image is written to the file or to stdout with the confirmation kept off stdout in json and agent mode; a forwarded stdin action works with
--screenshot -; a terminal on stdout is refused with no runner resolved; an answer without an image, a non-JPEG, and an unwritable destination each exit 4, say the action happened and do not invite a repeat; refusals, lost answers and timed-out requests read as before.Checklist
🤖 Generated with Claude Code