Skip to content

fix(computer): align default scroll distance with web (~70% of screen)#2527

Open
quanru wants to merge 1 commit into
mainfrom
fix/computer-scroll-default-distance
Open

fix(computer): align default scroll distance with web (~70% of screen)#2527
quanru wants to merge 1 commit into
mainfrom
fix/computer-scroll-default-distance

Conversation

@quanru
Copy link
Copy Markdown
Collaborator

@quanru quanru commented May 22, 2026

Summary

  • The desktop computer device previously used a hardcoded 500 px (libnut / AppleScript path) and 480 px (RDP path) as the default scroll distance when the model did not provide an explicit distance.
  • On 1080p / 1440p displays this only nudges the viewport by a small fraction, so prompts like "scroll down" appear to do nothing and the model has to retry many times to make progress through long lists.
  • Web puppeteer / chrome-extension have long defaulted to innerHeight * 0.7 (and innerWidth * 0.7 for horizontal). This PR brings the desktop backends into the same convention so the default behavior is consistent across web, mobile, and desktop.

Changes

  • libnut / AppleScript (Windows / Linux / macOS)packages/computer/src/device.ts

    • Default distance is now screen.height * 0.7 for vertical scrolls and screen.width * 0.7 for horizontal scrolls.
    • The macOS AppleScript PageUp / PageDown fallback now uses the real screen height instead of the hardcoded APPROX_VIEWPORT_HEIGHT_PX = 600 approximation when computing how many page-key presses are needed.
    • Removed the unused APPROX_VIEWPORT_HEIGHT_PX constant; introduced DEFAULT_SCROLL_VIEWPORT_RATIO = 0.7.
  • RDPpackages/computer/src/rdp/device.ts

    • Single-action scroll default is now computed from connectionInfo.size using the same 0.7 ratio (per axis).
    • Falls back to the previous hardcoded 480 if connectionInfo is not available yet.
    • Edge-scroll step amount (scrollToTop / scrollToBottom / etc.) is intentionally left at 480 because it is the per-tick wheel amount, not a "one screen" semantic.

The shared actionScrollParamSchema (in @midscene/core) is not changed — this PR only affects what happens when the model does not pass a distance. Callers can still override with an explicit value.

User-visible effect

Before, on Windows / Linux:

"scroll down" → 500 px → barely moves on 1080p, looks broken on 1440p.

After:

"scroll down" → ~70% of screen → roughly one screen worth of content, matching the web experience.

The Kylin Linux target that's coming next inherits the same behavior automatically (single shared core).

Test plan

  • npx nx test computer — 9 test files / 44 tests pass
  • pnpm run lint — clean
  • Manual smoke on Windows desktop with a long list page, with and without explicit distance
  • Manual smoke on macOS (phased-scroll path) and AppleScript fallback path
  • Manual smoke on an RDP session

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented May 22, 2026

Deploying midscene with  Cloudflare Pages  Cloudflare Pages

Latest commit: 33cc422
Status: ✅  Deploy successful!
Preview URL: https://38d0e559.midscene.pages.dev
Branch Preview URL: https://fix-computer-scroll-default.midscene.pages.dev

View logs

Previously the desktop computer device used a hardcoded 500 px (libnut /
AppleScript) and 480 px (RDP) as the default scroll distance when the
model did not provide one. On 1080p / 1440p screens this only moves the
viewport by a tiny fraction, so requests like "scroll down" effectively
appear stuck and the model has to retry many times.

Web puppeteer and the chrome-extension have long defaulted to
`innerHeight * 0.7` / `innerWidth * 0.7`, giving roughly one screen of
movement per single scroll action. This change brings the desktop
backends in line:

- libnut / AppleScript (Win / Linux / Mac): default distance is now
  `screen.height * 0.7` for vertical and `screen.width * 0.7` for
  horizontal scrolls. The mac AppleScript fallback also uses the real
  screen height when computing PageDown / PageUp count instead of the
  hardcoded 600 px approximation.
- RDP: default distance is computed from `connectionInfo.size` using
  the same 0.7 ratio, with a 480 px fallback if the connection info is
  not yet available. The edge-scroll step amount is left unchanged
  since it is the per-tick wheel amount, not a "one screen" semantic.

Users can still pass an explicit `distance` to override the default.
The shared scroll schema is unchanged.
@quanru quanru force-pushed the fix/computer-scroll-default-distance branch from ed335ab to 33cc422 Compare May 25, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants