fix(computer): use physical pixels for Windows DPI scaling - #3064
fix(computer): use physical pixels for Windows DPI scaling#3064quanru wants to merge 16 commits into
Conversation
Deploying midscene with
|
| Latest commit: |
54fc561
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7ae836c7.midscene.pages.dev |
| Branch Preview URL: | https://fix-windows-dpi-coordinate-r.midscene.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9449bedff6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| await this.inputDriver.calibrateMouseCoordinates( | ||
| this.displayGeometry.bounds, | ||
| ); |
There was a problem hiding this comment.
When displayId selects a monitor whose DPI transform differs from the monitor currently containing the cursor, this calibration is valid only inside the selected bounds. However, calibrateMouseCoordinates() saves the current global cursor position and later restores it through moveMouse() after installing this selected-display calibration. For example, calibrating a 175%-scaled secondary while the cursor is on a 100%-scaled primary applies the secondary inverse transform to a primary coordinate, leaving the cursor at a different location after every connection. Restore the saved position without this calibration, or use the transform for the monitor containing that position.
Useful? React with 👍 / 👎.
1abb29c to
74138d2
Compare
11c2f61 to
2f34f9d
Compare
quanru
changed the title
1 participant
Summary
Screen.Bounds,Graphics.CopyFromScreen, andCursor.PositionSetThreadDpiAwarenessContextin memory with Reflection.Emit, avoiding a runtimecsc.exedependencyRoot cause
Windows PowerShell 5.1 starts DPI-unaware. At 125% scaling, WinForms reported a logical desktop of
819 x 614, while the runner desktop was physically1024 x 768.Graphics.CopyFromScreenread physical pixels into a bitmap sized from the virtualized WinForms bounds, so Midscene captured only the upper-left819 x 614physical pixels. The green test target was reported by WinForms around(564, 229)but appeared in the screenshot around(704, 286), exactly the 1.25 scaling factor. SettingCursor.Positionwith that screenshot coordinate caused Windows to scale it again, so the cursor missed to the lower right.The failure is invisible at 100% because logical and physical coordinates are identical. The fix makes the short-lived PowerShell threads Per-Monitor V2 aware before any DPI-sensitive API is called, so capture, geometry, movement, and verification all use physical pixels.
CI reproduction and evidence
819 x 6141024 x 768physical pixelsThe hosted 1024 x 768 display exposes 100% and 125% in Windows Settings, not 175%. The 125% run exercises the same DPI-virtualization boundary that caused the reported 175% failure; the production fix does not use a scale-specific conversion.
Validation
pnpm run lintpnpm exec nx test @midscene/computer --skip-nx-cacheAI_TEST_TYPE=computer pnpm exec nx test @midscene/computer --skip-nx-cache -- tests/ai/windows-desktop-smoke.test.ts --retry=0pnpm exec nx build @midscene/computer --skip-nx-cachePrepatch release
1.12.3-beta-20260828110230.0@midscene/core,@midscene/computer, and@midscene/computer-win