Skip to content

fix(core): normalize GPT locate coordinates - #3057

Closed
quanru wants to merge 1 commit into
mainfrom
fix/core-gpt-normalized-locate
Closed

fix(core): normalize GPT locate coordinates#3057
quanru wants to merge 1 commit into
mainfrom
fix/core-gpt-normalized-locate

Conversation

@quanru

@quanru quanru commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • request GPT-5 bounding boxes in the existing normalized 0-1000 coordinate format
  • map normalized boxes against Midscene's authoritative prepared screenshot dimensions
  • retain strict out-of-bounds validation instead of clamping invalid model output
  • add an edge-element regression covering the reported 1440 x 3200 screenshot

Why

GPT does not know the authoritative pixel dimensions of the image after provider-side image handling. Asking it for absolute pixels allowed a valid bottom-edge element to be returned beyond the local screenshot height. Normalizing the model contract removes that size guess and lets Midscene perform the only pixel conversion.

Fixes #2929

Validation

  • pnpm exec nx test @midscene/core -- tests/unit-test/model-adapter/gpt.test.ts (11 tests)
  • pnpm exec nx test @midscene/core -- tests/unit-test/prompt/prompt.test.ts (34 tests)
  • pnpm exec nx build @midscene/core
  • pnpm run lint
  • git diff --check

No live GPT request was run because the required model credentials were not available; the adapter prompt and coordinate conversion are covered directly.

@quanru
quanru marked this pull request as draft August 27, 2026 09:17
@EAGzzyCSL EAGzzyCSL closed this Aug 27, 2026
@quanru quanru reopened this Aug 27, 2026
@quanru quanru closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Element positioning exceeds the screen size

3 participants