feat(core): record exact model input screenshots - #2856
Conversation
1d4bba9 to
2b30d4f
Compare
2b30d4f to
c970fa9
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c970fa95c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| uiScreenshot && | ||
| screenshotContentHash(uiScreenshot.base64) === contentHash | ||
| ? uiScreenshot | ||
| : ScreenshotItem.create(imageBase64, Date.now()); |
There was a problem hiding this comment.
When an input is WebP—as explicitly exercised by the new recorder test—this creates a ScreenshotItem that silently classifies it as PNG. Its rawBase64 accessor only removes PNG/JPEG prefixes, so file-backed report serialization decodes the entire data:image/webp;base64,... URI as base64 and writes corrupt bytes; inline references also advertise the wrong MIME type. Extend ScreenshotItem and ScreenshotRef to preserve the actual format rather than wrapping arbitrary data:image/* as PNG.
AGENTS.md reference: AGENTS.md:L9-L12
Useful? React with 👍 / 👎.
| continue; | ||
| } | ||
|
|
||
| const uiScreenshot = task.uiContext?.screenshot ?? sourceScreenshot; |
There was a problem hiding this comment.
When a UIObserver assertion supplies uiContext.screenshotSequence, extraction sends every buffered frame to the model, but this lookup considers only the representative screenshot. Each earlier frame therefore receives a new ScreenshotItem ID, after which recordAndReleaseScreenshotSequence adds the original frame objects to the same task. Because report persistence deduplicates by ID rather than content hash, multi-frame assertions persist and render every earlier screenshot twice; match against screenshotSequence as well before creating a new item.
Useful? React with 👍 / 👎.
|
Closing this outdated implementation. The final model image bytes are now determined at the model-request preparation boundary, while this PR hooks older task-level call sites and no longer guarantees that the recorded image matches the payload sent to the provider. Exact request payloads can already be inspected through model-call recording. If report-visible model input evidence is still needed, it should be redesigned on top of the current request pipeline rather than rebasing this implementation. |
2 participants
Summary
Stack
This is PR 3 of 4 for the WebP foundation split. It is intentionally based on
mainand can be reviewed and merged independently from #2854 and #2855.The original foundation PR #2851 remains unchanged as a rollback option.
Validation
pnpm exec nx run-many -t build -p @midscene/shared @midscene/corepnpm --filter @midscene/core test(124 files, 1,347 passed, 8 skipped)pnpm run type-check:testspnpm run lint