feat(web-integration): use WebP for browser screenshot producers

[quanruzhuoxiu]

Summary

• use native WebP capture APIs where the browser runtime supports them
• disable CDP WebP capture for a Playwright page after its first failure, avoiding repeated failures and timeouts
• route PNG fallback captures through the shared canonical WebP encoder
• preserve each screenshot's real format in Chrome Recorder ZIP and Markdown exports
• add unit and runtime-backed browser coverage for real WebP captures and report export
• capture the exact image bytes passed to chat.completions.create() and verify that Markdown exports the same WebP bytes, without depending on the independent model-input recorder PR

The reusable PNG/JPEG-to-WebP implementation lives in packages/shared/src/img/transform.ts: canonicalizeScreenshotBase64() calls convertImgBufferToWebp(), which uses Sharp in Node and the browser encoder for RGBA pixels. This PR wires browser producers to native WebP capture where possible and uses that shared encoder only for fallback paths.

WebP generation paths

Scenario	Where WebP is generated	Encoding path
Chrome extension page	packages/web-integration/src/chrome-extension/screenshot.ts	Native CDP Page.captureScreenshot({ format: 'webp', quality: 90 })
Puppeteer	packages/web-integration/src/puppeteer/base-page.ts	Native page.screenshot({ type: 'webp', quality: 90 })
Playwright on Chromium	packages/web-integration/src/puppeteer/base-page.ts	Native CDP WebP capture; the first CDP failure disables this optimization for the current Midscene page
Playwright after CDP is disabled	packages/web-integration/src/puppeteer/base-page.ts and packages/shared/src/img/transform.ts	Capture PNG, then canonicalizeScreenshotBase64() encodes WebP with Sharp
Chrome Recorder	apps/chrome-extension/src/utils/screenshot.ts	captureVisibleTab() produces PNG, then OffscreenCanvas.convertToBlob({ type: 'image/webp', quality: 0.9 }) encodes WebP
Recorder ZIP and Markdown export	apps/chrome-extension/src/extension/recorder/screenshot-export.ts	No encoding; detect the actual bytes and preserve the correct extension and MIME type

Stack

• Depends on feat(report): support WebP screenshot consumers #2857
• Replaces the clean-stack role of feat(web-integration): use WebP for browser screenshot producers #2852
• feat(web-integration): use WebP for browser screenshot producers #2852 remains unchanged as a rollback option

Validation

• pnpm --filter chrome-extension test (10 files, 40 tests passed)
• pnpm --filter @midscene/playground test (19 files, 274 tests passed)
• pnpm --filter @midscene/web test (50 files, 412 passed, 1 skipped)
• pnpm exec vitest --run tests/unit-test/base-page-screenshot.test.ts from packages/web-integration (5 tests passed)
• pnpm exec playwright test tests/ai/web/playwright/screenshot-cdp-webp.spec.ts --config=tests/playwright.config.ts from packages/web-integration (1 test passed)
• pnpm exec nx test:ai @midscene/web --skip-nx-cache -- tests/ai/web/puppeteer/webp-report.test.ts (4 consecutive runs passed)
• pnpm exec nx build @midscene/web
• pnpm exec nx run-many -t build -p chrome-extension @midscene/playground @midscene/web
• pnpm run type-check:tests
• pnpm run lint

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c31d6c41a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep WebP encoding within the recorder's one-second budget

On high-resolution displays or slower machines, decoding the full PNG and completing OffscreenCanvas.convertToBlob() can take more than the fixed 1,000 ms timeout in event-recorder-bridge.ts:78-97. Because the response is now delayed until this conversion completes, the bridge times out and records the event without its screenshot even when captureVisibleTab succeeded; extend the capture budget or move conversion off the response-critical path.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve the PNG when WebP encoding fails

If createImageBitmap, the canvas context, or WebP encoding fails, this converts a successfully captured PNG into a null response; the recorder interprets that as no screenshot and silently omits the before/after image. Return the original PNG as a fallback or propagate an explicit failure instead of discarding valid capture data.

AGENTS.md reference: AGENTS.md:L6-L8

Useful? React with 👍 / 👎.

[quanruzhuoxiu]

Closing this superseded WebP stack PR. The replacement stack is #3012 → #3013 → #3014 and incorporates the latest main branch plus the follow-up architecture, format-validation, and CI fixes. PR #2856 remains open because it is an independent model-input evidence change.