Skip to content

feat(device): use WebP for device screenshot producers - #2859

Closed
quanruzhuoxiu wants to merge 3 commits into
feat/webp-report-consumersfrom
feat/webp-device-producers-v2
Closed

feat(device): use WebP for device screenshot producers#2859
quanruzhuoxiu wants to merge 3 commits into
feat/webp-report-consumersfrom
feat/webp-device-producers-v2

Conversation

@quanruzhuoxiu

@quanruzhuoxiu quanruzhuoxiu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • route Android ADB, iOS WDA, macOS, and Windows PNG captures through the shared canonical WebP encoder
  • encode Android scrcpy H.264 frames as WebP through its dedicated FFmpeg-to-Sharp pipeline
  • preserve native JPEG streams where another lossy conversion would add cost and reduce fidelity
  • retain the desktop smoke-test stability changes in this final downstream PR
  • update Android, iOS, Computer, Windows, and RDP regression coverage

The reusable PNG/JPEG-to-WebP implementation lives in packages/shared/src/img/transform.ts: canonicalizeScreenshotBase64() validates the actual image bytes and calls convertImgBufferToWebp(), which uses Sharp.webp({ quality: 90, effort: 1 }) in Node. Device adapters call this shared boundary unless the source already provides a JPEG stream that should be preserved.

WebP generation paths

Scenario Source format Where WebP is generated
Android ADB PNG packages/android/src/device.ts calls shared canonicalizeScreenshotBase64(), which encodes with Sharp
Android scrcpy H.264 keyframe packages/android/src/scrcpy-manager.ts decodes H.264 to raw RGB with FFmpeg, then encodes once with Sharp.webp({ quality: 90, effort: 1 })
iOS WDA screenshot PNG packages/ios/src/device.ts calls shared canonicalizeScreenshotBase64(), which encodes with Sharp
macOS desktop PNG packages/computer/src/device.ts calls shared canonicalizeScreenshotBase64(), which encodes with Sharp
Windows desktop PNG packages/computer/src/device.ts calls shared canonicalizeScreenshotBase64(), which encodes with Sharp
RDP JPEG or PNG, depending on backend packages/computer/src/rdp/device.ts uses preserveJpeg: true: JPEG remains byte-for-byte JPEG, while PNG is encoded as WebP
iOS MJPEG frame source JPEG No WebP encoding; native JPEG frames pass through to avoid a second lossy encode

Stack

Validation

  • pnpm exec nx run-many -t build -p @midscene/android @midscene/ios @midscene/computer
  • pnpm --filter @midscene/android test (16 files, 325 tests passed)
  • pnpm --filter @midscene/ios test (12 files, 146 tests passed)
  • pnpm --filter @midscene/computer test (13 files, 89 tests passed)
  • pnpm run type-check:tests
  • pnpm run lint

@quanruzhuoxiu
quanruzhuoxiu force-pushed the feat/webp-report-consumers branch from 88e097e to 675a785 Compare July 23, 2026 09:50
@quanruzhuoxiu
quanruzhuoxiu force-pushed the feat/webp-device-producers-v2 branch 2 times, most recently from 87f794f to 688cb95 Compare July 23, 2026 09:55
@quanruzhuoxiu
quanruzhuoxiu force-pushed the feat/webp-report-consumers branch from 675a785 to 92530ad Compare July 23, 2026 11:56
@quanruzhuoxiu
quanruzhuoxiu force-pushed the feat/webp-device-producers-v2 branch from 688cb95 to a55dbdf Compare July 23, 2026 11:56
@quanruzhuoxiu
quanruzhuoxiu force-pushed the feat/webp-report-consumers branch from 92530ad to ebfa101 Compare July 24, 2026 07:45
@quanruzhuoxiu
quanruzhuoxiu force-pushed the feat/webp-device-producers-v2 branch from a55dbdf to e83a7ea Compare July 24, 2026 07:47
@quanruzhuoxiu

Copy link
Copy Markdown
Collaborator Author

Closing this superseded WebP stack PR. The replacement stack is #3012#3013#3014 and incorporates the latest main branch plus the follow-up architecture, format-validation, and CI fixes. PR #2856 remains open because it is an independent model-input evidence change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant