Skip to content

Commit 7bea55e

Browse files
Add Flutter plugin bridge and experimental native CPU capture
Convert package:tugboat into a Flutter plugin with a Pigeon host API. Keep RepaintBoundary as the default pixel source and offer native CPU capture as an opt-in backend that falls back without double-publishing. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
1 parent f42d7a1 commit 7bea55e

29 files changed

Lines changed: 3389 additions & 141 deletions
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: Flutter adapter
2+
3+
on:
4+
pull_request:
5+
types: [opened, synchronize, reopened, ready_for_review]
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
pigeon:
12+
name: Verify Pigeon outputs
13+
runs-on: ubuntu-latest
14+
steps:
15+
- name: Checkout
16+
uses: actions/checkout@v4
17+
- name: Set up Flutter
18+
uses: subosito/flutter-action@v2
19+
with:
20+
channel: stable
21+
- name: Install Dart dependencies
22+
run: flutter pub get
23+
- name: Verify generated native capture bridges
24+
run: bash tool/ci/verify-native-capture-pigeon.sh

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,10 @@ build/
1010
.local-maven/
1111
platforms/android/local.properties
1212
**/.gradle/
13+
**/.gradle/**
14+
.cxx/
1315
**/.cxx/
16+
**/.cxx/**
1417
crash-*
1518
leak-*
1619
timeout-*

docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ should be verified in their own repositories.
1010

1111
- [SDK README](../sdks/flutter/packages/tugboat/README.md) — install, public API, configuration, and limits
1212
- [Collector integration](integration/collector.md) — Flutter WebSocket and HTTP wire behavior
13+
- [Experimental native CPU capture](integration/native-cpu-experimental.md) — opt-in Android PixelCopy backend and fallback rules
1314
- [Production replay acceptance](integration/production-replay-acceptance.md) — release, Blend canary, and manual production replay gate
1415
- [Blend gesture check, 2026-08-26](integration/blend-gesture-check-2026-08-26.md) — local Android pinch/pan evidence and remaining recorder gaps
1516
- [Gesture PR review, 2026-08-27](integration/gesture-pr-review-2026-08-27.md) — Cursor review, follow-up fixes, tests, and device limits
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Experimental native CPU screenshot capture
2+
3+
Status: experimental opt-in, Android CPU path only
4+
5+
`TugboatScreenshotCaptureBackend.flutterRepaintBoundary` remains the default.
6+
Set `screenshotCaptureBackend` to
7+
`TugboatScreenshotCaptureBackend.nativeCpuExperimental` only when you intend
8+
to exercise the native Android CPU backend.
9+
10+
The native path:
11+
12+
- Captures the Flutter engine `SurfaceView` with `PixelCopy`
13+
- Applies privacy masks and dHash in the portable C++ core
14+
- Encodes JPEG on the platform (quality 80)
15+
- Returns masked JPEG bytes and bounded metadata to Dart
16+
17+
It automatically falls back to `RepaintBoundary` when the runtime is
18+
unsupported, the surface is unavailable, PixelCopy fails, processing fails, or
19+
the native timeout fires. Cancellation and disposal do not fall back. Native
20+
and Flutter results are never published for the same request.
21+
22+
Raw pixels never cross the platform channel. Do not enable this backend in
23+
production until the privacy and performance gates in
24+
[cpu-capture-baseline.md](../performance/cpu-capture-baseline.md) pass.
25+
26+
The Apple plugin currently reports native capture as unavailable.

docs/plans/2026-08-31-001-native-capture-cpu-plan.md

Lines changed: 48 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -568,54 +568,54 @@ Use the Flutter backend on older supported devices.
568568

569569
### Phase 5 - Convert the Flutter SDK into a plugin
570570

571-
- [ ] P5.01 Add Android plugin metadata to `tugboat`.
572-
- [ ] P5.02 Add Apple plugin metadata to `tugboat`.
573-
- [ ] P5.03 Add the Android plugin class.
574-
- [ ] P5.04 Add an Apple capability stub.
575-
- [ ] P5.05 Add Pigeon to the development toolchain.
576-
- [ ] P5.06 Define `getCapabilities`.
577-
- [ ] P5.07 Define `capture`.
578-
- [ ] P5.08 Define `dispose`.
579-
- [ ] P5.09 Generate the Dart bridge.
580-
- [ ] P5.10 Generate the Kotlin bridge.
581-
- [ ] P5.11 Generate the Swift bridge.
582-
- [ ] P5.12 Add generated-file verification to CI.
583-
- [ ] P5.13 Add `TugboatScreenshotCaptureBackend`.
584-
- [ ] P5.14 Add `flutterRepaintBoundary`.
585-
- [ ] P5.15 Add `nativeCpuExperimental`.
586-
- [ ] P5.16 Make `flutterRepaintBoundary` the default.
587-
- [ ] P5.17 Add `screenshotCaptureBackend` to replay configuration.
588-
- [ ] P5.18 Document the experimental status.
589-
- [ ] P5.19 Keep current capture-scale defaults.
590-
- [ ] P5.20 Keep JPEG quality 80.
591-
- [ ] P5.21 Keep the current dHash threshold.
592-
- [ ] P5.22 Keep the current frame transport schema.
593-
- [ ] P5.23 Add a capture backend interface.
594-
- [ ] P5.24 Move the current capturer behind that interface.
595-
- [ ] P5.25 Add the native CPU backend.
596-
- [ ] P5.26 Convert mask rectangles to normalized app coordinates.
597-
- [ ] P5.27 Send only mask metadata to native code.
598-
- [ ] P5.28 Send the previous dHash.
599-
- [ ] P5.29 Send the forced-capture flag.
600-
- [ ] P5.30 Receive only masked JPEG data.
601-
- [ ] P5.31 Convert native timings into capture diagnostics.
602-
- [ ] P5.32 Record the requested backend.
603-
- [ ] P5.33 Record the resolved backend.
604-
- [ ] P5.34 Record capture coverage.
605-
- [ ] P5.35 Record the fallback reason.
606-
- [ ] P5.36 Ignore stale native results.
607-
- [ ] P5.37 Prevent duplicate frame publication.
608-
- [ ] P5.38 Preserve current capture scheduling.
609-
- [ ] P5.39 Preserve current session semantics.
610-
- [ ] P5.40 Fall back when native capture is unsupported.
611-
- [ ] P5.41 Fall back when the render surface is unavailable.
612-
- [ ] P5.42 Fall back after a native timeout.
613-
- [ ] P5.43 Fall back after `PixelCopy` failure.
614-
- [ ] P5.44 Fall back after processing failure.
615-
- [ ] P5.45 Do not fall back after cancellation.
616-
- [ ] P5.46 Do not publish native and Flutter results together.
617-
- [ ] P5.47 Limit retries after a known native failure.
618-
- [ ] P5.48 Reset retry state after a lifecycle change.
571+
- [x] P5.01 Add Android plugin metadata to `tugboat`.
572+
- [x] P5.02 Add Apple plugin metadata to `tugboat`.
573+
- [x] P5.03 Add the Android plugin class.
574+
- [x] P5.04 Add an Apple capability stub.
575+
- [x] P5.05 Add Pigeon to the development toolchain.
576+
- [x] P5.06 Define `getCapabilities`.
577+
- [x] P5.07 Define `capture`.
578+
- [x] P5.08 Define `dispose`.
579+
- [x] P5.09 Generate the Dart bridge.
580+
- [x] P5.10 Generate the Kotlin bridge.
581+
- [x] P5.11 Generate the Swift bridge.
582+
- [x] P5.12 Add generated-file verification to CI.
583+
- [x] P5.13 Add `TugboatScreenshotCaptureBackend`.
584+
- [x] P5.14 Add `flutterRepaintBoundary`.
585+
- [x] P5.15 Add `nativeCpuExperimental`.
586+
- [x] P5.16 Make `flutterRepaintBoundary` the default.
587+
- [x] P5.17 Add `screenshotCaptureBackend` to replay configuration.
588+
- [x] P5.18 Document the experimental status.
589+
- [x] P5.19 Keep current capture-scale defaults.
590+
- [x] P5.20 Keep JPEG quality 80.
591+
- [x] P5.21 Keep the current dHash threshold.
592+
- [x] P5.22 Keep the current frame transport schema.
593+
- [x] P5.23 Add a capture backend interface.
594+
- [x] P5.24 Move the current capturer behind that interface.
595+
- [x] P5.25 Add the native CPU backend.
596+
- [x] P5.26 Convert mask rectangles to normalized app coordinates.
597+
- [x] P5.27 Send only mask metadata to native code.
598+
- [x] P5.28 Send the previous dHash.
599+
- [x] P5.29 Send the forced-capture flag.
600+
- [x] P5.30 Receive only masked JPEG data.
601+
- [x] P5.31 Convert native timings into capture diagnostics.
602+
- [x] P5.32 Record the requested backend.
603+
- [x] P5.33 Record the resolved backend.
604+
- [x] P5.34 Record capture coverage.
605+
- [x] P5.35 Record the fallback reason.
606+
- [x] P5.36 Ignore stale native results.
607+
- [x] P5.37 Prevent duplicate frame publication.
608+
- [x] P5.38 Preserve current capture scheduling.
609+
- [x] P5.39 Preserve current session semantics.
610+
- [x] P5.40 Fall back when native capture is unsupported.
611+
- [x] P5.41 Fall back when the render surface is unavailable.
612+
- [x] P5.42 Fall back after a native timeout.
613+
- [x] P5.43 Fall back after `PixelCopy` failure.
614+
- [x] P5.44 Fall back after processing failure.
615+
- [x] P5.45 Do not fall back after cancellation.
616+
- [x] P5.46 Do not publish native and Flutter results together.
617+
- [x] P5.47 Limit retries after a known native failure.
618+
- [x] P5.48 Reset retry state after a lifecycle change.
619619

620620
### Phase 6 - Validate privacy and correctness
621621

pubspec.lock

Lines changed: 61 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -73,14 +73,30 @@ packages:
7373
url: "https://pub.dev"
7474
source: hosted
7575
version: "2.1.2"
76+
built_collection:
77+
dependency: transitive
78+
description:
79+
name: built_collection
80+
sha256: "376e3dd27b51ea877c28d525560790aee2e6fbb5f20e2f85d5081027d94e2100"
81+
url: "https://pub.dev"
82+
source: hosted
83+
version: "5.1.1"
84+
built_value:
85+
dependency: transitive
86+
description:
87+
name: built_value
88+
sha256: "31b24be6615ec7fcf70b3aa5a7469fe35826485e639a16dd7eb83ba30e4cc6a8"
89+
url: "https://pub.dev"
90+
source: hosted
91+
version: "8.12.7"
7692
characters:
7793
dependency: transitive
7894
description:
7995
name: characters
80-
sha256: f71061c654a3380576a52b451dd5532377954cf9dbd272a78fc8479606670803
96+
sha256: faf38497bda5ead2a8c7615f4f7939df04333478bf32e4173fcb06d428b5716b
8197
url: "https://pub.dev"
8298
source: hosted
83-
version: "1.4.0"
99+
version: "1.4.1"
84100
charcode:
85101
dependency: transitive
86102
description:
@@ -121,6 +137,14 @@ packages:
121137
url: "https://pub.dev"
122138
source: hosted
123139
version: "1.1.2"
140+
code_builder:
141+
dependency: transitive
142+
description:
143+
name: code_builder
144+
sha256: aa5932e94c6c39c2f9ec4e5e06dfdd11a9430a61f6c41b6ba75b28ce0c481baf
145+
url: "https://pub.dev"
146+
source: hosted
147+
version: "4.12.0"
124148
collection:
125149
dependency: transitive
126150
description:
@@ -185,6 +209,14 @@ packages:
185209
url: "https://pub.dev"
186210
source: hosted
187211
version: "0.2.1"
212+
dart_style:
213+
dependency: transitive
214+
description:
215+
name: dart_style
216+
sha256: "8a0e5fba27e8ee025d2ffb4ee820b4e6e2cf5e4246a6b1a477eb66866947e0bb"
217+
url: "https://pub.dev"
218+
source: hosted
219+
version: "3.1.1"
188220
dbus:
189221
dependency: transitive
190222
description:
@@ -257,6 +289,14 @@ packages:
257289
url: "https://pub.dev"
258290
source: hosted
259291
version: "7.0.1"
292+
fixnum:
293+
dependency: transitive
294+
description:
295+
name: fixnum
296+
sha256: b6dc7065e46c974bc7c5f143080a6764ec7a4be6da1285ececdc37be96de53be
297+
url: "https://pub.dev"
298+
source: hosted
299+
version: "1.1.1"
260300
flutter:
261301
dependency: transitive
262302
description: flutter
@@ -372,18 +412,18 @@ packages:
372412
dependency: transitive
373413
description:
374414
name: matcher
375-
sha256: dc58c723c3c24bf8d3e2d3ad3f2f9d7bd9cf43ec6feaa64181775e60190153f2
415+
sha256: "31bd099b47c10cd1aeb55146a2d46ce0277630ecef3f7dae54ad7873f36696cd"
376416
url: "https://pub.dev"
377417
source: hosted
378-
version: "0.12.17"
418+
version: "0.12.20"
379419
material_color_utilities:
380420
dependency: transitive
381421
description:
382422
name: material_color_utilities
383-
sha256: f7142bb1154231d7ea5f96bc7bde4bda2a0945d2806bb11670e30b850d56bdec
423+
sha256: "9c337007e82b1889149c82ed242ed1cb24a66044e30979c44912381e9be4c48b"
384424
url: "https://pub.dev"
385425
source: hosted
386-
version: "0.11.1"
426+
version: "0.13.0"
387427
melos:
388428
dependency: "direct dev"
389429
description:
@@ -396,10 +436,10 @@ packages:
396436
dependency: transitive
397437
description:
398438
name: meta
399-
sha256: e3641ec5d63ebf0d9b41bd43201a66e3fc79a65db5f61fc181f04cd27aab950c
439+
sha256: "307249ce4ff29d58a18e97f6345f539382eb9c9c29ecda628900f31de0443dd9"
400440
url: "https://pub.dev"
401441
source: hosted
402-
version: "1.16.0"
442+
version: "1.19.0"
403443
mime:
404444
dependency: transitive
405445
description:
@@ -464,6 +504,14 @@ packages:
464504
url: "https://pub.dev"
465505
source: hosted
466506
version: "7.0.2"
507+
pigeon:
508+
dependency: transitive
509+
description:
510+
name: pigeon
511+
sha256: fe36bc5ad43ab7f4d8a9d974850aef1148f6d88a53bd14b3d87475ec403a3cdb
512+
url: "https://pub.dev"
513+
source: hosted
514+
version: "26.1.0"
467515
platform:
468516
dependency: transitive
469517
description:
@@ -585,10 +633,10 @@ packages:
585633
dependency: transitive
586634
description:
587635
name: test_api
588-
sha256: "522f00f556e73044315fa4585ec3270f1808a4b186c936e612cab0b565ff1e00"
636+
sha256: "2a122cbe059f8b610d3a5415f42e255b6c17b1f21eee1d960f31080237fb4f11"
589637
url: "https://pub.dev"
590638
source: hosted
591-
version: "0.7.6"
639+
version: "0.7.12"
592640
typed_data:
593641
dependency: transitive
594642
description:
@@ -609,10 +657,10 @@ packages:
609657
dependency: transitive
610658
description:
611659
name: vector_math
612-
sha256: d530bd74fea330e6e364cda7a85019c434070188383e1cd8d9777ee586914c5b
660+
sha256: f36f9f3be64c6198714492bb455c11056e33e2f85d9a0b676a48301e44fdcf47
613661
url: "https://pub.dev"
614662
source: hosted
615-
version: "2.2.0"
663+
version: "2.4.2"
616664
vm_service:
617665
dependency: transitive
618666
description:
@@ -678,5 +726,5 @@ packages:
678726
source: hosted
679727
version: "2.2.4"
680728
sdks:
681-
dart: ">=3.9.2 <4.0.0"
729+
dart: ">=3.11.0-0 <4.0.0"
682730
flutter: ">=3.35.0"

sdks/flutter/packages/tugboat/CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ Patch release after `0.8.11`.
66

77
- Record a regenerable JPEG size envelope for the current Dart encoder so the
88
native CPU capture path can compare codec output against the same buffers.
9+
- Convert `tugboat` into a Flutter plugin with an experimental opt-in Android
10+
native CPU capture backend (`nativeCpuExperimental`). The default remains
11+
`flutterRepaintBoundary`, with automatic fallback when native capture is
12+
unavailable or fails safely.
913

1014
## 0.8.11
1115

0 commit comments

Comments
 (0)