Skip to content

Native CPU capture: monorepo, plugin bridge, and privacy contracts - #47

Merged
Chinmay-KB merged 25 commits into
mainfrom
cursor/native-capture-cpu-ae5d
Sep 1, 2026
Merged

Native CPU capture: monorepo, plugin bridge, and privacy contracts#47
Chinmay-KB merged 25 commits into
mainfrom
cursor/native-capture-cpu-ae5d

Conversation

@Chinmay-KB

@Chinmay-KB Chinmay-KB commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Retargets native CPU screenshot capture (opt-in nativeCpuExperimental) and overlay identity onto main. Default backend stays flutterRepaintBoundary.

This was previously stacked on feat/native-capture-cpu (docs-only). The default branch is main, not master. Published 0.8.12 on main is the session identity stamp (#48); this PR is 0.8.13.

Why this PR

  • Experimental native CPU capture for Android (PixelCopy) and iOS (UIKit snapshot), opt-in only.
  • Overlay identity on route_change (MUST 1–5, SHOULD 6).
  • Flutter plugin compiles native runtimes from the monorepo when present; published pub archive uses a stub HostApi (no unpublished Maven/CocoaPods).
  • Keeps main's userId/traits stamping on every session lifecycle POST.
  • Public docs live in docs/; working notes in internal/docs/.

Test plan

  • Merge main (includes fix: stamp userId on traits_updated session events #48) and resolve version collision (0.8.12 published vs native-capture notes)
  • flutter test in sdks/flutter/packages/tugboat after retarget (collector identity tests + full suite)
  • Version policy vs main (0.8.120.8.13)
  • GitHub Actions on this branch vs main

Follow-ups (not blocking)

  • Publish capture-runtime Maven / CocoaPods when we want native capture in the pub package.
  • Metal texture readback if Core Graphics misses CAMetalLayer.
  • iOS 16+ UIView.drawHierarchy for SwiftUI.
Open in Web Open in Cursor 

Chinmay-KB and others added 3 commits August 31, 2026 17:32
Document the Dart capture stage mapping, NDK/CMake pins, and a regenerable
JPEG size envelope. Keep Dart clock names from becoming the native
comparison contract. Bump tugboat to 0.8.12 for the envelope test.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Add repository scope, an implementable native capture contract, and
decision records. Architecture files are authoritative; the plan is
sequencing. Mask math, fallback, coverage, and clock ownership live in
the contract so later phases do not invent privacy-critical rules.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@cursor cursor Bot changed the title Record native CPU capture Phase 0 baseline Native CPU capture: Phase 0 baseline and Phase 1 contracts Aug 31, 2026
cursoragent and others added 4 commits August 31, 2026 12:52
Relocate tugboat and tugboat_dio under sdks/flutter/packages, add
core/platforms/docs trees, and point workspace, Melos, and CI at the
new paths. GitHub remains tugboat-flutter until the rename is applied.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Portable CPU core for mask fill, buffer validation, and 9-by-8 dHash
parity with Dart. Tests run under ASan/UBSan; mask and metadata
fuzzers land in CI. JPEG and SHA-256 stay out of this library.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
PixelCopy of FlutterSurfaceView feeds the C++ core over JNI. The
release AAR ships arm64-v8a, armeabi-v7a, and x86_64. The sample
consumes com.tugboat.sdk:capture-runtime:0.1.0 from .local-maven.
Device rotation and API-matrix tests wait on a Flutter fixture.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Convert package:tugboat into a Flutter plugin with a Pigeon host API.
Keep RepaintBoundary as the default pixel source and offer native CPU
capture as an opt-in backend that falls back without double-publishing.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@cursor cursor Bot changed the title Native CPU capture: Phase 0 baseline and Phase 1 contracts Native CPU capture: monorepo through Flutter plugin bridge Aug 31, 2026
Exercise the experimental native backend with known private tiles,
decode the JPEG, and record the remaining device-only privacy rows.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@cursor cursor Bot changed the title Native CPU capture: monorepo through Flutter plugin bridge Native CPU capture: monorepo, plugin bridge, and privacy contracts Aug 31, 2026
cursoragent and others added 3 commits August 31, 2026 14:09
Stage timings already exist on the Android runtime and Pigeon bridge.
Expose pixelReadback on the Dart diagnostic trace so a device lab can
confirm it stays zero after PixelCopy. Document the Phase 7 protocol
and an empty results table; gates are unmeasured without a physical
device, so nativeCpuExperimental remains experimental.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Document the mobile layout, native capture architecture, coverage,
fallback, privacy pipeline, compatibility table, and roadmaps.
Replace the always-on Flutter version bump with a path-aware policy:
docs and C++ tests skip the adapter bump; public runtime API and C ABI
changes bump capture-runtime; adapter source updates the compatibility
table. Add license, Android API-surface, Swift skip, AAR CI, and pub
dry-run checks. Native CPU capture stays experimental on 0.8.12.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Keep nativeCpuExperimental opt-in on the 0.8.12 line. Do not tag
capture-runtime-v0.1.0, publish Maven Central, or bump Flutter to 0.9.0
while device privacy rows and performance gates are unmeasured.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB

Copy link
Copy Markdown
Collaborator Author

Controlled Android emulator A/B result

I benchmarked PR head ff8c3d54d848d5b6c1fbc481b9e24d0f6db7eaac on the same running emulator with the same benchmark application. The only application change between runs was screenshotCaptureBackend.

Test conditions:

  • AVD: device_api35, API 35, ARM64
  • Display: 1080 x 2400, density 420
  • Renderer: host gfxstream / surfaceView
  • Build: Flutter debug ARM64 APK
  • Capture output: 276 x 613 at pixel ratio 0.67
  • Workload: 30 warm-up captures and 200 measured forced captures
  • Screenshot budget skipping: disabled
  • Screen and two TugboatSensitive masks: identical for both runs

End-to-end wall latency:

Metric Native CPU Flutter RepaintBoundary
Average 16.977 ms 46.732 ms
p50 16.843 ms 47.784 ms
p90 20.959 ms 55.096 ms
p95 21.834 ms 56.889 ms
Worst 29.198 ms 63.434 ms

Native was 2.75x faster on average and 2.61x faster at p95. This is a 63.7% average latency reduction and a 61.6% p95 latency reduction.

The native run requested and resolved nativeCpuExperimental. It reported coverage=engineSurface and renderMode=surfaceView. It did not fall back to Flutter.

Average stage diagnostics:

Stage Native CPU Flutter RepaintBoundary
Frame wait 7.152 ms 11.156 ms
Mask collection 3.067 ms 3.124 ms
Surface copy 3.317 ms N/A
Flutter readback N/A 1.620 ms
Flutter encode pipeline N/A 29.984 ms
Native dHash 0.133 ms included above
Native JPEG 0.828 ms included above
Native SHA-256 0.199 ms included above
Native mask fill 0.001 ms included above
Platform-channel round trip 5.859 ms N/A

Native produced a 5,738-byte JPEG. Flutter produced a 10,556-byte JPEG. The native JPEG was 45.6% smaller for the same screen.

Privacy proof

Both results reported masked=true. I decoded each final JPEG and averaged pixels from the interiors of both masks and one unmasked red control region.

Sample Native RGB Flutter RGB
Top-left mask (25, 26, 26) (26, 26, 26)
Bottom-right mask (26, 27, 25) (26, 26, 26)
Unmasked red control (254, 0, 0) (254, 0, 0)

The small native channel differences are normal JPEG rounding. Both private regions match the expected dark mask color. The public red region remains visible.

Final JPEG SHA-256 values:

  • Native: 18252c9a277c7e096ea0259bea4276e0673035222eaee84304d451d63c0024f9
  • Flutter: d8ee8aa83bb6260f9694308d254de4917f2d24a66fb699845b03aa4ef1cb9b18

Timing-accounting issue

The native reportedCaptureMicros average was 20.560 ms, while measured wall latency was 16.977 ms. The current native accounting adds platformChannelMicros to native stage times. The platform-channel interval already contains those native stages. These values overlap and must not be added.

The wall latency above is the valid backend comparison. I recommend changing the native telemetry so captureMicros does not double-count the nested native work.

This result is strong emulator evidence. A release build on physical Android devices is still necessary for the production performance gate.

cursoragent and others added 2 commits September 1, 2026 06:58
TugboatFrame.captureMicros sums frame wait, mask collect, capture, and
encode. Native encodeMicros was adding PixelCopy/dHash/JPEG/SHA-256 on
top of the Pigeon round-trip that already includes those stages, so
reported time exceeded wall latency. Use platformChannel as the Dart
encode clock and keep nested stages on the backend trace. Record the
API 35 emulator A/B as non-gate evidence.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB

Copy link
Copy Markdown
Collaborator Author

iOS build and fallback verification

Fixed in d47aead.

Root cause: TugboatPlugin declared Pigeon protocol methods as public, but their generated request and result types are module-internal. Swift rejected those public signatures.

Changes:

  • Kept the public Flutter plugin registration entry point.
  • Made the four Pigeon protocol witnesses module-internal.
  • Added a CI source check to reject public Pigeon protocol methods in this adapter.

Verification on Xcode 26.6 and an iPhone 17 Pro simulator with iOS 26.5:

  • Reproduced the original Swift compiler error before the change.
  • verify-native-capture-pigeon.sh passes.
  • flutter build ios --simulator --debug --no-codesign passes.
  • A request for nativeCpuExperimental resolved to flutterRepaintBoundary with fallbackReason=unsupportedApi.
  • The manual frame finished with outcome=fresh_accepted.
  • The captured JPEG was 181 x 393, 6,119 bytes, and reported masked=true.
  • Both mask samples decoded to RGB (26, 26, 26).
  • The unmasked red control decoded to RGB (254, 0, 0).

Apple is still a capability stub in this milestone. This change fixes the iOS build and verifies the intended safe Flutter fallback. It does not add native CPU capture on Apple platforms.

cursoragent and others added 3 commits September 1, 2026 07:26
Raise the example wrapper to Gradle 8.14 / AGP 8.11.1 / Kotlin 2.2.20
so device-lab release builds are not rejected by Flutter's Gradle
floor. Add host native JPEG tests for capture scale and density, plus
landscape and view-inset mapping, and MaskMapper coverage for
portrait, landscape, and typical bitmap sizes. PixelCopy on physical
devices remains the open privacy and performance gate.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Introduce unpublished TugboatCaptureRuntime 0.1.0 (SwiftPM + CocoaPod) with
an Objective-C++ bridge to the C ABI. The Flutter iOS plugin draws the
Flutter view, masks and hashes in the portable core, and returns masked
ImageIO JPEG only. Example uses a local CocoaPods path; Xcode builds are
still unrun.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB

Copy link
Copy Markdown
Collaborator Author

iOS live Flutter-layer result

I pushed 3878e33 to this PR.

The missing Metal content came from the capture target. snapshotView(afterScreenUpdates:) creates a copy that does not keep FlutterView's engine-backed layer delegate. Rendering the live FlutterView layer with layer.render(in:) keeps that delegate, so Flutter rerenders its last layer tree into the BGRA bitmap.

The Flutter plugin now finds the nested live FlutterView and uses this path by default. It reports coverage=engineSurface. The old drawHierarchy path remains available as the explicit AppleCaptureMode.viewHierarchy compatibility mode for UIKit platform views.

Simulator proof

  • iPhone 17 Pro Simulator, iOS 26.5, Flutter debug
  • Result: status=ok, coverage=engineSurface, incomplete=false
  • JPEG: 270×586, 21,078 bytes
  • SHA-256: 223f3f21c7c507617547a0c3a1fe21aa3cecb07b4fd4a8a3fc363fc1538ee1c0
  • Visual check: Flutter Metal content present; both privacy masks opaque

Static capture used 30 warm-ups and 200 measured captures:

Metric Live Flutter layer
Wall average 7.179 ms
Wall p50 7.134 ms
Wall p95 7.603 ms
Wall worst 8.033 ms
Surface-copy average 5.335 ms
Surface-copy p95 5.676 ms
Failures 0

The scroll test moved through 5,000 rows by 12,000 points over 8 seconds. It made 60 captures:

Metric Live Flutter layer
Capture wall average 11.896 ms
Capture wall p95 13.485 ms
Surface-copy average 8.192 ms
Surface-copy p95 9.255 ms
Frames 482
Frame-time p95 23.731 ms
Frames over 33.33 ms 0
Capture failures 0

The two no-capture baselines rendered 481 and 482 frames. Their frame-time p95 values were 21.978 ms and 22.018 ms. The prior drawHierarchy surface-copy stage was 36.090 ms static and 34.329 ms during scroll. The live-layer path reduced those stages by 85.2% and 76.1%.

These direct Pigeon measurements exclude the SDK frame wait and Dart mask collection. They are debug Simulator results, not production gates. Physical-device release tests remain open. Engine-surface capture also does not guarantee embedded UIKit platform-view content.

Checks passed:

  • Apple iOS Simulator tests: 12 passed
  • Flutter package tests: 384 passed
  • Pigeon generated-output check
  • Swift public-API check

cursoragent and others added 4 commits September 1, 2026 10:11
Drop the duplicate AppleCaptureMode enum and select the path with
CaptureCoverage. Pass C ABI pixel formats as int32 instead of a
force-unwrapped NS_ENUM, share one reply helper for capture outcomes,
and keep Pigeon mapping out of TugboatPlugin.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Production A/B needs to know which pixel source produced each JPEG without
relying on exploration-only capture_diagnostic events. Copy the closed
backend names and fallback token onto TugboatFrame at publish time.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Split routeName/routeType/routeNamed without changing fromRoute, route, or
navigation. Classify overlays as page|sheet|dialog|popup|unknown, record
presentedOver/hostPage and a bounded routeStack, copy cause fingerprints
from claimed taps, and capture overlay after-frames under exploration
screenshot suppression.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Claim-time snapshots miss tap target fingerprints that resolve after
pointer-up. Refresh causeTargetFingerprint and causeGesture from the
claimed interaction when the event is published. Widget tests now
observe /home before overlay pushes and avoid inventing hostPageRoute
when the initial page was not recorded.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Add a publishing policy: docs/ is contracts and integrator pages.
Move plans, lab method/results, sign-off matrices, Blend canaries,
PR transcripts, and roadmap sketches to internal/docs. Redact host
app ids, session ids, and machine paths in those notes. Architecture
pages are unchanged.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB
Chinmay-KB marked this pull request as ready for review September 1, 2026 16:34
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T17:13:53.338234Z 630a92b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac754fdc36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}

dependencies {
implementation("com.tugboat.sdk:capture-runtime:0.1.0")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Package the Android runtime before requiring it

Avoid making this unpublished Maven coordinate an unconditional dependency. Every Android app using the Flutter package must resolve capture-runtime during Gradle configuration—even when the default flutterRepaintBoundary backend is selected—but the commit contains no tracked AAR/POM and the documented artifact exists only after manually populating the ignored .local-maven directory. Consequently, normal consumers and clean checkouts cannot build until the runtime is published, bundled, or included as a source project.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d541ed. The plugin now compiles platforms/android/capture-runtime from source when that tree is present, so Gradle no longer needs unpublished com.tugboat.sdk:capture-runtime:0.1.0. Pub/pub-cache checkouts without platforms/ get a stub HostApi that reports unsupportedApi and keep the default Flutter backend.

s.source = { :path => '.' }
s.source_files = 'Classes/**/*'
s.dependency 'Flutter'
s.dependency 'TugboatCaptureRuntime', '0.1.0'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Make the Apple runtime resolvable for package consumers

Do not unconditionally depend on an unpublished CocoaPod. A normal Flutter consumer receives this plugin podspec but not the monorepo example's explicit local-path override, while TugboatCaptureRuntime 0.1.0 is intentionally absent from CocoaPods; therefore pod install fails for every iOS consumer, including those retaining the default Flutter capture backend. Publish or vendor the runtime before adding this dependency, or otherwise keep it out of the default plugin resolution path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d541ed. The plugin podspec no longer depends on unpublished TugboatCaptureRuntime. Monorepo checkouts compile Apple runtime sources through ios/NativeRuntime (symlinks into platforms/apple and core/). Published packages exclude those links and stub native capture as unsupported, so pod install works for default flutterRepaintBoundary consumers.

s.source_files = 'Classes/**/*'
s.dependency 'Flutter'
s.dependency 'TugboatCaptureRuntime', '0.1.0'
s.platform = :ios, '15.0'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve support for pre-iOS-15 default-backend apps

Avoid raising the Flutter plugin's deployment target to iOS 15 for all users when native capture is advertised as opt-in. Any existing app targeting an earlier iOS version will be rejected by CocoaPods even when it uses flutterRepaintBoundary, so adopting this otherwise compatible Dart update forces an unrelated OS-support drop. The native implementation needs availability gating or packaging that does not raise the default plugin's minimum target.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d541ed. The Flutter plugin deployment target is iOS 12 again. Native capture is still runtime-gated to iOS 15 (unsupportedApi below that) and falls back to flutterRepaintBoundary. Window lookup and SHA-256 use availability / CommonCrypto fallbacks so pre-iOS-15 apps can keep the default backend.

Comment on lines +39 to +41
val completed = latch.await(timeoutMs, TimeUnit.MILLISECONDS)
if (!isCurrent()) return CaptureStatus.Cancelled
if (!completed) return CaptureStatus.Timeout

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the PixelCopy destination alive after local timeout

When the local latch.await times out before Android invokes the PixelCopy callback, this return unwinds CaptureRuntime.captureSync, whose finally immediately recycles the destination bitmap even though the asynchronous PixelCopy request still owns and may write to it. On slow or stalled copies this creates a native use-after-recycle race; defer recycling until the callback has completed, or retain a separate destination for outstanding requests.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d541ed. If the local latch.await times out, PixelCopyCapture.copy now blocks on a second latch.await() until the PixelCopy callback runs. CaptureRuntime.captureSync only recycles the destination after copy returns, so PixelCopy cannot write to a recycled bitmap.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces experimental native CPU capture for Flutter while retaining the privacy-masked RepaintBoundary fallback and establishing the supporting monorepo and release contracts.

Changes:

  • Adds portable image processing plus Android and Apple capture runtimes.
  • Adds Flutter plugin bridges, backend metadata, and overlay route identity.
  • Adds native/Flutter tests, CI checks, documentation, and release gates.

Reviewed changes

Copilot reviewed 149 out of 326 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
.github/workflows/* Adds native, Flutter, API, and version-policy CI.
core/image-processing/** Implements and tests masking and dHash core.
platforms/android/** Adds Android runtime, PixelCopy capture, JNI, and tests.
platforms/apple/** Adds Apple runtime, bridge, API golden, and tests.
sdks/flutter/packages/tugboat/** Adds plugin integration, fallback, metadata, routing, and tests.
sdks/flutter/packages/tugboat_dio/** Moves the Dio adapter into the monorepo structure.
sdks/react-native/README.md Reserves the future React Native adapter.
docs/** Defines public architecture, integration, compatibility, and publishing contracts.
internal/docs/** Moves private plans, lab results, and release evidence.
tool/benchmarks/** Documents device benchmark procedures.
tool/ci/** Adds build, test, API, licensing, generation, and publication checks.
tool/release/** Documents blocked experimental artifact publication.
Package.swift Defines the Apple Swift package.
TugboatCaptureRuntime.podspec Defines the experimental CocoaPod.
README.md Documents monorepo compatibility and capture status.
Suppressed comments (2)

platforms/android/capture-runtime/src/main/java/com/tugboat/capture/CaptureRuntime.kt:85

  • This recursively reads ViewGroup.childCount/getChildAt on the capture executor, even though Android view hierarchies are UI-thread confined and can be mutating during rotation or detach. Snapshot/find the Flutter surface on the main looper before continuing the bitmap work on the serial executor.
    platforms/android/capture-runtime/src/main/java/com/tugboat/capture/CaptureRuntime.kt:168
  • This always recycles the destination when the local wait times out, but timing out the latch does not cancel the already-issued PixelCopy.request. A late PixelCopy callback may therefore write into a recycled bitmap. Keep ownership until the platform callback completes (or arrange for the callback to recycle a timed-out destination) before releasing it.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +81 to +83
if (request.pixelWidth <= 0 || request.pixelHeight <= 0) {
return result(request, CaptureStatus.ProcessingFailed)
}
Comment on lines +12 to +16
if (mask.width <= 0.0 || mask.height <= 0.0) continue
val left = kotlin.math.floor(mask.x * width).toInt().coerceIn(0, width)
val top = kotlin.math.floor(mask.y * height).toInt().coerceIn(0, height)
val right =
kotlin.math.ceil((mask.x + mask.width) * width).toInt().coerceIn(0, width)
Comment on lines +14 to +17
let left = clamp(Int(floor(mask.x * Double(width))), 0, width)
let top = clamp(Int(floor(mask.y * Double(height))), 0, height)
let right = clamp(Int(ceil((mask.x + mask.width) * Double(width))), 0, width)
let bottom = clamp(Int(ceil((mask.y + mask.height) * Double(height))), 0, height)
Comment on lines +49 to +50
if (maskIntCount < 0 || (maskIntCount % 4) != 0) {
return [self fail:TB_IMAGE_INVALID_ARGUMENT];
Comment on lines +23 to +24
- name: Verify generated native capture bridges
run: bash tool/ci/verify-native-capture-pigeon.sh
Comment on lines +142 to +147
val jpegStart = System.nanoTime()
val jpeg = encodeJpeg(bitmap)
val jpegMicros = elapsedMicros(jpegStart)
val shaStart = System.nanoTime()
val digest = sha256Hex(jpeg)
val shaMicros = elapsedMicros(shaStart)
Comment on lines +34 to +37
PixelCopy.request(surfaceView, destination, { result ->
code.set(result)
latch.countDown()
}, handler)
Comment on lines +51 to +54
if indent == 0 and stripped.startswith(
("class ", "data class ", "object ")
):
lines.append(stripped.split("{")[0].strip())
Comment on lines +57 to +59
```sh
swift test --package-path . # needs Xcode / iOS SDK
```
Comment thread docs/releases/compatibility.md Outdated
Comment on lines +5 to +6
| Flutter `tugboat` | 0.8.12 | Android `com.tugboat.sdk:capture-runtime` 0.1.x (local Maven; experimental). Apple `TugboatCaptureRuntime` 0.1.x (local CocoaPod / SwiftPM; experimental). |
| Flutter `tugboat` | 0.9.0 (planned) | Android `capture-runtime` 0.1.x and Apple `TugboatCaptureRuntime` 0.1.x |
Compile Android and Apple capture runtimes from monorepo sources inside
the plugin, and stub native capture when those trees are absent. Drop the
Maven and CocoaPods coordinates, keep the plugin iOS floor at 12, and
wait for PixelCopy callbacks before recycling the destination bitmap.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB

Copy link
Copy Markdown
Collaborator Author

@codex review

Addressed the four review threads on unpublished Android/Apple packaging, the iOS 15 plugin floor, and the PixelCopy recycle-on-timeout race. Please re-review 2d541ed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d541edc80

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +45 to +48
if (!completed) {
// PixelCopy may still write to `destination`. Block until the
// callback runs so CaptureRuntime's finally can recycle safely.
latch.await()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the timeout path bounded

When the main looper or PixelCopy callback remains delayed past timeoutMs, this unconditional second await() has no deadline, so the advertised timeout never reaches Dart. Because captures share a single-thread executor, the in-flight request and every later request remain wedged instead of falling back to the Flutter backend; retain callback ownership of the bitmap without blocking the request indefinitely.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a7da9c7. Removed the unbounded second await(). On timeout the copy returns Timeout immediately, Dart can fall back, and the PixelCopy callback owns recycle of the in-flight destination bitmap.

if (!completed) {
// PixelCopy may still write to `destination`. Block until the
// callback runs so CaptureRuntime's finally can recycle safely.
latch.await()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the bitmap when the ownership wait is interrupted

When dispose() calls executor.shutdownNow() after the timed wait expires but before the PixelCopy callback, it interrupts this new latch.await(). The resulting InterruptedException unwinds through captureSync, whose finally recycles the destination while PixelCopy can still write to it, and no completion is delivered. This interrupt path is fresh evidence that the original recycle race remains; the outstanding callback must retain ownership even when the capture worker is interrupted.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a7da9c7. latch.await() is now only the original bounded timeoutMs wait. Timeout and InterruptedException (including dispose()shutdownNow()) transfer recycle to the PixelCopy callback via PixelCopyRecycleGuard, so CaptureRuntime does not recycle a bitmap that PixelCopy may still write. The capture worker returns immediately and later requests are not wedged.

Keep the capture timeout bounded: do not wait forever for the PixelCopy
callback. If the wait times out or dispose interrupts the worker, hand
recycle to the callback so CaptureRuntime does not recycle a bitmap
PixelCopy may still write.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB

Copy link
Copy Markdown
Collaborator Author

@codex review

Addressed the PixelCopy P2s on unbounded drain-wait and dispose-interrupt recycle. Timeout and interrupt now transfer bitmap ownership to the callback and return immediately. Please re-review a7da9c7.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 155 out of 335 changed files in this pull request and generated 9 comments.

Suppressed comments (4)

platforms/android/capture-runtime/src/main/java/com/tugboat/capture/CaptureRuntime.kt:103

  • The configured timeout currently covers only PixelCopyCapture.copy; mask/dHash, JPEG compression, and SHA-256 run afterward without any deadline check, so this runtime can return Ok well past the documented 2-second end-to-end native timeout. Track one capture deadline, pass the remaining budget to PixelCopy, and return Timeout after each processing stage if it has expired.
    platforms/android/capture-runtime/src/main/java/com/tugboat/capture/CaptureRuntime.kt:83
  • These dimensions are only checked for positivity before allocating the bitmap. Dart permits much larger values, while the image core's contract caps edges at 8192 and pixels at 16,777,216; oversized requests therefore allocate first and can throw OutOfMemoryError on the executor, bypassing onComplete and leaving the Pigeon capture future unresolved. Enforce the core limits before allocation and convert allocation failures to ProcessingFailed.
    platforms/apple/Sources/TugboatImageCoreBridge/TugboatImageCoreBridge.mm:58
  • A positive maskIntCount with masksPacked == nullptr is currently treated as an empty mask list, allowing processing and JPEG publication without the requested privacy masks. Because the public header declares this pointer nullable, reject this inconsistent pair instead of silently dropping masks.
    .github/workflows/flutter-adapter.yml:13
  • Neither workflow job runs the repository's Flutter test suite or analyzer; Pigeon regeneration and pub publish --dry-run do not execute the hundreds of behavioral tests added or moved in this PR. Add a required job that runs flutter analyze and bash tool/ci/run-flutter-tests.sh so Dart and plugin regressions cannot merge unchecked.

Comment on lines +44 to +49
val completed = latch.await(timeoutMs, TimeUnit.MILLISECONDS)
if (!completed) {
// PixelCopy may still write to `destination`. Block until the
// callback runs so CaptureRuntime's finally can recycle safely.
latch.await()
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already addressed in a7da9c7 (current code no longer uses an unbounded second await()). Timeout and InterruptedException transfer bitmap recycle to the PixelCopy callback and return immediately, so CaptureRuntime does not recycle a bitmap PixelCopy may still write.

Comment on lines +96 to +102
case .engineSurface:
// FlutterView implements CALayerDelegate.draw(_:in:) by asking the
// engine to rerender its last layer tree into readable memory. Rendering
// the live layer preserves Flutter Metal content. A snapshotView copy
// loses that engine delegate and produces an empty Flutter surface.
view.layer.render(in: context)
return false

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing the capture implementation in this PR. CALayer.render remains the experimental CPU path. Comments and integrator docs now state that Core Graphics does not copy CAMetalLayer contents and that physical Metal coverage is an open device-lab gate, with fallback still available via flutterRepaintBoundary.

Comment on lines +96 to +101
private func keyWindow() -> UIWindow? {
if #available(iOS 13.0, *) {
let windows = UIApplication.shared.connectedScenes
.compactMap { $0 as? UIWindowScene }
.flatMap { $0.windows }
return windows.first(where: \.isKeyWindow) ?? windows.first

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 630a92b. Capture now uses registrar.viewController for this plugin's engine and returns surfaceUnavailable when that view is missing, instead of scanning the key window.

Comment thread sdks/flutter/packages/tugboat/ios/tugboat.podspec Outdated
Comment thread docs/architecture/native-capture.md Outdated
Comment on lines +3 to +4
Native CPU capture is an experimental Android path. Flutter still owns
scheduling, mask discovery, capture scale, and session/frame publication.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 630a92b. The architecture page and pipeline diagram now include the Apple CALayer.render branch under the same Pigeon HostApi.

Comment thread docs/architecture/repository-map.md Outdated
Comment on lines +21 to +23
Do not copy the C++ core into the pub package. Files above a published pub
archive are not part of that archive. Flutter consumes published (or local
Maven / CocoaPods path) native artifacts.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 630a92b. The repository map now matches the 0.8.13 packaging contract: monorepo Flutter builds compile runtime sources; published pub archives stub native capture; Maven/CocoaPods remain for native Android/Apple apps only.

Comment thread docs/integration/apple-development.md Outdated
Comment on lines +28 to +32
The default runtime renders the live Flutter layer with
`view.layer.render(in:)` into a BGRA bitmap whose size is the Dart request
(`pixelWidth` × `pixelHeight`). Flutter's live layer delegate rerenders the
last Flutter layer tree, so Metal content is present. Coverage is
`engineSurface`. Embedded UIKit platform views are not guaranteed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in 630a92b. The experimental path is still CALayer.render of the live Flutter view; we no longer claim that Core Graphics copies CAMetalLayer contents. Physical Metal coverage stays an open device-lab gate. A Metal texture readback is out of scope for this adapter PR.

Comment thread sdks/flutter/packages/tugboat/lib/src/screenshot_capture_backend.dart Outdated
Comment thread tool/release/README.md Outdated
Comment on lines +12 to +13
The 0.8.12 adapter may consume `capture-runtime` `0.1.0` from untracked
`.local-maven` after `bash tool/ci/build-android-runtime.sh`.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 630a92b. Release notes now say the 0.8.13 Flutter plugin compiles the Android runtime from source; .local-maven is only for the standalone native sample.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: a7da9c7839

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Keep Android/iOS plugin versions at 0.8.13, capture from the registrar's
Flutter view, and document that CALayer.render is not a Metal texture
readback. PixelCopy.request now fails safe if the surface dies between
the validity check and the main-thread copy.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB

Copy link
Copy Markdown
Collaborator Author

@codex review

Latest head is 630a92b (plugin 0.8.13 metadata, registrar-bound iOS view, packaging/docs). Previous review of a7da9c7 found no major issues.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 630a92bb72

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Chinmay-KB
Chinmay-KB changed the base branch from feat/native-capture-cpu to main September 1, 2026 17:22
Keep published 0.8.12 as the session identity stamp. Fold native CPU
capture and overlay identity into 0.8.13, and lockstep tugboat_dio.

Co-authored-by: Chinmay Kabi <chinmay@blend.to>
@Chinmay-KB
Chinmay-KB merged commit f2a058d into main Sep 1, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants