Improve screenshot capture performance (0.7.1)

docs/design/capture-and-fingerprint.md

@@ -83,7 +83,7 @@ those fields are optional historic data in older sessions only.
 
 The session stores:
 
-- frame metadata plus PNG bytes;
+- frame metadata plus JPEG bytes;
 - ordered events;
 - optional scroll samples;
 - app/platform/viewport metadata;
@@ -245,8 +245,9 @@ telemetry and avoid putting user data in them.
 ## Screenshot pipeline
 
 Screenshots are taken from the SDK `RepaintBoundary` at the configured pixel
-ratio (default `0.75`). Before PNG encoding the SDK collects mask rectangles
-using the shared anchor resolver and paints them onto the raster.
+ratio (default `0.75`). Before JPEG encoding the SDK collects mask rectangles
+using the shared anchor resolver and applies them as opaque fills on the RGBA
+buffer inside the encode isolate (avoiding a second full-size GPU raster).
 
 The default mask policy is profile-dependent:
 
@@ -256,14 +257,22 @@ The default mask policy is profile-dependent:
 The public mask levels are `explicitOnly`, `allTextAndMedia`, `allText`,
 `allTextExceptActionable`, and `sensitiveInputsOnly`.
 
-Capture uses a 9x8 perceptual dHash before PNG encoding to skip a visually
-unchanged raster, then SHA-256 content hashing to deduplicate encoded frames.
-Capture requests are serialized and coalesced; repeated state signatures are
-also skipped unless a caller forces capture.
-
-PNG readback and encoding still happen through Flutter image APIs on the UI
-isolate. Platform views, video textures, maps, and native overlays may be absent
-or incomplete in repaint-boundary output.
+Capture computes a 9x8 perceptual dHash from the masked RGBA buffer inside the
+encode isolate and skips JPEG encoding when the Hamming distance to the last
+accepted hash is at most 2 bits (tolerating minor anti-alias shimmer). SHA-256
+content hashing then deduplicates encoded frames. Capture requests are
+serialized and coalesced. When the capture boundary's paint generation has not
+advanced since the last accepted frame, the controller skips the entire GPU
+readback/encode path and reuses a compatible frame (unless the caller forces
+capture or requires a fresh paint).
+
+Mask fills, dHash, JPEG encoding, and content hashing run on a persistent
+background isolate after a full-frame RGBA readback on the UI isolate. RGBA
+bytes are packed into [TransferableTypedData] for the worker: packing still
+copies once on the UI isolate, but the worker materializes the buffer without
+a second full-frame copy (unlike a plain isolate/`compute` send). Platform
+views, video textures, maps, and native overlays may be absent or incomplete
+in repaint-boundary output.
 
 When the exploration WebSocket connects and there is no HTTP collector, the
 controller suppresses new Flutter screenshots for UI-thread performance.
@@ -345,9 +354,11 @@ cross-build equivalence.
 
 ### 2. Screenshot budget device baselines
 
-Unit thresholds live in `benchmark/screenshot_budget_baseline.dart`. Record
-multi-tier device measurements before enabling aggressive degradation in
-production profiles.
+Unit thresholds live in `benchmark/screenshot_budget_baseline.dart`. The default
+rolling budget is 60 ms per 5 s window so eligible captures skip under load
+sooner now that paint-generation / dHash coalesce replace the old post-capture
+state-signature short circuit. Record multi-tier device measurements before
+enabling more aggressive degradation in production profiles.
 
 ### 3. Stronger collector acknowledgement
 

docs/integration/collector.md

@@ -65,7 +65,7 @@ The SDK sends:
   `fingerprintSchemaVersion`;
 - `type: event`: serialized event payload plus available session/run/action
   correlation fields;
-- `type: frame`: frame metadata followed by a binary PNG message;
+- `type: frame`: frame metadata followed by a binary JPEG message;
 - `type: control_ack`: acknowledgement for supported exploration commands.
 
 Incoming JSON control messages are forwarded to the controller. The current
@@ -161,7 +161,7 @@ The SDK calls:
 | --- | --- |
 | `POST /v1/sessions` | Session lifecycle and identity: `session_start`, `session_identify`, `session_end`, `traits_updated`, `user_changed` |
 | `POST /v1/events/batch` | JSON event batches |
-| `POST /v1/frames` | multipart PNG frame upload |
+| `POST /v1/frames` | multipart JPEG frame upload |
 
 Every request includes both `X-PMKit-API-Key` and `X-Tugboat-API-Key`, plus
 platform, build number, version name, and app ID headers. Mobile API keys are
@@ -204,8 +204,13 @@ Event payloads contain:
   schema version.
 
 Frame uploads are sorted by numeric frame suffix and sent as multipart files
-named `<frameNo>.png`, with `sessionId` and comma-separated `frameNos` fields.
+named `<frameNo>.jpg`, with `sessionId` and comma-separated `frameNos` fields.
 Malformed frame IDs and frames belonging to a stale SDK session are dropped.
+While frames are still queued, newer captures supersede pending `scroll`
+samples and any pending frame with the same content hash so intermediate
+burst rasters are not uploaded. The same policy marks in-flight uploads
+superseded; if that HTTP request later fails, superseded frames are dropped
+before retry instead of being requeued.
 
 ### Batching, retry, and backpressure
 

packages/tugboat/CHANGELOG.md

@@ -1,3 +1,19 @@
+## 0.7.1
+
+### Changed
+
+- **Screenshot capture performance** — remove the post-capture state-signature
+  short circuit and replace it with a paint-generation gate that skips the full
+  GPU readback/encode path when the capture boundary has not painted.
+  Diagnostic outcome `state_signature_short_circuit` is replaced by
+  `paint_generation_unchanged`.
+- **Encode path** — JPEG encoding, SHA-256, mask fills, and dHash now run on a
+  persistent background isolate with transferable RGBA input. dHash coalesce
+  tolerates Hamming distance ≤ 2. Default screenshot budget is 60 ms / 5 s.
+- **Collector uploads** — pending scroll samples and same-contentHash frames are
+  superseded before upload during capture bursts. Frame wire format docs
+  corrected to JPEG.
+
 ## 0.7.0
 
 ### Added

packages/tugboat/README.md

@@ -5,7 +5,7 @@ checkpoints around meaningful interactions, compact structural anchors, route
 transitions, scrolling evidence, and optional viewport semantic maps. Capture
 can be sent to the local exploration WebSocket, the HTTP collector, or both.
 
-The current package version is `0.7.0`. Session JSON writers emit schema
+The current package version is `0.7.1`. Session JSON writers emit schema
 version `9`; compatibility readers should accept versions `6` through
 `9`. Structural fingerprints use fingerprint schema version `6`.
 
@@ -23,7 +23,7 @@ The package requires Dart 3.9.2 or newer and Flutter 3.35.0 or newer.
 
 ```yaml
 dependencies:
-  tugboat_dio: ^0.7.0
+  tugboat_dio: ^0.7.1
 ```
 
 See `packages/tugboat_dio/README.md`.
@@ -303,7 +303,7 @@ Call `TugboatReplay.clearDurableOutbox()` on logout/consent revocation.
 | `viewportSemanticMapMaxBytes` | 48000 | emitted map byte budget |
 | `sinkFactories` | empty | extra `TugboatCaptureSinkFactory` adapters |
 | `outbox` | disabled | durable HTTP outbox configuration |
-| `screenshotBudget` | defaults | degraded-capture skip window / budget |
+| `screenshotBudget` | 60ms / 5s window | degraded-capture skip window / budget |
 
 ### Legacy gesture projection deprecation
 
@@ -433,9 +433,10 @@ cancellations, evidence, legacy projections, and diagnostics.
 
 Frames can be triggered by initial startup, taps, scrolls, routes, lifecycle,
 or explicit controller calls. Capture requests are serialized and coalesced.
-The SDK first skips repeated state signatures, then uses a small dHash to avoid
-PNG encoding for visually unchanged content, and finally deduplicates encoded
-frames by content hash.
+When the capture boundary has not painted since the last accepted frame, the
+SDK reuses that frame without GPU readback. Otherwise it uses a small dHash
+(Hamming distance ≤ 2) to avoid JPEG encoding for near-identical content, and
+finally deduplicates encoded frames by content hash.
 
 Pointer coordinates in event data (`x`, `y`, and swipe `startX`/`startY`) are
 Flutter global logical-pixel coordinates from the pointer event. The SDK
@@ -515,7 +516,7 @@ to an inferred event. The closed outcome vocabulary is:
 | `fresh_accepted` | A fresh frame was accepted. |
 | `exact_content_reused` | An exact content hash reused a compatible frame. |
 | `perceptual_hash_coalesced` | A perceptual hash reused a compatible frame. |
-| `state_signature_short_circuit` | Compatible semantic state made capture unnecessary. |
+| `paint_generation_unchanged` | The capture boundary had not painted since the last accepted frame. |
 | `screenshot_budget_skip` | Degraded screenshot budget skipped eligible work. |
 | `superseded_route_epoch` | Navigation superseded the request's route epoch. |
 | `paint_readiness_timeout` | A fresh paint did not become available in time. |
@@ -547,7 +548,8 @@ sink registration API has not been published.
 
 - Platform views, maps, video textures, and native overlays may be absent or
   incomplete in repaint-boundary screenshots and structural walks.
-- Screenshot readback and PNG encoding perform UI-thread work at checkpoints.
+- Screenshot readback and JPEG encoding perform UI-thread and background-isolate
+  work at checkpoints.
 - Runtime activation/deactivation requires a host rebuild, and activation IDs
   are not yet the emitted session IDs.
 - There is no automatic Android intent-extra/deep-link bridge, offline file

packages/tugboat/benchmark/screenshot_budget_baseline.dart

@@ -5,7 +5,7 @@
 /// enabling aggressive degradation in production profiles.
 class ScreenshotBudgetBaseline {
   static const window = Duration(seconds: 5);
-  static const budgetMicros = 80 * 1000;
+  static const budgetMicros = 60 * 1000;
   static const maxAvgEncodeMicros = 50 * 1000;
   static const maxAvgReadbackMicros = 40 * 1000;
 }

packages/tugboat/example/pubspec.yaml

@@ -32,7 +32,7 @@ resolution: workspace
 dependencies:
   flutter:
     sdk: flutter
-  tugboat: ^0.7.0
+  tugboat: ^0.7.1
 
   # The following adds the Cupertino Icons font to your application.
   # Use with the CupertinoIcons class for iOS style icons.

packages/tugboat/lib/src/collector_http_sink.dart

@@ -51,6 +51,13 @@ class CollectorHttpSink implements TugboatCaptureSink {
   bool _framesNeedRetry = false;
   final List<Map<String, Object?>> _pendingEvents = [];
   final List<_PendingFrameUpload> _pendingFrames = [];
+  /// Frame numbers superseded while an upload batch was in flight. Failed
+  /// batches are re-filtered against this set before requeue so a newer frame
+  /// can drop superseded scroll/duplicate uploads that left the pending queue.
+  final Set<int> _supersededInFlightFrameNos = <int>{};
+
+  /// Uploads currently awaiting an HTTP response, if any.
+  List<_PendingFrameUpload>? _inFlightFrameUploads;
   final List<List<Map<String, Object?>>> _retryBatches = [];
   final List<_PendingSessionLifecycle> _pendingLifecycle = [];
 
@@ -104,6 +111,8 @@ class CollectorHttpSink implements TugboatCaptureSink {
     _collectorSessionId = null;
     _pendingEvents.clear();
     _pendingFrames.clear();
+    _supersededInFlightFrameNos.clear();
+    _inFlightFrameUploads = null;
     _retryBatches.clear();
     _pendingLifecycle.clear();
     _framesNeedRetry = false;
@@ -170,7 +179,15 @@ class CollectorHttpSink implements TugboatCaptureSink {
       );
       return;
     }
-    _pendingFrames.add(_PendingFrameUpload(frameNo: frameNo, bytes: bytes));
+    _supersedePendingFrames(frame, frameNo);
+    _pendingFrames.add(
+      _PendingFrameUpload(
+        frameNo: frameNo,
+        bytes: bytes,
+        trigger: frame.trigger,
+        contentHash: frame.contentHash,
+      ),
+    );
     _trimPendingFrames();
     // While a frame upload is retrying, rely on the periodic flush timer.
     if (!_framesNeedRetry) {
@@ -321,6 +338,8 @@ class CollectorHttpSink implements TugboatCaptureSink {
     _client.close();
     _pendingEvents.clear();
     _pendingFrames.clear();
+    _supersededInFlightFrameNos.clear();
+    _inFlightFrameUploads = null;
     _retryBatches.clear();
     _pendingLifecycle.clear();
     _flushInFlight = null;
@@ -553,10 +572,13 @@ class CollectorHttpSink implements TugboatCaptureSink {
     final uploads = List<_PendingFrameUpload>.from(_pendingFrames)
       ..sort((a, b) => a.frameNo.compareTo(b.frameNo));
     _pendingFrames.clear();
+    _supersededInFlightFrameNos.clear();
+    _inFlightFrameUploads = uploads;
 
     // Drop extracted uploads when the session was reset mid-flush rather than
     // sending them under a stale collector id.
     if (!_isCurrentEpoch(epoch)) {
+      _inFlightFrameUploads = null;
       return;
     }
     final request = http.MultipartRequest(
@@ -586,17 +608,41 @@ class CollectorHttpSink implements TugboatCaptureSink {
       final result = _classifyResponse(response.statusCode);
       _framesNeedRetry = result == _SendResult.retry;
       if (_framesNeedRetry) {
-        _pendingFrames.insertAll(0, uploads);
-        _trimPendingFrames();
+        _requeueFailedUploads(uploads);
       }
     } catch (_) {
       if (!_isCurrentEpoch(epoch)) return;
       _framesNeedRetry = true;
-      _pendingFrames.insertAll(0, uploads);
-      _trimPendingFrames();
+      _requeueFailedUploads(uploads);
+    } finally {
+      if (identical(_inFlightFrameUploads, uploads)) {
+        _inFlightFrameUploads = null;
+      }
+      _supersededInFlightFrameNos.clear();
     }
   }
 
+  void _requeueFailedUploads(List<_PendingFrameUpload> uploads) {
+    final retained = uploads
+        .where(
+          (upload) => !_supersededInFlightFrameNos.contains(upload.frameNo),
+        )
+        .toList(growable: false);
+    final dropped = uploads.length - retained.length;
+    if (dropped > 0) {
+      debugPrint(
+        '[tugboat] collector dropped $dropped superseded in-flight '
+        'frame(s) before retry',
+      );
+    }
+    if (retained.isEmpty) {
+      _framesNeedRetry = false;
+      return;
+    }
+    _pendingFrames.insertAll(0, retained);
+    _trimPendingFrames();
+  }
+
   void _trimPendingEvents() {
     if (_pendingEvents.length <= _config.maxPendingEvents) return;
     final dropped = _pendingEvents.length - _config.maxPendingEvents;
@@ -606,6 +652,46 @@ class CollectorHttpSink implements TugboatCaptureSink {
     );
   }
 
+  void _supersedePendingFrames(TugboatFrame incoming, int incomingFrameNo) {
+    if (_pendingFrames.isNotEmpty) {
+      final before = _pendingFrames.length;
+      _pendingFrames.removeWhere(
+        (pending) => _shouldSupersedePending(pending, incoming),
+      );
+      final dropped = before - _pendingFrames.length;
+      if (dropped > 0) {
+        debugPrint(
+          '[tugboat] collector superseded $dropped pending frame(s) '
+          'before enqueueing frame $incomingFrameNo',
+        );
+      }
+    }
+
+    final inFlight = _inFlightFrameUploads;
+    if (inFlight == null || inFlight.isEmpty) return;
+    var marked = 0;
+    for (final upload in inFlight) {
+      if (_shouldSupersedePending(upload, incoming) &&
+          _supersededInFlightFrameNos.add(upload.frameNo)) {
+        marked++;
+      }
+    }
+    if (marked > 0) {
+      debugPrint(
+        '[tugboat] collector marked $marked in-flight frame(s) superseded '
+        'by frame $incomingFrameNo',
+      );
+    }
+  }
+
+  bool _shouldSupersedePending(
+    _PendingFrameUpload pending,
+    TugboatFrame incoming,
+  ) {
+    return pending.trigger == TugboatFrameTrigger.scroll ||
+        pending.contentHash == incoming.contentHash;
+  }
+
   void _trimPendingFrames() {
     if (_pendingFrames.length <= _config.maxPendingFrames) return;
     final dropped = _pendingFrames.length - _config.maxPendingFrames;
@@ -674,8 +760,15 @@ class _PendingSessionLifecycle {
 }
 
 class _PendingFrameUpload {
-  const _PendingFrameUpload({required this.frameNo, required this.bytes});
+  const _PendingFrameUpload({
+    required this.frameNo,
+    required this.bytes,
+    required this.trigger,
+    required this.contentHash,
+  });
 
   final int frameNo;
   final Uint8List bytes;
+  final TugboatFrameTrigger trigger;
+  final String contentHash;
 }