Improve screenshot capture performance (0.7.1) - #37
Merged
Conversation
Skip the full GPU readback/encode path when the capture boundary has not painted since the last accepted frame, and remove the post-capture stateAnchor short circuit that only avoided store/upload. Document JPEG frame format and the new paint_generation_unchanged diagnostic outcome. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Return JPEG bytes and SHA-256 together from compute() so hashing no longer blocks the UI isolate after every non-coalesced capture. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Replace per-frame compute() spawns with a long-lived worker that accepts TransferableTypedData, keeping JPEG encode and SHA-256 off the UI isolate without a second full-frame copy. Widget tests fall back to compute under FakeAsync. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Replace the second full-size picture.toImage mask composite with in-place RGBA fills on the encode worker, cutting one full-frame GPU raster on masked captures while preserving the dark mask fill color. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Remove the UI-isolate 9x8 thumbnail round-trip (picture.toImage + toByteData) and derive the perceptual hash from the masked RGBA buffer on the encode worker before deciding whether JPEG encoding is needed. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Treat perceptual hashes within 2 bits as visually unchanged so minor anti-alias shimmer no longer forces full JPEG encode, store, and upload. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Drop queued scroll samples and same-contentHash frames when a newer frame is enqueued, so capture bursts upload the latest observation instead of every intermediate raster. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Engage eligible-capture skipping sooner under load now that paint-generation and dHash coalesce replace the old post-capture state-signature short circuit. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Bump tugboat and tugboat_dio to 0.7.1 and document the paint-generation gate, encode-isolate path, dHash tolerance, frame supersession, and tighter screenshot budget. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Co-authored-by: Chinmay Kabi <chinmay@blend.to>
|
Cursor Agent can help with this pull request. Just |
There was a problem hiding this comment.
Pull request overview
Optimizes screenshot capture and releases Tugboat 0.7.1.
Changes:
- Adds paint-generation gating and persistent isolate encoding.
- Introduces tolerant dHash coalescing and upload supersession.
- Updates budgets, tests, documentation, and package versions.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
packages/tugboat/lib/src/screenshot_capturer.dart |
Integrates capture gating and worker encoding. |
packages/tugboat/lib/src/screenshot_encode.dart |
Implements masking, hashing, and JPEG encoding. |
packages/tugboat/lib/src/screenshot_encode_isolate.dart |
Adds the persistent encode worker. |
packages/tugboat/lib/src/perceptual_hash.dart |
Adds tolerant dHash comparison. |
packages/tugboat/lib/src/controller.dart |
Handles new capture outcomes and lifecycle. |
packages/tugboat/lib/src/collector_http_sink.dart |
Supersedes queued frame uploads. |
packages/tugboat/lib/src/replay_config.dart |
Lowers the screenshot budget. |
packages/tugboat/lib/src/health.dart |
Updates budget tracking defaults. |
packages/tugboat/lib/src/sdk_version.dart |
Bumps the SDK version. |
packages/tugboat/test/tugboat_replay_test.dart |
Tests dHash tolerance. |
packages/tugboat/test/replay/screenshot_fresh_paint_test.dart |
Uses deterministic inline encoding. |
packages/tugboat/test/replay/screenshot_encode_isolate_test.dart |
Tests isolate encoding and masking. |
packages/tugboat/test/replay/capture_diagnostics_test.dart |
Updates diagnostic vocabulary. |
packages/tugboat/test/collector_http_sink_test.dart |
Tests upload supersession. |
packages/tugboat/benchmark/screenshot_budget_baseline.dart |
Updates the benchmark budget. |
packages/tugboat/pubspec.yaml |
Bumps Tugboat to 0.7.1. |
packages/tugboat/example/pubspec.yaml |
Updates the example dependency. |
packages/tugboat/README.md |
Documents capture behavior and defaults. |
packages/tugboat/CHANGELOG.md |
Records the 0.7.1 release. |
packages/tugboat_dio/pubspec.yaml |
Bumps the adapter and dependency. |
packages/tugboat_dio/README.md |
Updates installation versions. |
packages/tugboat_dio/CHANGELOG.md |
Records the compatibility release. |
docs/integration/collector.md |
Documents JPEG uploads and supersession. |
docs/design/capture-and-fingerprint.md |
Updates the screenshot pipeline design. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| commands.send( | ||
| ScreenshotEncodeIsolateCommand( | ||
| jobId: jobId, | ||
| rgba: TransferableTypedData.fromList([input.rgba]), |
Comment on lines
+159
to
+166
| await ensureStarted(); | ||
| final commands = _commands; | ||
| if (commands == null) { | ||
| throw StateError('IsolateScreenshotEncoder failed to start'); | ||
| } | ||
| final jobId = _nextJobId++; | ||
| final completer = Completer<ScreenshotEncodeResult>(); | ||
| _pending[jobId] = completer; |
Comment on lines
+446
to
+447
| if (paintGeneration != null) { | ||
| _lastAcceptedPaintGeneration = paintGeneration; |
Comment on lines
+363
to
+366
| if (allowPaintGenerationSkip && | ||
| !force && | ||
| paintGeneration != null && | ||
| paintGeneration == _lastAcceptedPaintGeneration) { |
Comment on lines
+620
to
+624
| _pendingFrames.removeWhere( | ||
| (pending) => | ||
| pending.trigger == TugboatFrameTrigger.scroll || | ||
| pending.contentHash == incoming.contentHash, | ||
| ); |
Fix isolate dispose races, commit paint generation only after frame acceptance (keyed by render-object identity), supersede in-flight uploads before retry, correct TransferableTypedData copy claims, and add paint-generation / in-flight supersession regression tests. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Suppressed comments (3)
packages/tugboat/lib/src/screenshot_capturer.dart:467
- Updating
_lastDHashfor every result makes the new tolerance drift: if stored frame H0 is reused for H1 at distance 2, state advances to H1, so H2 can be distance 2 from H1 but 4 from the still-reused H0 and also be skipped; repeated changes can diverge arbitrarily. This also commits hashes from attempts later discarded as stale. Compare against and commit only the hash of an actually accepted/stored frame.
if (encoded.dHash != null) {
_lastDHash = encoded.dHash;
packages/tugboat/lib/src/screenshot_encode.dart:95
computeDHashFromRgbasamples only one source pixel for each of the 72 hash cells (perceptual_hash.dart:13-22). The removed path first rendered the full screenshot down to a filtered 9×8 thumbnail; calling this helper directly on a viewport makes changes away from those sample points invisible, so meaningful text or control updates can be coalesced. Downsample/aggregate the full image to 9×8 in the worker before comparing adjacent pixels.
final dHash = computeDHashFromRgba(rgba, input.width, input.height);
packages/tugboat/lib/src/controller.dart:2180
- Coalescing is still allowed when this request has no compatible frame. In that case either the paint-generation gate or dHash can return an empty result,
_reuseWithoutCapture/the dHash branch returnsnoCompatibleFrame, and unchanged subsequent requests repeat the same skip instead of ever creating usable evidence (for example after a boundary-transform generation change). Bypass coalescing whenever_compatibleFrameFor(context)is null.
force: force || requiresFreshPaint,
Comment on lines
+384
to
+388
| if (allowPaintGenerationSkip && | ||
| !force && | ||
| paintGeneration != null && | ||
| identical(boundary, _lastAcceptedBoundary) && | ||
| paintGeneration == _lastAcceptedPaintGeneration) { |
Comment on lines
+691
to
+692
| return pending.trigger == TugboatFrameTrigger.scroll || | ||
| pending.contentHash == incoming.contentHash; |
Paint-generation coalesce now uses a subtree paint signature so nested RepaintBoundary paints invalidate the gate. Collector no longer drops pending/in-flight frames by trigger or content hash, since events reference exact frame IDs with no upload alias. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Commit dHash only after controller accept/reuse, box-filter dHash cells on large buffers, bypass paint/dHash coalesce when no compatible frame exists, and guard isolate encode against dispose races. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.
Suppressed comments (4)
packages/tugboat/lib/src/screenshot_capturer.dart:475
- Do not advance the dHash baseline before the controller accepts or successfully reuses this result.
captureAttemptcan discard the result after encoding when its route/session becomes stale, and dHash coalescing can also return no compatible frame; the next attempt can then coalesce against an unaccepted image, return empty bytes, and again produce no frame. Commit the dHash alongside the paint signature only after acceptance/reuse.
height: imageHeight,
maskRects: scaledMasks,
packages/tugboat/lib/src/capture_boundary.dart:52
- This signature misses compositing-only visual updates. Retained layers such as
OpacityLayer,TransformLayer, andOffsetLayercan mutate alpha/transform/offset in place while both the layer andPictureLayer.pictureidentities remain unchanged, so the gate can reuse a stale screenshot even though the rendered pixels changed. Track the relevant layer state or a generation that also advances for composited-layer updates before using equality as a no-readback proof.
signature = Object.hash(signature, identityHashCode(layer));
if (layer is PictureLayer) {
signature = Object.hash(signature, identityHashCode(layer.picture));
}
packages/tugboat/lib/src/controller.dart:2236
- A paint-signature match does not guarantee that a compatible frame exists. For example,
_captureContextadvancesboundaryTransformGenerationwhen the boundary moves, while the retained boundary pixels/signature may remain unchanged; this branch then calls_reuseWithoutCapture, getsnoCompatibleFrame, and never retries the capture. Only enable the gate when a compatible frame is known, or retry without the gate when reuse fails.
);
}
_lastCaptureFailure = null;
if (result.skippedByPaintGeneration) {
packages/tugboat/pubspec.yaml:4
- The release bump leaves the repository's release documentation inconsistent:
docs/README.md:31,docs/integration/production-replay-acceptance.md:15, anddocs/integration/production-replay-acceptance.md:167still identify 0.7.0 as current/under test. Update those references as part of the 0.7.1 release.
version: 0.7.1
Hash composited layer offset/transform/opacity in subtree signatures, retry full capture when paint-skip reuse finds no compatible frame, and align docs to 0.7.1. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Patch release 0.7.1 that makes screenshot capture cheaper so the post-capture
stateAnchorshort circuit can be removed safely.Changes
RepaintBoundarylayers) has not painted; replacestate_signature_short_circuitwithpaint_generation_unchangedpicture.toImagebeforeFrame/afterFrameand multipart has no hash alias)Docs updated for JPEG wire format, paint-signature skip, and budget defaults.
Perf impact (expected)
Test plan
capture_diagnostics_test,screenshot_fresh_paint_test(incl. nestedRepaintBoundarygate),screenshot_encode_isolate_testflutter testsuite in CI