Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ verified in their own repositories.

## Current compatibility

- package version: `0.4.9`;
- package version: `0.4.11`;
- session JSON schema: `7`;
- fingerprint schema: `6`;
- minimum Dart SDK: `3.9.2`;
Expand Down
56 changes: 52 additions & 4 deletions docs/design/capture-and-fingerprint.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Capture and fingerprint architecture

Status: current implementation audit (2026-07-16)
Status: current implementation audit (2026-07-26)
Scope: `packages/tugboat` in this repository

This document describes what the Flutter SDK implements today. It deliberately
Expand Down Expand Up @@ -107,6 +107,52 @@ Events may carry `beforeFrame`, `afterFrame`, `stateAnchor`, `targetAnchor`,
type-specific `data`. Route transition values live in `route_change.data`, not
in a session-level route dictionary.

### Capture lifecycle and attribution

`wrapApp` starts a session only after its repaint boundary has a non-zero
viewport. The session begins with `session_start` and an initial capture
request. Pointer-down records `tap` plus a compatible pre-interaction frame,
then pointer-up either records a swipe or creates one `tap_settled` outcome.
The settled event refers to the initial tap through `relatedEventId` and is
intended to attach an after-frame only when that frame's provenance matches the
observed route epoch. A capture that is unavailable, cancelled, superseded, or
timed out is represented by bounded capture/attachment diagnostics instead of
borrowing the latest frame from another screen.

Frame requests are serialized, may coalesce, and use fresh-paint/readback
checks before publishing. Their provenance records the capture context and
completion state, so exact-content and perceptual deduplication reuse frames
only within a compatible context. `paused` and `hidden` request a delivery
flush after 500 ms; `resumed` cancels that pending flush; `detached`, wrapper
disposal, and deactivation end the session once and initiate sink shutdown.

Input event coordinates (`x`, `y`, and swipe start/delta values) originate as
Flutter global logical-pixel coordinates. The resolver converts a global point
to capture-boundary local coordinates only to hit-test and normalize it against
the viewport; stored coordinates are not capture-boundary-normalized for replay
playback, are neither device pixels nor widget-local, and can therefore produce
fractional overlay drift.

### Navigator and modal routes

Installing `TugboatReplay.navigatorObserver` is intended to record the standard
Navigator push, pop, replace, and remove callbacks automatically. No
per-navigation SDK call is required. Dialogs and `showModalBottomSheet`
instances can participate when their routes use that observed Navigator. Nested
navigators require their own observer integration; native/system overlays
remain outside the Flutter Navigator and repaint-boundary surface.

Every visible navigation change advances a route epoch and has a single route
capture barrier after transition settlement. A later visible navigation
supersedes stale work. This is the implemented intended invariant: a delayed
destination capture or tap-settle operation should not attach the prior route's
frame, and an unavailable compatible frame should remain explicitly degraded.

It is not yet a production-accepted guarantee. Production acceptance #13/#14
remains open: rapid/nested modal chains and programmatic/automatic navigation
can still be absent or degraded. Treat those observations as SDK capture gaps,
not as coherent replay evidence.

Pointer work and post-interaction capture are serialized through a controller
queue. An error in one queued task is caught and logged so later taps, scrolls,
and route captures continue.
Expand Down Expand Up @@ -229,11 +275,13 @@ frames captured before connection are still sent.
| --- | --- | --- | --- |
| `off` | no | no | no |
| `tapResolutionOnly` | yes for active profiles | no | no |
| `full` | yes | yes | no |
| `fullWithDebugLogs` | yes | yes | yes |
| `full` | yes | exploration only | no |
| `fullWithDebugLogs` | yes | exploration only | exploration only |

Exploration holds a persistent Flutter `SemanticsHandle` when semantics are
enabled. Production uses transient semantics. Emitted maps are bounded by
enabled. Production uses transient semantics and never emits semantic-map
events, even in `full` modes; it can still build maps locally for tap
resolution. Emitted exploration maps are bounded by
`viewportSemanticMapMaxNodes` (default `120`) and
`viewportSemanticMapMaxBytes` (default `48000`).

Expand Down
15 changes: 13 additions & 2 deletions docs/integration/production-replay-acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@ The procedure is intentionally strict about build identity. Do not mix sessions
from different SDK revisions, infer the SDK revision from capture time, or use
database receipt alone as proof that a replay is correct.

## Current acceptance status

Production acceptance #13/#14 remains open. The SDK's route-epoch and frame
provenance behavior is an intended invariant, but rapid/nested modal chains and
programmatic/automatic navigation can still be absent or degraded in a
production replay. Record those observations as SDK capture gaps; do not infer
route/action coherence from the intended contract or repair the evidence in the
dashboard. Stored tap coordinates are global logical pixels and are not
capture-boundary-normalized for playback, so fractional overlay drift is also a
known limitation.

## Roles and evidence

Record these people before starting:
Expand Down Expand Up @@ -140,7 +151,7 @@ flows share a session, list the event IDs or timestamps that delimit each flow.

Wait until the collector session has finalized and the replay is available in
the production website. Filter to the recorded Blend build and SDK version
`0.4.9` (or the version under test), then open every recorded session.
`0.4.10` (or the version under test), then open every recorded session.

For each interaction, inspect the actual replay UI and verify:

Expand Down Expand Up @@ -169,7 +180,7 @@ Use one row per production session:

| Session ID | UTC range | Blend build | SDK version / SHA | Flows | Frame availability | Route/action coherence | Verdict | Follow-up |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `<collector-session-id>` | `<start> - <end>` | `<version+build>` | `0.4.9 / <sha>` | `<flows>` | pass/fail | pass/fail | accept/reject | `<issue or none>` |
| `<collector-session-id>` | `<start> - <end>` | `<version+build>` | `0.4.10 / <sha>` | `<flows>` | pass/fail | pass/fail | accept/reject | `<issue or none>` |

The cohort passes only when:

Expand Down
4 changes: 4 additions & 0 deletions packages/tugboat/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@

### Changed

- **Replay capture contract documentation** — documents the implemented route/frame
attribution invariant separately from the still-open production acceptance
gaps for rapid modal chains, automatic navigation, and playback tap-coordinate
alignment.
- **Route-capture barrier** — tap settlement now joins the matching route epoch
instead of reusing the previous route's latest frame. Supersession,
lifecycle cancellation, capture failure, and bounded timeout outcomes
Expand Down
40 changes: 39 additions & 1 deletion packages/tugboat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ checkpoints around meaningful interactions, compact structural anchors, route
transitions, scrolling evidence, and optional viewport semantic maps. Capture
can be sent to the local exploration WebSocket, the HTTP collector, or both.

The current package version is `0.4.9`. Session JSON uses schema version `7`
The current package version is `0.4.11`. Session JSON uses schema version `7`
(readers still accept `6`), and structural fingerprints use fingerprint schema
version `6`.

Expand Down Expand Up @@ -43,6 +43,27 @@ work, but route-change events and route-backed anchors are incomplete. Without
`wrapApp`, no capture controller, repaint boundary, or input/scroll listener is
installed.

### Navigation and overlays

The supplied observer is intended to record standard Navigator `push`, `pop`,
`replace`, and `remove` callbacks without application code calling the replay
controller for each navigation. Dialog and modal-bottom-sheet routes can
participate when they use that observed Navigator. Nested navigators need their
own observer wiring, and native/system overlays are outside the Flutter
Navigator/repaint-boundary contract.

Each visible route change creates a route epoch and waits for the transition
plus the configured settle delay before taking the destination capture. A newer
visible route supersedes an older pending capture. Consequently, a route event
and a related `tap_settled` event are intended to reference a frame compatible
with that route epoch, or report bounded degraded/capture diagnostics rather
than attach an origin-route frame merely because it was the latest frame.

This is an implemented SDK invariant, not a production-accepted guarantee.
Production acceptance for #13/#14 remains open: rapid or nested modal chains
and programmatic/automatic navigation can still be absent or degraded. Treat
those cases as an SDK capture gap, not as coherent replay evidence.

## Capture profiles and runtime state

`TugboatReplayConfig.profile` controls whether the wrapper installs capture
Expand Down Expand Up @@ -177,6 +198,23 @@ The SDK first skips repeated state signatures, then uses a small dHash to avoid
PNG encoding for visually unchanged content, and finally deduplicates encoded
frames by content hash.

Pointer coordinates in event data (`x`, `y`, and swipe `startX`/`startY`) are
Flutter global logical-pixel coordinates from the pointer event. The SDK
converts a copy into its capture boundary's local space only for hit-testing and
normalizing target/viewport-semantic bounds; stored event coordinates are not
capture-boundary-normalized for replay playback. Do not interpret them as
physical pixels or as coordinates relative to an individual widget. Fractional
overlay drift is therefore still possible.

For a tap, `beforeFrame` is selected at pointer-down only if its provenance is
compatible with the tap's route epoch. At pointer-up, a single `tap_settled`
event links back to the `tap` via `relatedEventId` and is intended to contain
an `afterFrame` only when the settled capture is compatible with the observed
route. A missing attachment is explicit in `frameAttachment`/settle diagnostics
rather than a fallback to an unrelated frame. Acceptance remains open for the
navigation cases above, so consumers must still treat absence and degradation
as a capture gap.

During local WebSocket exploration, connecting without an HTTP collector
suppresses new Flutter screenshot capture for UI-thread performance. Events,
anchors, inventories, and semantic evidence continue to stream; the CLI's ADB
Expand Down
2 changes: 1 addition & 1 deletion packages/tugboat/lib/src/sdk_version.dart
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
// Keep this in sync with packages/tugboat/pubspec.yaml. The SDK version test
// reads pubspec.yaml directly so release bumps fail fast if this drifts.
const tugboatSdkVersion = '0.4.10';
const tugboatSdkVersion = '0.4.11';
2 changes: 1 addition & 1 deletion packages/tugboat/pubspec.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: tugboat
description: >-
Screenshot-based session replay with compact interaction anchors for Tugboat.
version: 0.4.10
version: 0.4.11
repository: https://github.com/blendto/tugboat-flutter
issue_tracker: https://github.com/blendto/tugboat-flutter/issues
homepage: https://github.com/blendto/tugboat-flutter
Expand Down
Loading