Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ verified in their own repositories.

## Current compatibility

- package version: `0.4.17`;
- session JSON schema: `8`;
- package version: `0.5.0`;
- session JSON schema: `9`;
- fingerprint schema: `6`;
- minimum Dart SDK: `3.9.2`;
- minimum Flutter SDK: `3.35.0`.
Expand Down
45 changes: 7 additions & 38 deletions docs/design/capture-and-fingerprint.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ current controller and keeps future calls to `wrapApp` inert. Runtime
requiring a host rebuild. `deactivate()` tears capture down through the same
gate. Pause/hidden flush pending delivery; detach ends the session once.

Identity fields (session schema **v7**; readers accept v6):
Identity fields (session schema **v9**; compatibility readers accept v6–v9):

- `activationRequestId` — host request correlation
- `captureSessionId` — SDK-emitted session (`session.id`)
Expand All @@ -76,7 +76,10 @@ emits exact build and fingerprint-schema provenance only.
## Session and event model

The controller owns one bounded, in-memory `TugboatSession`. Serialized session
JSON is schema version `7`. Readers accept schema versions `6` and `7`.
JSON is schema version `9`. Compatibility readers accept schema versions
`6` through `9`. Schema v9 does not write `controlValue`,
`controlValueTransition`, or `semanticAnnotation` in event `data`;
those fields are optional historic data in older sessions only.

The session stores:

Expand Down Expand Up @@ -235,43 +238,9 @@ Developer-authored identity strings can still be emitted:
- widget type names or configured `widgetNames` replacements;
- canonical structural paths.

Interaction events may also carry a `controlValue` payload (schema version 4) for
valued controls (checkbox, switch, radio, slider, dropdown / menu item, chip)
and for hit targets that expose Flutter semantic annotations:

- bools and numbers are emitted literally;
- enums and developer identifiers are emitted literally;
- arbitrary strings, including numeric strings and single-word values, are
emitted literally;
- explicit custom-control values can be supplied with
`TugboatControlValueScope`, including a stable `controlKey`, optional unit,
and numeric `min`, `max`, and `step` metadata.

`tap` includes a `controlValue` snapshot sampled at pointer-down.
`tap_settled` uses the distinct `controlValueTransition` contract with
`before` / `after` snapshots. Its post-callback sample stays bound to the
original hit element, so later taps, route changes, or dismissed overlays
cannot donate unrelated control state. Slider drags that become `swipe` events
carry a `controlValue` snapshot sampled at pointer-up.

When a typed widget value is unavailable (custom GestureDetector rows, bottom
sheets, etc.), the SDK still samples `SemanticsProperties` / live semantics
nodes under the pointer and records raw `semanticValue` / `semanticLabel`.
Standard controls may include both widget
state and semantic annotations under `sources: ["semantics","widget"]`.

Independently, every interaction event (`tap`, `tap_settled`, `swipe`,
`scroll_start`, `scroll_end`) may carry a top-level `semanticAnnotation`
payload (schema version 2) whenever Flutter semantics expose an identifier, label, value, or
selection flag on the target. This covers ordinary buttons and scrollables as
well as valued controls. The field is named `semanticAnnotation` to avoid
colliding with `tap_settled.data.settleObservation.semantic` (state-signature
change evidence).

Bounds, pointer coordinates, scroll metrics, and masked screenshot pixels are
also capture data. Apps must treat tags, route names, subview labels, and
semantic value/label tokens as telemetry and avoid putting raw user PII in
them.
also capture data. Apps must treat tags, route names, and subview labels as
telemetry and avoid putting user data in them.

## Screenshot pipeline

Expand Down
18 changes: 11 additions & 7 deletions docs/integration/production-replay-acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,16 @@ database receipt alone as proof that a replay is correct.

## Current acceptance status

Interaction consolidation shipped in SDK **0.4.15** (canonical `interaction`
events, 1,250 ms delayed claim window, diagnostic stream isolation). Use
The current SDK release candidate is **0.5.0**, which writes session schema
**v9**. It preserves structural interaction replay while no longer emitting
`controlValue`, `controlValueTransition`, or `semanticAnnotation` in event
data. Treat the absence of those fields as the expected privacy boundary, not
as missing capture evidence.

[`production-replay-acceptance-0.4.15.md`](./production-replay-acceptance-0.4.15.md)
for the Blend scoring gates. Collector/Context Graph migration onto
`stream: semantic` interactions remains a follow-up before legacy
`tap`/`tap_settled` projection can be removed.
remains a historical acceptance record for interaction consolidation. Do not
use its SDK version, legacy-projection assumptions, or scoring baseline as the
current release contract.

Production acceptance #13/#14 remains open for rapid/nested modal chains and
programmatic/automatic navigation gaps. Record those observations as SDK
Expand Down Expand Up @@ -160,7 +164,7 @@ flows share a session, list the event IDs or timestamps that delimit each flow.

Wait until the collector session has finalized and the replay is available in
the production website. Filter to the recorded Blend build and SDK version
`0.4.10` (or the version under test), then open every recorded session.
under test (`0.5.0` for this release), then open every recorded session.

For each interaction, inspect the actual replay UI and verify:

Expand Down Expand Up @@ -189,7 +193,7 @@ Use one row per production session:

| Session ID | UTC range | Blend build | SDK version / SHA | Flows | Frame availability | Route/action coherence | Verdict | Follow-up |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `<collector-session-id>` | `<start> - <end>` | `<version+build>` | `0.4.10 / <sha>` | `<flows>` | pass/fail | pass/fail | accept/reject | `<issue or none>` |
| `<collector-session-id>` | `<start> - <end>` | `<version+build>` | `<sdk-version> / <sha>` | `<flows>` | pass/fail | pass/fail | accept/reject | `<issue or none>` |

The cohort passes only when:

Expand Down
25 changes: 25 additions & 0 deletions packages/tugboat/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,28 @@
## 0.5.0

### Breaking changes

- **Control-value and semantic-annotation telemetry** — session JSON writers
now emit schema version 9 and no longer write `controlValue`,
`controlValueTransition`, or `semanticAnnotation` in event `data`.
Readers that support historic schemas should continue to tolerate versions
6–8, where those fields may be present.
- **Removed public barrel exports** —
`TugboatEncodedControlScalar`, `TugboatVisibleControlValue`,
`TugboatControlValueScope`, `TugboatControlValue`, and
`TugboatSemanticAnnotation`.
- **Removed public schema constants** —
`tugboatControlValueSchemaVersion`,
`tugboatControlValueTransitionSchemaVersion`, and
`tugboatSemanticAnnotationSchemaVersion`.
- **Removed public extraction and merge helpers** —
`tugboatControlValueForWidget`,
`tugboatControlValueFromSemanticsProperties`,
`tugboatControlValueFromSemanticsNode`,
`tugboatSemanticAnnotationFromProperties`,
`tugboatSemanticAnnotationFromNode`,
`tugboatMergeSemanticAnnotations`, and `tugboatMergeControlValues`.

## 0.4.18

### Fixed
Expand Down
71 changes: 34 additions & 37 deletions packages/tugboat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ checkpoints around meaningful interactions, compact structural anchors, route
transitions, scrolling evidence, and optional viewport semantic maps. Capture
can be sent to the local exploration WebSocket, the HTTP collector, or both.

The current package version is `0.4.18`. Session JSON uses schema version `8`
(readers still accept `6`), and structural fingerprints use fingerprint schema
version `6`.
The current package version is `0.5.0`. Session JSON writers emit schema
version `9`; compatibility readers should accept versions `6` through
`9`. Structural fingerprints use fingerprint schema version `6`.

## Install

Expand All @@ -19,6 +19,29 @@ import 'package:tugboat/tugboat.dart';

The package requires Dart 3.9.2 or newer and Flutter 3.35.0 or newer.

## Migrating to 0.5.0

This is a breaking release. Session JSON written by 0.5.0 uses schema version
`9` and no longer includes `controlValue`, `controlValueTransition`, or
`semanticAnnotation` in event `data`. Consumers reading historical schemas
`6`–`8` should treat those fields as optional historic data; new captures do
not provide them.

The public `package:tugboat/tugboat.dart` barrel no longer exports:

- `TugboatEncodedControlScalar`, `TugboatVisibleControlValue`,
`TugboatControlValueScope`, `TugboatControlValue`, and
`TugboatSemanticAnnotation`;
- `tugboatControlValueSchemaVersion`,
`tugboatControlValueTransitionSchemaVersion`, and
`tugboatSemanticAnnotationSchemaVersion`;
- `tugboatControlValueForWidget`,
`tugboatControlValueFromSemanticsProperties`,
`tugboatControlValueFromSemanticsNode`,
`tugboatSemanticAnnotationFromProperties`,
`tugboatSemanticAnnotationFromNode`,
`tugboatMergeSemanticAnnotations`, and `tugboatMergeControlValues`.

## Minimal integration

Install both the app wrapper and navigator observer. Capture is dormant by
Expand Down Expand Up @@ -209,11 +232,10 @@ Available mask levels are `explicitOnly`, `allTextAndMedia`, `allText`,
stay visible; other custom-painted or decorated image surfaces are not
classified by this mode, so wrap them in `TugboatSensitive` when needed).

Control values and semantic strings are retained verbatim so session summaries
and aggregate analysis can use values such as slider positions, video duration,
and selected templates. Dynamic list discriminators remain hashed before they
enter canonical paths. Telemetry also includes developer-authored routing and
identity strings where applicable:
The structural telemetry does not retain arbitrary `Text`, accessibility,
tooltip, or icon label strings. Dynamic list discriminators are hashed before
they enter canonical paths. Telemetry does include developer-authored routing
and identity strings where applicable:

- route names in `route_change.data` and anchor `routeKey` fields;
- `TugboatSubView.label` in state/scroll context;
Expand All @@ -223,35 +245,10 @@ identity strings where applicable:
- normalized bounds, pointer coordinates, scroll metrics, and screenshot
pixels after the configured masking policy is applied.

Screenshots and telemetry can contain rendered or semantic user content. Choose
an explicit production masking policy and test custom widgets, platform views,
overlays, and semantic labels before enabling production capture.

### Explicit custom-control values

Standard Flutter controls expose their typed state automatically. Wrap custom
controls when the app knows a more useful stable key, unit, or range:

```dart
TugboatControlValueScope(
controlKey: 'video_duration',
role: 'slider',
unit: 'milliseconds',
min: 1_000,
max: 60_000,
step: 1_000,
value: TugboatVisibleControlValue.duration(
const Duration(seconds: 15),
),
child: MyDurationSlider(),
)
```

For template or preset selection, use a stable enum identifier:

```dart
value: TugboatVisibleControlValue.enumId('modern_minimal'),
```
Screenshots are the only captured surface that can contain rendered user
content. Choose an explicit production masking policy and test custom widgets,
platform views, and overlays in the target app before enabling production
capture.

## Event and frame model

Expand Down
23 changes: 0 additions & 23 deletions packages/tugboat/example/lib/screens/profile_screen.dart
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ class _ProfileScreenState extends State<ProfileScreen> {
bool _darkMode = false;
double _notificationVolume = 0.6;
String _language = 'English';
int _generationCount = 1;

@override
Widget build(BuildContext context) {
Expand Down Expand Up @@ -144,28 +143,6 @@ class _ProfileScreenState extends State<ProfileScreen> {
),
],
),
DemoSection(
title: 'Generation settings',
children: [
Wrap(
spacing: 8,
children: [
for (final count in [1, 2, 3, 4])
Semantics(
label: 'Number of generations',
value: count.toString(),
selected: _generationCount == count,
child: FilledButton(
key: Key('generation-count-$count'),
onPressed: () =>
setState(() => _generationCount = count),
child: Text(count.toString()),
),
),
],
),
],
),
DemoSection(
title: 'Account actions',
children: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tugboat/example/pubspec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ resolution: workspace
dependencies:
flutter:
sdk: flutter
tugboat: ^0.4.0
tugboat: ^0.5.0

# The following adds the Cupertino Icons font to your application.
# Use with the CupertinoIcons class for iOS style icons.
Expand Down
51 changes: 1 addition & 50 deletions packages/tugboat/example/test/widget_test.dart
Original file line number Diff line number Diff line change
@@ -1,23 +1,7 @@
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:tugboat/tugboat.dart';
import 'package:tugboat_example/main.dart';

Future<void> _waitForTugboatEvents(WidgetTester tester) async {
for (var attempt = 0; attempt < 12; attempt++) {
await tester.pump(const Duration(milliseconds: 50));
}
}

Map<String, Object?> _semanticAnnotation(TugboatEvent event) {
final raw = event.data['semanticAnnotation'];
return Map<String, Object?>.from(raw! as Map);
}
import 'package:flutter_test/flutter_test.dart';

void main() {
setUp(TugboatReplay.resetForTest);
tearDown(TugboatReplay.resetForTest);

testWidgets('demo app loads home screen', (tester) async {
await tester.pumpWidget(const ReplayDemoApp());
await tester.pump();
Expand All @@ -28,37 +12,4 @@ void main() {
await tester.pump(const Duration(milliseconds: 300));
}
});

testWidgets(
'generation count tap emits its semantic parameter label and value',
(tester) async {
await tester.binding.setSurfaceSize(const Size(800, 1200));
addTearDown(() => tester.binding.setSurfaceSize(null));
TugboatReplay.activate(
activationRequestId: 'example-semantic-parameter-test',
profile: TugboatCaptureProfile.productionLean,
);
await tester.pumpWidget(const ReplayDemoApp());
await _waitForTugboatEvents(tester);

await tester.tap(find.text('Profile & settings'));
await tester.pumpAndSettle();
await _waitForTugboatEvents(tester);

await tester.tap(find.byKey(const Key('generation-count-3')));
await _waitForTugboatEvents(tester);

final tap = TugboatReplay.controller!.session!.events.lastWhere(
(event) => event.type == 'tap',
);
final annotation = _semanticAnnotation(tap);

expect(annotation['label'], {
'kind': 'string',
'value': 'Number of generations',
});
expect(annotation['value'], {'kind': 'string', 'value': '3'});
expect(annotation['selected'], isFalse);
},
);
}
Loading
Loading