feat(replay): control values + semanticAnnotation on all interactions

[Chinmay-KB]

Summary

Per-interaction capture for enrichment:

1. controlValue for typed standard controls with before/after on settle.
2. semanticAnnotation on every interaction when Flutter semantics expose an identifier, label, value, or selection flags.

Custom controls

Use Flutter semantics for custom hit targets, or TugboatControlValueScope when the app knows a stable key, typed value, unit, or numeric range.

Semantics(
  button: true,
  value: '30',
  label: 'Duration 30 seconds',
  selected: true,
  child: GestureDetector(...),
)

Privacy

Control values and semantic labels are retained raw. Host apps must treat semantic values as telemetry and avoid placing user PII in them; use TugboatSensitive for content that must remain private.

Release documentation

• Package/runtime version: 0.4.17
• Changelog: packages/tugboat/CHANGELOG.md
• Integration and data-contract details: packages/tugboat/README.md and docs/design/capture-and-fingerprint.md

Verification

• dart format packages/tugboat/lib packages/tugboat/test
• dart analyze packages/tugboat
• flutter test test/control_value_test.dart (25 passing)

[Copilot]

Pull request overview

This PR adds privacy-safe per-interaction enrichment to Tugboat replay capture by (1) sampling typed “control values” for common controls and (2) attaching a unified semanticAnnotation payload to all interaction events when Flutter semantics expose identifier/label/value/selection flags.

Changes:

• Capture and serialize controlValue snapshots (including before/after on tap_settled) for standard widgets and semantics-backed targets.
• Attach semanticAnnotation to tap, tap_settled, swipe, scroll_start, and scroll_end, with ancestor/descendant semantics merged for better labels.
• Add comprehensive widget tests and update design docs describing payload shapes and privacy rules.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/tugboat/test/control_value_test.dart	Adds tests covering control value capture, hashing rules, semantic annotations for taps and scrolls.
packages/tugboat/lib/tugboat.dart	Exposes new control/semantics capture APIs and schema version constants via public exports.
packages/tugboat/lib/src/widget_roles.dart	Avoids generic callback cast hazards for radio enablement checks by using dynamic access.
packages/tugboat/lib/src/semantics_flags_compat.dart	Adds cross-SDK helpers to read enabled/checked/toggled/selected from semantics flags.
packages/tugboat/lib/src/controller.dart	Plumbs controlValue and semanticAnnotation into emitted interaction events (tap/settled/swipe/scroll).
packages/tugboat/lib/src/control_value.dart	Introduces privacy-safe scalar encoding, TugboatControlValue, and TugboatSemanticAnnotation + merge/build helpers.
packages/tugboat/lib/src/anchors.dart	Wires the new control/semantics implementation into the anchors library via a new part file.
packages/tugboat/lib/src/anchor_resolver.dart	Adds hit-test + semantics-tree sampling entry points for controlValueAt and semanticAnnotationAt.
docs/design/capture-and-fingerprint.md	Documents the new payloads and privacy considerations for semantic tokens.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

packages/tugboat/lib/src/semantics_flags_compat.dart:31

• semanticsCheckedFromFlags references CheckedState.isTrue, but this package supports Flutter >=3.35 and CheckedState is only present in newer SDKs, so this will fail to compile on 3.35. Since checked is treated dynamically already, avoid referencing the enum symbol directly and derive the bool from toString() instead.

  try {
    if (checked.toString().endsWith('.none')) return null;
    if (checked.toString().endsWith('.mixed')) return null;
    return checked == CheckedState.isTrue;
  } catch (_) {

packages/tugboat/lib/src/control_value.dart:52

• When parsing numeric strings, num.tryParse will also accept "NaN"/"Infinity" and produce non-finite doubles, which will later break JSON encoding. Filter those out (or encode them as string tokens) before returning a number kind.

      final asNum = num.tryParse(trimmed);
      if (asNum != null) {
        return TugboatEncodedControlScalar._(kind: 'number', value: asNum);
      }

packages/tugboat/lib/src/control_value.dart:37

• TugboatEncodedControlScalar.encode returns numbers as-is, which can include non-finite doubles (NaN/Infinity). Those values will cause jsonEncode(...) of session payloads to throw, breaking capture/export. Consider encoding non-finite doubles as string tokens instead.

This issue also appears on line 49 of the same file.

    if (raw is num) {
      return TugboatEncodedControlScalar._(kind: 'number', value: raw);
    }

packages/tugboat/lib/src/controller.dart:2307

• recordPointerDown calls resolver.buildTapContext(...) and then immediately calls resolver.controlValueAt(position) / resolver.semanticAnnotationAt(position), which each perform their own hit-test walk. This introduces multiple hit tests per tap-down and can be a noticeable overhead in complex trees. Consider extending buildTapContext (or the underlying hit-test loop) to also return controlValue/semanticAnnotation so the tap path is only walked once.

      target = tapContext.target;
      tapInventory = tapContext.inventory;
      controlValue = resolver.controlValueAt(position);
      semantic = resolver.semanticAnnotationAt(position);