Skip to content

[Replay] Define SDK release and manual production replay acceptance gate #13

Description

@Chinmay-KB

Problem

Local tests cannot prove that real Blend routes, rendering delays, masking, upload timing, and production replay reconstruction are coherent. The SDK work must not be declared complete merely because unit/widget tests pass.

Scope

Document and execute a bounded SDK release/canary procedure whose final acceptance is manual inspection of production replays from the updated SDK. This issue owns verification and evidence collection, not behavioral implementation.

Prerequisites

  • All behavioral issues in the SDK replay-correctness milestone are closed.
  • The integration race matrix passes.
  • A new SDK version is published and the host app uses that exact revision/version.
  • Production events expose the SDK version and capture outcome/provenance needed to identify the cohort.

Manual session set

Capture multiple real sessions containing:

  • normal same-route taps and true no-ops;
  • slow route push and route replacement;
  • rapid consecutive navigation;
  • modal/bottom-sheet open and close;
  • tap immediately after destination screen appears;
  • scrolling followed by navigation;
  • external picker/background/foreground flow;
  • at least one screenshot-budget degradation or classified capture failure when reproducible.

Acceptance checklist

For every sampled session verify:

  • tap marker lands on the control visible in its before-frame;
  • target anchor, route, signature, and frame describe the same screen;
  • navigation-producing taps do not emit an early unrelated noVisibleChange;
  • destination-screen actions do not reuse origin-screen frames;
  • changed outcomes have either a fresh visual frame or an explicit semantic-only classification;
  • route events show the rendered destination rather than splash/stale UI;
  • frame reuse is explainable through classified diagnostics;
  • no referenced frame is unavailable;
  • SDK/app version and session IDs are recorded with the evidence.

Exit criteria

  • The reviewer records session IDs and timestamps for every tested flow.
  • No unexplained cross-route/cross-step settle, wrong-screen marker, stale route frame, or unavailable referenced frame appears in the acceptance cohort.
  • Any failure opens a new narrowly scoped issue and keeps this gate open.
  • Broad rollout is not approved until the user manually confirms the sampled production replays look correct.

Dependencies

Non-goals

  • Automatically modifying dashboard events.
  • Treating local fixtures as production acceptance.
  • Expanding capture to continuous video.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions