|
| 1 | +# Production replay release and acceptance |
| 2 | + |
| 3 | +This is the release gate for changes that affect screenshot timing, route |
| 4 | +ownership, state signatures, target anchors, or interaction outcomes. Local |
| 5 | +tests are necessary but do not satisfy this gate. Acceptance requires manually |
| 6 | +playing back a bounded cohort from the released SDK in the production replay |
| 7 | +website. |
| 8 | + |
| 9 | +The procedure is intentionally strict about build identity. Do not mix sessions |
| 10 | +from different SDK revisions, infer the SDK revision from capture time, or use |
| 11 | +database receipt alone as proof that a replay is correct. |
| 12 | + |
| 13 | +## Roles and evidence |
| 14 | + |
| 15 | +Record these people before starting: |
| 16 | + |
| 17 | +| Role | Responsibility | |
| 18 | +| --- | --- | |
| 19 | +| SDK releaser | Lands the SDK stack and records the exact main commit | |
| 20 | +| Blend releaser | Pins Blend to that commit and records the app build | |
| 21 | +| Replay reviewer | Executes the flow matrix and inspects production replays | |
| 22 | + |
| 23 | +The evidence record must contain: |
| 24 | + |
| 25 | +- Tugboat package version and exact Git commit; |
| 26 | +- Blend version, build number, and Git commit; |
| 27 | +- device platform and OS version; |
| 28 | +- capture start and end timestamps in UTC; |
| 29 | +- collector session ID for every inspected replay; |
| 30 | +- the flows present in each session; |
| 31 | +- a per-session verdict and links to any follow-up issues. |
| 32 | + |
| 33 | +## Entry gate |
| 34 | + |
| 35 | +Do not start the production cohort until every item is true: |
| 36 | + |
| 37 | +- all behavioral replay-correctness PRs are merged to `main`; |
| 38 | +- the navigation and interaction race matrix passes on the merged commit; |
| 39 | +- `flutter analyze` and the complete `packages/tugboat` test suite pass; |
| 40 | +- `packages/tugboat/pubspec.yaml` and |
| 41 | + `packages/tugboat/lib/src/sdk_version.dart` contain the same new version; |
| 42 | +- Blend is pinned to the exact merged SDK commit; |
| 43 | +- the deployed Blend build is available to the reviewer; |
| 44 | +- production collection is enabled for the test account/device; |
| 45 | +- the production replay website can filter or otherwise identify the SDK and |
| 46 | + app build cohort. |
| 47 | + |
| 48 | +The HTTP collector sends the package version as `X-Sdk-Version`. App version, |
| 49 | +build number, app ID, and platform are sent with the same requests. Record both |
| 50 | +the SDK version and exact Git commit because a version string alone cannot |
| 51 | +distinguish two builds made from different commits. |
| 52 | + |
| 53 | +## 1. Land and verify the SDK |
| 54 | + |
| 55 | +Merge dependent PRs from the bottom of the stack upward. Never merge a child |
| 56 | +before its parent. After the final PR lands: |
| 57 | + |
| 58 | +```sh |
| 59 | +git switch main |
| 60 | +git pull --ff-only |
| 61 | +git status --short |
| 62 | +git rev-parse HEAD |
| 63 | +``` |
| 64 | + |
| 65 | +The worktree must be clean. From `packages/tugboat` run: |
| 66 | + |
| 67 | +```sh |
| 68 | +flutter analyze |
| 69 | +flutter test --concurrency=1 --reporter compact |
| 70 | +``` |
| 71 | + |
| 72 | +Then verify the release identity: |
| 73 | + |
| 74 | +```sh |
| 75 | +sed -n '1,8p' pubspec.yaml |
| 76 | +sed -n '1,8p' lib/src/sdk_version.dart |
| 77 | +``` |
| 78 | + |
| 79 | +Record the main commit as `SDK_GIT_SHA`. This repository currently distributes |
| 80 | +the package as a Git dependency; there is no separate pub.dev release whose |
| 81 | +contents can substitute for that commit. |
| 82 | + |
| 83 | +## 2. Pin and deploy the Blend canary |
| 84 | + |
| 85 | +Create a Blend canary branch from its current release base. In Blend's |
| 86 | +`pubspec.yaml`, pin the Tugboat dependency to `SDK_GIT_SHA`, not a moving |
| 87 | +branch: |
| 88 | + |
| 89 | +```yaml |
| 90 | +tugboat: |
| 91 | + git: |
| 92 | + url: https://github.com/blendto/tugboat-flutter |
| 93 | + path: packages/tugboat |
| 94 | + ref: <SDK_GIT_SHA> |
| 95 | +``` |
| 96 | +
|
| 97 | +Resolve dependencies using Blend's checked-in Flutter toolchain: |
| 98 | +
|
| 99 | +```sh |
| 100 | +flutter pub get |
| 101 | +``` |
| 102 | + |
| 103 | +Verify `pubspec.lock` contains both the expected package version and |
| 104 | +`resolved-ref: <SDK_GIT_SHA>`. Commit the manifest and lockfile together. |
| 105 | + |
| 106 | +Build and deploy through Blend's normal internal canary channel. Record: |
| 107 | + |
| 108 | +- Blend Git commit; |
| 109 | +- version and build number; |
| 110 | +- deployment environment/channel; |
| 111 | +- platform artifact identifier; |
| 112 | +- installation time on the test device. |
| 113 | + |
| 114 | +Launch the installed artifact, not a locally patched example application. |
| 115 | +Confirm that its production collector configuration is active. Do not use the |
| 116 | +local exploration collector for this acceptance cohort. |
| 117 | + |
| 118 | +## 3. Capture the manual session matrix |
| 119 | + |
| 120 | +Use a dedicated test account where practical. Keep each session focused enough |
| 121 | +that event order is easy to inspect, but include multiple related actions when |
| 122 | +the race itself requires them. |
| 123 | + |
| 124 | +| Flow | Required actions | Expected evidence | |
| 125 | +| --- | --- | --- | |
| 126 | +| Same-route tap | Perform one state-changing tap and one true no-op | Each tap points at its visible control; the no-op is not confused with delayed navigation | |
| 127 | +| Slow push | Open a route with noticeable rendering or loading delay | Route event and settled tap use the rendered destination frame | |
| 128 | +| Route replacement | Exercise a replacement-style transition | No frame or state from the replaced route is attached to the destination | |
| 129 | +| Rapid navigation | Trigger two consecutive route changes | Only the visible successor owns destination evidence | |
| 130 | +| Modal and sheet | Open and close a dialog and bottom sheet | Overlay events and frames describe the overlay that is actually visible | |
| 131 | +| Immediate destination tap | Tap a control as soon as the destination appears | Destination action does not reuse the origin frame | |
| 132 | +| Scroll then navigate | Begin/end a scroll and immediately navigate | Scroll evidence stays on its origin epoch; route capture shows the destination | |
| 133 | +| External picker | Open a system picker, background/foreground, and return | Lifecycle cancellation/resumption is explicit and the resumed frame is current | |
| 134 | +| Degraded capture | Reproduce budget pressure or a classified capture failure when feasible | Missing visual evidence has a bounded diagnostic outcome and never borrows a stale frame | |
| 135 | + |
| 136 | +Record session IDs and UTC time ranges immediately after each flow. If several |
| 137 | +flows share a session, list the event IDs or timestamps that delimit each flow. |
| 138 | + |
| 139 | +## 4. Inspect the production website |
| 140 | + |
| 141 | +Wait until the collector session has finalized and the replay is available in |
| 142 | +the production website. Filter to the recorded Blend build and SDK version |
| 143 | +`0.4.9` (or the version under test), then open every recorded session. |
| 144 | + |
| 145 | +For each interaction, inspect the actual replay UI and verify: |
| 146 | + |
| 147 | +- the tap marker lands on the control visible in its before-frame; |
| 148 | +- target anchor, route, state signature, and frame describe the same screen; |
| 149 | +- a navigation-producing tap does not emit an early unrelated |
| 150 | + `noVisibleChange`; |
| 151 | +- an action on a destination screen does not reuse an origin-screen frame; |
| 152 | +- a changed outcome has either a fresh visual frame or an explicit |
| 153 | + semantic-only/degraded classification; |
| 154 | +- a route event shows the rendered destination rather than a splash, outgoing |
| 155 | + route, or partially advanced UI; |
| 156 | +- reused visual evidence has an explainable capture diagnostic; |
| 157 | +- every referenced frame opens successfully; |
| 158 | +- taps, route changes, scroll boundaries, and lifecycle events appear in |
| 159 | + chronological order; |
| 160 | +- the replay identifies the recorded app build and session. |
| 161 | + |
| 162 | +ClickHouse, logs, or raw event payloads may help diagnose a failure, but they |
| 163 | +cannot replace this visual inspection. A frame row returning from storage does |
| 164 | +not prove that the replay attached it to the correct screen or action. |
| 165 | + |
| 166 | +## 5. Record the verdict |
| 167 | + |
| 168 | +Use one row per production session: |
| 169 | + |
| 170 | +| Session ID | UTC range | Blend build | SDK version / SHA | Flows | Frame availability | Route/action coherence | Verdict | Follow-up | |
| 171 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | |
| 172 | +| `<collector-session-id>` | `<start> - <end>` | `<version+build>` | `0.4.9 / <sha>` | `<flows>` | pass/fail | pass/fail | accept/reject | `<issue or none>` | |
| 173 | + |
| 174 | +The cohort passes only when: |
| 175 | + |
| 176 | +- every required flow was inspected; |
| 177 | +- no referenced frame is unavailable; |
| 178 | +- no unexplained cross-route or cross-step settle is present; |
| 179 | +- no tap marker points at UI absent from its before-frame; |
| 180 | +- no destination action uses origin-screen pixels or signatures; |
| 181 | +- every degraded capture is explicit and bounded; |
| 182 | +- the user manually confirms the sampled replays look correct. |
| 183 | + |
| 184 | +Any failure keeps the gate open. Save the session ID and timestamps, open one |
| 185 | +narrow issue for the observed defect, and link it from the verdict. Do not |
| 186 | +approve broad rollout based on the remaining successful sessions. |
| 187 | + |
| 188 | +## Rollback |
| 189 | + |
| 190 | +If the canary fails: |
| 191 | + |
| 192 | +1. stop broad rollout; |
| 193 | +2. retain the rejected session IDs and deployed artifact; |
| 194 | +3. restore Blend's Tugboat ref and lockfile to the last accepted |
| 195 | + `resolved-ref`; |
| 196 | +4. redeploy through the same channel; |
| 197 | +5. open a granular SDK issue with the failed flow, event IDs, frame IDs, |
| 198 | + version/build identity, and production replay link. |
| 199 | + |
| 200 | +Do not repair acceptance replays with dashboard post-processing. The SDK must |
| 201 | +emit causally aligned evidence at capture time. |
0 commit comments