Skip to content

Commit 3c74a8f

Browse files
committed
Add UI Startup lazy-loading worked example to performance-attribution
Second worked example (13.31.0 -> 13.35.x, UI Startup) demonstrating the data-fidelity pitfalls the skill warns about, drawn from a real production Sentry dashboard: - High-count page-startup wins (/home, /popup, /notification, /sidepanel p90 -54% to -69%) are verdict-grade; the UI Startup tail 'regression' (p90 +7%, p95 +16%) rests on 30-80x thinner 13.35.x samples -> watch item. - Two traps: anchoring 'best' on a thin superseded patch (13.34.0 @3.3M vs 13.34.1 @35M), and an unweighted baseline mixing thin + adopted releases manufacturing a phantom regression. - Span breakdown: Load Scripts dominates startup tail. Generalize both lessons into skill.md (separate confirmed wins from watch items; build baselines by adoption/weight, not release count).
1 parent 4e97c4b commit 3c74a8f

2 files changed

Lines changed: 20 additions & 0 deletions

File tree

domains/analytics/skills/performance-attribution/repos/metamask-extension.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,3 +94,21 @@ Example findings:
9494
| /notification.html | 1.36s → 1.05s (-23%) | 4.30s → 4.71s (+9%, **high variance — inconclusive**) |
9595

9696
Most UI Startup and /home.html gains landed in 13.12 (p95 UI Startup -40% in one release); Asset Details improved across 13.14 → 13.15. Treat the per-release header deltas as measured totals and attribute individual code changes as likely contributors only.
97+
98+
## Worked Example 2: UI Startup Lazy Loading (13.31.0 → 13.35.x) — Fidelity Cautionary Tale
99+
100+
Same 30d production spans dataset, two findings of **opposite reliability** — the textbook reason to run query hygiene before attribution:
101+
102+
- **Solid (large, consistent deltas):** page-startup transactions improved sharply vs 13.31.0 — `/home.html` p90 3,225ms → 1,014ms (-69%), `/popup.html` 2,614ms → 1,002ms (-62%), `/notification.html` -54%, `/sidepanel.html` -67%. Lazy loading is a real, felt win.
103+
- **Fragile (sample-thin):** the `UI Startup` transaction tail reads as a *regression* (p90 +7%, p95 +16% vs a "recent baseline"). But 13.35.0 / 13.35.1 carry only **662K / 1.3M** spans vs **35–52M** for 13.33.0 / 13.34.1 — 30–80× thinner. This is a **watch item, not a verdict** (see `sentry-mcp-queries` → Percentile Fidelity).
104+
105+
Two methodology traps this dashboard demonstrates:
106+
107+
| Trap | What happened | Fix |
108+
|---|---|---|
109+
| Anchoring "best" on a thin, superseded patch | 13.34.0 (3.3M count) was flagged "best recent p90/p95" — but it was superseded by 13.34.1 (35M) | Anchor on the last *widely-adopted* patch (13.34.1 / 13.33.0), never the sample-thin `.0` |
110+
| Unweighted baseline mixing thin + adopted releases | the "recent baseline" averaged 13.32.0 (4.8M) and 13.34.0 (3.3M) **equally** with 13.33.0 (52.6M) and 13.34.1 (35M) | Weight by count, or compare only to widely-adopted patches — an unweighted mean lets thin low-latency patches drag the baseline down and **manufacture a phantom regression** in the next release |
111+
112+
Span breakdown (13.35.x, p90): `Load Scripts` 3,775ms dominates `UI Startup` 6,159ms; `Background Connect` 985ms; `First Render` 116ms and `Setup Store` 49ms are negligible → keep the next investigation on script-load and background-connect cost, not render/store setup.
113+
114+
Before promoting the `UI Startup` readout to a verdict: re-run once 13.35.x reaches count parity with 13.34.1, and prefer the 90d window over this dashboard's 30d per the primary-source rule.

domains/analytics/skills/performance-attribution/skill.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,8 @@ App-repo diffs miss work shipped as `@metamask/*` version bumps — it surfaces
7777
- "High confidence" = mechanism + timing + population align
7878
- Always keep an **Unattributed** section for movement no change explains
7979
- Flag high-variance metrics (a noisy confirmation-popup p95) as inconclusive, not as wins
80+
- **Separate confirmed findings from watch items.** A delta backed by large counts on both ends is a verdict; a tail movement on a sample-thin or not-yet-adopted release is a watch item — never present them at equal confidence in the same summary.
81+
- **Build the baseline by adoption, not by count of releases.** When averaging a "recent baseline", weight by sample count or include only widely-adopted patches. An unweighted mean that folds in thin, superseded `.0` patches can drag the baseline down and *manufacture a phantom regression* in the next release.
8082

8183
## Limitations
8284

0 commit comments

Comments
 (0)