|
| 1 | +--- |
| 2 | +name: backtest-change |
| 3 | +description: > |
| 4 | + Validate a data-driven change against LIVE historical data before it ships — |
| 5 | + replay old-vs-new over a real window, report whether it achieves its goal, and |
| 6 | + refuse to ship when the data disproves the premise. Fires whenever someone |
| 7 | + proposes a measurable change and names a number: "add a monitor at 700MB", |
| 8 | + "set the threshold to N", "warn at X / critical at Y", "alert when it exceeds |
| 9 | + N", "raise the timeout to 5s", "change the sampling rate", "bump the cache |
| 10 | + TTL", "tighten this alert", "loosen the threshold", "this should reduce the |
| 11 | + noise", "that will fix the p95" — and before opening any PR for a monitor |
| 12 | + threshold, alert routing or renotify cadence, metric/log/trace query, sampling |
| 13 | + rate, rate limit, autoscaling parameter, or a perf change with a latency or |
| 14 | + throughput target. A proposed number is a hypothesis, not a decision: backtest |
| 15 | + it and let the data override it. Always report old N vs new M with the window |
| 16 | + and data source. The /backtest-change command loads this same skill. |
| 17 | +allowed-tools: Bash, Read, Grep, Glob, AskUserQuestion |
| 18 | +model: opus |
| 19 | +--- |
| 20 | + |
| 21 | +# Backtest a change before you ship it |
| 22 | + |
| 23 | +Validate a **data-driven change against real historical data before opening the |
| 24 | +PR** — and be willing to abandon or redirect the approach when the data says it |
| 25 | +won't work. This is the gate that stops a plausible-but-ineffective change from |
| 26 | +shipping. |
| 27 | + |
| 28 | +Use it for any change whose success is measurable: monitor thresholds, alert |
| 29 | +routing / re-notify cadence, metric/log/trace queries, sampling rates, cache |
| 30 | +TTLs, rate limits, autoscaling params, or a perf optimization with a |
| 31 | +latency/throughput target. |
| 32 | + |
| 33 | +## When this fires without being asked |
| 34 | + |
| 35 | +The most valuable case is the one nobody invokes deliberately: **someone hands |
| 36 | +you a number.** "Add a monitor at 700 MB warn / 1.2 GB critical." "Set the |
| 37 | +timeout to 5s." A named threshold arrives with an implicit claim attached — that |
| 38 | +it separates the bad cases from the good ones. That claim is testable, and it is |
| 39 | +often wrong, because the person proposing it has seen the incident population and |
| 40 | +not the healthy population. |
| 41 | + |
| 42 | +Treat a user-supplied number as a **hypothesis to test**, never as a |
| 43 | +specification to implement. Then say plainly what the data did to it. |
| 44 | + |
| 45 | +Worked example. A request arrived for a per-host memory monitor at "700 MB warn / |
| 46 | +1.2 GB critical", motivated by two hosts that had OOM-wedged at a 2 GiB limit. A |
| 47 | +30-day replay found a third host that had held **1.34–1.37 GB flat for ~21 |
| 48 | +consecutive hours with no incident** — so the proposed critical would have paged |
| 49 | +continuously for most of a day. Shipped 1.6 GB instead. The warning was kept |
| 50 | +exactly as proposed, because a *non-paging* tier is allowed to sit inside normal |
| 51 | +range when its job is lead time. Both halves of that outcome came from the |
| 52 | +backtest, not from the proposal. |
| 53 | + |
| 54 | +## Inputs |
| 55 | + |
| 56 | +You need: the change intended (and the file(s) if known), and the **goal** it |
| 57 | +should achieve — which metric/signal should move, in which direction, by how |
| 58 | +much. If the goal isn't stated, ask. A backtest is meaningless without a target. |
| 59 | + |
| 60 | +When invoked as `/backtest-change`, parse `$ARGUMENTS` for the same two things. |
| 61 | + |
| 62 | +## The discipline (why this exists) |
| 63 | + |
| 64 | +A change that *looks* right is not the same as a change the data supports. The |
| 65 | +common failure is shipping a fix whose premise is wrong — the real driver was |
| 66 | +something else, so the metric never moves. Catch that **before** the PR, not in a |
| 67 | +post-merge validation. |
| 68 | + |
| 69 | +## Workflow |
| 70 | + |
| 71 | +1. **State the hypothesis precisely.** "Changing X will move metric M from ~A to |
| 72 | + ~B because C." Write it down. If you can't name the metric and the expected |
| 73 | + direction, stop and clarify. |
| 74 | + |
| 75 | +2. **Find the authoritative data source** and respect sampling: |
| 76 | + - **Metrics** (standard Datadog metrics, `trace.*`, CloudWatch) are ~100% — |
| 77 | + use these to count rates/volumes/percentiles. |
| 78 | + - Spans and logs are often heavily sampled on the success path; don't count |
| 79 | + volume from them. |
| 80 | + - For alert/page/incident questions, pull the alert system's own event history |
| 81 | + (e.g. incident.io alerts), not a proxy. |
| 82 | + - Beware aggregation defaults that hide the shape you're testing — e.g. a |
| 83 | + scalar query that silently averages a `max:` series returns avg-of-max and |
| 84 | + will understate peaks. Set the aggregator explicitly. |
| 85 | + |
| 86 | +3. **Pull a representative window** (typically 7–30 days; long enough to include |
| 87 | + the conditions the change targets). |
| 88 | + |
| 89 | +4. **Replay old logic vs new logic over that same window.** Compute concrete |
| 90 | + deltas: old **N** vs new **M** — alerts fired, pages, error rate, p95, cost, |
| 91 | + rows, whatever the goal metric is. For threshold/monitor changes, evaluate |
| 92 | + both the old and the new condition against the historical series and count |
| 93 | + transitions. Identify *which groups/series* change, not just the aggregate. |
| 94 | + For a brand-new monitor, "old" is 0 — say so explicitly rather than omitting it. |
| 95 | + |
| 96 | +5. **Separate the two populations.** The threshold's whole job is to divide |
| 97 | + incident from healthy. Report the highest *legitimate* value observed and the |
| 98 | + lowest *incident* value. If they overlap, the threshold cannot work at any |
| 99 | + setting and the signal itself needs to change — say that instead of picking a |
| 100 | + number in the overlap. |
| 101 | + |
| 102 | +6. **Classify the result:** |
| 103 | + - **EFFECTIVE** — data shows the change achieves the goal. Capture the |
| 104 | + old-vs-new numbers for the PR body. |
| 105 | + - **PARTIAL** — moves the metric but not enough / not for the cases that |
| 106 | + matter. Note the gap. |
| 107 | + - **INEFFECTIVE / PREMISE DISPROVED** — the data shows the real driver is |
| 108 | + elsewhere, or the change barely moves M. **Stop. Do not open the PR.** |
| 109 | + Report what the data actually shows and propose the lever that *would* work. |
| 110 | + - **REVISED** — the goal is sound but the proposed number isn't. Ship the |
| 111 | + corrected value and state prominently what you changed and why. |
| 112 | + |
| 113 | +7. **Only if it holds up**, proceed to the change + PR, and put the backtest in |
| 114 | + the PR body: the hypothesis, the window, old-vs-new numbers, and a link to the |
| 115 | + live dashboard/query (prefer a link over stale typed numbers). |
| 116 | + |
| 117 | +## Output |
| 118 | + |
| 119 | +A short backtest report: |
| 120 | + |
| 121 | +- **Hypothesis** and goal metric. |
| 122 | +- **Window + data source** (and any sampling caveat applied). |
| 123 | +- **Old vs new** with hard numbers and which groups changed. |
| 124 | +- **Population separation** — highest healthy value vs lowest incident value. |
| 125 | +- **Verdict** (EFFECTIVE / PARTIAL / INEFFECTIVE / REVISED) + recommendation. If |
| 126 | + INEFFECTIVE, the alternative lever. |
| 127 | + |
| 128 | +## Principles |
| 129 | + |
| 130 | +- Backtest **before** acting; never claim a change works without replaying data. |
| 131 | +- A user-supplied number is a hypothesis. Testing it is the job, not overriding |
| 132 | + the request — but when the data rejects it, say so and ship the corrected value. |
| 133 | +- Be willing to **reverse** — a disproved premise is a successful backtest, not a |
| 134 | + failure. |
| 135 | +- Prefer **dashboard/query links** over typed numbers that go stale. |
| 136 | +- Distinguish tiers by consequence: a chatty non-paging warning can be |
| 137 | + acceptable; a chatty page destroys trust in the monitor. |
| 138 | +- When the change spans owners (e.g. an external-config change + a repo change), |
| 139 | + say which half the data supports and which is out of scope. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4&size=40){kind=avatar}
0 commit comments