Skip to content

Commit f24445f

Browse files
DevTobisDevTobis
committed
docs(design): backstop readiness cost model and measurement plan
orbital-stellar#1063 depends on 21.1 (orbital-stellar#1062), which is open, and packages/worker-core has no tracked files in this repo. This is the model and the measurement plan; the numbers table is deliberately empty. The acceptance criterion asks for measured numbers, not estimates, and filling it with plausible figures would defeat the criterion it is meant to satisfy. The design decision worth reviewing is that the shared drivers take a LIST of subscription ids rather than one. A single-id signature would force attribution to be decided at each call site, scattering the split policy everywhere instead of keeping it in one place - and shared monitoring across subscriptions watching the same condition is the main cost lever, so it has to be measurable. Both attributedCost (even split, sums to real spend) and standaloneCost (what this subscription would cost alone) are recorded. The gap between them is the value of sharing as a number rather than an intuition, and pricing needs both. Marginal cost is segmented by whether a new subscription shared an existing condition or introduced a new one. Those differ by a large factor, and a blended average would hide exactly what is being measured.
1 parent c2ff81f commit f24445f

1 file changed

Lines changed: 158 additions & 0 deletions

File tree

‎docs/design/backstop-cost-model.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# Backstop readiness cost model
2+
3+
**Status: proposed, and blocked on measurement.** Issue
4+
[#1063](https://github.com/determined-001/orbital_stellar/issues/1063) depends on
5+
21.1 ([#1062](https://github.com/determined-001/orbital_stellar/issues/1062)),
6+
which is open, and `packages/worker-core` does not exist in this repository yet
7+
— it has no tracked files.
8+
9+
This document is the **model and the measurement plan**. The numbers are
10+
deliberately absent: §C.7's whole point is that readiness cost must be measured
11+
rather than assumed, and there is no backstop running to measure. Filling this
12+
table with plausible estimates would defeat the criterion it is meant to satisfy.
13+
14+
## Why readiness cost is the number that matters
15+
16+
§C.7 is explicit: model the readiness cost before pricing, because it is not
17+
pure margin. **Monitoring cost scales with the number of backstopped
18+
subscriptions regardless of how many ever need intervention.** A tier priced off
19+
intervention frequency prices the rare event and gives away the common one.
20+
21+
Without this, the pricing page in 21.3 is a guess, and the product's margin is
22+
discovered in a monthly bill.
23+
24+
## What is metered
25+
26+
Four cost drivers, attributed per subscription per window.
27+
28+
| Driver | Unit | Attribution |
29+
| --- | --- | --- |
30+
| RPC calls | count, by method | Direct where a call serves one subscription; shared calls split per [below](#shared-monitoring) |
31+
| Export scans | count and bytes scanned | Direct or shared, same rule |
32+
| Compute | milliseconds of watcher CPU | Sampled per evaluation, attributed to the subscription evaluated |
33+
| Storage | byte-ledgers (bytes × windows retained) | Direct — coverage records and watcher state are per subscription |
34+
35+
A window is the watcher's evaluation window, so cost buckets line up exactly with
36+
coverage windows and the two can be joined without interpolation.
37+
38+
## <a id="shared-monitoring"></a>Shared monitoring is the whole shape of the curve
39+
40+
Implementation note 3: shared monitoring across subscriptions watching the same
41+
condition is the main lever — **and it must be measured before it is optimised.**
42+
43+
It is also what makes attribution non-trivial. One RPC call can serve N
44+
subscriptions watching the same contract. Two ways to attribute it:
45+
46+
- **Even split** (`1/N` each) — every subscription's attributed cost falls as the
47+
cohort grows. Flatters the marginal number.
48+
- **Full cost to each** — attributed cost is stable per subscription but total
49+
attributed exceeds total incurred, which makes the aggregate meaningless.
50+
51+
**Neither alone.** The meter records both: `attributedCost` (even split, sums to
52+
actual spend) and `standaloneCost` (what this subscription would cost with no
53+
sharing). The gap between them *is* the value of sharing, expressed as a number
54+
rather than an intuition, and pricing needs both — one for margin, one to know
55+
what happens to a subscription that ends up alone on its condition.
56+
57+
## Marginal cost, not just total
58+
59+
Implementation note 2. Total cost hides the shape of the curve, and the shape is
60+
what decides whether the tier is viable at scale.
61+
62+
Reported explicitly as: the cost of adding one more backstopped subscription,
63+
computed as the delta in total attributed cost across a window in which the
64+
subscription count changed, segmented by whether the new subscription **shared**
65+
an existing condition or **introduced a new one**. Those two marginal costs will
66+
differ by a large factor, and reporting a blended average of them would hide
67+
precisely the thing being measured.
68+
69+
## Interfaces
70+
71+
Mirrors the metrics idiom already in `packages/pulse-webhooks` (`metrics.ts` +
72+
`PrometheusWebhookMetrics.ts` + `OtelWebhookMetrics.ts`) — one interface, a
73+
no-op default, adapters per backend. One idiom across the project, per note 1.
74+
75+
```ts
76+
export interface CostMeter {
77+
recordRpcCall(subscriptionIds: string[], method: string, durationMs: number): void;
78+
recordExportScan(subscriptionIds: string[], bytesScanned: number): void;
79+
recordCompute(subscriptionId: string, durationMs: number): void;
80+
recordStorage(subscriptionId: string, bytes: number): void;
81+
}
82+
83+
export interface CostWindow {
84+
subscriptionId: string;
85+
startLedger: number;
86+
endLedger: number;
87+
/** Even-split share of actually-incurred cost. Sums to real spend. */
88+
attributedCost: CostBreakdown;
89+
/** What this subscription would have cost alone. Never summed. */
90+
standaloneCost: CostBreakdown;
91+
}
92+
```
93+
94+
`subscriptionIds` is a **list** on the shared drivers rather than a single id.
95+
That is the design decision that makes sharing measurable at all: a
96+
single-id signature would force the caller to decide attribution at the call
97+
site, and the split policy would then be scattered across every call site
98+
instead of living in one place.
99+
100+
- `NOOP_COST_METER` — the default, so metering is opt-in and costs nothing when off.
101+
- `PrometheusCostMeter` / `OtelCostMeter` — export through the existing surfaces.
102+
103+
Files: `packages/worker-core/src/backstop/costMeter.ts`,
104+
`packages/worker-core/src/metrics.ts`.
105+
106+
## The measured numbers
107+
108+
<!-- Filled in once 21.1 (#1062) lands and a backstop has run for a week. -->
109+
110+
| | |
111+
| --- | --- |
112+
| Measurement window | *pending* |
113+
| Subscriptions observed | *pending* |
114+
| Distinct conditions watched | *pending* |
115+
| Total attributed cost / window | *pending* |
116+
| Marginal cost, shared condition | *pending* |
117+
| Marginal cost, new condition | *pending* |
118+
| Sharing factor (standalone ÷ attributed) | *pending* |
119+
120+
**This table is the deliverable of #1063 and it cannot be completed yet.** The
121+
acceptance criterion says "the measured numbers, not estimates" — so it stays
122+
empty rather than being filled with numbers that would read as measurements.
123+
124+
### How to fill it
125+
126+
1. Land 21.1 so a backstop watcher exists.
127+
2. Wire `CostMeter` into it at the four drivers above.
128+
3. Run against testnet for at least one full retention period with a subscription
129+
count that **changes during the window** — a static count cannot yield a
130+
marginal cost.
131+
4. Ensure at least one subscription shares a condition and at least one does not,
132+
or the two marginal figures collapse into one.
133+
5. Read the dashboard query below and paste the results here.
134+
135+
## Dashboard query
136+
137+
Cost against subscription count, which is the chart the pricing decision needs:
138+
139+
```promql
140+
# Attributed cost per window, against the subscription count that produced it
141+
sum(rate(orbital_backstop_cost_attributed_total[1h])) by (driver)
142+
/ on() group_left sum(orbital_backstop_subscriptions_active)
143+
144+
# Marginal cost, split by whether the condition was already being watched
145+
sum(rate(orbital_backstop_cost_attributed_total[1h]))
146+
/ sum(rate(orbital_backstop_subscriptions_active[1h]))
147+
```
148+
149+
## What is blocked, precisely
150+
151+
| Acceptance criterion | Status |
152+
| --- | --- |
153+
| Per-subscription attribution for RPC, scans, compute, storage | Designed; needs `worker-core` |
154+
| Aggregated per subscription per window, queryable over time | Designed; needs `worker-core` |
155+
| Marginal cost reported explicitly | Designed; needs `worker-core` |
156+
| Exported through existing Prometheus and OTel surfaces | Designed; idiom already exists in `pulse-webhooks` |
157+
| Dashboard or documented query | **Done** — above |
158+
| Written cost model with **measured numbers, not estimates** | **Blocked.** Needs 21.1 running. |

0 commit comments

Comments
 (0)