|
| 1 | +--- |
| 2 | +adr: "0021" |
| 3 | +title: "Power metric standardization (cpu/gpu/total) and per-platform source layering" |
| 4 | +status: accepted |
| 5 | +date: "2026-07-02" |
| 6 | +supersedes: null |
| 7 | +superseded_by: null |
| 8 | +deciders: |
| 9 | + - "Kinn Coelho Juliao" |
| 10 | +--- |
| 11 | + |
| 12 | +# 0021 — Power metric standardization (cpu/gpu/total) and per-platform source layering |
| 13 | + |
| 14 | +## 1. Context |
| 15 | + |
| 16 | +Power reporting had grown platform-specific and inconsistent, and fleet |
| 17 | +observation surfaced concrete wrongness: |
| 18 | + |
| 19 | +- `power.pkg` meant **"CPU package"** on Linux (RAPL) but **"whole system"** on |
| 20 | + macOS (SMC `PSTR`), so the same key was not comparable across hosts. |
| 21 | +- On Linux `power.cpu` was the RAPL *core* subdomain while `power.pkg` was the |
| 22 | + package — two CPU-ish numbers that didn't reconcile. |
| 23 | +- **AMD APU (Strix Halo)** under-reported: only the amdgpu rail showed, so a box |
| 24 | + that btop measured at ~139 W read ~52 W. |
| 25 | +- A busy discrete GPU (RTX PRO 6000 at ~500 W) was hidden behind a ~30 W CPU |
| 26 | + package figure, because the panel headlined `power.pkg`. |
| 27 | +- **Apple Pro/Max** appeared to have no CPU power at all — IOReport and |
| 28 | + `powermetrics` both report `CPU Power: 0 mW` there. |
| 29 | +- **Windows** had no CPU power path. |
| 30 | + |
| 31 | +The metric-adapter contract (ADR-0003) and the optional privileged helper |
| 32 | +(ADR-0004) still hold; this decision layers on top of them. |
| 33 | + |
| 34 | +## 2. Goals / Non-Goals |
| 35 | + |
| 36 | +**Goals:** |
| 37 | +- One comparable power model on every OS/chip: `power.cpu`, `power.gpu`, |
| 38 | + `power.total`. |
| 39 | +- Read the *best available* source per platform; never fabricate a value. |
| 40 | +- Explain every missing reading (Unavailable-with-reason), never a silent blank. |
| 41 | +- Backward compatible across a mixed-version fleet. |
| 42 | + |
| 43 | +**Non-Goals:** |
| 44 | +- Shipping our own kernel driver (Windows RAPL) — we read a user-installed one. |
| 45 | +- A full per-chip SMC key map for every Apple SoC (we map what we can verify). |
| 46 | +- Per-core wattage (no vendor exposes it; see §5). |
| 47 | + |
| 48 | +## 3. Proposal |
| 49 | + |
| 50 | +**Standard trio, everywhere:** |
| 51 | +- `power.cpu` — CPU power (the whole CPU complex). |
| 52 | +- `power.gpu` — GPU power. |
| 53 | +- `power.total` — whole machine. Retire `power.pkg`. |
| 54 | + |
| 55 | +**Per-platform source, layered (first that yields a value wins):** |
| 56 | + |
| 57 | +| Platform | `power.cpu` | `power.total` | |
| 58 | +|---|---|---| |
| 59 | +| Linux (Intel/AMD) | RAPL package (root, via helper) | `cpu + gpu (+ npu)` | |
| 60 | +| Linux (ARM/GB10) | Unavailable — no RAPL | `gpu` | |
| 61 | +| Windows | Scaphandre scrape (its Hubblo RAPL driver) | `cpu + gpu` | |
| 62 | +| macOS **base** (M1–M4) | IOReport per-domain CPU | SMC `PSTR` | |
| 63 | +| macOS **Pro/Max/Ultra** | raw SMC cluster keys (`PC02+PC03+PC42+PC43` on M3 Max) | SMC `PSTR` | |
| 64 | + |
| 65 | +`power.total` is a source-aware sum: on Apple it is the SMC whole-system rail |
| 66 | +directly (never `cpu+gpu`, which would double-count); everywhere else the GPU is |
| 67 | +a physically separate rail, so it is `cpu + gpu (+ npu)`. |
| 68 | + |
| 69 | +**`power.cpu` source selection (first that yields a value wins):** |
| 70 | + |
| 71 | +```mermaid |
| 72 | +flowchart TD |
| 73 | + start([collect power.cpu]) --> os{OS?} |
| 74 | +
|
| 75 | + os -->|Linux| rapl{RAPL readable?<br/>root / helper} |
| 76 | + rapl -->|yes| lcpu[power.cpu = RAPL package] |
| 77 | + rapl -->|no ARM/GB10| lnone[Unavailable: no RAPL] |
| 78 | +
|
| 79 | + os -->|Windows| scaph{Scaphandre<br/>endpoint up?} |
| 80 | + scaph -->|yes| wcpu[power.cpu = Scaphandre scrape] |
| 81 | + scaph -->|no| wnone[Unavailable: run Scaphandre] |
| 82 | +
|
| 83 | + os -->|macOS| io{IOReport CPU<br/>reads nonzero?<br/>base dies} |
| 84 | + io -->|yes| mcpu[power.cpu = IOReport CPU] |
| 85 | + io -->|no, Pro/Max/Ultra| smc{SMC cluster keys<br/>present and in bound?} |
| 86 | + smc -->|yes| scpu["power.cpu = sum of SMC cluster keys<br/>'CPU complex SMC'"] |
| 87 | + smc -->|no / unmapped die| snone[Unavailable: no per-domain CPU power] |
| 88 | +``` |
| 89 | + |
| 90 | +The macOS path is the novel one: base dies report CPU power through IOReport, but |
| 91 | +Pro/Max/Ultra report `0` there, so Heimdall falls to the raw SMC cluster keys with |
| 92 | +a sanity bound (below) before giving up. |
| 93 | + |
| 94 | +**Apple SMC per-domain CPU** (the key novelty): the IOReport energy model reports |
| 95 | +0 for CPU on Pro/Max, but the raw AppleSMC per-cluster power keys still carry it |
| 96 | +(the same source Stats reads). `smcReadFloat(key)` reads any AppleSMC `flt ` key; |
| 97 | +`smcCPUPower()` sums the CPU cluster keys, **gated by a sanity bound** (reject |
| 98 | +NaN/Inf/negative/`>200 W`) so a chip whose keys mean something else falls back to |
| 99 | +Unavailable rather than showing garbage. |
| 100 | + |
| 101 | +**Helper is in-process-first:** the daemon reads privileged metrics itself first; |
| 102 | +it only consults the helper when it *can't* read a CPU power rail (an unprivileged |
| 103 | +Linux daemon needing RAPL). On macOS the daemon has power via SMC/IOReport, so it |
| 104 | +never calls the (slow, `powermetrics`-bound) helper — running the helper on Apple |
| 105 | +Silicon is harmless. |
| 106 | + |
| 107 | +## 4. Alternatives Considered |
| 108 | + |
| 109 | +| Option | Pros | Cons | Why Rejected | |
| 110 | +|---|---|---|---| |
| 111 | +| Keep platform-specific names (`power.pkg` etc.) | no churn | not comparable across fleet; APU under-count persists | the whole point is comparability | |
| 112 | +| Ship our own Windows RAPL driver | works standalone | kernel-signing, liability, maintenance | read Scaphandre's signed driver instead | |
| 113 | +| Full per-chip Apple SMC key map (Stats-style) | accurate on every chip | large, ongoing reverse-engineering | verify what we have; safe fallback elsewhere | |
| 114 | +| Helper-first (prior behaviour) | one source of truth | a slow helper (macOS `powermetrics`) blanks *all* privileged metrics | in-process-first + bounded helper | |
| 115 | +| Naive `power.total = pkg + gpu` on all | simple | double-counts Apple SMC total and AMD-APU integrated GPU | source-aware sum | |
| 116 | + |
| 117 | +## 5. Trade-offs and Risks |
| 118 | + |
| 119 | +- **Reverse-engineered SMC keys are per-generation.** Verified on M3 Max; other |
| 120 | + Pro/Max dies may use different keys. Mitigated: sanity bound + IOReport-first + |
| 121 | + Unavailable fallback — never a wrong number, worst case an honest dash. |
| 122 | +- **`power.cpu` on Apple Pro/Max is the CPU-cluster sum**, labelled |
| 123 | + `CPU complex (SMC)` — it is the P/E cluster rails, close to but not a |
| 124 | + vendor-blessed "package" figure. |
| 125 | +- **In-process-first costs a redundant `nvidia-smi` on Linux** (in-process, then |
| 126 | + the helper) — a few hundred ms per cycle at the collector cadence. Acceptable. |
| 127 | +- **No per-core wattage anywhere** — Apple SMC is per-cluster, Intel/AMD RAPL is |
| 128 | + per-package/domain. Per-core *utilisation* exists; per-core power does not. |
| 129 | +- **Mixed-version fleet** during rollout: an un-updated host emits the old key and |
| 130 | + reads the old way until upgraded. Transient, non-breaking. |
| 131 | + |
| 132 | +## 6. Impact |
| 133 | + |
| 134 | +**FinOps:** none — all reads are local, unprivileged where possible; no new |
| 135 | +services except the operator-chosen Scaphandre on Windows. |
| 136 | + |
| 137 | +**SRE:** strictly safer. The helper can no longer blank a host under latency; every |
| 138 | +gap explains itself; a mis-mapped chip degrades to Unavailable, not garbage. |
| 139 | +Recovery for a wrong reading is "it can't happen" (bounded + fallback). |
| 140 | + |
| 141 | +**Security:** all Apple/Linux reads are unprivileged (AppleSMC, IOReport, sysfs |
| 142 | +RAPL via the root helper). Windows depends on the operator installing Scaphandre's |
| 143 | +signed driver — a documented, user-controlled dependency, not shipped by us. |
| 144 | + |
| 145 | +**Team:** new-chip support = probe the SMC keys on that machine and add them |
| 146 | +(§Next Steps). The SMC key catalogue is reverse-engineered and needs occasional |
| 147 | +maintenance, exactly like Stats' per-chip temperature keys. |
| 148 | + |
| 149 | +## 7. Decision |
| 150 | + |
| 151 | +Standardize power on `power.cpu` / `power.gpu` / `power.total` across every |
| 152 | +platform, sourced by a layered, best-available strategy and a source-aware total |
| 153 | +that never double-counts. Read Apple Pro/Max CPU power from the raw SMC cluster |
| 154 | +keys (with a sanity bound), read Windows CPU power from a user-run Scaphandre, and |
| 155 | +make the helper in-process-first so it is additive, never subtractive. Retire |
| 156 | +`power.pkg`; explain every unavailable rail. Backward compatibility is a hard |
| 157 | +constraint met by keeping the metric schema stable. Shipped across v2.3.0–v2.5.2 |
| 158 | +and verified live on Intel/AMD Linux, Windows, and Apple base (M4) + Pro/Max (M3 |
| 159 | +Max). |
| 160 | + |
| 161 | +Status: **accepted** |
| 162 | + |
| 163 | +## 8. Next Steps |
| 164 | + |
| 165 | +- [ ] Map SMC CPU keys for more Apple dies (M1/M2/M4 Pro/Max, Ultra) as hardware |
| 166 | + is available; unmapped chips already fall back safely. |
| 167 | +- [ ] Top view: per-core % grid labelled by core type (P/E), plus per-cluster |
| 168 | + power where the SMC exposes it. |
| 169 | +- [ ] Multi-GPU per-device breakout (currently aggregated). |
0 commit comments