Skip to content

Commit 6fbc334

Browse files
tariknztariknz
authored
docs: record PR 656 Fuel performance A/B (#658)
1 parent 0097bf2 commit 6fbc334

2 files changed

Lines changed: 103 additions & 12 deletions

File tree

‎docs/PERFORMANCE_TEST_LOG.md‎

Lines changed: 92 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,86 @@ When new scenarios are added (e.g. to test specific findings), document them her
8080
8181
---
8282

83+
### 2026-08-06 · Phase 3 Fuel pilot · PR #656 deterministic A/B · Fuel-only
84+
85+
**Scenario:** Fuel-only dashboard replaying the curated ten-minute AI-race tape
86+
from frame zero. Baseline and candidate used separate worktrees on the same
87+
Windows machine, identical saved dashboard/settings, two-overlay window layout,
88+
and the same 348,187,160-byte tape (`gapCount: 0`). Each build received a
89+
separate 60-second warm-up followed by a 420-second capture; analysis discarded
90+
the first 60 seconds of each measured run.
91+
92+
**Baseline:** `0d74f2766fd27a9db3f86c5ca27cf2626b3b5e7a` (after PR #652)
93+
94+
**Candidate:** `e3dd77aad84f0a5ec8393ae29478459bbce8aff1` (after PR #656)
95+
96+
**Raw artifacts (local-only; not committed):**
97+
98+
- Baseline worktree: `perf-results/2026-08-06T10-29-50-363Z-pr656-baseline-fuel-r1.log`
99+
- Candidate worktree: `perf-results/2026-08-06T10-38-42-515Z-pr656-candidate-fuel-r1.log`
100+
- Candidate A/B summary: `perf-results/2026-08-06T10-38-42-515Z-pr656-candidate-fuel-r1.comparison.summary.md`
101+
102+
**Top-line numbers (60-second analysis warm-up, 5.84-minute steady-state
103+
window, 71 main samples):**
104+
105+
| Metric | Baseline | Candidate | Change |
106+
| ------------------------------------------- | ----------------: | ----------------: | ----------------------: |
107+
| App CPU | 0.88% | 0.76% | −0.12 pp (−13.6%) |
108+
| Main-process CPU | 0.135% | 0.117% | −0.018 pp (−13.3%) |
109+
| Fuel renderer CPU | 0.309% | 0.265% | −0.044 pp (−14.2%) |
110+
| App-wide renderer wake-ups | 42.27/s | 24.37/s | **−42.4%** |
111+
| Legacy telemetry deliveries | 42.27/s | **0.00/s** | **−100%** |
112+
| `fuel.projection` delivery to Fuel renderer | not instrumented¹ | ~4.87 Hz measured | candidate-only evidence |
113+
| `processTelemetry` mean | 0.733 ms | 0.612 ms | −16.5% |
114+
| `processTelemetry` p99 mean | 1.736 ms | 1.362 ms | −21.5% |
115+
| Telemetry average / minimum | 21.14 / 20.94 Hz | 21.11 / 20.20 Hz | stable; both pass |
116+
| Frames over 50 ms | 0.000% | 0.000% | unchanged |
117+
| Working-set slope | 16.35 MB/min | 11.16 MB/min | −31.8%; both fail gate |
118+
| Private-memory slope | 15.73 MB/min | 10.63 MB/min | −32.4%; both fail gate |
119+
120+
¹ PR #652 did not record channel-callback metrics, so its channel delivery rate
121+
cannot be reconstructed directly from the baseline log. Both builds configure
122+
`fuel.projection` at 5 Hz. The candidate recorded 24.37 local callback
123+
invocations per second across five Fuel consumers, equivalent to approximately
124+
4.87 renderer deliveries per second.
125+
126+
**Correctness and lap coverage:** Both captures crossed completed-lap
127+
boundaries. Visual captures showed populated five-lap average, min/max,
128+
fuel-to-add, pit-window, fuel-at-end, time-empty, and confidence calculations in
129+
both builds. No visible behavior difference was found. There were no replay
130+
gaps, dropped-update messages, application failures, or unexpected process
131+
exits. Both runs emitted the same single Chromium Mojo startup warning.
132+
133+
**Gate result:** `processTelemetry` p99, minimum telemetry cadence, and frames
134+
over 50 ms passed for both builds. The candidate mechanically passed the FPS
135+
A/B gates, but the tape's iRacing FPS/GPU variables are historical and are not
136+
host-performance evidence. The steady-state memory-slope gate failed in both
137+
builds.
138+
139+
**Findings:**
140+
141+
1. **PR #656 validates the Phase 3 Fuel-only subscription boundary.** The
142+
candidate Fuel renderer received `fuel.projection` at approximately 4.87 Hz
143+
under its 5 Hz configuration and stopped receiving the legacy telemetry
144+
firehose entirely. The baseline channel rate was not instrumented.
145+
2. **The deterministic wake-up reduction is conclusive.** App-wide renderer
146+
wake-ups fell 42.4%, and the renderer without active Fuel widgets correctly
147+
received neither telemetry nor channel wake-ups.
148+
3. **No functional or frame-pacing regression was observed.** Fuel remained
149+
populated after completed laps and both captures recorded zero steady-state
150+
frames over 50 ms.
151+
4. **CPU and memory movements are directional, not conclusive.** App CPU, Fuel
152+
renderer CPU, `processTelemetry`, peak memory, and memory slopes all moved
153+
favorably, but this is one A/B pair. Repeat before assigning the exact
154+
magnitude to PR #656. The memory-slope gate remains open.
155+
156+
**Conclusion:** PR #656 improved the intended architectural performance path
157+
without a detected regression. It proves the Fuel pilot can opt out of legacy
158+
telemetry; it does not, by itself, establish that the broader memory-slope
159+
problem is resolved.
160+
161+
---
162+
83163
### 2026-05-18 · Combined Phase 2a · PCC race spectated · GR86 at Navarra
84164

85165
**Scenario:** Real multiplayer PCC race at Navarra, 4 classes, 42 drivers. User joined and spectated the first ~15 min of the race rather than driving. 23.5 min log capturing warmup → race start → 15 min of spectating → end.
@@ -931,18 +1011,18 @@ Live record of which architecture review findings have empirical support. Update
9311011

9321012
### Confirmed implicated, fix validated
9331013

934-
| ID | Description | Validation |
935-
| ---------------------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
936-
| **L1** | Synchronous `writeFileSync` on main process | Phase 0.5 — tick dips eliminated |
937-
| **L2** | YAML re-parsed on every `getSessionData` call | Phase 0.5 — multi-class startup cost eliminated |
938-
| **S5** | Analytics forwards every warn/error log line to PostHog | Phase 0.5 — NetworkService returned to 53 MB baseline |
939-
| **P1** | Full 340-key telemetry payload structured-cloned to every overlay 25 Hz | Phase 1 — Primary slope −97%, broadcast latency −54% |
940-
| **P2** | `useDriverPositions` subscribes to raw `CarIdxLapDistPct` | Phase 1 — implicit in P1 fix; Primary leak elimination consistent with replacement |
941-
| **P4** | `SectorTimingStore.tick()` subscribes to raw `LapDistPct`/`SessionTime` | Phase 1 — implicit in P1 fix |
942-
| **H3** | Spurious `reset()` calls at boot on undefined→0 SDK-connect transitions | Phase 2a Tier 1 — no duplicate "Resetting lap time history" log lines in P5 boot |
943-
| **H4** | `PitLapStore.updatePitLaps` clone storm on uneventful ticks | Phase 2a Tier 1 — Left renderer slope dropped 95%; combined PR shows Standings at −0.4 MB/min (declining) |
944-
| **Standings widget** | Per-driver allocation pattern in Standings | Phase 2a Tier 1 (via H4) and H1 rewrite — slope now under target across multi-class scenarios |
945-
| **Main process slope** | Persistent +2–6 MB/min steady-state growth in main process | Phase 2a combined — Main slope dropped to +0.5 MB/min in combined PR test, at target. Likely attributed to Tier 2a session-boundary cleanup. |
1014+
| ID | Description | Validation |
1015+
| ---------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
1016+
| **L1** | Synchronous `writeFileSync` on main process | Phase 0.5 — tick dips eliminated |
1017+
| **L2** | YAML re-parsed on every `getSessionData` call | Phase 0.5 — multi-class startup cost eliminated |
1018+
| **S5** | Analytics forwards every warn/error log line to PostHog | Phase 0.5 — NetworkService returned to 53 MB baseline |
1019+
| **P1** | Full 340-key telemetry payload structured-cloned to every overlay 25 Hz | Phase 1 — Primary slope −97%, broadcast latency −54%. Phase 3 Fuel pilot / PR #656 — Fuel-only legacy deliveries −100% and app-wide renderer wake-ups −42.4%. |
1020+
| **P2** | `useDriverPositions` subscribes to raw `CarIdxLapDistPct` | Phase 1 — implicit in P1 fix; Primary leak elimination consistent with replacement |
1021+
| **P4** | `SectorTimingStore.tick()` subscribes to raw `LapDistPct`/`SessionTime` | Phase 1 — implicit in P1 fix |
1022+
| **H3** | Spurious `reset()` calls at boot on undefined→0 SDK-connect transitions | Phase 2a Tier 1 — no duplicate "Resetting lap time history" log lines in P5 boot |
1023+
| **H4** | `PitLapStore.updatePitLaps` clone storm on uneventful ticks | Phase 2a Tier 1 — Left renderer slope dropped 95%; combined PR shows Standings at −0.4 MB/min (declining) |
1024+
| **Standings widget** | Per-driver allocation pattern in Standings | Phase 2a Tier 1 (via H4) and H1 rewrite — slope now under target across multi-class scenarios |
1025+
| **Main process slope** | Persistent +2–6 MB/min steady-state growth in main process | Phase 2a combined — Main slope dropped to +0.5 MB/min in combined PR test, at target. Likely attributed to Tier 2a session-boundary cleanup. |
9461026

9471027
### Partially fixed, work outstanding
9481028

‎docs/PERFORMANCE_TEST_SUMMARY.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
11
# irDashies Performance — Empirical Test Summary
22

3+
> **August 2026 Phase 3 Fuel update:** A deterministic 420-second A/B of
4+
> `0d74f276` against `e3dd77aa` validated the first channel-only widget path.
5+
> The candidate Fuel renderer measured approximately 4.87 Hz
6+
> `fuel.projection` delivery under its 5 Hz configuration and received zero
7+
> legacy telemetry callbacks; the baseline channel rate was not instrumented.
8+
> App-wide renderer wake-ups fell 42.4% with no visible Fuel or frame-pacing
9+
> regression. CPU and memory metrics also moved favorably, but their exact
10+
> changes remain inconclusive from one pair, and both builds failed the
11+
> `<5 MB/min` memory-slope gate. See the newest entry in
12+
> [`PERFORMANCE_TEST_LOG.md`](./PERFORMANCE_TEST_LOG.md).
13+
>
314
> **July 2026 controlled-replay update:** This historical live-session summary is
415
> now complemented by the deterministic, layer-by-layer
516
> [`TELEMETRY_PERFORMANCE_REPORT.html`](./TELEMETRY_PERFORMANCE_REPORT.html).

0 commit comments

Comments
 (0)