test: reference board + measured baseline for the failure knee (#74 step 0) - #244
Merged
Conversation
…p 0) Step 0 of the failure-knee work, and deliberately no product code: the pre-implementation critique produced five different "reference boards" and five different collapse points (10-20 rps), which is a way of saying none of the design's claims were falsifiable. A board described in prose is not a measurement. tests/helpers/reference-board.mjs pins one: exact wiring and tiers, a fixed traffic mix (kept out of CONFIG so a survival balance tweak cannot silently move every baseline), mulberry32 seeding, and a frame loop that mirrors animate()'s order and its game-scaled dt. sweepAt() reports only quantities the design makes claims about — failures, min reputation, residency and mean dwell in the readable band, and time from the first drop to a reputation floor — so a claim can be falsified without writing new instrumentation. Harness guards throw on a board that did not build or a sweep with no RPS, because a silently dead sweep reports a clean board and a passing objective. The measured baseline on main (60s per point, seed 0x5eed): rps 4: 0 fail, rep 100 rps 8: 165 fail, rep -43.5 rps 5: 3 fail, rep 97.7 rps 9: 252 fail, rep -129.8 rps 6: 7 fail, rep 95.3 rps 10: 288 fail, rep -164.5 rps 7: 54 fail, rep 59.3 rps 20: 691 fail, rep -556.8 Two properties are asserted, both currently true and both meant to become false as the work lands: 1. The readable middle is a KNIFE EDGE. This corrected the design's own wording: a point where the board drops requests and still survives does exist (5-6 rps). What does not exist is WIDTH — 12.5% of the collapse load, against a 20% target — so the band cannot be found by a player adjusting an architecture rather than turning an rps dial. Width is the metric, not existence. 2. The band is a STROBE. Mean dwell of the bottleneck's instantaneous utilization inside (0.90, 1.20) never exceeds 0.245s at any load. No curve on this axis can produce a readable middle, because the state it describes does not persist long enough to perceive. Also lands the design doc at rev 2. Rev 1's central mechanic (congestion latency) was killed by the critique with measurements before a line was written: multiplying processingTime feeds back into the very quantity totalLoad measures, and at the specced k=3 a board that was fine at 8 rps went from 463 completed / 0 failures to 53 completed / 416 failures. Its payoff was imaginary too — recordServiceSuccess is called only from finishRequest, so a congested intermediate node never records a latency sample at all. Rev 2 changes the AXIS (an EWMA) rather than the curve, and keeps rev 1's errors written down, including the gate-vs-switch ambiguity that would have made 16 shipped campaign levels unlosable. 826/826, eslint clean.
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 0 of the work on #74 ("bored in 10 minutes"), and deliberately no product code.
Why this comes first
The pre-implementation critique of the knee design produced five different "reference boards" and five different collapse points (10–20 rps). That is another way of saying none of the design's claims were falsifiable. A board described in prose is not a measurement.
tests/helpers/reference-board.mjspins one: exact wiring and tiers, a fixed traffic mix (kept out of CONFIG so a survival balance tweak cannot silently move every baseline), mulberry32 seeding, and a frame loop mirroringanimate()'s order and its game-scaled dt.sweepAt()reports only the quantities the design makes claims about — failures, min reputation, band residency, mean dwell, time from first drop to a reputation floor — so a claim can be falsified without writing new instrumentation. Harness guards throw on a board that did not build or a sweep with no RPS, because a silently dead sweep otherwise reports a clean board and a passing objective.The measured baseline on main (60 s per point)
Two properties, asserted — both true today, both meant to become false
1. The readable middle is a knife edge. This measurement corrected the design's own wording: a point where the board drops requests and still survives does exist (5–6 rps). What does not exist is width — 12.5 % of the collapse load, against a 20 % target. A player adjusting an architecture, rather than turning an rps dial, cannot find it. Width is the metric, not existence.
2. The band is a strobe. Mean dwell of the bottleneck's instantaneous utilization inside (0.90, 1.20) never exceeds 0.245 s at any load. No curve placed on this axis can produce a readable middle, because the state it describes does not persist long enough to perceive. This is why the design changes the axis (an EWMA), not the curve.
Also lands the design doc at rev 2
Rev 1's central mechanic — congestion latency — was killed by the critique with measurements, before a line of product code: multiplying
processingTimefeeds back into the very quantitytotalLoadmeasures, and at the specced k=3 a board healthy at 8 rps went from 463 completed / 0 failures to 53 completed / 416 failures (stability ceiling ≈ 0.25–0.4). Its payoff was imaginary too —recordServiceSuccessis called only fromfinishRequest, so a congested intermediate node never records a latency sample at all.Rev 1's errors are kept in the doc on purpose, including the gate-vs-switch ambiguity that would have made 16 shipped campaign levels unlosable (level 15's design comment says outright that its pre-built Compute "sits ~10 % over its own throughput").
826/826, eslint clean. Steps 1–5 (ramp interpolation →
smoothedLoad→ rings/alert → the knee switch → ASG) land as their own PRs, each verified against this baseline.