Skip to content

test: reference board + measured baseline for the failure knee (#74 step 0) - #244

Merged
pshenok merged 1 commit into
mainfrom
feat/knee-step0-reference-board
Aug 8, 2026
Merged

test: reference board + measured baseline for the failure knee (#74 step 0)#244
pshenok merged 1 commit into
mainfrom
feat/knee-step0-reference-board

Conversation

@pshenok

@pshenok pshenok commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Step 0 of the work on #74 ("bored in 10 minutes"), and deliberately no product code.

Why this comes first

The pre-implementation critique of the knee design produced five different "reference boards" and five different collapse points (10–20 rps). That is another way of saying none of the design's claims were falsifiable. A board described in prose is not a measurement.

tests/helpers/reference-board.mjs pins one: exact wiring and tiers, a fixed traffic mix (kept out of CONFIG so a survival balance tweak cannot silently move every baseline), mulberry32 seeding, and a frame loop mirroring animate()'s order and its game-scaled dt. sweepAt() reports only the quantities the design makes claims about — failures, min reputation, band residency, mean dwell, time from first drop to a reputation floor — so a claim can be falsified without writing new instrumentation. Harness guards throw on a board that did not build or a sweep with no RPS, because a silently dead sweep otherwise reports a clean board and a passing objective.

The measured baseline on main (60 s per point)

rps failures min reputation
4 0 100
5 3 97.7
6 7 95.3
7 54 59.3
8 165 −43.5
20 691 −556.8

Two properties, asserted — both true today, both meant to become false

1. The readable middle is a knife edge. This measurement corrected the design's own wording: a point where the board drops requests and still survives does exist (5–6 rps). What does not exist is width — 12.5 % of the collapse load, against a 20 % target. A player adjusting an architecture, rather than turning an rps dial, cannot find it. Width is the metric, not existence.

2. The band is a strobe. Mean dwell of the bottleneck's instantaneous utilization inside (0.90, 1.20) never exceeds 0.245 s at any load. No curve placed on this axis can produce a readable middle, because the state it describes does not persist long enough to perceive. This is why the design changes the axis (an EWMA), not the curve.

Also lands the design doc at rev 2

Rev 1's central mechanic — congestion latency — was killed by the critique with measurements, before a line of product code: multiplying processingTime feeds back into the very quantity totalLoad measures, and at the specced k=3 a board healthy at 8 rps went from 463 completed / 0 failures to 53 completed / 416 failures (stability ceiling ≈ 0.25–0.4). Its payoff was imaginary too — recordServiceSuccess is called only from finishRequest, so a congested intermediate node never records a latency sample at all.

Rev 1's errors are kept in the doc on purpose, including the gate-vs-switch ambiguity that would have made 16 shipped campaign levels unlosable (level 15's design comment says outright that its pre-built Compute "sits ~10 % over its own throughput").

826/826, eslint clean. Steps 1–5 (ramp interpolation → smoothedLoad → rings/alert → the knee switch → ASG) land as their own PRs, each verified against this baseline.

…p 0)

Step 0 of the failure-knee work, and deliberately no product code: the
pre-implementation critique produced five different "reference boards"
and five different collapse points (10-20 rps), which is a way of saying
none of the design's claims were falsifiable. A board described in prose
is not a measurement.

tests/helpers/reference-board.mjs pins one: exact wiring and tiers, a
fixed traffic mix (kept out of CONFIG so a survival balance tweak cannot
silently move every baseline), mulberry32 seeding, and a frame loop that
mirrors animate()'s order and its game-scaled dt. sweepAt() reports only
quantities the design makes claims about — failures, min reputation,
residency and mean dwell in the readable band, and time from the first
drop to a reputation floor — so a claim can be falsified without writing
new instrumentation. Harness guards throw on a board that did not build
or a sweep with no RPS, because a silently dead sweep reports a clean
board and a passing objective.

The measured baseline on main (60s per point, seed 0x5eed):

  rps  4: 0 fail, rep 100     rps  8: 165 fail, rep -43.5
  rps  5: 3 fail, rep 97.7    rps  9: 252 fail, rep -129.8
  rps  6: 7 fail, rep 95.3    rps 10: 288 fail, rep -164.5
  rps  7: 54 fail, rep 59.3   rps 20: 691 fail, rep -556.8

Two properties are asserted, both currently true and both meant to become
false as the work lands:

1. The readable middle is a KNIFE EDGE. This corrected the design's own
   wording: a point where the board drops requests and still survives
   does exist (5-6 rps). What does not exist is WIDTH — 12.5% of the
   collapse load, against a 20% target — so the band cannot be found by
   a player adjusting an architecture rather than turning an rps dial.
   Width is the metric, not existence.
2. The band is a STROBE. Mean dwell of the bottleneck's instantaneous
   utilization inside (0.90, 1.20) never exceeds 0.245s at any load.
   No curve on this axis can produce a readable middle, because the
   state it describes does not persist long enough to perceive.

Also lands the design doc at rev 2. Rev 1's central mechanic (congestion
latency) was killed by the critique with measurements before a line was
written: multiplying processingTime feeds back into the very quantity
totalLoad measures, and at the specced k=3 a board that was fine at 8
rps went from 463 completed / 0 failures to 53 completed / 416 failures.
Its payoff was imaginary too — recordServiceSuccess is called only from
finishRequest, so a congested intermediate node never records a latency
sample at all. Rev 2 changes the AXIS (an EWMA) rather than the curve,
and keeps rev 1's errors written down, including the gate-vs-switch
ambiguity that would have made 16 shipped campaign levels unlosable.

826/826, eslint clean.
@pshenok
pshenok merged commit 9ccf553 into main Aug 8, 2026
1 check passed
@pshenok
pshenok deleted the feat/knee-step0-reference-board branch August 8, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant