generator-cost-baseline.json

{
  "$comment": [
    "PER-CAPABILITY IS NOW RECORDED BESIDE PER-FLOW, and it is the unit the argument about this budget needs. Bytes per flow is right for the plan emitter -- one plan per flow -- and wrong for the error catalogue, which is derived per capability; the synthetic project declares about five capabilities per flow, so a per-flow figure mixes two rates. Both are flat across 25 and 50 flows (772,522 / 769,272 per flow; 148,562 / 146,808 per capability), which is what an absolute unit should look like and what the +8 % criterion is not: the same generator measures +0.4 % on the one-flow reference sample and +67.1 % at 200 flows, because the denominator moves with the subject. Measuring the catalogue against a capability count is what lets its cost be stated once -- about 65,000 bytes per capability, from the arm with the reader stubbed out -- instead of restated per project shape.",
    "",
    "RE-RECORDED 2026-08-10 BECAUSE THE HARNESS WAS MEASURING ITS OWN CACHE. The probe parsed the synthetic project's syntax trees once and handed the same instances to every run. That was correct while nothing downstream remembered anything about a tree; ErrorCatalogueReader now memoises a capability's catalogue against the identity of the trees it read, so run 2 onward was a memo hit and the probe reported a 39 % improvement that did not exist. The probe now parses fresh trees per run, outside the measured window. Against the fixed probe the memo costs +0.4 % on a cold build and the previous baseline was ~4.4 % optimistic -- the figures below are the first honest cold numbers.",
    "",
    "Committed baseline for FlowPlanGenerator's compile-time cost, in bytes allocated per",
    "run against a synthetic project of N flows. Regenerate deliberately:",
    "",
    "    ./scripts/measure-generator-cost.py --json /tmp/cost.json",
    "    ./scripts/check-generator-cost.py /tmp/cost.json --record",
    "",
    "and commit the diff. That is the same contract docs/benchmarks/baseline.json and",
    "samples/ecommerce/flowx.manifest.baseline.json already use, for the same stated",
    "reason: a real change then arrives as a reviewable diff rather than as a number in a",
    "log that nobody reads.",
    "",
    "WHY BYTES AND NOT MILLISECONDS. docs/benchmarks/B12-scale.md measures build overhead",
    "in wall clock, because wall clock is what the +8 % budget means. Its own A/A control",
    "puts that instrument's noise floor at 3.9-8.7 % on a quiet container, and a full run",
    "costs half an hour. Across twelve independent runs of THIS measurement, taken on a",
    "4-core container at 1-minute load averages between 5.8 and 21.1, the gated figure",
    "moved by at most 0.069 % while wall clock over the same runs moved by 123 %. That is",
    "the whole argument for the choice, and docs/benchmarks/generator-cost-gate.md shows",
    "the table.",
    "",
    "It is the same split scripts/check-benchmark-budgets.py already makes and argues for",
    "at run time: allocation counts are exact and gateable on shared hardware, timings are",
    "not.",
    "",
    "THIS IS A RELATIVE GATE AND IT IS NOT THE BUDGET. See absoluteCriterion below, which",
    "check-generator-cost.py reprints on every run, passing or failing.",
    "",
    "RE-RECORDED 2026-08-05, AND NOT BY THE COMMIT NAMED BELOW. `recordedOn.commit` says",
    "which tree the numbers were taken on, which reads as an attribution and here is not",
    "one. The previous figures were recorded at fc842e8 on 2026-07-31, 96 commits to",
    "src/FlowX.Compiler ago, and dev had already drifted past the +2.0 % threshold before",
    "this commit existed. Measured on both trees, same machine, same run parameters:",
    "",
    "    dev      25 flows  18,393,360 B  +2.28 %   50 flows  36,740,280 B  +2.28 %",
    "    e4b650a  25 flows  18,394,328 B  +2.29 %   50 flows  36,670,448 B  +2.09 %",
    "",
    "Emitted characters are identical on both — 401,535 and 805,563 — because the probe",
    "references only Abstractions, Core and Runtime, so the two emitters e4b650a added are",
    "gated off for it and cost it nothing. What this commit contributes is +0.01 % at 25",
    "flows and -0.19 % at 50.",
    "",
    "WHAT IS NOT DISCHARGED: the +2.3 % itself. It arrived across 96 commits and nobody",
    "bisected it, so this re-recording moves the line without naming what crossed it. The",
    "gate is relative and works from here; the drift behind it is unattributed."
  ],
  "recordedOn": {
    "commit": "f9025ec",
    "recordedAt": "2026-08-15",
    "roslyn": "4.14.0-3.25262.10+8edf7bcd4f1594c3d68a6a567469f41dbd33dd1b",
    "roslynAssemblyVersion": "4.14.0.0",
    "framework": ".NET 10.0.10",
    "sdk": "10.0.110",
    "nproc": 4,
    "load1DuringRun": 1.48,
    "runsPerSize": 5,
    "warmupPerSize": 2
  },
  "tolerances": {
    "$comment": [
      "regressionPercent 2.0 is chosen from the A/A distribution, not from intuition.",
      "The gated statistic is the median of 5 runs after 2 discarded warm-ups. Across 12",
      "independent processes on a container at load 5.8-21.1 its worst deviation from the",
      "central value was 0.069 %, and the worst deviation of any single un-medianed sample",
      "was 0.576 %. 2.0 % is 29x the first and 3.5x the second, so a degenerate run whose",
      "median happens to land on a worst-case sample still passes.",
      "",
      "At the other end: the commit B12-scale.md section 5.2 bisects, which took the generator",
      "from 5.60 to 27.28 ms per flow, moves this figure by +101 %. The threshold is 1/50",
      "of that, so the gate catches a regression fifty times smaller than the one that got",
      "through, and would have failed the pull request that introduced it.",
      "",
      "It can be tightened to 1 % once the same distribution has been measured on a hosted",
      "runner. It has not been, and widening it is the wrong response if a runner ever",
      "disagrees: re-record the baseline where the gate runs. See generator-cost-gate.md.",
      "",
      "staleBaselinePercent is symmetric with it on purpose. A generator that got 2 %",
      "cheaper and left the baseline alone has quietly bought itself 2 % of headroom for",
      "the next regression, so an improvement past the noise floor asks to be recorded too",
      "-- as a notice, never as a build failure.",
      "",
      "maxSpreadPercent 2.0 is the INCONCLUSIVE limit, against a worst observed spread of",
      "0.59 %. Tripping it does not mean the machine was busy; this measurement does not",
      "care. It means the same generator gave different answers about the same sources."
    ],
    "regressionPercent": 2.0,
    "staleBaselinePercent": 2.0,
    "maxSpreadPercent": 2.0
  },
  "absoluteCriterion": {
    "$comment": [
      "Reprinted by check-generator-cost.py on every run, including passing ones. A relative",
      "gate answers 'did this change make it worse'. It cannot answer 'is it good enough', and",
      "this package exists because a criterion nobody could act on stopped being watched for",
      "four working packages.",
      "",
      "RE-EXPRESSED 2026-08-10, when ADR-0014 was decided. The old criterion was '<= +8 %",
      "overhead at 200 flows', and its denominator was the user's code: the same generator",
      "measured +0.4 % on the one-flow reference sample and +67.1 % at 200 flows. A ratio whose",
      "bottom half moves with the subject cannot be passed or failed, only re-argued.",
      "",
      "The unit is bytes and not milliseconds, which is where this departs from what ADR-0014",
      "recommended. generator-cost-gate.md measured twelve identical runs of one tree",
      "disagreeing on wall clock by 139 % and on allocation by 0.069 %. Milliseconds stay",
      "reported and advisory.",
      "",
      "The ceilings are the measured figures with headroom, not aspirations: 772,522 and 769,272",
      "bytes per flow, 148,562 and 146,808 per capability, flat across 25 and 50 flows. About",
      "65,000 of the per-capability figure is the derived error catalogue, from the arm with the",
      "reader stubbed out -- which is the cost ADR-0014 decided was worth paying."
    ],
    "criterion": "the generator allocates <= 800,000 bytes per flow and <= 160,000 bytes per capability, at any subject size (docs/20-Roadmap.md, P1)",
    "bytesPerFlowCeiling": 800000,
    "bytesPerCapabilityCeiling": 160000,
    "lastMeasured": "772,522 B/flow and 148,562 B/capability at 25 flows; 769,272 and 146,808 at 50",
    "measuredBy": "./scripts/measure-generator-cost.py (allocation)",
    "recordedIn": "docs/adr/ADR-0014-derived-error-catalogue-vs-build-budget.md",
    "status": "PASS",
    "supersededCriterion": {
      "criterion": "a 200-flow synthetic solution builds with <= +8 % overhead",
      "lastMeasured": "+46.5 % [+42.4, +51.0] at 50 flows; +67.1 % [+61.9, +73.6] at 200 flows, A/A noise floor 10.6 %",
      "measuredBy": "./scripts/measure-scale-overhead.sh --rounds 12 --sizes 50,200 (wall clock)",
      "recordedIn": "docs/benchmarks/B12-scale.md section 5.4",
      "status": "FAIL",
      "$comment": "Kept, not deleted. A budget that was missed by fifty-nine points and then replaced is a thing a reader is entitled to see, and B12-scale.md's measurements remain valid measurements of what they measured."
    }
  },
  "sizes": {
    "25": {
      "allocatedBytes": 18232784,
      "allocatedBytesPerFlow": 729311.4,
      "sourcesSha256": "c80fa5b5f3bc074258b2a7707f3eaedfb0e192ffc5599ee5c3397250737743f8",
      "generatedTrees": 26,
      "generatedChars": 401560,
      "diagnostics": 24,
      "observedSpreadPercent": 0.1543
    },
    "50": {
      "allocatedBytes": 36328176,
      "allocatedBytesPerFlow": 726563.5,
      "sourcesSha256": "63dacc54694291ce27e7110ced5bd4f9061dbeac8554e0ad7eb1aa58b61c7a65",
      "generatedTrees": 51,
      "generatedChars": 805613,
      "diagnostics": 49,
      "observedSpreadPercent": 0.2868
    }
  }
}