baseline.json

{
  "$comment": [
    "Committed performance baseline. Regenerate with scripts/run-benchmarks.sh.",
    "",
    "allocatedBytes is EXACT and BLOCKING, unless the entry declares allocationTolerancePercent,",
    "in which case it is a ceiling and the entry says why it cannot be exact.",
    "budgetNs is a hard ceiling and BLOCKING.",
    "absoluteNs and ratioToBaseline are ADVISORY: two runs of the same commit on the",
    "same container disagreed by up to 159% on absolute time and 63% on ratio, because",
    "the fastest benchmark sits at ~10-20 ns, on the measurement noise floor. WP-3",
    "claimed ratios were machine-independent; the data refuted it and the claim is",
    "withdrawn.",
    "",
    "Re-recorded at WP-11 with 10 warmups and 30 iterations (was 3 and 10) for the P0",
    "kill-criterion report. Still shared container hardware, and still advisory for",
    "timing -- see docs/benchmarks/P0.md for why that does not weaken the verdict.",
    "",
    "RE-RECORDED 2026-08-09, FOR THREE ALLOCATION FIGURES AND NOTHING ELSE.",
    "StepLoopBenchmarks.BuildPlan 520 -> 528 B, CompilerBenchmarks.GeneratorOnly 588937 ->",
    "857070 B, CompilerBenchmarks.WithGenerator 1508524 -> 2072971 B. The gate had been red on",
    "every push since these figures were taken on 2026-08-01, which is a gate nobody can act on.",
    "",
    "WHY THEY MOVED. The generator does more than it did on 2026-08-01: capability and dispatcher",
    "registrations are generated (b6691ea), an assembly's declarations are wired by one call",
    "(fdd4278), and [RabbitMqTrigger] and [ServiceBusTrigger] are read (458bc28) -- all after this",
    "file was written. More emitted code allocates more while emitting it.",
    "",
    "AND WHAT WAS NOT DONE: this was not bisected. The attribution above is from the commit log",
    "over the window, not from measuring each commit, so it is the likely cause rather than the",
    "proven one. A 45% rise in the generator's own allocations is worth somebody's afternoon even",
    "though it tracks features that shipped; recording it here makes the next rise visible instead",
    "of hiding it under a gate that was already red.",
    "",
    "MEASURED, AND FROM WHERE. Two CI runs on 2026-08-09 (f1d6745 and d8ad054) agreed to within",
    "0.4%: GeneratorOnly 853998 and 857070, WithGenerator 2068928 and 2072971, BuildPlan exactly",
    "528 both times. The higher of each pair is recorded. A developer machine reads WithGenerator",
    "at about 2.39 MB -- 15% above the CI figure -- because the allocation being measured is",
    "Roslyn's and it moves with the SDK's own paths. The gate runs in CI; CI is what is recorded."
  ],
  "recordedOn": {
    "runtime": ".NET 10.0.10 (10.0.1026.32716)",
    "sdk": "10.0.110",
    "os": "Linux Ubuntu 24.04.4 LTS (Noble Numbat)",
    "cpu": "Intel Xeon Processor 2.10GHz, 1 CPU, 4 logical and 4 physical cores",
    "arch": "X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI",
    "iterationCount": 30,
    "warmupCount": 10,
    "recordedAt": "WP-14",
    "note": "Shared container hardware. Timing figures are indicative only; allocations are the contract -- except the four banded entries: the three CompilerBenchmarks, which measure Roslyn's allocations rather than ours, and StepLoopBenchmarks.BuildPlan, whose last 64 B the JIT elides on some machines and not others. Each says which at its own entry. B1 measured 172.3 ns when P0.md was written and 129.5 ns here, on the same commit: a 25% swing that is exactly the noise P0.md documents, and that leaves its verdict untouched at a ~30x margin either way."
  },
  "benchmarks": {
    "DispatchBenchmarks.Direct": {
      "absoluteNs": 17.54,
      "ratioToBaseline": 1.0,
      "allocatedBytes": 0,
      "budget": "B3",
      "budgetNs": 150
    },
    "DispatchBenchmarks.ViaInterface": {
      "absoluteNs": 29.3,
      "ratioToBaseline": 1.67,
      "allocatedBytes": 0,
      "budget": "B3",
      "budgetNs": 150,
      "note": "Not slower than a direct call: the JIT devirtualises a sealed monomorphic call site."
    },
    "DispatchBenchmarks.ViaDelegate": {
      "absoluteNs": 18.71,
      "ratioToBaseline": 1.067,
      "allocatedBytes": 0,
      "budget": "B3",
      "budgetNs": 150
    },
    "DispatchBenchmarks.ViaReflection": {
      "absoluteNs": 80.77,
      "ratioToBaseline": 4.605,
      "allocatedBytes": 48,
      "budget": "reference",
      "budgetNs": 150,
      "note": "Not a target. The 48 B is the argument ADR-0002 actually rests on."
    },
    "EngineBenchmarks.Query": {
      "absoluteNs": 129.52,
      "ratioToBaseline": 1.0,
      "allocatedBytes": 0,
      "budget": "B1",
      "budgetNs": 5000,
      "note": "B1 proper: a 4-step flow through the real engine."
    },
    "EngineBenchmarks.SagaSuccess": {
      "absoluteNs": 160.69,
      "ratioToBaseline": 1.241,
      "allocatedBytes": 0,
      "budget": "B1",
      "budgetNs": 5000,
      "note": "What declaring compensations costs when nothing fails."
    },
    "EngineBenchmarks.SagaFailure": {
      "absoluteNs": 201.68,
      "ratioToBaseline": 1.557,
      "allocatedBytes": 56,
      "budget": "B1",
      "budgetNs": 5000,
      "note": "Failure path with a full unwind. Still one object, and still the Unwind iterator -- what moved is its size, from 40 B to 56 B. An iterator's state machine carries the value it yields, and CompensationStack.Unwind stopped yielding a bare StepNode: it now yields CompensationEntry, which grew from 8 B to 24 B over two commits. 744b005 (WP-29 ForEach) added FlowContext? Scope, so a compensation inside a loop undoes the element its own step processed rather than whichever one the loop ended on. 16b6988 (WP-57 compensation policies) added StepScope JournalScope, so the compensation row written for the third element is keyed as the third element instead of colliding with the first. Both are correctness, neither is removable, and 40 B is therefore not reachable without giving one of them up. The iterator itself stays for the reason AllocationBudgetTests.UnwindingAllocatesOneIteratorPerFailedFlow already records: once per FAILED flow, immediately before a compensation makes a network call. That test now pins 56 B exactly, so the next byte is caught by the unit-test job rather than only here."
    },
    "StepLoopBenchmarks.WalkPlan": {
      "absoluteNs": 3.03,
      "ratioToBaseline": 1.0,
      "allocatedBytes": 0,
      "budget": "B1-precursor",
      "budgetNs": 5000
    },
    "StepLoopBenchmarks.WalkAndDispatch": {
      "absoluteNs": 28.45,
      "ratioToBaseline": 9.389,
      "allocatedBytes": 0,
      "budget": "B1-precursor",
      "budgetNs": 5000
    },
    "StepLoopBenchmarks.CompensateAll": {
      "absoluteNs": 89.76,
      "ratioToBaseline": 29.624,
      "allocatedBytes": 440,
      "budget": "none",
      "budgetNs": 5000,
      "note": "Constructs a stack per call; the engine reuses a pooled one instead. Moved from 328 B to 440 B for the same reason EngineBenchmarks.SagaFailure moved, and in the same commit: CompensationEntry grew from 8 B to 24 B, so the Stack<CompensationEntry> this benchmark fills with four entries grew with it, and the duplicate check's key went from int to (int, FlowContext?). Verified rather than assumed -- 328 B reproduces exactly at e6fcd37, the commit that recorded this baseline, and at 1c654eb, the commit before WP-29."
    },
    "StepLoopBenchmarks.BuildPlan": {
      "absoluteNs": 164.07,
      "ratioToBaseline": 54.149,
      "allocatedBytes": 464,
      "budget": "build-time",
      "budgetNs": 100000,
      "note": "Runs once per flow at startup, not per execution. BANDED RATHER THAN EXACT, and the only entry outside CompilerBenchmarks that is -- for a different reason, which is the point. Four figures are on record for the same benchmark: 456 B on a container at e6fcd37, 464 B on a container at dev, 528 B on the hosted runner at dev, 520 B in the file. The 64 B between the two clusters is not allocated by FlowX on every run: ExecutionPlan.Create builds a temporary the JIT can prove does not escape, and when escape analysis reaches it the object never touches the heap. Reproduced at 4871849 by toggling the JIT and nothing else -- default 464 B; DOTNET_JitObjectStackAllocation=0 528 B; DOTNET_TieredPGO=0 528 B, because dynamic PGO is what supplies the inlining escape analysis needs. Whether the JIT gets there is decided by profile-guided inlining, which follows tier-up timing and the machine, not this repository, so a byte-exact gate here asserts something about the runtime rather than about the code. Its neighbours do not have that property and stay exact: CompensateAll's Stack<CompensationEntry> genuinely escapes, no JIT decision can remove it, and it read exactly 440 B under every configuration above. 464 B is recorded because it is what the shipping JIT costs; the 15% band is CompilerBenchmarks' figure, puts the ceiling at 533 B, admits the un-elided 528 B, and still fails anything that adds an allocation. Two things were checked and refused. PLAN §9 item 15 names Roslyn pool sizing as the suspect -- ruled out: BuildPlan calls ExecutionPlan.Create and reaches no Roslyn at all. And pinning DOTNET_TieredPGO=0 in the benchmark job would make the figure byte-exact at 528 B -- refused, because it buys exactness for one allocation by measuring every timing budget in this harness under a JIT nobody deploys, and records a number production never pays.",
      "allocationTolerancePercent": 15
    },
    "CompilerBenchmarks.WithoutGenerator": {
      "absoluteNs": 4044320.78,
      "ratioToBaseline": 1.414,
      "allocatedBytes": 678312,
      "budget": "B12",
      "budgetNs": 60000000,
      "note": "Build overhead. The ratio here does NOT answer the 8% budget -- the arms are not like-for-like. See docs/benchmarks/B12.md. Allocations are banded rather than exact because this benchmark measures Roslyn's allocations, not ours, and those move between runs of the same commit. Committed so a generator that became much slower or much greedier shows as a regression while the ratio question is open. THE RATIO QUESTION IS CLOSED as of 2026-08-10: ADR-0014 replaced the 8% ratio with generator allocation per flow and per capability, checked by check-generator-cost.py. These three entries still measure what they always measured and are still worth committing, but they no longer sit beside an open question.",
      "allocationTolerancePercent": 15
    },
    "CompilerBenchmarks.GeneratorOnly": {
      "absoluteNs": 2860116.13,
      "ratioToBaseline": 1.0,
      "allocatedBytes": 857070,
      "budget": "B12",
      "budgetNs": 60000000,
      "note": "Build overhead. The ratio here does NOT answer the 8% budget -- the arms are not like-for-like. See docs/benchmarks/B12.md. Allocations are banded rather than exact because this benchmark measures Roslyn's allocations, not ours, and those move between runs of the same commit. Committed so a generator that became much slower or much greedier shows as a regression while the ratio question is open. THE RATIO QUESTION IS CLOSED as of 2026-08-10: ADR-0014 replaced the 8% ratio with generator allocation per flow and per capability, checked by check-generator-cost.py. These three entries still measure what they always measured and are still worth committing, but they no longer sit beside an open question. Re-recorded 2026-08-09: 588937 -> 857070 B.",
      "allocationTolerancePercent": 15
    },
    "CompilerBenchmarks.WithGenerator": {
      "absoluteNs": 11173681.82,
      "ratioToBaseline": 3.907,
      "allocatedBytes": 2072971,
      "budget": "B12",
      "budgetNs": 60000000,
      "note": "Build overhead. The ratio here does NOT answer the 8% budget -- the arms are not like-for-like. See docs/benchmarks/B12.md. Allocations are banded rather than exact because this benchmark measures Roslyn's allocations, not ours, and those move between runs of the same commit. Committed so a generator that became much slower or much greedier shows as a regression while the ratio question is open. THE RATIO QUESTION IS CLOSED as of 2026-08-10: ADR-0014 replaced the 8% ratio with generator allocation per flow and per capability, checked by check-generator-cost.py. These three entries still measure what they always measured and are still worth committing, but they no longer sit beside an open question. Re-recorded 2026-08-09: 1508524 -> 2072971 B.",
      "allocationTolerancePercent": 15
    }
  },
  "tolerances": {
    "absolutePercent": 40,
    "ratioPercent": 15,
    "allocatedBytesExact": true
  }
}