Skip to content

fix(bench): fail loud on a missing latency percentile instead of defaulting to 0#767

Merged
ELares merged 1 commit into
mainfrom
fix/bench-p999-extraction
Jul 24, 2026
Merged

fix(bench): fail loud on a missing latency percentile instead of defaulting to 0#767
ELares merged 1 commit into
mainfrom
fix/bench-p999-extraction

Conversation

@ELares

@ELares ELares commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Why

The last c7g #676 run was wasted by an "empty p999" -- and the harness reported a result anyway.

Root cause is a silent default that can't fire:

p999="$(json_field "${open_json}" p999_us)"
[[ -n "${p999}" ]] || p999="0"

jq -r .missing_key prints the string "null", which is non-empty, so the -n guard never triggers. A missing percentile became "null" (or 0 via the sed fallback) and flowed into the report and the moat ratio as a plausible-looking 0us tail.

That's the worst failure mode: the number looks real, so a paid bench run is silently invalidated.

What

Adds json_num FILE KEY LABEL -- requires a non-negative integer, else aborts naming the file, key, offending value, and dumping the first 20 lines of the results file. The four open-loop percentiles now use it: the p99.9 is the #518 moat metric and the #676 A/B verdict, so it must never be defaulted.

Verified

Unit-checked standalone against all five shapes: good file -> 30000 (rc=0); missing key / JSON null / non-numeric / empty file -> each exits 1 with a FATAL diagnostic. bash -n clean.

Pre-flight for the sanctioned c7g run, so a bad extraction aborts in seconds instead of producing a garbage tail.

…ulting it to 0

The last c7g #676 run was wasted by an "empty p999" and the harness did not say so -- it reported
a result anyway. Root cause: the extraction used a silent default,

    p999="$(json_field "${open_json}" p999_us)"
    [[ -n "${p999}" ]] || p999="0"

but `jq -r .missing_key` prints the STRING "null", which is non-empty, so the `-n` guard never
fires. A missing percentile therefore became the literal "null" (or, via the sed fallback, ""->0)
and flowed into the report and the moat ratio as a plausible-looking 0us tail. On a paid bench
host that is a silently invalidated run -- the worst failure mode, because the number LOOKS real.

Adds `json_num FILE KEY LABEL`: requires a non-negative integer and otherwise aborts naming the
file, the key, the offending value, and dumping the first 20 lines of the results file. The four
open-loop percentiles (p50/p99/p999/p9999) now use it -- the p99.9 IS the #518 moat metric and the
#676 A/B verdict, so it must never be defaulted.

Unit-checked standalone against all five shapes: a good file extracts 30000 (rc=0); a missing key,
a JSON null, a non-numeric value, and an empty file each exit 1 with a FATAL diagnostic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfFZ8gkkNhDBASuntB72HR
@github-actions

Copy link
Copy Markdown

perf-gate (A5)

Same-runner ratchet of HEAD against the merge-base (both rebuilt and measured in this job).
PASS = within the noise band, WARN = a real move inside budget (does not fail), FAIL = past budget in the bad direction.

metric base head delta% band budget verdict
qps_median (peak) 82455.98 82336.02 -0.15% +/-5.00% drop <= 15% PASS
bytes_per_key int 47.59 46.98 -1.28% det rise <= 5% PASS
bytes_per_key embstr 60.86 60.83 -0.05% det rise <= 5% PASS
bytes_per_key raw 332.61 332.78 0.05% det rise <= 5% PASS

Overall: PASS

  • qps: noisy on shared CI, so the band comes from the base reps spread (floored at 5%); a drop is only a regression past the 15% budget.
  • bytes_per_key: deterministic (allocator-true memmodel), so a tight 5% rise budget; any rise beyond it FAILs.
  • Open-loop tails / criterion micro-benches are reported-not-failed (tail noise is high) and are not part of this ratchet.
  • An intentional perf trade is landed by raising the relevant budget in this PR with a documented reason (CI never auto-commits a baseline).

@ELares
ELares merged commit 629ef37 into main Jul 24, 2026
2 checks passed
@ELares
ELares deleted the fix/bench-p999-extraction branch July 24, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant