fix(benchmarks): quote the median run, not the best, and use the CI numbers

The first CI run showed why best-of-N is the wrong statistic here: node-tls-client
ranged from 900 to 3095 req/s across its eleven runs, so its best would have put
it above impit while its median sits a third below. That spread is intrinsic to
the client rather than machine noise, which is exactly the case best-of-N
flatters. The table now quotes the median, the result files keep best and worst,
and a client whose runs swing by more than 1.5x gets a footnote saying so.

The committed numbers are now the ones the workflow measured on a runner instead
of the ones from a loaded laptop.

Author: barjin

README.md

@@ -34,35 +34,37 @@ async fn main() {
 <!-- comparison:start -->
 ### Comparison
 
-Sequential requests from a single client against the local HTTP/2 origin in [`benchmarks/`](benchmarks), 1 KiB JSON response, best of 11 runs of 2000 requests. Every client negotiated h2. Each one keeps a single connection warm for the whole run unless a footnote says otherwise. `Profiles` counts the distinct impersonation targets each public API accepts, ignoring aliases that resolve to another target. Python sizes are the platform wheel; Node.js sizes are what `npm install <package>` leaves on disk, transitive dependencies included.
+Sequential requests from a single client against the local HTTP/2 origin in [`benchmarks/`](benchmarks), 1 KiB JSON response, median of 11 runs of 2000 requests. Every client negotiated h2. Each one keeps a single connection warm for the whole run unless a footnote says otherwise. `Profiles` counts the distinct impersonation targets each public API accepts, ignoring aliases that resolve to another target. Python sizes are the platform wheel; Node.js sizes are what `npm install <package>` leaves on disk, transitive dependencies included.
 
 **Python**
 
 | Package | req/s | Wheel | Profiles | Backend |
 | --- | --- | --- | --- | --- |
-| [`primp`](https://github.com/deedy5/primp) | 2750 | 5.9 MB | —[^1] | Rust |
-| [`rnet`](https://github.com/0x676e67/rnet) | 2735 | 3.7 MB | 75 | Rust |
-| **`impit`** | 2000 | 4.2 MB | 20 | Rust |
-| [`tls-client`](https://github.com/FlorianREGAZ/Python-Tls-Client) | 1338 | 41.3 MB | 51 | Go |
-| [`curl_cffi`](https://github.com/lexiforest/curl_cffi) | 1116 | 13.5 MB | 38 | C (libcurl) |
-| `httpx` (no impersonation) | 797 | 0.1 MB | — | Python |
+| [`primp`](https://github.com/deedy5/primp) | 6214 | 5.9 MB | —[^1] | Rust |
+| [`rnet`](https://github.com/0x676e67/rnet) | 5104 | 3.7 MB | 75 | Rust |
+| **`impit`** | 3780 | 4.2 MB | 20 | Rust |
+| [`curl_cffi`](https://github.com/lexiforest/curl_cffi) | 3420 | 13.5 MB | 38 | C (libcurl) |
+| [`tls-client`](https://github.com/FlorianREGAZ/Python-Tls-Client) | 3211 | 41.3 MB | 51 | Go |
+| `httpx` (no impersonation) | 2323 | 0.1 MB | — | Python |
 
 **Node.js**
 
 | Package | req/s | Install | Profiles | Backend |
 | --- | --- | --- | --- | --- |
-| [`node-tls-client`](https://github.com/Sahil1337/node-tls-client) | 1566 | 30.7 MB | 63 | Go |
-| **`impit`** | 997 | 8.7 MB | 20 | Rust |
-| [`got-scraping`](https://github.com/apify/got-scraping) | 896 | 4.7 MB | 3[^2] | Node.js TLS |
-| [`cycletls`](https://github.com/Danny-Dasilva/CycleTLS) | 201[^3] | 133.0 MB | —[^4] | Go subprocess |
-| `undici` (no impersonation) | 2010 | 1.9 MB | — | Node.js |
+| **`impit`** | 2289 | 8.7 MB | 20 | Rust |
+| [`got-scraping`](https://github.com/apify/got-scraping) | 2234 | 4.7 MB | 3[^2] | Node.js TLS |
+| [`node-tls-client`](https://github.com/Sahil1337/node-tls-client) | 1659[^3] | 30.7 MB | 63 | Go |
+| [`cycletls`](https://github.com/Danny-Dasilva/CycleTLS) | 626[^4][^5] | 133.0 MB | —[^6] | Go subprocess |
+| `undici` (no impersonation) | 5064 | 1.9 MB | — | Node.js |
 
-Measured on linux-x64 with CPython 3.12.13 and Node.js v24.16.0 on 2026-08-31. Hardware moves these numbers around, so rerun `benchmarks/` yourself before drawing conclusions.
+Measured on linux-x64 with CPython 3.12.3 and Node.js v24.19.0 on 2026-08-31. Hardware moves these numbers around, so rerun `benchmarks/` yourself before drawing conclusions.
 
 [^1]: `primp` does not expose its profile list, and an unknown name silently falls back to a random profile rather than erroring, so the set cannot be counted.
 [^2]: `got-scraping` matches cipher suite and signature algorithm order only; it has no control over extension order, GREASE, or HTTP/2 `SETTINGS`. Its three profiles are not enumerable through the public API, so this count is hard-coded from its bundled cipher table.
-[^3]: `cycletls` opens a new connection for every request, so its figure includes a TLS handshake each time instead of reusing a warm one.
-[^4]: `cycletls` is configured with a raw JA3 string instead of named profiles, so it has no fixed set to count.
+[^3]: `node-tls-client` was erratic across runs — 900 to 3095 req/s — so its median says less than the others'.
+[^4]: `cycletls` was erratic across runs — 409 to 640 req/s — so its median says less than the others'.
+[^5]: `cycletls` opens a new connection for every request, so its figure includes a TLS handshake each time instead of reusing a warm one.
+[^6]: `cycletls` is configured with a raw JA3 string instead of named profiles, so it has no fixed set to count.
 <!-- comparison:end -->
 
 ### Other projects

benchmarks/README.md

@@ -25,10 +25,12 @@ download` to size each wheel.
 
 [`server.mjs`](server.mjs) is the origin: a Node.js `http2` server on a self-signed certificate,
 serving a fixed 1 KiB JSON body over `h2` or `http/1.1`, whichever the client negotiates. Each
-client then issues sequential requests over one connection; the best of N runs is reported, which
-absorbs scheduler noise without flattering a client that is genuinely slow. Sequential single-client
-traffic is deliberate — it isolates per-request client overhead, which is what differs between these
-libraries, rather than measuring how well each one saturates a socket.
+client then issues sequential requests over one connection, N runs of it, and the **median** run is
+what the table quotes. Best and worst go into the result file too: some clients swing by 3x between
+runs on the same machine, and quoting the best would reward them for one lucky pass — where that
+spread is wide, `update-readme.mjs` footnotes it instead of pretending one number describes them.
+Sequential single-client traffic is deliberate: it isolates per-request client overhead, which is
+what differs between these libraries, rather than measuring how well each one saturates a socket.
 
 The server also reports its connection count at `/__stats`, and both scripts record how many
 connections a client opened while being measured. That is what surfaces `cycletls` handshaking on

benchmarks/harness.mjs

@@ -19,10 +19,11 @@ export function parseArgs(argv, defaults) {
 }
 
 /**
- * Runs `requests` sequential requests `runs` times and keeps the best run.
- * Sequential traffic over one warm connection isolates per-request client
- * overhead, which is what the comparison is about; best-of-N absorbs scheduler
- * noise without hiding a client that is genuinely slow.
+ * Runs `requests` sequential requests `runs` times over one warm connection,
+ * which isolates per-request client overhead — the thing that differs between
+ * these libraries. The median is what the table quotes; the best and worst runs
+ * come along because some clients swing by 3x between runs, and a best-of-N
+ * figure would quietly reward them for one lucky pass.
  */
 export async function measure(request, { requests, runs, warmup }) {
   for (let i = 0; i < warmup; i += 1) await request();

benchmarks/update-readme.mjs

@@ -40,12 +40,19 @@ const dominantAlpn = [...python.results, ...node.results]
   .map((result) => result.alpn)
   .reduce((agreed, alpn) => (agreed === alpn ? agreed : null));
 
+/** Above this best-to-worst ratio a client's throughput is too unsteady to quote as one number. */
+const UNSTABLE_RATIO = 1.5;
+
 /** Notes about how the throughput figure was reached, rendered next to it. */
 function throughputNotes(result, report) {
   const notes = [];
   if (result.alpn !== dominantAlpn) {
     notes.push(footnote(`\`${result.label}\` negotiated ${result.alpn} rather than ${dominantAlpn}.`));
   }
+  if (result.rps > result.rpsWorst * UNSTABLE_RATIO) {
+    notes.push(footnote(`\`${result.label}\` was erratic across runs — ${result.rpsWorst.toFixed(0)} to `
+      + `${result.rps.toFixed(0)} req/s — so its median says less than the others'.`));
+  }
   const total = report.options.runs * report.options.requests;
   if (result.connections >= total / 2) {
     notes.push(footnote(`\`${result.label}\` opens a new connection for every request, so its figure `
@@ -57,12 +64,13 @@ function throughputNotes(result, report) {
 }
 
 function table(report, sizeHeading) {
-  const ordered = [...report.results].sort((a, b) => (a.baseline - b.baseline) || (b.rps - a.rps));
+  const ordered = [...report.results]
+    .sort((a, b) => (a.baseline - b.baseline) || (b.rpsMedian - a.rpsMedian));
   const rows = ordered.map((result) => {
     const name = result.repo ? `[\`${result.label}\`](${result.repo})` : `\`${result.label}\``;
     return [
       result.baseline ? `${name} (no impersonation)` : (result.repo ? name : `**${name}**`),
-      `${result.rps.toFixed(0)}${throughputNotes(result, report)}`,
+      `${result.rpsMedian.toFixed(0)}${throughputNotes(result, report)}`,
       formatMB(result.sizeBytes),
       `${result.profiles ?? '—'}${result.note ? footnote(result.note) : ''}`,
       result.backend,
@@ -78,7 +86,7 @@ function table(report, sizeHeading) {
 const { requests, runs, bodyBytes } = python.options;
 const caption = [
   `Sequential requests from a single client against the local HTTP/2 origin in [\`benchmarks/\`](benchmarks),`,
-  `${bodyBytes / 1024} KiB JSON response, best of ${runs} runs of ${requests} requests.`,
+  `${bodyBytes / 1024} KiB JSON response, median of ${runs} runs of ${requests} requests.`,
   dominantAlpn ? `Every client negotiated ${dominantAlpn}.` : '',
   'Each one keeps a single connection warm for the whole run unless a footnote says otherwise.',
   '`Profiles` counts the distinct impersonation targets each public API accepts, ignoring aliases that',