ci(perf-gate): loosen per-kernel thresholds to absorb between-run drift

The first real-CI run of the perf gate (CI run 25614982597 on this
PR's own no-op change) tripped a hard fail on `Magnitude` at 0.840x
and soft-warned on five other kernels in [0.913, 0.965], despite
this PR not changing any kernel code. The PR-side and main-side
binaries should be byte-identical (same source, same auto-detected
SIMD features, same `-C target-cpu=x86-64-v3`), so the only
explanation is between-run noise on the same VM.

Empirical observation: between-run drift on otherwise-identical
binaries on the same runner VM hits ~10-15% per kernel, which is
considerably higher than the within-run CV% the bench reports
(typically <1%). Cache state across the two consecutive
`./openvx-mark` process launches, thermal headroom, and VM-host
neighbour load are the usual culprits. The within-run CV% filter
(`--max-cv 5.0`) doesn't catch this because it only inspects
samples within a single bench process.

Recalibration:

  --kernel-floor 0.85 -> 0.75
    Per-kernel hard fail now requires >25% regression. Generous
    enough to absorb the worst between-run drift we've observed
    (the 16% Magnitude blip on the failed CI run sits comfortably
    above the new floor).

  --warn-floor   0.97 -> 0.90
    Soft-warn band moves from "any kernel slower than 3%" to
    "individual kernels in [-25%, -10%)". Below 10% is treated as
    noise and not flagged.

  --geomean-floor 0.97 (unchanged)
    Aggregate move > 3% across 50+ verified kernels stays the
    primary gate signal. That magnitude of aggregate drift is
    essentially impossible to fake with single-kernel noise: it
    requires a real software-side regression that touches the hot
    path. Keeping this strict.

Self-tests on the four reference input pairs (PR12 vs pre-PR12
main, reversed, identity, same-side) still behave correctly with
the new thresholds: PASS with verdict 1.375x on the real perf wins,
FAIL with verdict 0.727x and 7 hard-failed kernels on the simulated
regression, PASS with 1.000x on the identity pair. Applying the new
thresholds to the offending CI run's data turns its 1 hard-fail /
5 soft-warn output into the PASS verdict it should have had on a
no-op PR.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: kiritigowda

.github/scripts/perf_gate.py

@@ -13,9 +13,23 @@
 
 Defaults:
     --geomean-floor 0.97   (no more than 3% aggregate slowdown)
-    --kernel-floor  0.85   (no kernel may regress more than 15%)
+    --kernel-floor  0.75   (no kernel may regress more than 25%)
+    --warn-floor    0.90   (soft-warn band for individual kernels in
+                            [0.75, 0.90); below 10% we treat as noise)
     --max-cv        5.0    (skip kernels above this run-to-run noise)
 
+The per-kernel floor is intentionally generous (0.75x = 25%
+allowed regression) because between-run drift on otherwise-identical
+binaries on the SAME runner VM measures ~10-15% per kernel in real
+CI — well above the within-run CV% the bench itself reports. Cache
+state, thermal headroom, and VM-host neighbour load are the usual
+suspects. A tighter per-kernel floor produced false positives on
+no-op PRs.
+
+Aggregate moves > 3% across 50+ verified kernels are essentially
+impossible to fake with noise, which is why the geomean floor is
+the real gate signal — it stays at 0.97x.
+
 Each filter is applied independently; a kernel that doesn't pass the
 filters (unverified, noisy, missing on either side) is reported in a
 "skipped" section but does not contribute to the gate decision.
@@ -284,10 +298,12 @@ def main(argv: list[str]) -> int:
     p.add_argument("pr_json", help="benchmark_results.json from PR's rustVX run")
     p.add_argument("--geomean-floor", type=float, default=0.97,
                    help="Aggregate geomean floor (default 0.97 = up to 3%% regression)")
-    p.add_argument("--kernel-floor", type=float, default=0.85,
-                   help="Per-kernel floor (default 0.85 = up to 15%% regression)")
-    p.add_argument("--warn-floor", type=float, default=0.97,
-                   help="Soft warn floor (default 0.97 = warn between -3%% and -15%%)")
+    p.add_argument("--kernel-floor", type=float, default=0.75,
+                   help="Per-kernel floor (default 0.75 = up to 25%% regression; "
+                        "generous to absorb ~10-15%% between-run noise on real CI)")
+    p.add_argument("--warn-floor", type=float, default=0.90,
+                   help="Soft warn floor (default 0.90 = warn for individual "
+                        "kernels in [-25%%, -10%%); below 10%% is treated as noise)")
     p.add_argument("--max-cv", type=float, default=5.0,
                    help="Skip kernels whose CV%% exceeds this threshold (default 5.0)")
     p.add_argument("--summary-out", default=None,

.github/workflows/conformance.yml

@@ -846,10 +846,28 @@ jobs:
   # the regression thresholds, posts a markdown verdict to the PR's job
   # summary, and exits non-zero (i.e. fails the workflow) on regression.
   #
-  # Defaults:
-  #   * geomean PR/main >= 0.97   (no more than 3% aggregate slowdown)
-  #   * per-kernel PR/main >= 0.85 (no kernel may regress more than 15%)
-  #   * skip kernels with stability_warning / verified=false / cv_percent > 5%
+  # Threshold rationale (see `.github/scripts/perf_gate.py` for full
+  # docstring and per-flag semantics):
+  #
+  #   * --geomean-floor 0.97     -> aggregate move > 3% slower fails. This
+  #                                 is the real signal for actual perf bugs
+  #                                 that affect multiple kernels.
+  #   * --kernel-floor  0.75     -> a SINGLE-kernel hard fail requires
+  #                                 > 25% regression. This is intentionally
+  #                                 generous: we measured ~10-15% between-
+  #                                 run drift on otherwise-identical
+  #                                 binaries on the same VM (cache state,
+  #                                 thermal, VM-host neighbour load), well
+  #                                 above the within-run CV% the bench
+  #                                 itself reports. A tighter per-kernel
+  #                                 floor produced false positives on
+  #                                 no-op PRs (CI run 25614982597).
+  #   * --warn-floor    0.90     -> soft-warn band [0.75, 0.90). Below 10%
+  #                                 we treat as noise.
+  #   * --max-cv        5.0      -> auto-skip kernels above this within-
+  #                                 run CV%; combined with the looser
+  #                                 per-kernel floor this gives us a clean
+  #                                 signal-to-noise ratio.
   #
   # Trigger:
   #   * pull_request only — push events to main do not gate against
@@ -892,7 +910,7 @@ jobs:
           python3 ${{ github.workspace }}/.github/scripts/perf_gate.py \
             "$MAIN" "$PR" \
             --geomean-floor 0.97 \
-            --kernel-floor 0.85 \
-            --warn-floor 0.97 \
+            --kernel-floor 0.75 \
+            --warn-floor 0.90 \
             --max-cv 5.0 \
             --summary-out "$GITHUB_STEP_SUMMARY"