fix: explicit unarmed marker for the placeholder latency baseline

The previous fail-on-zero behavior made every trusted run red until a
CI-captured baseline was committed — and this PR cannot ship one, since
it runs from a fork without secrets. Per review, the comparison is now
gated on the baseline file itself:

- baseline.json declares latency_guard: unarmed — the latency test
  skips with a loud arming message while the recall guard stays active
  (dropping --baseline-file entirely would have disabled recall too)
- committing a CI-captured benchmark-results-<sha> artifact as
  baseline.json arms the guard automatically: artifacts carry no
  latency_guard flag
- a zero p95 WITHOUT the unarmed marker still fails as a misconfigured
  baseline, and a non-zero p95 WITH the marker fails as inconsistent,
  so the guard can never be silently inactive by accident

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: Sravan1011

benchmarks/ci/README.md

@@ -69,9 +69,16 @@ The harness compares the current run's metrics against `baseline.json`:
 - **Latency**: Fails if P95 increases by more than the threshold (default 20%)
 - **Recall**: Fails if Recall@5 drops by more than the threshold (default 5pp)
 
-A zero latency baseline **fails** comparison runs rather than skipping — the
-guard cannot stay silently inactive. To arm it, commit a CI-captured
-baseline (see "Updating the baseline" below).
+The latency guard arms itself from the baseline file:
+
+- The checked-in placeholder declares `"latency_guard": "unarmed"` — the
+  latency test skips with a loud message while recall remains guarded, so
+  trusted runs are not red before the first baseline exists.
+- To arm the guard, download the `benchmark-results-<sha>` artifact from a
+  trusted CI run and commit it as `baseline.json`. Artifacts carry no
+  `latency_guard` flag, so committing one arms the guard automatically.
+- A zero p95 **without** the unarmed flag fails the run as a misconfigured
+  baseline — the guard can never be silently inactive by accident.
 
 Thresholds are configurable via CLI flags:
 

benchmarks/ci/baseline.json

@@ -1,7 +1,8 @@
 {
   "commit": "f4732a1",
   "timestamp": "2026-07-20T05:35:00+00:00",
-  "_note": "Recall values are measured (hardware-independent) so the recall guard is active. Latency values are intentionally zero and the latency guard FAILS on zero baselines in comparison runs \u2014 the first trusted CI run will be red until a CI-captured baseline is committed: run the Benchmark workflow (update_baseline=true also works), download the benchmark-results-<sha> artifact, and commit it here. Latency baselines must come from CI runners; numbers from other hardware are not comparable.",
+  "latency_guard": "unarmed",
+  "_note": "Recall values are measured (hardware-independent) so the recall guard is active. Latency values are zero and 'latency_guard: unarmed' marks this file as the explicit placeholder: the latency regression test skips (loudly) instead of failing every trusted run. To arm the latency guard, run the Benchmark workflow on a trusted ref, download the benchmark-results-<sha> artifact, and commit it as this file \u2014 artifacts carry no latency_guard flag, so committing one arms the guard automatically. A zero p95 WITHOUT the unarmed flag fails the run as a misconfigured baseline. Latency baselines must come from CI runners; numbers from other hardware are not comparable.",
   "latency_ms": {
     "p50": 0,
     "p95": 0,

benchmarks/ci/test_bench_ci_moss.py

@@ -420,16 +420,32 @@ def test_no_latency_regression(self, request, benchmark_results):
 
         _assert_baseline_compatible(baseline, benchmark_results)
 
+        if baseline.get("latency_guard") == "unarmed":
+            # The checked-in placeholder declares itself unarmed: latency
+            # baselines must come from CI runners, and none has been captured
+            # yet. Skip loudly (recall is still guarded) instead of failing
+            # every trusted run until the first artifact lands.
+            if baseline_p95 != 0:
+                pytest.fail(
+                    "baseline.json marks latency_guard as 'unarmed' but contains "
+                    "a non-zero p95 — remove the latency_guard flag to arm the "
+                    "guard."
+                )
+            pytest.skip(
+                "LATENCY GUARD NOT ARMED — baseline.json is the explicit "
+                "placeholder (latency_guard: unarmed). To arm it: download the "
+                "benchmark-results-<sha> artifact from a trusted CI run and "
+                "commit it as benchmarks/ci/baseline.json (the artifact carries "
+                "no latency_guard flag, so committing it arms the guard)."
+            )
+
         if baseline_p95 == 0:
-            # A zero baseline means the latency guard has never been armed.
-            # Skipping here would let every run pass with the guard silently
-            # inactive — fail instead, with the arming procedure.
+            # Zero without the explicit unarmed marker is a misconfigured
+            # baseline, not a placeholder — never a silent pass.
             pytest.fail(
-                "Baseline p95 is zero — the latency guard is not armed. Run the "
-                "Benchmark workflow with update_baseline=true, download the "
-                "benchmark-results-<sha> artifact, and commit it as "
-                "benchmarks/ci/baseline.json (values must come from CI runners; "
-                "this run's artifact works too)."
+                "Baseline p95 is zero but baseline.json does not declare "
+                "latency_guard: unarmed — the baseline is misconfigured. Commit "
+                "a CI-captured baseline or restore the explicit placeholder."
             )
 
         regression = (current_p95 - baseline_p95) / baseline_p95