fix(bench): make the gate loud on dropped data and the Makefile its single source

Review fixes for the bench-gate PR:

- Silence the sync engine's global logger inside the sync benchmarks:
  interleaved log.Printf output corrupted the 'BenchmarkX' result
  lines, so benchfmt parsed zero sync results and all three sync
  benchmarks silently dropped out of the gate on both sides.
- benchgate now treats unparseable result lines as a corrupted
  capture (report + exit 2) instead of skipping benchfmt syntax
  errors, and reports a gated unit missing from one side ('missing
  from baseline/candidate, not gated') instead of a bare continue.
  Custom ReportMetric units are reported as ungated.
- Split the sample-count policy: too few candidate samples stays a
  config error, but a short baseline (a legitimately partial base
  run) is reported and not gated.
- Print detected regressions even when a config issue forces exit 2,
  so a real violation is never hidden behind a configuration error.
- bench.yml now runs 'make bench-gate' on both sides; the Makefile's
  BENCH_GATE_PACKAGES/COUNT/TIME are the single source of truth, and
  each side benchmarks its own commit's package list, so growing the
  gate cannot empty the baseline. A failing base run degrades to its
  partial output plus a workflow warning instead of truncating the
  whole baseline into a vacuous pass. Run steps use set -euo
  pipefail and the workflow gains a Go-paths filter.
- Absorb the leading-edge O(history) signal recompute before the
  timer in BenchmarkSyncPathsIncrementalAppend (it was ~58% of the
  measured allocs, masking steady-state regressions), stretch the
  debounce window so the flush timer cannot fire nondeterministically
  inside a worst-run-gated loop, and pre-build appended lines.
- Build BenchmarkInsertMessagesBatch's 200-message fixture once
  outside the timed loop; only the SessionID is rewritten per
  iteration.
- Widen testDB to testing.TB, drop the bench-only openBenchDB copy,
  and fix BenchmarkGetDailyUsage's detached testing.T hack (its
  require failures and cleanups were wired to nothing).
- Split benchgate's compare()/main() into evalGate/compareUnits/
  render/parseFlags to meet function-size and complexity limits, and
  drop the dead NaN guard.
- Document the new mechanics in docs/internal/performance-gates.md
  and reformat it with mdformat --wrap 80.

Author: wesm

.github/workflows/bench.yml

@@ -11,19 +11,33 @@ name: Bench Gate
 #   - bulk ingest throughput (#411)
 #   - per-row query-shape regressions in usage aggregation (#309)
 #
-# cmd/benchgate gates on allocs/op and B/op (deterministic on a given
-# machine, tight thresholds) and on ns/op with a loose 2x threshold
-# that only catches algorithmic blowups; both sides run on the same
-# runner within one job, so the comparison is apples to apples.
+# Both sides run `make bench-gate` — the Makefile is the single
+# source of truth for the gated package list, sample count, and
+# iteration count — and cmd/benchgate compares the outputs. It gates
+# on allocs/op and B/op (deterministic on a given machine, tight
+# thresholds) and on ns/op with a loose 2x threshold that only
+# catches algorithmic blowups; both sides run on the same runner
+# within one job, so the comparison is apples to apples.
+#
 # Benchmarks that only exist on one side are reported but never fail
-# the gate — which is what makes running every benchmark in the
-# gated packages safe: a benchmark added by a PR has no baseline and
-# is reported without gating, then gates automatically once merged.
-# BENCH_PACKAGES is the gate boundary; keep it in sync with the
-# Makefile's bench-gate target.
+# the gate: a benchmark added by a PR has no baseline and is reported
+# without gating, then gates automatically once merged. Because each
+# side benchmarks its own Makefile's package list, a PR that adds a
+# package to the gate cannot break the base run. A partially failing
+# base run degrades to a partial baseline (whatever benchmarks it
+# produced still gate) rather than silently disabling the whole gate.
 
 on:
   pull_request:
+    # Docs/frontend-only PRs cannot change the gated Go paths. If
+    # this check is ever made required on branch protection, pair it
+    # with a no-op sibling workflow on the inverse paths.
+    paths:
+      - "**.go"
+      - "go.mod"
+      - "go.sum"
+      - "Makefile"
+      - ".github/workflows/bench.yml"
 
 concurrency:
   group: bench-${{ github.head_ref || github.ref }}
@@ -32,19 +46,6 @@ concurrency:
 permissions:
   contents: read
 
-env:
-  BENCH_PACKAGES: ./internal/sync ./internal/db ./internal/secrets
-  # Must stay >= 5: benchgate's time gate needs at least 5 samples
-  # per side for the significance test and treats fewer as a
-  # configuration error.
-  BENCH_COUNT: "6"
-  # Fixed iterations (not a duration) so the baseline and candidate
-  # run identical workloads: two of the gated benchmarks grow their
-  # fixture as they iterate (appending to a session, adding sessions
-  # to a DB), and a duration-based -benchtime would let a slower side
-  # run fewer iterations against a different final size.
-  BENCH_TIME: 20x
-
 jobs:
   bench-gate:
     name: Benchmark Gate
@@ -59,38 +60,26 @@ jobs:
         with:
           go-version-file: go.mod
 
-      - name: Restore pricing snapshot
-        run: go run ./internal/pricing/cmd/litellm-snapshot -restore
-
       - name: Run benchmarks (PR head)
-        env:
-          CGO_ENABLED: "1"
         run: |
-          set -o pipefail
-          go test -tags "fts5" -run '^$' -bench . \
-            -benchmem -count "$BENCH_COUNT" -benchtime "$BENCH_TIME" \
-            -timeout 25m $BENCH_PACKAGES | tee /tmp/bench-new.txt
+          set -euo pipefail
+          make -s bench-gate | tee /tmp/bench-new.txt
 
       - name: Run benchmarks (merge base)
         env:
-          CGO_ENABLED: "1"
           BASE_REF: ${{ github.base_ref }}
-        # A failing merge-base run (or one with none of the gated
-        # benchmarks yet) degrades to an empty baseline: benchgate
-        # then reports every benchmark as new and passes, so a broken
-        # base never blocks a PR. The baseline file is truncated on
-        # failure so partial output from packages that ran before the
-        # failure cannot gate against incomplete data.
+        # A failing merge-base run keeps whatever benchmark output it
+        # produced: go test emits results per package, so one broken
+        # package (or a base predating the bench-gate target) leaves
+        # a partial or empty baseline and benchgate gates only what
+        # exists on both sides. The warning makes the degraded run
+        # visible instead of a silently green vacuous pass.
         run: |
+          set -euo pipefail
           base=$(git merge-base HEAD "origin/$BASE_REF")
           git worktree add /tmp/bench-base "$base"
-          cd /tmp/bench-base
-          go run ./internal/pricing/cmd/litellm-snapshot -restore || true
-          if ! go test -tags "fts5" -run '^$' -bench . \
-            -benchmem -count "$BENCH_COUNT" -benchtime "$BENCH_TIME" \
-            -timeout 25m $BENCH_PACKAGES > /tmp/bench-old.txt; then
-            echo "merge-base benchmarks failed; comparing against empty baseline"
-            : > /tmp/bench-old.txt
+          if ! make -s -C /tmp/bench-base bench-gate > /tmp/bench-old.txt; then
+            echo "::warning title=Bench Gate::merge-base benchmark run exited non-zero; gating against its partial output"
           fi
 
       - name: Compare against merge base

Makefile

@@ -267,14 +267,15 @@ bench-backends: pricing-snapshot ensure-embed-dir
 		CGO_ENABLED=1 go test -tags "fts5,benchdb" ./internal/backendbench $(BENCH_BACKENDS_FLAGS)
 
 # Hot-path benchmark gate. Runs every benchmark in the gated packages
-# — the same set CI's bench.yml compares against a PR's merge base
 # (sync engine warm/cold/append, message write paths, usage
-# aggregation, secret scanning). Run it before and after touching a
-# sync or DB hot path, then compare with
-# `go run ./cmd/benchgate -old old.txt -new new.txt`. Keep the
-# package list in sync with BENCH_PACKAGES in bench.yml.
+# aggregation, secret scanning). This target is the single source of
+# truth for the gate configuration: CI's bench.yml runs it on both
+# the PR head and the merge base, then compares the outputs with
+# `go run ./cmd/benchgate -old old.txt -new new.txt`. Run it before
+# and after touching a sync or DB hot path.
+BENCH_GATE_PACKAGES ?= ./internal/sync ./internal/db ./internal/secrets
 # Count must stay >= 5: benchgate's time gate needs at least 5
-# samples per side for its significance test.
+# candidate samples for its significance test.
 BENCH_GATE_COUNT ?= 6
 # Fixed iterations, not a duration: some gated benchmarks grow their
 # fixture as they iterate, so baseline and candidate must run the
@@ -284,7 +285,7 @@ bench-gate: pricing-snapshot ensure-embed-dir
 	CGO_ENABLED=1 go test -tags "fts5" -run '^$$' \
 		-bench . -benchmem \
 		-count $(BENCH_GATE_COUNT) -benchtime $(BENCH_GATE_TIME) \
-		-timeout 25m ./internal/sync ./internal/db ./internal/secrets
+		-timeout 25m $(BENCH_GATE_PACKAGES)
 
 # Start test PostgreSQL container
 postgres-up: