fix: verify published benchmark artifacts

.github/workflows/ci.yml

@@ -20,5 +20,6 @@ jobs:
       - run: python -m pip install -e ".[dev]"
       - run: ruff check .
       - run: pytest
+      - run: triton-kernel-artifacts
       - run: python -m compileall -q src tests
       - run: triton-kernel-lab --help

README.md

@@ -1,5 +1,7 @@
 # Triton Kernel Lab
 
+[![CI](https://github.com/WaffleBits/triton-kernel-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/WaffleBits/triton-kernel-lab/actions/workflows/ci.yml)
+
 ![Triton vs torch.compile vs eager p50 latency on RTX 5070 Ti](assets/rmsnorm-swiglu-p50.png)
 
 FP16 RMSNorm and SwiGLU p50 latency on an RTX 5070 Ti, cache-cold, 500 timed samples per case, Triton against torch.compile and PyTorch eager.
@@ -26,6 +28,8 @@ regression gate guards against slowdowns.
 - Performance reasoning that distinguishes a logical bandwidth model from
   hardware-counter evidence.
 - CPU-only CI for parsers, statistics, report contracts, linting, and CLI shape.
+- CPU-only validation that recomputes every committed timing summary and derived
+  roofline case from the raw JSON evidence.
 - A lab-oriented regression gate for controlled GPU runners.
 
 ## Quick Start
@@ -132,6 +136,17 @@ Each JSON report includes:
 - Speedup against both PyTorch baselines and transparent logical
   effective-bandwidth estimates.
 
+Committed artifacts have an additional CPU-only integrity gate. It rejects
+non-finite JSON, sensitive fields, failed correctness records, duplicate cases,
+mismatched raw sample counts, stale percentile/speedup/bandwidth summaries, and
+roofline cases that no longer reproduce from their named source artifacts:
+
+```bash
+triton-kernel-artifacts
+```
+
+This validates the published records; it does not rerun a GPU measurement.
+
 See [docs/METHODOLOGY.md](docs/METHODOLOGY.md) for assumptions and limitations.
 
 ## Development
@@ -142,6 +157,7 @@ CPU-side validation does not install the GPU stack:
 python -m pip install -e ".[dev]"
 ruff check .
 pytest
+triton-kernel-artifacts
 python -m compileall -q src tests
 ```
 

docs/ARTIFACT_VALIDATION.md

@@ -0,0 +1,20 @@
+# Committed artifact validation plan
+
+## Market-backed gap
+
+Inference performance roles repeatedly ask for correctness gates, reproducible measurements, root-causeable telemetry, and explicit model boundaries. This repository publishes raw benchmark and derived roofline JSON, but CI currently checks only Python contracts. It does not prove that committed summaries still match their raw samples or that the roofline report can be reproduced from its named sources. The latest public workflow is also red because the chart generator does not pass the repository lint gate.
+
+## Selected change
+
+Add one CPU-only artifact validation path to the existing kernel lab:
+
+1. Parse every committed JSON artifact without accepting non-finite values or
+   fields that could carry prompts, credentials, private keys, or response bodies.
+2. Validate benchmark identity, environment, methodology, correctness, unique case keys, raw sample counts, recomputed min/p50/p95/p99/max summaries, bandwidth values, and speedup ratios.
+3. Rebuild the committed roofline cases from the named source artifacts and compare the derived values and claim boundary.
+4. Expose the validator as `triton-kernel-artifacts` and run it in CI.
+5. Repair the existing chart-script lint failure without changing the chart or benchmark values.
+
+## Verification boundary
+
+The validator checks internal integrity and reproducibility of already committed RTX 5070 Ti artifacts on a CPU runner. It does not rerun GPU kernels, create new performance measurements, or convert logical-bandwidth projections into hardware-counter evidence.

pyproject.toml

@@ -46,6 +46,7 @@ dev = [
 triton-kernel-lab = "triton_kernel_lab.benchmark:main"
 triton-kernel-roofline = "triton_kernel_lab.roofline:main"
 triton-kernel-nsight = "triton_kernel_lab.nsight:main"
+triton-kernel-artifacts = "triton_kernel_lab.artifacts:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]

scripts/plot_benchmarks.py

@@ -1,9 +1,11 @@
 """Hero chart of Triton vs torch.compile vs eager p50 latency from a committed artifact."""
+
 import json
 import sys
 from pathlib import Path
 
 import matplotlib
+
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
 
@@ -44,16 +46,20 @@ def main(artifact_path: str, out_path: str) -> None:
         vals = [g[1][b]["p50_ms"] for g in groups]
         bars = ax.bar([x + offs for x in xs], vals, width=width,
                       color=COLORS[b], label=LABELS[b])
-        for rect, v in zip(bars, vals):
+        for rect, v in zip(bars, vals, strict=True):
             ax.text(rect.get_x() + rect.get_width() / 2, v, f"{v:.3f}",
                     ha="center", va="bottom", color=FG, fontsize=8)
 
     ax.set_yscale("log")
     ax.set_xticks(xs)
     ax.set_xticklabels([g[0] for g in groups], color=FG, fontsize=9.5)
     ax.set_ylabel("p50 latency (ms, log scale, lower is better)", color=FG, fontsize=11)
-    ax.set_title(f"Triton kernel p50 latency on {gpu}\nFP16, cache-cold, 500 timed samples per case",
-                 color=FG, fontsize=13, pad=12)
+    ax.set_title(
+        f"Triton kernel p50 latency on {gpu}\nFP16, cache-cold, 500 timed samples per case",
+        color=FG,
+        fontsize=13,
+        pad=12,
+    )
     ax.tick_params(colors=FG)
     for spine in ax.spines.values():
         spine.set_color(GRID)