Skip to content

EVAL-TRUST.md

Latest commit

 

History

History
111 lines (89 loc) · 6.94 KB

EVAL-TRUST.md

File metadata and controls

111 lines (89 loc) · 6.94 KB
Raw

Eval trust & verifiability

How sparkinfer's SN74 evaluation is made trustworthy — what you can verify today, what's on the roadmap, and the honest boundaries. We'd rather state this plainly than over-claim.

TL;DR

  • Today: every result is reproducible from source — anyone can rebuild main and the PR on an RTX 5090 and recover the same same-box delta. The frontier has already been independently reproduced by a community member on a rented 5090.
  • Roadmap: attested, multi-source eval — the deterministic scoring runs in a CPU TEE (Intel TDX) that returns a hardware-signed receipt; run logs are committed immutably; and independent validators re-measure for consensus.
  • Honest boundary: a consumer RTX 5090 has no GPU Confidential Computing, so the speed number itself can't be sealed in a GPU enclave. We make it trustworthy through reproduction + consensus, not a cryptographic proof of tok/s — and we keep it that way on purpose (below).

What the evaluator does

For each PR commit, on one RTX 5090, in one run: 0. Verify the baseline reference before anything is scored: the Q4_K_M weights are checked against a pinned sha256 and re-fetched if they don't match, and llama.cpp is rebuilt from a pinned commit with a clean tree — so a tampered persisted copy on a reused box can't skew the verdict. (bench/scripts/reference.lock.)

  1. Build main and the PR from source on the same box.
  2. Warm up the GPU and pin/record the graphics clock — pinned via nvidia-smi -lgc where the box permits (bare-metal/datacenter), otherwise the observed clock (and its spread) is recorded with the result, so the absolute tok/s is reproducible and clock-checkable, not just same-box-cancelled.
  3. Bench decode for both, interleaved, and score the same-box delta % — so box-to-box hardware variance (~2%) cancels and the score is hardware-independent.
  4. Gate correctness on a held-out prompt: top-1 token agreement ≥ 0.90 and KL ≤ 0.20 (preferred ≤ 0.15) vs llama.cpp on the same GGUF — strict bars that hold even on hard held-out text. The prompt is chosen by a fresh, unpredictable per-eval seed (a random window of a multi-domain corpus + fuzzed length), so a submission can't overfit a fixed in-repo prompt; the seed is logged so the exact prompt is reproducible. The KL is measured at matched top-k depth (sparkinfer dumps a deep top-k so llama's tail isn't floored) — the true divergence is ~0.01–0.03 (top-1 0.96–0.98), so the strict 0.20 holds with large margin. A speedup that erodes accuracy is REJECTed regardless of how fast it is — accuracy is the moat.
  5. Label = a deterministic function of the measurements (XS … XL, none, BASELINE, REJECT), so independent validators converge on the same verdict. The verdict carries its provenance (clock, prompt seed, reference pins) so the immutable log is self-describing.

Every frontier advance is also appended to an immutable, GitHub-timestamped frontier ledger (ledger.jsonl in the eval-log repo): (date, PR, author, commit, Δ%, prev→new frontier, proof) — append-only, so the frontier history is auditable line-by-line against the per-run logs.

The bot runs the same code you can: eval/ (pr_eval_bot.py, vast_eval.py), with the scoring math in bench/scripts/label.py.

Reproduce it yourself

# on any RTX 5090 (sm_120, CUDA 12.8+)
git clone https://github.com/gittensor-ai-lab/sparkinfer && cd sparkinfer
bench/scripts/bench.sh --download            # sparkinfer decode tok/s
bench/scripts/bench.sh --download --compare  # head-to-head vs llama.cpp (same GGUF)
bench/scripts/accuracy.sh --download         # top-1 / KL / perplexity vs llama.cpp

A note on the head-to-head: build llama.cpp with full CUDA optimization — an under-tuned llama build will understate its tok/s and overstate our lead. Our published figures (+24.0% at 128-tok, +21.6% at 256, +17.6% at 512) are against a fully-built llama.cpp (CUDA, sm_120), measured same-box, warm, and interleaved; please compare apples to apples.

Deterministic vs non-deterministic (why the trust model is split)

Correctness (top-1 / KL) Speed (tok/s)
nature deterministic — fixed model + inputs + greedy ⇒ exact non-deterministic — clocks/thermal/box vary
verify by re-running → identical numbers (or a TEE proof) reproduction + consensus within tolerance

There is no cryptographic proof of a benchmark number — only agreement among independent measurers. We design for that honestly.

Roadmap to attested eval

  1. Reproducible from source + published protocol ← today.
  2. Immutable public run log — each eval's full log + result committed to a public eval-log repo (GitHub-timestamped), one verifiable page per run.
  3. Attested scoring — run label.py + the accuracy gate inside Intel TDX (e.g. via a CPU-TEE provider); the receipt binds the exact code + commit + inputs + verdict, checkable offline. Kills "did they edit the harness or the numbers."
  4. Multi-validator consensus — independent validators re-measure the same-box delta %; stake-weighted trimmed median, commit-reveal. Removes the single scorer.

Why RTX 5090 ≠ Confidential Computing — and why we keep it that way

GPU Confidential Computing (encrypted-VRAM TEE + attestation) exists only on datacenter parts (H100/H200, B200/GB200), not consumer RTX. So we could chase cryptographic GPU attestation by moving to datacenter GPUs — but we won't, for three reasons:

  1. It's our whole point. sparkinfer optimizes the consumer/edge Blackwell GPUs people actually own (RTX 5090, RTX PRO 6000, RTX Spark, Jetson Thor) — the gap the datacenter engines leave. Optimizing H100/B200 means a different product in a different market.
  2. The eval must run on the target hardware. Measuring H100 speed isn't measuring 5090 speed.
  3. Datacenter TEE wouldn't even prove the real number. CC mode adds overhead and changes the performance it's trying to attest — you'd get a hardware-signed receipt of a CC-degraded speed, not the production one.

So the right design for our hardware is reproduction + consensus + attested (CPU) scoring — not a GPU enclave. That's a more honest claim, and it matches the machines our users run.

The honest claim

Every result is reproducible from source, the scoring is deterministic and (on the roadmap) hardware-attested, and the speed is corroborated by independent reproduction + consensus — a stake-backed, stabilized, openly-checkable protocol. Not a cryptographic proof of GPU tok/s (no consumer GPU can give one) — and we'd rather say so.

Found a hole in this? Open an issue — adversarial review of the eval is exactly what makes it trustworthy.