Skip to content

0041-deep-benchmark.md

Latest commit

 

History

History
87 lines (68 loc) · 5.12 KB

0041-deep-benchmark.md

File metadata and controls

87 lines (68 loc) · 5.12 KB
Raw

ADR-0041 — Deep, large-data benchmark dimensions

Status: Accepted Date: 2026-06-19 Deciders: Achref Soua

Context

ADR-0037 established the scientific multi-DB benchmark suite (eight competitor adapters; recall@k, QPS, latency percentiles, build time, on-disk size, steady-state RSS; a reproducibility manifest). To date only a full multi-DB smoke has run (SIFTSMALL, 10k×128) plus a partial SIFT1M; that is not enough to understand Quiver's position against the field on realistic data. The state-of-Quiver assessment (docs/analysis/state-of-quiver-v0.17.md) records the coverage gaps: larger datasets, filtered search, concurrent throughput, churn/incremental workloads, memory-under-load, restart/cold-open timing, denser recall↔QPS Pareto curves, and quantization tradeoff curves.

The benchmark environment is already provisioned: the bench/ uv venv has FAISS, LanceDB, Chroma, Milvus Lite, Qdrant, Weaviate, and the Quiver SDK; Docker 29.4.3 is available; SIFT1M is extracted at bench/datasets/sift/. The constraint is honesty under the PRIME RULE and the risk register: this is a resource-shared WSL2 box (i7-12700H, 20 logical cores, 15.5 GiB RAM, ext4-on-VHDX). Per R6, comparisons on identical hardware are a fair, publishable result; per R5, absolute headline RSS and the 10M disk path are distorted by a VM and stay reference-hardware-pending.

Decision

Extend the suite (not replace it) with deep dimensions, and run locally what is honest to run now.

Datasets

  • SIFT1M (1M×128, L2) — run now; data already local.
  • GIST1M (1M×960, L2) — add a checksummed downloader (datasets._load_dataset already accepts a .fvecs directory); run if the box has headroom, else mark pending.
  • Deep1M / Deep10M (96-d, L2) — Deep1M run if feasible; Deep10M stays [reference-hardware-pending] (the memory-frugality headline the VM distorts).

New measurement dimensions

  1. Concurrent throughput (multi-thread QPS). The methodology already names QPS(NT); add a harness-level concurrent driver (--concurrency N) that runs query_one from a thread pool and records saturated QPS alongside single-thread QPS. Adapter-agnostic; each adapter's client must be usable from multiple threads (documented per adapter).
  2. Denser recall↔QPS Pareto curve. Already supported by widening the --ef sweep; standardise a dense default for the published curves.
  3. Quantization tradeoff curve (Quiver, single-DB). Recall@10 vs RSS across scalar / product / binary quantization and the disk-resident path — the memory-frugality wedge. Lives in the single-DB run.py path (no competitor needed); feeds quantization-tradeoffs.md.
  4. Build + memory at scale. build_s and steady-state rss_mb are already captured; publish them at SIFT1M for every runnable competitor, with the absolute-RSS caveat.

Deferred to a follow-up (stated, not faked)

  • Filtered-search selectivity sweep and churn / incremental insert-delete workloads require per-adapter insert / delete / filtered-query support that the competitors implement differently; they are scoped as a follow-up ADR once the larger-data comparison lands, rather than half-done here.
  • Restart / cold-open timing is Quiver-specific and folded into the single-DB path opportunistically.

Honesty rules (binding)

  • Every result carries the manifest.json machine spec (captured automatically). Comparative head-to-head numbers run on the identical box are published as real results. Only absolute RSS and the 10M disk path are labelled indicative — reference-hardware-pending.
  • Publish wins and losses. Pin competitor versions and configs. No fabricated or extrapolated number; a run that OOMs or cannot complete is reported as such, not estimated.
  • Output goes to docs/benchmarks/results/comparison-v0.18.0/; the report is regenerated by just bench-report. The methodology and reference-hardware runbook gain the new dimensions for the eventual reference run.

Consequences

  • A genuine large-data picture of Quiver vs the field (recall, QPS single+concurrent, build, memory) on SIFT1M, with the memory wedge shown via the quantization curve — answering "where does Quiver stand" with real, reproducible numbers.
  • The harness gains a reusable concurrency driver and dataset downloaders.
  • Filtered/churn coverage is explicitly deferred (not silently missing), keeping this change bounded.
  • No CI change: heavy runs stay manual; the smoke dataset remains the regression gate (ADR-0037).

Alternatives considered

  • Run the full filtered + churn matrix across all eight competitors now. Rejected for this round: competitors expose filtering and incremental updates too differently to compare fairly without significant per-adapter work; doing it half-way would produce misleading numbers. Deferred to a dedicated follow-up.
  • Publish absolute RSS / 10M as headline numbers from this box. Rejected: a VM distorts exactly those metrics (R5); publishing them as official would misrepresent — under- or over-stating — Quiver's memory wedge. They stay reference-hardware-pending.