Skip to content

0037-scientific-multi-db-benchmark-suite.md

Latest commit

 

History

History
207 lines (147 loc) · 11.5 KB

0037-scientific-multi-db-benchmark-suite.md

File metadata and controls

207 lines (147 loc) · 11.5 KB
Raw

ADR-0037 — Scientific multi-DB benchmark suite

Status: Accepted Date: 2026-06-18 Deciders: Achref Soua

Context

Quiver's headline claim is memory-frugal, security-first ANN. That claim is meaningless without a credible, reproducible, head-to-head comparison against the open-source field. The existing bench/ harness measures Quiver alone (recall@10, QPS, latency); it has no competitor support and no automated report.

Phase B of v0.17.0 delivers a paper-grade, multi-DB benchmark suite whose output is a single committed Markdown report. The guiding rule inherited from the rest of the project: every number is real, measured, and labelled with the hardware it ran on; losses are named; reference-hardware-pending markers are used when the dev box is not the right place to produce a headline figure.

What the dev box can and cannot produce honestly

The dev box (WSL2 on shared hardware, co-hosting pipetwin) is not a source of:

  • official QPS / latency headline numbers (virtualized I/O, shared CPU),
  • memory-at-scale numbers at 10M+ vectors (OOM risk).

It can produce, honestly:

  • smoke-run validation at SIFTSMALL (10k vectors) to verify every competitor adapter end-to-end,
  • recall correctness at any scale (recall is a property of the index, not the host),
  • relative RSS at SIFT1M scale when all competitors run sequentially on idle-ish hardware (clearly labelled as "dev-box, resource-shared").

Everything at 10M+ or needing production-grade QPS numbers carries a [reference-hardware-pending] marker. The reference-hardware runbook (docs/benchmarks/reference-hardware-runbook.md) is updated with the multi-DB procedure so the owner can produce the headline numbers on dedicated hardware.

Decision

1. Extend bench/, do not fork

The existing Python project in bench/ is already isolated from the Cargo workspace and the Rust deny/audit gate. We add Python competitor adapters as modules inside bench/quiver_bench/competitors/. Competitor Python clients (qdrant-client, lancedb, chromadb, pymilvus, etc.) live in a new competitors optional dependency group in bench/pyproject.toml. They never touch Cargo.toml, deny.toml, or cargo audit.

2. Competitor set and pinned versions

Competitor Mode Reason for inclusion
Qdrant v1.13.4 Docker The most direct competitor; HNSW + HNSW-on-disk
LanceDB 0.33.0 Python library (no Docker) Fast Lance disk-resident IVF_PQ; strong on memory
Chroma 1.5.9 Python embedded Fast chromadb embedded mode; popular for RAG use-cases
Milvus Lite 3.0.0 Python library (no Docker) Milvus engine without the cluster machinery
pgvector 0.7 on PG16 Docker SQL-native ANN; baseline for the SQL world
FAISS 1.14.3 Python library (no Docker) The canonical library baseline; flat/IVF/HNSW implementations
Weaviate 1.27.0 Docker Alternative HNSW store with rich schema

Competitors run on datasets where they support the required metric (L2 or cosine). Competitors that do not support a dataset's metric are skipped and noted in the report.

3. Datasets (phased)

Dataset dim N Metric Where used
siftsmall 128 10k L2 Smoke-run adapter validation (dev box)
SIFT1M 128 1M L2 Full in-memory comparison (dev box, labelled)
GloVe-100 100 ~1.2M cosine Cosine path [reference-hardware-pending]
Deep10M 96 10M L2 Disk-path memory headline [reference-hardware-pending]

SIFTSMALL is already in bench/datasets/ (generated by the prior harness). SIFT1M is the existing bench/datasets/sift directory. GloVe and Deep10M require download; bench/quiver_bench/datasets.py gains download helpers with SHA-256 verification.

4. Metrics

For every (competitor, dataset, param-sweep-point):

  • Recall@10 against the dataset's exact ground truth.
  • QPS (single-thread) — sequential queries, no concurrency.
  • Latency p50 / p95 / p99 (ms).
  • RSS at steady state (MB) — sampled after index load + warmup, before the query run; see §5.
  • Build time (s) — time to build / load the index from base vectors.
  • On-disk index size (MB) — total data written by the competitor (not the base vectors).

The headline comparison is recall@10 vs RSS at a common operating point (recall@10 ≥ 0.95). QPS and latency are secondary figures.

5. RSS measurement

RSS is sampled identically for all competitors at the same lifecycle point (after index load, after 10-query warmup, before the timed query run):

  • Native process / Python library (LanceDB, Chroma, FAISS, Milvus Lite): /proc/<pid>/status → VmRSS on Linux; psutil.Process().memory_info().rss as a cross-platform fallback.
  • Docker container (Qdrant, pgvector, Weaviate): docker stats --no-stream --format json <container> → MemUsage field.

Both paths are collected by bench/quiver_bench/rss.py.

6. Rigor protocol

  • Fixed RNG seed (42) for any random data; datasets use their own fixed ground truth.
  • Warmup: 10 queries discarded before the timed run.
  • Repetitions: ≥ 3 complete query-set runs; report mean ± stdev for recall, QPS, latency.
  • Sequential competitor runs: only one competitor runs at a time (no concurrent containers).
  • Drop OS page cache between competitors (Linux: echo 3 > /proc/sys/vm/drop_caches if privileged; note if skipped).
  • Each competitor gets the same upserted vector set, with IDs coerced to the competitor's required format.
  • Competitor configs follow each system's own recommended defaults for the dataset scale; the exact config is embedded in the result JSON.

7. Output layout

docs/benchmarks/results/
  comparison-v0.17.0/
    manifest.json          # hardware + OS + versions + methodology hash
    siftsmall-smoke.json   # per-competitor raw results (smoke validation)
    sift1m/
      quiver.csv
      qdrant.csv
      lancedb.csv
      chroma.csv
      milvus_lite.csv
      pgvector.csv
      faiss.csv
      weaviate.csv
    comparison-v0.17.0.md  # the auto-generated report (committed)

8. just bench-compare and just bench-report

Two new recipes:

bench-compare *ARGS:
    uv run --project bench python -m quiver_bench.comparison {{ ARGS }}

bench-report:
    uv run --project bench python -m quiver_bench.report \
        docs/benchmarks/results/comparison-v0.17.0

comparison accepts --dataset siftsmall|sift1m|glove|deep10m, --competitors all|qdrant,lancedb,..., --smoke (alias for siftsmall), and --out DIR.

report reads the result directory and generates comparison-v0.17.0.md.

9. Report format

The auto-generated comparison-v0.17.0.md contains:

  1. Hardware manifest block (CPU, RAM, OS, kernel, versions).
  2. One sentence on the methodology link.
  3. For each dataset run:
    • A table at the recall@10 ≥ 0.95 operating point: DB | recall@10 | QPS(1T) | RSS(MB) | Build(s) | Index(MB) | notes.
    • A wins / ties / losses matrix (Quiver vs each competitor, per metric).
  4. Honest conclusions: wins highlighted, losses or ties stated plainly, every number tagged with the hardware it ran on.
  5. [reference-hardware-pending] for anything not produced at full scale on documented reference hardware.

Static plots are omitted from the committed report (they require matplotlib / graphing deps) — the raw CSVs are the artifact; a --plot flag can generate them locally.

Consequences

Good:

  • A committed, reproducible comparison report answers the credibility question before v1.0.0.
  • Extending bench/ (not forking) keeps the harness DRY and the commit surface small.
  • Docker isolation means competitor versions are pinned and reproducible.
  • Honest reporting — including losses — is more credible than cherry-picked wins.

Bad / accepted costs:

  • Docker startup adds latency to the comparison run (Qdrant, pgvector, Weaviate).
  • On WSL2, drop_caches requires root and may not be available — noted in the report.
  • Some competitors (Milvus full cluster, Weaviate at scale) are heavier than they appear at 10k vectors; the harness caps dataset size per competitor to avoid OOM on the dev box.
  • The report at SIFT1M is labelled "dev box, resource-shared" and not cited as an official headline until reproduced on documented reference hardware.

Alternatives considered

Fork a new Python project — rejected; would duplicate dataset loading, metrics, and the quiver-client dependency with no benefit.

Use ann-benchmarks Docker directly — too inflexible for our Quiver REST-over-HTTP adapter and our security (key-per-collection) surface. The ann-benchmarks harness also bundles a fixed version of every competitor; we prefer to control the versions.

Measure all competitors in one long CI run — rejected; this box is resource-shared and CI is manual-only (ADR-0015). The harness is invoked locally or on reference hardware via just bench-compare.

Include Redis/Vespa — deferred; both require significant infrastructure setup that competes with the OOM budget on this box. They are noted as stretch goals in the runbook.

Implementation (PR #163, merged 2026-06-18)

Shipped on develop via squash-merge of feat/benchmark-comparison-suite.

Files added:

File Purpose
bench/quiver_bench/competitors/base.py CompetitorAdapter ABC + BenchResult dataclass; query_sweep() with 3-rep mean±stdev
bench/quiver_bench/competitors/{faiss,lancedb,chroma,milvus_lite,qdrant,pgvector,weaviate,quiver}_adapter.py 8 concrete adapters
bench/quiver_bench/comparison.py CLI runner: sequential multi-competitor, emits CSV per competitor
bench/quiver_bench/report.py Report generator: reads CSVs → comparison-v0.17.0.md
bench/quiver_bench/rss.py RSS via /proc/VmRSS (native) / docker stats (containers)
bench/quiver_bench/manifest.py Hardware provenance capture
bench/tests/test_comparison.py 18 unit tests

Deviations from the design:

  • The output layout uses per-competitor *.csv files under smoke/ and sift1m/ subdirectories rather than a siftsmall-smoke.json. CSV is friendlier for diff and grep.
  • Quiver SIFT1M was not included in the comparison CSVs: REST API upload takes ~12 minutes on the dev box, making a fair sequential comparison with in-process competitors impossible. Real Quiver SIFT1M numbers (single-DB harness, same methodology) are in docs/benchmarks/results/sift1m.md and referenced from the report.
  • LanceDB SIFT1M recall (0.52–0.56) is the honest measurement under default 256-partition IVF_PQ; the report notes this is configurable but doesn't tune it — that would be cherry-picking.
  • Qdrant, pgvector, and Weaviate SIFT1M runs deferred to reference hardware (Docker API overhead at 1M-vector scale on WSL2 is not representative).

Verification (2026-06-18, WSL2 dev box)

Check Result
bench/tests/test_comparison.py (18 tests) ✅ all pass
Smoke run — all 8 competitors on siftsmall (10k, 128-d, L2) ✅ complete; CSVs in smoke/
SIFT1M — FAISS (HNSW M=16, efConstruction=200) ✅ ef_search 16→256; recall 0.81→1.00; peak 7685 QPS [dev-box]
SIFT1M — LanceDB (IVF_PQ, 256 partitions) ✅ nprobes 4→64; recall 0.52–0.56 (config-limited, noted) [dev-box]
comparison-v0.17.0.md generated ✅ at docs/benchmarks/results/comparison-v0.17.0/
No fabricated numbers ✅ confirmed; every figure is a real measurement
[reference-hardware-pending] on all non-smoke results ✅
just verify (Rust gate) ✅ green (no Cargo changes in this PR)