wiki-dpr-disk-rescore-benchmark

Benchmark harness and published results for disk-rescore vector search on kenhktsui/wiki_dpr_e5: 21M passages, 768-dimensional e5-base-v2 embeddings, recall@100 over 10,000 queries.

Configuration: TurboQuant 4-bit, originals on disk, async_scorer enabled, two rescore patterns swept across hnsw_ef at oversampling=1.

Published results (~0.96 recall@100, Pattern B, closed-loop concurrency 4)

Cluster	Recall@100	QPS	Avg latency	P99
3 × 2 vCPU / 8 GB per node, RF=1	0.9596	67.2	59.5 ms	86.0 ms
3 × 4 vCPU / 16 GB per node, RF=1	0.9596	111.9	35.7 ms	83.1 ms

Full sweeps (Patterns A and B) are in results/published/.

Quick start

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

export QDRANT_URL=https://your-host:6333
export QDRANT_API_KEY=your-key

# queries.parquet at repo root — see docs/REPRODUCE.md
python3 queryset.py queries.parquet
python3 dryrun.py
scripts/run_profile.sh qdrant_2vcpu8gb   # or qdrant_4vcpu16gb

What's in the repo

Path	Purpose
benchmark.py, runengine.py, queryset.py	Benchmark harness (Pattern A + B, ef sweep)
dryrun.py	Sanity check on 20 queries before a full run
scripts/load_data.py	Load kenhktsui/wiki_dpr_e5 (21M × 768d)
scripts/run_profile.sh	Run a named profile from configs/runs/
configs/qdrant_config.yaml	Self-hosted async_scorer toggle
configs/node_specs.json	Reference node sizing for published runs
configs/runs/qdrant_2vcpu8gb.env	Small-node profile
configs/runs/qdrant_4vcpu16gb.env	Large-node profile
results/published/qdrant_2vcpu8gb.csv	Published results — 3 × 2 vCPU / 8 GB
results/published/qdrant_4vcpu16gb.csv	Published results — 3 × 4 vCPU / 16 GB
docs/REPRODUCE.md	Full reproduction steps

Rescore patterns

• Pattern A — single-stage rescore: quantized HNSW pre-selects, engine rescores against on-disk originals in one call
• Pattern B — two-stage rescore: prefetch an oversampled candidate set from in-RAM quantized vectors, then rescore a bounded set against on-disk originals

Both patterns are swept in every run. Headline numbers above use Pattern B at hnsw_ef=50, oversampling=1.

Not included

• queries.parquet (~56 MB) — gitignored; see docs/REPRODUCE.md
• API keys or cluster URLs

License

Benchmark code: use and adapt with attribution. Dataset kenhktsui/wiki_dpr_e5 is subject to its Hugging Face license.