[Feature Request] Add a Large-TopK benchmark case: LAION-100M with top-1M ground truth

[frankleaf]

Summary

I'd like to request support for a large-TopK benchmark scenario: the LAION-100M dataset
with ground truth extended to top-1,000,000 neighbors per query, so that VDBBench can
measure recall/latency/QPS when a single search returns up to 1M results.

Motivation

Large-TopK retrieval (roughly K from 16K up to a few million) is a real production
workload, distinct from the usual K=10/100 setting:

• Batch similarity retrieval — fetch the top 100K–1M most similar items for one query
  vector.
• Data mining / candidate generation — extract large candidate sets for downstream
  filtering, ranking or model training.
• Regression / evaluation corpora — materialize large result sets to build test
  corpora.

Vector databases have started shipping dedicated support for it. Milvus 2.6 has a
collection-level query_mode=large_topk that raises the topK limit from 16,384 to 1,000,000

Today there is no standard, reproducible benchmark for this regime, which makes it hard to
compare systems or to validate optimizations.

Current gap in VDBBench

• Existing performance cases target small K (e.g. k=100), and the shipped
  neighbors.parquet ground truth is correspondingly narrow.
• Recall computation assumes ground truth is at least as wide as the requested K, so a
  k=1_000_000 run cannot be scored against the current GT files.
• As a result, evaluating large-TopK today requires generating custom ground truth
  out-of-band, which is exactly the duplicated effort a shared benchmark should remove.

Proposed scope

A new case (name TBD, e.g. Performance768D100M_TopK1M) based on the existing LAION-100M
base data, plus a wide ground-truth artifact:

1. Wide GT: top-1M neighbor IDs per query. To keep the artifact manageable, a small
   query set is sufficient — large-TopK latency is seconds per query, so a large nq is
   neither necessary nor practical.

   Rough sizing (int64 IDs):

   • 100 queries × 1M × 8 B ≈ 800 MB
   • 1,000 queries × 1M × 8 B ≈ 8 GB

   So something like 50–200 queries seems like the right trade-off; optionally ship
   distances as well (doubles the size), or make GT width configurable
   (e.g. top-100K variant for a lighter run).

2. Multi-level recall reporting: report recall@100 / @1K / @10K / @100K / @1M from a
   single run, since accuracy typically degrades non-uniformly across the result depth.

3. Latency percentiles + throughput: p50/p95/p99 per query, plus steady-state QPS.
   In this regime a single query can take seconds, so a fixed-duration load phase is more
   meaningful than a fixed number of queries.

Questions for maintainers

1. Is a large-TopK case something the project is open to hosting, given the GT artifact
   size? Would a configurable GT width (e.g. 100K default, 1M optional download) be a
   better fit than a single 1M-wide file?
2. Would you prefer this as a new CaseType, or as a parameterized variant of the existing
   Performance768D100M case with a --k / GT-width option?

I'm happy to help with generating the ground truth and contributing the case if there's
interest in the direction.

[xiaofan-luan]

@frankleaf

that's a good suggestion, we are happy to take a new benchmark and dataset.

search top 1m on 100m(might be too small) or 1B dataset would be a good start. Is thre a real dataset on your production?

[frankleaf]

@xiaofan-luan
Thanks for the reply. I'm currently using the Performance768D100M dataset. My workflow is to first create a FLAT index and query the top 1M results to serve as the Ground Truth (GT), and then validate the recall rate of other indexes, such as DISKANN. I don't have access to any larger datasets at the moment.

[xiaofan-luan]

brute force might be too slow to create a ground truth on 100m dataset.

we might release a large topk benchmark later.

[xiaofan-luan]

Following up on my own comment above — I went and did the arithmetic, and I was too pessimistic about the ground-truth cost. Brute-force GT for top-1M on 100M is tractable. Also, while looking at how a large-K run would actually execute, I found two blockers in the current code that will bite before the GT artifact even matters. Writing both down so this becomes actionable.

1. Brute-force GT at 100M × top-1M is affordable

The key is to batch all queries into a single streaming pass over the base data instead of one pass per query.

LAION-100M is 768-dim fp32 = ~307 GB, already sharded as 100 shuffled parquet files of ~1M rows. That chunking is exactly what the algorithm wants:

• Compute is a non-issue. For nq queries the whole job is one nq × 100M × 768 GEMM = 2 × nq × 7.68e10 FLOPs. At nq=100 that is ~15 TFLOP total — seconds of GPU time, tens of seconds even on CPU. This is not where the cost is.
• Selection is the real CPU cost, and it is bounded. Keep a running top-1M (id, dist) per query; for each 1M-row chunk, concatenate with the running best and argpartition back down to 1M. That is ~2M elements per (chunk, query), so 100 chunks × 100 queries × 2M ≈ 2e10 elements — order of 10–20 minutes with numpy, less on GPU.
• The wall clock is dominated by reading 307 GB once, plus parquet decode. On local NVMe this is a single-digit-hours job, and it checkpoints naturally per chunk.

Two practical notes for whoever generates it:

• Don't reach for faiss-gpu IndexFlatL2 here — its GPU top-k is capped (2048/4096), so it can't produce top-1M directly. Either CPU faiss, or a hand-rolled chunked merge with torch.topk / cuvs.
• nq can stay small. Recall@1M is already an average over a million items per query, so per-query variance is low; 50–200 queries is statistically plenty, which is also what keeps the artifact to 800 MB at nq=100 (int64 ids only).

So I'd retract "too slow" — the blocker isn't generating the GT, it's the two items below.

2. calc_recall / calc_ndcg do not terminate at K=1M

This is the hard one, and it is independent of the dataset.

metric.py:97-103:

def calc_recall(count: int, ground_truth: list[int], got: list[int]) -> float:
    recalls = np.zeros(count)
    for i, result in enumerate(got):
        if result in ground_truth:   # <-- linear scan of a Python list
            recalls[i] = 1
    return np.mean(recalls)

ground_truth is a Python list, so result in ground_truth is O(|GT|). The loop is O(|got|). At k=100 that's 1e2 × 1e2 = 1e4 comparisons per query — invisible. At k=1_000_000 it is 1e12 comparisons per query, i.e. hours per query in CPython.

calc_ndcg (metric.py:114-121) has the same shape via ground_truth.index(got_id), and get_ideal_dcg is a Python loop of k iterations.

Both are straightforward to fix and the fix is a win at every K:

• recall → hash the GT once per query (set), or np.isin / sorted np.searchsorted → O(k)
• ndcg → build an id -> rank dict once per query instead of .index() per hit
• get_ideal_dcg → vectorize (np.sum(1/np.log2(np.arange(k)+2)))

Worth doing as a standalone PR regardless of whether the large-K case lands.

3. Ground truth is materialized as Python lists

dataset.py:389-391:

self.test_data = self._read_file(test_file)[self.data.test_vector_field].to_list()
self.gt_data   = self._read_file(gt_file)[self.data.gt_neighbors_field].to_list()

.to_list() turns the GT into a list-of-lists of Python int objects. Those IDs are well above the CPython small-int cache, so each costs a 28-byte PyLongObject plus an 8-byte slot ≈ 36 B. At nq=100 × 1M that's 1e8 ints ≈ 3.6 GB of Python objects for an 800 MB artifact — before the search runner forks.

Keeping the GT as an arrow/numpy array and slicing per query would avoid the blowup entirely, and again helps at any K.

4. On @frankleaf's question 2 (new CaseType vs. parameterized case)

I'd go parameterized, because the k plumbing already exists end to end:

• CaseConfig.k — models.py:194, defaulting to config.K_DEFAULT = 100
• --k CLI flag — cli/cli.py:351-360
• threaded through task_runner.py into both SerialSearchRunner and MultiProcessingSearchRunner

There is no MAX_K and no validation anywhere, so --k 1000000 is already accepted today — it just silently produces a wrong answer, because serial_runner.py:204 does gt[: self.k] and calc_recall divides by self.k regardless of how wide the GT actually is. A GT narrower than k therefore deflates recall with no warning. A guard there (raise or warn when len(gt) < k) is worth adding on its own.

So the missing pieces are really: the wide GT artifact, a GT-width-aware recall guard, and the O(k) metric rewrite — not a new case class.

Suggested scope

1. Metric fixes + GT-width guard (small, independent, useful now).
2. GT as arrow/numpy rather than Python lists.
3. Publish tiered GT for LAION-100M — I'd suggest top-100K as the default download (~80 MB) and top-1M as an opt-in (~800 MB), rather than one 1M-wide file, so the common case stays cheap.
4. Report recall@{100, 1K, 10K, 100K, 1M} from a single run by slicing the one wide GT — this is nearly free once the metrics are O(k), and is more informative than a single number since accuracy degrades unevenly with depth.

Happy to have our side generate and host the GT under assets.zilliz.com/benchmark/ once we agree on the width and nq.

[frankleaf]

@xiaofan-luan
Thanks for your reply! I'm very glad that Zilliz can provide the 1M ground truth data for the LAION-100M dataset. We've also noticed the suggested scope issues with the current benchmark, and I'd be happy to collaborate on optimizing it if needed.

[jamesgao-jpg]

@frankleaf @xiaofan-luan
Hi, thank you for the suggestion! This is definitely a good case to add.
We have taken the liberty of creating a 1m ground truth file for laion 100m datasets, and have uploaded them to vdbbench's benchmark data storage at both Aliyun and AWS. We are implementing and collecting benchmark results on the new largetopk case as we speak at https://github.com/zilliztech/VectorDBBench/tree/LargeTopk, and any contribution is appreciated.

[frankleaf]

@jamesgao-jpg That's great! We hope this scenario can be supported as soon as possible, and we'd be happy to contribute to it.