feat: add cuVS vector search backend

[yuanqingz]

English | 中文

Summary

• add an opt-in cuvs VectorDB backend while reusing OpenViking's local storage, scalar/path indexes, sparse retrieval, and recovery paths
• support cuVS brute-force exact search and CAGRA ANN with correct bitset prefiltering
• keep the default CPU behavior unchanged: cuVS is optional and lazily imported; explicit backend=cuvs is fail-fast
• add an opt-in, memory-aware auto_cuvs path under backend=local, with per-query latency routing for selective filters
• keep a host-side shadow and lazily rebuild the GPU index after upsert, delete, or restart
• add English/Chinese documentation, a smoke example, and reusable index/collection benchmark harnesses

Non-invasive data-type boundary

Enabling cuVS does not change the default backend, migrate records, or rewrite native CPU index metadata. Normal collection/service paths keep OpenViking's current per-vector-scale int8 native index, while cuVS maintains an independent float32 GPU shadow. Native fallback therefore retains the existing CPU behavior.

The index-only exact results below compare FP32 CPU with FP32 GPU. Collection-level results preserve application defaults and therefore compare native int8 with cuVS FP32; they are not equal-dtype or equal-memory claims. Recall@K and the selected execution route must be reported with those results. Auto mode can select either representation per query; applications requiring one fixed numerical representation can use an explicit backend or set both native-routing thresholds to zero.

Native bitmap bridge and filtered-query routing

The original prototype evaluated a new filter against every host-side Python record. This PR now reuses OpenViking's existing scalar/path indexes:

1. after each GPU rebuild, cuVS registers its label order with the native engine once;
2. the native filter parser and scalar/path indexes evaluate the DSL and produce their existing bitmap;
3. a small ABI bridge projects native logical offsets into the cuVS row order and returns packed uint32 words;
4. cuVS copies those packed words to the device and caches the prepared bitset;
5. upsert/delete invalidates the registered layout and filter cache together.

This preserves native semantics for and, or, must, must_not, contains, ranges, URI depth/prefix filters, date_time, and geo_point, instead of maintaining a second Python implementation. Sparse/hybrid search and operations outside dense top-k remain native.

In auto mode, the eligible count also drives request-level routing:

• ordinary scalar filters with at most auto_filter_native_threshold=2000 candidates use native vector recall;
• URI/path filters use the more conservative auto_path_filter_native_threshold=200 because Trie traversal and subtree-bitmap union can dominate wider path scopes;
• either threshold can be set to zero to disable that route;
• explicit backend=cuvs continues to use cuVS for supported dense queries.

Mutation and GPU search share a cross-backend lock so a native bitmap cannot be paired with a stale GPU row layout. Native fallback executes outside that lock and keeps the native engine's existing read concurrency.

Memory-aware auto mode and VRAM

With backend=local and cuvs.auto_enable=true, OpenViking checks free device memory before every lazy build/rebuild. Admission estimates the float32 vector payload, CAGRA retained/intermediate graphs, and configured filter-bitset cache; applies a default 2.0 safety factor; and preserves 1 GiB. If the budget does not fit, cuVS is unavailable, or an admitted allocation still fails, that query uses the unchanged native index. The dirty GPU index is retried by later queries, so it can activate after memory becomes available.

The brute-force vector payload is approximately N * dimension * 4 bytes. CAGRA additionally retains about N * graph_degree * 4 graph bytes and may need N * intermediate_graph_degree * 4 during build. A cached bitset costs about N / 8 bytes. Prior five-process runs recorded these retained build deltas:

Dataset	cuVS algorithm	GPU memory delta
100K x 768D	brute-force	294 MiB
1M x 768D	brute-force	2.9 GiB
100K x 1024D	brute-force	392 MiB
1M x 1024D	brute-force	3.9 GiB
1,183,514 x 100D	brute-force	452 MiB
1,183,514 x 100D	CAGRA	872 MiB

These are retained deltas, not sampled peaks, and exclude the approximately 327 MiB CUDA runtime/context baseline observed before build. Allocator state, parameters, query batch size, and concurrent GPU work can raise peak VRAM, so the auto gate is intentionally conservative.

Preliminary vector-index results

The public index benchmark uses ann-benchmarks glove-100-angular: 1,183,514 x 100D base vectors, 10,000 queries, K=10, cosine ranking, and one NVIDIA H20. Each value is the median +/- median absolute deviation (MAD) from five independent processes. "GPU exact" means cuVS brute-force; CAGRA is approximate.

Batch size 1, after ten warm-up batches:

Backend	Recall@10	p50 (ms/query)	QPS
OpenViking native CPU exact, FP32	1.0000 +/- 0.0000	40.209 +/- 0.071	24.7 +/- 0.1
cuVS GPU brute-force exact, FP32	1.0000 +/- 0.0000	0.796 +/- 0.005	1,254.5 +/- 7.1
cuVS CAGRA, itopk_size=512	0.9633 +/- 0.0003	1.730 +/- 0.016	562.8 +/- 1.7
cuVS CAGRA, itopk_size=2048	0.9944 +/- 0.0002	1.797 +/- 0.019	549.0 +/- 0.9

For this single-query shape, GPU brute-force exact delivered a 50.5x warm-p50 speedup and 50.7x higher QPS than the current FP32 native exact path. First-search medians were 85.7 ms native and 104.0 ms GPU exact, so cold start remains separate from warm search.

At batch size 128, CAGRA reached 43,595 +/- 280 QPS (1.23x GPU exact) at Recall@10=0.9628. At Recall@10=0.9943 it reached 21,711 +/- 79 QPS (0.61x GPU exact). CAGRA therefore only wins this initial throughput point when accepting lower recall.

For deterministic normalized Gaussian vectors, warm batch=1 exact search showed:

Dim	Vectors	Native p50 (ms)	GPU exact p50 (ms)	Speedup
768	100K	21.991 +/- 0.177	0.341 +/- 0.003	64.5x
768	1M	229.728 +/- 0.580	1.435 +/- 0.002	160.1x
1024	100K	28.443 +/- 0.262	0.376 +/- 0.001	75.7x
1024	1M	306.730 +/- 0.838	1.708 +/- 0.002	179.6x

The observed warm crossover was between 2K and 5K vectors at 768D and between 1K and 2K at 1024D. These boundaries are hardware- and workload-specific, not universal thresholds.

Preliminary collection-level results

The collection benchmark calls CollectionAdapter.query() and includes filter handling, label-to-record lookup, result normalization, persistence, mutation, and lazy rebuild. It uses 100K x 768D normalized synthetic vectors, 50 queries per scenario, K=10, and five independent processes.

The first prepared-bitset version still spent 119--141 ms scanning Python records for a new scalar filter. The native bitmap bridge reduces the corresponding first-use latency to 1.38--2.98 ms, a paired 47--86x reduction. Auto-mode medians are:

Scenario	First query (ms)	Warm p50 (ms)	Route
Unfiltered	1,902.930 +/- 6.045	0.972 +/- 0.015	cuVS; first query includes lazy GPU build
Uniform scalar 10%	2.984 +/- 0.034	1.119 +/- 0.029	cuVS
Uniform scalar 1%	2.096 +/- 0.019	0.643 +/- 0.018	native
Uniform scalar 0.1%	1.486 +/- 0.014	0.404 +/- 0.009	native
URI path 10%	12.860 +/- 0.569	1.126 +/- 0.024	cuVS
URI path 1%	3.087 +/- 0.031	1.020 +/- 0.027	cuVS
URI path 0.1%	1.635 +/- 0.014	0.455 +/- 0.010	native

A dirty-index selective-first follow-up on clean revision d2a74c1 ran a 0.1% scalar filter before any GPU build. Across five processes it took 11.670 +/- 0.156 ms with a 0 +/- 0 byte GPU-memory delta; the subsequent unfiltered first query still took 1,889.281 +/- 11.155 ms. This confirms native routing now happens before GPU admission/rebuild and leaves the GPU index dirty for a later GPU-routed query.

The path first-use cost depends on subtree width: the 10% URI prefix spends about 12.9 ms traversing and unioning native path bitmaps, but repeated searches reuse the device bitset. The 1% path remains on GPU because repeatedly rebuilding its native path bitmap was slower than cached cuVS search; only the 0.1% path crosses the lower path threshold. Clustered scalar results follow the same routing pattern and are included in the full report.

These collection rows preserve normal defaults: native uses int8 and cuVS uses float32. In the earlier explicit-backend comparison, native Recall@10 against cuVS brute-force was 0.982 unfiltered and 0.978--0.994 with filters. The auto table above is a latency-routing result, not an equal-dtype quality comparison.

Validation

• 70 targeted VectorDB unit/integration tests pass
• 9 collection benchmark and cross-process summary tests pass
• 7 stable-ABI loader tests pass
• the standalone C++ engine test covers projection into an arbitrary external label order
• real-GPU brute-force and CAGRA smoke tests pass, including combined scalar + URI filtering, update, delete, and rebuild
• Python lint, C++ syntax checks, and git diff --check pass
• the staged and committed diffs were scanned for credentials, private keys, and non-public environment identifiers before push

WIP / follow-ups

• evaluate a read-safe GPU snapshot with concurrent CUDA streams or micro-batching
• evaluate background/coalesced rebuild and atomic index swap for write-active collections
• preserve the non-invasive dtype boundary and evaluate configurable cuVS float16 first
• evaluate CAGRA int8/VPQ on an explicit recall/latency/memory frontier
• treat native-compatible scaled-int8 as separate scale-aware distance/reranking work, not a cast
• add full HTTP/embedding tests and a real agent-memory workload
• repeat selected performance points across hardware and software versions

Integration plan: docs/design/openviking-cuvs-integration-plan.md.

Full methodology and all preliminary tables: benchmark/cuvs/PRELIMINARY_RESULTS.md.

---

中文版

概要

• 新增可选的 cuvs VectorDB 后端，同时复用 OpenViking 现有的本地存储、标量/路径索引、稀疏检索和恢复路径
• 支持 cuVS brute-force 精确检索和 CAGRA ANN，并使用正确的 bitset 前置过滤
• 默认 CPU 行为保持不变：cuVS 是可选依赖并延迟导入；显式 backend=cuvs 保持 fail-fast
• 在 backend=local 下增加显式开启的显存感知 auto_cuvs，并对高选择性过滤做请求级延迟路由
• host 端保留向量 shadow，在 upsert、delete 或重启后延迟重建 GPU index
• 增加中英文文档、smoke 示例和可复用的 index/collection benchmark 工具

非侵入的数据类型边界

启用 cuVS 不会改变默认后端，不迁移记录，也不重写 native CPU index metadata。正常 collection/service 路径继续使用 OpenViking 当前的逐向量 scale int8 native index，cuVS 独立维护 float32 GPU shadow；native fallback 因此保持现有 CPU 行为。

下面 index-only exact 数据是 FP32 CPU 与 FP32 GPU 的对比。collection-level 数据保留应用默认配置，因此是 native int8 与 cuVS FP32 的对比，不代表等 dtype 或等内存；解读时必须同时报告 Recall@K 和实际执行路径。auto 模式可能按查询选择任一表示；要求固定数值表示的应用可以使用显式 backend，或将两个 native 路由阈值都设为 0。

Native bitmap bridge 与 filtered-query 路由

初始原型会让每个新 filter 扫描所有 host-side Python records。本 PR 现在直接复用 OpenViking 的 scalar/path index：

1. 每次 GPU rebuild 后，只向 native engine 注册一次 cuVS label 顺序；
2. native filter parser 和 scalar/path index 按原有语义计算 bitmap；
3. 小型 ABI bridge 将 native logical offset 投影到 cuVS row 顺序，返回打包后的 uint32 words；
4. cuVS 将这些 words 复制到 GPU，并缓存 prepared bitset；
5. upsert/delete 同时使 layout 和 filter cache 失效。

因此 and、or、must、must_not、contains、range、URI depth/prefix、date_time、geo_point 都继承 native 过滤语义，不再维护第二套 Python evaluator。sparse/hybrid 以及 dense top-k 之外的能力仍走 native。

auto 模式还使用 eligible count 做请求级路由：

• 普通 scalar filter 的候选数不超过 auto_filter_native_threshold=2000 时使用 native vector recall；
• URI/path filter 使用更保守的 auto_path_filter_native_threshold=200，因为宽路径的 Trie 遍历和 subtree bitmap union 本身可能占主要开销；
• 任一阈值设为 0 即可关闭对应路由；
• 显式 backend=cuvs 对支持的 dense query 继续固定使用 cuVS。

mutation 与 GPU search 共享一把跨后端锁，避免 native bitmap 与过期 GPU row layout 配对；native fallback 在该锁外执行，保留 native engine 原有的读并发。

显存感知 auto 模式与 VRAM

配置 backend=local 和 cuvs.auto_enable=true 后，OpenViking 会在每次 lazy build/rebuild 前检查空闲显存。准入估算 float32 vector payload、CAGRA retained/intermediate graph 和 filter-bitset cache，默认乘 2.0 safety factor，并保留 1 GiB。如果预算不足、cuVS 不可用，或者准入后仍发生 allocation failure，本次查询继续使用未改变的 native index；GPU index 保持 dirty，后续查询会重试，因此显存释放后可以自动启用。

brute-force vector payload 约为 N * dimension * 4 bytes。CAGRA 还常驻约 N * graph_degree * 4 graph bytes，并可能在 build 时需要 N * intermediate_graph_degree * 4。每个缓存 bitset 约占 N / 8 bytes。既有 5 进程测试记录的 build 后显存增量为：

数据集	cuVS 算法	GPU 显存增量
100K x 768D	brute-force	294 MiB
1M x 768D	brute-force	2.9 GiB
100K x 1024D	brute-force	392 MiB
1M x 1024D	brute-force	3.9 GiB
1,183,514 x 100D	brute-force	452 MiB
1,183,514 x 100D	CAGRA	872 MiB

这些是常驻增量，不是 sampled peak，也不包含 build 前观测到的约 327 MiB CUDA runtime/context 基线。allocator 状态、参数、query batch 和并发 GPU workload 都可能提高 peak VRAM，所以 auto gate 有意采用保守预算。

初步 vector-index 结果

公开 benchmark 使用 ann-benchmarks glove-100-angular：1,183,514 x 100D base vectors、10,000 条 query、K=10、cosine ranking、单张 NVIDIA H20。每项均为 5 个独立进程的中位数 +/- 中位绝对偏差（MAD）。“GPU exact”特指 cuVS brute-force；CAGRA 是近似检索。

Batch size 1，先执行 10 个 warm-up batch：

后端	Recall@10	p50（ms/query）	QPS
OpenViking native CPU exact，FP32	1.0000 +/- 0.0000	40.209 +/- 0.071	24.7 +/- 0.1
cuVS GPU brute-force exact，FP32	1.0000 +/- 0.0000	0.796 +/- 0.005	1,254.5 +/- 7.1
cuVS CAGRA，itopk_size=512	0.9633 +/- 0.0003	1.730 +/- 0.016	562.8 +/- 1.7
cuVS CAGRA，itopk_size=2048	0.9944 +/- 0.0002	1.797 +/- 0.019	549.0 +/- 0.9

在这个单查询场景下，相比当前 FP32 native exact 路径，GPU brute-force exact 的 warm p50 快 50.5 倍，QPS 高 50.7 倍。第一次检索的中位数分别为 native 85.7 ms、GPU exact 104.0 ms，因此 cold start 与 warm search 需要分开考虑。

Batch size 128 下，CAGRA 在 Recall@10=0.9628 时达到 43,595 +/- 280 QPS，是 GPU exact 的 1.23 倍；在 Recall@10=0.9943 时为 21,711 +/- 79 QPS，只有 GPU exact 的 0.61 倍。因此本轮只有在接受较低 recall 时 CAGRA 才体现吞吐优势。

确定性归一化 Gaussian vectors 的 warm、batch=1 exact 结果为：

维度	向量数	Native p50（ms）	GPU exact p50（ms）	加速比
768	100K	21.991 +/- 0.177	0.341 +/- 0.003	64.5x
768	1M	229.728 +/- 0.580	1.435 +/- 0.002	160.1x
1024	100K	28.443 +/- 0.262	0.376 +/- 0.001	75.7x
1024	1M	306.730 +/- 0.838	1.708 +/- 0.002	179.6x

观察到的 warm 交叉点在 768D 下位于 2K 到 5K，在 1024D 下位于 1K 到 2K。这些边界与硬件和 workload 有关，不是通用阈值。

初步 collection-level 结果

Collection benchmark 调用 CollectionAdapter.query()，覆盖 filter、label-to-record lookup、结果归一化、持久化、mutation 和 lazy rebuild。数据集为 100K x 768D 归一化合成向量，每个场景 50 条 query、K=10，并运行 5 个独立进程。

上一版 prepared-bitset 实现仍需为新 scalar filter 扫描 Python records，首次使用耗时 119--141 ms。native bitmap bridge 将对应延迟降到 1.38--2.98 ms，配对改善 47--86 倍。auto 模式的中位数为：

场景	首次查询（ms）	Warm p50（ms）	路由
无过滤	1,902.930 +/- 6.045	0.972 +/- 0.015	cuVS；首次查询包含 lazy GPU build
均匀 scalar 10%	2.984 +/- 0.034	1.119 +/- 0.029	cuVS
均匀 scalar 1%	2.096 +/- 0.019	0.643 +/- 0.018	native
均匀 scalar 0.1%	1.486 +/- 0.014	0.404 +/- 0.009	native
URI path 10%	12.860 +/- 0.569	1.126 +/- 0.024	cuVS
URI path 1%	3.087 +/- 0.031	1.020 +/- 0.027	cuVS
URI path 0.1%	1.635 +/- 0.014	0.455 +/- 0.010	native

在 clean revision d2a74c1 上增加了 dirty-index selective-first 验证：GPU 尚未 build 时先执行 0.1% scalar filter，5 个进程的延迟为 11.670 +/- 0.156 ms，GPU 显存增量为 0 +/- 0 byte；随后的 unfiltered 首查仍为 1,889.281 +/- 11.155 ms。这证明 native 路由已经发生在 GPU admission/rebuild 之前，并且 dirty 状态会保留给后续真正走 GPU 的查询。

path 首次使用成本取决于 subtree 宽度：10% URI prefix 约花 12.9 ms 遍历并 union native path bitmap，但重复查询会复用 device bitset。1% path 保持在 GPU，是因为反复重建 native path bitmap 比缓存后的 cuVS search 更慢；只有 0.1% path 越过更低的 path threshold。clustered scalar 结果遵循相同路由模式，完整表见详细报告。

这些 collection 数据保留正常默认配置：native 使用 int8，cuVS 使用 float32。之前显式 backend 对比中，以 cuVS brute-force 为基准，native Recall@10 在无过滤时为 0.982，有过滤时为 0.978--0.994。上面的 auto 表是延迟路由结果，不是等 dtype 的质量对比。

验证状态

• 70 个目标 VectorDB 单元/集成测试通过
• 9 个 collection benchmark 与跨进程汇总测试通过
• 7 个 stable-ABI loader 测试通过
• 独立 C++ engine 测试覆盖任意外部 label 顺序的 bitmap 投影
• 真实 GPU 上 brute-force 和 CAGRA smoke 通过，覆盖 scalar + URI 组合过滤、更新、删除和重建
• Python lint、C++ syntax check 和 git diff --check 通过
• 推送前对 staged diff 和 commit diff 做了凭据、私钥和非公开环境标识扫描

WIP / 后续工作

• 评估 read-safe GPU snapshot 配合并发 CUDA streams 或 micro-batching
• 为写活跃 collection 评估后台/合并重建以及原子 index swap
• 保持非侵入的 dtype 边界，优先评估可配置的 cuVS float16
• 在显式 recall/latency/memory frontier 下评估 CAGRA int8/VPQ
• 将 native-compatible scaled-int8 作为 scale-aware distance/reranking 的独立工作，而不是简单 cast
• 增加完整 HTTP/embedding 测试和真实 agent-memory workload
• 跨硬件与软件版本重复选定的性能点

集成计划：docs/design/openviking-cuvs-integration-plan.md。

完整方法和所有初步数据见：benchmark/cuvs/PRELIMINARY_RESULTS.md。

[qin-ctx]

Requesting changes for the two blocking inline comments: auto selective filters rebuild cuVS before falling back to native, and the collection benchmark scenario test is stale and fails on this branch.