[Feature] Launch Buffer Storage for Memory-Bounded Query Caching

[iaojnh]

Background

zvec currently relies primarily on mmap for index access. When an index is significantly larger than available memory, mmap provides no explicit budget for RSS or resident pages. Multiple Collections and application code sharing the same process may also compete for memory.

Buffer Storage provides a process-wide shared cache budget for index pages. Its primary use case is when the full index cannot be economically kept in memory, but the frequently reused working set can fit in the Buffer Pool.

Representative validated results:

• IVF, Cohere 1M, 80/20 hotspot workload, 256 MiB Pool: peak RSS was 303 MiB versus 2,888 MiB with mmap, with 1.72× higher QPS/GiB.
• DiskANN, Cohere 1M, 80/20 semantic workload, 512 MiB Pool: QPS increased by 39%, P99 decreased by 14%, physical reads decreased by 80%, and Recall@10 remained unchanged.

Goals

Release Buffer Storage as an explicit opt-in feature, initially targeting IVF and DiskANN query workloads:

• Allow users to configure a process-wide shared memory budget.
• Support querying IVF and DiskANN indexes through Buffer Storage.
• Bring the Buffer Pool, RocksDB, and required metadata under a unified memory-budget framework.
• Improve QPS/GiB and reduce repeated physical reads under hotspot and long-tail workloads.
• Keep mmap as the default behavior without affecting existing users.

Scope

• Harden Buffer Pool lifecycle, concurrency, eviction, writeback, and memory accounting.
• Support Linux io_uring, libaio, and pread fallback paths, with macOS and Windows compatibility.
• Support Buffer Storage reads and writes.
• Integrate IVF, including cross-page posting-list access.
• Integrate DiskANN file access with Buffer Storage.
• Give RocksDB an internal fixed share of the memory budget without exposing a separate ratio through the Python API.
• Provide explicit opt-in APIs for Python and C++.
• Add correctness, memory, performance, recall, and cross-platform tests.
• Provide user documentation and reproducible benchmarks.

Out of Scope

• Index-build peak memory is not guaranteed to stay within memory_limit_mb. A Buffer Collection may use mmap temporarily during index construction. Memory-bounded index building will be addressed separately.
• Process RSS is not expected to equal the Buffer Pool size. RSS also includes query workspaces, thread stacks, runtimes, allocators, and required metadata.
• Strict per-Collection memory isolation is not provided. All Buffer Collections in a process share the same budget.
• Flat indexes scan most vectors and offer limited page reuse, so they are not a target workload for Buffer Storage.
• HNSW graph access has some locality, but its graph and vector access patterns have not demonstrated a clear advantage over DiskANN under the same memory budget. No HNSW performance benefit will be promoted in this release.

User API

import zvec

# Must be called before any Collection operation.
# The budget is shared by all Buffer Collections in the process.
zvec.init(memory_limit_mb=2048, query_threads=8)

collection = zvec.open(
    "./items.zvec",
    option=zvec.CollectionOption(
        read_only=True,
        enable_mmap=False,
    ),
)

Release behavior:

• enable_mmap=True remains the default.
• Buffer Storage is explicitly enabled with enable_mmap=False.
• The storage mode is persisted when a Collection is created.
• Opening an existing Collection with the opposite option does not automatically convert its storage format.
• The cache starts empty after a process restart and may be warmed with representative queries.

Implementation Phases

• PR1: Harden the Buffer Pool core, including budgeting, concurrency, eviction, writeback, I/O fallback, and core tests.
• PR2: Implement Buffer Storage read/write paths, lifecycle management, and unified memory accounting.
• PR3: Integrate IVF and support cross-page posting lists.
• PR4: Integrate DiskANN with direct I/O and Buffer Pool query paths.
• PR5: Finalize public APIs, build behavior, user documentation, and benchmark tooling.
• Complete integration regression testing before release.

Acceptance Criteria

Correctness

• mmap and Buffer Storage produce identical query-result fingerprints.
• IVF and DiskANN recall remains unchanged across storage modes.
• Write, flush, optimize, reopen, and read-only workflows behave correctly.
• Cache eviction, background reclamation, and writeback do not cause data corruption or sustained memory growth.

Memory

• The Buffer Pool and RocksDB follow the process-wide shared soft budget.
• Low-budget configurations do not exhibit uncontrolled growth from page tables, locks, or low-reuse pages.
• Documentation clearly states that memory_limit_mb is not a hard process RSS limit.
• Multiple Collections share the budget without creating independent, duplicated caches.

Performance

Use identical data, index parameters, recall targets, query sequences, and concurrency. Record at least:

• QPS
• P95/P99
• Peak RSS
• Physical reads
• Recall or result fingerprint
• QPS/GiB

When real query logs are unavailable, cover at least:

• Uniform unique
• 80/20 hotspot
• Zipf α=1.0
• 90/10 exact repeat

Use 80/20 and Zipf as the primary results. Exact-repeat workloads represent only the upper bound of cache benefits.

Compatibility

• Linux, macOS, and Windows CI pass.
• The default mmap path has no functional regression.
• The I/O path safely falls back to pread when io_uring or libaio is unavailable.
• Public Python and C++ APIs remain backward compatible.

Definition of Done

Close this issue after all implementation phases are merged, correctness, memory, performance, and cross-platform tests pass, and the user documentation explains supported workloads, budget configuration, capability boundaries, and validation methods.