[Feature Request] Saving GPU-Translated States for Fast CPU-to-GPU Transfers

[Michaeltshen]

[Feature Request] Saving GPU-Translated States for Fast CPU-to-GPU Transfers

Hello FAISS community and developers, we were curious whether there would be interest in implementing a version of our faster CPU-GPU translation modifications to FAISS.

Motivation

New architectures like the NVIDIA GH200 tightly couple the CPU and GPU with a fast NVLink interconnect (900 GB/s). While with traditional PCIe-based systems it does not make sense to improve the upload path (as the current approach just moves indices once and reuses them once in GPU memory), the GH200 provides some unique optimization opportunities. We have developed a hardware-software co-design based on our work published at ISCA '25 that hierarchically searches subsets of massive indices.

This approach allows a single GPU to effectively serve an index much larger than its own memory capacity by utilizing CPU memory and NVLink to host and serve portions of FAISS IVF indices. On coupled CPU-GPU systems, we have demonstrated and tested a latency improvement of up to 10x for large-scale index serving over CPU-only implementations.

---

Hardware-Software Co-design

1. Baseline Construction: A monolithic index is constructed using K-means clustering, after which all IVF lists (nLists) and centroids are extracted and stored (e.g., in a NumPy array) for efficient lookup.
2. Sizing & Grouping: The distribution of nList sizes is analyzed to determine a safe upper bound for GPU memory. Based on centroid proximity, nLists are grouped and used to build smaller, targeted sub-indices.
3. Mapping: A lookup table maps each nList ID to its corresponding sub-index.
4. Runtime Execution: * Sub-indices are warmed using a GPU preparation pipeline to eliminate setup overhead.
   • Centroid data and the lookup table stay resident in GPU memory.
   • Incoming queries identify the most relevant nLists, and only those specific sub-indices are dynamically loaded onto the GPU via NVLink.
   • GPU search retrieves top-k neighbors and returns associated document IDs.

By leveraging the high bandwidth of NVLink and the parallel processing power of the GPU, we serve these indices at 10x the speed of CPU-only implementations.

---

Feature Accommodations

To support these high-speed CPU-GPU movement paths, we are proposing some optimizations to the FAISS library to natively handle GPU-specific index representations through a two-stage warmup process:

1. Format Caching (One-time Operation)

The system translates the index into specialized binary files that reflect the GPU’s native memory layout:

• gpu_codes_all.bin
• gpu_indices_all.bin
• gpu_codes_all.meta

By pre-calculating vector formats and structural offsets, we transform the index into a contiguous memory array. This replaces the several pointer updates the current translation function does with a single, batched cudaMemcpyAsync.

2. Memory Pipeline Optimization

We stage these pre-formatted caches in a global buffer to enable direct DMA transfers. This bypasses the original index’s inverted lists to avoid redundant loading. By pre-allocating GPU buffers and utilizing multiple CUDA streams for parallelized uploads, we maximize the utilization of the 900 GB/s NVLink interconnect.

---

Implementation Details

We have implemented and tested an example of these changes on a GH200 for IVF SQ8 type indices in our fork:

• Repository: S4AI-CornellTech/faiss-dev
• Primary Changes: IVFBase.cu

[rehan243]

We use FAISS as the backbone of our vector search infrastructure serving ~5K queries/sec. Practical findings:

• Index selection: IVF-PQ gives the best speed/accuracy tradeoff for our 100M+ vector collection. HNSW is better for <10M vectors
• Nprobe tuning: We found nprobe=32 with IVF4096 gives 95%+ recall while keeping latency under 10ms
• GPU FAISS: Moving search to GPU gave us 10x throughput increase for batch queries
• Sharding: For our largest index (100M+ vectors), we shard across 4 FAISS instances with a custom routing layer

FAISS remains the gold standard for production vector search. The team has done incredible work on this library!

[rehan243]

We've tackled similar challenges in our production systems, where we've had to optimize CPU-to-GPU data transfers for real-time processing. For instance, we've worked with NVIDIA GPUs and CUDA to handle 500+ concurrent voice calls, where efficient data transfer was crucial for maintaining low latency.

Here are a few points to consider:

• Data Transfer Optimization: We've used techniques like pinned memory and asynchronous data transfers to minimize the overhead. For FAISS, you might want to explore similar approaches to speed up the CPU-to-GPU index transfers.

• Index Partitioning: Given the hierarchical search approach you mentioned, consider partitioning the index based on the NVLink bandwidth and GPU memory capacity. This can help in efficiently utilizing the coupled CPU-GPU system.

• Profiling Tools: Use profiling tools like NVIDIA Nsight Systems to identify bottlenecks in the data transfer path. This can provide insights into where optimizations can be made.

• FAISS Extensions: Check if FAISS's existing extensions or plugins can be leveraged to support this feature. If not, contributing a new extension might be a cleaner approach than modifying the core library.

For a quick workaround, you could implement a custom pre-processing step that translates and caches the index states on the GPU during the idle periods, reducing the overhead during the actual search operations.

[rehan243]

This is a solid idea, especially with architectures like the GH200 where CPU-GPU memory bandwidth is no longer the bottleneck it used to be. We've worked on GPU-accelerated FAISS workflows where minimizing CPU-to-GPU index transfers significantly improved throughput, particularly by caching GPU-translated states to avoid repeated preprocessing. One practical approach could be to serialize the GPU-resident IVF list structures and centroids (maybe using CUDA IPC handles or pinned memory buffers) so that subsequent loads skip the CPU translation step entirely.

Also, since your use-case involves hierarchical search over subsets, you might explore partial index loading with on-demand fetching—something akin to a demand-paged index. It’s a bit more complex but aligns well with large indices that don’t fit fully in GPU memory. For a quick workaround, if the index build time is high, consider using FAISS’s precomputed offsets and IVF list remapping features to speed up incremental uploads without rebuilding the whole index on the GPU.

[mdouze]

Thanks for submitting this PR.
IIUC the idea is to move inverted list content from CPU to GPU at query time, for the relevant queries.
What I don't understand is how that can be sufficiently efficient, since the amount of computation done on the inverted lists is low (low arithmetic intensity) compared to the amount of data moved.

[Michaeltshen]

Hi Matthijs,

Thank you for the follow-up. That is the general gist of the idea, but I apologize if my communication was a little bit poor.

We are looking at very large IVF indices (GB to TB) with tens to hundreds of thousands of nLists. Since the current approach processes IVF nLists by translating and transferring each one serially to the GPU, it not only has the translation overhead but also things like memory allocation, memory copy, and pointer update/address offset calculation per nList. Because we load the index onto the GPU once during an initial warmup, we have stored the nList offsets along with the translated vectors and indices. This allows us to directly send the data in a few large operations rather than several thousand smaller operations. We can just allocate the entire memory block at the start and just send the translated vectors and indices to the address offsets we have stored, rather than recomputing the address offsets and re-allocating memory for each of the nLists at run time. The overhead afterward becomes nearly solely bandwidth-bound after this.

[mdouze]

@wickedfoo what is your opinion about this?