Skip to content

Storing IVF indexes on disk

Jump to bottom
Matthijs Douze edited this page Jun 23, 2020 · 5 revisions

For large indexes, it is useful to store the index content on disk. This incurs a high runtime penalty compared to standard indexes stored in RAM, but the size of the index can become larger.

The OnDiskInvertedLists object

Building an ondisk index

Home

Tutorial

Installing Faiss

Getting started

Faster search

Lower memory footprint

Running on GPUs

Basics

MetricType and distances

Faiss building blocks: clustering, PCA, quantization

Guidelines to choose an index

Faiss indexes

Basic indexes

Binary indexes

Composite indexes

Pre- and post-processing

The index factory

Index IO, cloning and hyper parameter tuning

Special operations on indexes

Additive quantizers

CPU Faiss + Intel SVS - Overview

CPU Faiss + Intel SVS - Usage

GPU Faiss

GPU overview

GPU versus CPU

Sample: GPU k-means

GPU Faiss + NVIDIA cuVS - Overview

GPU Faiss + NVIDIA cuVS - Usage

Advanced topics

Faiss code structure

Threads and asynchronous calls

Inverted list objects and scanners

Indexes that do not fit in RAM

Vector codecs

Brute force search without an index

Fast accumulation of PQ and AQ codes (FastScan)

Implementation notes

Setting search parameters for one query

How to make Faiss run faster

Typical use cases and benchmarks

Low level benchmarks

Indexing 1M vectors

Indexing 1G vectors

Indexing 1T vectors

Vector codec benchmarks

Binary hashing index benchmark

Hybrid CPU/GPU and multiple GPUs

Case studies

Additional information

FAQ

Python/C++ code snippets

Troubleshooting

Related projects

Clone this wiki locally