Fix peak-memory spike when loading IVF invlists via IO_FLAG_MMAP_IFC (#5122)

Summary:
On a 148 GB `IndexIVFRaBitQ`, `faiss.read_index(path, IO_FLAG_MMAP_IFC)` currently spikes to **165 GB of committed private heap** for ~50 seconds during load, before dropping back to the intended ~9 GB steady state. The spike is transient but matches the entire invlist payload, which defeats the purpose of `IO_FLAG_MMAP_IFC` (keep invlist codes and ids file-cache-backed, not pagefile-committed).

Root cause is a two-loop pattern in `read_InvertedLists_up` (`faiss/impl/index_read.cpp`) for both the `ilar` (`ArrayInvertedLists`) and `ilpn` (`ArrayInvertedListsPanorama`) branches:

```cpp
// Loop A: pre-allocate every list's owning std::vector (= full invlist size on heap)
for (i = 0 .. nlist) {
    ails->codes[i].resize(sizes[i] * code_size);
    ails->ids[i].resize(sizes[i]);
}
// Loop B: replace each owning vector with a view into mmap
for (i = 0 .. nlist) {
    read_vector_with_known_size(ails->codes[i], ...);
    read_vector_with_known_size(ails->ids[i], ...);
}
```

`read_vector_with_known_size` detects a `MappedFileIOReader` and replaces the target `MaybeOwnedVector` via `create_view`, correctly releasing the owning storage. But Loop A has already committed 100% of the invlist size to the heap before Loop B can release any of it.

## Fix

Merge the two loops into a single pass per list, so each list's owning heap allocation is released via the view-substitution before the next list's is made. Peak heap during load is now bounded by a single list's worth (~one nprobe cluster, typically under 1 MB), not the sum of all lists'.

Applied to both the `ilar` and `ilpn` branches. End state is byte-identical: with `MappedFileIOReader` every `MaybeOwnedVector` ends up as a view; with a regular `FileIOReader` every `MaybeOwnedVector` ends up as owning storage.

## Measured impact

Same IndexIVFRaBitQ file (IVF262144_HNSW32,RaBitQ8, 99.7 M vectors, nlist=262144, code_size=1556, 148 GB on disk), loaded via `faiss.read_index(path, IO_FLAG_MMAP_IFC)`:

| Metric              |  Before | After |
|---------------------|--------:|------:|
| Peak private memory | 164 GB  | 10 GB |
| Load time           |  99 s   | 16 s  |
| Steady-state RSS    |   9 GB  |  9 GB |

Load time drops because ~90 s of the original was spent in `std::vector::resize` zero-filling pages that were about to be discarded.

Pull Request resolved: #5122

Test Plan:
- [x] Manual: `faiss.read_index(path, IO_FLAG_MMAP_IFC)` on the 148 GB production index; `search(q, 20)` at `nprobe=64` returns the same top-k ids before and after the patch.
- [x] Manual: 1 Hz private-memory polling during a 10-query, `k=500`, `nprobe=256` stress — RSS grows identically to the pre-patch behaviour (file-cache page-fault growth, not private heap), confirming the view substitution still works.
- [ ] Existing FAISS CI — no new tests added. The bug is a transient peak that existing tests cannot observe, and the fix is an ordering change with no new code path.

## Safety

- No API, ABI, serialization-format, or behavioural change.
- `FAISS_CHECK_DESERIALIZATION_LOOP_LIMIT` hardening still runs in the same place, before the merged loop.
- Works identically on `FileIOReader` (heap-owning end state) and `MappedFileIOReader` (view end state).

Made with [Cursor](https://cursor.com)

Reviewed By: mdouze

Differential Revision: D103055438

Pulled By: mnorris11

fbshipit-source-id: 5cdc371aa78547f00a4e8e854cac84945d81b6b5

Author: yantis

faiss/impl/index_read.cpp

@@ -518,16 +518,19 @@ std::unique_ptr<InvertedLists> read_InvertedLists_up(
                 nlist, code_size, n_levels, bs);
         std::vector<size_t> sizes(nlist);
         read_ArrayInvertedLists_sizes(f, sizes);
+        // Do resize + read in a single pass per list. See the matching
+        // comment in the `ilar` branch below for rationale.
         size_t byte_limit = get_deserialization_vector_byte_limit();
         for (size_t i = 0; i < nlist; i++) {
+            size_t n = sizes[i];
             FAISS_THROW_IF_NOT_FMT(
-                    sizes[i] <= byte_limit / sizeof(idx_t),
+                    n <= byte_limit / sizeof(idx_t),
                     "inverted list %zu ids size %zu exceeds "
                     "deserialization byte limit",
                     i,
-                    sizes[i]);
-            ailp->ids[i].resize(sizes[i]);
-            size_t num_elems = ((sizes[i] + bs - 1) / bs) * bs;
+                    n);
+            ailp->ids[i].resize(n);
+            size_t num_elems = ((n + bs - 1) / bs) * bs;
             size_t codes_bytes = mul_no_overflow(
                     num_elems, code_size, "inverted list codes");
             FAISS_THROW_IF_NOT_FMT(
@@ -549,9 +552,6 @@ std::unique_ptr<InvertedLists> read_InvertedLists_up(
                     i,
                     cum_sums_count);
             ailp->cum_sums[i].resize(cum_sums_count);
-        }
-        for (size_t i = 0; i < nlist; i++) {
-            size_t n = sizes[i];
             if (n > 0) {
                 read_vector_with_known_size(
                         ailp->codes[i], f, ailp->codes[i].size());
@@ -627,27 +627,30 @@ std::unique_ptr<InvertedLists> read_InvertedLists_up(
         ails->codes.resize(ails->nlist);
         std::vector<size_t> sizes(ails->nlist);
         read_ArrayInvertedLists_sizes(f, sizes);
+        // Resize + read in a single pass per list so that each list's
+        // heap allocation is released by the mmap view-substitution
+        // before the next list is allocated. This bounds peak heap to
+        // one list's worth of memory, which matters for large IVF
+        // indexes (hundreds of GB) under IO_FLAG_MMAP_IFC.
         size_t ilar_byte_limit = get_deserialization_vector_byte_limit();
-        for (size_t i = 0; i < ails->nlist; i++) {
+        for (size_t i = 0; i < sizes.size(); i++) {
+            size_t n = sizes[i];
             FAISS_THROW_IF_NOT_FMT(
-                    sizes[i] <= ilar_byte_limit / sizeof(idx_t),
+                    n <= ilar_byte_limit / sizeof(idx_t),
                     "inverted list %zu ids size %zu exceeds "
                     "deserialization byte limit",
                     i,
-                    sizes[i]);
-            ails->ids[i].resize(sizes[i]);
-            size_t codes_bytes = mul_no_overflow(
-                    sizes[i], ails->code_size, "inverted list codes");
+                    n);
+            ails->ids[i].resize(n);
+            size_t codes_bytes =
+                    mul_no_overflow(n, ails->code_size, "inverted list codes");
             FAISS_THROW_IF_NOT_FMT(
                     codes_bytes <= ilar_byte_limit,
                     "inverted list %zu codes size %zu exceeds "
                     "deserialization byte limit",
                     i,
                     codes_bytes);
             ails->codes[i].resize(codes_bytes);
-        }
-        for (size_t i = 0; i < ails->nlist; i++) {
-            size_t n = ails->ids[i].size();
             if (n > 0) {
                 read_vector_with_known_size(
                         ails->codes[i],

tests/test_io.py

@@ -574,7 +574,39 @@ def test_mmap(self):
                 #   unless index2 is collected by a GC
                 try:
                     os.unlink(fname)
-                except:
+                except Exception:
+                    pass
+
+    @unittest.skipIf(
+        platform.system() not in ["Windows", "Linux", "Darwin"],
+        "supported OSes only"
+    )
+    def test_mmap_ivf(self):
+        d, nlist = 32, 64
+        xt, xb, xq = get_dataset_2(d, 2000, 5000, 50)
+        index = faiss.index_factory(d, f"IVF{nlist},Flat")
+        index.train(xt)
+        index.add(xb)
+        index.nprobe = 8
+        Dref, Iref = index.search(xq, 10)
+
+        fd, fname = tempfile.mkstemp()
+        os.close(fd)
+
+        index2 = None
+        try:
+            faiss.write_index(index, fname)
+            index2 = faiss.read_index(fname, faiss.IO_FLAG_MMAP_IFC)
+            index2.nprobe = 8
+            Dnew, Inew = index2.search(xq, 10)
+            np.testing.assert_array_equal(Iref, Inew)
+            np.testing.assert_array_equal(Dref, Dnew)
+        finally:
+            del index2
+            if os.path.exists(fname):
+                try:
+                    os.unlink(fname)
+                except Exception:
                     pass
 
     def test_zerocopy(self):