Bound recursion depth in index deserialization (CWE-674) (#5468)

Summary:
Pull Request resolved: #5468

Agentic fuzzing (T278233426) found an uncontrolled-recursion stack
overflow in `faiss::read_index_binary_up`. The serialized index format is
self-describing and recursively nestable: wrapper types (`IBMp`/`IBM2`
IndexBinaryIDMap, `IBHf`/`IBHc` HNSW storage, `IBFf` IndexBinaryFromFloat)
each recurse back into the reader to deserialize their inner index, with
no depth bound. A stream of a few thousand nested wrapper markers — each
only a 4-byte fourcc plus a small header — drives recursion until the
thread stack is exhausted, killing the process (SIGSEGV on the guard
page). The float reader `read_index_up` has the identical structure
(`IxMp`, `IxPT`, `IxRF`, HNSW storage, ...).

Add a shared, thread-local nesting-depth guard (RAII) at the entry of both
`read_index_up` and `read_index_binary_up`, capped at 50 levels. The
check runs before incrementing so a thrown limit leaves the counter
consistent; the destructor decrements on unwind. Legitimate indexes nest
only a handful of levels, so the limit is generous while turning the
stack-exhaustion crash into a catchable `FaissException`.

Reviewed By: alibeklfc

Differential Revision: D113119945

fbshipit-source-id: 7c4015131ca6c0bc7de4a3b8ba764ecff0119dae

Author: Michael Norris

faiss/impl/index_read.cpp

@@ -1608,7 +1608,29 @@ static std::unique_ptr<IndexIVFPQ> read_ivfpq(
 
 int read_old_fmt_hack = 0;
 
+namespace {
+
+constexpr int kMaxIndexNestingDepth = 50;
+thread_local int index_read_nesting_depth = 0;
+
+struct IndexNestingGuard {
+    IndexNestingGuard() {
+        FAISS_THROW_IF_NOT_FMT(
+                index_read_nesting_depth < kMaxIndexNestingDepth,
+                "faiss index nesting depth exceeds limit of %d; "
+                "input may be corrupt or malicious",
+                kMaxIndexNestingDepth);
+        ++index_read_nesting_depth;
+    }
+    ~IndexNestingGuard() {
+        --index_read_nesting_depth;
+    }
+};
+
+} // namespace
+
 std::unique_ptr<Index> read_index_up(IOReader* f, int io_flags) {
+    IndexNestingGuard nesting_guard;
     std::unique_ptr<Index> idx;
     uint32_t h;
     READ1(h);
@@ -3275,6 +3297,7 @@ static void read_binary_multi_hash_map(
 }
 
 std::unique_ptr<IndexBinary> read_index_binary_up(IOReader* f, int io_flags) {
+    IndexNestingGuard nesting_guard;
     std::unique_ptr<IndexBinary> idx;
     uint32_t h;
     READ1(h);