Cannot open HNSW zvec collection on Apple Silicon (arm64): Map segment failed, segment id HnswT2S0

[cc1234475]

Summary

Opening a zvec collection (HNSW index) fails on Apple Silicon (macOS arm64) with a segment-mapping error. The exact same collection opens successfully on x86_64 Linux, so this is architecture-specific.

Error

On macOS 26.5.2, Apple M-series (arm64):

index_mapping.cc:379      Map segment failed, segment id HnswT2S0
hnsw_chunk.cc:42          Storage append segment failed for Duplicate
hnsw_streamer_entity.cc:380  Open index failed for Duplicate
index.cc:337              Failed to open streamer, path: .../collection.zvec
Zvec error: open collection failed err=8

err=8 on macOS is ENOEXEC. The failing path is the HNSW segment mmap in index_mapping.cc.

On x86_64 Linux, the identical collection opens fine.

Environment

• macOS 26.5.2 (build 25F84), Apple Silicon arm64
• zvec built from source, cmake -DBUILD_C_BINDINGS=ON -DBUILD_TOOLS=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET=14.0
• CMake detects turbo: ARM64 detected, NEON enabled by default
• zvec commit tested: e5e799221b45bbb90f5fdc5c69b6b8dfbf017e78
• SDK: macOS 26.x

Steps to reproduce

1. Build zvec C bindings natively on Apple Silicon (see env above).
2. Open any existing HNSW zvec collection (created on x86_64).
3. Observe the Map segment failed, segment id HnswT2S0 error and open collection failed err=8.

Notes

• Reproduced with both a GitHub-release prebuilt binary and a freshly source-built library, so it is not a packaging/artifact issue.
• The collection was created on Linux x86_64; cross-architecture open fails at the HNSW segment map step.
• Related: zvec-ai/zvec-rust#13 (same failure surfaced through the Rust binding).

[cc1234475]

Resolved: page-size portability bug in HNSW segment mmap

Root cause identified and fixed on the writer side. Closing this issue.

Root cause

CalcPageAlignedSize in src/core/framework/index_mapping.cc aligns index segments to the host page size of the machine that wrote the index. Apple Silicon macOS uses 16 KB pages; Linux x86_64 uses 4 KB. An index written on a 4 K host therefore has 4 K-aligned segment offsets that a 16 K host cannot mmap (offsets must be page-aligned on the opening host), producing:

index_mapping.cc:379   Map segment failed, segment id HnswT2S0
Zvec error: open collection failed err=8

err=8 is the zvec error enum, not a POSIX errno. It is not a codegen/NEON fault: a native source build on arm64 fails identically, while the same index opens fine on x86_64 Linux.

Evidence

• Shipped (4 K-written) HNSW segments: 0/adaface.index.1.proxima = 5263360 B (mod 4K = 0, mod 16K = 4096).
• A collection created on the Mac (16 K host): segment = 5324800 B (mod 16K = 0), and opens on the Mac.
• A collection written on Linux with the patched writer: segment = 92913664 B (mod 16K = 0), and opens on the Mac with an unpatched reader (134,638 docs).

So the fix is writer-side only; readers need no change.

Fix applied (local)

In CalcPageAlignedSize, floor the alignment to 16 KB (matches the align_size = 4096 * 4 constant already used in src/core/framework/index_helper.cc):

size_t page_size = ailego::MemoryHelper::PageSize();
if (page_size < 4096 * 4) page_size = 4096 * 4;   // portable across 16K-page hosts (Apple Silicon)

16 KB is a multiple of 4 KB, so a 16 K-aligned index still opens on 4 K-page hosts (Linux x86_64, Windows x86_64), and opens exactly on 16 K-page Apple Silicon.

Suggested upstream fix

Rather than host page size, align HNSW segment files to the maximum supported page size at write time (16 KB today; 64 KB if future hosts need it). This makes a written index portable across all architectures without reader changes.