Add pip install support via scikit-build-core + cibuildwheel (#4862) by alibeklfc · Pull Request #4862 · facebookresearch/faiss

alibeklfc · 2026-03-04T21:03:35Z

Summary:

Add official pip wheel packaging for faiss-cpu, enabling
pip install faiss-cpu from PyPI. Builds binary wheels across Linux
x86_64/aarch64, macOS arm64/x86_64, and Windows x86_64 for Python 3.10-3.14.
Linux and macOS use Python Stable ABI (abi3) — a single cp310-abi3 wheel
per platform works on all Python 3.10+ versions without per-version rebuilds.

BLAS strategy:

Linux (x86_64 and aarch64): Threaded OpenBLAS (libopenblaso). The
CMake build prefers libopenblaso (OpenMP-threaded) or libopenblasp
(pthreads-threaded) over the serial default on RHEL/Fedora-based
manylinux containers.
macOS: Apple Accelerate (found automatically).
Windows: Prebuilt OpenBLAS from GitHub releases (v0.3.30).

SIMD optimization strategy:

Linux and macOS: Dynamic Dispatch (FAISS_OPT_LEVEL=dd) compiles SIMD-hot
kernel files at multiple ISA levels (generic + AVX2 + AVX-512 on x86_64,
NEON + SVE on aarch64) into a single library. Runtime CPUID detection
selects the optimal code path automatically.
Windows: MSVC does not support DD, so uses FAISS_OPT_LEVEL=generic.
Shared libraries (BUILD_SHARED_LIBS=ON) are used to avoid MSVC's
single-pass static archive scanning issue.

New files:

pyproject.toml: scikit-build-core build backend config with cibuildwheel
settings. Version is dynamically extracted from CMakeLists.txt. Enables
abi3 on Linux/macOS via SWIG 4.2+ and NumPy 2.0+ Limited API support.
Enables LTO and dead-code stripping for smaller wheel sizes.
.github/workflows/build-pip.yml: CI workflow using cibuildwheel to build
binary wheels on 5 platform runners. All GitHub Actions are pinned to
commit SHAs for supply chain security. Publishes wheels to PyPI via OIDC
trusted publishers on tag push.
THIRD_PARTY_NOTICES: License notices for bundled dependencies (OpenBLAS,
LLVM OpenMP).
tests/test_wheel_smoke.py: 12-test smoke suite validating import,
OpenMP, BLAS (FlatL2/FlatIP), index factory (IVF+PQ), HNSW, serialization
roundtrip, GC safety, contrib imports, and SIMD level detection.

Modified files:

faiss/CMakeLists.txt: Added threaded OpenBLAS fallback — prefers
libopenblaso (OpenMP-threaded) or libopenblasp (pthreads-threaded)
over the serial default on RHEL/Fedora-based manylinux containers.
python/CMakeLists.txt: Added install() targets for scikit-build-core
wheel packaging, including all Dynamic Dispatch SWIG variants
(avx2/avx512/avx512_spr/sve). Added abi3 support: conditionally uses
Python::SABIModule, defines Py_LIMITED_API, and sets .abi3.so suffix
when SKBUILD_SABI_VERSION is set.
.github/workflows/build.yml: Wired build-pip.yml into the root CI
trigger so pip builds run alongside conda builds.
tests/BUCK: Added python_pytest target for the smoke test.

Reviewed By: mnorris11

Differential Revision: D95258115

meta-codesync · 2026-03-04T21:03:45Z

@alibeklfc has exported this pull request. If you are a Meta employee, you can view the originating Diff in D95258115.

…kresearch#4862) Summary: Add official pip wheel packaging for `faiss-cpu`, enabling `pip install faiss-cpu` from PyPI. Builds wheels across Linux x86_64/aarch64, macOS arm64/x86_64, and Windows x86_64 for Python 3.10-3.13. **New files:** - `pyproject.toml`: scikit-build-core build backend config with cibuildwheel settings. Uses Dynamic Dispatch (`FAISS_OPT_LEVEL=dd`) for runtime SIMD selection, OpenBLAS for BLAS, and disables GPU/MKL/SVS. - `.github/workflows/build-pip.yml`: CI workflow using cibuildwheel to build wheels on 5 platform runners. Publishes to PyPI via OIDC trusted publishers on tag push. - `tests/test_wheel_smoke.py`: 11-check smoke test suite validating import, OpenMP, BLAS (FlatL2/FlatIP), index factory (IVF+PQ), HNSW, serialization roundtrip, GC safety, contrib imports, and SIMD level detection. **Modified files:** - `python/CMakeLists.txt`: Added `install()` targets so scikit-build-core can package the SWIG extension module, Python source files, and contrib subpackage into wheels. - `.github/workflows/build.yml`: Wired `build-pip.yml` into the root CI trigger so pip builds run alongside conda builds. Differential Revision: D95258115

kyamagu · 2026-03-10T00:49:56Z

+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["cp310", "cp311", "cp312", "cp313"]


No cp314 support yet?

Added cp314 support. Enabled abi3 on Linux/macOS, for Windows build wheel for each python version

kyamagu · 2026-03-10T00:51:26Z

+          python-version: '3.12'
+
+      - name: Install cibuildwheel
+        run: pip install cibuildwheel==2.22.0


Latest cibuildwheel is v3.4.0. Any reason to pin the version?

Updated cibuildwheel version

kyamagu · 2026-03-10T00:56:09Z

+      - name: Build sdist
+        run: pipx run build --sdist
+
+      - name: Upload sdist


IMO, sdist format for a large project like Faiss would only introduce user confusion with a lot of build error reports appearing on GitHub issues. If GPU wheels are planned to be officially supported, I believe we should exclude sdist from PyPI upload.

Removed sdist

kyamagu · 2026-03-10T00:58:47Z

 import inspect

+
+def _preload_gpu_libs():


Just a comment. Actually, Intel MKL or BLAS libraries can be treated the same way, as long as the build-time configuration is properly set up. That would be a future enhancement to reduce the package size.

kyamagu · 2026-03-10T01:06:51Z

+
+[build-system]
+# cuVS requires rapids-cmake which needs cmake >= 3.30.4.
+requires = ["scikit-build-core>=0.10", "numpy>=2.0", "cmake>=3.30.4"]


Just a comment. It is possible to include swig from PyPI as a build-time dependency.

Included swig as a build-time dependency.

kyamagu · 2026-03-10T01:14:51Z

+dependencies = [
+  "numpy>=1.25",
+  "packaging",
+  "nvidia-cuda-runtime-cu12",


I'm not an expert on this, but do we need to worry about CUDA minor version compatibility? It is fine if a CUDA 12.6 wheel runs on a CUDA 12.4 runtime. https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html

Will be addressed in next PR, this PR is not only for faiss-cpu

kyamagu · 2026-03-10T01:18:07Z

+# location (C:\mkl) so FindMKL.cmake can discover it regardless of
+# PEP 517 build isolation.  DLL collection includes MKL, Intel OpenMP
+# (libiomp5md.dll), and TBB — all needed at runtime.
+before-all = "pip install virtualenv && choco install -y swig && pip install mkl==2024.2.2 mkl-devel==2024.2.2 && python -c \"import os,sys,shutil; os.makedirs('C:/mkl/lib',exist_ok=True); os.makedirs('C:/mkl/bin',exist_ok=True); os.makedirs('C:/mkl/include',exist_ok=True); [shutil.copy2(os.path.join(r,f),'C:/mkl/lib/') for r,_,fs in os.walk(sys.prefix) for f in fs if f.endswith('.lib') and 'mkl' in f]; [shutil.copy2(os.path.join(r,f),'C:/mkl/bin/') for r,_,fs in os.walk(sys.prefix) for f in fs if f.endswith('.dll') and ('mkl' in f or 'iomp' in f or 'tbb' in f)]; [shutil.copy2(os.path.join(r,f),'C:/mkl/include/') for r,_,fs in os.walk(sys.prefix) for f in fs if f=='mkl.h']; print('MKL lib:',os.listdir('C:/mkl/lib')); print('MKL bin:',os.listdir('C:/mkl/bin'))\""


I would refactor this in a script file for better maintainability.

Removed this part, since we will use BLAS for now

kyamagu · 2026-03-10T01:29:37Z

+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-14, windows-2022, 2-core-ubuntu-arm]


PR description says macOS x86_64 is included, but the CI configuration drops the Intel support; macos-14 is arm64. Better to update the PR description

Added macos-15-intel support

alibeklfc · 2026-04-03T22:28:29Z

Hi, @kyamagu ! I have updated the PR. This PR is only for faiss-cpu support right now. I will work on faiss-gpu after that. I addressed all your comments. Made this PR cleaner and more efficient.
Can you please review this PR again?
Thank you!

kyamagu

The implementation looks great! The only thing I found is the duplicated wheel content. After fixing this, I believe the pip support is ready.

I would initiate the transfer of the PyPI repository when you're ready. Let me know your or your designated PyPI account.

kyamagu · 2026-04-08T00:36:44Z

+
+# Exclude C++ headers, cmake configs, static libs, and binaries from the wheel.
+# Only the Python package (faiss/) and bundled shared libs remain.
+wheel.exclude = ["include/**", "share/**", "lib/**", "bin/**"]


The list misses lib64, and Linux wheels get duplicated libfaiss.so files.

tar tvf faiss_cpu-1.14.1-cp310-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl | grep libfaiss

-rwxr-xr-x 0 0 0 9798425 Apr 6 21:41 faiss/libfaiss.so -rwxr-xr-x 0 0 0 9798425 Apr 6 21:41 lib64/libfaiss.so

I have fixed this issue. Thank you!

…kresearch#4862) Summary: Add official pip wheel packaging for `faiss-cpu`, enabling `pip install faiss-cpu` from PyPI. Builds binary wheels across Linux x86_64/aarch64, macOS arm64/x86_64, and Windows x86_64 for Python 3.10-3.14. Linux and macOS use Python Stable ABI (abi3) — a single cp310-abi3 wheel per platform works on all Python 3.10+ versions without per-version rebuilds. **BLAS strategy:** - Linux (x86_64 and aarch64): Threaded OpenBLAS (`libopenblaso`). The CMake build prefers `libopenblaso` (OpenMP-threaded) or `libopenblasp` (pthreads-threaded) over the serial default on RHEL/Fedora-based manylinux containers. - macOS: Apple Accelerate (found automatically). - Windows: Prebuilt OpenBLAS from GitHub releases (v0.3.30). **SIMD optimization strategy:** - Linux and macOS: Dynamic Dispatch (`FAISS_OPT_LEVEL=dd`) compiles SIMD-hot kernel files at multiple ISA levels (generic + AVX2 + AVX-512 on x86_64, NEON + SVE on aarch64) into a single library. Runtime CPUID detection selects the optimal code path automatically. - Windows: MSVC does not support DD, so uses `FAISS_OPT_LEVEL=generic`. Shared libraries (`BUILD_SHARED_LIBS=ON`) are used to avoid MSVC's single-pass static archive scanning issue. **New files:** - `pyproject.toml`: scikit-build-core build backend config with cibuildwheel settings. Version is dynamically extracted from CMakeLists.txt. Enables abi3 on Linux/macOS via SWIG 4.2+ and NumPy 2.0+ Limited API support. Enables LTO and dead-code stripping for smaller wheel sizes. - `.github/workflows/build-pip.yml`: CI workflow using cibuildwheel to build binary wheels on 5 platform runners. All GitHub Actions are pinned to commit SHAs for supply chain security. Publishes wheels to PyPI via OIDC trusted publishers on tag push. - `THIRD_PARTY_NOTICES`: License notices for bundled dependencies (OpenBLAS, LLVM OpenMP). - `tests/test_wheel_smoke.py`: 12-test smoke suite validating import, OpenMP, BLAS (FlatL2/FlatIP), index factory (IVF+PQ), HNSW, serialization roundtrip, GC safety, contrib imports, and SIMD level detection. **Modified files:** - `faiss/CMakeLists.txt`: Added threaded OpenBLAS fallback — prefers `libopenblaso` (OpenMP-threaded) or `libopenblasp` (pthreads-threaded) over the serial default on RHEL/Fedora-based manylinux containers. - `python/CMakeLists.txt`: Added `install()` targets for scikit-build-core wheel packaging, including all Dynamic Dispatch SWIG variants (avx2/avx512/avx512_spr/sve). Added abi3 support: conditionally uses `Python::SABIModule`, defines `Py_LIMITED_API`, and sets `.abi3.so` suffix when `SKBUILD_SABI_VERSION` is set. - `.github/workflows/build.yml`: Wired `build-pip.yml` into the root CI trigger so pip builds run alongside conda builds. - `tests/BUCK`: Added `python_pytest` target for the smoke test. Reviewed By: mnorris11 Differential Revision: D95258115

meta-codesync · 2026-05-01T22:29:28Z

This pull request has been merged in 585ba79.

meta-cla Bot added the CLA Signed label Mar 4, 2026

meta-codesync Bot added fb-exported meta-exported labels Mar 4, 2026

alibeklfc force-pushed the export-D95258115 branch from f227da1 to e33de6c Compare March 4, 2026 21:12

alibeklfc force-pushed the export-D95258115 branch from e33de6c to 3ad6165 Compare March 4, 2026 21:23

alibeklfc force-pushed the export-D95258115 branch from 3ad6165 to 786dfab Compare March 4, 2026 21:34

alibeklfc force-pushed the export-D95258115 branch from 786dfab to 2eda6b4 Compare March 4, 2026 21:49

alibeklfc force-pushed the export-D95258115 branch from 2eda6b4 to 6506c6a Compare March 4, 2026 22:27

alibeklfc force-pushed the export-D95258115 branch from 6506c6a to dc28e4f Compare March 4, 2026 22:38

alibeklfc force-pushed the export-D95258115 branch from dc28e4f to e579b8f Compare March 5, 2026 01:24

alibeklfc force-pushed the export-D95258115 branch from e579b8f to a61f4fd Compare March 5, 2026 01:47

alibeklfc force-pushed the export-D95258115 branch from a61f4fd to 7a273c2 Compare March 5, 2026 01:53

alibeklfc force-pushed the export-D95258115 branch from 7a273c2 to 61d917a Compare March 5, 2026 02:08

alibeklfc force-pushed the export-D95258115 branch from 61d917a to 4ec2d4f Compare March 5, 2026 02:22

alibeklfc force-pushed the export-D95258115 branch from 4ec2d4f to 983edc0 Compare March 5, 2026 02:31

alibeklfc force-pushed the export-D95258115 branch from 983edc0 to 45bb916 Compare March 5, 2026 02:49

alibeklfc force-pushed the export-D95258115 branch 2 times, most recently from e378593 to 242a16e Compare March 5, 2026 19:37

alibeklfc force-pushed the export-D95258115 branch from 242a16e to 5ae5f0d Compare March 5, 2026 20:38

alibeklfc force-pushed the export-D95258115 branch from 5ae5f0d to cd2c57d Compare March 5, 2026 20:50

alibeklfc force-pushed the export-D95258115 branch from cd2c57d to 7c0347d Compare March 5, 2026 21:56

alibeklfc force-pushed the export-D95258115 branch 2 times, most recently from 1657e14 to 6d58298 Compare March 6, 2026 05:26

alibeklfc force-pushed the export-D95258115 branch from 6d58298 to f1994e2 Compare March 6, 2026 07:18

alibeklfc force-pushed the export-D95258115 branch 2 times, most recently from 1b8df4a to f0d431f Compare March 6, 2026 20:21

alibeklfc force-pushed the export-D95258115 branch 2 times, most recently from 8b7e98f to 874592b Compare March 9, 2026 15:16

kyamagu reviewed Mar 10, 2026

View reviewed changes

kyamagu approved these changes Apr 8, 2026

View reviewed changes

Conversation

alibeklfc commented Mar 4, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meta-codesync Bot commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alibeklfc commented Apr 3, 2026

Uh oh!

kyamagu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meta-codesync Bot commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alibeklfc commented Mar 4, 2026 •

edited by meta-codesync Bot

Loading