[RFC] Fix the Python packaging layer around mori CMake build (scikit-build-core)

[carlushuang]

Context

mori's Python package imports torch in 13 modules, and its extension libmori_pybinds.so is built by CMake and shipped prebuilt through package_data. The C++ build is not the problem — the hand-rolled packaging layer between CMake and the wheel is, and every gap below is silent at build time.

Gaps

	evidence	consequence
torch undeclared	setup.py has no install_requires at all	nothing records which torch a wheel was built against
interpreter guessed	src/pybind/CMakeLists.txt:4 runs python3 -c ... off PATH, despite the comment saying "same Python that built PyTorch"; falls back to a literal /usr/local/bin/python3.10 (line 13)	configure under one interpreter, install under another, nothing notices
no ABI tag	CMake emits libmori_pybinds.so; convention is ...cpython-312-x86_64-linux-gnu.so	EXTENSION_SUFFIXES ends in a bare .so, so 3.11 matches a 3.12 build and dies on symbols. Worked around by loading via explicit path (python/mori/cpp/__init__.py:43)
pybind11 unpinned	src/pybind/CMakeLists.txt:30 uses pybind11.get_include()	the standalone copy drifts from torch's bundled one
wheel metadata	a prebuilt, interpreter- and ROCm-specific .so	tagged as though it were none of those things

CMake has to stay

find_package(mori) for external C++ users (CMakeLists.txt:342-346 installs include/mori plus mori-config.cmake, exporting mori::shmem, mori::ops, mori::io); the umbp_master / umbp_client / umbp_standalone_server / spdk_proxy executables; spdk, spdlog, gRPC and Protobuf; ~25 option() flags; and BUILD_XLA_FFI_OPS, which builds JAX support with no torch involved — routing mori's extension build through a torch-specific tool would make JAX support require torch.

Proposal

Adopt scikit-build-core as the PEP 517 backend. It drives the existing CMake unchanged and takes over exactly the hand-rolled parts: interpreter selection, ABI-tagged module names, wheel tags, dependency metadata. Declare torch as a dependency either way.

torch.utils.cpp_extension is worth using only for modules that genuinely link libtorch. Torch-agnostic ones, mori_pybinds included, should not be built that way.

Minimal alternative, if changing backend is too big a step: pin Python_EXECUTABLE and drop the 3.10 fallback, set SUFFIX from EXT_SUFFIX, prefer torch's pybind11 when importable, add install_requires.

Questions

1. Move the build backend, or fix by hand?
2. Should torch be a declared dependency? Thirteen modules import it.
3. Is src/pybind/torch_utils.hpp dead code? It includes <torch/torch.h> and defines ScalarTypeToHipDataType (line 54), but nothing compiles it, and it is the only C++ reference to torch in the tree.

[carlushuang]

Moved to #557 — the allocator design over CCO now has its own RFC, keeping this issue on packaging.