Migrate static metadata to pyproject.toml (meta-pytorch#2136)

Summary:

Migrates Monarch to more modern Python packaging standards by moving static metadata to pyproject.toml, adopting a declarative config approach.

Ideally, everything is kept within pyproject.toml and is easily installed with `pip install .`

Concretely this means, getting rid of setup.py altogether and all `*-requirements.txt`. But there are a few challenges that I want to cover:

Why we need setup.py still:

`setup.py` cannot be fully eliminated because Monarch has dynamic build requirements:
  1. C++ extensions that link against PyTorch (requires torch library paths and headers)
  2. Rust extensions via setuptools-rust (requires LIBTORCH_LIB environment variables)
  3. Dynamic CUDA detection and CXX11 ABI detection at build time
  4. Platform-specific rpath configuration for linking
  5. Environment variable overrides (MONARCH_PACKAGE_NAME, MONARCH_VERSION,
     USE_TENSOR_ENGINE)
  6. Custom build commands (Clean command)

Why we still need build-requirements.txt in this change:

build-requirements.txt (mirroring [build-system.requires]) is required because:
1. setup.py detects torch paths at module import time (lines 99-103) BEFORE the build
2. This requires torch to be pre-installed in the build environment
3. torch cannot be declared in [build-system.requires] because it needs custom index URLs (e.g., --index-url https://download.pytorch.org/whl/nightly/cu126)
4. Therefore we use `--no-build-isolation` which then disables automatic installation of `[build-system.requires]`, requiring manual installation via build-requirements.txt

To eliminate build-requirement.txt, this would require us to enable standard isolated builds. Concretely:
1. Refactor setup.py to move torch detection from module-level into build command methods (e.g., inside CustomBuildExt.run())
2. Delay all torch path detection until the build_ext command actually executes
3. This would allow build isolation to work because torch would be installed by the user before running `pip install .`, but build deps would be auto-installed

Changes:
- Add [build-system] configuration per PEP 517/518
- Move all static metadata to [project] section per PEP 621 (dependencies, authors, license, entry points, etc.)
- Migrate runtime dependencies from requirements.txt to pyproject.toml
- Migrate test dependencies to [project.optional-dependencies.test]
- Simplify setup.py to only contain dynamic configuration (torch detection, C++/Rust extensions, environment variables)
- Updates in CI and READMEs
- Migrates from `python setup.py bdist_wheel` to `python -m build`. Setup.py is being deprecated: https://packaging.python.org/en/latest/discussions/setup-py-deprecated/

Differential Revision: D89066231

Author: allenwang28

.github/workflows/build-cpu.yml

@@ -34,4 +34,7 @@ jobs:
         pip install -r build-requirements.txt
 
         # Build monarch (No tensor engine, CPU version)
-        USE_TENSOR_ENGINE=0 python setup.py bdist_wheel
+        USE_TENSOR_ENGINE=0 python -m build --no-isolation --wheel
+
+        # Fix permissions for artifact upload
+        chmod -R 755 dist/

.github/workflows/build-cuda.yml

@@ -42,4 +42,7 @@ jobs:
         setup_tensor_engine
 
         # Build monarch (CUDA version)
-        python setup.py bdist_wheel
+        python -m build --no-isolation --wheel
+
+        # Fix permissions for artifact upload
+        chmod -R 755 dist/

.github/workflows/wheels.yml

@@ -50,7 +50,10 @@ jobs:
         export MONARCH_PACKAGE_NAME="torchmonarch-nightly"
         export MONARCH_VERSION=$(date +'%Y.%m.%d')
 
-        python setup.py bdist_wheel
+        python -m build --no-isolation --wheel
+
+        # Fix permissions for artifact upload
+        chmod -R 755 dist/
 
         # hacky until the right distribution wheel can be made...
         find dist -name "*linux_x86_64.whl" -type f -exec bash -c 'mv "$1" "${1/linux_x86_64.whl/manylinux2014_x86_64.whl}"' _ {} \;

README.md

@@ -3,12 +3,18 @@
 **Monarch** is a distributed programming framework for PyTorch based on scalable
 actor messaging. It provides:
 
-1. Remote actors with scalable messaging: Actors are grouped into collections called meshes and messages can be broadcast to all members.
-2. Fault tolerance through supervision trees: Actors and processes form a tree and failures propagate up the tree, providing good default error behavior and enabling fine-grained fault recovery.
-3. Point-to-point RDMA transfers: cheap registration of any GPU or CPU memory in a process, with the one-sided transfers based on libibverbs
-4. Distributed tensors: actors can work with tensor objects sharded across processes
-
-Monarch code imperatively describes how to create processes and actors using a simple python API:
+1. Remote actors with scalable messaging: Actors are grouped into collections
+   called meshes and messages can be broadcast to all members.
+2. Fault tolerance through supervision trees: Actors and processes form a tree
+   and failures propagate up the tree, providing good default error behavior and
+   enabling fine-grained fault recovery.
+3. Point-to-point RDMA transfers: cheap registration of any GPU or CPU memory in
+   a process, with the one-sided transfers based on libibverbs
+4. Distributed tensors: actors can work with tensor objects sharded across
+   processes
+
+Monarch code imperatively describes how to create processes and actors using a
+simple python API:
 
 ```python
 from monarch.actor import Actor, endpoint, this_host
@@ -33,8 +39,9 @@ fut = trainers.train.call(step=0)
 fut.get()
 ```
 
-
-The [introduction to monarch concepts](https://meta-pytorch.org/monarch/generated/examples/getting_started.html) provides an introduction to using these features.
+The
+[introduction to monarch concepts](https://meta-pytorch.org/monarch/generated/examples/getting_started.html)
+provides an introduction to using these features.
 
 > ⚠️ **Early Development Warning** Monarch is currently in an experimental
 > stage. You should expect bugs, incomplete features, and APIs that may change
@@ -45,16 +52,21 @@ The [introduction to monarch concepts](https://meta-pytorch.org/monarch/generate
 
 ## 📖 Documentation
 
-View Monarch's hosted documentation [at this link](https://meta-pytorch.org/monarch/).
+View Monarch's hosted documentation
+[at this link](https://meta-pytorch.org/monarch/).
 
 ## Installation
-Note for running distributed tensors and RDMA, the local torch version must match the version that monarch was built with.
-Stable and nightly distributions require libmxl and libibverbs (runtime).
+
+Note for running distributed tensors and RDMA, the local torch version must
+match the version that monarch was built with. Stable and nightly distributions
+require libmxl and libibverbs (runtime).
 
 ## Fedora
+
 `sudo dnf install -y libibverbs rdma-core libmlx5 libibverbs-devel rdma-core-devel`
 
 ## Ubuntu
+
 `sudo apt install -y rdma-core libibverbs1 libmlx5-1 libibverbs-dev`
 
 ### Stable
@@ -64,21 +76,22 @@ Stable and nightly distributions require libmxl and libibverbs (runtime).
 torchmonarch stable is built with the latest stable torch.
 
 ### Nightly
+
 `pip install torchmonarch-nightly`
 
 torchmonarch-nightly is built with torch nightly.
 
 ### Build and Install from Source
 
-If you're building Monarch from source, you should be building it with the nightly PyTorch as well for ABI compatibility.
-
+If you're building Monarch from source, you should be building it with the
+nightly PyTorch as well for ABI compatibility.
 
 #### On Fedora distributions
 
 ```sh
 
 # Create and activate the conda environment
-conda create -n monarchenv python=3.10 -y
+conda create -n monarchenv python=3.12 -y
 conda activate monarchenv
 
 # Install nightly rust toolchain
@@ -101,10 +114,11 @@ conda update -n monarchenv --all -c conda-forge -y
 # If you are building with RDMA support, build monarch with `USE_TENSOR_ENGINE=1 pip install --no-build-isolation .` and dnf install the following packages
 sudo dnf install -y libibverbs rdma-core libmlx5 libibverbs-devel rdma-core-devel
 
-# Install build dependencies
-pip install -r torch-requirements.txt -r build-requirements.txt
-# Install test dependencies
-pip install -r python/tests/requirements.txt
+# Install PyTorch nightly (required for building)
+pip install -r torch-requirements.txt
+
+# Install build dependencies (required when using --no-build-isolation)
+pip install -r build-requirements.txt
 
 # Build and install Monarch
 pip install --no-build-isolation .
@@ -140,10 +154,11 @@ export CXX=clang++
 # Install the correct cuda and cuda-toolkit versions for your machine
 sudo apt install -y cuda-toolkit-12-8 cuda-12-8
 
-# Install build dependencies
-pip install -r torch-requirements.txt -r build-requirements.txt
-# Install test dependencies
-pip install -r python/tests/requirements.txt
+# Install PyTorch nightly (required for building)
+pip install -r torch-requirements.txt
+
+# Install build dependencies (required when using --no-build-isolation)
+pip install -r build-requirements.txt
 
 # Build and install Monarch (with tensor engine support)
 pip install --no-build-isolation .
@@ -161,27 +176,28 @@ pip list | grep monarch
 
 #### On non-CUDA machines
 
-You can also build Monarch to run on non-CUDA machines, e.g. locally on a MacOS system.
-
-Note that this does not support tensor engine, which is tied to CUDA and RDMA (via ibverbs).
+You can also build Monarch to run on non-CUDA machines, e.g. locally on a MacOS
+system.
 
+Note that this does not support tensor engine, which is tied to CUDA and RDMA
+(via ibverbs).
 
 ```sh
 
 # Create and activate the conda environment
-conda create -n monarchenv python=3.10 -y
+conda create -n monarchenv python=3.12 -y
 conda activate monarchenv
 
 # Install nightly rust toolchain
 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
 rustup toolchain install nightly
 rustup default nightly
 
-# Install build dependencies
+# Install PyTorch nightly (CPU version)
 pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu
+
+# Install build dependencies (required when using --no-build-isolation)
 pip install -r build-requirements.txt
-# Install test dependencies
-pip install -r python/tests/requirements.txt
 
 # Build and install Monarch
 USE_TENSOR_ENGINE=0 pip install --no-build-isolation .
@@ -192,10 +208,10 @@ USE_TENSOR_ENGINE=0 pip install --no-build-isolation -e .
 pip list | grep monarch
 ```
 
-
 ## Running examples
 
-Check out the `examples/` directory for demonstrations of how to use Monarch's APIs.
+Check out the `examples/` directory for demonstrations of how to use Monarch's
+APIs.
 
 We'll be adding more examples as we stabilize and polish functionality!
 
@@ -205,6 +221,7 @@ We have both Rust and Python unit tests. Rust tests are run with `cargo-nextest`
 and Python tests are run with `pytest`.
 
 Rust tests:
+
 ```sh
 # We use cargo-nextest to run our tests, as they can provide strong process isolation
 # between every test.
@@ -213,12 +230,15 @@ Rust tests:
 cargo install cargo-nextest --locked
 cargo nextest run
 ```
+
 cargo-nextest supports all of the filtering flags of "cargo test".
 
 Python tests:
+
 ```sh
-# Make sure to install test dependencies first
-pip install -r python/tests/requirements.txt
+# Install test dependencies if not already installed
+pip install -e '.[test]'
+
 # Run unit tests. consider -s for more verbose output
 pytest python/tests/ -v -m "not oss_skip"
 ```

build-requirements.txt

@@ -1,4 +1,15 @@
-setuptools
+# Build dependencies for Monarch
+#
+# This file mirrors [build-system.requires] in pyproject.toml for convenience.
+# It's needed when using --no-build-isolation (required because setup.py needs
+# torch installed before it runs, and torch can't be declared in pyproject.toml
+# due to needing custom index URLs like nightly/cu126).
+#
+# Modern build tools (pip, python -m build) will automatically install these
+# dependencies when using isolated builds, but Monarch requires --no-build-isolation.
+
+setuptools>=64
 setuptools-rust
 wheel
 numpy
+build

pyproject.toml

@@ -1,5 +1,56 @@
-[tool.pytest.ini_options]
+[build-system]
+requires = ["setuptools>=64", "setuptools-rust", "wheel", "numpy"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "monarch"
+version = "0.0.1"
+description = "Monarch: Single controller library"
+readme = "README.md"
+requires-python = ">=3.10"
+license = {text = "BSD-3-Clause"}
+authors = [
+    {name = "Meta", email = "[email protected]"}
+]
+
+dependencies = [
+    "pyzmq",
+    "requests",
+    "numpy",
+    "pyre-extensions",
+    "typing-extensions>=4.12",
+    "cloudpickle",
+    "torchx-nightly",
+    "lark",
+    "tabulate",
+    "opentelemetry-api",
+    "clusterscope",
+]
+
+[project.optional-dependencies]
+examples = [
+    "bs4",
+    "ipython",
+]
+test = [
+    "pytest",
+    "pytest-timeout",
+    "pytest-asyncio",
+    "pytest-xdist",
+    "pyright",
+]
+
+[project.scripts]
+monarch = "monarch.tools.cli:main"
+monarch_bootstrap = "monarch._src.actor.bootstrap_main:invoke_main"
 
+[tool.setuptools]
+packages = {find = {where = ["python"], exclude = ["tests*", "tests.*"]}}
+
+[tool.setuptools.package-dir]
+"" = "python"
+
+[tool.pytest.ini_options]
 markers = [
     "oss_skip: marks tests to skip in OSS CI",
 ]

python/tests/requirements.txt

@@ -49,7 +49,7 @@ setup_rust_toolchain() {
 # Install Python test dependencies
 install_python_test_dependencies() {
     echo "Installing test dependencies..."
-    pip install -r python/tests/requirements.txt
+    pip install -e '.[test]'
     dnf install -y rsync # required for code sync tests
 }
 

requirements.txt

@@ -11,7 +11,7 @@
 import sys
 import sysconfig
 
-from setuptools import Command, find_packages, setup
+from setuptools import Command, setup
 from setuptools.command.build_ext import build_ext
 from setuptools.extension import Extension
 
@@ -198,12 +198,6 @@ def run(self):
         subprocess.run(["cargo", "clean"])
 
 
-with open("requirements.txt") as f:
-    reqs = f.read()
-
-with open("README.md", encoding="utf8") as f:
-    readme = f.read()
-
 if sys.platform.startswith("linux"):
     # Always include the active env's lib (Conda-safe)
     conda_lib = os.path.join(sys.prefix, "lib")
@@ -278,35 +272,10 @@ def run(self):
 setup(
     name=package_name,
     version=package_version,
-    packages=find_packages(
-        where="python",
-        exclude=["python/tests.*", "python/tests"],
-    ),
-    package_dir={"": "python"},
-    python_requires=">= 3.10",
-    install_requires=reqs.strip().split("\n"),
-    extras_require={
-        "examples": [
-            "bs4",
-            "ipython",
-        ],
-    },
-    license="BSD-3-Clause",
-    author="Meta",
-    author_email="[email protected]",
-    description="Monarch: Single controller library",
-    long_description=readme,
-    long_description_content_type="text/markdown",
     ext_modules=[
         controller_C,
         common_C,
     ],
-    entry_points={
-        "console_scripts": [
-            "monarch=monarch.tools.cli:main",
-            "monarch_bootstrap=monarch._src.actor.bootstrap_main:invoke_main",
-        ],
-    },
     rust_extensions=rust_extensions,
     cmdclass={
         "build_ext": build_ext,