Migrate static metadata to pyproject.toml

[allenwang28]

Summary:
Migrates Monarch to more modern Python packaging standards by moving static metadata to pyproject.toml, adopting a declarative config approach.

Ideally, everything is kept within pyproject.toml and is easily installed with pip install .

Concretely this means, getting rid of setup.py altogether and all *-requirements.txt. But there are a few challenges that I want to cover:

Why we need setup.py still:

setup.py cannot be fully eliminated because Monarch has dynamic build requirements:

1. C++ extensions that link against PyTorch (requires torch library paths and headers)
2. Rust extensions via setuptools-rust (requires LIBTORCH_LIB environment variables)
3. Dynamic CUDA detection and CXX11 ABI detection at build time
4. Platform-specific rpath configuration for linking
5. Environment variable overrides (MONARCH_PACKAGE_NAME, MONARCH_VERSION,
   USE_TENSOR_ENGINE)
6. Custom build commands (Clean command)

Why we still need build-requirements.txt in this change:

build-requirements.txt (mirroring [build-system.requires]) is required because:

1. setup.py detects torch paths at module import time (lines 99-103) BEFORE the build
2. This requires torch to be pre-installed in the build environment
3. torch cannot be declared in [build-system.requires] because it needs custom index URLs (e.g., --index-url https://download.pytorch.org/whl/nightly/cu126)
4. Therefore we use --no-build-isolation which then disables automatic installation of [build-system.requires], requiring manual installation via build-requirements.txt

To eliminate build-requirement.txt, this would require us to enable standard isolated builds. Concretely:

1. Refactor setup.py to move torch detection from module-level into build command methods (e.g., inside CustomBuildExt.run())
2. Delay all torch path detection until the build_ext command actually executes
3. This would allow build isolation to work because torch would be installed by the user before running pip install ., but build deps would be auto-installed

Changes:

• Add [build-system] configuration per PEP 517/518
• Move all static metadata to [project] section per PEP 621 (dependencies, authors, license, entry points, etc.)
• Migrate runtime dependencies from requirements.txt to pyproject.toml
• Migrate test dependencies to [project.optional-dependencies.test]
• Simplify setup.py to only contain dynamic configuration (torch detection, C++/Rust extensions, environment variables)
• Updates in CI and READMEs
• Migrates from python setup.py bdist_wheel to python -m build. Setup.py is being deprecated: https://packaging.python.org/en/latest/discussions/setup-py-deprecated/

Differential Revision: D89066231

[meta-codesync]

@allenwang28 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D89066231.