Skip to content

Add quadrants - #34661

Merged
traversaro merged 20 commits into
conda-forge:mainfrom
jeongseok-meta:add-quadrants-gpu
Aug 31, 2026
Merged

Add quadrants#34661
traversaro merged 20 commits into
conda-forge:mainfrom
jeongseok-meta:add-quadrants-gpu

Conversation

@jeongseok-meta

@jeongseok-meta jeongseok-meta commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Adds Quadrants 1.3.0 as a standalone staged recipe. This is split from #33293 so its native build and GPU backends can be reviewed independently.

Backend coverage:

  • Linux x86_64: CPU, NVIDIA CUDA, AMDGPU/ROCm, and Vulkan
  • Linux AArch64: CPU, NVIDIA CUDA, and Vulkan
  • Windows x86_64: CPU, NVIDIA CUDA, and Vulkan
  • macOS x86_64 and arm64: CPU and Metal

The Linux AArch64 CUDA backend has no architecture guard or compile-time dependency on the CUDA toolkit; it emits PTX through LLVM and loads the host NVIDIA driver at runtime. Upstream currently documents this combination as unsupported, so CI can validate compilation and packaging but not GPU execution. Windows and Intel macOS were enabled in response to review so their native CI jobs can establish whether the source build works there.

Source and dependency handling:

  • Uses the hashed GitHub release archive rather than a mutable Git checkout.
  • Supplies only the submodules that do not have usable conda-forge coverage on every required target as separately hashed sources.
  • Uses conda-forge packages for spdlog/fmt, Eigen, DLPack, SPIR-V Tools/Headers, Vulkan Headers, and volk.
  • Links shared LLVM on Unix instead of embedding the static component libraries. On Linux this reduces the extension from about 68 MiB to about 11 MiB and the final package to 5.48 MiB.
  • Includes the NVIDIA and ROCm device-library license texts for the bundled backend bitcode.

The build logic now lives in build.sh and build.bat, uses CPU_COUNT, and documents why a separate Clang executable is needed to emit embedded LLVM bitcode. AMDGPU is enabled on Linux x86_64 only because upstream's implementation currently hard-disables it on macOS and supports only Linux in its runtime loader.

Tests now include imports, pip check, backend payload checks, a CPU kernel smoke test, and selected upstream Python tests (test_smoke.py and test_range_loops) forced to the CPU backend.

GPU runtime packaging:

  • quadrants remains the actual package. Its ordinary CUDA kernels need only the host NVIDIA driver, so installing it does not automatically pull a CUDA toolkit.
  • quadrants-cuda is an opt-in noarch: generic convenience metapackage that pins to the matching Quadrants patch release and installs cuBLAS, cuSPARSE, and cuSOLVER for the optional linear-algebra APIs. Upstream PR #896 teaches Quadrants 1.3's dynamic loader to recognize the CUDA 13 cuBLAS SONAME while preserving its existing preference for PyTorch-provided CUDA 11/12 libraries.
  • quadrants-rocm is an opt-in noarch: generic convenience metapackage that pins to the matching Quadrants patch release and installs hip-runtime-amd >=5.2, matching upstream's documented minimum; that dependency currently limits installation to Linux x86_64.

Local validation:

  • conda-smithy recipe-lint --conda-forge recipes/quadrants
  • shellcheck recipes/quadrants/build.sh
  • Full native rattler-build Linux x86_64 build with Python 3.13
  • Import tests, pip check, CPU smoke test, and three selected upstream test cases: all passed
  • Actual CUDA kernel executed successfully on an NVIDIA RTX 3080 using the locally built conda package
  • CUDA 13.0 runtime validation passed on the RTX 3080 with cuBLAS 13.1.1, cuSPARSE 12.6.3, and cuSOLVER 12.0.4 by running a real sparse conjugate-gradient solve
  • Full Linux x86_64 multi-output build and tests passed against current conda-forge releases, including spdlog 1.17, fmt 12, Eigen 5, DLPack 1.3, and current SPIR-V/Vulkan packages
  • Both quadrants-cuda and quadrants-rocm build once as noarch: generic packages and passed explicit package tests
  • Rendered and solved Linux AArch64, macOS x86_64, macOS arm64, and Windows x86_64 outputs
  • Current-head staged-recipes CI on b67b914c12 passed all 12 checks: Linux, Windows, macOS, rerender, CUDA 12.9, CUDA 13.0, skip/status, staged-recipes linter, and conda-forge linter (Azure build 1577212)
  • git diff --check

Checklist

  • Title is meaningful.
  • Recipe uses the v1 recipe.yaml format.
  • License files are packaged.
  • Source archives are immutable and hash-verified.
  • Available vendored dependencies are replaced with conda-forge packages.
  • No static libraries are shipped.
  • Build number is 0.
  • The proposed maintainer is the PR author.

@conda-forge-admin

Copy link
Copy Markdown
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/quadrants/recipe.yaml) and found it was in an excellent condition.

@jeongseok-meta
jeongseok-meta marked this pull request as ready for review August 28, 2026 14:50
@jeongseok-meta

jeongseok-meta commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

@conda-forge/help-python-c, ready for review! The Quadrants recipe passed the complete matrix before the branch update. Current main is now merged cleanly as head 6d3a12b, with no changes under recipes/quadrants; fresh CI is running.

@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

@traversaro, this is the Quadrants split from #33293. Thanks again for offering to take a look.

Comment thread recipes/quadrants/run_test.py
Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
@traversaro

Copy link
Copy Markdown
Contributor

Thanks a lot for working on this! Quadrants is really an interesting library. I did a bit of research on this on my own on this (see https://chatgpt.com/share/6a92d65f-17b0-83eb-b8b5-f31edf899a58), so most of my inline comments are based on that.

Comment thread recipes/quadrants/conda_build_config.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml Outdated
@traversaro

Copy link
Copy Markdown
Contributor

One thing that I wonder is how to handle the gpu specific dependencies. quadrants has this really nice design in which a single package variant (as one in PyPI) can target vulkan, rocm and cuda, as the required library are then loaded at runtime. However, that means that if a user just install quadrants and needs cuda support, it needs to manually install the needed libraries.

I think a reasonable design could be to switch this recipe to be multiple output, and add the following outputs:

  • quadrants-cuda that run-depends the required dependencies to use quadrants with cuda
  • quadrants-rocm that run depends on the required dependencies to use quadrants with rocm (that is just hip-runtime-amd

I think for the vulkan support, probably it is small enough that we can just depend on it directly in the quadrants main package, but I do not have a super strong opinion on this.

This would permit easily for users to install the dependencies required to run on gpu if they want, but it would have a blind spot: a user that would just install quadrants on a system where __cuda is available, will end up in an environment without full CUDA support of quadrants (due to the missing dependencies). That is different behaviour from most of the cuda-enabled libraries such as pytorch, jax, warp, onnxruntime, where if you have __cuda available, the cuda-enabled package gets installed.

To deal with it, we could think of having a quadrant-base package that is the one that actually contains the quandrants file, and then having the quadrants output having two variants:

  • A cuda variant that depends on __cuda and quadrants-cuda, that has higher priority
  • A normal variant that just depends on quadrants-base

In this way a, users that install in an environment with __cuda avaialble would end up with both quadrants-base and quadrants-cuda being installed, while a user without __cuda will end up with just quadrants-base installed.

Comment thread recipes/quadrants/recipe.yaml Outdated
@traversaro

Copy link
Copy Markdown
Contributor
  • (upstream does not support CUDA on Linux ARM64)

It does not provide wheels, or there is something actually preventing it to work?

@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

@traversaro, on the GPU dependency/output question: I traced the runtime loaders and tested this in a fresh environment before deciding on the output layout.

The core CUDA backend only dynamically loads the NVIDIA driver (libcuda.so / nvcuda.dll). A real Quadrants CUDA kernel ran successfully on an RTX 3080 in a fresh environment whose filtered package list contained quadrants but no CUDA toolkit, cuBLAS, cuSPARSE, or cuSOLVER packages. The latter libraries are loaded lazily only by the conjugate-gradient/sparse APIs. ROCm is different: its core backend loads libamdhip64.so, so hip-runtime-amd is relevant there.

Given that evidence, I kept quadrants as the actual package for this initial recipe. Automatically selecting a CUDA meta-output from __cuda would install several large optional libraries even for users who only need ordinary JIT kernels. I think opt-in quadrants-cuda / quadrants-rocm convenience metapackages are a useful feedstock follow-up, while avoiding automatic __cuda selection. If you consider those outputs required for the initial merge, I can add them here.

@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

@traversaro, on Linux AArch64 CUDA: it appears to be an upstream support/documentation and wheel-availability limitation, not a hard source-code guard. The CUDA backend emits PTX through LLVM and dynamically loads the host driver; it has no compile-time CUDA toolkit dependency or AArch64 exclusion.

I therefore enabled CUDA for linux-aarch64. The AArch64 variant renders and solves for all supported Python versions and includes the CUDA payload checks. We cannot validate runtime execution in conda-forge CI without ARM NVIDIA hardware, so the PR still states that upstream currently documents this combination as unsupported rather than claiming hardware validation.

@traversaro

Copy link
Copy Markdown
Contributor

@traversaro, on the GPU dependency/output question: I traced the runtime loaders and tested this in a fresh environment before deciding on the output layout.

The core CUDA backend only dynamically loads the NVIDIA driver (libcuda.so / nvcuda.dll). A real Quadrants CUDA kernel ran successfully on an RTX 3080 in a fresh environment whose filtered package list contained quadrants but no CUDA toolkit, cuBLAS, cuSPARSE, or cuSOLVER packages. The latter libraries are loaded lazily only by the conjugate-gradient/sparse APIs. ROCm is different: its core backend loads libamdhip64.so, so hip-runtime-amd is relevant there.

Given that evidence, I kept quadrants as the actual package for this initial recipe. Automatically selecting a CUDA meta-output from __cuda would install several large optional libraries even for users who only need ordinary JIT kernels. I think opt-in quadrants-cuda / quadrants-rocm convenience metapackages are a useful feedstock follow-up, while avoiding automatic __cuda selection. If you consider those outputs required for the initial merge, I can add them here.

I see, thanks for the analysis! Give that, I agree in avoiding the quadrants-core/quadrants split, but as agreed please add quadrants-rocm and quadrants-cuda outputs, and for rocm please keep 5.2 as minimum version as documented in quadrants readme?

Comment thread recipes/quadrants/recipe.yaml Outdated
@jeongseok-meta

jeongseok-meta commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Implemented on the current head.

  • quadrants remains the actual package, without automatic __cuda selection.
  • quadrants-cuda pins to the matching Quadrants patch release and installs libcublas, libcusparse, and libcusolver for the optional CUDA linear-algebra paths. 30f2f93da2 adds CUDA 13 cuBLAS SONAME support while preserving the loader's existing CUDA 11/12 preference.
  • quadrants-rocm pins to the matching Quadrants patch release and installs hip-runtime-amd >=5.2, limited to Linux x86_64 where this upstream backend is supported.
  • The convenience metapackages are Python-independent and render once per applicable platform, while quadrants still builds for every supported Python version; see 32a9a27245.

Adding these outputs here, rather than immediately restructuring the generated feedstock, lets staged-recipes validate and create all three package names together.

The full Linux multi-output build and tests passed locally. Both convenience packages also passed explicit package tests, and their output sets render/solve correctly on Linux x86_64, Linux AArch64, Windows x86_64, macOS x86_64, and macOS arm64. Fresh CI is running now.

@traversaro

Copy link
Copy Markdown
Contributor

quadrants-cuda exact-pins the matching quadrants build and installs libcublas, libcusparse, and libcusolver for the optional CUDA linear-algebra paths. I constrained cuda-version \u003C13 because Quadrants 1.3's dynamic loader recognizes cuBLAS SONAMEs through version 12; ordinary CUDA kernels still only require the host driver.

This seems to a big an upstream bug we may want to solve/patch rather then just propagate in the metadata?

@jeongseok-meta

jeongseok-meta commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Agreed. I removed the metadata cap and patched the loader instead in 30f2f93da2.

The patch adds the CUDA 13 cuBLAS SONAME after the existing CUDA 11/12 choices, so the current PyTorch-compatible preference is preserved while libcublas.so.13 / cublas64_13.dll can be used when that is the installed runtime.

I validated more than loading: a fresh environment with quadrants-cuda, cuda-version 13.0, cuBLAS 13.1.1, cuSPARSE 12.6.3, and cuSOLVER 12.0.4 completed a real sparse conjugate-gradient solve on the RTX 3080 and matched NumPy. The full local multi-output build and existing tests also pass. Fresh CI is running on this head.

@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

@traversaro, thank you for the careful reviews and package-specific guidance here—you've contributed substantially to shaping this recipe. If you're interested, would you like to be listed as a co-maintainer of the resulting quadrants-feedstock?

No pressure at all; I would only add you after explicit confirmation. If you are interested, adding you to the staged recipe before feedstock creation would be the cleanest point to do it.

@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

All requested recipe changes are implemented on 32a9a27245, and the fresh CI run is fully green:

  • Linux, Windows, and macOS native builds
  • dedicated CUDA 12.9 and CUDA 13.0 builds
  • rerender, skip/status, staged-recipes linter, and conda-forge linter

The CUDA 13 loader fix is in 30f2f93da2, with both dedicated CUDA CI and the previously reported RTX 3080 conjugate-gradient runtime validation now supporting it.

I also resolved the now-outdated backend thread: the build logic uses build.sh/build.bat, AMDGPU is enabled on Linux x86_64, and Windows AMDGPU remains unavailable because upstream force-disables it and asserts a Linux-only runtime implementation.

@traversaro, when convenient, could you please take another look?

@traversaro

Copy link
Copy Markdown
Contributor

@traversaro, thank you for the careful reviews and package-specific guidance here—you've contributed substantially to shaping this recipe. If you're interested, would you like to be listed as a co-maintainer of the resulting quadrants-feedstock?

No pressure at all; I would only add you after explicit confirmation. If you are interested, adding you to the staged recipe before feedstock creation would be the cleanest point to do it.

Thanks for the asking, indeed I am interested in this package. However, I think we can discuss about this after the feedstock has been created, I would prefer to avoid approving and merging a recipe in which I am a maintainer.

Comment thread recipes/quadrants/recipe.yaml Outdated
Comment thread recipes/quadrants/recipe.yaml
Comment thread recipes/quadrants/recipe.yaml Outdated
@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

That makes sense. I will leave the staged recipe maintainer list unchanged and revisit this after the feedstock has been created. Thank you for being open to it.

@jeongseok-meta

Copy link
Copy Markdown
Contributor Author

All three remaining inline requests are addressed and the threads are resolved on b67b914.

Current-head CI is fully green: all 12 checks passed, including native Linux, Windows, and macOS; dedicated CUDA 12.9 and CUDA 13.0 builds; rerender; skip/status; and both linters. Azure build: https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_build/results?buildId=1577212

The PR description now carries this current-head evidence. As discussed, the staged recipe maintainer list remains unchanged; we can revisit co-maintenance after feedstock creation.

@traversaro, when convenient, could you please take another look?

@traversaro traversaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, great work!

@traversaro
traversaro merged commit a1ad5c3 into conda-forge:main Aug 31, 2026
14 checks passed
@jeongseok-meta
jeongseok-meta deleted the add-quadrants-gpu branch August 31, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants