Replace conda-build with rattler-build

[vyasr]

RAPIDS currently builds conda packages in CI using conda-build. The rattler-build tool is a newer alternative. It is written in Rust, and should be faster than conda-build (I haven't seen any official benchmarks yet, though). It only supports a limited subset of the meta.yaml recipe format, but that subset is designed to still enable all the same features, just with a more limited syntax (see CEPS 13 and 14). conda-build overhead is nontrivial (I've never benchmarked it, but I know it can stretch into multiple minutes beyond the environment solve when doing local CI reproductions), and reducing that would be quite valuable for us in improving our CI turnaround. Moreover, switching to the more restricted syntax described in the above CEPs would be beneficial because it would convert our conda recipes into pure YAML rather than the extended YAML currently used by meta.yaml. That change is important because the YAML extensions currently in our recipe make it impossible to parse or write with standard YAML parsers, which is a big reason why we have struggled to do things like support meta.yaml files in rapids-dependency-file-generator.

We should do a PoC of replacing conda-build with rattler-build in one repo (preferably something reasonably complex like cudf or cugraph) to see what it would take to make this transition, and how much we would benefit.

rattler-build porting progress

• rmm (Port to rattler-build rmm#1796)
• raft (Port all conda recipes to rattler-build raft#2623)
• cudf (Port all conda recipes to rattler-build cudf#18054)
• kvikio (Port all conda recipes to rattler-build kvikio#678)
• cuml (Port all conda recipes to rattler-build cuml#6440) (merged 2025-03-20)
• cucim (Port all conda recipes to rattler-build & use strict channel priority cucim#864)
• cugraph (Port all conda recipes to rattler-build cugraph#4999)
• cugraph-ops
• cumlprims_mg (https://github.com/rapidsai/cumlprims_mg/pull/237)
• cuspatial (Port all conda recipes to rattler-build cuspatial#1555)
• cuvs (feat(libcuvs): port libcuvs to rattler-build cuvs#751) (merged 2025-03-19)
• cuxfilter (Port all conda recipes to rattler-build cuxfilter#669)
• dask-cuda (Port all conda recipes to rattler-build dask-cuda#1460)
• dependency-file-generator
• integration
• jupyterlab-nvdashboard
• ptxcompiler
• pynvjitlink
• rapids-cmake (refactor(conda): remove unused conda package and associated CI resources rapids-cmake#810)
• rapids-build-backend
• rapids-dask-dependency (refactor: port conda-build to rattler-build rapids-dask-dependency#93)
• rapids-metadata
• rapidsmpf
• ucx-py (Port all conda recipes to rattler-build ucx-py#1126)
• ucxx (Switch from conda-build to rattler-build ucxx#374)
• wholegraph

Few more:

• nx-cugraph feat(rattler): port conda build recipe to rattler build nx-cugraph#121
• cugraph-gnn feat(rattler): port conda build recipe to rattler build cugraph-gnn#178
• cuopt https://github.com/rapidsai/cuopt/pull/2420

Follow-up work

I'm going to track a few rattler-build followups here and update as they get resolved:

general

Look into rolling the sccache cache-busting fix for -fdebug-prefix-map into rapids-cmake (xref rapidsai/rapids-cmake#798 (comment))

cudf

• Add default build arg generation to rapids-rattler-channel-string #156
• Follow up and remove extra wrapping quotes in context once Values of context variables can have different types depending on the underlying shell prefix-dev/rattler-build#1451 is resolved
• Follow up and remove sed workaround for -fdebug-prefix-map once Cache build including output version in environment variables is causing sccache misses prefix-dev/rattler-build#1458 is resolved
• Remove cuda-nvcc-impl dependency once it is provided by numba-cuda (xref [BUG] Add proper dependencies to conda package NVIDIA/numba-cuda#146)
• Strip out secrets and env from cudf-polars recipe
• Test removing cuda-version from all pure Python packages (xref Properly support building pure Python packages #43)
• Confirm rapids-dask-dependency isn't needed by custreamz and then remove it
• Figure out if python-confluent-kafka should have run exports
• Confirm streamz is only a test dependency of custreamz and then remove it

rmm

• Add default build arg generation to rapids-rattler-channel-string #156
• Follow up and remove extra wrapping quotes in context once Values of context variables can have different types depending on the underlying shell prefix-dev/rattler-build#1451 is resolved

cuvs

• Relax/extend pin mkl=2023 to mkl>=2023 or similar
• combine cuvs-bench and cuvs-bench-cpu recipes

kvikio

• The libcufile dependencies are an absolute mess and we would probably benefit from adding a few variants at the intersection of architecture and cuda version Port all conda recipes to rattler-build kvikio#678 (comment)

[vyasr]

I now have two POCs of using rattler-build

• Build conda packages using rattler-build ucxx#248
• Replace conda-build with rattler-build cugraph#4551

Some notes:

• rattler-build requires strict channel priority (Allow to change channel priority prefix-dev/rattler-build#343) so we need to address Switch to using strict channel priority during RAPIDS builds #84 first
• Our general model of "build-once, install into separate outputs" requires a feature that rattler-build still marks as experimental, the multi-output cache. This doesn't personally concern me all that much since they are developing quite fast and I expect bug fixes to happen far more quickly than with conda-build, but it is something to be aware of.
• There are some issues around cache run exports that would be best to have addressed upstream before we adopt.
  • Update: This is now fixed in the latest release (0.19).

I've littered the PRs liberally with comments, so those should provide additional information.

[msarahan]

Nice! Looks like you got some pretty good speedup there - maybe around 5 minutes of speedup, from what I can see. Considering the whole build time was 8-10 minutes, saving 5 minutes is huge.

[vyasr]

Yup, similar speedups in absolute numbers for cugraph (smaller percentage, but still not bad to go from 28 mins to 22 mins. Also there's probably an extra 30 s in there for installing rattler-build itself in each job since it's not yet preinstalled in the image (but it will be).

[bdice]

I looked at the two POCs for ucxx and cugraph and I am supportive of this effort. It seems like there are a lot of steps to take first (like fixing strict priority conda builds, maybe altering librmm run exports, and maybe some rattler-build issues -- basically all the things that are marked in the comments) but overall I think this is mature enough that we can invest some time in this direction.

[vyasr]

FYI @wolfv @ruben-arts here's the follow-up to our discussions from SciPy 🙂

[vyasr]

We're actively discussing #84, which is IMO a blocker to making this change. We can work around it, but it requires a lot of extra boilerplate in recipes that I don't think it's worth proliferating.

[msarahan]

Adding repo tracker for progress tracking:

• rmm (Port to rattler-build rmm#1796)
• raft
• cudf (Port all conda recipes to rattler-build cudf#18054)
• kvikio
• cuml
• cucim
• cugraph (Replace conda-build with rattler-build cugraph#4551)
• cugraph-ops
• cumlprims_mg
• cuspatial
• cuvs
• cuxfilter
• dask-cuda
• rapids-cmake
• rapids-dask-dependency
• ucx-py
• ucxx(Build conda packages using rattler-build ucxx#248)
• wholegraph

[bdice]

We should re-evaluate this now that prefix-dev/rattler-build#343 is closed. I am not sure if strict channel priority is still a blocker for us to migrate to rattler-build. We need to see if prefix-dev/rattler-build#1211 solved the problems we were seeing.

cc: @msarahan @gforsyth @vyasr @jakirkham

[vyasr]

prefix-dev/rattler-build#343 is indeed the issue that was blocking us, and it looks like it was resolved about a month ago which is great news. It should allow us to proceed on both fronts in parallel. I think enabling strict channel priority is worthwhile enough that we may as well move forward with that as well now given that we have enough information about how we can do it and approval to do so. It should give additional improvements to both user experiences and our solve times.