Port all conda recipes to `rattler-build` #6440

gforsyth · 2025-03-14T20:50:25Z

Description

Port all condabuild recipes over to use rattler-build instead.

Contributes to rapidsai/build-planning#47

To satisfy rattler-build, this changes all the licenses in the pyproject.toml files to the SPDX-compliant Apache-2.0 instead of Apache 2.0

copy-pr-bot · 2025-03-14T20:50:30Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

gforsyth · 2025-03-14T20:51:46Z

/ok to test

gforsyth · 2025-03-14T21:00:27Z

/ok to test

gforsyth · 2025-03-17T15:39:15Z

/ok to test

There isn't a GPU on the build worker, so we can't do an import test here

gforsyth · 2025-03-17T17:25:07Z

/ok to test

gforsyth · 2025-03-17T19:43:48Z

/ok to test

gforsyth · 2025-03-17T19:47:40Z

/ok to test

gforsyth · 2025-03-19T15:14:06Z

Currently waiting on #6400

gforsyth · 2025-03-20T13:59:19Z

Here are the finalized run dependencies for the cuda 11 cpp builds (posting this because we currently don't have this tested on cuda 11.4):

libcuml:

  │ │ Finalized run dependencies:
 │ │ ╭──────────────────┬────────────────────────────────────────────────────────╮
 │ │ │ Name             ┆ Spec                                                   │
 │ │ ╞══════════════════╪════════════════════════════════════════════════════════╡
 │ │ │ Run dependencies ┆                                                        │
 │ │ │ cuda-version     ┆ >=11,<12.0a0 (PC)                                      │
 │ │ │ cudatoolkit      ┆                                                        │
 │ │ │ libcublas        ┆ >=11.5.2.43,<=11.11.3.6                                │
 │ │ │ libcufft         ┆ >=10.5.0.43,<=10.9.0.58                                │
 │ │ │ libcurand        ┆ >=10.2.5.43,<=10.3.0.86                                │
 │ │ │ libcusolver      ┆ >=11.2.0.43,<=11.4.1.48                                │
 │ │ │ libcusparse      ┆ >=11.6.0.43,<=11.7.5.86                                │
 │ │ │ libcumlprims     ┆ 25.4.*                                                 │
 │ │ │ libcuvs          ┆ 25.4.*                                                 │
 │ │ │ librmm           ┆ 25.4.*                                                 │
 │ │ │ rapids-logger    ┆ 0.1.*                                                  │
 │ │ │ treelite         ┆ 4.4.1.*                                                │
 │ │ │ __glibc          ┆ >=2.28,<3.0.a0 (RE of [build: sysroot_linux-64])       │
 │ │ │ libgcc-ng        ┆ >=12 (RE of [cache-build: gcc_linux-64])               │
 │ │ │ libstdcxx-ng     ┆ >=12 (RE of [cache-build: gxx_linux-64])               │
 │ │ │ libgcc-ng        ┆ >=12 (RE of [cache-build: gxx_linux-64])               │
 │ │ │ sysroot_linux-64 ┆ >=2.17,<3.0a0 (RE of [cache-build: nvcc_linux-64])     │
 │ │ │ __glibc          ┆ >=2.28,<3.0.a0 (RE of [cache-build: sysroot_linux-64]) │
 │ │ ╰──────────────────┴────────────────────────────────────────────────────────╯

libcuml-tests

 │ │ Finalized run dependencies:
 │ │ ╭──────────────────┬────────────────────────────────────────────────────────╮
 │ │ │ Name             ┆ Spec                                                   │
 │ │ ╞══════════════════╪════════════════════════════════════════════════════════╡
 │ │ │ Run dependencies ┆                                                        │
 │ │ │ cuda-version     ┆ >=11,<12.0a0 (PC)                                      │
 │ │ │ libcuml          ┆ ==25.4.0a150 cuda11_250319_43c5643e (PS)               │
 │ │ │ cudatoolkit      ┆                                                        │
 │ │ │ __glibc          ┆ >=2.28,<3.0.a0 (RE of [build: sysroot_linux-64])       │
 │ │ │ libgcc-ng        ┆ >=12 (RE of [cache-build: gcc_linux-64])               │
 │ │ │ libstdcxx-ng     ┆ >=12 (RE of [cache-build: gxx_linux-64])               │
 │ │ │ libgcc-ng        ┆ >=12 (RE of [cache-build: gxx_linux-64])               │
 │ │ │ sysroot_linux-64 ┆ >=2.17,<3.0a0 (RE of [cache-build: nvcc_linux-64])     │
 │ │ │ __glibc          ┆ >=2.28,<3.0.a0 (RE of [cache-build: sysroot_linux-64]) │
 │ │ ╰──────────────────┴────────────────────────────────────────────────────────╯

jameslamb

I have some questions, but I strongly suspect most are just from my relative lack of familiarity with rattler-build and will have answers like "yeah yeah this is expected it's just how rattler-build works and we've already talked about this in previous migration PRs". So marking this "approve" from a packaging-codeowners perspective, so that you don't need to wait on a re-review from me.

To review this, after looking through the diff I also looked through logs for stuff like this:

dependency metadata looks roughly correct
versions numbers look correct (correctly include the git tag distance)
the packages built in conda-{cpp,python}-build jobs are actually getting installed in conda-*-test jobs
no significant warnings / errors in any builds or tests

All look ok to me!

Just 1 question... do we care about these warnings and errors for cuml? I'm not sure what they mean?

 ⚠ warning Overdepending against dask-cudf
 ⚠ warning Overdepending against joblib
 ⚠ warning Overdepending against cuda-cudart
 Compiling 1 .py files to .pyc
 ⚠ warning Error compiling "/tmp/cumlWaVN1x/share/gdb/auto-load/tmp/conda-bld-output/bld/rattler-build_cuml/host_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/libarrow.so.1900.1.0-gdb.py" to .pyc:
 ⚠ warning [Errno 2] No such file or directory: '/tmp/cumlWaVN1x/share/gdb/auto-load/tmp/conda-bld-output/bld/rattler-build_cuml/host_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/libarrow.so.1900.1.0-gdb.py'

(build link)

And these for cuml-cpu?

⚠ warning Overdepending against numpy
⚠ warning Overdepending against pandas
⚠ warning Overdepending against scikit-learn
⚠ warning Overdepending against hdbscan
⚠ warning Overdepending against umap-learn
⚠ warning Overdepending against nvtx

(build link)

jameslamb · 2025-03-20T14:06:04Z

ci/build_cpp.sh

+# more info is available at
+# https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build
+rattler-build build --recipe conda/recipes/libcuml \
+                    --experimental \


Sorry, haven't reviewed one of these PRs before... why are we opting into experimental stuff from rattler-build? What specifically do we need?

The main usage of --experimental is the pre-build or cache-build option, otherwise we'd have to compile everything for each output. It's been marked experimental for a while, but we're confident it's not going anywhere (I suspect the main reason it's still gated is that upstream may want to change some names around).

Wolf and I were discussing how the phrase multiple output cache can cause confusion when paired with sccache and compilation caches -- so maybe pre-build would be a better name, but that will require breaking changes to the yaml spec.

jameslamb · 2025-03-20T14:19:06Z

ci/build_cpp.sh


 sccache --show-adv-stats

+# remove build_cache directory


For my understanding... is this necessary because rattler-build puts some files into $RAPIDS_CONDA_BLD_OUTPUT_DIR and rapids-upload-conda-to-s3 is doing something like trying to zip up the entire contents of that directory?

I think it'd be helpful to modify this comment to answer the question "why?" instead of "what?"... it's clear to me from the code that this is removing a directory called build_cache, but not clear why.

Yeah, there's an upstream issue tracking this that I can also link in there.

jameslamb · 2025-03-20T14:22:23Z

ci/build_python.sh


 sccache --show-adv-stats
+sccache --zero-stats


Suggested change

sccache --zero-stats

This looks like it's unnecessary. It's wasted time on CUDA 11 builds (where no other compilation will happen after this line), and redundant with this existing call on CUDA 12 builds (where we compile cuml-cpu):

cuml/ci/build_python.sh

Line 37 in 84946a9

sccache --zero-stats

gforsyth · 2025-03-20T15:05:10Z

Thanks @jameslamb ! I'm going to push up fixes for the things you flagged.

We are currently not worrying about overdepending, partially because it's hard to trace out, and also because it's non-critical -- it usually means a dependency is being overspecified, but so long as those are proper duplicates (and not in conflict) then it's not a big issue.

We care a lot about overlinking errors, since those might point to missing libraries.

The libarrow warning is new to me -- I'm going to check and see if that shows up in the current conda-build logs -- we probably still want to open a tracking issue, but that would at least confirm it's not because of the port to rattler-build

gforsyth · 2025-03-20T15:15:23Z

Welp, I can't find a similar error in the conda build logs, so let me rebuild locally and see if I can either fix that, or confirm that it's a non-issue.

gforsyth · 2025-03-20T15:58:05Z

Ok, here's a diff of the contents of the cuml conda package between the most recent commit on this PR and the most recent push to rapidsai-nightly:

2,3d1
< info/files 
< info/git 
5d2
< info/has_prefix 
7d3
< info/licenses/LICENSE 
9d4
< info/recipe/build.sh 
11,20c6,16
< info/recipe/meta.yaml 
< info/recipe/meta.yaml.template 
< info/test/run_test.py 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/direct_url.json 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/INSTALLER 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/licenses/LICENSE 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/METADATA 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/RECORD 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/REQUESTED 
< lib/python3.12/site-packages/cuml-25.4.0a132.dist-info/WHEEL 
---
> info/recipe/recipe.yaml 
> info/recipe/rendered_recipe.yaml 
> info/recipe/variant_config.yaml 
> info/tests/tests.yaml 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/direct_url.json 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/INSTALLER 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/licenses/LICENSE 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/METADATA 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/RECORD 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/REQUESTED 
> lib/python3.12/site-packages/cuml-25.4.0a150.dist-info/WHEEL 
674a671
> share/gdb/auto-load/tmp/conda-bld-output/bld/rattler-build_cuml/host_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/libarrow.so.1900.1.0-gdb.py

All of the changes are expected because of different commit counts and recipe files -- the only other difference is this weird libarrow file that also has it's entire placeholder prefix path in place.

gforsyth · 2025-03-20T18:14:28Z

Ok, because that libarrow file doesn't ship in the current cuml packages, I've added it to an explicit exclude list in the recipe, which has the side-effect of also removing the warning from the build logs.

If we want to ship it, we can always add it back in, but I think it's just an artifact of the earlier compilation stage.

gforsyth · 2025-03-20T20:11:38Z

/merge

gforsyth added 9 commits March 14, 2025 15:40

feat(libcuml): port libcuml to rattler-build

ae89c27

fix(libcuml): resolve all overlinking errors

0a4555c

refactor(libcuml): remove old conda recipe and scripts

a68bed0

fix(libcuml): disable binary prefix patching

b4fa3f4

feat(cuml): port cuml recipe to rattler-build

def6cc6

feat(cuml-cpu): port cuml-cpu recipe to rattler-build

39f854e

refactor(cuml): remove old conda recipe and scripts

e40fb57

refactor(cuml-cpu): remove old conda recipe and scripts

5abc251

refactor(ci): use rattler-build for cpp and python conda ci builds

5a4fd8b

gforsyth added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Mar 14, 2025

github-actions bot added conda conda issue Cython / Python Cython or Python issue ci labels Mar 14, 2025

chore: fix shellcheck error in accel test script

a607378

Merge branch 'branch-25.04' into rattler-build

1548b5b

gforsyth added 2 commits March 17, 2025 13:18

fix(cuml): disable import test

e9cbe40

There isn't a GPU on the build worker, so we can't do an import test here

refactor(libcuml,cuml): make sccache prefix more granular

56bf912

fix(cuml-cpu): add missing secrets and env section

2b492cd

gforsyth and others added 2 commits March 17, 2025 15:46

prepend cuda to part of sccache key prefix

fa4aa01

Merge branch 'branch-25.04' into rattler-build

6847b50

gforsyth mentioned this pull request Mar 18, 2025

Replace conda-build with rattler-build rapidsai/build-planning#47

Open

43 tasks

gforsyth added 2 commits March 19, 2025 12:25

chore: remove default values from recipe

b105e7b

Merge branch 'branch-25.04' into rattler-build

f6c8af7

gforsyth marked this pull request as ready for review March 19, 2025 16:29

gforsyth requested review from a team as code owners March 19, 2025 16:29

gforsyth requested a review from AyodeAwe March 19, 2025 16:29

gforsyth and others added 3 commits March 19, 2025 16:01

fix(libcuml): use host_version for host deps

099c435

fix(libcuml): ignore cudatoolkit on cuda11

97fc0d0

Merge branch 'branch-25.04' into rattler-build

43c5643

jameslamb approved these changes Mar 20, 2025

View reviewed changes

jameslamb removed the request for review from AyodeAwe March 20, 2025 14:51

gforsyth added 4 commits March 20, 2025 14:12

chore(rattler): improve comment about build cache removal

01c688b

chore: remove extraneous zero-stats call

5005bb0

chore(rattler): fix copyright year

6ad24e1

fix(cuml): skip libarrow gdb file and disable prefix patching

9e57403

Merge branch 'branch-25.04' into rattler-build

5d1405f

rapids-bot bot merged commit d950b30 into rapidsai:branch-25.04 Mar 20, 2025
75 checks passed

gforsyth deleted the rattler-build branch March 20, 2025 20:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port all conda recipes to `rattler-build` #6440

Port all conda recipes to `rattler-build` #6440

gforsyth commented Mar 14, 2025

copy-pr-bot bot commented Mar 14, 2025

gforsyth commented Mar 14, 2025

gforsyth commented Mar 14, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 19, 2025

gforsyth commented Mar 20, 2025 •

edited

Loading

jameslamb left a comment

jameslamb Mar 20, 2025

gforsyth Mar 20, 2025

jameslamb Mar 20, 2025

gforsyth Mar 20, 2025

jameslamb Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

Port all conda recipes to rattler-build #6440

Port all conda recipes to rattler-build #6440

Conversation

gforsyth commented Mar 14, 2025

Description

copy-pr-bot bot commented Mar 14, 2025

gforsyth commented Mar 14, 2025

gforsyth commented Mar 14, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 17, 2025

gforsyth commented Mar 19, 2025

gforsyth commented Mar 20, 2025 • edited Loading

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb Mar 20, 2025

Choose a reason for hiding this comment

gforsyth Mar 20, 2025

Choose a reason for hiding this comment

jameslamb Mar 20, 2025

Choose a reason for hiding this comment

gforsyth Mar 20, 2025

Choose a reason for hiding this comment

jameslamb Mar 20, 2025

Choose a reason for hiding this comment

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

gforsyth commented Mar 20, 2025

Port all conda recipes to `rattler-build` #6440

Port all conda recipes to `rattler-build` #6440

gforsyth commented Mar 20, 2025 •

edited

Loading