Skip to content

Port all conda recipes to rattler-build #6440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Mar 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
ae89c27
feat(libcuml): port libcuml to rattler-build
gforsyth Feb 3, 2025
0a4555c
fix(libcuml): resolve all overlinking errors
gforsyth Mar 14, 2025
a68bed0
refactor(libcuml): remove old conda recipe and scripts
gforsyth Mar 14, 2025
b4fa3f4
fix(libcuml): disable binary prefix patching
gforsyth Mar 14, 2025
def6cc6
feat(cuml): port cuml recipe to rattler-build
gforsyth Mar 14, 2025
39f854e
feat(cuml-cpu): port cuml-cpu recipe to rattler-build
gforsyth Mar 14, 2025
e40fb57
refactor(cuml): remove old conda recipe and scripts
gforsyth Mar 14, 2025
5abc251
refactor(cuml-cpu): remove old conda recipe and scripts
gforsyth Mar 14, 2025
5a4fd8b
refactor(ci): use rattler-build for cpp and python conda ci builds
gforsyth Mar 14, 2025
a607378
chore: fix shellcheck error in accel test script
gforsyth Mar 14, 2025
1548b5b
Merge branch 'branch-25.04' into rattler-build
gforsyth Mar 17, 2025
e9cbe40
fix(cuml): disable import test
gforsyth Mar 17, 2025
56bf912
refactor(libcuml,cuml): make sccache prefix more granular
gforsyth Mar 17, 2025
2b492cd
fix(cuml-cpu): add missing secrets and env section
gforsyth Mar 17, 2025
fa4aa01
prepend `cuda` to part of sccache key prefix
gforsyth Mar 17, 2025
6847b50
Merge branch 'branch-25.04' into rattler-build
gforsyth Mar 17, 2025
b105e7b
chore: remove default values from recipe
gforsyth Mar 18, 2025
f6c8af7
Merge branch 'branch-25.04' into rattler-build
gforsyth Mar 19, 2025
099c435
fix(libcuml): use `host_version` for host deps
gforsyth Mar 19, 2025
97fc0d0
fix(libcuml): ignore `cudatoolkit` on cuda11
gforsyth Mar 19, 2025
43c5643
Merge branch 'branch-25.04' into rattler-build
gforsyth Mar 19, 2025
01c688b
chore(rattler): improve comment about build cache removal
gforsyth Mar 20, 2025
5005bb0
chore: remove extraneous zero-stats call
gforsyth Mar 20, 2025
6ad24e1
chore(rattler): fix copyright year
gforsyth Mar 20, 2025
9e57403
fix(cuml): skip libarrow gdb file and disable prefix patching
gforsyth Mar 20, 2025
5d1405f
Merge branch 'branch-25.04' into rattler-build
gforsyth Mar 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions ci/build_cpp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@

set -euo pipefail

rapids-configure-conda-channels

source rapids-configure-sccache

source rapids-date-string
Expand All @@ -17,9 +15,25 @@ rapids-logger "Begin cpp build"

sccache --zero-stats

RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry build \
conda/recipes/libcuml
RAPIDS_PACKAGE_VERSION=$(rapids-generate-version)
export RAPIDS_PACKAGE_VERSION

source rapids-rattler-channel-string

# --no-build-id allows for caching with `sccache`
# more info is available at
# https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build
rattler-build build --recipe conda/recipes/libcuml \
--experimental \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, haven't reviewed one of these PRs before... why are we opting into experimental stuff from rattler-build? What specifically do we need?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main usage of --experimental is the pre-build or cache-build option, otherwise we'd have to compile everything for each output. It's been marked experimental for a while, but we're confident it's not going anywhere (I suspect the main reason it's still gated is that upstream may want to change some names around).

Wolf and I were discussing how the phrase multiple output cache can cause confusion when paired with sccache and compilation caches -- so maybe pre-build would be a better name, but that will require breaking changes to the yaml spec.

--no-build-id \
--channel-priority disabled \
--output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \
"${RATTLER_CHANNELS[@]}"

sccache --show-adv-stats

# remove build_cache directory to avoid uploading the entire source tree
# tracked in https://github.com/prefix-dev/rattler-build/issues/1424
rm -rf "$RAPIDS_CONDA_BLD_OUTPUT_DIR"/build_cache

rapids-upload-conda-to-s3 cpp
43 changes: 33 additions & 10 deletions ci/build_python.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@

set -euo pipefail

rapids-configure-conda-channels

source rapids-configure-sccache

source rapids-date-string
Expand All @@ -19,27 +17,52 @@ rapids-logger "Begin py build"

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION)
export RAPIDS_PACKAGE_VERSION

# populates `RATTLER_CHANNELS` array
source rapids-rattler-channel-string

rapids-logger "Prepending channel ${CPP_CHANNEL} to RATTLER_CHANNELS"

RATTLER_CHANNELS=("--channel" "${CPP_CHANNEL}" "${RATTLER_CHANNELS[@]}")

sccache --zero-stats

# TODO: Remove `--no-test` flag once importing on a CPU
# node works correctly
RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \
--no-test \
--channel "${CPP_CHANNEL}" \
conda/recipes/cuml
rapids-logger "Building cuml"

# --no-build-id allows for caching with `sccache`
# more info is available at
# https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build
rattler-build build --recipe conda/recipes/cuml \
--experimental \
--no-build-id \
--channel-priority disabled \
--output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \
"${RATTLER_CHANNELS[@]}"

sccache --show-adv-stats

# Build cuml-cpu only in CUDA 12 jobs since it only depends on python
# version
RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
if [[ ${RAPIDS_CUDA_MAJOR} == "12" ]]; then
rapids-logger "Building cuml-cpu"

sccache --zero-stats

RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry build \
conda/recipes/cuml-cpu
rattler-build build --recipe conda/recipes/cuml-cpu \
--experimental \
--no-build-id \
--channel-priority disabled \
--output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \
"${RATTLER_CHANNELS[@]}"

sccache --show-adv-stats
fi

# remove build_cache directory to avoid uploading the entire source tree
# tracked in https://github.com/prefix-dev/rattler-build/issues/1424
rm -rf "$RAPIDS_CONDA_BLD_OUTPUT_DIR"/build_cache

rapids-upload-conda-to-s3 python
5 changes: 0 additions & 5 deletions conda/recipes/cuml-cpu/build.sh

This file was deleted.

51 changes: 0 additions & 51 deletions conda/recipes/cuml-cpu/meta.yaml

This file was deleted.

78 changes: 78 additions & 0 deletions conda/recipes/cuml-cpu/recipe.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Copyright (c) 2025, NVIDIA CORPORATION.
schema_version: 1

context:
version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }}
date_string: '${{ env.get("RAPIDS_DATE_STRING") }}'
py_version: ${{ env.get("RAPIDS_PY_VERSION") }}
py_buildstring: ${{ py_version | version_to_buildstring }}
head_rev: ${{ git.head_rev(".")[:8] }}

package:
name: cuml-cpu
version: ${{ version }}

source:
path: ../../..

build:
string: py${{ py_buildstring }}_${{ date_string }}_${{ head_rev }}
dynamic_linking:
overlinking_behavior: "error"
script:
content: |
./build.sh cuml-cpu -v
secrets:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
env:
CMAKE_C_COMPILER_LAUNCHER: ${{ env.get("CMAKE_C_COMPILER_LAUNCHER") }}
CMAKE_CXX_COMPILER_LAUNCHER: ${{ env.get("CMAKE_CXX_COMPILER_LAUNCHER") }}
CMAKE_GENERATOR: ${{ env.get("CMAKE_GENERATOR") }}
SCCACHE_BUCKET: ${{ env.get("SCCACHE_BUCKET") }}
SCCACHE_IDLE_TIMEOUT: ${{ env.get("SCCACHE_IDLE_TIMEOUT") }}
SCCACHE_REGION: ${{ env.get("SCCACHE_REGION") }}
SCCACHE_S3_USE_SSL: ${{ env.get("SCCACHE_S3_USE_SSL") }}
SCCACHE_S3_NO_CREDENTIALS: ${{ env.get("SCCACHE_S3_NO_CREDENTIALS") }}
SCCACHE_S3_KEY_PREFIX: cuml-cpu/${{ env.get("RAPIDS_CONDA_ARCH") }}

requirements:
build:
- ${{ compiler("c") }}
- ${{ compiler("cxx") }}
- ${{ stdlib("c") }}
- cmake ${{ cmake_version }}
- ninja
host:
- cython >=3.0.0
- pip
- python =${{ py_version }}
- rapids-build-backend>=0.3.0,<0.4.0.dev0
- scikit-build-core >=0.10.0
run:
- numpy >=1.23,<3.0a0
- python
- pandas
- packaging
- scikit-learn 1.5.*
- hdbscan>=0.8.39,<0.8.40
- umap-learn=0.5.6
- nvtx
ignore_run_exports:
by_name:
- cuda-cudart
- cuda-version
- if: cuda_major == "11"
then: cudatoolkit

tests:
- python:
imports:
- cuml
pip_check: false

about:
homepage: ${{ load_from_file("python/cuml/pyproject.toml").project.urls.Homepage }}
license: ${{ load_from_file("python/cuml/pyproject.toml").project.license.text }}
summary: cuML-CPU Library
5 changes: 0 additions & 5 deletions conda/recipes/cuml/build.sh

This file was deleted.

109 changes: 0 additions & 109 deletions conda/recipes/cuml/meta.yaml

This file was deleted.

Loading
Loading