WIP: doc changes for Github Artifacts

resources/github-actions.md

@@ -123,45 +123,59 @@ As an example, `rapids-print-env` is used to print common environment informatio
 
 ## Downloading CI Artifacts
 
-For NVIDIA employees with VPN access, artifacts from both pull-requests and branch builds can be accessed on [https://downloads.rapids.ai/](https://downloads.rapids.ai/).
+Artifacts from both pull-requests and branch builds can be accessed on the GitHub UI for the specific workflow run, found in the Actions tab of the repository.
 
-There is a link provided at the end of every C++ and Python build job where the build artifacts for that particular workflow run can be accessed.
+![](/assets/images/workflow-ui.png)
 
-![](/assets/images/downloads.png)
+There is also a link provided in the `Run actions/upload-artifact@v4` step of every C++ and Python build job where the build artifacts for that particular job can be accessed.
+
+![](/assets/images/downloads-github.png)
+
+This link downloads the required artifact as a zip file.
+
+This can also be done using the following `gh` CLI command to download the artifact. This command also unzips the artifact to the destination location:
+
+```sh
+gh run download <workflow-run-id> --repo <repo-name> --name <artifact-name> --dir <destination-directory>
+```
+
+For example, to download the artifact `rmm_conda_python_cuda11_py311_x86_64` from the workflow run ID `14437867406` on the `rmm` repository into the `/artifacts` directory, you can use this command:
+
+```sh
+gh run download 14437867406 --repo rapidsai/rmm --name rmm_conda_python_cuda11_py311_x86_64 --dir /artifacts
+```
 
 ## Using Conda CI Artifacts Locally
 
-The artifacts that result from running `conda build` are conda channels. RAPIDS' CI system then compresses these conda channels into tarballs and uploads them to [https://downloads.rapids.ai/](https://downloads.rapids.ai/).
+The artifacts that result from running `conda build` are conda channels. RAPIDS' CI system uploads these conda channels to GitHub Artifacts as zip files.
 
-The packages in the conda channel can be used by extracting the tarball to your local filesystem and using the resulting path in your conda commands.
+The packages in the conda channel can be used by downloading the artifact to your local filesystem using `gh` CLI and using the resulting path in your conda commands.
 
-For example, the following snippet will download a pull request artifact for `librmm` and install it into the active conda environment:
+For example, the following snippet will download an artifact for `librmm` from workflow run ID `14437867406` and install it into the active conda environment:
 
 ```sh
-wget https://downloads.rapids.ai/ci/rmm/pull-request/1376/5124d43/rmm_conda_cpp_cuda11_x86_64.tar.gz
-mkdir local_channel
-tar xzf rmm_conda_cpp_cuda11_x86_64.tar.gz -C local_channel/
+# Download and extract the artifact
+gh run download 14437867406 --repo rapidsai/rmm -name rmm_conda_cpp_cuda11_x86_64 --dir local_channel
 mamba install --channel file://local_channel --channel rapidsai-nightly --channel conda-forge --channel nvidia librmm
 ```
 
-Note that CI artifacts can only be downloaded while connected to the NVIDIA VPN.
+Note: Make sure you have the GitHub CLI (`gh`) installed and authenticated on your host machine to download artifacts from GitHub Artifacts. To download artifacts made for a specific PR, replace the workflow run ID with the ID of the run triggered by the PR branch.
 
 ## Using Wheel CI Artifacts Locally
 
-RAPIDS' CI system compresses the wheels that it builds into tarballs and uploads them to [https://downloads.rapids.ai/](https://downloads.rapids.ai/).
+RAPIDS' CI system compresses the wheels that it builds into zip files and uploads them to GitHub Artifacts.
 
-The wheels can be used by extracting the tarball to your local filesystem and using the resulting path in your pip commands.
+The wheels can be used by downloading the artifact to your local filesystem using `gh` CLI and using the resulting path in your pip commands.
 
-For example, the following snippet will download a pull request artifact for `librmm` and install it into the active conda environment:
+For example, the following snippet will download an artifact for `librmm` and install it into the active conda environment:
 
 ```sh
-wget https://downloads.rapids.ai/ci/rmm/pull-request/1376/5124d43/rmm_wheel_python_rmm_cu12_39_x86_64.tar.gz
-mkdir wheels
-tar xzf rmm_wheel_python_rmm_cu12_39_x86_64.tar.gz -C wheels/
-pip install wheels/rmm_cu12-24.2.0a1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+# Download and extract the artifact
+gh run download 14437867406 --repo rapidsai/rmm --name rmm_wheel_python_rmm_cu12_py312_x86_64 --dir wheels
+pip install wheels/rmm_cu12-25.6.0a23-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
 ```
 
-Note that CI artifacts can only be downloaded while connected to the NVIDIA VPN.
+Note: Make sure you have the GitHub CLI (`gh`) installed and authenticated on your host machine to download artifacts from GitHub Artifacts. To download artifacts made for a specific PR, replace the workflow run ID with the ID of the run triggered by the PR branch.
 
 ## Using Conda CI Artifacts in Other PRs
 
@@ -184,8 +198,8 @@ Add a new file called `ci/use_conda_packages_from_prs.sh`.
 # ci/use_conda_packages_from_prs.sh
 
 # download CI artifacts
-LIBRAFT_CHANNEL=$(rapids-get-pr-conda-artifact raft 1388 cpp)
-LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp)
+LIBRAFT_CHANNEL=$(rapids-get-pr-conda-artifact-github raft 1388 cpp)
+LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact-github rmm 1095 cpp)
 
 # make sure they can be found locally
 conda config --system --add channels "${LIBRAFT_CHANNEL}"
@@ -207,9 +221,9 @@ So, for example, Python testing jobs that use the `rmm` Python package also need
 # ci/use_conda_packages_from_prs.sh
 
 # download CI artifacts
-LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp)
-LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 cpp)
-RMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 python)
+LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact-github kvikio 224 cpp)
+LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact-github rmm 1223 cpp)
+RMM_CHANNEL=$(rapids-get-pr-conda-artifact-github rmm 1223 python)
 
 # make sure they can be found locally
 conda config --system --add channels "${LIBKVIKIO_CHANNEL}"
@@ -223,7 +237,7 @@ Then copy the following into every script in the `ci/` directory that is doing `
 source ./ci/use_conda_packages_from_prs.sh
 ```
 
-**Note:** By default `rapids-get-pr-conda-artifact` uses the most recent commit from the specified PR.
+**Note:** By default `rapids-get-pr-conda-artifact-github` uses the most recent commit from the specified PR.
 A commit hash from the dependent PR can be added as an optional 4th argument to pin testing to a specific commit.
 
 ## Using Wheel CI Artifacts in Other PRs
@@ -246,10 +260,10 @@ RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")
 
 # download wheels, store the directories holding them in variables
 LIBRMM_WHEELHOUSE=$(
-  RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 cpp
+  RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github rmm 1678 cpp
 )
 LIBRAFT_WHEELHOUSE=$(
-  RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2433 cpp
+  RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github raft 2433 cpp
 )
 
 # write a pip constraints file saying e.g. "whenever you encounter a requirement for 'librmm-cu12', use this wheel"
@@ -283,13 +297,13 @@ RAPIDS_PY_CUDA_SUFFIX=$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")
 
 # download wheels, store the directories holding them in variables
 LIBKVIKIO_WHEELHOUSE=$(
-  RAPIDS_PY_WHEEL_NAME="libkvikio_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact kvikio 510 cpp
+  RAPIDS_PY_WHEEL_NAME="libkvikio_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github kvikio 510 cpp
 )
 LIBRMM_WHEELHOUSE=$(
-  RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 cpp
+  RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github rmm 1678 cpp
 )
 RMM_WHEELHOUSE=$(
-  RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1678 python
+  RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact-github rmm 1678 python
 )
 
 # write a pip constraints file saying e.g. "whenever you encounter a requirement for 'librmm-cu12', use this wheel"
@@ -311,7 +325,7 @@ source ./ci/use_wheels_from_prs.sh
 As above, if any of the other CI scripts are already setting the environment variable `PIP_CONSTRAINT`, you may need to
 modify them slightly to ensure they **append to**, instead of **overwriting**, the constraints set up by `use_wheels_from_prs.sh`.
 
-**Note:** By default `rapids-get-pr-wheel-artifact` uses the most recent commit from the specified PR.
+**Note:** By default `rapids-get-pr-wheel-artifact-github` uses the most recent commit from the specified PR.
 A commit hash from the dependent PR can be added as an optional 4th argument to pin testing to a specific commit.
 
 ## Skipping CI for Commits

resources/reproducing-ci.md

@@ -51,12 +51,14 @@ docker run \
   -it \
   --gpus all \
   --pull=always \
-  --network=host \
+  --env GITHUB_TOKEN=$(gh auth token) \
   --volume $PWD:/repo \
   --workdir /repo \
   rapidsai/ci-conda:cuda11.8.0-ubuntu22.04-py3.10
 ```
 
+Note: The `--env GITHUB_TOKEN=$(gh auth token)` flag is required to authenticate with GitHub for accessing artifacts. This token is automatically generated from your GitHub CLI authentication and passed to the container. Make sure you have the GitHub CLI (`gh`) installed and authenticated on your host machine.
+
 Once the container has started, you can run any of the CI scripts inside of it:
 
 ```sh
@@ -83,11 +85,11 @@ The `docker` command above makes the follow assumptions:
 
 - Your current directory is the repository that you wish to troubleshoot
 - Your current directory has the same commit checked out as the pull-request whose jobs you are trying to debug
+- You have the GitHub CLI (`gh`) installed and authenticated on your host machine
 
 A few notes about the `docker` command flags:
 
 - Most of the RAPIDS conda builds occur on machines without GPUs. Only the tests require GPUs. Therefore, you can omit the `--gpus` flag when running local conda builds
-- The `--network` flag ensures that the container has access to the VPN connection on your host machine. VPN connectivity is required for test jobs since they need access to [downloads.rapids.ai](https://downloads.rapids.ai) for downloading build artifacts from a particular pull-request. This flag can be omitted for build jobs
 
 
 ## Additional Considerations
@@ -108,13 +110,14 @@ In RAPIDS CI workflows, the builds and tests occur on different machines.
 
 Machines without GPUs are used for builds, while the tests occur on machines with GPUs.
 
-Due to this process, the artifacts from the build jobs must be downloaded from [downloads.rapids.ai](https://downloads.rapids.ai) in order for the test jobs to run.
+Due to this process, the artifacts from the build jobs must be downloaded from GitHub Artifacts in order for the test jobs to run.
 
 In CI, this process happens transparently.
 
 Local builds lack the context provided by the CI environment and therefore will require input from the user in order to ensure that the correct artifacts are downloaded.
 
-Any time the `rapids-download-conda-from-s3` command (e.g. [here](https://github.com/rapidsai/cugraph/blob/b50850f0498e163e56b0374c1c64e551a5898f26/ci/test_python.sh#L22-L23)) is encountered in a local test run, the user will be prompted for any necessary environment variables that are missing.
+Any time the `rapids-download-conda-from-github` command (e.g. [here](https://github.com/rapidsai/cugraph/blob/b50850f0498e163e56b0374c1c64e551a5898f26/ci/test_python.sh#L22-L23)) is encountered in a local test run, the user will be prompted for any necessary environment variables that are missing.
+<!-- The reference to the file needs to be changed once the downloading from GH PRs are merged -->
 
 The screenshot below shows an example.
 
@@ -140,21 +143,21 @@ There are a few limitations to keep in mind when running CI scripts locally.
 
 ### Local Artifacts Cannot Be Uploaded
 
-Build artifacts from local jobs cannot be uploaded to [downloads.rapids.ai](https://downloads.rapids.ai).
+Build artifacts from local jobs cannot be uploaded to GitHub Artifacts.
 
 If builds are failing in CI, developers should fix the problem locally and then push their changes to a pull-request.
 
 Then CI jobs can run and the fixed build artifacts will be made available for the test job to download and use.
 
-To attempt a complete build and test workflow locally, you can manually update any instances of `CPP_CHANNEL` and `PYTHON_CHANNEL` that use `rapids-download-conda-from-s3` (e.g. [1](https://github.com/rapidsai/cuml/blob/dc38afc584154ebe7332d43f69e3913492f7a273/ci/build_python.sh#L14),[2](https://github.com/rapidsai/cuml/blob/dc38afc584154ebe7332d43f69e3913492f7a273/ci/test_python_common.sh#L22-L23)) with the value of the `RAPIDS_CONDA_BLD_OUTPUT_DIR` environment variable that is [set in our CI images](https://github.com/rapidsai/ci-imgs/blob/d048ffa6bfd672fa72f31aeb7cc5cf2363aff6d9/Dockerfile#L105).
+To attempt a complete build and test workflow locally, you can manually update any instances of `CPP_CHANNEL` and `PYTHON_CHANNEL` that use `rapids-download-conda-from-github` (e.g. [1](https://github.com/rapidsai/cuml/blob/dc38afc584154ebe7332d43f69e3913492f7a273/ci/build_python.sh#L14),[2](https://github.com/rapidsai/cuml/blob/dc38afc584154ebe7332d43f69e3913492f7a273/ci/test_python_common.sh#L22-L23)) with the value of the `RAPIDS_CONDA_BLD_OUTPUT_DIR` environment variable that is [set in our CI images](https://github.com/rapidsai/ci-imgs/blob/d048ffa6bfd672fa72f31aeb7cc5cf2363aff6d9/Dockerfile#L105).
 
 This value is used to set the `output_folder` of the `.condarc` file used in our CI images (see [docs](https://conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#specify-conda-build-build-folder-conda-build-3-16-3-output-folder)). Therefore, any locally built packages will end up in this directory.
 
 For example:
 
 ```sh
-# Replace all local uses of `rapids-download-conda-from-s3`
-sed -ri '/rapids-download-conda-from-s3/ s/_CHANNEL=.*/_CHANNEL=${RAPIDS_CONDA_BLD_OUTPUT_DIR}/' ci/*.sh
+# Replace all local uses of `rapids-download-conda-from-github`
+sed -ri '/rapids-download-conda-from-github/ s/_CHANNEL=.*/_CHANNEL=${RAPIDS_CONDA_BLD_OUTPUT_DIR}/' ci/*.sh
 
 # Run the sequence of build/test scripts
 ./ci/build_cpp.sh
@@ -165,11 +168,9 @@ sed -ri '/rapids-download-conda-from-s3/ s/_CHANNEL=.*/_CHANNEL=${RAPIDS_CONDA_B
 ./ci/build_docs.sh
 ```
 
-### VPN Access
-
-Currently, [downloads.rapids.ai](https://downloads.rapids.ai) is only available via the NVIDIA VPN.
+Similarly, you can manually update any instance of the environment variable `RAPIDS_WHEEL_BLD_OUTPUT_DIR` to set the destination location for newly built wheels. (e.g. [1](https://github.com/rapidsai/cudf/blob/05646df0b8dd4b69f7cfed6fbf9e882df795210c/ci/build_wheel_cudf.sh#L29),[2](https://github.com/rapidsai/cuml/blob/e02797cabbd54a328fb22a48b5f1f02c4b6c81ee/ci/build_wheel_cuml.sh#L38)).
 
-If you want to run any test jobs locally, you'll need to be connected to the VPN to download CI build artifacts.
+This value is set to `/tmp/wheelhouse` in our [CI images](https://github.com/rapidsai/ci-imgs/blob/adc9f61a0c9d37b21b9ce0a978681e406a00bc64/ci-wheel.Dockerfile#L27)
 
 ### Some Builds Rely on Versioning Information in Git Tags