ci: run NCCL unit tests on SNL CI #179
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- NCCL is built from source, packaged into an OS-agnostic tarball & installed from that tarball. For now, I am unsure if this will actually work in the container. - Kokkos version is explicitly set to 4.7.01 (equivalent to master at the time of this commit). In the future, we hope to have a container that installs the latest versions of Kokkos/Open MPI/MPICH/NCCL and have it cached by GHA. Submitted CI jobs will just pull down this container image with all dependencies set up, and only build KokkosComm. - For now, KokkosComm's MPI and NCCL backends are configured/built/tested one after the other. This may be improved later on, as both are independent and could theoretically run concurrently.
dssgabriel
added
A-ci
cwpearson
added
SNL-CI-APPROVAL
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
cwpearson
removed
the
SNL-CI-APPROVAL
Signed-off-by: Carl Pearson <[email protected]>
cwpearson
force-pushed
the
ci/run-nccl-tests
branch
from
f97f811 to
cb44b75
Compare
cwpearson
reviewed
Nov 7, 2025
Collaborator
cwpearson
left a comment
cwpearson
left a comment
There was a problem hiding this comment.
Fixed up a few things. I'm seeing invalid datatype for Kokkos::complex<float> in MPI, but I'm not sure this PR is responsible. @dssgabriel thoughts?
Signed-off-by: Carl Pearson <[email protected]>
cwpearson
force-pushed
the
ci/run-nccl-tests
branch
from
0be51f7 to
55dc946
Compare
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
cwpearson
force-pushed
the
ci/run-nccl-tests
branch
from
55dc946 to
af5f435
Compare
Signed-off-by: Carl Pearson <[email protected]>
dssgabriel
force-pushed
the
ci/run-nccl-tests
branch
from
f71a381 to
2473c1f
Compare
Do not use MPI sessions, rely on MPI being initialized (done by `test_main.cpp`) and use `MPI_COMM_WORLD`. This is because OpenMPI 5.0.8 does not support Sessions. Also simplify NCCL init using their examples. Improve logging infrastructure. Signed-off-by: Gabriel Dos Santos <[email protected]>
dssgabriel
force-pushed
the
ci/run-nccl-tests
branch
from
2473c1f to
454db3c
Compare
Collaborator
Author
|
@cwpearson NCCL tests are passing on our system 🎉 Node configuration is:
I am still investigating the MPI tests failing on the |
Signed-off-by: Gabriel Dos Santos <[email protected]>
Signed-off-by: Gabriel Dos Santos <[email protected]>
dssgabriel
force-pushed
the
ci/run-nccl-tests
branch
4 times, most recently
from
1016eab to
81f8069
Compare
Signed-off-by: Gabriel Dos Santos <[email protected]>
dssgabriel
force-pushed
the
ci/run-nccl-tests
branch
from
81f8069 to
5b18715
Compare
cwpearson
added
SNL-CI-SPECIAL-APPROVAL
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Signed-off-by: Carl Pearson <[email protected]>
Collaborator
|
@dssgabriel and I identified a few more problems:
|
dssgabriel
commented
Nov 21, 2025
Collaborator
Author
dssgabriel
left a comment
dssgabriel
left a comment
There was a problem hiding this comment.
Looks good to me!
We will need to properly check error codes returned by all NCCL calls in the backend. I will open an issue for tracking that.
We should also consider implementing this for the MPI backend and develop a more robust solution for proper error handling.
cwpearson
approved these changes
Nov 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR enables running the NCCL backend unit tests on the SNL H100 CI workflow.
Some notes: