Conversation
yhmtsai
added
reg:ci-cd
yhmtsai
force-pushed
the
spack_job
branch
13 times, most recently
from
a3d12fd to
58c1e15
Compare
yhmtsai
added a commit
that referenced
this pull request
May 17, 2025
|
yhmtsai
force-pushed
the
spack_job
branch
2 times, most recently
from
faeb610 to
083f458
Compare
pratikvn
reviewed
Aug 27, 2025
Member
Author
pratikvn
approved these changes
Sep 4, 2025
pratikvn
left a comment
pratikvn
left a comment
Member
There was a problem hiding this comment.
Mostly LGTM! I am little concerned that so many jobs might overload the TUM system.
Comment on lines
+164
to
+192
| build/cuda120/openmpi/gcc/cuda/release/static: | ||
| extends: | ||
| - .build_and_test_tum_template | ||
| - .default_variables | ||
| - .full_test_condition | ||
| - .use_tum-nvidia | ||
| variables: | ||
| BUILD_CUDA: "ON" | ||
| BUILD_HWLOC: "OFF" | ||
| ENABLE_HALF: "ON" | ||
| BUILD_MPI: "ON" | ||
| BUILD_SHARED_LIBS: "OFF" | ||
| BUILD_TYPE: "Release" | ||
| MODULE_LOAD: "cmake/3.18.6 cuda/12.0.1 gcc/12.4.0 openmpi/4.1.8" | ||
|
|
||
| build/cuda122/openmpi/gcc/cuda/release/shared: | ||
| extends: | ||
| - .build_and_test_tum_template | ||
| - .default_variables | ||
| - .full_test_condition | ||
| - .use_tum-nvidia | ||
| variables: | ||
| BUILD_CUDA: "ON" | ||
| BUILD_HWLOC: "OFF" | ||
| ENABLE_HALF: "ON" | ||
| BUILD_TYPE: "Release" | ||
| MODULE_LOAD: "cmake/3.20.6 cuda/12.2.2 gcc/12.4.0 openmpi/5.0.7" | ||
|
|
||
| build/cuda124/mpich/gcc/cuda/release/shared: |
Member
There was a problem hiding this comment.
I feel so many jobs might overload the TUM system ? Maybe we can:
- Switch off MPI or CUDA wherever possible ?
- Not test multiple versions of the same CUDA major version ?
Member
Author
There was a problem hiding this comment.
It is not the scope of this pr.
We will allow a GPU can serve multiple instance.
also, we are working on another project to make the setup more widely cover with limited resource
yhmtsai
force-pushed
the
spack_job
branch
2 times, most recently
from
770d8bb to
3c3e805
Compare
side note: - add cuda 12.2 and do not compile bfloat16 below cuda 12.2 - cuda 11.8.0 starts to support 8.9 and 9.0 - current docker image does not support A770. set up another without docker for A770 - tum does not contain HWLOC yet
try disable cuda home turn off CMAKE_INSTALL_RPATH_USE_LINK_PATH to see what happens now
yhmtsai
added
1:ST:ready-to-merge
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enables more jobs running on our server and they are reusing the packages and spack from the system rather than building everything through spack.
MODULE_LOADandSPACK_LOADto load the package through the variables. They are simply expanded tomodule load ${MODULE_LOAD}andspack load ${SPACK_LOAD}if they contains context. cuda module from spack does not includeLD_LIBRARY_PATH, so the script contains extendingLD_LIBRARY_PATHbefore test_install.RPATH_USE_LINKonly helps pure cpu build but not for the hip part.This PR will only extend the job relying on the same linux version and the packages from system.
Loading the system packages on different linux version or distribution might still work but it is not the purpose of this PR.
TODO: