-
Notifications
You must be signed in to change notification settings - Fork 70
Dockerfile - Support cuda12.8 for Blackwell arch #682
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 18 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
8d937f5
add docker build with cuda12.8 for blackwell support
dpower4 72dd661
missing ansible_runner pkg
dpower4 85905cb
update msccl to support build for sm_100
dpower4 6a51193
typo in Makefile
dpower4 eee6464
add blackwell arch for compiling micro bm binaries
dpower4 3f1af3c
Merge branch 'main' into feat/GB_12_8_build
polarG af675c2
Merge branch 'main' into feat/GB_12_8_build
dpower4 b416f42
Merge branch 'main' into feat/GB_12_8_build
dpower4 366af39
update cuda samples to v12.8
dpower4 3f0a0a8
add 12.8 build
dpower4 5d22706
add 12.8 build arm64 build
dpower4 c57907a
address PR comments. use cmake for cuda sampels
dpower4 fa2f4c6
update nccl
dpower4 c67864d
Merge branch 'microsoft:main' into feat/GB_12_8_build
dpower4 86e50b5
Merge branch 'main' into feat/GB_12_8_build
polarG f103d8e
add missing rsync pkg to docker 12.8
dpower4 709563e
Merge branch 'main' into feat/GB_12_8_build
polarG 8583ea5
add nccl build for archs 80,90
dpower4 3ee480e
fix typos, remove 12.4 arm docker build
dpower4 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
FROM nvcr.io/nvidia/pytorch:25.02-py3 | ||
|
||
# OS: | ||
# - Ubuntu: 24.04 | ||
# - OpenMPI: 4.1.7+ | ||
# - Docker Client: 20.10.8 | ||
# NVIDIA: | ||
# - CUDA: 12.8.0.38 | ||
# - cuDNN: 9.7.1.26 | ||
# - cuBLAS: 12.8.3.14 | ||
# - NCCL: v2.25.1 | ||
# - TransformerEngine 2.0 | ||
# Mellanox: | ||
# - MOFED_VERSION; 5.4-rdmacore39.0 | ||
# - HPC-X: v2.21.0-CUDA12.x | ||
# Intel: | ||
# - mlc: v3.11 | ||
|
||
LABEL maintainer="SuperBench" | ||
|
||
ENV DEBIAN_FRONTEND=noninteractive | ||
|
||
RUN apt-get update && \ | ||
apt-get install -y --no-install-recommends \ | ||
autoconf \ | ||
automake \ | ||
bc \ | ||
build-essential \ | ||
curl \ | ||
dmidecode \ | ||
ffmpeg \ | ||
git \ | ||
iproute2 \ | ||
jq \ | ||
libaio-dev \ | ||
libavcodec-dev \ | ||
libavformat-dev \ | ||
libavutil-dev \ | ||
libboost-program-options-dev \ | ||
libcap2 \ | ||
libcurl4-openssl-dev \ | ||
libnuma-dev \ | ||
libpci-dev \ | ||
libswresample-dev \ | ||
libncurses-dev \ | ||
libtool \ | ||
lshw \ | ||
python3-mpi4py \ | ||
net-tools \ | ||
nlohmann-json3-dev \ | ||
openssh-client \ | ||
openssh-server \ | ||
pciutils \ | ||
sudo \ | ||
util-linux \ | ||
vim \ | ||
wget \ | ||
rsync \ | ||
&& \ | ||
apt-get autoremove && \ | ||
apt-get clean && \ | ||
rm -rf /var/lib/apt/lists/* /tmp/* | ||
|
||
ARG NUM_MAKE_JOBS= | ||
ARG TARGETPLATFORM | ||
ARG TARGETARCH | ||
|
||
# Install Docker | ||
ENV DOCKER_VERSION=20.10.8 | ||
RUN TARGETARCH_HW=$(uname -m) && \ | ||
wget -q https://download.docker.com/linux/static/stable/${TARGETARCH_HW}/docker-${DOCKER_VERSION}.tgz -O docker.tgz && \ | ||
tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/ && \ | ||
rm docker.tgz | ||
|
||
# Update system config | ||
RUN mkdir -p /root/.ssh && \ | ||
touch /root/.ssh/authorized_keys && \ | ||
mkdir -p /var/run/sshd && \ | ||
sed -i "s/[# ]*PermitRootLogin prohibit-password/PermitRootLogin yes/" /etc/ssh/sshd_config && \ | ||
sed -i "s/[# ]*PermitUserEnvironment no/PermitUserEnvironment yes/" /etc/ssh/sshd_config && \ | ||
sed -i "s/[# ]*Port.*/Port 22/" /etc/ssh/sshd_config && \ | ||
echo "* soft nofile 1048576\n* hard nofile 1048576" >> /etc/security/limits.conf && \ | ||
echo "root soft nofile 1048576\nroot hard nofile 1048576" >> /etc/security/limits.conf | ||
|
||
# Install OFED | ||
ENV OFED_VERSION=24.10-1.1.4.0 | ||
RUN TARGETARCH_HW=$(uname -m) && \ | ||
cd /tmp && \ | ||
wget -q https://content.mellanox.com/ofed/MLNX_OFED-${OFED_VERSION}/MLNX_OFED_LINUX-${OFED_VERSION}-ubuntu24.04-${TARGETARCH_HW}.tgz && \ | ||
tar xzf MLNX_OFED_LINUX-${OFED_VERSION}-ubuntu24.04-${TARGETARCH_HW}.tgz && \ | ||
MLNX_OFED_LINUX-${OFED_VERSION}-ubuntu24.04-${TARGETARCH_HW}/mlnxofedinstall --user-space-only --without-fw-update --without-ucx-cuda --force --all && \ | ||
rm -rf /tmp/MLNX_OFED_LINUX-${OFED_VERSION}* | ||
|
||
# Install HPC-X | ||
ENV HPCX_VERSION=v2.21 | ||
RUN TARGETARCH_HW=$(uname -m) && \ | ||
cd /opt && \ | ||
rm -rf hpcx && \ | ||
wget https://content.mellanox.com/hpc/hpc-x/${HPCX_VERSION}/hpcx-${HPCX_VERSION}-gcc-doca_ofed-ubuntu24.04-cuda12-${TARGETARCH_HW}.tbz -O hpcx.tbz && \ | ||
tar xf hpcx.tbz && \ | ||
mv hpcx-${HPCX_VERSION}-gcc-doca_ofed-ubuntu24.04-cuda12-${TARGETARCH_HW} hpcx && \ | ||
rm hpcx.tbz | ||
|
||
# Installs specific to amd64 platform | ||
RUN if [ "$TARGETARCH" = "amd64" ]; then \ | ||
# Install Intel MLC | ||
cd /tmp && \ | ||
wget -q https://downloadmirror.intel.com/793041/mlc_v3.11.tgz -O mlc.tgz && \ | ||
tar xzf mlc.tgz Linux/mlc && \ | ||
cp ./Linux/mlc /usr/local/bin/ && \ | ||
rm -rf ./Linux mlc.tgz && \ | ||
# Install AOCC compiler | ||
wget https://download.amd.com/developer/eula/aocc-compiler/aocc-compiler-4.0.0_1_amd64.deb && \ | ||
apt install -y ./aocc-compiler-4.0.0_1_amd64.deb && \ | ||
rm -rf aocc-compiler-4.0.0_1_amd64.deb && \ | ||
# Install AMD BLIS | ||
wget https://download.amd.com/developer/eula/blis/blis-4-0/aocl-blis-linux-aocc-4.0.tar.gz && \ | ||
tar xzf aocl-blis-linux-aocc-4.0.tar.gz && \ | ||
mv amd-blis /opt/AMD && \ | ||
rm -rf aocl-blis-linux-aocc-4.0.tar.gz; \ | ||
else \ | ||
echo "Skipping Intel MLC, AOCC and AMD Bliss installations for non-amd64 architecture: $TARGETARCH"; \ | ||
fi | ||
|
||
# Install NCCL 2.25.1 | ||
RUN cd /tmp && \ | ||
git clone -b v2.25.1-1 https://github.com/NVIDIA/nccl.git && \ | ||
cd nccl && \ | ||
make -j ${NUM_MAKE_JOBS} src.build \ | ||
NVCC_GENCODE="-gencode=arch=compute_100,code=sm_100 \ | ||
-gencode=arch=compute_100,code=sm_100 \ | ||
-gencode=arch=compute_100,code=sm_100" && \ | ||
dpower4 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
make install && \ | ||
rm -rf /tmp/nccl | ||
|
||
# Install UCX with multi-threading support | ||
ENV UCX_VERSION=1.18.0 | ||
RUN cd /tmp && \ | ||
wget https://github.com/openucx/ucx/releases/download/v${UCX_VERSION}-rc1/ucx-${UCX_VERSION}.tar.gz && \ | ||
tar xzf ucx-${UCX_VERSION}.tar.gz && \ | ||
cd ucx-${UCX_VERSION} && \ | ||
./contrib/configure-release-mt --prefix=/usr/local && \ | ||
make -j ${NUM_MAKE_JOBS} && \ | ||
make install | ||
|
||
ENV PATH="${PATH}" \ | ||
LD_LIBRARY_PATH="/usr/local/lib:/usr/local/mpi/lib:${LD_LIBRARY_PATH}" \ | ||
SB_HOME=/opt/superbench \ | ||
SB_MICRO_PATH=/opt/superbench \ | ||
ANSIBLE_DEPRECATION_WARNINGS=FALSE \ | ||
ANSIBLE_COLLECTIONS_PATH=/usr/share/ansible/collections | ||
|
||
RUN echo PATH="$PATH" > /etc/environment && \ | ||
echo LD_LIBRARY_PATH="$LD_LIBRARY_PATH" >> /etc/environment && \ | ||
echo SB_MICRO_PATH="$SB_MICRO_PATH" >> /etc/environment && \ | ||
echo "source /opt/hpcx/hpcx-init.sh && hpcx_load" | tee -a /etc/bash.bashrc >> /etc/profile.d/10-hpcx.sh | ||
|
||
# Add config files | ||
ADD dockerfile/etc /opt/microsoft/ | ||
|
||
WORKDIR ${SB_HOME} | ||
|
||
ADD third_party third_party | ||
RUN make -C third_party cuda_with_msccl | ||
|
||
ADD . . | ||
RUN python3 -m pip install --upgrade setuptools==70.3.0 && \ | ||
python3 -m pip install --no-cache-dir .[nvworker] && \ | ||
make cppbuild && \ | ||
make postinstall && \ | ||
rm -rf .git |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule nccl-tests
updated
15 files
+9 −4 | README.md | |
+1 −0 | doc/PERFORMANCE.md | |
+18 −3 | src/Makefile | |
+3 −5 | src/all_gather.cu | |
+2 −2 | src/all_reduce.cu | |
+5 −5 | src/alltoall.cu | |
+2 −2 | src/broadcast.cu | |
+81 −6 | src/common.cu | |
+2 −2 | src/common.h | |
+6 −6 | src/gather.cu | |
+3 −3 | src/hypercube.cu | |
+2 −2 | src/reduce.cu | |
+3 −5 | src/reduce_scatter.cu | |
+6 −6 | src/scatter.cu | |
+2 −2 | src/sendrecv.cu |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.