Skip to content

Commit 964ee6b

Browse files
hcho3hcho3
authored
[CI] Pack R package tarball with pre-built xgboost.so (with GPU support) (#6827) (#6836)
* Add scripts for packaging R package with GPU-enabled libxgboost.so * [CI] Automatically build R package tarball * Add comments * Don't build tarball for pull requests * Update the installation doc
1 parent 04fedef commit 964ee6b

File tree

4 files changed

+182
-6
lines changed

4 files changed

+182
-6
lines changed

‎Jenkinsfile

+19
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ pipeline {
6565
'build-gpu-cuda10.1': { BuildCUDA(cuda_version: '10.1') },
6666
'build-gpu-cuda10.2': { BuildCUDA(cuda_version: '10.2', build_rmm: true) },
6767
'build-gpu-cuda11.0': { BuildCUDA(cuda_version: '11.0') },
68+
'build-gpu-rpkg': { BuildRPackageWithCUDA(cuda_version: '10.0') },
6869
'build-jvm-packages-gpu-cuda10.0': { BuildJVMPackagesWithCUDA(spark_version: '3.0.0', cuda_version: '10.0') },
6970
'build-jvm-packages': { BuildJVMPackages(spark_version: '3.0.0') },
7071
'build-jvm-doc': { BuildJVMDoc() }
@@ -264,6 +265,24 @@ def BuildCUDA(args) {
264265
}
265266
}
266267

268+
def BuildRPackageWithCUDA(args) {
269+
node('linux && cpu_build') {
270+
unstash name: 'srcs'
271+
def container_type = 'gpu_build_r_centos6'
272+
def docker_binary = "docker"
273+
def docker_args = "--build-arg CUDA_VERSION_ARG=10.0"
274+
if (env.BRANCH_NAME == 'master' || env.BRANCH_NAME.startsWith('release')) {
275+
sh """
276+
${dockerRun} ${container_type} ${docker_binary} ${docker_args} tests/ci_build/build_r_pkg_with_cuda.sh ${commit_id}
277+
"""
278+
echo 'Uploading R tarball...'
279+
path = ("${BRANCH_NAME}" == 'master') ? '' : "${BRANCH_NAME}/"
280+
s3Upload bucket: 'xgboost-nightly-builds', path: path, acl: 'PublicRead', includePathPattern:'xgboost_r_gpu_linux_*.tar.gz'
281+
}
282+
deleteDir()
283+
}
284+
}
285+
267286
def BuildJVMPackagesWithCUDA(args) {
268287
node('linux && mgpu') {
269288
unstash name: 'srcs'

‎doc/build.rst

+18-6
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,15 @@
22
Installation Guide
33
##################
44

5-
.. note:: Pre-built binary wheel for Python
5+
.. note:: Pre-built binary wheel for Python: now with GPU support
66

7-
If you are planning to use Python, consider installing XGBoost from a pre-built binary wheel, available from Python Package Index (PyPI). You may download and install it by running
7+
If you are planning to use Python, consider installing XGBoost from a pre-built binary wheel, to avoid the trouble of building XGBoost from the source. You may download and install it by running
88

99
.. code-block:: bash
1010
11-
# Ensure that you are downloading one of the following:
12-
# * xgboost-{version}-py2.py3-none-manylinux1_x86_64.whl
13-
# * xgboost-{version}-py2.py3-none-win_amd64.whl
1411
pip3 install xgboost
1512
16-
* The binary wheel will support GPU algorithms (`gpu_hist`) on machines with NVIDIA GPUs. Please note that **training with multiple GPUs is only supported for Linux platform**. See :doc:`gpu/index`.
13+
* The binary wheel will support the GPU algorithm (``gpu_hist``) on machines with NVIDIA GPUs. Please note that **training with multiple GPUs is only supported for Linux platform**. See :doc:`gpu/index`.
1714
* Currently, we provide binary wheels for 64-bit Linux, macOS and Windows.
1815
* Nightly builds are available. You can go to `this page
1916
<https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/list.html>`_, find the
@@ -23,6 +20,21 @@ Installation Guide
2320
2421
pip install <url to the wheel>
2522
23+
.. note:: (EXPERIMENTAL) Pre-built binary package for R: now with GPU support
24+
25+
If you are planning to use R, consider installing ``{xgboost}`` from a pre-built binary package, to avoid the trouble of building XGBoost from the source. The binary package will let you use the GPU algorithm (``gpu_hist``) out of the box, as long as your machine has NVIDIA GPUs.
26+
27+
Download the binary package from the Releases page. The file name will be of the form ``xgboost_r_gpu_linux_[version].tar.gz``. Then install XGBoost by running:
28+
29+
.. code-block:: bash
30+
31+
# Install dependencies
32+
R -q -e "install.packages(c('data.table', 'magrittr', 'jsonlite', 'remotes'))"
33+
# Install XGBoost
34+
R CMD INSTALL ./xgboost_r_gpu_linux.tar.gz
35+
36+
Currently, we provide the binary package for 64-bit Linux.
37+
2638

2739
****************************
2840
Building XGBoost from source

‎tests/ci_build/Dockerfile.gpu_build_r_centos6

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
ARG CUDA_VERSION_ARG
2+
FROM nvidia/cuda:$CUDA_VERSION_ARG-devel-centos6
3+
ARG CUDA_VERSION_ARG
4+
5+
# Environment
6+
ENV DEBIAN_FRONTEND noninteractive
7+
ENV DEVTOOLSET_URL_ROOT http://vault.centos.org/6.9/sclo/x86_64/rh/devtoolset-4/
8+
9+
COPY CentOS-Base.repo /etc/yum.repos.d/
10+
11+
# Install all basic requirements
12+
RUN \
13+
yum install -y epel-release && \
14+
yum -y update && \
15+
yum install -y tar unzip wget xz git patchelf readline-devel libX11-devel libXt-devel \
16+
xorg-x11-server-devel openssl-devel texlive-* && \
17+
yum install -y $DEVTOOLSET_URL_ROOT/devtoolset-4-gcc-5.3.1-6.1.el6.x86_64.rpm \
18+
$DEVTOOLSET_URL_ROOT/devtoolset-4-gcc-gfortran-5.3.1-6.1.el6.x86_64.rpm \
19+
$DEVTOOLSET_URL_ROOT/devtoolset-4-libquadmath-devel-5.3.1-6.1.el6.x86_64.rpm \
20+
$DEVTOOLSET_URL_ROOT/devtoolset-4-gcc-c++-5.3.1-6.1.el6.x86_64.rpm \
21+
$DEVTOOLSET_URL_ROOT/devtoolset-4-binutils-2.25.1-8.el6.x86_64.rpm \
22+
$DEVTOOLSET_URL_ROOT/devtoolset-4-runtime-4.1-3.sc1.el6.x86_64.rpm \
23+
$DEVTOOLSET_URL_ROOT/devtoolset-4-libstdc++-devel-5.3.1-6.1.el6.x86_64.rpm
24+
25+
ENV PATH=/opt/python/bin:/usr/local/ninja:/opt/software/packages/bin:/opt/R/3.3.0/bin:$PATH
26+
ENV LD_LIBRARY_PATH=/opt/software/packages/lib:/opt/R/3.3.0/lib64:$LD_LIBRARY_PATH
27+
ENV CC=/opt/rh/devtoolset-4/root/usr/bin/gcc
28+
ENV CXX=/opt/rh/devtoolset-4/root/usr/bin/c++
29+
ENV CPP=/opt/rh/devtoolset-4/root/usr/bin/cpp
30+
ENV F77=/opt/rh/devtoolset-4/root/usr/bin/gfortran
31+
32+
# A few packages have to be built from the source because CentOS 6 is a very old distribution and
33+
# the system packages are not sufficiently up-to-date to build R 3.3.0. We'll want to update to
34+
# CentOS 7 after the 1.4.0 release. Tracking issue: dmlc/xgboost#6791.
35+
#
36+
# Why choose an old Linux distro? This is so that the resulting xgboost.so is compatible with a
37+
# wide range of Linux OSes currently in operation. See https://www.python.org/dev/peps/pep-0571/
38+
RUN \
39+
wget https://zlib.net/fossils/zlib-1.2.5.tar.gz && \
40+
wget https://sourceware.org/pub/bzip2/bzip2-1.0.6.tar.gz && \
41+
wget http://tukaani.org/xz/xz-5.2.2.tar.gz && \
42+
wget https://ftp.pcre.org/pub/pcre/pcre-8.40.tar.gz && \
43+
wget https://www.openssl.org/source/old/1.0.0/openssl-1.0.0k.tar.gz && \
44+
wget --no-check-certificate https://curl.se/download/curl-7.47.1.tar.gz && \
45+
tar xf zlib-1.2.5.tar.gz && \
46+
tar xf bzip2-1.0.6.tar.gz && \
47+
tar xf xz-5.2.2.tar.gz && \
48+
tar xf pcre-8.40.tar.gz && \
49+
tar xf openssl-1.0.0k.tar.gz && \
50+
tar xf curl-7.47.1.tar.gz && \
51+
cd zlib-1.2.5 && \
52+
./configure --prefix=/opt/software/packages && \
53+
make -j$(nproc) && \
54+
make install && \
55+
cd ../bzip2-1.0.6 && \
56+
sed -i 's/CFLAGS=-Wall/CFLAGS=-fPIC -Wall/g' Makefile && \
57+
make -f Makefile-libbz2_so && \
58+
make clean && \
59+
make -j$(nproc) && \
60+
make -n install PREFIX=/opt/software/packages && \
61+
make install PREFIX=/opt/software/packages && \
62+
cd ../xz-5.2.2 && \
63+
./configure --prefix=/opt/software/packages && \
64+
make -j$(nproc) && \
65+
make install && \
66+
cd ../pcre-8.40 && \
67+
./configure --enable-utf8 --prefix=/opt/software/packages && \
68+
make -j$(nproc) && \
69+
make install && \
70+
cd ../curl-7.47.1 && \
71+
./configure --prefix=/opt/software/packages --with-ssl && \
72+
make -j$(nproc) && \
73+
make install && \
74+
export CFLAGS="-I/opt/software/packages/include" && \
75+
export LDFLAGS="-L/opt/software/packages/lib" && \
76+
cd .. && \
77+
# R 3.3.0
78+
wget -nv -nc https://cran.r-project.org/src/base/R-3/R-3.3.0.tar.gz && \
79+
tar xf R-3.3.0.tar.gz && \
80+
cd R-3.3.0 && \
81+
./configure --prefix=/opt/R/3.3.0 --enable-R-shlib && \
82+
make -j$(nproc) && \
83+
make install && \
84+
# Python
85+
wget -nv -nc -O Miniconda3.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
86+
bash Miniconda3.sh -b -p /opt/python && \
87+
/opt/python/bin/python -m pip install auditwheel && \
88+
# CMake
89+
wget -nv -nc https://cmake.org/files/v3.13/cmake-3.13.0-Linux-x86_64.sh --no-check-certificate && \
90+
bash cmake-3.13.0-Linux-x86_64.sh --skip-license --prefix=/usr && \
91+
# Ninja
92+
mkdir -p /usr/local && \
93+
cd /usr/local/ && \
94+
wget -nv -nc https://github.com/ninja-build/ninja/archive/v1.10.0.tar.gz --no-check-certificate && \
95+
tar xf v1.10.0.tar.gz && mv ninja-1.10.0 ninja && rm -v v1.10.0.tar.gz && \
96+
cd ninja && \
97+
/opt/python/bin/python ./configure.py --bootstrap
98+
99+
ENV GOSU_VERSION 1.10
100+
101+
# Install lightweight sudo (not bound to TTY)
102+
RUN set -ex; \
103+
wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-amd64" && \
104+
chmod +x /usr/local/bin/gosu && \
105+
gosu nobody true
106+
107+
# Default entry-point to use if running locally
108+
# It will preserve attributes of created files
109+
COPY entrypoint.sh /scripts/
110+
111+
WORKDIR /workspace
112+
ENTRYPOINT ["/scripts/entrypoint.sh"]

‎tests/ci_build/build_r_pkg_with_cuda.sh

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
set -e
3+
set -x
4+
5+
if [ "$#" -ne 1 ]
6+
then
7+
echo "Build the R package tarball with CUDA code. Usage: $0 [commit hash]"
8+
exit 1
9+
fi
10+
11+
commit_hash="$1"
12+
13+
make Rpack
14+
mv xgboost/ xgboost_rpack/
15+
16+
mkdir build
17+
cd build
18+
cmake .. -GNinja -DUSE_CUDA=ON -DR_LIB=ON
19+
ninja
20+
cd ..
21+
22+
rm xgboost
23+
# This super wacky hack is found in cmake/RPackageInstall.cmake.in and
24+
# cmake/RPackageInstallTargetSetup.cmake. This hack lets us bypass the normal build process of R
25+
# and have R use xgboost.so that we've already built.
26+
rm -v xgboost_rpack/configure
27+
rm -rfv xgboost_rpack/src
28+
mkdir -p xgboost_rpack/src
29+
cp -v lib/xgboost.so xgboost_rpack/src/
30+
echo 'all:' > xgboost_rpack/src/Makefile
31+
echo 'all:' > xgboost_rpack/src/Makefile.win
32+
mv xgboost_rpack/ xgboost/
33+
tar cvzf xgboost_r_gpu_linux_${commit_hash}.tar.gz xgboost/

0 commit comments

Comments
 (0)