Skip to content

Commit bf8de86

Browse files
verdurinverdurin
authored
Merge pull request #7 from boegel/eb510
add blog post on EasyBuild v5.1.0
2 parents 9fd5da1 + 145af02 commit bf8de86

File tree

1 file changed

+194
-0
lines changed

1 file changed

+194
-0
lines changed

‎docs/posts/2025/05/EasyBuild-5.1.0.md‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
---
2+
authors:
3+
- boegel
4+
date: 2025-05-27
5+
slug: easybuild-5.1.0
6+
hide:
7+
- navigation
8+
---
9+
10+
# EasyBuild v5.1.0
11+
12+
EasyBuild v5.1.0 was released on 26 May 2025.
13+
14+
This version includes a couple of changes and enhancements we want to briefly highlight in this blog post.
15+
16+
For a detailed overview of all changes included in EasyBuild v5.1.0, [see the release notes](https://docs.easybuild.io/release-notes/#release_notes_eb510).
17+
18+
<!-- more -->
19+
20+
---
21+
22+
23+
## Build summary
24+
25+
EasyBuild v5.1.0 will print a summary of the installations that were planned for a particular session,
26+
and indicate whether each installation was a *success*, which ones *failed*, and which installations were *skipped*
27+
(for example because a required dependency failed to install).
28+
29+
For example:
30+
31+
```
32+
$ eb matplotlib-3.9.2-gfbf-2024a.eb -r
33+
...
34+
== Build succeeded for 2 out of 4
35+
== Summary:
36+
* [SUCCESS] Python/3.12.3-GCCcore-13.3.0
37+
* [SUCCESS] SciPy-bundle/2024.05-gfbf-2024a
38+
* [FAILED] libpng/1.6.43-GCCcore-13.3.0
39+
* [SKIPPED] matplotlib/3.9.2-gfbf-2024a
40+
```
41+
42+
With this summary you can more easily assess the overall result of the EasyBuild session, and quickly focus on the failing installations.
43+
44+
---
45+
46+
## CUDA sanity check
47+
48+
Starting with EasyBuild v5.1.0, additional checks are being done during the sanity check step
49+
for installations in which CUDA is used as a dependency.
50+
51+
Binaries, libraries, and compiled Python modules are inspected by EasyBuild using the
52+
[`cuobjdump` binary utility](https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html)
53+
that comes with the CUDA SDK, to determine whether they have the expected *device and/or PTX code*.
54+
55+
CUDA device code is a binary file format for executables that can be run on NVIDIA GPUs.
56+
57+
PTX is an intermediate binary format for NVIDIA GPUs, which can be [Just-In-Time (JiT) compiled](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#just-in-time-compilation)
58+
to run on a specific NVIDIA GPU.
59+
60+
In short: EasyBuild verifies in the CUDA sanity check to what extent the hardware targets supported by the binaries,
61+
libraries, and compiled Python modules correspond with the CUDA compute capabilities that EasyBuild was configured
62+
to use (see the `cuda-compute-capabilities` configuration setting, and the `cuda_compute_capabilities` easyconfig parameter).
63+
64+
Various related configuration settings are supported since EasyBuild v5.1.0, which all start with `cuda-sanity-check`.
65+
They specify how strict EasyBuild should be when performing the CUDA sanity check:
66+
67+
* Does PTX code *need* to be available? (`cuda-sanity-check-accept-missing-ptx`, disabled by default);
68+
* Is device code for the specified CUDA compute capabilities required, or is equivalent PTX code sufficient?
69+
(`cuda-sanity-check-accept-ptx-as-devcode`, disabled by default);
70+
* Is device code for additional CUDA compute capabilities allowed? (`cuda-sanity-check-strict`, disabled by default)
71+
72+
For now, EasyBuild will by default only print and/or log warnings when the specified conditions are not met,
73+
unless the `cuda-sanity-check-error-on-failed-checks` configuration setting is enabled.
74+
75+
For installations in which CUDA is used as a (direct) dependency,
76+
the trace output for the sanity check step will include a brief summary like:
77+
```
78+
>> CUDA sanity check summary report:
79+
>> Number of CUDA files checked: 7
80+
>> Number of files missing one or more CUDA Compute Capabilities: 2
81+
>> (not running with --cuda-sanity-check-error-on-failed-checks, so not considered failures)
82+
>> Number of files with device code for more CUDA Compute Capabilities than requested: 3
83+
>> (not running with --cuda-sanity-check-error-on-failed-checks, so not considered failures)
84+
>> Number of files missing PTX code for the highest configured CUDA Compute Capability: 5
85+
>> (not running with --cuda-sanity-check-error-on-failed-checks, so not considered failures)
86+
>> You may consider rerunning with --cuda-sanity-check-accept-ptx-as-devcode to accept suitable PTX code instead of device code.
87+
>> You may consider running with --cuda-sanity-check-accept-missing-ptx to accept binaries missing PTX code for the highest configured CUDA Compute Capability.
88+
>> See build log for detailed lists of files not passing the CUDA Sanity Check
89+
```
90+
91+
For in-depth details on this, see the changes that were made in [`easybuild-framework` PR #4692](https://github.com/easybuilders/easybuild-framework/pull/4692/files).
92+
93+
---
94+
95+
## Data installations
96+
97+
EasyBuild v5.1.0 adds support for installing datasets as modules.
98+
99+
Sam Moors (VUB) gave a talk about this idea at EUM'24 (see [slides](https://users.ugent.be/~kehoste/eum24/019_eum24_datasets.pdf) + [recording](https://www.youtube.com/watch?v=13q_aKDDv9k&list=PLhnGtSmEGEQild9FmlP8Qmz9Csc_gOJKF&index=20&pp=gAQBiAQB)).
100+
101+
Datasets get special treatment in terms of where they are installed, and how their "source" files are managed.
102+
103+
For more information, see [https://docs.easybuild.io/datasets](https://docs.easybuild.io/datasets).
104+
105+
---
106+
107+
## Downloading pull request diff via GitHub API
108+
109+
The mechanism for determining the files that were changed in a pull request when `--from-pr` or
110+
`--include-easyblocks-from-pr` is used has been changed in EasyBuild v5.1.0, to use the GitHub API rather than downloading a `*.diff`
111+
file from `https://github.com`.
112+
113+
This was done to mitigate the impact of [changes made by GitHub to impose stricter rate
114+
limits](https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/)
115+
for the GitHub web interface, which can quickly lead to an HTTP error when downloading too frequently:
116+
```
117+
HTTP Error 429: Too Many Requests
118+
```
119+
120+
For more details, see [framework issue #4869](https://github.com/easybuilders/easybuild-framework/issues/4869) and [framework PR #4878](https://github.com/easybuilders/easybuild-framework/pull/4878).
121+
122+
---
123+
124+
## Cleaner output for sanity check
125+
126+
The output produced by the sanity check step has been cleaned up in EasyBuild v5.1.0,
127+
by replacing the standard (noisy) trace output for commands executed during the sanity check with a one-line message.
128+
129+
The output for the "import" tests that are run for every extension that is being installed is now a lot cleaner.
130+
It used to be multiple lines with way too much detail for each extension:
131+
132+
```
133+
>> running shell command:
134+
/software/Python/3.12.3-GCCcore-13.3.0/bin/python -c "import matplotlib"
135+
[started at: 2025-05-23 09:15:19]
136+
[working dir: /software/matplotlib/3.9.2-gfbf-2024a]
137+
[output and state saved to /tmp/eb-sec1umjh/run-shell-cmd-output/python-d3adb33f]
138+
>> command completed: exit 0, ran in < 1s
139+
```
140+
141+
It now is just a single line per extension:
142+
143+
```
144+
>> Extension sanity check command '/software/Python/3.12.3-GCCcore-13.3.0/bin/python -c "import matplotlib"': OK
145+
```
146+
147+
Likewise for the `pip check` command that is run in the sanity check step when Python packages are being installed.
148+
This used to result in output like:
149+
```
150+
== sanity checking...
151+
>> running shell command:
152+
python -m pip check
153+
[started at: 2025-05-23 10:26:11]
154+
[working dir: /tmp/easybuild/Python/3.13.1/GCCcore-14.2.0]
155+
[output and state saved to /tmp/eb-d_ec_z2f/run-shell-cmd-output/python-d3adb33f]
156+
>> command completed: exit 0, ran in < 1s
157+
```
158+
159+
Now, it's a lot shorter:
160+
161+
```
162+
== sanity checking...
163+
>> Check on requirements for installed Python packages with 'pip check': OK
164+
```
165+
166+
---
167+
168+
## 2025a common toolchains
169+
170+
Easyconfig files for the `2025a` update of the common toolchains `foss` and `intel` are included with EasyBuild v5.1.0.
171+
172+
Note that the `2024b` version of the common toolchains was skipped, mostly due to focusing on releasing EasyBuild
173+
v5.0.0 in the period in which the `2024b` update of the common toolchains was supposed to be defined.
174+
175+
For more details, see the [overview of common toolchains](https://docs.easybuild.io/common-toolchains/#common_toolchains_overview).
176+
177+
---
178+
179+
## Notable bug fixes
180+
181+
Beyond the highlights covered above, there are a couple of small bug fixes included in EasyBuild
182+
v5.1.0:
183+
184+
- The `pip check` command is now executed only *once* in the sanity check for an installation that involves Python packages, which makes sense since this is a global check (not for a specific Python package).
185+
- Earlier versions of EasyBuild included a silly bug that could lead to messages like "`Fetching files: 100% (4/3)`" being shown in
186+
the download progress bar. This silly problem has been resolved in EasyBuild v5.1.0.
187+
- Since EasyBuild v5.0.0, the path to the `lib` subdirectory of a software installation directory was being added to the `$CMAKE_LIBRARY_PATH` environment variable in environment module files generated
188+
by EasyBuild, while this was not the intention. This bug has been squashed in EasyBuild v5.1.0.
189+
- The `--ignore-test-failure` EasyBuild configuration setting was partially broken since EasyBuild v5.0.0: errors that
190+
were raised during the test step were not actually being ignored. This rendered `--ignore-test-failure` useless for
191+
ignoring a handful of failing tests when installing PyTorch, for example.
192+
This regression was fixed in EasyBuild v5.1.0.
193+
- Several small issues have been fixed in the custom easyblock for LLVM, which was thoroughly revised in EasyBuild
194+
v5.0.0.

0 commit comments

Comments
 (0)