Skip to content

EasyBuild 5.0 sync meetings

Jump to bottom
Kenneth Hoste edited this page Feb 24, 2025 · 2 revisions

2025-02-17

  • attending: Kenneth, Bart, Jasper, Alex, Mikael

Notes

2025-02-12

Merged PRs

framework

  • Adapt module show command run to cope with non-zero exit code for non-existing module (required for Environment Modules v5.5+ and Lmod 8.7.56+) (framework PR #4739)
  • Small improvements to ModuleLoadEnvironment (framework PR #4754)
  • enable non-blocking reads for streaming outputs (framework PR #4755)
  • Remove superflous is_string method (PR #4756)
  • add BOTH_TO_DIR state to LibSymlink (framework PR #4761)

easyblocks

Active PRs

  • Deprecate use of parallel easyconfig parameter and fix updating the template value (framework PR #4580)
  • add module-search-path-headers configuration option to control how modules set search paths to header files (framework PR #4655)
  • Let jobs retweak easyconfigs themselves (framework PR #4669)
  • 14 open easyblocks PRs related to EasyBuild 5.0 (link)

TODO

  • port CUDAcompat easyblock to module_load_environment => obscure thing, just make educated guess on how to port, and assume it's OK

2025-02-05

Merged PRs

2025-01-20

attending: Kenneth, Simon, Mikael, Alexandre, Alex, Bart

Sync PRs

Merged PRs

  • refactor generation of required environment variables in module files + deprecate make_module_req_guess method in EasyBlock class (framework PR #4653)
  • fix error reporting for unknown value of unrecognized_configure_options in ConfigureMake (easyblocks PR #3563)

Known issues

  • RPATH wrappers break SystemCompiler easyblock (easyblocks issue #3558)
  • installation of Qt6-6.6.3-GCCcore-13.2.0.eb fails with EB 5.0dev, works with EB v4.9.4
  • $CMAKE_PREFIX_PATH is being updated in all module files generated with EasyBuild 5.x?
    • no, already done with EasyBuild 4.x as well
    • => @Alex: please open issue to check if we need to make this conditional so $CMAKE_PREFIX_PATH is only expanded when it makes sense
    • Bart: (also) used by CMake to find libraries/header files...

WIP

2025-01-13

attending: Kenneth, Alex, Mikael, Bart, Simon, Alex

Merged PRs

WIP

  • (Alex,Kenneth) refactor generation of required environment variables in module files + deprecate make_module_req_guess method in EasyBlock class (framework PR #4653)
  • (Mikael) Deprecate use of parallel easyconfig parameter and fix updating the template value #4580
    • some concerns w.r.t. always adding -j to build_cmd
    • parallel = False use case was broken (cfr. Corsika), no equivalent to it via maxparallel
        1. (NO) should we auto-remove -j 1 in <build_cmd> -j 1?
        1. (YES) only add -j if self.cfg.parallel > 1 in ConfigureMake
        • maxparallel = 1 would still work for easyconfigs that customize build_cmd
        • make -j 1 is equivalent to make (see Make docs)
        1. (not needed if 2 is done) add support build_j_option = False to ConfigureMake (as equivalent to parallel = False)?
        • to avoid maxparallel = 1 workaround?
        1. (NO, see Rust) only add -j if build_cmd has make?
        1. (YES) help text for build_cmd should mention that -j is added to build command (if > 1 cores is used)
  • complete overview of changes in EasyBuild v5.0 in docs

Discuss

2025-01-08

attending: Alex, Jasper, Simon, Kenneth, Mikael, Bart

project board: https://github.com/orgs/easybuilders/projects/18/views/2

TODO before we can release EasyBuild v5.0.0

  • [Alex,Kenneth] refactor easyblocks to not use deprecated make_module_req_guess method
  • [Mikael] deprecate parallel easyconfig parameter framework PR #4580
    • will also require updates in easyblocks that use self.cfg['parallel']
  • [Bart,Mikael] checksum-related PRs in framework
    • support for using dict-value in checksums (as opposed to list of dicts...) is a nice-to-have
  • [Mikael] let jobs retweak themselves framework PR #4669
  • [Jasper] RPATH-related issues
  • [Kenneth] enhanced LLVM easyblock
  • [Simon,Kenneth] complete overview of changes in EasyBuild v5.0 in docs
    • work through list of merged PRs to 5.0.x branches, make sure all of them are somehow reflected in overview of changes

Open PRs

  • revised landing page with overview of EasyBuild v5.0 changes docs PR #289

2024-12-18

attending: Kenneth, Alex, Simon, Jasper

project board: https://github.com/orgs/easybuilders/projects/18/views/2

  • beta release of EasyBuild 5.0 (5.0.0beta1)
    • which PRs should we really get merged before tagging 5.0.0beta1?
      • not running sanity check commands from install dir (no PR yet) [Alex]
      • Add dashes directly in compiler optimization flags (framework PR #4698) [Kenneth]
      • update ConfigureMake easyblock to error out on unknown configure options (easyblocks PR #3025) [Kenneth]
      • Rename post_install_step to post_processing_step (framework PR #4715 + easyblocks PR #3525)
      • checksum PRs #4578 + #4579 [Mikael]
      • update Python easyblock to move sitecustomize.py into site-packages (easyblocks #3514) [Kenneth]
      • sync with develop for all repos [Kenneth]
    • what do we need?
      • tags in GitHub repos (no releases on PyPI)
      • easyconfig file to make installation with EasyBuild v4.x easy (using git_config)
      • instructions to set up venv
      • docs page with basic info on release candidates for EasyBuild 5.0 (https://docs.easybuild.io/easybuild-v5/release-candidates)
        • highlighted changes
          • requires Python >= 3.6
          • requires Lmod >= 8.0 or Environment Modules version >= 4.3.0
          • run_shell_cmd
          • changed defaults: RPATH, trace output, use_pip, depends_on in Lua modules
          • deprecated functionality
            • run_cmd + run_cmd_qa
          • easyconfigs using an old unsupported toolchains have been moved to the easyconfigs archive
        • planned changes
          • selected open framework PRs
          • deprecating of module_req_guess method
          • deprecating of parallel easyconfig parameter
      • dedicated issue to give feedback (in easybuild repo)
        • positive/negative
        • include output of eb --version and eb --show-config and eb --show-system-info
        • what did you try

2024-12-11

attending: Mikael, Jasper, Simon, Kenneth, Bart, Alex

project board: https://github.com/orgs/easybuilders/projects/18/views/2

Merged PRs

  • tweak setup.py to fall back to importing setup from setuptools if distutils is not available (framework PR #4478)
  • Rename 'source' step to 'extract' (affects skipsteps easyconfig parameter + --stop option) (framework PR #4629)
  • add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
  • Refactor make_extension_string (framework PR #4690)
  • add option search-path-linker to control linker options at build time (framework PR #4697)
  • enhance get_software_libdir to return full paths if requested (framework PR #4699)
  • sync with develop (20241207) (framework PR #4718)
  • retain custom easyconfig parameters supported by MesonNinja easyblock in custom easyblock for scipy (easyblocks PR #3526)

Known bugs to fix

Open PRs

Release candidate

  • Hopefully an initial release candidate of EasyBuild v5.0 still this year...
    • tagged version in GitHub repos
    • easyconfig file to make it easy to install
      • instructions on setting up Python virtual env to play with it
    • feedback mechanisms
      • #eb5 channel in Slack
      • dedicated meta-issue in GitHub to report problems with this RC?

2024-12-04

2024-11-28

attending: Simon, Alex, Jasper, Mikael, Kenneth

Merged PRs

  • make LooseVersion('1.0') == LooseVersion('1') #4691
  • simplify code for determining the PYTHONPATH module entries #4686
  • allow use of custom delimiter for paths in module generator #4687
  • change semantics of --dry-run, so it doesn't imply --robot (PR #4704)

2024-11-13

attending: Alex, Mikael, Bart, Kenneth, Simon

  • WIP PRs
    • Add dashes directly in compiler optimization flags (PR #4698)
      • should produce a warning when leading - is missing
      • ideally as late as possible, when we're actually setting up the build env?
    • tarball PR reviewed, small changes needed
    • refactor generation of required environment variables in module files + deprecate make_module_req_guess method in EasyBlock class (PR #4653)
      • changes to easyblocks are WIP by Alex already
    • overview of changes in docs

2024-11-06

attending:

  • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • blockers + breaking changes + changed defaults columns were updated to reflect what's worth blocking the EasyBuild 5.0 release over
    • to be revisited in next sync meeting, a handful of issues/PRs still on the table may be pushed out still...
  • syncing of 5.0.x branches with develop
    • framework: in sync with develop
    • easyblocks: PR #3500
    • easyconfigs: TODO
  • open PRs to focus on:
    • improve portability of reproducible tarballs PR #4660 ready for review [Kenneth]
    • $CPATH related PRs
      • for build env: PR #4645 => ready for review [Kenneth]
      • refactoring + deprecating make_module_req_guess: PR #4653
        • will require changing easyblocks that customize make_module_req_guess method
      • for CPATH in module files: PR #4655
    • Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
    • Let jobs retweak easyconfigs themselves (PR #4669) [Kenneth]
    • Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)
      • still being considered to be included in EasyBuild 5.0
      • we can already disallow setting of parallel in easyconfig files by checking for it in easyconfigs test suite
      • support for parallel easyconfig parameter should be deprecated, not removed
      • impact on easyblocks?

2024-10-02

attending: Kenneth, Alex, Alexandre, Simon, Jasper, Bart

  • project board: https://github.com/orgs/easybuilders/projects/18/views/2

  • syncing of 5.0.x branches with develop

    • framework PR #4654
    • easyblocks PR #3459
    • easyconfigs PR #21497
    • collapse of 5.0.x into develop is postponed a bit, we need to be closer to EasyBuild 5.0 release for that
      • mainly to avoid impact on people using EasyBuild 4.9.x release with --from-pr + having to re-trigger CI in easyconfig PRs
      • doesn't affect --new-pr: will still work with EasyBuild 4.x even when develop is already prepared for EasyBuild 5.0 release
      • probably only makes sense when we have a release candidate for EasyBuild 5.0

Merged PRs

  • framework
    • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
    • also define functions in interactive shell session created by cmd.sh script that is produced by run_shell_cmd (PR #4662)
  • easyblocks
    • update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option (PR #3343)
    • Make CMakeMake respect the toolchainopts when selecting build type (easyblocks #3452)

Open PRs

framework

  • [Kenneth] Enable module-depends-on by default (PR #4500)
  • [Alex] Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
  • [Kenneth] add option search-path-cpp-headers to control how EasyBuild sets paths to headers at build time (PR #4645)
    • affects build environment set up by EaysBuild
  • [Kenneth] refactor generation of required environment variables in module files (PR #4653)
    • merge conflict on easybuild/tools/config.py
    • ModuleLoadEnvironment should not be a singleton?
    • deprecates make_module_req_guess method
    • easyblocks will have to be updated accordingly
      • Alex is working on this already
      • self.module_load_environment should still be accessed in make_module_step (taking into account --module-only)
  • [Kenneth] add option module-search-path-headers to control how modules set search paths to header files (PR #4655)
  • [Kenneth] use Slurm as the default job back-end (PR #4659)
    • shold be ready to go now
  • use more portable --date argument for touch command used in reproducible tarballs (PR #4660)
  • Add flushing and possible closing of stdin on run_shell_cmd (PR #4664)
  • Proposed changest to add a timeout to run_shell_cmd (PR #4665)

easyblocks

  • enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
  • [Jasper] Make MesonNinja respect the toolchainopts with buildtype as well as --debug and --optimization flags (PR #3454)

easyconfigs

  • stop using modextrapaths to update $PYTHONPATH with standard path to installed Python packages (lib/python%(pyshortver)s/site-packages) (PR #20960)

2024-09-23

Open PRs

  • framework:
    • Enable module-depends-on by default (PR #4500)
      • deprecate modules_depends_on easyconfig parameter
      • => set it to None as default, print warning from module_generator.load_module + honor it when set to False/True
      • => also accept use of modules_depends_on = False in tests (will trigger deprecated warning)
      • modules_depends_on easyconfig parameter was introduced in framework PR #2391, but use case for it was unclear, maybe to "ease in" to using depends_on?
        • was originally introduced to mirror recursive_module_unload which was already there;
    • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
      • should be merged along with companion PRs in easyblocks (PR #3343) + easyconfigs (#20960)
      • so should be done before 5.0.x is collapsed into develop
      • see also easyconfigs issue #21456 for easyconfigs that explictly update $PYTHONPATH
    • add option cpp-headers-search-paths to control how EasyBuild sets paths to headers at build time (PR #4645)
      • this PR is about $CPATH set in build environment that EasyBuild uses
      • tests fixed, extra test added
      • other (independent) PR to not update $CPATH in generated module files is coming up, see PR #4653
        • keep in mind there's a temporary module file as well, which is loaded before extensions are being installed
        • make_module_req_guess is often redefined by easyblocks
    • Reverse order for parsing files in XDG_CONFIG_DIRS (PR #4630)
  • easyblocks:
    • Use prefer_ebpythonprefixes flag to use $EBPYTHONPREFIXES (PR #3343)
    • Make CMakeMake respect the toolchainopts when selecting build type (PR #3452)
    • Make MesonNinja respect the toolchainopts with buildtype as well as --debug and --optimization flags (PR #3454 + PR #3453)

2024-09-18 (15:00 CEST)

merged PRs

  • framework
    • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
    • use more granular exit codes when EasyBuildError is raised (PR #4534)
    • move verifying of checksums from source to fetch step, to include it with --fetch (PR #4624)
    • derive EnvironmentModules class directly from ModulesTool rather than from to be deprecated EnvironmentModulesTcl (PR #4625)
    • sync with develop (20240915) (PR #4639)
  • easyblocks
    • sync with develop (20240915) (PR #3447)

open PRs

  • framework
    • Replace parallel easyconfig parameter by maxparallel (PR #4398)
    • Enable module-depends-on by default (PR #4500)
    • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
      • Mikael is looking into suggested changes
    • Deprecate use of ec['parallel'] and fix updating the template value (PR #4580)
    • Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
  • easyblocks
    • Use prefer_ebpythonprefixes flag to use $EBPYTHONPREFIXES (PR #3343)
    • run "pip check" only once for PythonBundle (PR #3432)

Tasks

  • ...

2024-09-09 (15:00 CEST)

  • project board: https://github.com/orgs/easybuilders/projects/18/views/2
  • merged PRs
    • several PRs related to Environment Modules support (contributions by Tmod maintainer)
      • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
      • add check_group support for module files in Tcl syntax (PR #4418)
      • Run unit tests on an updated versions of Modules: v4.5.3 + v5.3.1 (PR #4415)
      • Use getenv modulefile command with EnvironmentModules >= 4.2.0 (PR #4614)
      • Add module cache build support on EnvironmentModules (PR #4615)
      • derive EnvironmentModules class directly from ModulesTool rather than from to be deprecated EnvironmentModulesTcl (PR #4625)
      • update to docs that suggests to install Environment Modules 3.2.10 (docs PR #264)
    • Use default value $XDG_CONFIG_DIRS from XDG basedir spec: /etc/xdg (instead of /etc) (PR #4591 )
      • including warn about finding /etc/easybuild.d/*.cfg instead of /etc/xdg/easybuild.d/*.cfg
    • move verifying of checksums from source to fetch step, to include it with --fetch (PR #4624)
  • open active PRs
    • Enable module-depends-on by default (framework PR #4500)
      • --module-depends-on should only mean use depends_on when using Lmod as modules tool
    • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (framework PR #4539)
      • companion PRs:
      • reviewed by Alex, some nitpicking on names of the options
    • More granular exit codes (framework PR #4534)
      • Kenneth is reviewing this in detail, has some (minor) suggestions coming soon...
      • enhancing tests for non-1 exit code is not that trivial...
    • run pip check only once for PythonBundle (PR #3428)
  • TODO:
    • ...

2024-09-02 (15:00 CEST)

  • Merged module extensions default to true.
  • Merged template constant list to dicts (+ fix).
  • Alex looked at EBPYTHONPREFIXES pr. Does not like the option names.
    • Alex: EBPYTHONPREFIXES means nothing to users, prefer some more descriptive option about user vs system precedence.
    • Mikael: Maybe we call the easyconfig/block option force_pythonpath (default false)?
    • Suggestions for command line option (currently --prefer-ebpythonprefixes)?
      • Mikaels wild idea: --use-system-level-python-path
      • Mikaels wild idea: --use-low-prio-python-path
  • Alex is ready to merge return codes PR #4534
    • Maybe we add a test for return codes?

2024-08-26 (15:00 CEST)

2024-08-19 (15:00 CEST)

  • merged PRs
    • ...
  • recent open PRs
    • Use default value $XDG_CONFIG_DIRS from XDG basedir spec: /etc/xdg (instead of /etc) (PR #4591)
      • breaking change
      • we should at least print a warning when finding /etc/easybuild.d/*.cfg, maybe even still pick up on them?
    • convert template constant lists to dicts and export the constants by name (PR #4595)
      • breaking (though mostly internal) change to value type for TEMPLATE_* constants
    • add qa_check_interval option to run_shell_cmd (PR #4613)
      • seemed necessary for WIEN2k easyblock, but looks like there's actually a bug to fix...
    • EBPYTHONPREFIXES vs PYTHONPATH (framework PR #4539, easyblocks PR #3343, easyconfigs PR #20960)
      • is now opt-in via --prefer-ebpythonprefixes
      • Mikael will do some more testing with this
      • only consider /lib/, since that's also what our sitecustomize.py does
  • plan towards EasyBuild 5.0
    • focus on breaking changes, put other efforts in the fridge for now
      • that includes PRs related to improving naming consistency of things
      • would be nice to get support for alternative names for toolchain options merged (framework PR #4587), since that opens door for "modernizing" easyconfig files
    • finish up run_shell_cmd effort
      • 3 easyblocks left
      • bug in running interactive commands with run_shell_cmd, should only look for question pattern matches if extra output was generated?

2024-08-12 (15:00 CEST)

(hosted by Mikael)

  • Work on EBPYTHONPREFIXES proceeding (see easyblock PR#33438. Needs lots of testing, preliminary results look fine. PRs are not that complex but needs all 3 (framework+blocks+configs) to test. Someone other than Mikael also needs to volunteer to test some of it at least.
  • Toolchain options basically done framework PR#4587, PR stuck on test that only runs on easybuilders repo (why do we have such tests?)
    • Mikael: I can't think of a good reason why this would only run within the easybuilders branches; just have it run always (and fix it)?
  • Maybe add a --accept-eula-for=anaconda (no issue yet, also not a 5.0.x blocker in Mikaels opinion)

2024-07-22 (15:00 CEST)

(hosted by Simon)

Limited progress since the previous meeting and several people unavailable for this meeting.

2024-07-03 (14:00 CEST)

Merged PRs

  • check presence of CWD at the end of run_shell_cmd and try to return to original working directory if non-existent (framework PR #4390)
  • remove CMAKE_INSTALL_LIBDIR settings from configopts + add test to enfore use of install_libdir instead (easyconfigs PR #20487)

Recently active PRs

  • Make generic EasyBlock usable to install software (framework PR #4531)
    • this is not a blocker for EasyBuild 5.0, so we should put this in the fridge until EasyBuild 5.0 is out

Notes

  • pre_configure_cmds in PR #4531 is run in separate subshell, which limits usefulness

    • also only picked up when configure_cmd is set (see EasyBlock.configure_step)
    single_shell_env = True
pre_configure_cmds = [
    "export FOO=bar",
    "ln -s something something_else",
    "sed ...",
    "make configure",
]
configure_cmd = "./configure"

    shell 0:

    set -e

echo '>>> running command: export FOO=bar'
export FOO=bar >&2 | tee /tmp/export.out
echo ">>> done running command: export FOO=bar'"

...

echo ">>> running command: ./configure"
./configure
  • progress has stalled, how do we get things back on track towards EasyBuild 5.0?

2024-06-17 (15:00 CEST)

Merged PRs

  • Use "alternative" instead of "alternate" PR#4555
  • include alternative names of easyconfig parameters in output of --avail-easyconfig-params PR#4549

Recently active PRs

2024-06-10 (15:00 CEST)

EasyBuild 4.9.2

  • prepared, regtest

Merged PRs

  • versions checks to avoid mixing major versions across the EasyBuild components #4520
    • will be included with EasyBuild v4.9.2 release
    • quite strict now, could loosen this up
      • new EasyBuild using easyconfigs for EB v4.x should be fine
      • should we add support for --allow-from-pr-api-mismatch so you can still consume (only) easyconfig PRs intended for EasyBuild 5.x using EasyBuild 4.9.2+
        • --from-pr on a merged PR => pull easyconfig from develop branch
        • using EB 4.9.2 --from-pr + when 5.0.x in collapsed into develop => ERROR
        • conclusion: for v4.9.2, we'll leave the strict API version check in place for both framework+easyblocks & framework+easyconfigs
        • if needed/desired, we could still do a v4.9.3 release that adds a way to opt-out
  • add renamed alternative constants/templates/parameters #4514
  • trigger pre-hook in run_shell_cmd as early as possible so dry-run + trace output is correct #4544

Recently active PRs

  • check presence of CWD at the end of run_shell_cmd and try to return to original working directory if non-existent #4390
  • include alternative names of easyconfig parameters in output of --avail-easyconfig-params #4549
  • Use "alternative" instead of "alternate" #4555
  • add support for alternative EasyBuild configuration options (WIP) #4556
  • More granular exit codes #4534 
    ┃ 74  class EasyBuildExit(IntEnum):
   """
   Table of exit codes
   """
   SUCCESS = 0
   ERROR = 0
   MISS_EASYCONFIG = 3
   # no_specific_easyconfig = 4
   FAIL_IMPORT_EASYBLOCK = 5
   FAIL_IMPORT_EASYBLOCK_CLASS = 6
   MISS_DEPENDENCY = 10
   MISS_SYSTEM_DEPENDENCY = 11
   MISS_SOURCES = 12
   FAIL_SANITY_CHECK = 20
   FAIL_MODULE_WRITE = 21
   ERRO_SYNTAX = 30
    • can use EasyBuildExit.ERROR
    • which exit code to use for failing sanity check command?
      • can't use exit code of command directly, since that would overlap with custom exit codes used by EasyBuild
    • MISS_DEPENDENCY vs MISSING_DEP
    • custom exit code for download failures, failing shell commands, ...
      • also separate exit code for extract failures?

TODO

  • parent issue with overview of what's TODO for renaming stuff (Kenneth)
    • toolchain options (Jasper)
    • cfg options (Kenneth)
    • capture renamed stuff in docs (see PR #)
    • feature to auto-rename in easyconfigs (see PR #4547)

2024-06-05 (14:00 CEST)

attendees: Kenneth, Simon, Jasper, Mikael, Bart

Sync with develop (20240605)

Merged PRs

  • create env.sh and cmd.sh helper scripts in run_shell_cmd (framework PR #4486)
  • fix test_update_branch_github which got broken because toy-0.0.eb easyconfig was modified (framework PR #4545)
  • use run_shell_cmd (and use f-strings) in custom easyblock for GAMESS-US (PR #3342)

Recent PRs

  • versions checks to avoid mixing major versions across the EasyBuild components (PR #4520)
  • add renamed alternate constants/templates/parameters (PR #4514)
    • policy for naming easyconfig parameters/templates/constants/configuration options
      • multiple words are separated by _ (as opposed to just concatenating them)
        • examples: build_opts (not buildopts), post_install_cmds (not postinstallcmds)
        • exceptions: symlink, sysroot
      • common long words can be abbreviated, but only if there's no room for confusion
        • examples: deps (instead of dependencies), opts (instead of options), env_mod (instead of environment_module)
        • deps, opts, stats, exts, vars, dir, ver, maj_ver, min_ver, short_ver, py (python), cuda_cc (CUDA compute capabilities)
        • exceptions:
          • configure in easyconfig parameters (to be consistent with configure step)
          • cfg in EasyBuild configuration options
      • confusion with commonly used terms (like 'module') is avoided by being more descriptive
        • example: don't use mod for module, but py_mod for Python module, env_mod for environment module, fortran_mod for Fortran module
    • support for automatically renaming in easyconfigs: WIP PR #4547
      • nice to have for EasyBuild 5.0 (not a blocker)
    • TODO
      • renaming of configuration options
      • reanming of toolchain options
        • usempi -> just mpi? or more specific like 'mpi_compilers': True
      • make eb -a & co also mention alternate names
      • docs

Notes

2024-05-29 (15:00 CEST)

attendees: Kenneth, Bart, Alex, Mikael, Jasper, Simon

Merged PRs

Recent PRs

2024-05-22 (15:00 CEST)

Status of 5.0.x branches

PRs to discuss

  • versions checks to avoid mixing major versions across the EasyBuild components (PR #4520)
    • who: Simon + Kenneth
    • action: review (Kenneth)
    • should into EasyBuild v4.9.2
  • add support for disabling strict RPATH sanity check + print a warning when mixing of non-RPATH and RPATH installations was detected (PR #4475)
    • who: Kenneth
    • action: needs to be reworked, RPATH sanity check needs to be changed
  • Log shell cmds to temp file which allow users to revisit them (PR #4486)
    • who: Mikael + Kenneth
    • action: merge it (Kenneth), then document it (Mikael?)
  • Prefer $EBPYTHONPREFIXES over $PYTHONPATH (PR #4496)
    • who: Mikael
    • plan: rework based on current ideas, then get feedback
    • concerns w.r.t. behaviour for Python installations not controlled by EasyBuild
    • PythonPackage should take into account self.prefer_ebpythonprefixes
      • self.prefer_ebpythonprefixes = build_option('prefer_ebpythonprefixes') and self.cfg['prefer_ebpythonprefixes']
    • let framework prepend to $EBPYTHONPREFIXES or $PYTHONPATH (similar to $LD_LIBRARY_PATH) for lib/python*/site-packages, if the easyconfig/easyblock is not prepending to either $EBPYTHONPREFIXES or $PYTHONPATH already, and only if Python is a dependency
    • clean up easyconfigs to not update $PYTHONPATH via modextrapaths + make easyconfigs test suite check that this doesn't get re-introduced (with a whitelist for special cases)
  • add support for alternate easyconfig parameters/templates/constants (PR #4511)
    • cfr. issue #4464
    • who: Jasper + Kenneth
    • action: Kenneth figures out trouble with the tests
    • follow-up PR: add renamed alternate constants/templates/parameters (PR #4514)
    • TODO:
      • document policy on naming stuff + cross-check with PR #4514
      • support for alternative names for configuration options
  • More granular exit codes (PR #4534)
    • who: Danilo + Alex
  • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)
    • who: Alex + Kenneth
    • status: ready to merge
  • several easyblocks to migrate from run_cmd to run_shell_cmd

2024-05-08 (14:00 CEST)

attending: Alex, Simon, Bart, Jasper, Mikael

Merged PRs

Open (active) PRs

  • Log shell cmds to temp file which allow users to revisit them (PR #4486)
    • Alex: will review and merge it asap
  • Prefer $EBPYTHONPREFIXES over $PYTHONPATH (framework PR #4496)
    • Mikael: EBPYTHONPREFIX is not understood by anything but EB, this makes it not a workable solution for software in SYSTEM (or Conda)
    • Alex: we need something that can be understood by upstream Python, is there something softer than PYTHONPATH?
    • Mikael: new logic with EBPYTHONPREFIX can be limited to PythonPackage easyblock (and its derivates), so moving from framework to easyblock
    • Bart: many users load virtual environments before loading software modules. Our Python modules set PYTHONPATH to a sitecustomization.py that sets sys.path
    • Mikael: complex to implement as opt-in as needs to consider all custom PYTHONPATH definitions in easyconfigs
  • add support for alternate easyconfig parameters/templates/constants (framework #4511)
    • Jasper: unit test fails in CI, but not when run standalone locally. Somewhere templates stopped to be resolved.
  • add 'archive' extra to optionally install easybuild-easyconfigs-archive (PR #20435)
    • Aled: ready to merge
  • set CMAKE_INSTALL_LIBDIR to lib by default
    • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)
    • remove CMAKE_INSTALL_LIBDIR settings from configopts and add test to enfore use of install_libdir instead (PR #20487)
    • Alex: ready to merge
  • Standarize behaviour on CWD changes
    • add new get_cwd function to tools.filetools to retrieve current working directory (PR #4525)
    • return to original working directory after executing a command in run_shell_cmd (PR #4390)
    • Alex: conbination of faulty storage and build on install dir makes os.getcwd to fail (storage in zombie state). Proposal to standarize behaviour of run_shell_cmd to switch to starting working directory at the end of execution
    • Mikael: this changes behaviour and should not be necessary as any changes to working directory inside the shell executing the command should not affect the CWD of the main Python process. Will be troublesome to execute multiple commands that rely on directory changes by previous commands.
    • Alex: I still hit the error with os.getcwd outside of the subprocess shell
  • Replace run_cmd with run_shell_cmd in custom easyblock for QuantumESPRESSO (PR #3269)
    • Simon: there is some on-going discussion about deciding which one of the two easyblocks for QuantumESPRESSO becomes the default one (aka EB_QuantumESPRESSO)
  • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)

Discussion

2024-04-29 sprint meeting (14:00 CEST)

attending: ???

Merged PRs

Open (active) PRs

  • Log shell cmds to temp file which allow users to revisit them #4486
    • ready to merge
  • Prefer $EBPYTHONPREFIXES over $PYTHONPATH (framework PR #4496)
    • may need to move logic to pick EBPYTHONPREFIXES vs PYTHONPATH to EasyBlock.make_module_req (rather than in ModuleGenerator.update_paths)
    • maybe need to logic in both make_module_req (default + make PythonPackage behave) + ModuleGenerator (to deal with PYTHONPATH used in modextrapaths)
    • EB should produce a warning if it detects that PYTHONPATH is used when it shouldn't be
    • Mikael will continue working on this
  • add support for alternate easyconfig parameters/templates/constants (framework #4511)
  • always include rpath configuration setting in output of --show-config (framework PR #4515)
  • 5.0.x consideration: ruff format filetools.py (PR #4521)
    • not so much about which tool/linter is used, but rather whether the community thinks it's worth adopting an approach like this
    • ruff can be heavily configured, we could configure it to match our current code style
    • definitely impacts usefulness of "git blame"
    • should we recommend that developers install a pre-commit hook to run ruff on their code changes?
    • should we adopt a bot that automatically add a commit to fix code style changes?
  • add 'archive' extra to optionally install easybuild-easyconfigs-archive (PR #20435)
    • need to introduce protected branch in archive repo => fixed

Discussion

  • how to mitigate impact of collapsing 5.0.x branches in develop
    • will (often) break use of --from-pr with EasyBuild 4.x
    • see also Simon's framework PR #4520
    • include easybuild_version = '5.0' in easyconfig that require EasyBuild v5.0 
      # requires EasyBuild 5.0 which enables use_pip by default
easybuild_version = '5.0'
      • also will be useful when we start renaming easyconfig parameters
      • error message should be improved
    • motivation to rename branches in GitHub?
      • --from-pr on merged PR currently pulls from develop
      • only use 5.0.x style branches going forward?
      • how does --new-pr know which branch to target by default?

2024-04-19 sprint meeting (14:00 CEST)

attending: ???

Next EasyBuild release

  • v4.9.1 released 2024-04-05 (14 days ago)
  • next release will hopefully be 5.0.0 (famous last words)

Status of 5.0.x branches

Merged PRs

framework

  • implement support for running interactive commands with run_shell_cmd (PR #4453)
    • fix getting all available output for interactive commands + fix logging end of output when no match was found in run_shell_cmd (PR #4504)
    • replace spaces/line breaks in question/wait patterns with regex pattern that matches one or more spaces/line breaks in run_shell_cmd (PR #4509)
  • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (PR #4433)

easyblocks

  • various PRs to migrate from run_cmd_qa to run_shell_cmd
  • remove a bunch of unused easyblocks (WRF-Fire, HEALPix, pplacer, Modeller, Molpro)
  • drop support for versions < 9.5 in custom easyblock for Geant4 (PR #3296)
  • add warning if .Renviron detected during install of RPackage (PR #3263)

easyconfigs

Open (active) PRs

framework

  • add support for disabling strict RPATH sanity check + print a warning when mixing of non-RPATH and RPATH installations was detected (PR #4475)
  • add mechanism for easyconfig template deprecation (PR #4485)
    • stuck on broken tests
  • Log shell cmds to temp file which allow users to revisit them (PR #4486)
  • Enable module-depends-on by default (PR #4500)
  • Make module-extensions true by default (PR #4501)
  • add support for alternate easyconfig parameters/templates/constants (PR #4511)
  • Prefer EBPYTHONPREFIX over PYTHONPATH #4496

easyblocks

  • Replace run_cmd with run_shell_cmd in custom easyblock for QuantumESPRESSO (quantumespresso.py) (PR #3269)

TODO

framework

  • go through open PRs/issues for EasyBuild 5.0: what do we really still need to do that is required for EasyBuild 5.0

easyblocks

  • complete migration to run_shell_cmd (ABAQUS, GAMESS-US, QuantumESPRESSO, WIEN2k)
  • go through open PRs/issues for EasyBuild 5.0: what do we really still need to do that is required for EasyBuild 5.0

Alternate plan for renaming easyconfig parameters & co

  • not a "big bang" approach by deprecated configopts & co for EasyBuild 5.0
  • first add support for new names, then gradually adopt them
    • not necessarily in EasyBuild 5.0
  • only deprecate when framework/easyblocks/easyconfigs have been migrated to new news (EasyBuild 5.x with x > 0)
  • drop support for old names in EasyBuild 6.0 (in X years)
  • need to take into account hooks using old/new names...

Notes

2024-04-03 sprint meeting (14:00 CEST)

attending: Alex, Bart, Jasper, Simon, Kenneth

Next EasyBuild release (v4.9.1)

  • being prepared

Status of 5.0.x branches

  • unchanged since 2024-03-28 meeting, except for merging PRs to sync with develop branch
    • framework: in sync with develop (since 202403011)
    • easyblocks: PR #3261
    • easyconfigs: PR #20236

Merged PRs

framework
  • hide readelf output in trace output in RPATH sanity check PR #4491
  • switch from ls to bash in tests that are expecting this to be a binary PR #4492
easyblocks
  • use sysconfig.get_config_vars from Python standard library in PythonPackage easyblock (instead of distutils.sysconfig.get_config_vars) PR #3264 (Python 3.12)
  • also run easyblocks test suite with Python 3.12 PR #3266
  • remove unused easyblock for ncurses (ncurses.py) PR #3265
easyconfigs

Open (active) PRs

framework
  • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
    • blocked by open run_shell_cmd PR #4453
  • implement support for running interactive commands with run_shell_cmd (WIP) (PR #4453)
  • add support for disabling strict RPATH sanity check (PR #4475)
    • our idea to detect a "mixed stack" (mix of RPATH and non-RPATH installations) isn't really working out...
    • we can merge support for --disable-strict-rpath-sanity-check, and always mention it in error message for failing RPATH sanity check for now
    • should also be controllable via easyconfig parameter (for OCaml for example)
    • then later come up with a good way of detected mixed stack
    • ldd check should always be run to check for missing libraries, regardless of using RPATH or not
    • we could/should only care of NEEDED libraries (direct dependencies, via readelf -d) if ldd reports them as not found => no problem in mixed stack situation
  • Rename easyconfig parameters (WIP PR #4483)
  • Log shell cmds to temp file which allow users to revisit them (PR #4486)
  • add mechanism for easyconfig template deprecation (PR #4485)
easyblocks
  • fix extension filter for Perl packages (PR #2699)
  • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)
  • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
  • use run_shell_cmd in custom easyblock for WRF PR #3270
easyconfigs
  • Replace parallel by maxparallel (PR #19375)
  • Remove whitespace at EOL from ECs (PR #20082)

2024-03-28 sprint meeting (09:00 CET)

attending: Mikael, Jasper, Kenneth, Simon

Next EasyBuild release (v4.9.1)

  • blocked by framework PR #4468 to add support for --from-commit and --include-easyblocks-from-commit

Status of 5.0.x branches

  • framework: in sync with develop (since 202403011)
  • easyblocks: PR #3261
  • easyconfigs: PR #20236

Merged PRs

framework
  • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
  • Eliminate use of distutils.version.StrictVersion, use LooseVersion instead (PR #4476)
  • also run unit tests with Python 3.12 + add it to classifiers in setup.py (PR #4484)
easyblocks

(none)

easyconfigs

(none)

Open (active) PRs

framework
  • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
    • blocked by open run_shell_cmd PR #4453
  • implement support for running interactive commands with run_shell_cmd (WIP) (PR #4453)
  • add support for disabling strict RPATH sanity check (PR #4475)
    • our idea to detect a "mixed stack" (mix of RPATH and non-RPATH installations) isn't really working out...
    • we can merge support for --disable-strict-rpath-sanity-check, and always mention it in error message for failing RPATH sanity check for now
    • should also be controllable via easyconfig parameter (for OCaml for example)
    • then later come up with a good way of detected mixed stack
  • Rename easyconfig parameters (WIP PR #4483)
  • Log shell cmds to temp file which allow users to revisit them (PR #4486)
  • add mechanism for easyconfig template deprecation (PR #4485)
easyblocks
  • fix extension filter for Perl packages (PR #2699)
  • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)
  • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
easyconfigs
  • Replace parallel by maxparallel (PR #19375)
  • Remove whitespace at EOL from ECs (PR #20082)

2024-03-11 sprint meeting (15:00 CET)

attending: Kenneth, Mikael, Simon, Jasper, Alex, Bart, Adam

Next EasyBuild release

  • working towards EasyBuild v4.9.1 release using develop branch, hopefully this week...

Status of 5.0.x branches

  • framework: sync with develop (20240311) PR #4482
  • easyblocks: sync with develop (20240311) PR #3251
  • easyconfigs: sync with develop (20240311) PR #20076

Merged PRs

framework
  • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
  • replace log.warn with log.warning (PR #4473)
  • Eliminate distutils.util.strtobool (PR #4477)
  • fix easyconfig parameter deprecation (PR #4479)
  • Clean up parameter deprecation warnings, and update easybuild doc urls (PR #4480)
easyblocks
  • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
easyconfigs
  • migrate archive of easyconfigs to easybuild-easyconfigs-archive (PR #20006)

Open PRs

framework
  • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
  • implement support for running interactive commands with run_shell_cmd (WIP) (PR #4453)
  • add support for disabling strict RPATH sanity check (PR #4475)
    • our idea to detect a "mixed stack" (mix of RPATH and non-RPATH installations) isn't really working out...
  • Eliminate use of distutils.version.StrictVersion (PR #4476)
  • remove distutils use from setup.py (PR #4478)
  • Rename easyconfig parameters (WIP PR #4483)
easyblocks
  • fix extension filter for Perl packages (PR #2699)
  • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
  • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)
easyconfigs
  • Replace parallel by maxparallel (PR #19375)

To discuss

  • refactor easyconfig parameters and template constants to use a consistent naming style (issue #4464)
  • enabling RPATH linking by default

TODO

2024-03-06 sync meeting (14:00 CET)

attending: Kenneth, Bart, Simon, Alex, Jasper, Mikael

Next EasyBuild release

  • working to push out EasyBuild v4.9.1 using current develop in next couple of days...

Status of 5.0.x branches

Merged PRs

  • framework
    • problematic test_toy_exts_parallel fixed (PR #4471)
    • broken test_github_merge_pr fixed (PR #4470, also applied to 5.0.x branch)
    • fix bug (wrong working dir) in run_shell_cmd for running commands asynchronously (PR #4471)
    • replace log.warn with log.warning (PR #4473)
  • easyblocks
    • update RPackage to make sure that asynchronous installation command is run in correct working directory (PR #3242)
    • more PRs to migrate easyblocks to run_shell_cmd
    • replace log.warn with log.warning (PR #3245)
  • easyconfigs
    • none

Open PRs

  • framework
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248) [Alex]
    • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400) [Kenneth]
    • implement support for running interactive commands with run_shell_cmd (WIP) (PR #4453) [Kenneth]
    • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433) [Kenneth]
  • easyblocks
    • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064) [Alex]
    • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227) [Simon]
    • replace run_cmd with run_shell_cmd in custom easyblock for LAMMPS (lammps.py) (PR #3244) [Kenneth]
      • failing test report when not all dependencies were installed with RPATH... :-/
  • easyconfigs

To discuss

  • Are we sticking to enabling RPATH by default? [Kenneth]
    • installing with RPATH on top of dependencies not installed with RPATH may fail...
      • cfr. Kenneth's recent experience with LAMMPS (issue #20051) 
        == FAILED: Installation ended unsuccessfully: Sanity check failed: Library libhdf5_hl.so.310 not found for
/software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/bin/lmp
Library libbz2.so.1.0 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/bin/lmp
Library libhdf5_hl.so.310 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib/liblammps.so.0
Library libbz2.so.1.0 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib/liblammps.so.0
Library libhdf5_hl.so.310 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib64/liblammps.so.0
Library libbz2.so.1.0 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib64/liblammps.so.0 (took 23 mins 13 secs)
    • for a mixed software stack (installations with RPATH + without RPATH), we need a configuration option to loosen up the RPATH sanity check (to not unset $LD_LIBRARY_PATH)
      • need to use --disable-strict-rpath-sanity-check
      • can we produce a warning to hint towards need for --disable-strict-rpath-sanity-check?
        • when we notice that a dependency was installed without RPATH (like netCDF, in case of LAMMPS)
      • Kenneth will look into implementation, review by Simon
  • stop relying on distutils (will no longer be available in Python 3.12) (issue #3963) [Bart]
  • refactor easyconfig parameters and template constants to use a consistent naming style (PR #4464) [Jasper]
    • eb --modernize-easyconfigs foo.eb bar.eb
    • TODO: revise EasyBuild configuration options
  • Alex: review issues tagged for 5.0 (see here)

2024-02-29 sprint meeting (17:00 CET)

attending: Mikael, Simon, Jasper, Simon, Kenneth, Alex

Next EasyBuild release

  • we're still weeks away from releasing EasyBuild 5.0
  • should look into EasyBuild v4.9.1 release...
  • and then collapse 5.0.x into develop (and keep 4.9.x as backup branch)
    • this will affect people using --from-pr and --include-easyblocks-from-pr

Status of 5.0.x branches

Merged PRs

  • docs:
    • RPATH is enabled by default in EasyBuild v5.0 (PR #241)
  • framework:
    • add support for running shell commands asynchronously with run_shell_cmd (PR #4444)
    • enable RPATH linking by default (PR #4448)
    • allow only alphanumeric characters in the output filename used in run_shell_cmd (PR #4454)
    • add cuda compute capability integer format templates (PR #4463)
    • Enhance download instructions by mentioning active source path (PR #4459)
  • easyblocks:
    • update run_async methods in RPackage, Rmpi, and Rserve easyblocks to use submit run_shell_cmd call into thread pool (PR #3163)
    • replace run_cmd with run_shell_cmd in custom easyblock for NWChem (PR #3197)
    • Revert "remove unneeded custom easyblock for OCaml (ocaml.py)" (PR #3237)
  • easyconfigs
    • fix source definition for dialog (PR #19906)
    • archive old style Kent_tools easyconfig (PR #19933)
    • archive oldest MATLAB easyconfigs (PR #19937)
    • lots of PRs to add missing download_instructions
    • detect easyconfigs where missing information about downloading sources (PR #19881)

Open (active) PRs/issues

  • archive (https://github.com/easybuilders/easybuild-easyconfigs-archive)

    • add README PR #1
    • support installation of archive via pip install easybuild-easyconfigs[archive] (or pip install easybuild-easyconfigs-archive)

  • docs

    • split out default config changes and remove proposed changes (PR #242)
    • download_instructions enhancements in EB5 (PR #243)

  • framework

    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
    • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
      • ready for final review + merge [Kenneth]
    • implement support for running interactive commands with run_shell_cmd (WIP) (PR #4453)
      • rebased after merge of PR #4444
      • still WIP...
      • missing support for qa_wait_patterns
      • more existing run_cmd_qa tests should also be implemented for run_shell_cmd
      • to be tested with real easyblock
    • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
    • refactor easyconfig parameters and template constants to use a consistent naming style (issue #4464)
      • can we reach consensus on which things we will rename, and what not?
      • dependencies vs deps
      • what about EasyBuild configuration options, do we need to rename something there too?
      • does current deprecation mechanism take into account setting of easyconfig parameters in a hook? (see EasyConfig.__setitem__?)
      • rules
        • separate words with underscores (versionsuffix -> version_suffix)
        • aim for consistency (mod + module -> all module)
        • if a shorthand is used, no room for confusion (deps/exts/doc/mod is OK), less room for typos, must be at least 50% gain
        • mod_/module_ -> env_mod_ (no room for confusion with other types of modules)
        • modulename -> import_name (see dedicated issue)
          • options = {'modulename': 'torch'} => import_name = 'torch' (avoid "module")
        • should also look at easyblock-specific easyconfig parameters
        • script or eb option to "modernize" easyconfigs (rename parameters/templates/constants)
    • Always set $EBPYTHONPREFIXES instead of $PYTHONPATH (issue #2887)
      • counter argument: $PYTHONPATH is ubiquitous
      • problem with setting $PYTHONPATH is getting worse due to switch to introduction of Python-bundle-PyPI
      • someone should do the homework to figure out what needs to be implemented to support this => Mikael
        • global EasyBuild configuration option: --prefer-ebpythonprefixes-over-pythonpath, --avoid-pythonpath=PythonPackage,PythonBundle
        • make PythonBundle + PythonPackage to be aware of it
      • what about corner cases like PLUMED/2.7.2-foss-2021a.lua which has prepend_path("PYTHONPATH", pathJoin(root, "lib/plumed/python")) in env. module
        • rename lib/plumed/python to lib/pythonX.Y/site-packages to avoid having to use $PYTHONPATH
        • similar: Graphviz, ROOT, PYTHIA

  • easyblocks

  • easyconfigs

    • Replace parallel by maxparallel (PR #19375)

2024-02-12 sprint meeting (15:00 CET)

attending: Kenneth, Alex, Simon, Bart, Sebastian, Lara

  • status of 5.0.x branches
    • framework: synced with develop on 8 Feb'24 (CI OK)
    • easyblocks: synced with develop on 12 Feb'24 (CI OK)
    • easyconfigs: synced with develop on 12 Feb'24 (CI broken)
      • "Missing dependencies: Gblocks/0.91b" => Gblocks easyconfig should be pulled back out of archive...
    • should start doing syncs of 5.0.x with develop via PRs to avoid breaking CI
  • merged PRs (5.0.x branches)
    • docs
      • fix typo in example code for run_shell_cmd (PR #240)
    • framework
      • create lib -> lib64 symlink (or vice versa) before running postinstallcmds (PR #4435)
        • make sure that Brunsli libraries are installed into /lib (PR #19805)
        • -DCMAKE_INSTALL_LIBDIR=lib should be set by CMakeMake easyblock => Alex will open PR
    • easyblocks
      • over 20 PRs to remove unused easyblocks
      • ~50 PRs to migrate from run_cmd to run_shell_cmd in software-specific easyblocks
      • fail on non-zero exit code in command to run Perl test suite (PR #3170)
      • enable use_pip for extensions that are installed along with Python (PR #3221)
        • may need to revisit this and come up with a way to inherit defaults from PythonPackage easyblock?
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
      • archive or delete old patches (PR #19827)
      • replace run_cmd with run_shell_cmd in easyconfigs testsuite (PR #19818)
  • open PRs
    • framework
      • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
        • what happens if a PythonPackage that has not been migrated yet is used along with an easyblock on top (like numpy) that has been migrated
        • we can probably detect this and produce a meaningful error that easyblocks must be migrated
      • add support for running shell commands asynchronously with run_shell_cmd (PR #4444)
      • enable RPATH linking by default (PR #4448)
        • see also docs PR #241
        • waiting for Kenneth to re-review both PRs & merge
    • easyblocks
      • ~5 PRs to remove unused easyblocks
      • ~18 PRs to migrate software-specific easyblocks to run_shell_cmd
        • OpenBLAS easyblock still uses parse_log_for_error (which we want to deprecate, cfr. framework PR #4433)
        • OpenSSL => should catch RunShellCmdError instead of EasyBuildError
      • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
      • update run_async methods in RPackage, Rmpi, and Rserve easyblocks to use submit run_shell_cmd call into thread pool (PR #3163)
    • easyconfigs
      • add check to make sure that download_dep_fail, sanity_pip_check, use_pip are not explicitly set to True in easyconfigs (PR #19830)
  • WIP
    • (Kenneth) support for running interactive commands in run_shell_cmd (no PR yet)
  • notes
  • TODO
    • ...

2024-02-07 monthly sync meeting (14:00 CET)

attending: ?

  • status of 5.0.x branches
    • synced with develop on 2024-02-07, only trivial merge conflicts had to be fixed
  • merged PRs
    • framework
      • deprecate support for EnvironmentModulesC (v3.2.x) and EnvironmentModulesTcl (ancient v1.x) module tools (PR #4439)
    • easyblocks
      • various PRs to port software-specific easyblocks to run_shell_cmd
        • for Perl, we are ignoring exit code of shell command in test_step, that seems like a bug (PR #3162)
        • being fixed in easyblocks PR #3170
        • no easyconfig for PALM (recently merged easyblock)
  • open PRs
    • framework
      • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • should also switch to depends_on with Lmod as default?
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
      • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
        • "see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information" => this really needs to be updated (although link does still work), in a separate framework PR
      • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
        • blocked by PR #4415 to fix running of test suite with Tmod >= v4.3
      • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
        • WIP until run_shell_cmd implementation is complete (incl. Q&A support)
      • create lib/lib64 symlink before running postinstallcmds (PR #4435)
        • requires fixes in easyconfigs that will get broken because of this
      • add support for running shell commands asynchronously with run_shell_cmd (PR #4444)
        • last place where framework itself still relies on run_cmd
        • implementation of EasyBlock.install_extensions_parallel can be improved further, but can be left for future PRs (even post-5.0 release)
        • ready for review/merge! => maybe Alex
        • requires small change in easyblocks (see easyblocks PR #3163)
      • enable RPATH linking by default (PR #4448)
    • easyblocks
      • 33 open PRs to port software-specific easyblocks to run_shell_cmd...
      • 9 open PRs to remove unused easyblocks
      • update run_async methods in RPackage, Rmpi, and Rserve easyblocks to use submit run_shell_cmd call into thread pool (PR #3163)
      • fix extension filter for Perl packages (PR #2699)
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
  • TODO
    • directories that don't contain any library files shouldn't be added to $LD_LIBRARY_PATH (issue #3504)
      • makes sense for (bundles of) Python packages
    • Clean up SLURM/Batchsystem environment before doing builds (issue #4434)
      • should be done opt-in, via --clean-up-slurm-env configuration setting
    • change default backend for --job to Slurm (+ deprecate GC3Pie support)
    • document how to start playing with development pre-release version of EasyBuild 5.0
      • create easyconfig to install EasyBuild/dev-5.0.x-<datestamp>

2024-01-29 sprint meeting (14:30 CET)

attending: Simon, Jasper, Kenneth, Adam, Bart

2024-01-22 sprint meeting (16:00 CET)

attending: Jasper, Alex, Simon, Sebastian, Kenneth (sort of)

  • status of 5.0.x branches
    • synced with develop on 20240119
  • merged PRs
    • framework
      • bump minimum required Lmod to 8.0.0 (PR #4424)
      • don't hardcode /bin/bash in eb script, RPATH wrapper script, and run_shell_cmd (PR #4441)
      • fix error reporting when test step fails (PR #4443)
    • easyblocks
      • remove hidden=True from run_shell_cmd in PythonPackage easyblock for consistency (PR #3074)
      • don't change installopts easyconfig parameter value in-place in PythonPackage easyblock (PR #3080)
      • stop running easyblocks test suite with Lmod 7.x (PR #3083)
    • easyconfigs
      • archive EasyBuild 3.x easyconfigs (PR #19656)
  • open PRs
    • framework
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
      • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
      • create lib/lib64 symlink before running postinstallcmds (PR #4435)
      • deprecate support for EnvironmentModulesC and EnvironmentModulesTcl module tools (PR #4439)
        • should just kick out these modules tools from testing configuration
      • add support for running shell commands asynchronously with run_shell_cmd (WIP) (PR #4444)
    • easyblocks
      • fix extension filter for Perl packages (PR #2699)
        • (still) awaiting more tests, need to compose list of relevant easyconfigs
      • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
        • good progress on dealing with easyconfigs that were not enabling these parameters yet, see checklist in PR comment
        • open issues:
      • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
        • opt-out mechanism is implemented, some more requested changes => active work-in-progress
      • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
      • disable use_pip by default for PyTorch, except for recent versions (>= 2.0) (PR #3079)
        • blocker for PR #3022
        • should be ready to merge (despite failed test report)
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
        • synced with 5.0.x by @boegel on 18 Jan'24
        • we should also add a check in easyconfigs test suite to prevent that use_pip = True & co are re-introduced in easyconfigs...
      • replace parallel by maxparallel (PR #19375)
    • docs
      • document how to transition from run_cmd to run_shell_cmd (PR #238)
  • TODO
    • (Kenneth) finish run_shell_cmd async PR + review Bart's docs PR
    • (Simon) auto-enable use_pip for PyTorch
    • (Sebastian) auto-enable use_pip & co in PythonPackage
    • (Alex) renaming run PR
    • (Jasper) look into PR for parallel -> maxparallel
    • porting easyblocks to run_shell_cmd
      • separate PRs: GCC, PyTorch, TensorFlow, ...

2024-01-15 sprint meeting (15:00 CET)

attending: Kenneth, Bart, Simon, Sebastian

  • merged PRs (since last meeting)
  • open PRs
    • easyblock PR #2699: fix extension filter for Perl packages
      • waiting until other easyconfigs using PerlModule have been checked/fixed
    • easyblocks PR #3022: enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock
      • Waiting for fixing easyconfigs mentioned it the list
      • once problem with CMakePythonPackage is fixed in PythonPackage easyblock, half of the easyconfigs not enabling use_pip will be working fine
    • easyblock PR #3025: update ConfigureMake easyblock to error out on unknown configure args
      • opt-out support has been added
    • easyblock PR #3074: pythonpackage: Remove hidden=True from run_shell_cmd for consistency
    • framework PR #3285: Deprecate failure to resolve a template value
    • framework PR #4248: Change tar command used in get_source_tarball_from_git to get reproducible tarballs
    • framework PR #4373: drop load storm safe guard for Environment Modules v4.2.4+
    • framework PR #4398: Replace parallel easyconfig parameter by maxparallel
    • framework PR #4400: rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions
      • needs more work, current approach doesn't actually work to make sure that easyblocks that haven't migrated yet to new methods
      • easyblock PR #3064: rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async
    • framework PR #4424: bump minimum required Lmod to 8.0.0
      • should we required a more recent Lmod 8.x?
      • harder to pick a particular 8.x than just using 8.0
      • except if we require Lmod >= 8.2.8 which supports extensions in module files
      • => ready to merge!
    • framework PR #4425: bump minimum required Tmod (4.x) to 4.3.0
    • framwork PR #4433: deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP)
      • should only be merged once implementation of run_shell_cmd
    • framwork PR #4435: Create lib/lib64 symlink before running postinstallcmds
      • will break some existing easyconfigs, so should only be done for EasyBuild 5.0
  • other
    • we should also deprecate support for Tcl-based modules tools, like ancient pure Tcl + Tmod 3.2.x
    • support for GC3Pie should be deprecated + use Slurm as default job backend
  • TODO
    • documentation on how to convert run_cmd to run_shell_cmd (Bart)
    • support testing easyconfigs + easyblocks with 5.0.x branch (Sebastian)
    • async option or thread pool for run_shell_cmd (Kenneth)
    • fix issue with use_pip = True + CMakePythonPackage (Kenneth)
    • Sebastian: review Bart's easyblock PR

2024-01-03 sync meeting (14:00 CET)

attending: Kenneth, Bart, Simon, Mikael, Sebastian

  • merged PRs (since last meeting)
    • (none)
  • status of 5.0.x branches
    • synced with develop after release of EasyBuild v4.9.0 (on 2023-12-31 + 2024-01-01)
    • for now, let's keep merging changes for EasyBuild v5.0 to 5.0.x branches, and keep develop for a potential v4.9.x release (but hopefully there won't be a need for one)
    • re-evaluate during sync meeting of Feb'24
  • open PRs
    • framework
      • deprecate failure to resolve a template value (PR #3285)
        • Kenneth should take a detailed look at this
        • raising an error when a template could not be resolved is fine, since we don't really want to continue anyway if that's the case
      • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
        • Jasper's testing shows that this won't really work as is
        • doesn't work on macOS
        • only works when .git folder is not retained
        • in the fridge for now, awaiting feedback from contributor + Jasper
      • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • need to think how to align this with Lmod
        • do we change to using depends_on by default? (framework issue #4397)
          • helps with keeping default behaviour with Tmod & Lmod
          • supported since July 2017 (cfr. this commit)
          • => so yes
        • should also bump required Tmod version to 4.3 in PR #4373
          • better in a separate PR
        • along with Lmod to 8.x, see issue #871
      • detect Fortran .mod files in GCCcore installations (PR #4389)
        • only prints warning, unless EasyBuild is configured with --fail-on-mod_files-gcccore, so good to go as is? => yep
        • see also issue #4203 for list of easyconfigs where we need to whitelist (incl. Clang, AOCC, etc.)
      • replace parallel easyconfig parameter by maxparallel (PR #4398)
        • see also easyconfigs PR #19375
          • trivial change parallel -> maxparallel, but does change 721 easyconfigs (457 of which archived easyconfigs)
        • do we need to "soften the blow" a bit w.r.t. easyconfigs still using parallel (deprecate it rather than hard remove it)?
      • rename unclear run methods to install extensions (PR #4400)
        • basically ready to merge, except that log.deprecated calls should use '6.0' instead of '5.0'
        • needs matching easyblocks PR
      • use run_shell_cmd in scripts (PR #4422)
        • ready to review/merge => Simon
      • implement support for 'stream_output' option in run_shell_cmd (PR #4423)
        • ready to review/merge => Simon
    • easyblocks
      • fix extension filter for Perl packages (PR #2699)
        • awaiting test of other easyconfigs that (indirectly) rely on PerlModule easyblock (Kenneth)
      • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
      • update ConfigureMake easyblock to error out on unknown configure args (PR #3025)
      • convert generic easyblocks to run_shell_cmd (PR #3046)
        • awaiting review/testing (Sebastian)
    • easyconfigs
      • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
  • TODO
    • run_shell_cmd
      • implement async option
      • complete transition to run_shell_cmd in framework
      • complete docs (Bart)

2023-12-12 sync meeting (16:00 CET)

attending: Jasper, Kenneth, Sebastian, Bart, Simon, Alex, Alexandre

  • rename unclear run methods to install extensions (PR #4400)
    • let's deprecate run & co rather than remove them
    • run should print deprecation warning + call install_extension, etc.
  • migrate generic easyblocks to run_shell_command (PR #3046)
  • kim-api (PR #19399 through PR #19405, except #19400)
    • required because of check for Fortran .mod files when using GCCcore
    • there may be others...
  • change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • didn't work on macOS, so introduces a regression
    • got different checksums when retaining .git subdirectory
      • maybe because order of hidden files is not controlled?

2023-12-06 sync meeting (14:00 CET)

attending: Simon, Jasper, Sebastian, Kenneth, Adam, Alex, Mikael, Bart

  • merged PRs
    • better structure in docs for changes in EasyBuild v5.0 docs #230
      • note: only in develop branch for now, not in live documentation yet
  • open PRs
    • [Simon] detect Fortran .mod files in GCCcore installations PR #4389
      • we should make some extra effort to filter out blatant false positives?
      • actual Fortran .mod files will have this as output from file command: gzip compressed data
      • one step further would be to unzip and look for something like GFORTRAN module version ...., but that's maybe a bridge too far?
      • for now, let's merge as is, we can try to weed out false positives later if there's a real need for it
      • need to update boegelbot to set $EASYBUILD_FAIL_ON_MOD_FILES_GCCCORE=1 when using EasyBuild v5.0
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
      • Jasper will take a look soon
    • [Kenneth, Sebastian review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
      • [Sebastian] Clean up easyconfigs that set these True => easyconfigs PR #19265
    • Deprecate failure to resolve a template value (framework PR #3285)
      • needs to be ported to 5.0.x
    • update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
      • see also easyblocks PR #3026 to start producing warnings for unknown configure options in EasyBuild 4.x
      • we should first fix the easyconfigs that are affected by this...
  • ideas: yay or nay?
    • --read-only-installdir by default? (framework issue #123)
      • impact:
        • annoying to remove install dirs later
        • potentially problematic for sites that have a group of users installing software with EasyBuild
    • change order of extract and prepare steps (framework issue #1376)
      • load (only build?) dependencies before extracting sources
    • don't run sanity check in install dir (framework issue #4044)
      • Simon has some use cases where they're currently "abusing" sanity check commands to make changes in install, which require that environment is correctly set up
    • PythonPackage easyblock: modulename should be a first-class parameter, not under the options parameter (issue #2352)
    • fix inconsistent license variable naming used in easyblocks (easyblocks issue #3043)
      • should always be $EB_SOFTWARENAME_LICENSE_SERVER
    • CMakeMake could default CMAKE_BUILD_TYPE to RelWithDebInfo instead of Release (issue #3032)
    • separate framework issue to open (Bart): enable -g by default (debug toolchain option)
    • including module extensions info in generated module file should be enabled by default
      • implies a sufficiently recent Lmod (>= 8.2.0)
      • will work even with older Lmod versions, because there's a version check in generated module file itself
    • should depends_on by enabled by default? => Mikael will open an issue
  • TODOs
    • [Kenneth,Sebastian] set up copy of boegelbot to test with EasyBuild 5.0.x branches
      • Sebastian has looked into this a bit
      • can add support for BRANCH=5.0.x
    • [Kenneth | Alex | Bart] Document run_shell_cmd changes
      • should be PR to easybuild-v5/run_shell_cmd/ page in docs (develop branch)
    • [Kenneth + Alex] finish implementation of run_shell_cmd
      • improve error output in case of long output produced by command
        • detect long output, point to output file?
      • stream_output
      • asynchronous
      • qa_patterns + qa_wait_patterns
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
    • [Sebastian] get easyblocks PR #3022 merged to auto-enable use_pip & co
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • this should ideally be done before easyblocks PR #3022 is merged...
      • should also check whether easyblocks that derive from PythonPackage will need to get use_pip = False...
      • check in easyconfigs test suite only checks easyconfigs that use PythonPackage or PythonBundle, not stuff that derives from them => should be fixed
    • [Alex] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
      • deprecating use of .run(...)
      • better name:
        • Extension.install_as_extension?
      • definitely also affects Extension.postrun and Extension.prerun
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain
  • other
    • Bart's cheat sheet on run_shell_cmd 
      run_cmd -> run_shell_cmd

result:
(out, code) = ... -> res = ...; res.output, res.exit_code

parameter changes:
    log_ok, log_all, simple, regexp -> (remove!)
    inp -> stdin
    log_output=False -> output_file=True (reversed default)
    path -> work_dir
    force_in_dry_run -> in_dry_run
    verbose=True -> verbose_dry_run=False (reversed default)
    shell=None (->True) -> use_bash=True
    trace=True -> hidden=False (reversed meaning)
    stream_output, with_hooks (unchanged)
    asynchronous (NYI, don't touch for now!)
new parameters:
    fail_on_error=True, split_error=False, env=None
    qa_patterns=None (NYI)
    qa_wait_patterns=None (NYI)

Derived from detailed notes:
run_cmd                run_shell_cmd

(out, code)            .output, .exit_code, .stderr (if split_stderr=True)
cmd                    cmd
log_ok=True            (removed: always True)
log_all=False          (removed: always True)
simple=False           (removed: always return named tuple)
inp=None               stdin=None
regexp=True            (removed: no automatic scanning for stuff that looks like errors)
log_output=False       output_file=True
path=None              work_dir=None
force_in_dry_run=False in_dry_run=False
verbose=True           verbose_dry_run=False
shell=None(->True)     use_bash=True
trace=True             hidden=False
stream_output=None     stream_output=False
asynchronous=False     asynchronous=False (NYI)
with_hooks=True        with_hooks=True
                       fail_on_error=True (NEW)
                       split_error=False (NEW)
                       env=None (NEW)
                       qa_patterns=None (NEW,NYI)
                       qa_wait_patterns=None (NEW,NYI)

run_cmd
    :param cmd: command to run
    :param log_ok: only run output/exit code for failing commands (exit code non-zero)
    :param log_all: always log command output and exit code
    :param simple: if True, just return True/False to indicate success, else return a tuple: (output, exit_code)
    :param inp: the input given to the command via stdin
    :param regexp: regex used to check the output for errors;  if True it will use the default (see parse_log_for_error)
    :param log_output: indicate whether all output of command should be logged to a separate temporary logfile
    :param path: path to execute the command in; current working directory is used if unspecified
    :param force_in_dry_run: force running the command during dry run
    :param verbose: include message on running the command in dry run output
    :param shell: allow commands to not run in a shell (especially useful for cmd lists), defaults to True
    :param trace: print command being executed as part of trace output
    :param stream_output: enable streaming command output to stdout
    :param asynchronous: run command asynchronously (old: returns subprocess.Popen instance if set to True)
    :param with_hooks: trigger pre/post run_shell_cmd hooks (if defined)

run_shell_cmd
    :param fail_on_error: fail on non-zero exit code (enabled by default)
    :param split_stderr: split of stderr from stdout output
    :param stdin: input to be sent to stdin (nothing if set to None)
    :param env: environment to use to run command (if None, inherit current process environment)
    :param hidden: do not show command in terminal output (when using --trace, or with --extended-dry-run / -x)
    :param in_dry_run: also run command in dry run mode
    :param verbose_dry_run: show that command is run in dry run mode (overrules 'hidden')
    :param work_dir: working directory to run command in (current working directory if None)
    :param use_bash: execute command through bash shell (enabled by default)
    :param output_file: collect command output in temporary output file
    :param stream_output: stream command output to stdout
    :param asynchronous: run command asynchronously
    :param with_hooks: trigger pre/post run_shell_cmd hooks (if defined)
    :param qa_patterns: list of 2-tuples with patterns for questions + corresponding answers
    :param qa_wait_patterns: list of 2-tuples with patterns for non-questions
                             and number of iterations to allow these patterns to match with end out command output
    :return: Named tuple with:
    - output: command output, stdout+stderr combined if split_stderr is disabled, only stdout otherwise
    - exit_code: exit code of command (integer)
    - stderr: stderr output if split_stderr is enabled, None otherwise

2023-11-27 sprint (10:00 CET)

attending: Mikael, Sebastian, Bart

  • merged PRs
    • start adopting run_shell_cmd in easyblock.py (+ use f-strings) (PR #4383)
  • open PRs
    • [Kenneth,Mikael] better structure in docs for changes in EasyBuild v5.0 docs #230
    • [Kenneth] detect Fortran .mod files in GCCcore installations PR #4389
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • [WHO? review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
      • [Sebastian] Clean up easyconfigs that set these true PR #19265
    • Deprecate failure to resolve a template value (framework PR #3285)
      • needs to be ported to 5.0.x
    • update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
      • see also easyblocks PR #3026 to start producing warnings for unknown configure options in EasyBuild 4.x
      • we should first fix the easyconfigs that are affected by this...
  • TODOs
    • [Kenneth] set up copy of boegelbot to test with EasyBuild 5.0.x branches
    • [Kenneth | Alex] Document run_shell_cmd changes
    • [Kenneth + Alex] finish implementation of run_shell_cmd
      • stream_output
      • asynchronous
      • qa_patterns + qa_wait_patterns
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
      • framework PR #4383 can be helpful for inspiration
    • [Sebastian] get easyblocks PR #3022 merged to auto-enable use_pip & co
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • this should ideally be done before easyblocks PR #3022 is merged...
      • should also check whether easyblocks that derive from PythonPackage will need to get use_pip = False...
      • check in easyconfigs test suite only checks easyconfigs that use PythonPackage or PythonBundle, not stuff that derives from them => should be fixed
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
      • deprecating use of .run(...)
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain

2023-11-20 sprint (10:00 CET)

attending: Kenneth, Sebastian, Adam, Mikael

  • merged PRs
    • rename shell option in run_shell_cmd to use_bash (PR #4378)
    • print error message in main_with_hooks function when EasyBuildError was raised (PR #4380)
    • start adopting run_shell_cmd in easyblock.py (+ use f-strings) (PR #4383)
  • open PRs
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • [WHO? review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
    • Deprecate failure to resolve a template value (framework PR #3285)
      • needs to be ported to 5.0.x
    • update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
      • see also easyblocks PR #3026 to start producing warnings for unknown configure options in EasyBuild 4.x
      • we should first fix the easyconfigs that are affected by this...
  • TODOs
    • [Kenneth + Alex] finish implementation of run_shell_cmd
      • stream_output
      • asynchronous
      • qa_patterns + qa_wait_patterns
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
      • framework PR #4383 can be helpful for inspiration
    • [Kenneth,Adam] better structure in docs for changes in EasyBuild v5.0
    • [Sebastian] get easyblocks PR #3022 merged to auto-enable use_pip & co
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • this should ideally be done before easyblocks PR #3022 is merged...
      • should also check whether easyblocks that derive from PythonPackage will need to get use_pip = False...
      • check in easyconfigs test suite only checks easyconfigs that use PythonPackage or PythonBundle, not stuff that derives from them => should be fixed
    • [Sebastian] clean up easyconfigs that explicitly set use_pip & co to True
      • only after easyblocks PR #3022 is merged
      • use_pip = True, 'use_pip': True => remove
      • sanity_pip_check = True => remove
      • download_dep_fail = True => remove
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
      • deprecating use of .run(...)
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain

2023-11-13 sprint (10:00 CET)

attending: Kenneth, Adam, (Mikael)

  • merged PRs
    • change run_shell_cmd to store command output in temporary file(s) by default + pass RunShellCmdResult instance to RunShellCmdError (framework PR #4356)
  • open PRs
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • [WHO? review/merge] enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • [Kenneth] submit test reports
    • [Adam to review/merge] rename shell option in run_shell_cmd to use_bash (framework PR #4378)
  • next steps
    • [Alex,Kenneth] fix error reporting in 5.0.x branch
      • failing to resolve dependencies doesn't produce an error message anymore (only non-zero exit code)
    • [Kenneth + Alex] finish implementation of run_shell_cmd so framework can be updated to only use run_shell_cmd (no more run_cmd)
      • implement support for stream_output option (required in easybuild/tools/containers/*.py)
      • implement support for asynchronous option (required for --parallel-extensions-install support)
    • [Kenneth?] implement support for Q&A commands in run_shell_cmd
      • implement support for qa_patterns and qa_wait_patterns options
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
    • get easyblocks PR #3022 merged to auto-enable use_pip & co
      • Kenneth can look into submitting a bunch of test reports for affected easyblocks
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • WIP script to identify the easyconfigs to update (which somehow triggers open file handle limit...)
    • [WHO?] clean up easyconfigs that explicitly set use_pip & co to True
    • [Alex(structure),Bart(run_shell_cmd transition table),Adam?] update documentation to have a more complete overview of changes in EasyBuild 5.0
      • cfr. https://docs.easybuild.io/easybuild-v5/overview-of-changes
      • should include an overview of how to transition from run_cmd to run_shell_cmd, inspired by how easyblocks are updated to adopt run_shell_cmd
      • organise based on users vs contributors vs developers
        • Changes relevant to end users => docs/roles/end-users/easybuild-v5
          • trace output by default => docs/easybuild-v5/trace_output
          • error reporting => docs/easybuild-v5/error_reporting
          • Lmod version
          • toolchain support policy
          • Python 2 no longer supported
          • RIP experimental .yeb
          • RIP bootstrap script
        • contributors
          • easyconfigs
            • toolchain support policy
            • enforce correct .patch(.*) extension for patch files
          • easyblocks/framework
            • run_shell_cmd
            • py2vs3
            • RIP Python 2
            • removal of deprecated functionality
            • changed defaults
              • change_dir in extract_file
        • developers/maintainers
          • toolchain support policy
          • (see easyblocks/framework contributors)
      • separate pages for:
        • run_shell_cmd
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain

2023-11-08 sync meeting (14:00 CET)

attending: Kenneth, Bart, Simon, Alexandre, Sebastian, Adam, Alex, Jasper

  • merged PRs
    • use 5.0.x branch for easyblocks + easyconfigs in CI workflows (for now) (framework PR #4358)
    • take into account that VERBOSE_VERSION imported from easybuild.easyblocks is now a string value (framework PR #4357)
  • open PRs
    • [Kenneth,Alex] change run_shell_cmd to store command output in temporary file(s) by default + pass RunShellCmdResult instance to RunShellCmdError (framework PR #4356)
      • status: reviewed + updated based on review, ready to merge?
    • [Jasper] change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
      • PR branch needs to be updated with current 5.0.x to re-trigger CI
      • someone should play with this across different systems/OSs to verify whether this is indeed sufficient to obtain reproducible tarballs for easyconfigs using git_config
    • deprecate failure to resolve a template value (framework PR #3285)
      • needs thorough review/testing
      • needs to be updated with current 5.0.x branch to re-trigger CI
      • Kenneth will ask Alexander to move to 5.0.x
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • needs review + testing with a handful of existing easyconfigs to ensure no bugs have crept in the refactoring, at least one test for each easyblock that was touched in the PR
    • [Alexander?] update ConfigureMake easyblock to error out on unknown configure args (easyblocks PR #3025)
  • next steps
    • [Alex] get framework PR #4356 merged to fix storing of command output to file(s)
    • [Alex] fix error reporting in 5.0.x branch
      • failing to resolve dependencies doesn't produce an error message anymore (only non-zero exit code)
    • [Kenneth + Alex] finish implementation of run_shell_cmd so framework can be updated to only use run_shell_cmd (no more run_cmd)
      • rename shell option to use_bash
      • implement support for stream_output option (required in easybuild/tools/containers/*.py)
      • implement support for asynchronous option (required for --parallel-extensions-install support)
    • [Kenneth?] implement support for Q&A commands in run_shell_cmd
      • implement support for qa_patterns and qa_wait_patterns options
    • [Bart,Alex,Adam,Jasper] start porting easyblocks to run_shell_cmd
      • start with commonly used ones like ConfigureMake, CMakeMake, PythonPackage, EB_GCC, etc.
    • [WHO?] rename Extension.run method that is used for installing extensions (cfr. framework issue #4113)
    • get easyblocks PR #3022 merged to auto-enable use_pip & co
      • Kenneth can look into submitting a bunch of test reports for affected easyblocks
    • [Kenneth] update easyconfigs that use an easyblock that derives from PythonPackage but do not enable use_pip & co yet to explicitly use False
      • WIP script to identify the easyconfigs to update (which somehow triggers open file handle limit...)
    • [WHO?] clean up easyconfigs that explicitly set use_pip & co to True
    • [Alex(structure),Bart(run_shell_cmd transition table)] update documentation to have a more complete overview of changes in EasyBuild 5.0
    • [WHO?] see if we can limit the damage done by archiving easyconfigs using a deprecated toolchain
      • supported software (excl. versions) in EasyBuild v4.8.2 vs current 5.0.x branch: 3,292 vs 2,365 => ~29% decrease :-/

2023-10-23 sprint (10:00 CEST)

attending: Kenneth, Mikael

  • merged PRs
    • document policy on supported toolchain generations (docs PRs #200 + #216)
    • improve error reporting for failing shell commands (and EasyBuild crashes) (PR #4351)
    • use LooseVersion from easybuild.tools in all easyblocks (easyblocks PR #3018)
      • VERSION and VERBOSE_VERSION constants in easybuild.easyblocks are now string values (used to be LooseVersion instance)
      • requires framework #4357: take into account that VERBOSE_VERSION imported from easybuild.easyblocks is now a string value
    • don't auto-enable use of -DSCOTCH_PTHREAD when using MPI library other than Intel MPI in SCOTCH easyblock (PR #3019)
    • improve error reporting for failing shell commands (and EasyBuild crashes) #4351
  • open PRs
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (framework PR #4248)
    • take into account that VERBOSE_VERSION imported from easybuild.easyblocks is now a string value (framwork PR #4357)
    • change run_shell_cmd to always store command output in temporary file(s) + pass RunShellCmdResult instance to RunShellCmdError (WIP (framework PR #4356)
      • tests are failing, needs more work
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (easyblocks PR #3022)
      • worth doing a couple of test build for this...
  • other
  • next 5 goals
  • timeline
    • EasyBuild 4.8.2 release soon
      • may be last EasyBuild 4.x release
      • 2023b common toolchain update could be reason for EasyBuild v4.9.0 release (~end of Nov'23)
    • for now, stick to 5.0.x branches, until we're sure that EasyBuild 5.0 can be released soon enough

2023-10-16 sprint (10:00 CEST)

attending: Kenneth, Alex, Simon, Mikael, Adam

  • merged PRs
    • stop importing from easybuild.tools.py2vs3 (+ minor cleanup in init easyblocks test) (easyblocks PR #3015)
    • scripts to archive easyconfigs (easyconfigs PR #18934)
      • script used to archive easyconfigs using SYSTEM toolchain needs a whitelist to avoid archiving things like GCC-system.eb, etc.
    • archiving of old easyconfigs
      • all done for all toolchain versions older than 2020a (across 14 PRs)
      • remove old archived easyconfigs (EasyBuild 4.x archive) (PR #18982)
      • use more recent easyconfigs in tests for --search (PR #18995)
      • archive SYSTEM level easyconfigs from before 2018-10-11 (PR #19013)
      • fallout
        • support for 937 (out of ~3,300) software packages will be lost in EasyBuild 5.0
          • ~28% of all software EasyBuild currently supports...
          • with EasyBuild 4.0, is was way less dramatic (1840 -> 1669, ~9%)
          • we can/should add easyconfigs back with more recent toolchains to partially mitigate this...
          • a bunch of these are:
            • toolchains we no longer use
            • stuff like pandas that's still supported as extension
            • complex ones like RELION, ...
      • we should also do a cleanup sweep of patches
      • overview of software in EasyBuild docs should also include extensions?
  • open PRs
    • improve error reporting for failing shell commands (and EasyBuild crashes) (framework PR #4351)
      • reviewed by Alex, suggested changes made by Kenneth, ready for re-review + merge...
      • in some way, it would be easier to always have stdout and stderr command output split
        • but then we lose "context" of errors
        • can we collect both stdout+stderr mixed and stderr output only when running a command?
    • Use LooseVersion from easybuild.tools and sort imports (easyblocks PR #3018)
      • use of LooseVersion in easybuild/easyblocks/__init__.py is causing trouble
    • don't auto-enable use of -DSCOTCH_PTHREAD when using MPI library other than Intel MPI in SCOTCH easyblock (easyblocks PR #3019)
  • next 5 goals
    • [Kenneth,Alex] get (framework PR #4351) merged (better error reporting)
    • [Kenneth] implement support for stream_output in run_cmd_shell function + switch to run_cmd_shell in easybuild/tools/containers/*.py
    • [Mikael,Kenneth] get (easyblocks PR #3018) merged (LooseVersion import)
      • need to stop using LooseVersion in easybuild/easyblocks/__init__.py
    • [Sebastian?] enable download_dep_fail + use_pip + sanity_pip_check by default in PythonPackage + PythonBundle easyblocks
      • easyconfigs cleanup should be less painful now
      • Kenneth is working on script to identify eaysconfigs that do not already enable these parameters
    • [Alex] start switching to run_shell_cmd in easyblocks...

2023-10-09 sprint (10:00 CEST)

attending: Kenneth, Mikael, Sebastian, Adam, Alexandre

  • merged PRs
  • open PRs
    • improve error reporting for failing shell commands (and EasyBuild crashes) (framework PR #4351)
      • => will be reviewed/tested/merged this week by ...
    • stop importing from easybuild.tools.py2vs3 (+ minor cleanup in init easyblocks test) (easyblocks PR #3015)
      • => will be reviewed/merged this week by ...
    • scripts to archive easyconfigs (easyconfigs PR #18934)
      • => will be reviewed/tested/merged this week by ...
  • next 5 goals
    • [Alex + Kenneth] get framework PR #4351 that implements better error reporting merged
    • [Kenneth] implement support for stream_output in run_cmd_shell function + switch to run_cmd_shell in easybuild/tools/containers/*.py
    • [Chia-jung Hsu] switch easyblocks to LooseVersion from easybuild.tools.* (only change import statements)
    • [Sebastian] enable download_dep_fail + use_pip + sanity_pip_check by default in PythonPackage + PythonBundle easyblocks
      • explicitely set these to False in easyconfigs not setting them to True
        • easyconfigs using any easyblock that derives from PythonPackage or PythonBundle
        • Kenneth can look into generating a list of easyconfigs that will need fixing (setting to False)
      • clean up easyconfigs explicitly setting these to True
      • easyconfigs test suite should report explicit setting these to True
    • [Mikael] archive easyconfigs
      • using a 2016a/2016b/2016.x toolchain
      • using GCC 5.4.0 or older as toolchain
      • using scripts implemented by Mikael (easyconfigs PR #18934)

2023-10-04

Attendees: Kenneth, Sebastian, Mikael, Alexandre, Bart, Simon, Jasper, Adam

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-10-02 (commit d450a80)
  • merged PRs:
    • ...
  • open PRs:
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
  • TODO
    • for run_shell_cmd
      • improve error reporting
        • raise EasyBuildError instead of subprocess.CalledProcessError
        • implement better error reporting, using a new report_error function that uses return value produced by run_shell_cmd (which cmd used, includes output, exit code, working dir, ...)
        • WIP: 
          == installing...
  >> running command:
        [started at: 2023-10-04 13:59:58]
        [working dir: /private/tmp/kehoste/bzip2/1.0.6/system-system/bzip2-1.0.6]
        [output logged in /tmp/eb-leexxvrd/easybuild-run-rrpla98w.log]
        xxmake -f Makefile-libbz2_so CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'

- full command: xxmake -f Makefile-libbz2_so CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'
- exit code: 127
- working directory: /private/tmp/kehoste/bzip2/1.0.6/system-system/bzip2-1.0.6
- output: /tmp/eb-leexxvrd/shell-cmd-errorq2fsapve/xxmake.out
- stderr: /tmp/eb-leexxvrd/shell-cmd-errorq2fsapve/xxmake.err

ERROR: shell command 'xxmake' failed!
        • run_shell_cmd could create script to source to start interactive debug shell, something like: 
          export TEST=foo
cd /tmp/test
history -s "test 1 2 3"
echo "Failing command was 'test 1 2 3'"
      • implement missing options in run_shell_cmd
        • stream_output
        • asynchronous
        • qa_patterns + qa_wait_patterns
      • complete transition to run_shell_cmd in easybuild-framework
        • TODO:
          • easybuild/framework/easyblock.py
          • easybuild/framework/extension.py
          • easybuild/toolchains/linalg/flexiblas.py
          • easybuild/tools/containers/*.py
          • easybuild/tools/job/slurm.py
          • easybuild/tools/options.py
          • easybuild/tools/package/utilities.py
      • gradually switch easyblocks to run_shell_cmd (in 5.0.x branch)
      • add support for dropping into a interactive shell if command failed (cfr. issue #3950)
        • global configuration option to opt-in to letting EasyBuild drop into an interactive shell if a shell command failed
        • only when command is not expected to fail
        • only when there's a tty
        • opt-out option in run_shell_cmd for specific commands
    • tackle other ideas, see https://github.com/easybuilders/easybuild-framework/labels/EasyBuild-5.0
  • how can we activate more people?
    • half-day "sprint" to outline next things to do?
    • better overview of easy things to do to make progress => open issues
  • archiving of easyconfigs
    • for oldest toolchains (older than 2020a)
      • generation by generation
      • step 1: grep '^toolchain.*2019b' easybuild/easyconfigs
      • step 2: grep '^toolchain.*GCC.*8.3.0' easybuild/easyconfigs
      • step 3: close easyocnfig PRs using these toolchains
      • don't worry about conflicts that get introduced by PRs currently open to develop
    • by letter subdir
      • doesn't really work due to dependencies
    • oldest easyconfigs using system toolchain => archive software versions older than 5 years
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-10-04 (commit b2d86f5)
    • trivial conflicts fixed
  • next steps
    • switching from run_cmd to run_shell_cmd
      • tricky for ones using run_cmd_qa
    • switching to LooseVersion from easybuild.tools.* (only change import statements)
    • enable use_pip + sanity_pip_check in PythonPackage
      • clean up easyconfigs after archiving
    • remove dead easyblocks
    • remove dead code in easyblocks (ancient software versions)
    • update config.guess in ConfigureMake (issue #2826)
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-10-04 (commit db2d79d)
    • trivial conflicts fixed
task assignments
  • [Kenneth] run_shell_cmd error reporting
  • [Mikael] archiving easyconfigs
    • {foss,intel}/2016a + GCC(core) 4.9.3-2.25
    • script in scripts/ subdirectory in easybuild-easyconfigs repo
  • [Mikael's colleague?] flip defaults for download_dep_fail + use_pip + sanity_check_pip in PythonPackage easyblock
  • [Mikael's colleague?] fix LooseVersion imports in easyblocks
  • [Sebastian] update config.guess in ConfigureMake (issue #2826)
  • [Mikael] clean up SCOTCH easyblock (issue #2972)
sprint meeting (max 1h)
  • Mon 9 Oct 10:00 CEST

2023-09-06

Attendees: Kenneth Hoste, Simon Branford, Alex Domingo, Jasper Grimm, Mikael Öhman, Bart Oldeman, Alexandre Strube

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-09-06 (commit 02ed943)
    • conflicts fixed, mostly due to support for run_shell_cmd hook that was added in develop via PR #4323
  • merged PRs:
    • Simon's PR to remove deprecated functionality:
      • PR #4273 remove deprecated descr option for simple_option
      • PR #4274 remove deprecated Toolchain.add_dependencies
      • PR #4275 remove deprecated functionality from filetools
      • PR #4276 remove deprecated log_error option from which function
      • PR #4277 remove deprecated skip_lower option from template_constant_dict
      • PR #4278 remove deprecated options from easyconfig.py
      • PR #4279 removed deprecated mod_exists_regex_template in ModulesTool.exist
    • progress on run_shell_cmd:
      • initial implementation of run function to replace run_cmd + run_cmd_qa (PR #4284)
      • copy load_source function from easybuild.tools.py2vs3.py3 to easybuild.tools.filetools so it can be used in easybuild.tools.hooks (PR #4306)
      • disable trace output for commands run in get_source_tarball_from_git (PR #4310)
      • fix broken github tests in options test.framework.options by disabling trace output in download_repo and fetch_files_from_pr functions (PR #4311)
      • fix for test_get_source_tarball_from_git broken by trace output (PR #4310)
      • fix broken github tests in options test.framework.options by disabling trace output in download_repo and fetch_files_from_pr functions (PR #4311)
      • implement fail_on_error/in_dry_run/output_file options + enable caching for run function, and switch from run_cmd to run function in systemtools (PR #4314)
      • implement support in run function for splitting stdout and stderr output (split_stderr) + passing down environment to use, and switch to run function for running module commands (PR #4321)
      • also include command used + working directory in return value of run function (PR #4322)
      • implement support in 'run' function for running command in different working directory + switch to run function in filetools (PR #4327)
      • trigger run_shell_cmd hook in run function (PR #4334)
        • run_shell_cmd hook will also be supported in EasyBuild v4.8.1
      • rename run function to run_shell_cmd (PR #4335 + PR #4336)
  • open PRs:
    • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
  • TODO
    • for run_shell_cmd
      • improve error reporting
        • raise EasyBuildError instead of subprocess.CalledProcessError
        • implement better error reporting, using a new report_error function that uses return value produced by run_shell_cmd (which cmd used, includes output, exit code, working dir, ...)
      • implement missing options in run_shell_cmd
        • stream_output
        • asynchronous
        • qa_patterns + qa_wait_patterns
      • complete transition to run_shell_cmd in easybuild-framework
        • TODO:
          • easybuild/framework/easyblock.py
          • easybuild/framework/extension.py
          • easybuild/toolchains/linalg/flexiblas.py
          • easybuild/tools/containers/*.py
          • easybuild/tools/job/slurm.py
          • easybuild/tools/options.py
          • easybuild/tools/package/utilities.py
      • gradually switch easyblocks to run_shell_cmd (in 5.0.x branch)
      • add support for dropping into a interactive shell if command failed (cfr. issue #3950)
        • global configuration option to opt-in to letting EasyBuild drop into an interactive shell if a shell command failed
        • only when command is not expected to fail
        • only when there's a tty
        • opt-out option in run_shell_cmd for specific commands
    • tackle other ideas, see https://github.com/easybuilders/easybuild-framework/labels/EasyBuild-5.0
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-09-06 (commit d7b8b45)
    • trivial conflicts fixed
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-09-06 (commit a82280d)
    • trivial conflicts fixed

2023-08-02

Attendees: Kenneth, Kurt, Alexandre, Sebastian, ...

Discussion

docs
Policy changes
  • toolchain support policy (issue #872)
  • supported Lmod versions (issue #871)
    • definitely drop support for Lmod 6
    • dropping support for Lmod 7 probably has low impact
    • should we require a minimal Lmod 8.x version (8.2 or 8.3)
      • for extensions, depends_on, etc.
      • bump to REQ_VERSION to 8.0 (or 8.3), DEPR_VERSION to 8.5
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • merged PRs:
    • PR #4272 remove support for 32-bit targets
  • open PRs:
    • PR #4284: initial progress by Kenneth, following discussion with Alex, on new run function (issue #4252)
  • several PRs for code cleanup in 5.0.x branch, by removing deprecated stuff
    • PR #4273 remove deprecated descr option for simple_option
    • PR #4274 remove deprecated Toolchain.add_dependencies
    • PR #4275 remove deprecated functionality from filetools
    • PR #4276 remove deprecated log_error option from which function
    • PR #4277 remove deprecated skip_lower option from template_constant_dict
    • PR #4278 remove deprecated options from easyconfig.py
    • PR #4279 removed deprecated mod_exists_regex_template in ModulesTool.exist
  • proposals for EasyBuild 5.0:
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • stop using deprecated py2vs3 module in easyblocks
    • start switching to new run function
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

2023-07-05

Attendees: Kenneth, Simon, Jasper, Kurt, Mikael, Alex, Adam, Sebastian, Bart

Discussion

docs
Policy changes
  • toolchain support policy (issue #872)
  • supported Lmod versions (issue #871)
    • definitely drop support for Lmod 6
    • dropping support for Lmod 7 probably has low impact
    • should we require a minimal Lmod 8.x version (8.2 or 8.3)
      • for extensions, depends_on, etc.
      • bump to REQ_VERSION to 8.0 (or 8.3), DEPR_VERSION to 8.5
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • merged PRs:
    • PR #4250: enabling --trace by default (by Simon)
    • PR #4270 update check_python_version to reflect that only Python >= 3.6 is supported
    • stop relying on imp (will no longer be available in Python 3.12) (issue #4264)
  • open PRs:
    • PR #4284: initial progress by Kenneth, following discussion with Alex, on new run function (issue #4252)
      • feedback from Simon in PR
      • next steps:
        • expand tests run with new run function (in same PR?)
        • start using new run function in framework where possible (no unsupported options used) - follow-up PR?
        • start using new run in easyblocks
          • probably split across multiple PRs
          • with reasonabe effort regarding testing, not testing all easyconfigs affected...
        • implement missing features by expanding basic run function
        • support for setting environment variables in the scope of the command being run
        • implement support for dropping into a shell when a command fails (issue #3950)
          • should be configurable (globally + per run invocation)
            • shell_on_error=True?
          • only do this when exit on error is enabled (fail_on_error=True)
          • only do this when eb is run in an interactive shell
          • Mikael has looked into this a bit
          • use pty.spawn + something like cmd || bash -l ?
  • several PRs for code cleanup in 5.0.x branch, by removing deprecated stuff
    • PR #4272 remove support for 32-bit targets
    • PR #4273 remove deprecated desc option for simple_option
    • PR #4274 remove deprecated Toolchain.add_dependencies
    • PR #4275 remove deprecated functionality from filetools
    • PR #4276 remove deprecated log_error option from which function
    • PR #4277 remove deprecated skip_lower option from template_constant_dict
    • PR #4278 remove deprecated options from easyconfig.py
    • PR #4279 removed deprecated mod_exists_regex_template in ModulesTool.exist
  • proposals for EasyBuild 5.0:
    • rename unclear ExtensionEasyblock.run() method (issue #4113)
    • include entire flag in optarch (instead of excluding the initial dash) and maybe make it only per-compiler (issue #4269)
      • --optarch='march=x86-64 -mtune=generic' vs --optarch='-march=x86-64 -mtune=generic'
    • stop relying on distutils (https://github.com/easybuilders/easybuild-framework/issues/3963)
    • require Lmod >= 7.0 ?
    • eb options simplification
      • John: "Using eb to build software is the last step. eb has a large number of arguments that continue to grow."
        • separate ec command doesn't seem like a good idea
      • Simon: It might be possible to make a split using subparser, but I have no firm idea or suggestion here and I'm unlikely to spend time on this.
        • eb build, eb github, ...
        • unclear where all options would end up
        • significant effort
        • some options would apply to multiple subcommands, so would be "top-level" (like --debug)
        • opportunity to review current options and see if they're still needed and make sense
        • re-design of CLI may result in significant delay for EasyBuild 5.0
          • so maybe something for a "quick" EasyBuild 6.0 release?
        • to some extent this "complaint" can be tackled by defining eb* aliases
        • main issue is that output of eb --help is massive
        • maybe this is more about shortening of help output
          • eb --help=github, ...
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • stop using deprecated py2vs3 module in easyblocks
    • start switching to new run function
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

2023-06-07

Attendees: Bart, Sebastian, Kenneth, Jasper, Simon, Mikael, Alexandre

Pre-meeting notes

Discussion

  • GitHub Actions no longer support testing on top of Python 2.7 (cfr. framework PR #4267)

  • proposal on splitting up extensions for Python/Perl/R

    • prepped by Alex + Mikael
    • see https://hackmd.io/K1QhWGfdQWa2ZtlXlWVCCA
    • progress in context of 2023a common toolchains:
      • Perl/5.36.1-GCCcore-12.3.0 merged: (PR #17946)
        • includes only a handful of extensions, those required by Autoconf/Automake (Thread::Queue, Data::Dumper)
        • separate Perl-bundle-CPAN for additional Perl modules from CPAN: (PR #17970)
      • Python/3.11.3-GCCcore-12.3.0 merged (PR #17955)
        • only includes setuptools + pip + wheel (+ flit_core dep) as extensions
        • next step would be Python-bundle-PyPI (+ maybe separate easyconfigs for virtualenv, hatchling, poetry, etc.)
      • R: coming soon...

Done

docs
  • WIP PR #137 to include overview of changes in EasyBuild 5.0
framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
  • some progress on enabling --trace by default by Simon (PR #4250)
    • Simon: fixing the tests is painful, but we're very close, only 1 test is failing (unexpected output being generated)
    • could consider setting $EASYBUILD_DISABLE_TRACE in test setup?
  • very little progress by Kenneth on new run function (issue #4252)
    • may be useful to pair up with someone on this in the coming weeks?
  • proposals for EasyBuild 5.0:
easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
  • no merged or WIP PRs for now
  • TODO
    • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
    • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
      • and then remove use_pip = True & co in all easyconfigs
    • update version of config.guess used by ConfigureMake (issue #2826)
      • important to test across a bunch of OSs & CPU architectures
        • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
        • Kenneth: power9le (emulated)
        • Sebastian: riscv64
easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

  • 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
  • no merged or WIP PRs for now
  • TODO
    • archive easyconfigs using a deprecated toolchains + toolchain policy
    • archive ancient easyconfigs using system toolchain
    • archive directories that only have patches in them + cleaned up unused patches
    • remove use_pip = True & co in all easyconfigs

TODO

Clone this wiki locally