New RFC: EESSI/EasyBuild and nf-core

[ocaisa]

Have you read the RFC docs?

• Yes, I have read and understood the RFC docs

Summary

Leverage the European Environment for Scientific Software Installations (EESSI, pronounced as "easy") to provide portability of pipelines between CPU and GPU architectures. EESSI currently supports 14 different CPU builds across x86 and ARM CPU families and all major CUDA compute capabilities, using a best-fit approach to match the instructions available on the current CPU to the builds that EESSI ships.

Champion

@ocaisa

Background & Motivation

EESSI is a collaborative project that creates a common, high-performance scientific software stack for use on high-performance computing (HPC) systems and other Linux-based devices like laptops, workstations and cloud environments. It provides a way for researchers to access the same software on their laptop, in the cloud, or across different European HPC sites, ensuring portability, performance, and reproducibility. The software is distributed via CernVM-FS (CVMFS), a read-only file system, making it easy to stream and use without complex installation processes.

In the past, EESSI has used Nextflow for demonstrations to show how it would be possible to ship a portable workflow pipeline including the complete execution environment via a GitHub repository. The utility of this is limited however without full support for the nf-core/modules required by the pipeline.

We (EESSI) and others within the EasyBuild community (EasyBuild is the backend that is used to install software within EESSI), are interested in exploring the support for nf-core pipelines via EasyBuild and EESSI.

Goals

Practically speaking, with respect to EasyBuild this would likely mean setting the following targets for specific pipelines (and pipeline versions):

• Map the nf-core/module requirements of a pipeline to software available in EasyBuild (adding new software if necessary)
  • This would be done inside an easyconfig which is a recipe for an installation (in this case the specific version of the pipeline)
  • Dependencies are explicitly listed (including their verions), required nf-core/module would also likely need to be explicitly listed. We note that https://github.com/nf-core/modules does not have releases, so we would pin the nf-core/module by commit.
  • a mapping is likely required between the EasyBuild naming and nf-core naming (this would be done centrally)
• Test the installation of the pipeline
  • This is done in a general way via an easyblock (Python code)
  • Requires the testing of the individual nf-core/modules as the versions the nf-core/modules use in their environment files are unlikely to match the current versions shipped via EasyBuild (EasyBuild has a strong preference for the latest versions of packages). We would be relying on the tests for nf-core/modules to verify a specific module works as expected using the EasyBuild supplied software stack.
  • We would also run the tests for pipeline itself before proceeding with the final installation.

Once the pipeline is installed an end user could then use environment modules to load the specific pipeline:

module load nf-core::sarek/3.6.1

Shipping the pipeline via EESSI is a relatively trivial step once a working EasyBuild recipe is in place.

From the Nextflow perspective, we would be interested in understanding how support for EESSI as an execution environment might be supported. Since we would control the installation of Nextflow being used (Nextflow itself would be a dependency of the EasyBuild recipe), there are likely a number of options here.

Non-Goals

• not looking to support all nf-core pipelines at once, it would be case-by-case with the idea of creating a general mechanism for adding support for a pipeline to EasyBuild/EESSI

References

• Introduction to EESSI presentation
• EESSI has used Nextflow for demonstration of how a workflow including the execution environment could be shipped in a Git repository
• Nextflow in EESSI demos repository

[nf-core-bot]

Approval status: 🕐 Pending

RFC has approvals from 1/7 required @nf-core/core quorum.

Review Status	Core Team members
✅ Approved	@maxulysse
🕐 Pending	@ewels, @mribeirodantas, @mashehu, @JoseEspinosa, @mirpedrol, @FriederikeHanssen, @kenibrewer, @jfy133, @edmundmiller, @sateeshperi, @FranBonath, @christopher-hakkaart, @nvnieuwk

[maxulysse]

/approve

I think this is a good idea, and leveraging modules that are already available and "just" mapping them in a custom config file for a specific pipeline/version doesn't seem too complex and not too dissimilar to what we did already at least once for arm in rnaseq (cf https://github.com/nf-core/rnaseq/blob/master/conf/arm.config).

I know that more and more people in the HPC world are using such solutions, and I know that at nf-core we took the road towards just having a conda yml environment file that gets converted into docker and singularity (either via biocontainers or seqera containers) seems a bit different in philosophy, and we should be aware of potential reproducibility risks that could arise due to the difference in building recipe.

[ewels]

Thanks for this @ocaisa , it looks interesting! Couple of points to start with:

I think you should be the champion, not @maxulysse - assuming that it'll be you that will be sheparding this feature through to completion (even if not doing the work, being responsible to keep it on track).

I'm a bit confused about the stuff regarding modules. In nf-core, modules are incorporated into pipelines by developers and the code is copied into the pipeline repo. So at run time (which is what we care about here), there is no involvement with nf-core/modules. I don't see why we need to do any version tracking or listing or anything. Could you elaborate on why this is needed please?

Much of what you describe sounds to me like it would happen outside of Nextflow and the nf-core pipeline (eg. pulling in Nextflow itself at a specific version), so I'd be curious about what changes (if any) would be needed within the nf-core pipeline code.

I don't know how similar this is to environment modules, but it might be worth noting that Nextflow already has a module -process directive to support those. See the docs. These can be specified within pipeline code, or also within separate Nextflow config files. We used to use these in the very early days of some nf-core pipelines (before nf-core itself started in fact), e.g here, but we dropped them because the module names varied between institutes, so it wasn't portable.

Phil

[ocaisa]

I think you should be the champion, not @maxulysse - assuming that it'll be you that will be sheparding this feature through to completion (even if not doing the work, being responsible to keep it on track).

Done

I'm a bit confused about the stuff regarding modules. In nf-core, modules are incorporated into pipelines by developers and the code is copied into the pipeline repo. So at run time (which is what we care about here), there is no involvement with nf-core/modules. I don't see why we need to do any version tracking or listing or anything. Could you elaborate on why this is needed please?

Indeed our main concern is run time. EESSI is providing the binaries that are used at runtime and we try very hard to ensure we run all the tests that the developers provide. My understanding here was that https://github.com/nf-core/modules/tree/master/modules/nf-core is ultimately the "source of truth" for end users of the modules, and our main interest is having access to the (most up to date) tests for the components of the pipeline. EESSI will provide the binaries for the components, the critical thing for us is to have a way to test that components work as they are expected to without having any understanding of what the component actually does (as we will most likely be using a different version of the component to that specified in environment.yml).

Much of what you describe sounds to me like it would happen outside of Nextflow and the nf-core pipeline (eg. pulling in Nextflow itself at a specific version), so I'd be curious about what changes (if any) would be needed within the nf-core pipeline code.

I would hope to see zero changes to nf-core/modules or pipeline code. EESSI would be consuming the tests from both of these and running them on a wide range of architectures before shipping the environment modules for the pipeline. Experience to date has told us that most changes that result from inclusion in EESSI are changes in test suites to adapt to architectures that the applications had not been previously exposed to.

I don't know how similar this is to environment modules, but it might be worth noting that Nextflow already has a module -process directive to support those. See the docs. These can be specified within pipeline code, or also within separate Nextflow config files. We used to use these in the very early days of some nf-core pipelines (before nf-core itself started in fact), e.g here, but we dropped them because the module names varied between institutes, so it wasn't portable.

Indeed, EESSI uses the Lmod implementation of environment modules to allow people to switch between versions. EESSI exactly targets portability, not just between sites but between CPU architectures (we try very hard to keep a "same software everywhere" promise, and are currently even working on RISC-V support). What EESSI does differently is that it has the capacity to distribute the binaries and the module files, and a detection mechanism to know which binaries it should run on the system where it is used. Within the EESSI environment you would no longer have the variance with respect to module names or software versions between institutes. For example, EESSI is part of the EuroHPC Federation Platform and will soon be available on all EuroHPC supercomputers (and is already available on the majority).

It is possible that simply providing a default Nextflow config file when using EESSI-shipped Nextflow is enough, indeed these are the types of things where we would like to collaborate on. For example, saying a configuration file may be enough leads me to ask: are there environment variables for configuration files, are they PATH-like, if a user configuration file is somehow provided are settings from the "default" configuration still picked up? (Important note: I'm not a Nextflow user so I am learning as we go here).

[ocaisa]

It's been raised that pipelines may really require a specific version of an nf-core module, as behaviour may change between versions.

There are ways for EasyBuild to deal with this. If we rely on test accuracy, we could run the test from upstream nf-core/modules: if those succeed, run the end-to-end tests from the pipeline; if the end-to-end fails, run the module tests shipped with the pipeline. We use the pipeline module test to identify the problem case, with the general guess that it is version specific behaviour.
In EasyBuild you can have support for components , which you could think of like a sub-installation. We could use components to install specific versions of tools in the cases where it is really needed, and do that in such a way that they are visible to the pipeline but don't leak into the wider environment.