ADR: Workflow modules

[bentsherman]

Spun out from #7213

This PR proposes an ADR for workflow modules -- extending the module system to support both processes and workflows.

Updated based on latest round of feedback

[netlify]

✅ Deploy Preview for nextflow-docs ready!

Name	Link
🔨 Latest commit	d691aa8
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a57dbcae494250008bc5f8e
😎 Deploy Preview	https://deploy-preview-7342--nextflow-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[bentsherman]

Addressing @pditommaso 's feedback in more detail:

1. The workflows/ prefix seems used a little inconsistently.

ADR updated -- both process modules and workflow modules are now stored in the modules directory

2. Worth clarifying that a workflow module can include any module.

The ADR does specify this, but I added an example of a workflow module's directory tree to make it more clear

3. The workflow interface is probably best defined by take:/emit: rather than duplicated in the spec.

Agreed. I added an example of inputs/outputs in the module spec. At the very least we need name and description. Not sure if type is really needed for v1.

4. Dependencies + versioning — there's a bit of tension here.

ADR updated -- workflow modules now get their own modules directory to store nested dependencies. It will make for some long file paths but it seems like the right trade-off.

5. The params/output mapping might be better as its own ADR.

I kept this section here for now so that I can continue to refine the mapping rules. I will probably just move it back to #7208

[bentsherman]

TODO: clarify mapping rules for CLI params -> workflow takes, beyond the one example

Consider moving back to #7208

[bentsherman]

TODO: evaluate the impact on project size

It should be minimal if we install only the script+spec and exclude tests. In principle you don't need the tests in your pipeline if they were verified when the module was published

[bentsherman]

We might also consider the impact on compile time, since Nextflow will end up parsing more scripts when duplication is allowed. But this should be minimal compared to overall runtime

[bentsherman]

Transitive dependencies also complicates how we sync nf-core subworkflows to the registry. They have includes based on nf-core's directory structure (flat, separate modules vs subworkflows) whereas ideally they should use the canonical module name as shown here

Perhaps we can be clever and re-write these include statements when importing to the registry? Should be possible with some regex, or an AST transform is always an option

[jorgee]

I have some differences about how modules are managed in workflows and pipelines. In modules, we propose to use the module.requires and they will be stored in the workflow modules folder during the installation. However, the current module version pinning in pipelines is done by committing the module to the modules folder. If the module is not committed, Nextflow uses the latest.

My doubt is why we do not apply the pipelines way for workflows? So, the version is pinned if the module is included in the folder; otherwise, use the latest. In this way, we will not need the module.requires and looking at the repo, it seems to match the current behaviour in the nf-core subworkflows repo. They use relative paths to the modules folder in the repo, so it is pointing to the latest version. Adopting it, the nf-core subworkflows import will be basically uploading `main.nf, changing the relative path in imports by the module references.

The problem that I could see is if there are a big amount modules pinned that could pass the 1MB limit for the upload compressed artifact.

[bentsherman]

Are you talking about bundling the transitive modules in the workflow module? i suppose we could do that with just a directory tree of .module-info files. But it just doesn't seem necessary when we already have the module spec

The version pinning approach we took for modules was mostly a way to avoid depending on the pipeline spec since we don't have it yet. Once we have the pipeline spec, I could see us moving the version pins to there. So I wouldn't worry too much about consistency there

[jorgee]

Are you talking about bundling the transitive modules in the workflow module?

Yes, I was thinking it also to simplify the import of nf-core subworkflow. But it can make publisehd subworkflows unreliable. I think we can extract the nf-core modules versions from the git commit.

[awgymer]

it seems to match the current behaviour in the nf-core subworkflows repo. They use relative paths to the modules folder in the repo, so it is pointing to the latest version

Yeah... my experience of this from working closely with nf-core components is that it was a design mistake and any new system should avoid it because it causes so many headaches.

If it's about concordance with the way modules operate currently I'd argue it would be better to just make people have to pin modules too.

The problem that I could see is if there are a big amount modules pinned that could pass the 1MB limit for the upload compressed artifact.

FWIW I tested running a simple zip on the entire nf-core/modules folder (only the .nf and meta yml files no test code) and that would be 4.4M for about ~2.5k modules. I think a 1MB limit on compressed size would be quite difficult to hit for even large subworkflows. All the equivalent files in one of my personal nf-core like pipelines was 153K .

EDIT:
Just thinking about this further, I think that it's fairly imperative that users are able to update the version of modules which are pinned within a workflow module type locally. i.e. I might install nf-core/complex_subworkflow and for some specific reason I may want to upgrade or downgrade process module within that workflow. With that in mind it may make sense to have the registry just carry the manifest and then the default install pull the versions in that manifest into the nested module tree and you would have some e.g. nextflow module update-deps command which would allow the user to manually update the deps in said manifest and then refresh the install.

Of course this would probably require the top-level manifest to carry the full nested module tree of dependencies because otherwise it would become tricky to update a nested workflow module.

[jorgee]

Just thinking about this further, I think that it's fairly imperative that users are able to update the version of modules which are pinned within a workflow module type locally. i.e. I might install nf-core/complex_subworkflow and for some specific reason I may want to upgrade or downgrade process module within that workflow. With that in mind it may make sense to have the registry just carry the manifest and then the default install pull the versions in that manifest into the nested module tree and you would have some e.g. nextflow module update-deps command which would allow the user to manually update the deps in said manifest and then refresh the install.

Not sure if I have really get your point. Do you want to update the dependencies of an already installed workflow (locally) or update the dependencies of a published workflow in the remote server? For the first one, I think you can do it with the current module install commands. If you go to modules/nf-core/complex_subworkflow and you run nextflow module install <module-to-update>@<version>, it will override the module in the current workflow installation. I am not sure if this command should also update the module.requires in the local workflow manifest or not. For the second case, you can publish a new version of the workflow with the updated manifest.

[jorgee]

is requires.modules just allowing to pin a version or are we going to support ranges or constraints (< 2.0.0)

[bentsherman]

My inclination is to not implement version ranges for v1, since we can always add it later

It depends on whether other module namespaces will want to use version ranges or if they will want to be like nf-core

[awgymer]

The first.

One issue right now is that nf-core does not have "real" versioning of modules. Its arbitrary git commits and a different "version" might mean a logic change or it might mean formatting, or a container which can be overridden in config anyway - or any combination of these.

My issue with needing to do modules install in arbitrarily nested locations is:

• highly manual per-module
• not traceable
• prone to human error

[jorgee]

Regarding this, is it required to validate if the required modules have been published before, or is it a workflow developer's responsibility? Ideally, it we will not publish incomplete workflows or with mistakes in the dependencies. However, it could be tricky because there are workflows that can be published in a registry depending on modules stored in other registries, and the dependency resolution will depend on the local configuration os the user (repositories set in registry.urls). For instance, an organization that is hosting its own registry used to store workflows that depend on modules in the public registry.

[awgymer]

What does eg cargo/pypi do for this?

[jorgee]

Asking Claude, seems both options are used and depends on the registries architecture. It seems cargo checks because they have the central (crates.io). However, Maven does not check dependencies and the error is discovered at install time.

[awgymer]

As a user I think it would be a much worse experience if the validation didn't happen at publish time. I think it's also worth considering that nextflow has so far been a primarily bioinformatics platform and that many users are not necessarily coming from engineering backgrounds and the friction of failing installs and/or not providing publish-time checks could carry more risk than in some languages

[bentsherman]

I figured we would validate this at publish time. Publishing a workflow module should require that all module dependencies are already published in the same registry

Cross-registry deps can be handled for now by mirroring the public registry into the private one

[bentsherman]

@awgymer pulling your question into a thread:

One issue right now is that nf-core does not have "real" versioning of modules. Its arbitrary git commits and a different "version" might mean a logic change or it might mean formatting, or a container which can be overridden in config anyway - or any combination of these.

My issue with needing to do modules install in arbitrarily nested locations is:

• highly manual per-module
• not traceable
• prone to human error

In theory, I think you would be able to edit the requires.modules section of an installed workflow module, changing versions to your liking, then run nextflow modules install/update once in the project root and Nextflow would recursively install modules to align it with the module specs. Nextflow would just warn about locally-modified modules for certain operations

This is different from how we currently manage modules though, since we don't have a pipeline spec to pin module versions right now but workflow modules will pin their dependencies. So we'll have to think about this more

[awgymer]

Yeah a pipeline really needs a way to pin module versions I think (other than at the include point)

[bentsherman]

@awgymer this is the main assumption I wanted to check from the nf-core side

My sense is that module developers will not want to deal with semantic versioning and version ranges across thousands of modules, which is why we are going down this route of just duplicating the modules in every workflow

I'm curious which trade-off you think folks would prefer?

[awgymer]

It will be a split I expect, but personally I would rather we moved to semantic versioning - although a transition period within nf-core would probably be needed

[muffato]

I think introducing another versioning on the modules/nf-core side will be confusing and a pain to maintain. Upstream software often don't follow semantic versioning. We shouldn't burden module developers with finding out how backwards-compatible each software update is to come up with a parallel semantic version.
If we want a version number, I'd stick to the version of the wrapped software itself, with a "build" number attached to it like in Conda. Every time the software is updated, the build number goes back to 0. Every time the module itself is updated, the build number is incremented.

[awgymer]

I don't think the version has anything to do with the internal tool really?

Semantic versioning is reasonably clear on what the different levels mean but it's also easy enough to define one for nextflow. Something like:

• major: i/o channel interface changes
• minor: non-breaking functional change (ie container bump)
• patch: formatting, meta yml etc

The current system of git sha gives the user absolutely 0 indication and also has no way to tell at a glance if a version is newer or older than another one.

You could pin it to the internal tool version but:

• some modules use multiple tools
• build versions again give no indication if the nextflow module has breaking changes or is purely formatting

[bentsherman]

After discussing with the team, I am less worried about this question. I think as long as workflow modules depend only on other registry modules, Nextflow should be able to duplicate or deduplicate the dependency tree at install-time

So I think the current proposal makes sense as a first step, and we can add support for version ranges / resolution in the future

[jorgee]

Apart from the install, publish and run, there are other implications of adding workflows in other module command.

• list: Include the module kind
• spec: Generate the spec for a workflow
• validate: Validate a workflow module (here we could check if dependencies exists according to the configuration)
• search: filter by 'kind'
• create: Add an option to generate a basic workflow module?
• view: It should render workflow modules

[jorgee]

In module run for processes, we support running with no static types, relying on the process input definitions (path, val, tuple) and the meta.yml to convert CLI to the input channels.

In workflows, I see the ADR focuses on typed workflows. Are we only going to support module run executions for static typed workflows, or both?

If it is just for typed workflows, nf-core subworkflows are not currently typed, so if we try to automatically import nf-core subworkflows in the current state, they will not be used in module run. So, I wonder if this automatic migration makes sense.

To support non-typed workflows, we should rely on a good meta.yml description as processes. Current, the ADR examples and the nf-core workflows schema only set a type per channel; however, it is not possible to describe the internal structure of the channel. Looking at the nf-core subworkflow meta.ymls, it looks like some of them have tried to define the channel structure. See below some examples:

1. structure: property — used but absent from the schema. mafft_align:

• ch_fasta:
  type: file
  pattern: ".{bam/cram/sam}" # ← stale copy-paste: it's a FASTA subworkflow
  structure:
  - - meta: {type: map, ...}
  - fasta: {type: file, pattern: ".{fa,fasta}", ontologies: [...]}
  structure nests the tuple's element definitions — the schema doesn't define it (0 occurrences), so it's unvalidated.

2. Structure encoded in free-text description. fastq_align_star:

• ch_reads:
  description: |
  List of input FastQ files ...
  Structure: [ val(meta), [ path(reads) ] ] # ← tuple shape as prose
  (note ch_reads here has no type at all.)

3. Input keys = flattened internal channel elements, NOT the take channels. bam_sort_stats_samtools:
   take: ch_bam, ch_fasta_fai # 2 channels
   input: meta, bam, fasta # 3 entries — the elements INSIDE the channels

So, in case we want to support the execution of non-typed workflow we should include a structure or payload property to describe the internal structure of the channel.

[bentsherman]

@jorgee I just don't think there is enough information in legacy workflows, even in the meta.yml, to support direct execution. That's why I de-scoped it, although we can always try in the future

I don't think it's a blocker for registry auto-import though. The module run command should just fail for legacy workflows. This functionality doesn't exist yet anyway so there's no regression. The core feature is the publish / install / inclusion