Skip to content

Support workflow modules (kind: Workflow) in the module system - #7363

Draft
jorgee wants to merge 9 commits into
adr-workflow-modulesfrom
260714-workflow-modules
Draft

Support workflow modules (kind: Workflow) in the module system#7363
jorgee wants to merge 9 commits into
adr-workflow-modulesfrom
260714-workflow-modules

Conversation

@jorgee

@jorgee jorgee commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Draft — stacked on #7342 (the workflow-modules ADR). Base is adr-workflow-modules so the diff shows only the implementation; retarget to master once #7342 lands.

Implements the workflow modules feature from the ADR (#7342): the module system is extended to support standalone workflow modules (kind: Workflow) alongside process modules — published, installed, resolved, included, and (via a follow-up PR) executed through the existing registry / nextflow module CLI infrastructure.

What's implemented

Spec model & schema

  • ModuleSpec/ModuleSpecFactory parse a top-level kind (Process|Workflow, default Process) and a nested requires.modules list of direct dependencies.
  • adr/module-spec-schema.json allows kind, requires.modules (exact-pin references), nested scope/name names, and adds channel / custom-record to the parameter type enum as documentation tags for statically-typed declarations.

Install / include

  • Nested vendoringModuleResolver installs a workflow module together with its transitive dependencies, each vendored under its own nested modules/ directory (modules/<scope>/<name>/modules/...). No cross-module flattening — duplication accepted per the ADR's diamond-dependency decision — plus dependency-cycle detection.
  • Context-relative include resolution — the runtime path (IncludeDef) and both compile-time paths (nf-lang ResolveIncludeVisitor and the source-discovery ModuleResolver) resolve a module's remote includes relative to the including file's directory, so a workflow module's own includes find its nested dependencies.
  • Integrity checksum covers the module's full installed subtree (own files + vendored deps), computed after vendoring — so a freshly installed workflow-with-deps reads VALID, and edits to either the module or a vendored dependency are detected as MODIFIED.
  • module install -update-deps — for an already-installed module, re-vendors its dependencies to match the (possibly locally-edited) meta.yml without reinstalling the module itself: installs new deps, updates changed versions, prunes removed ones (a locally-modified dependency is never silently overwritten or pruned). Ignored when the module is not installed; mutually exclusive with -force.

Publish / validate

  • kind-aware validation: a workflow module must define exactly one workflow; when its meta.yml declares input/output, the counts must match the workflow's take:/emit: arity.
  • requires.modules are validated against the registry at publish time (they exist at their pinned version), since dependencies are re-resolved from the registry by consumers at install time.
  • Nested vendored dependencies are excluded from the publish bundle (consumers re-resolve them from the registry).
  • Schema-location override (-schema flag, NXF_MODULE_SPEC_SCHEMA env, or meta.yml $schema).

nextflow module CLI — kind awareness

  • list — shows a Kind column (and kind JSON field).
  • create-kind Process|Workflow and -typed. Typed scaffolds set nextflow.enable.types and use typed input/output (process) or take/emit (workflow); the generated meta.yml follows suit, and typed process/workflow declare the minimum Nextflow version that introduced them (25.10.0 / 26.04.0). An untyped workflow scaffold documents its channel take/emit.
  • spec — supports workflow modules, deriving input/output from the workflow's take:/emit: (types inferred when statically typed; TODO placeholder otherwise).
  • search / view — show the module kind (formatted + JSON), defaulting to Process.

Registry dependency (npr-api 0.24.10 → seqeralabs/nextflow-registry#366)

The kind shown by module search / module view comes from the registry API, which added kind to ModuleSearchResult / ModuleMetadata in seqeralabs/nextflow-registry#366 (npr-api 0.24.10). This branch bumps io.seqera:npr-api / npr-client to 0.24.10.

CI will not pass until npr-api 0.24.10 is published to the Seqera Maven repository. To build and test locally in the meantime:

# 1. Build the registry API client from PR #366 (VERSION 0.24.10) and publish it to local Maven
cd <nextflow-registry checkout on seqeralabs/nextflow-registry#366>
./gradlew :npr-api:publishToMavenLocal :npr-client:publishToMavenLocal

# 2. In this repo, add mavenLocal() to the repositories block in build.gradle (do NOT commit this):
#    repositories {
#        mavenLocal()
#        mavenCentral()
#        ...
#    }

# 3. Build / run nextflow as usual
make compile

Not in this PR (follow-ups)

  • Direct typed execution nextflow module run scope/name — entry-workflow synthesis from take:/emit:, samplesheet input loading, and work-dir-path outputs — is in a separate PR.
  • module search --kind filter (registry-side support needed).
  • Typed-record samplesheet loading and the ADR's no-output-directory / index-file outputs (both follow-ups of the module-run PR).

Testing

  • Unit: ModuleSpecFactoryTest (process + workflow spec, typed types, channel/custom-record), ModuleValidatorTest, ModuleSchemaValidatorTest, ModuleResolverDependencyTest (nested install layout, per-consumer duplication, cycle detection, auto-install, checksum/integrity, -update-deps), ModuleStorageTest (bundle excludes nested modules/), CmdModuleCreateTest, CmdModuleListTest, CmdModulePublishTest, CmdModuleSearchTest, CmdModuleViewTest.
  • Manual e2e: published a kind: Workflow module (adapted from nf-core mafft_align + its mafft/align dependency) to a dev registry, ran module install, and included/composed it in a pipeline — verified working.

Notes

  • The registry-side changes (persisting/serving kind + requires.modules, exposing kind on search/view) are in seqeralabs/nextflow-registry#366.

jorgee and others added 7 commits July 20, 2026 10:31
Extend the module system to support standalone *workflow* modules
alongside process modules, per ADR 20260608-workflow-modules (PR #7342).
This covers installing, resolving and including workflow modules; direct
typed execution (`module run`) is still WIP.

- ModuleSpec/ModuleSpecFactory: parse a top-level `kind`
  (Process|Workflow, default Process) and a nested `requires.modules`
  list of direct dependencies.
- ModuleResolver: install a workflow module together with its transitive
  dependencies, vendored per-module under nested `modules/` directories
  (no cross-module flattening; duplication accepted); dependency-cycle
  detection.
- Context-relative include resolution in all three paths — runtime
  (IncludeDef) and compile-time (nf-lang ResolveIncludeVisitor and
  ModuleResolver source discovery) — so a workflow module's own includes
  resolve under its nested `modules/` directory.
- CmdModuleInstall/CmdModuleRun install with dependencies.
- Publish/validate: branch on `kind` (a workflow module must define a
  `workflow`), plus a schema-location override (`-schema` flag,
  NXF_MODULE_SPEC_SCHEMA env, or meta.yml `$schema`) to bridge the
  not-yet-published remote schema.
- adr/module-spec-schema.json: allow `kind` and `requires.modules`
  (exact-pin references), and nested scope/name module names.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Add a Kind column to `nextflow module list` (and a `kind` field to its
JSON output), sourced from the installed module's meta.yml and defaulting
to Process when absent.

- InstalledModule: carry the module `kind`.
- ModuleStorage: populate it from the parsed spec (default Process).
- CmdModuleList: render the Kind column and JSON field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Extend `nextflow module validate` (and thus `module publish`, which runs the
same validation) with two checks:

- requires.modules: each declared `scope/name@version` dependency must be
  vendored under the module's own nested `modules/` at the pinned version.
  The publish bundle ships these vendored deps, so a declared-but-unvendored
  or version-drifted dependency would produce a broken bundle. The check is
  local/offline (no registry access), so `-dry-run` still works.
- workflow interface: a workflow module must define exactly one workflow; when
  its meta.yml declares input/output, the counts must match the workflow's
  take:/emit: arity (mirrors the process input/output count check). When the
  meta.yml omits them, the take:/emit: sections remain the source of truth.

ModuleSpecFactory.workflowInterfaces extracts take/emit arity via a syntactic
parse, so includes of not-yet-installed dependencies still parse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
A workflow module's requires.modules dependencies are vendored under its own
nested `modules/` directory for local development and validation, but they are
re-resolved from the registry at install time. Bundling them into the parent
module's publish artifact duplicated registry content and bloated the bundle.

Skip the module-root `modules/` directory when creating the bundle. Deeper
directories such as `resources/modules/` are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
The integrity checksum was saved by installModule *before* a module's
requires.modules dependencies were vendored into its nested modules/ directory,
so a freshly installed workflow module with dependencies always read as
MODIFIED. The checksum covers the whole installed subtree (own files + vendored
deps, excluding .module-info), so it must be computed once the subtree is
complete.

Refresh each (re)installed module's checksum in walkDependencies after its
dependencies are vendored (post-order). A module that is reused as-is (already
present at the requested version) is left untouched, preserving any local
modification status. As a result a clean install reads VALID, while a later
edit -- to the module's own files or to a vendored dependency -- is detected
as a modification.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Add a `-update-deps` option to `nextflow module install` that, for an
already-installed module, re-vendors its dependencies to match the module's
(possibly locally-edited) meta.yml requires.modules, without reinstalling the
module itself. This automates the manual workflow of bumping a dependency
version in meta.yml and pulling the change into the nested modules/ directory.

- ModuleResolver.updateDependencies: install/update each declared dependency at
  its pinned version (transitively), and prune vendored dependencies that are no
  longer declared. A dependency with local modifications is neither overwritten
  nor pruned -- an error is raised instead. The parent module is left untouched
  (its checksum is not refreshed), preserving its unpublished modification status.
- CmdModuleInstall: -update-deps is ignored when the module is not installed
  (a normal install runs), and is mutually exclusive with -force.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
… tags

Extend the module CLI and spec tooling for workflow modules and typed modules
(no npr-api changes; builds against the current npr-api).

- module create: add -kind (Process|Workflow) and -typed. Typed scaffolds set
  nextflow.enable.types and use typed input/output (process) or take/emit
  (workflow). meta.yml is generated accordingly: typed workflow input/output
  derived from take/emit; untyped workflow take/emit documented as channels;
  typed process/workflow declare the minimum Nextflow version that introduced
  them (25.10.0 / 26.04.0).
- module spec: support workflow modules, deriving input/output from the single
  workflow's take:/emit: (types inferred when statically typed, TODO placeholder
  otherwise).
- module-spec schema: add 'channel' and 'custom-record' to the parameter type
  enum as documentation tags for statically-typed declarations; ModuleSpecVisitorV2
  maps Channel<T> and record types accordingly.
- docs: document the new create options and workflow spec generation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
jorgee force-pushed the 260714-workflow-modules branch from e1f20cc to 39715ef Compare July 24, 2026 10:50
jorgee and others added 2 commits July 24, 2026 14:37
The requires.modules publish check previously required each declared dependency
to be installed (vendored) under the module's own nested modules/ directory. That
was wrong: dependencies are not bundled with a module (they are excluded from the
publish bundle) and are re-resolved from the registry by consumers at install
time, so a dependency need not be installed locally to publish -- it needs to
exist in the registry.

- Remove the local-vendored dependency check from ModuleValidator (module
  validate remains offline; the schema still validates the reference format).
- Add ModuleResolver.findMissingDependencies, which checks each declared
  dependency resolves in the registry (getModuleRelease at the pinned version).
  The registry client falls back across all configured registries, so a
  dependency is accepted if it exists in any configured repo.
- CmdModulePublish verifies declared dependencies against the registry before
  publishing (and in -dry-run), only when the module declares requires.modules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Surface the module kind (Process/Workflow) in the search and view commands,
defaulting to Process when the registry does not report one.

- CmdModuleSearch: add kind to the formatted and JSON output.
- CmdModuleView: add a Kind line and a kind JSON field.
- Bump io.seqera:npr-api / npr-client to 0.24.10, which adds the kind field to
  ModuleSearchResult and ModuleMetadata (seqeralabs/nextflow-registry#366).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant