Support workflow modules (kind: Workflow) in the module system - #7363
Draft
jorgee wants to merge 9 commits into
Draft
Support workflow modules (kind: Workflow) in the module system#7363jorgee wants to merge 9 commits into
jorgee wants to merge 9 commits into
Conversation
Extend the module system to support standalone *workflow* modules alongside process modules, per ADR 20260608-workflow-modules (PR #7342). This covers installing, resolving and including workflow modules; direct typed execution (`module run`) is still WIP. - ModuleSpec/ModuleSpecFactory: parse a top-level `kind` (Process|Workflow, default Process) and a nested `requires.modules` list of direct dependencies. - ModuleResolver: install a workflow module together with its transitive dependencies, vendored per-module under nested `modules/` directories (no cross-module flattening; duplication accepted); dependency-cycle detection. - Context-relative include resolution in all three paths — runtime (IncludeDef) and compile-time (nf-lang ResolveIncludeVisitor and ModuleResolver source discovery) — so a workflow module's own includes resolve under its nested `modules/` directory. - CmdModuleInstall/CmdModuleRun install with dependencies. - Publish/validate: branch on `kind` (a workflow module must define a `workflow`), plus a schema-location override (`-schema` flag, NXF_MODULE_SPEC_SCHEMA env, or meta.yml `$schema`) to bridge the not-yet-published remote schema. - adr/module-spec-schema.json: allow `kind` and `requires.modules` (exact-pin references), and nested scope/name module names. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Add a Kind column to `nextflow module list` (and a `kind` field to its JSON output), sourced from the installed module's meta.yml and defaulting to Process when absent. - InstalledModule: carry the module `kind`. - ModuleStorage: populate it from the parsed spec (default Process). - CmdModuleList: render the Kind column and JSON field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Extend `nextflow module validate` (and thus `module publish`, which runs the same validation) with two checks: - requires.modules: each declared `scope/name@version` dependency must be vendored under the module's own nested `modules/` at the pinned version. The publish bundle ships these vendored deps, so a declared-but-unvendored or version-drifted dependency would produce a broken bundle. The check is local/offline (no registry access), so `-dry-run` still works. - workflow interface: a workflow module must define exactly one workflow; when its meta.yml declares input/output, the counts must match the workflow's take:/emit: arity (mirrors the process input/output count check). When the meta.yml omits them, the take:/emit: sections remain the source of truth. ModuleSpecFactory.workflowInterfaces extracts take/emit arity via a syntactic parse, so includes of not-yet-installed dependencies still parse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
A workflow module's requires.modules dependencies are vendored under its own nested `modules/` directory for local development and validation, but they are re-resolved from the registry at install time. Bundling them into the parent module's publish artifact duplicated registry content and bloated the bundle. Skip the module-root `modules/` directory when creating the bundle. Deeper directories such as `resources/modules/` are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
The integrity checksum was saved by installModule *before* a module's requires.modules dependencies were vendored into its nested modules/ directory, so a freshly installed workflow module with dependencies always read as MODIFIED. The checksum covers the whole installed subtree (own files + vendored deps, excluding .module-info), so it must be computed once the subtree is complete. Refresh each (re)installed module's checksum in walkDependencies after its dependencies are vendored (post-order). A module that is reused as-is (already present at the requested version) is left untouched, preserving any local modification status. As a result a clean install reads VALID, while a later edit -- to the module's own files or to a vendored dependency -- is detected as a modification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Add a `-update-deps` option to `nextflow module install` that, for an already-installed module, re-vendors its dependencies to match the module's (possibly locally-edited) meta.yml requires.modules, without reinstalling the module itself. This automates the manual workflow of bumping a dependency version in meta.yml and pulling the change into the nested modules/ directory. - ModuleResolver.updateDependencies: install/update each declared dependency at its pinned version (transitively), and prune vendored dependencies that are no longer declared. A dependency with local modifications is neither overwritten nor pruned -- an error is raised instead. The parent module is left untouched (its checksum is not refreshed), preserving its unpublished modification status. - CmdModuleInstall: -update-deps is ignored when the module is not installed (a normal install runs), and is mutually exclusive with -force. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
… tags Extend the module CLI and spec tooling for workflow modules and typed modules (no npr-api changes; builds against the current npr-api). - module create: add -kind (Process|Workflow) and -typed. Typed scaffolds set nextflow.enable.types and use typed input/output (process) or take/emit (workflow). meta.yml is generated accordingly: typed workflow input/output derived from take/emit; untyped workflow take/emit documented as channels; typed process/workflow declare the minimum Nextflow version that introduced them (25.10.0 / 26.04.0). - module spec: support workflow modules, deriving input/output from the single workflow's take:/emit: (types inferred when statically typed, TODO placeholder otherwise). - module-spec schema: add 'channel' and 'custom-record' to the parameter type enum as documentation tags for statically-typed declarations; ModuleSpecVisitorV2 maps Channel<T> and record types accordingly. - docs: document the new create options and workflow spec generation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
jorgee
force-pushed
the
260714-workflow-modules
branch
from
July 24, 2026 10:50
e1f20cc to
39715ef
Compare
The requires.modules publish check previously required each declared dependency to be installed (vendored) under the module's own nested modules/ directory. That was wrong: dependencies are not bundled with a module (they are excluded from the publish bundle) and are re-resolved from the registry by consumers at install time, so a dependency need not be installed locally to publish -- it needs to exist in the registry. - Remove the local-vendored dependency check from ModuleValidator (module validate remains offline; the schema still validates the reference format). - Add ModuleResolver.findMissingDependencies, which checks each declared dependency resolves in the registry (getModuleRelease at the pinned version). The registry client falls back across all configured registries, so a dependency is accepted if it exists in any configured repo. - CmdModulePublish verifies declared dependencies against the registry before publishing (and in -dry-run), only when the module declares requires.modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Surface the module kind (Process/Workflow) in the search and view commands, defaulting to Process when the registry does not report one. - CmdModuleSearch: add kind to the formatted and JSON output. - CmdModuleView: add a Kind line and a kind JSON field. - Bump io.seqera:npr-api / npr-client to 0.24.10, which adds the kind field to ModuleSearchResult and ModuleMetadata (seqeralabs/nextflow-registry#366). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the workflow modules feature from the ADR (#7342): the module system is extended to support standalone workflow modules (
kind: Workflow) alongside process modules — published, installed, resolved, included, and (via a follow-up PR) executed through the existing registry /nextflow moduleCLI infrastructure.What's implemented
Spec model & schema
ModuleSpec/ModuleSpecFactoryparse a top-levelkind(Process|Workflow, defaultProcess) and a nestedrequires.moduleslist of direct dependencies.adr/module-spec-schema.jsonallowskind,requires.modules(exact-pin references), nestedscope/namenames, and addschannel/custom-recordto the parametertypeenum as documentation tags for statically-typed declarations.Install / include
ModuleResolverinstalls a workflow module together with its transitive dependencies, each vendored under its own nestedmodules/directory (modules/<scope>/<name>/modules/...). No cross-module flattening — duplication accepted per the ADR's diamond-dependency decision — plus dependency-cycle detection.IncludeDef) and both compile-time paths (nf-langResolveIncludeVisitorand the source-discoveryModuleResolver) resolve a module's remote includes relative to the including file's directory, so a workflow module's own includes find its nested dependencies.VALID, and edits to either the module or a vendored dependency are detected asMODIFIED.module install -update-deps— for an already-installed module, re-vendors its dependencies to match the (possibly locally-edited)meta.ymlwithout reinstalling the module itself: installs new deps, updates changed versions, prunes removed ones (a locally-modified dependency is never silently overwritten or pruned). Ignored when the module is not installed; mutually exclusive with-force.Publish / validate
kind-aware validation: a workflow module must define exactly oneworkflow; when itsmeta.ymldeclaresinput/output, the counts must match the workflow'stake:/emit:arity.requires.modulesare validated against the registry at publish time (they exist at their pinned version), since dependencies are re-resolved from the registry by consumers at install time.-schemaflag,NXF_MODULE_SPEC_SCHEMAenv, or meta.yml$schema).nextflow moduleCLI — kind awarenesslist— shows aKindcolumn (andkindJSON field).create—-kind Process|Workflowand-typed. Typed scaffolds setnextflow.enable.typesand use typedinput/output(process) ortake/emit(workflow); the generatedmeta.ymlfollows suit, and typed process/workflow declare the minimum Nextflow version that introduced them (25.10.0 / 26.04.0). An untyped workflow scaffold documents its channel take/emit.spec— supports workflow modules, derivinginput/outputfrom the workflow'stake:/emit:(types inferred when statically typed;TODOplaceholder otherwise).search/view— show the modulekind(formatted + JSON), defaulting toProcess.Registry dependency (npr-api 0.24.10 → seqeralabs/nextflow-registry#366)
The
kindshown bymodule search/module viewcomes from the registry API, which addedkindtoModuleSearchResult/ModuleMetadatain seqeralabs/nextflow-registry#366 (npr-api 0.24.10). This branch bumpsio.seqera:npr-api/npr-clientto0.24.10.CI will not pass until npr-api 0.24.10 is published to the Seqera Maven repository. To build and test locally in the meantime:
Not in this PR (follow-ups)
nextflow module run scope/name— entry-workflow synthesis fromtake:/emit:, samplesheet input loading, and work-dir-path outputs — is in a separate PR.module search --kindfilter (registry-side support needed).Testing
ModuleSpecFactoryTest(process + workflow spec, typed types,channel/custom-record),ModuleValidatorTest,ModuleSchemaValidatorTest,ModuleResolverDependencyTest(nested install layout, per-consumer duplication, cycle detection, auto-install, checksum/integrity,-update-deps),ModuleStorageTest(bundle excludes nestedmodules/),CmdModuleCreateTest,CmdModuleListTest,CmdModulePublishTest,CmdModuleSearchTest,CmdModuleViewTest.kind: Workflowmodule (adapted from nf-coremafft_align+ itsmafft/aligndependency) to a dev registry, ranmodule install, and included/composed it in a pipeline — verified working.Notes
kind+requires.modules, exposingkindon search/view) are in seqeralabs/nextflow-registry#366.