Skip to content

Commit ded64b5

Browse files
committed
Document the package-provider system
docs/package.md (registered in the docs toctree): the provider/plugin/Wave-compatibility matrix, manifest auto-detection and custom-named manifest rules, per-process options, and the v2 parser requirement. Add the `packages` scope to the config reference, the `nextflow.preview.package` entry to the feature-flags reference, and the `package` directive to the process reference. Record the design decision as an ADR (adr/20260707-package-provider-plugins.md) following the convention on master. Signed-off-by: Evan Rees <evanroyrees@gmail.com>
1 parent 58f1119 commit ded64b5

7 files changed

Lines changed: 420 additions & 238 deletions

File tree

PACKAGE_MANAGEMENT.md

Lines changed: 0 additions & 219 deletions
This file was deleted.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Package provider plugins
2+
3+
- Authors: Evan Rees
4+
- Status: proposed
5+
- Deciders: Nextflow maintainers
6+
- Date: 2026-07-07
7+
- Tags: plugins, packages
8+
9+
## Summary
10+
11+
Provide a single generic `package` process directive whose package managers are implemented as pluggable providers, each shipped as its own `nf-<provider>` plugin.
12+
13+
## Problem Statement
14+
15+
Nextflow supports software dependencies through per-manager process directives (`conda`, `spack`), and each new package manager requires its own directive and core runtime changes. Requests for additional managers keep arriving (nix and guix in #1526, uv in #7023, pixi and R managers in the #6342 review), and the per-manager approach does not scale: every addition grows the core, the process DSL, and the config surface.
16+
17+
## Goals or Decision Drivers
18+
19+
- One directive for all package managers, so processes declare dependencies the same way regardless of the tool.
20+
- New managers must be addable without core runtime changes.
21+
- Alignment with Wave: providers that Wave can build (conda-based) hand off to Wave; others build locally.
22+
- The feature must be opt-in (preview flag) and must not affect existing pipelines.
23+
24+
## Non-goals
25+
26+
- Implementing Wave container builds for non-conda providers.
27+
- Removing or deprecating the legacy `conda` and `spack` directives.
28+
- Extracting the existing conda support out of core (the legacy `conda` directive still depends on it).
29+
30+
## Considered Options
31+
32+
### Per-manager directives
33+
34+
Continue adding one process directive per package manager (`uv`, `pixi`, `nix`, ...).
35+
36+
- **Pro:** Explicit and familiar; mirrors the existing `conda`/`spack` directives.
37+
38+
- **Con:** Every manager requires parser, DSL, config, and runtime changes in core; N managers means N directives to document and maintain.
39+
40+
### Wave as the single integration point
41+
42+
Route all package specifications through Wave and let it build the environments.
43+
44+
- **Pro:** One container-based path for every manager.
45+
46+
- **Con:** Wave only builds conda-based package environments; forcing other managers through it is not possible today, and it makes containers mandatory where local environments suffice.
47+
48+
### Generic directive with pluggable providers (chosen)
49+
50+
A `package` directive backed by a `PackageProvider` SPI in core, with each manager implemented in its own `nf-<provider>` plugin.
51+
52+
- **Pro:** Managers ship, version, and test independently; third parties can add providers without touching core.
53+
54+
- **Con:** `package` is a Groovy/Java keyword, so the strict (v2) parser needs a narrowly scoped exception, and the directive cannot be offered under the legacy parser.
55+
56+
- **Con:** A new SPI surface in core to keep stable.
57+
58+
## Solution
59+
60+
Add a provider-agnostic SPI to core (`PackageProvider`, `PackageProviderExtension`, `PackageManager`, `PackageSpec`, and the `packages` config scope) and implement each manager as a plugin: `nf-conda`, `nf-pixi`, `nf-uv`, `nf-nix`, `nf-guix`, `nf-pak`, `nf-install2r`.
61+
62+
- The directive is gated behind the `nextflow.preview.package` feature flag and requires the v2 syntax parser; the `package` keyword is allowed only in the process-directive section, and the v1 parser is untouched.
63+
64+
- When Wave is enabled, conda-family and pixi specifications map to Wave's CONDA build type; local-only providers skip Wave and their plugin creates the environment on the local file system, so mixed pipelines work.
65+
66+
- When a process has no `package` directive, the process module directory is scanned for a provider manifest file (e.g. `environment.yml`, `requirements.txt`), analogous to Wave's Dockerfile auto-detection; this can be disabled with `packages.autoDetect = false`.
67+
68+
- Conda's environment cache remains in core because the legacy `conda` directive depends on it; the `nf-conda` plugin reuses those classes. Full extraction into the plugin is a follow-up once the legacy directive is deprecated.

docs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ vscode
9494
git
9595
container
9696
conda
97+
package
9798
spack
9899
wave
99100
```

0 commit comments

Comments
 (0)