|
| 1 | +# Package provider plugins |
| 2 | + |
| 3 | +- Authors: Evan Rees |
| 4 | +- Status: proposed |
| 5 | +- Deciders: Nextflow maintainers |
| 6 | +- Date: 2026-07-07 |
| 7 | +- Tags: plugins, packages |
| 8 | + |
| 9 | +## Summary |
| 10 | + |
| 11 | +Provide a single generic `package` process directive whose package managers are implemented as pluggable providers, each shipped as its own `nf-<provider>` plugin. |
| 12 | + |
| 13 | +## Problem Statement |
| 14 | + |
| 15 | +Nextflow supports software dependencies through per-manager process directives (`conda`, `spack`), and each new package manager requires its own directive and core runtime changes. Requests for additional managers keep arriving (nix and guix in #1526, uv in #7023, pixi and R managers in the #6342 review), and the per-manager approach does not scale: every addition grows the core, the process DSL, and the config surface. |
| 16 | + |
| 17 | +## Goals or Decision Drivers |
| 18 | + |
| 19 | +- One directive for all package managers, so processes declare dependencies the same way regardless of the tool. |
| 20 | +- New managers must be addable without core runtime changes. |
| 21 | +- Alignment with Wave: providers that Wave can build (conda-based) hand off to Wave; others build locally. |
| 22 | +- The feature must be opt-in (preview flag) and must not affect existing pipelines. |
| 23 | + |
| 24 | +## Non-goals |
| 25 | + |
| 26 | +- Implementing Wave container builds for non-conda providers. |
| 27 | +- Removing or deprecating the legacy `conda` and `spack` directives. |
| 28 | +- Extracting the existing conda support out of core (the legacy `conda` directive still depends on it). |
| 29 | + |
| 30 | +## Considered Options |
| 31 | + |
| 32 | +### Per-manager directives |
| 33 | + |
| 34 | +Continue adding one process directive per package manager (`uv`, `pixi`, `nix`, ...). |
| 35 | + |
| 36 | +- **Pro:** Explicit and familiar; mirrors the existing `conda`/`spack` directives. |
| 37 | + |
| 38 | +- **Con:** Every manager requires parser, DSL, config, and runtime changes in core; N managers means N directives to document and maintain. |
| 39 | + |
| 40 | +### Wave as the single integration point |
| 41 | + |
| 42 | +Route all package specifications through Wave and let it build the environments. |
| 43 | + |
| 44 | +- **Pro:** One container-based path for every manager. |
| 45 | + |
| 46 | +- **Con:** Wave only builds conda-based package environments; forcing other managers through it is not possible today, and it makes containers mandatory where local environments suffice. |
| 47 | + |
| 48 | +### Generic directive with pluggable providers (chosen) |
| 49 | + |
| 50 | +A `package` directive backed by a `PackageProvider` SPI in core, with each manager implemented in its own `nf-<provider>` plugin. |
| 51 | + |
| 52 | +- **Pro:** Managers ship, version, and test independently; third parties can add providers without touching core. |
| 53 | + |
| 54 | +- **Con:** `package` is a Groovy/Java keyword, so the strict (v2) parser needs a narrowly scoped exception, and the directive cannot be offered under the legacy parser. |
| 55 | + |
| 56 | +- **Con:** A new SPI surface in core to keep stable. |
| 57 | + |
| 58 | +## Solution |
| 59 | + |
| 60 | +Add a provider-agnostic SPI to core (`PackageProvider`, `PackageProviderExtension`, `PackageManager`, `PackageSpec`, and the `packages` config scope) and implement each manager as a plugin: `nf-conda`, `nf-pixi`, `nf-uv`, `nf-nix`, `nf-guix`, `nf-pak`, `nf-install2r`. |
| 61 | + |
| 62 | +- The directive is gated behind the `nextflow.preview.package` feature flag and requires the v2 syntax parser; the `package` keyword is allowed only in the process-directive section, and the v1 parser is untouched. |
| 63 | + |
| 64 | +- When Wave is enabled, conda-family and pixi specifications map to Wave's CONDA build type; local-only providers skip Wave and their plugin creates the environment on the local file system, so mixed pipelines work. |
| 65 | + |
| 66 | +- When a process has no `package` directive, the process module directory is scanned for a provider manifest file (e.g. `environment.yml`, `requirements.txt`), analogous to Wave's Dockerfile auto-detection; this can be disabled with `packages.autoDetect = false`. |
| 67 | + |
| 68 | +- Conda's environment cache remains in core because the legacy `conda` directive depends on it; the `nf-conda` plugin reuses those classes. Full extraction into the plugin is a follow-up once the legacy directive is deprecated. |
0 commit comments