feat: add plugin lockfile integrity (plugins.lock) - #7394
Open
pditommaso wants to merge 3 commits into
Open
Conversation
Add an opt-in, offline-verifiable integrity check for the local plugin cache, complementing the extracted-directory guard proposed in #7308. A committed plugins.lock file pins the sha512 hash of each plugin archive, computed over the downloaded bytes and keyed by id@version. Following the go.sum / package-lock.json model, the file is populated automatically on first download (trust-on-first-use) — there is no dedicated command; enable it by creating an empty plugins.lock and running the pipeline once. An existing entry is never rewritten silently: a downloaded archive whose hash differs from a committed entry is a verification failure, gated by NXF_PLUGINS_LOCK_MODE (warn default, strict, off). Verification is network-free: it re-hashes the archive retained alongside the extracted plugin and never re-fetches from the registry as a remedy. A locked plugin whose retained archive is unavailable (e.g. a cache created before this feature) is reported but never aborts the run — integrity of the extracted code that executes remains the job of the #7308 directory guard. The trust anchor lives in the pipeline repo under version control, so it survives registry compromise and silent drift with no runtime dependency on the registry. See adr/20260727-plugin-lockfile-integrity.md. Assisted-by: Claude Opus 4.8 (via Claude Code) Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
✅ Deploy Preview for nextflow-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
christopher-hakkaart
left a comment
christopher-hakkaart
left a comment
Collaborator
There was a problem hiding this comment.
Suggestions for docs
| When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). | ||
|
|
||
| :::note | ||
| The plugin cache is shared across pipelines and is not access-controlled. On multi-tenant or shared systems, use a private cache directory per user (set `NXF_PLUGINS_DIR` to a location only you can write) to avoid loading plugin artifacts populated by another user. |
Collaborator
There was a problem hiding this comment.
Suggested change
| The plugin cache is shared across pipelines and is not access-controlled. On multi-tenant or shared systems, use a private cache directory per user (set `NXF_PLUGINS_DIR` to a location only you can write) to avoid loading plugin artifacts populated by another user. | |
| The plugin cache is shared across pipelines and has no access controls. On multi-tenant or shared systems, set `NXF_PLUGINS_DIR` to a directory only you can write to. This prevents Nextflow from loading plugin artifacts from other users. |
|
|
||
| <AddedInVersion version="26.07" /> | ||
|
|
||
| A `plugins.lock` file pins the exact plugin artifacts a pipeline expects. For each plugin it records the `sha512` checksum of the plugin archive, keyed by `id@version`. The file is meant to be committed to the pipeline repository so that everyone running the pipeline resolves the same plugin artifacts. |
Collaborator
There was a problem hiding this comment.
Suggested change
| A `plugins.lock` file pins the exact plugin artifacts a pipeline expects. For each plugin it records the `sha512` checksum of the plugin archive, keyed by `id@version`. The file is meant to be committed to the pipeline repository so that everyone running the pipeline resolves the same plugin artifacts. | |
| A `plugins.lock` file pins the exact plugin artifacts a pipeline expects. For each plugin, it records the `sha512` checksum of the plugin archive, keyed by `id@version`. Commit the file to the pipeline repository so that everyone who runs the pipeline resolves the same plugin artifacts. |
Comment on lines
+69
to
+75
| The lockfile is populated automatically, like `go.sum` or `package-lock.json` — there is no separate command. To enable it, create an empty file in the pipeline directory and run the pipeline once: | ||
|
|
||
| ```bash | ||
| touch plugins.lock | ||
| ``` | ||
|
|
||
| The first time each plugin is downloaded, its archive checksum is added to `plugins.lock`. Review the resulting file and commit it. On subsequent runs Nextflow verifies each plugin against the committed checksum. When no `plugins.lock` file is present, the feature is dormant and has no effect. |
Collaborator
There was a problem hiding this comment.
Suggested change
| The lockfile is populated automatically, like `go.sum` or `package-lock.json` — there is no separate command. To enable it, create an empty file in the pipeline directory and run the pipeline once: | |
| ```bash | |
| touch plugins.lock | |
| ``` | |
| The first time each plugin is downloaded, its archive checksum is added to `plugins.lock`. Review the resulting file and commit it. On subsequent runs Nextflow verifies each plugin against the committed checksum. When no `plugins.lock` file is present, the feature is dormant and has no effect. | |
| To enable the lockfile: | |
| 1. Create an empty `plugins.lock` file in the pipeline directory: | |
| ```bash | |
| touch plugins.lock | |
| ``` | |
| 2. Run the pipeline once. The first time Nextflow downloads a plugin, it adds that archive's checksum to `plugins.lock`. | |
| 3. Review the resulting file and commit it. | |
| On later runs, Nextflow verifies each plugin against the committed checksum. Without a `plugins.lock` file, the feature does nothing. |
Comment on lines
+77
to
+79
| Verification is fully offline: Nextflow re-computes the checksum of the plugin archive from a copy retained in the local cache and compares it to the lock entry, without contacting the plugin registry. An existing entry is never rewritten automatically — if a plugin archive legitimately changes, delete its entry and run again to re-pin it. | ||
|
|
||
| Use [`NXF_PLUGINS_LOCK_MODE`][using-plugins-env-vars] to control what happens on a checksum mismatch: `warn` (default) logs a warning and continues, `strict` aborts the run, and `off` skips verification. A plugin whose retained archive is missing (for example, a cache populated before this feature existed) cannot be verified offline; it is reported but never aborts the run, and is never re-downloaded just to verify it. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Verification is fully offline: Nextflow re-computes the checksum of the plugin archive from a copy retained in the local cache and compares it to the lock entry, without contacting the plugin registry. An existing entry is never rewritten automatically — if a plugin archive legitimately changes, delete its entry and run again to re-pin it. | |
| Use [`NXF_PLUGINS_LOCK_MODE`][using-plugins-env-vars] to control what happens on a checksum mismatch: `warn` (default) logs a warning and continues, `strict` aborts the run, and `off` skips verification. A plugin whose retained archive is missing (for example, a cache populated before this feature existed) cannot be verified offline; it is reported but never aborts the run, and is never re-downloaded just to verify it. | |
| Verification runs offline. Nextflow re-computes the checksum from the copy of the archive retained in the local cache and compares it to the lock entry, without contacting the plugin registry. | |
| Nextflow never rewrites an existing entry. If a plugin archive changes legitimately, delete its entry and run the pipeline again to re-pin it. | |
| Use [`NXF_PLUGINS_LOCK_MODE`][using-plugins-env-vars] to control what happens when a checksum does not match: | |
| | Mode | Behavior | | |
| | ---------------- | ----------------------------- | | |
| | `warn` (default) | Logs a warning and continues. | | |
| | `strict` | Aborts the run. | | |
| | `off` | Skips verification. | | |
| Nextflow cannot verify a plugin whose retained archive is missing. For example, in a cache populated before this feature existed. It reports the plugin, does not abort the run, and does not re-download the archive to verify it. |
|
|
||
| Use [`NXF_PLUGINS_LOCK_MODE`][using-plugins-env-vars] to control what happens on a checksum mismatch: `warn` (default) logs a warning and continues, `strict` aborts the run, and `off` skips verification. A plugin whose retained archive is missing (for example, a cache populated before this feature existed) cannot be verified offline; it is reported but never aborts the run, and is never re-downloaded just to verify it. | ||
|
|
||
| The lockfile complements, but does not replace, the private-cache guidance above: the cache isolation prevents untrusted artifacts from being loaded, while the lockfile ensures the artifacts that are loaded match what the pipeline pinned. |
Collaborator
There was a problem hiding this comment.
Suggested change
| The lockfile complements, but does not replace, the private-cache guidance above: the cache isolation prevents untrusted artifacts from being loaded, while the lockfile ensures the artifacts that are loaded match what the pipeline pinned. | |
| The lockfile does not replace a private cache directory. A private cache prevents Nextflow from loading untrusted artifacts. The lockfile confirms only that the artifacts it loads match what the pipeline pinned. |
Rework the plugin lockfile to hash the extracted $id-$version/ directory (the code Nextflow actually loads and executes) rather than the downloaded archive. Hashing the unpacked tree lets a single, content-addressed check cover both integrity surfaces: - supply chain / registry compromise / silent drift (a tampered archive extracts to a different tree), and - cache poisoning of the executed code (a lower-trust user editing already -extracted files on a shared cache) — detected regardless of directory ownership or permissions, with no ownership heuristic and therefore no false-positive warnings on legitimate shared caches. This supersedes the archive-hash approach and the directory-ownership guard proposed in #7308: it is stronger (catches modifications an ownership check misses) and quieter (silent on any healthy cache; output only on a genuine mismatch). Verification runs once per plugin in load0(), before loading, on both cold and warm caches. The retained-archive machinery and the "archive absent" case are removed (the extracted tree is always present when a plugin loads). Full re-hash on every load is acceptable for now; the ADR documents a local fingerprint-cache optimisation (gated on a private, non-writable cache) as a measure-first follow-up for large plugins. Assisted-by: Claude Opus 4.8 (via Claude Code) Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Member
Author
|
@christopher-hakkaart for some reason I'm not able to apply your changes. Can you submit them again ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in, offline-verifiable integrity check for plugins: a committed
plugins.lockfile that pins a canonical SHA-512 of each plugin's extracted$id-$version/directory — the code Nextflow actually loads and executes — keyed byid@version, following thego.sum/package-lock.jsonmodel.Because it hashes the code that runs (not the archive it came from), one mechanism covers both integrity surfaces:
Full design rationale in
adr/20260727-plugin-lockfile-integrity.md.Relationship to #7308 (supersedes)
This started as the "committed lockfile" long-term direction raised in the review of #7308. In its final form it supersedes #7308's
PluginSecuritydirectory-ownership guard rather than complementing it:The content hash is both stronger (catches modifications an ownership check misses) and quieter (no ownership heuristic, so no false-positive warnings on legitimate shared/admin caches). I'd suggest closing #7308 in favour of this. The zero-config baseline for the tree surface remains the operational guidance to keep the plugin cache per-user private; this is the committed, enforceable layer on top.
How it works
plugins.lock(touch plugins.lock) and running once, then review the diff and commit.NXF_PLUGINS_LOCK_MODE(warndefault /strict/off). Re-pin explicitly: delete the entry and re-run.PluginUpdater.load0()before loading (cold and warm caches, same path); never re-fetches from the registry. No retained archive, no "archive absent" case.Performance
Full re-hash on every load is fine for small plugins; for large cloud plugins it's material. The ADR documents a follow-up optimisation (a local, non-committed
(path,size,mtime)fingerprint cache to skip the re-hash when nothing changed, trusted only on a private non-writable cache — ownership selects the strategy, never emits a warning), deliberately left as measure-first rather than built up front.Testing
PluginLockFileTest— round-trip, blank/touched file, malformed file, lookup, add/update.PluginLockVerifierTest— mode resolution;sha512Treedeterminism & content-sensitivity; dormant vs enabled-by-presence; TOFU auto-pin writes the tree hash; match passes; mismatch aborts (strict) / warns once (warn) / ignored (off); cache-poisoning caught regardless of ownership.PluginUpdaterTest,PluginsFacadeTest,CmdPluginTest— unchanged, still green.Docs:
NXF_PLUGINS_LOCK_MODEinenv-vars.mdx; a Lockfile section + shared-cache note inusing-plugins.mdx.Supersedes #7308.
🤖 Generated with Claude Code