refactor: pin extracted-tree hash instead of archive hash

Rework the plugin lockfile to hash the extracted $id-$version/ directory
(the code Nextflow actually loads and executes) rather than the downloaded
archive. Hashing the unpacked tree lets a single, content-addressed check
cover both integrity surfaces:

- supply chain / registry compromise / silent drift (a tampered archive
  extracts to a different tree), and
- cache poisoning of the executed code (a lower-trust user editing already
  -extracted files on a shared cache) — detected regardless of directory
  ownership or permissions, with no ownership heuristic and therefore no
  false-positive warnings on legitimate shared caches.

This supersedes the archive-hash approach and the directory-ownership guard
proposed in #7308: it is stronger (catches modifications an ownership check
misses) and quieter (silent on any healthy cache; output only on a genuine
mismatch).

Verification runs once per plugin in load0(), before loading, on both cold
and warm caches. The retained-archive machinery and the "archive absent"
case are removed (the extracted tree is always present when a plugin loads).
Full re-hash on every load is acceptable for now; the ADR documents a local
fingerprint-cache optimisation (gated on a private, non-writable cache) as a
measure-first follow-up for large plugins.

Assisted-by: Claude Opus 4.8 (via Claude Code)
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Author: pditommaso

adr/20260727-plugin-lockfile-integrity.md

@@ -64,21 +64,21 @@ The plugin cache is shared across pipelines and is not access-controlled. On mul
 
 <AddedInVersion version="26.07" />
 
-A `plugins.lock` file pins the exact plugin artifacts a pipeline expects. For each plugin it records the `sha512` checksum of the plugin archive, keyed by `id@version`. The file is meant to be committed to the pipeline repository so that everyone running the pipeline resolves the same plugin artifacts.
+A `plugins.lock` file pins the exact plugin code a pipeline expects. For each plugin it records a `sha512` hash of the **extracted plugin directory** — the code Nextflow actually loads and runs — keyed by `id@version`. The file is meant to be committed to the pipeline repository so that everyone running the pipeline executes the same plugin code.
 
 The lockfile is populated automatically, like `go.sum` or `package-lock.json` — there is no separate command. To enable it, create an empty file in the pipeline directory and run the pipeline once:
 
 ```bash
 touch plugins.lock
 ```
 
-The first time each plugin is downloaded, its archive checksum is added to `plugins.lock`. Review the resulting file and commit it. On subsequent runs Nextflow verifies each plugin against the committed checksum. When no `plugins.lock` file is present, the feature is dormant and has no effect.
+The first time each plugin is loaded, its hash is added to `plugins.lock`. Review the resulting file and commit it. On subsequent runs Nextflow re-hashes each plugin's extracted directory and verifies it against the committed hash. When no `plugins.lock` file is present, the feature is dormant and has no effect.
 
-Verification is fully offline: Nextflow re-computes the checksum of the plugin archive from a copy retained in the local cache and compares it to the lock entry, without contacting the plugin registry. An existing entry is never rewritten automatically — if a plugin archive legitimately changes, delete its entry and run again to re-pin it.
+Verification is fully offline — it re-hashes the files already in the local cache and never contacts the plugin registry. Because it hashes the extracted code rather than the download, it detects both a tampered or compromised download and a plugin directory that was modified after extraction (for example by another user on a shared cache), independently of file ownership or permissions. An existing entry is never rewritten automatically — if a plugin legitimately changes, delete its entry and run again to re-pin it.
 
-Use [`NXF_PLUGINS_LOCK_MODE`][using-plugins-env-vars] to control what happens on a checksum mismatch: `warn` (default) logs a warning and continues, `strict` aborts the run, and `off` skips verification. A plugin whose retained archive is missing (for example, a cache populated before this feature existed) cannot be verified offline; it is reported but never aborts the run, and is never re-downloaded just to verify it.
+Use [`NXF_PLUGINS_LOCK_MODE`][using-plugins-env-vars] to control what happens on a mismatch: `warn` (default) logs a warning and continues, `strict` aborts the run, and `off` skips verification.
 
-The lockfile complements, but does not replace, the private-cache guidance above: the cache isolation prevents untrusted artifacts from being loaded, while the lockfile ensures the artifacts that are loaded match what the pipeline pinned.
+The lockfile complements the private-cache guidance above: keeping the cache private prevents untrusted code from being written in the first place, while the lockfile detects any change to the plugin code that is actually loaded.
 
 ## Offline usage
 

docs/plugins/using-plugins.mdx

@@ -214,7 +214,7 @@ The path where the plugin archives are loaded and stored (default: `$NXF_HOME/pl
 
 <AddedInVersion version="26.07" />
 
-Controls how Nextflow reacts when a downloaded plugin artifact does not match the entry recorded in the `plugins.lock` file: `warn` logs a warning once per plugin and continues, `strict` aborts the run, and `off` skips verification silently (default: `warn`). Verification is dormant when no `plugins.lock` file is present.
+Controls how Nextflow reacts when a plugin's extracted directory does not match the hash recorded in the `plugins.lock` file: `warn` logs a warning once per plugin and continues, `strict` aborts the run, and `off` skips verification silently (default: `warn`). Verification is dormant when no `plugins.lock` file is present.
 
 ##### `NXF_PLUGINS_REGISTRY_URL`
 

docs/reference/env-vars.mdx

@@ -16,8 +16,11 @@
 
 package nextflow.plugin
 
+import java.nio.file.FileVisitResult
 import java.nio.file.Files
 import java.nio.file.Path
+import java.nio.file.SimpleFileVisitor
+import java.nio.file.attribute.BasicFileAttributes
 import java.security.MessageDigest
 
 import groovy.transform.CompileStatic
@@ -26,30 +29,26 @@ import nextflow.SysEnv
 import nextflow.exception.AbortOperationException
 
 /**
- * Verifies plugin archives against the {@code plugins.lock} file.
+ * Verifies an extracted plugin directory against the {@code plugins.lock} file.
  *
  * The feature is opt-in <em>by the presence of the lock file</em>: it is dormant (a no-op) when no
- * {@code plugins.lock} exists. When the file is present, the sha512 of the retained plugin archive
- * is re-computed locally (no network) and compared to the committed lock entry.
+ * {@code plugins.lock} exists. When the file is present, a canonical hash of the extracted
+ * {@code $id-$version/} directory - the code that Nextflow actually loads and executes - is
+ * re-computed locally (no network) and compared to the committed lock entry. Because it hashes the
+ * unpacked tree rather than the archive, it detects both a tampered/compromised download and a
+ * poisoned cache directory (a lower-trust user editing already-extracted files on a shared cache).
  *
- * Following the {@code go.sum} / {@code package-lock.json} model, the lock is populated
- * automatically: the first time a coordinate is downloaded and it is missing from the lock, its
- * archive checksum is appended (trust-on-first-use). An existing entry is never rewritten silently
- * — a mismatch against a committed entry is a verification failure.
- *
- * The behaviour on a checksum mismatch is gated by the {@code NXF_PLUGINS_LOCK_MODE} environment
- * variable:
+ * Following the {@code go.sum} / {@code package-lock.json} model the lock is populated
+ * automatically: the first time a coordinate is seen and it is missing from the lock, its tree
+ * hash is appended (trust-on-first-use). An existing entry is never rewritten silently — a
+ * mismatch against a committed entry is a verification failure, gated by
+ * {@code NXF_PLUGINS_LOCK_MODE}:
  * <ul>
  *   <li>{@code strict} - abort with an {@link AbortOperationException}</li>
  *   <li>{@code warn} (default) - log a warning once per coordinate and proceed</li>
  *   <li>{@code off} - skip verification silently</li>
  * </ul>
  *
- * A locked plugin whose archive is not available locally (e.g. a cache extracted before this
- * feature existed) is never aborted: it cannot be verified offline and its archive is never
- * re-downloaded just to verify it. Integrity of the extracted code that actually runs is the
- * responsibility of the plugin directory guard, not of this archive check.
- *
  * @author Paolo Di Tommaso <paolo.ditommaso@gmail.com>
  */
 @Slf4j
@@ -103,46 +102,39 @@ class PluginLockVerifier {
     }
 
     /**
-     * Verify - and, on first sight, pin - a plugin archive against the lock.
+     * Verify - and, on first sight, pin - an extracted plugin directory against the lock.
      *
      * @param fqid The plugin fully-qualified id ie. {@code id@version}
-     * @param zip The retained plugin archive; may be {@code null} or missing
+     * @param pluginDir The extracted plugin directory ({@code $id-$version/})
      */
-    void verify(String fqid, Path zip) {
+    void verify(String fqid, Path pluginDir) {
         if( !enabled )
             return
-        final entry = lock.getEntry(fqid)
-        final present = zip != null && Files.exists(zip)
-
-        // coordinate not yet locked: pin it on first download (trust-on-first-use), never fail
-        if( entry == null ) {
-            if( present )
-                pin(fqid, sha512(zip))
-            else
-                log.debug "Plugin '$fqid' is not in the plugins lock file and its archive is not available to pin"
+        if( pluginDir == null || !Files.isDirectory(pluginDir) ) {
+            log.debug "Cannot verify plugin '$fqid' against the plugins lock file - directory not available: $pluginDir"
             return
         }
 
-        // locked, but the archive is not available (e.g. a cache created before this feature):
-        // it cannot be verified offline - never abort and never re-download to verify
-        if( !present ) {
-            if( getMode() != Mode.OFF && notified.add(fqid) )
-                log.warn "Cannot verify plugin '$fqid' against the plugins lock file - its archive is not available in the cache"
+        final actual = sha512Tree(pluginDir)
+        final entry = lock.getEntry(fqid)
+
+        // coordinate not yet locked: pin it on first sight (trust-on-first-use), never fail
+        if( entry == null ) {
+            pin(fqid, actual)
             return
         }
-
-        // verify the retained archive against the committed checksum
-        final actual = sha512(zip)
+        // matches the committed hash
         if( actual == entry.sha512 )
             return
 
-        final reason = "Plugin '$fqid' checksum does not match the plugins lock file\n- expected: ${entry.sha512}\n- actual  : ${actual}"
+        // mismatch: the extracted plugin differs from what the lock pinned
+        final reason = "Plugin '$fqid' does not match the plugins lock file\n- expected: ${entry.sha512}\n- actual  : ${actual}"
         switch( getMode() ) {
             case Mode.OFF:
                 log.debug "Plugins lock verification failed (ignored, mode=off) - $reason"
                 break
             case Mode.STRICT:
-                throw new AbortOperationException("$reason\n- delete the entry from the plugins lock file and re-run to re-pin, or restore the expected plugin archive")
+                throw new AbortOperationException("$reason\n- delete the entry from the plugins lock file and re-run to re-pin, or restore the expected plugin")
             case Mode.WARN:
                 // warn only once per coordinate to avoid log spam
                 if( notified.add(fqid) )
@@ -154,7 +146,7 @@ class PluginLockVerifier {
     /**
      * Append a new entry to the lock and persist it (trust-on-first-use). Existing entries are
      * never overwritten by this path — {@link #verify} routes an already-locked coordinate through
-     * the checksum comparison instead.
+     * the hash comparison instead.
      */
     private synchronized void pin(String fqid, String sha512) {
         lock.addEntry(fqid, new PluginLockFile.Entry(sha512))
@@ -164,18 +156,35 @@ class PluginLockVerifier {
     }
 
     /**
-     * Compute the sha512 hex digest of the given file.
+     * Compute a canonical sha512 digest over the content of a directory tree. The digest covers,
+     * for every regular file (visited in sorted relative-path order for determinism), its relative
+     * path and its bytes - not timestamps or permissions - so it is stable across extractions and
+     * platforms while still detecting any change to the files that will be executed.
      *
-     * @param file The file to hash
+     * @param dir The directory to hash
      * @return The lowercase hex-encoded sha512 digest (128 chars)
      */
-    static String sha512(Path file) {
+    static String sha512Tree(Path dir) {
         final md = MessageDigest.getInstance('SHA-512')
-        try (InputStream is = Files.newInputStream(file)) {
-            final buffer = new byte[8192]
-            int read
-            while( (read = is.read(buffer)) != -1 )
-                md.update(buffer, 0, read)
+        // TreeMap keyed by relative path -> deterministic, sorted iteration order
+        final files = new TreeMap<String, Path>()
+        Files.walkFileTree(dir, new SimpleFileVisitor<Path>() {
+            @Override
+            FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
+                files.put(dir.relativize(file).toString().replace('\\', '/'), file)
+                return FileVisitResult.CONTINUE
+            }
+        })
+        final buffer = new byte[8192]
+        for( Map.Entry<String, Path> it : files.entrySet() ) {
+            md.update(it.key.getBytes('UTF-8'))
+            md.update((byte) 0)
+            try (InputStream is = Files.newInputStream(it.value)) {
+                int read
+                while( (read = is.read(buffer)) != -1 )
+                    md.update(buffer, 0, read)
+            }
+            md.update((byte) 0)
         }
         return HexFormat.of().formatHex(md.digest())
     }

modules/nf-commons/src/main/nextflow/plugin/PluginLockVerifier.groovy

@@ -253,7 +253,7 @@ class PluginUpdater extends UpdateManager {
     }
 
     /**
-     * The {@code plugins.lock} file location used to verify downloaded plugin artifacts.
+     * The {@code plugins.lock} file location used to verify plugins against their pinned hash.
      * Defaults to a {@code plugins.lock} file in the current working directory.
      */
     protected Path lockFilePath() {
@@ -269,13 +269,6 @@ class PluginUpdater extends UpdateManager {
         return lockVerifier
     }
 
-    /**
-     * @return the path to the retained plugin zip artifact used for lock verification
-     */
-    private Path retainedZip(String id, String version) {
-        return pluginsStore.resolve("${id}-${version}.zip")
-    }
-
     private Path download0(String id, String version) {
         // 0. check if version is specified
         if( !version )
@@ -291,21 +284,9 @@ class PluginUpdater extends UpdateManager {
         // 2. download to temporary location
         Path downloaded = safeDownloadPlugin(id, version);
 
-        // 3. unzip the content
+        // 3. unzip the content and delete downloaded file
         Path dir = FileUtils.expandIfZip(downloaded)
-
-        // 3.1 when the plugins lock is enabled retain the downloaded zip next to the extracted
-        // dir and verify (or pin) its checksum against the lock (cold cache). Otherwise delete it.
-        if( getLockVerifier().isEnabled() ) {
-            final retained = retainedZip(id, version)
-            FileHelper.deletePath(retained)
-            FileHelper.copyPath(downloaded, retained)
-            FileHelper.deletePath(downloaded)
-            getLockVerifier().verify("${id}@${version}", retained)
-        }
-        else {
-            FileHelper.deletePath(downloaded)
-        }
+        FileHelper.deletePath(downloaded)
 
         // 4. move the final destination the plugin directory
         assert pluginPath.getFileName() == dir.getFileName()
@@ -434,17 +415,16 @@ class PluginUpdater extends UpdateManager {
         if( !FilesEx.exists(pluginPath) ) {
             pluginPath = safeDownload(id, version)
         }
-        else {
-            // warm cache: re-hash the retained artifact and verify it against the plugins lock
-            // (missing retained zip is treated as a lock-miss by the verifier)
-            getLockVerifier().verify("${id}@${version}", retainedZip(id, version))
-        }
 
         // verify the plugin install path contains the expected manifest path
         if( !FilesEx.exists(pluginPath.resolve('classes/META-INF/MANIFEST.MF')) ) {
             log.warn("Plugin '${pluginPath.getFileName()}' installation looks corrupted - Delete the following directory and run nextflow again: $pluginPath")
         }
 
+        // verify (or pin) the extracted plugin directory against the plugins lock, before loading
+        // and executing its code. This covers both a fresh download and a reused (warm) cache.
+        getLockVerifier().verify("${id}@${version}", pluginPath)
+
         // load the plugin from the file system
         PluginWrapper wrapper = pluginManager.loadPluginFromPath(pluginPath)