Skip to content

Use registry config for plugin repos and default module fallback#7180

Draft
jorgee wants to merge 8 commits into
masterfrom
multi-plugin-registry
Draft

Use registry config for plugin repos and default module fallback#7180
jorgee wants to merge 8 commits into
masterfrom
multi-plugin-registry

Conversation

@jorgee

@jorgee jorgee commented May 27, 2026

Copy link
Copy Markdown
Contributor

Related to #6771

Problem

The registry config scope and the plugin/module resolution paths handled the
default Nextflow registry inconsistently:

  • Modules: a registry block with a custom url fully replaced the default
    registry — the configured list was authoritative and the public registry was
    dropped.
  • Plugins: the registry scope was not wired into plugin resolution at all;
    only a single plugin repository could be set, via indexUrl /
    NXF_PLUGINS_REGISTRY_URL.

Summary

This PR wires the registry config scope into plugin resolution and makes
both modules and plugins treat the configured registries as authoritative:

  • When no registry is configured, the default public registry
    (https://registry.nextflow.io/api) is used.
  • When one or more registry.url entries are set, Nextflow queries exactly
    those
    , in the order listed. The default is not added implicitly — to
    keep using the public registry alongside your own, list it explicitly.

This restores the pre-existing module behaviour and applies the same rule to
plugins.

Plugins (nf-commons)

  • PluginUpdater.addRegistryRepos(RegistryConfig) replaces the default
    registry repository (registry/nextflow.io, derived from indexUrl /
    NXF_PLUGINS_REGISTRY_URL) with an HttpPluginRepository for each configured
    URL. Repo ids are generated as registry-N with a collision guard; test repos
    and the offline local repo are preserved.
  • PluginsFacade.applyRegistryConfig(config) extracts the registry block and
    calls addRegistryRepos on the existing updater (no updater recreation), but
    only when registry.url is explicitly set — so the NXF_PLUGINS_REGISTRY_URL
    default and API authentication are left untouched.
  • HttpPluginRepository.refresh() is null-safe — required because
    UpdateManager.addRepository() calls refresh() immediately, before
    prefetch() has populated the metadata map.

Modules (nextflow)

  • RegistryClientFactory.forConfig() builds the client from cfg.allUrls, which
    falls back to the default only when nothing is configured. No implicit default
    is prepended.

RegistryConfig itself is unchanged — it stays a faithful representation of the
user's config block, returning the default only when no url is configured.

NXF_PLUGINS_REGISTRY_URL vs registry.url (behaviour after this PR)

NXF_PLUGINS_REGISTRY_URL registry.url Plugin registry Module registry
unset unset built-in default built-in default
set unset env var URL built-in default
unset set registry.url registry.url
set set registry.url (env var overridden) registry.url
  • registry.url is applied at config-load time, after the env-var-based default
    repo is created, so registry.url takes precedence over
    NXF_PLUGINS_REGISTRY_URL for plugins.
  • NXF_PLUGINS_REGISTRY_URL affects plugins only; it never influences module
    resolution. Modules fall back to the built-in public registry whenever
    registry.url is unset.
  • The env var continues to work unchanged when registry.url is not set
    (including pointing at the legacy plugins.json index).

This behaviour is also documented in docs/plugins/plugin-registry.md and the
registry config scope reference.

Out of scope

  • API authentication for additional registries — apiKey still authenticates
    only the primary registry.
  • Deprecating NXF_PLUGINS_REGISTRY_URL — tracked separately. It is not a
    1:1 replacement for registry.url: the env var is plugin-only and supports the
    legacy plugins.json index, which registry.url does not.

Test plan

  • ./gradlew :nf-commons:testPluginUpdaterTest: configured registries
    replace the default, replacement even when a configured URL matches the
    default, offline no-op, null-config no-op
  • ./gradlew :nextflow:testRegistryClientFactoryTest: configured
    registries queried in order, only the configured registry queried (no
    default fallback)
  • Full nextflow + nf-commons suites pass locally
  • Manual test: include registry-dev in config and run with a plugin only
    available in registry-dev

Implementation note

We also considered routing the plugin calls through RegistryClient — it takes
a list of URLs and performs the calls until one replies successfully. This works
well for modules, but it doesn't fit pf4j's PluginUpdater and how plugins are
managed, where it expects to manage them separately.

Moreover, there is a chicken-and-egg problem between plugin loading and config
resolution: PluginUpdater is initialized before the config is resolved,
because some SCM and filesystem extensions are themselves provided by plugins and
are required at that stage. At that point only the default plugin registry is
available. Later, at load time — once the config is resolved — the registries
from the registry scope replace the default, and plugins are resolved using
the configured registries.

Bootstrap-window caveat

Because applyRegistryConfig() runs inside Plugins.load(config), the
registry scope is not in effect during the steps that precede it in
CmdRun.run() — resolving the main script (getScriptFile) and parsing the
config (ConfigBuilder.build). Plugins lazily started in that window via
Plugins.startIfMissing(...) — e.g. a filesystem provider for a remote script
or an includeConfig location (s3://, gs://, az://, triggered from
FileHelper.getOrCreateFileSystemFor), or the nf-codecommit SCM provider —
are resolved against the default plugin registry (or
NXF_PLUGINS_REGISTRY_URL) only.

Consequences:

  • Plugins already installed/cached load fine, regardless of registry config.
  • A plugin that must be downloaded in this window resolves from the default
    registry — so the default registry may still be fetched even when it is not
    listed in registry.url
    . A plugin that lives only in a custom
    registry.url registry cannot be resolved at this stage; it must be available
    in the default registry, set via NXF_PLUGINS_REGISTRY_URL, or pre-installed.
  • This window is unchanged by this PR (the plugin path never consulted
    registry.url before config load), and it's also why NXF_PLUGINS_REGISTRY_URL
    remains useful — it is the only setting that influences pre-config plugin
    resolution. Documented in docs/plugins/plugin-registry.md.

jorgee added 2 commits May 26, 2026 17:19
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@netlify

netlify Bot commented May 27, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 5493fab
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6a2a85bb461bc10008dce94e

@jorgee jorgee marked this pull request as draft May 27, 2026 09:03
Align module registry resolution with the plugin behaviour: the default
Nextflow registry is always queried first, then the registries configured
in the `registry` scope. Document the `registry` config scope and update
the module/plugin registry guides accordingly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
jorgee and others added 5 commits June 10, 2026 14:47
Change the `registry` scope so that configured registries fully replace the
default public registry instead of being queried after it. When no registry
is configured the default is used; when one or more URLs are set, Nextflow
queries exactly those, in the order listed. Users that want the public
registry alongside their own must list it explicitly.

This restores the pre-existing module-registry behaviour and applies the same
rule to the plugin registry:

- RegistryClientFactory: build the client from `cfg.allUrls` (which falls back
  to the default when unset); drop the always-prepended default fallback.
- PluginUpdater.addRegistryRepos: replace the default registry repository with
  the configured URLs rather than appending and de-duplicating.
- PluginsFacade.applyRegistryConfig: only override plugin repositories when
  `registry.url` is explicitly set, leaving API authentication and the
  NXF_PLUGINS_REGISTRY_URL env default untouched.

Docs and tests updated accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Add a note clarifying that both settings override the default plugin
registry, that registry.url takes precedence when both are set, and that
NXF_PLUGINS_REGISTRY_URL applies to plugin resolution only (whereas
registry.url drives both modules and plugins).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
The `registry` scope is applied only after config resolution. Plugins
needed earlier — filesystem providers for a remote script or
`includeConfig` location, or SCM providers — are resolved against the
default registry (or NXF_PLUGINS_REGISTRY_URL). Warn that the default
registry may still be fetched even when it is not listed in registry.url.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
HttpPluginRepository.plugins is now eagerly initialised to an empty map
(master commit 1c47215), so refresh()'s 'plugins == null' guard never
fired and every addRepository() call triggered a live registry request
for an empty plugin set. Use Groovy truthiness so an empty or null map
skips the round-trip, matching the prefetch() guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants