Use registry config for plugin repos and default module fallback#7180
Draft
jorgee wants to merge 8 commits into
Draft
Use registry config for plugin repos and default module fallback#7180jorgee wants to merge 8 commits into
jorgee wants to merge 8 commits into
Conversation
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
Align module registry resolution with the plugin behaviour: the default Nextflow registry is always queried first, then the registries configured in the `registry` scope. Document the `registry` config scope and update the module/plugin registry guides accordingly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Change the `registry` scope so that configured registries fully replace the default public registry instead of being queried after it. When no registry is configured the default is used; when one or more URLs are set, Nextflow queries exactly those, in the order listed. Users that want the public registry alongside their own must list it explicitly. This restores the pre-existing module-registry behaviour and applies the same rule to the plugin registry: - RegistryClientFactory: build the client from `cfg.allUrls` (which falls back to the default when unset); drop the always-prepended default fallback. - PluginUpdater.addRegistryRepos: replace the default registry repository with the configured URLs rather than appending and de-duplicating. - PluginsFacade.applyRegistryConfig: only override plugin repositories when `registry.url` is explicitly set, leaving API authentication and the NXF_PLUGINS_REGISTRY_URL env default untouched. Docs and tests updated accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Add a note clarifying that both settings override the default plugin registry, that registry.url takes precedence when both are set, and that NXF_PLUGINS_REGISTRY_URL applies to plugin resolution only (whereas registry.url drives both modules and plugins). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
The `registry` scope is applied only after config resolution. Plugins needed earlier — filesystem providers for a remote script or `includeConfig` location, or SCM providers — are resolved against the default registry (or NXF_PLUGINS_REGISTRY_URL). Warn that the default registry may still be fetched even when it is not listed in registry.url. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
HttpPluginRepository.plugins is now eagerly initialised to an empty map (master commit 1c47215), so refresh()'s 'plugins == null' guard never fired and every addRepository() call triggered a live registry request for an empty plugin set. Use Groovy truthiness so an empty or null map skips the round-trip, matching the prefetch() guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #6771
Problem
The
registryconfig scope and the plugin/module resolution paths handled thedefault Nextflow registry inconsistently:
registryblock with a customurlfully replaced the defaultregistry — the configured list was authoritative and the public registry was
dropped.
registryscope was not wired into plugin resolution at all;only a single plugin repository could be set, via
indexUrl/NXF_PLUGINS_REGISTRY_URL.Summary
This PR wires the
registryconfig scope into plugin resolution and makesboth modules and plugins treat the configured registries as authoritative:
(
https://registry.nextflow.io/api) is used.registry.urlentries are set, Nextflow queries exactlythose, in the order listed. The default is not added implicitly — to
keep using the public registry alongside your own, list it explicitly.
This restores the pre-existing module behaviour and applies the same rule to
plugins.
Plugins (
nf-commons)PluginUpdater.addRegistryRepos(RegistryConfig)replaces the defaultregistry repository (
registry/nextflow.io, derived fromindexUrl/NXF_PLUGINS_REGISTRY_URL) with anHttpPluginRepositoryfor each configuredURL. Repo ids are generated as
registry-Nwith a collision guard; test reposand the offline local repo are preserved.
PluginsFacade.applyRegistryConfig(config)extracts theregistryblock andcalls
addRegistryReposon the existing updater (no updater recreation), butonly when
registry.urlis explicitly set — so theNXF_PLUGINS_REGISTRY_URLdefault and API authentication are left untouched.
HttpPluginRepository.refresh()is null-safe — required becauseUpdateManager.addRepository()callsrefresh()immediately, beforeprefetch()has populated the metadata map.Modules (
nextflow)RegistryClientFactory.forConfig()builds the client fromcfg.allUrls, whichfalls back to the default only when nothing is configured. No implicit default
is prepended.
RegistryConfigitself is unchanged — it stays a faithful representation of theuser's config block, returning the default only when no
urlis configured.NXF_PLUGINS_REGISTRY_URLvsregistry.url(behaviour after this PR)NXF_PLUGINS_REGISTRY_URLregistry.urlregistry.urlregistry.urlregistry.url(env var overridden)registry.urlregistry.urlis applied at config-load time, after the env-var-based defaultrepo is created, so
registry.urltakes precedence overNXF_PLUGINS_REGISTRY_URLfor plugins.NXF_PLUGINS_REGISTRY_URLaffects plugins only; it never influences moduleresolution. Modules fall back to the built-in public registry whenever
registry.urlis unset.registry.urlis not set(including pointing at the legacy
plugins.jsonindex).This behaviour is also documented in
docs/plugins/plugin-registry.mdand theregistryconfig scope reference.Out of scope
apiKeystill authenticatesonly the primary registry.
NXF_PLUGINS_REGISTRY_URL— tracked separately. It is not a1:1 replacement for
registry.url: the env var is plugin-only and supports thelegacy
plugins.jsonindex, whichregistry.urldoes not.Test plan
./gradlew :nf-commons:test—PluginUpdaterTest: configured registriesreplace the default, replacement even when a configured URL matches the
default, offline no-op, null-config no-op
./gradlew :nextflow:test—RegistryClientFactoryTest: configuredregistries queried in order, only the configured registry queried (no
default fallback)
nextflow+nf-commonssuites pass locallyavailable in registry-dev
Implementation note
We also considered routing the plugin calls through
RegistryClient— it takesa list of URLs and performs the calls until one replies successfully. This works
well for modules, but it doesn't fit pf4j's
PluginUpdaterand how plugins aremanaged, where it expects to manage them separately.
Moreover, there is a chicken-and-egg problem between plugin loading and config
resolution:
PluginUpdateris initialized before the config is resolved,because some SCM and filesystem extensions are themselves provided by plugins and
are required at that stage. At that point only the default plugin registry is
available. Later, at load time — once the config is resolved — the registries
from the
registryscope replace the default, and plugins are resolved usingthe configured registries.
Bootstrap-window caveat
Because
applyRegistryConfig()runs insidePlugins.load(config), theregistryscope is not in effect during the steps that precede it inCmdRun.run()— resolving the main script (getScriptFile) and parsing theconfig (
ConfigBuilder.build). Plugins lazily started in that window viaPlugins.startIfMissing(...)— e.g. a filesystem provider for a remote scriptor an
includeConfiglocation (s3://,gs://,az://, triggered fromFileHelper.getOrCreateFileSystemFor), or thenf-codecommitSCM provider —are resolved against the default plugin registry (or
NXF_PLUGINS_REGISTRY_URL) only.Consequences:
registry — so the default registry may still be fetched even when it is not
listed in
registry.url. A plugin that lives only in a customregistry.urlregistry cannot be resolved at this stage; it must be availablein the default registry, set via
NXF_PLUGINS_REGISTRY_URL, or pre-installed.registry.urlbefore config load), and it's also whyNXF_PLUGINS_REGISTRY_URLremains useful — it is the only setting that influences pre-config plugin
resolution. Documented in
docs/plugins/plugin-registry.md.