Skip nf-schema path validation for annotation_cache and igenomes_base#2184
Open
pinin4fjords wants to merge 3 commits into
Open
Skip nf-schema path validation for annotation_cache and igenomes_base#2184pinin4fjords wants to merge 3 commits into
pinin4fjords wants to merge 3 commits into
Conversation
…_base nf-schema 2.5+ resolves params declared with `format: directory-path` to verify the path exists on the storage backend. For `snpeff_cache`, `vep_cache` and `igenomes_base` the defaults point at S3 buckets that are not always readable from the launch environment (e.g. SLURM clusters without credentials for s3://annotation-cache/, SSO-scoped AWS sessions). Validation fails with 403 / "Key cannot be empty" before the pipeline can decide whether those caches are even needed. PR #2083 dropped `exists: true` from the two cache params but the `format: directory-path` declaration still triggers the existence check. Follow the nf-schema maintainer's recommendation (see nextflow-io/nf-schema#204): remove the format declaration and add the params to `defaultIgnoreParams` so the validator skips them. Refs #2079 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pinin4fjords
requested review from
FriederikeHanssen and
maxulysse
as code owners
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 3.5.1. For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation. |
|
maxulysse
approved these changes
May 13, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
snpeff_cache,vep_cacheandigenomes_baseare declared in the schema withformat: directory-pathand default to S3 buckets. nf-schema 2.5+ resolves these defaults against the storage backend and fails the launch if the path is unreachable - no credentials, no network, or an SSO/IAM session that can't see the bucket. Any user without read access tos3://annotation-cache/(e.g. on-prem clusters, restricted cloud accounts) currently can't launch the pipeline at all, even when they have no intention of running snpEff or VEP.Details
The full debate of where this fix belongs is in nextflow-io/nf-schema#204. The plugin maintainer's settled position is that:
format: "directory-path"so the existence probe never runs, andvalidation.defaultIgnoreParamsso nf-schema skips it entirely.That guidance has been applied in nf-core/rnaseq for
igenomes_base(PRs #1696 and #1739). This PR applies the same treatment to sarek's three S3-defaulted directory params.PR #2083 already removed
exists: truefromsnpeff_cache/vep_cache, butformat: directory-pathalone is enough to trigger the existence check, so launches still fail for users without access to the default bucket.Fixes #2079
Changes
nextflow.config: addigenomes_base,snpeff_cache,vep_cachetovalidation.defaultIgnoreParams.nextflow_schema.json: drop"format": "directory-path"from the same three params. Derived/resolved paths are still validated individually downstream.Test plan
s3://annotation-cache/- validation should now pass.--snpeff_cache/--vep_cacheoverrides - annotation should still work.--tools snpeff,vepand confirm download-cache flows still error appropriately if the path is bad at the point of use (not at schema validation).