Workflow fails intermittently due to UnknownHostException from s3://ngi-igenomes/igenomes/

[oneillkza]

Description of the bug

* --igenomes_base (s3://ngi-igenomes/igenomes/): could not validate file format of 's3://ngi-igenomes/igenomes/': Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.

This happens stochastically maybe one run run in 20, and re-running seems to resolve the error. But it is concerning, and I'm wondering if there's a way to run the workflow more robustly?

Command used and terminal output

Relevant files

No response

System information

No response

[scwatts]

Hi @oneillkza, does this error have any further information in the .nextflow.log file?

[oneillkza]

@scwatts there are these lines too.

May-19 12:16:08.473 [main] DEBUG n.v.parameters.ParameterValidator - Starting parameters validation
May-19 12:16:09.515 [main] DEBUG n.v.v.evaluators.SchemaEvaluator - Started validating /projects/koneill_prj/prodev/POG/oncoanalyser/test_run_from_cram_mini/samplesheet.csv
May-19 12:16:09.946 [main] DEBUG n.v.v.evaluators.SchemaEvaluator - Validation of file 'samplesheet.csv' passed!
May-19 12:16:10.397 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 250ms (attempt=1)
May-19 12:16:10.871 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 750ms (attempt=2)
May-19 12:16:11.858 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 2250ms (attempt=3)
May-19 12:16:14.579 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 250ms (attempt=1)
May-19 12:16:15.063 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 750ms (attempt=2)
May-19 12:16:16.040 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 2250ms (attempt=3)
May-19 12:16:18.692 [main] DEBUG n.plugin.HttpPluginRepository - Registry request: https://registry.nextflow.io/api/v1/plugins/dependencies?plugins=nf-amazon%403.4.4&nextflowVersion=25.10.4
- code: 200
- body: {"plugins":[{"id":"nf-amazon","releases":[{"version":"3.4.4","url":"https://registry.nextflow.io/api/v1/plugins/nf-amazon/3.4.4/download/nf-amazon-3.4.4.zip","date":"2026-02-10T15:21:20.535204Z","sha512sum":"76c37dc3b526e6d06b7fb5c171cd0679da4a5f7c6545eb16fba4ca5e7d812c60f555fb7a71718955a93aa02a22fec73b0e220ca0998725a0dbda0333b0f106ac","requires":">=25.09.2-edge","dependsOn":[],"status":"PUBLISHED"}],"projectUrl":"https://github.com/nextflow-io/nf-amazon","provider":"nextflow-io"}]}
May-19 12:16:18.694 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 3.4.4
May-19 12:16:18.768 [main] INFO  org.pf4j.AbstractPluginManager - Plugin 'nf-amazon@3.4.4' resolved
May-19 12:16:18.768 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin 'nf-amazon@3.4.4'
May-19 12:16:22.871 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-amazon@3.4.4
May-19 12:16:23.114 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
May-19 12:16:23.114 [main] DEBUG nextflow.file.FileHelper - Started plugin 'nf-amazon' required to handle file: s3://ngi-igenomes/igenomes/
May-19 12:16:23.284 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider; config=[:]
May-19 12:16:23.475 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {max_error_retry=5}
May-19 12:16:39.701 [main] ERROR n.v.parameters.ParameterValidator - Validation of pipeline parameters failed!

(And further down, a repeat of the error message about DNS failures that I posted above.)

[scwatts]

Hi @oneillkza, I missed your reply - were you able to solve this issue? It seems most likely an issue with the compute network connection, or firewall. One change to try would be setting Nextflow / oncoanalyser to run in offline mode if you haven't already done that

[oneillkza]

Thanks @scwatts !

Do you mean using the nf-core offline functionality via setting NXF_OFFLINE='true'? Will that also pre-cache things?

I have been running with local reference data (via a refdata.local.config), but it still seems to be trying to download from igenomes...

[scwatts]

Give this section of the nf-core docs a try: https://nf-co.re/docs/usage/offline#transfer-nextflow-offline. The NXF_OFFLINE can also be set in the BASH environment to avoid editing your ~/.bashrc

I would generally recommend for large analyses fully running offline to decrease reliance (and points of failure) on external services

If the above doesn't solve unwanted iGenomes requests, I'll take a closer look and see whether I can replicate