test_full: writing a downloaded database to an s3:// storeDir fails, aws: command not found

[erikrikarddaniel]

Description of the bug

-profile test_full cannot write a downloaded database into an s3:// storeDir, because the
AWS CLI is not available in the environment:

.command.run: line 217: aws: command not found

The download itself succeeds. In the failing run CAZy.dmnd, dbCAN-sub.hmm and
fam-substrate-mapping.tsv were all fetched, and the failure came afterwards, when Nextflow tried
to copy them into the storeDir.

How the databases are obtained

This is worth stating explicitly, because the parameter names suggest otherwise:

Tool	Mechanism	In conf/test_full.config
dbCAN	downloads, caches via storeDir (conf/modules.config:409)	dbcan_dbpath -> s3://
EggNOG	downloads, caches via storeDir (conf/modules.config:353)	eggnog_dbpath -> s3://
KOfamScan	downloads, caches via storeDir (conf/modules.config:374)	kofam_dir not set -> local ./kofam/
EUKulele	consumes a pre-existing database via fromPath	eukulele_dbpath -> s3://

So dbcan_dbpath/eggnog_dbpath/kofam_dir are cache destinations, not sources. The
databases do not need to be uploaded anywhere -- the pipeline fetches them. Only EUKulele takes a
database as input, and that one is present in the bucket (test-data/metatdenovo/gtdb_eukulele/)
and works.

Consistent with this, s3://ngi-igenomes/test-data/metatdenovo/dbcan/ is empty. That is not a
missing upload: it is a write destination that has never been successfully written to, because the
copy has always failed for the reason above.

Consequences

• dbCAN and EggNOG both fail this way in the full test. dbCAN is currently skipped in
  conf/test_full.config to get the profile running; that is a workaround for the missing CLI, not
  a fix.
• Once the AWS CLI is available, dbCAN can simply be re-enabled -- nothing needs uploading.
• KOfamScan can be enabled at the same time. It is skipped in the full test today, but it works
  exactly like the other two. Pointing kofam_dir at a bucket prefix as well would let it cache
  between runs instead of re-downloading from genome.jp every time.
• Since awsfulltest is the workflow that exercises this profile, it is worth checking whether its
  recent runs have been failing for this reason.

Worth confirming when fixing: which staging/publishing path generates the aws call. Nextflow's
base command-run.txt template contains no aws invocation, so it comes from a dynamically
generated block. The ~15 lines around line 217 of a failing .command.run would show whether the
fix is "make awscli available in that environment" or "use a different storeDir scheme".

Second, separate bug: new File() on an S3 path

workflows/metatdenovo.nf:848:

d = new File("${params.eukulele_dbpath}")
if ( ! d.exists() ) {
    d.mkdirs()
}

new File() is a local-filesystem path, so with an s3:// --eukulele_dbpath it can never exist.
This creates a literal directory named s3: in the launch directory and then proceeds. The
following channel.fromPath(params.eukulele_dbpath, checkIfExists: true) resolves the real S3 path
independently, so the effect is stray directories rather than a wrong result -- but the existence
check does nothing useful for any non-local path and should not assume a local filesystem.

System information

• Nextflow 26.04.6
• Found while running -profile test_full on branch issue-460-protein-consolidation; none of the
  above is related to that branch, all of it predates it.

[erikrikarddaniel]

Rewrote the description above — the original framing was wrong and worth flagging in case anyone read it first.

I had reported that the dbCAN database "was never uploaded", on the evidence that s3://ngi-igenomes/test-data/metatdenovo/dbcan/ lists zero keys. The listing is accurate; the inference was not. dbcan_dbpath is a storeDir (conf/modules.config:409) — a cache destination — and the pipeline downloads the database itself. The empty prefix is a write target that has never been written to, not a missing upload.

What exposed it: the failing run had already downloaded CAZy.dmnd, dbCAN-sub.hmm and fam-substrate-mapping.tsv before it failed, which is impossible if the database had to come from the bucket.

Two consequences, both in the rewritten description: re-enabling dbCAN needs nothing uploaded, only the AWS CLI available; and KOfamScan works the same way, so it can be enabled alongside rather than being a separate piece of work.