awsbatch not publishing output files

[xmeng]

Description of the bug

Hello! I’m new to nf-core/mag and recently ran some tests on an EC2 machine, which completed successfully—thank you to the developers for the great work. However, when I tried running the same tests on AWS Batch, the pipeline didn’t behave as expected. In most cases, MAG didn’t publish any output files except the pipeline_info folders, and in other cases it failed midway through the workflow. Thanks.

Command used and terminal output

Add the following lines into nextflow.config
process {
executor = 'awsbatch'
queue = 'default-pipelines'
cleanup = true
cache = false
}
aws.region = 'us-west-2'

aws batch submit-job \
    --job-name nf-core-mag_minimal \
    --job-queue priority-pipelines \
    --job-definition nextflow-production \
    --container-overrides command="nextflow-pipeline/mag, \
    "-profile", "test_minimal,debug,docker", \
    "--outdir", "s3://genomics-workflow-core/Results/nf-mag/test_minimal" "

Relevant files

nextflow_aws_test_minimal.log

System information

Version: 25.04.8 build 5956
Created: 06-10-2025 21:19 UTC
System: Linux 4.14.232-176.381.amzn2.x86_64
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 17.0.18+9-LTS
Encoding: UTF-8 (ANSI_X3.4-1968)

[xmeng]

Fixed the issue