Skip to content

Refactor resource scaling in nextflow.config tower profile #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamjtaylor wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Refactor resource scaling in nextflow.config tower profile #28

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 58 additions & 10 deletions nextflow.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,76 @@
process.container = 'ghcr.io/sage-bionetworks-workflows/nf-artist:latest'
docker.enabled = true

// Global retry strategy for Docker OOM errors
process {
errorStrategy = {
// Handle Docker out-of-memory errors (exit codes 125, 137, 139)
if (task.exitStatus in [125, 137, 139]) {
return task.attempt <= 3 ? 'retry' : 'ignore'
}
// Handle general failures
else if (task.exitStatus != 0) {
return task.attempt <= 2 ? 'retry' : 'ignore'
}
return 'ignore'
}
maxRetries = 3

// Default resource scaling
cpus = { 2 * task.attempt }
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The global default CPU allocation grows unbounded with retry attempts (2, 4, 6, 8, etc.). Consider adding a Math.min() cap similar to the tower profile's process_medium and process_high labels to prevent excessive CPU allocation on later retries.

Suggested change
cpus = { 2 * task.attempt }
cpus = { Math.min(2 * task.attempt, 8) }

Copilot uses AI. Check for mistakes.
memory = { 4.GB * Math.pow(2, task.attempt - 1) } // Exponential memory scaling
}

profiles {
test { includeConfig 'conf/test.config'}
sage { includeConfig 'conf/sage.config'}
tower {
process {
withLabel: process_low {
cpus = {1 * task.attempt}
memory = {2.GB * task.attempt}
cpus = { 1 * task.attempt }
memory = { 2.GB * Math.pow(2, task.attempt - 1) } // 2GB, 4GB, 8GB
maxRetries = 3
errorStrategy = {task.attempt <= 2 ? 'retry' : 'ignore' }
errorStrategy = {
if (task.exitStatus in [125, 137, 139]) {
return task.attempt <= 3 ? 'retry' : 'ignore'
}
return task.attempt <= 2 ? 'retry' : 'ignore'
}
}
withLabel: process_medium {
cpus = {4 * task.attempt}
memory = {8.GB * task.attempt}
cpus = { Math.min(4 * task.attempt, 8) }
memory = { 8.GB * Math.pow(2, task.attempt - 1) } // 8GB, 16GB, 32GB
maxRetries = 3
errorStrategy = {task.attempt <= 3 ? 'retry' : 'ignore' }
errorStrategy = {
if (task.exitStatus in [125, 137, 139]) {
return task.attempt <= 3 ? 'retry' : 'ignore'
}
return task.attempt <= 3 ? 'retry' : 'ignore'
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error strategy for process_medium has redundant logic - both the OOM error case and the default case return the same condition (task.attempt <= 3). This can be simplified to just 'return task.attempt <= 3 ? 'retry' : 'ignore'' without the if statement, or the non-OOM case should have a different retry limit to match the differentiation in other process labels.

Suggested change
return task.attempt <= 3 ? 'retry' : 'ignore'
return task.attempt <= 2 ? 'retry' : 'ignore'

Copilot uses AI. Check for mistakes.
}
}
withLabel: process_high {
cpus = {8 * task.attempt}
memory = {16.GB * task.attempt}
maxRetries = 3
errorStrategy = {task.attempt <= 3 ? 'retry' : 'ignore' }
cpus = { Math.min(8 * task.attempt, 16) }
memory = { 16.GB * Math.pow(2, task.attempt - 1) } // 16GB, 32GB, 64GB
maxRetries = 4 // One extra retry for high-memory processes
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment states 'high-memory processes' but this is for the process_high label, not process_high_memory. Either the comment should say 'high processes' or it may be confusing since process_high_memory is introduced later.

Suggested change
maxRetries = 4 // One extra retry for high-memory processes
maxRetries = 4 // One extra retry for high processes

Copilot uses AI. Check for mistakes.
errorStrategy = {
if (task.exitStatus in [125, 137, 139]) {
return task.attempt <= 4 ? 'retry' : 'ignore'
}
return task.attempt <= 3 ? 'retry' : 'ignore'
}
}

// Special handling for very memory-intensive processes
withLabel: process_high_memory {
cpus = { Math.min(4, 2 * task.attempt) }
memory = { 32.GB * Math.pow(2, task.attempt - 1) } // 32GB, 64GB, 128GB
maxRetries = 4
errorStrategy = {
if (task.exitStatus in [125, 137, 139]) {
return task.attempt <= 4 ? 'retry' : 'ignore'
}
return task.attempt <= 2 ? 'retry' : 'ignore'
}
}
}
}
Expand Down