Skip to content

Consolidate GPU destinations #1931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
domgz merged 3 commits into from
Mar 5, 2026
Merged

Consolidate GPU destinations #1931

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c01e119
Consolidate GPU destinations
domgz Mar 4, 2026
6a1f42d
Remove various parameters from `condor_container_gpu` destination
domgz Mar 4, 2026
7da29be
Do not run Singularity containers in `condor_container_gpu_divide4`
domgz Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 28 additions & 50 deletions files/galaxy/tpv/destinations.yml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -457,6 +457,8 @@ destinations:
- name: APPTAINER_CACHEDIR
value: /scratch/singularity/
params:
request_gpus: "{gpus or 0}"

# we could set this to false, and usually, afaik, this would then fallback to conda-resolution, however, I put in here a `container:` fallback, so maybe we can leave that to true
require_container: true
#
Expand Down Expand Up @@ -502,6 +504,32 @@ destinations:
- docker
- singularity

condor_container_gpu:
inherits: condor_container
max_accepted_cores: 128
max_accepted_mem: 500
min_accepted_gpus: 1
max_accepted_gpus: 4
params:
requirements: 'GalaxyGroup == "pxe-gpu"'
docker_run_extra_arguments: "{entity.params.get('docker_run_extra_arguments') or ''} --gpus all --env CUDA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs --env NVIDIA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs"
singularity_run_extra_arguments: "{entity.params.get('singularity_run_extra_arguments') or ''} --nv --env CUDA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs"
env:
GPU_AVAILABLE: 1

# This means a GPU can be shared by max 4 jobs at the same time
condor_container_gpu_divide4:
inherits: condor_container_gpu
max_accepted_gpus: 1
params:
singularity_enabled: false # sharing GPUs has not been tested with Singularity
requirements: 'GalaxyGroup == "pxe-gpu-div4"'
scheduling:
require:
- gpu-divided
Comment on lines +527 to +529

@domgz domgz Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mira-miracoli Will shared GPUs actually work with Singularity containers?

@gsaudade99 gsaudade99 Mar 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this destination actually needs a tag and we currently only have two tools with that tag that are explicit docker I don't think it will be a problem. But agreed that we should probably check if this works with singularity

@bgruening bgruening Mar 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it will work with Singularity, at least we never tested this, afaik.

@domgz domgz Mar 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will then add a "reject: singularity" and set "singularity_enabled: false" for this destination. It'll be interesting to see what happens when TPV finds both accept: singularity and reject: singularity 😆.

@domgz domgz Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 7da29be.

reject:
- singularity # sharing GPUs has not been tested with Singularity

condor_singularity_with_conda:
inherits: basic_singularity_destination
runner: condor
Expand Down Expand Up @@ -536,53 +564,3 @@ destinations:
scheduling:
require:
- internal

condor_docker_gpu_pxe:
inherits: basic_docker_destination
# shorter than inheriting from condor_gpu
runner: condor
max_accepted_cores: 128
max_accepted_mem: 500
min_accepted_gpus: 1
max_accepted_gpus: 4
scheduling:
# Try to make this the default GPU destination if a tool is untagged (e.g. from upstream)
accept:
- docker
env:
GPU_AVAILABLE: 1
params:
requirements: 'GalaxyGroup == "pxe-gpu"'
request_gpus: "{gpus or 0}"
docker_run_extra_arguments: "{entity.params.get('docker_run_extra_arguments') or ''} --gpus all --env CUDA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs --env NVIDIA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs"

# This means a GPU can be shared by max 4 jobs at the same time
condor_docker_gpu_pxe_divide4:
inherits: condor_docker_gpu_pxe
max_accepted_gpus: 1
params:
requirements: 'GalaxyGroup == "pxe-gpu-div4"'
scheduling:
require:
- gpu-divided

condor_singularity_gpu_pxe:
inherits: basic_singularity_destination
# shorter than inheriting from condor_gpu
runner: condor
max_accepted_cores: 128
max_accepted_mem: 500
min_accepted_gpus: 1
max_accepted_gpus: 1
env:
- name: SINGULARITY_CACHEDIR
value: /scratch/singularity/
- name: APPTAINER_CACHEDIR
value: /scratch/singularity/
scheduling:
require:
- singularity
params:
requirements: 'GalaxyGroup == "pxe-gpu"'
request_gpus: "{gpus or 0}"
singularity_run_extra_arguments: "{entity.params.get('singularity_run_extra_arguments') or ''} --nv --env CUDA_VISIBLE_DEVICES=$_CONDOR_AssignedGPUs"
Loading