Inconsistent handling of gres in profiles

[w8jcik]

Software Versions

Snakemake 9.1.3
snakemake-executor-plugin-slurm 1.1.0

Describe the bug

This works

default-resources:
  gres: "'gpu:1'"

but with gres set in a rule, in the same profile file

set-resources:
  "some_rule":
    ...
    gres: "'gpu:1'"

following error appears

WorkflowError:
Invalid GRES format: 'gpu:1'. Expected format: '<name>:<number>' or '<name>:<type>:<number>' (e.g., 'gpu:1' or 'gpu:tesla:2')

Aside from failing submission, the message suggests to use format that is already used.

More details in case parameters interfere with each other

default-resources:
  tasks: 1
  threads: 12
  cpus_per_task: 12
  runtime: 5
  slurm_partition: "deflt"
  gres: "'gpu:1'"
  slurm_extra: "'--exclude node5'"

set-resources:
  "failing_rule":
    runtime: 30240
    slurm_partition: "long"
    nodes: 1
    tasks: 4
    mpi: "srun"
    gres: "'gpu:4'"

[cmeesters]

Interesting observation - I am sorry, we are still putting some work in the documentation.

Please try not to nest this string, so gres: "gpu:1" instead of gres: "'gpu:2'". If your cluster setup allows for it, you can simply write gpu: 1, too.

I will put an extra check and a more informative error message in the upcoming release.

[w8jcik]

Removing the extra single quotes leads to parsing error and suggestion to add them.

As far as I know the gpu/gpus parameter uses different mechanism than Slurm GRES so it won't work when using GRES. Just to confirm I tried gpu switch and GRES is not requested (no GPU is requested).

[cmeesters]

I will look into this.

Regarding the gpu resource (which translates to --gpus in SLURM: This ought to work. You can try yourself with srun -p deflt --pty -t 10 --gpus=1 bash -i and then, when the job is started on you are on the gpu-node: nvidia-smi. However, it might be, that your cluster setup does not work with this flag. This is why the upcoming documentation will state, that you need to refer to your local documentation. Which version of SLURM are you using?