Problems with the internal jobstep submission

[jd3234]

Software Versions

• snakemake: v8.29.3
• snakemake-executor-plugin-slurm 0.15.1 pypi_0 pypi
• snakemake-executor-plugin-slurm-jobstep 0.2.1 pypi_0 pypi
• slurm 23.11.3

Describe the bug
Often - but not always reproducible- I am getting this error:

srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.0 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted

This error seems to appear when running too many samples in parallel for the same rules. But when only few samples run it might not happen in the same rule.

I have also seen it when asking for too many cpus_per_thread that are then unused (e.g. I accidently asked for 20 cpus, but the tool does not support multithreading, when testing with fewer cpus (5) it worked again, with 1 cpu it always works).

Here is an example of a rule where I have seen the error:

rule clustering_1:
    name: TOOL + '.clustering_1'
    message: "Clustering reads to make a list of most abundant, representative reads. Rule {rule}."
    input:
        fasta = rules.read_filter.output.fasta
    output:
        clusterfasta = 'clustering_1/{sample}/{sample}_cluster_representatives.fasta',
        cluster = 'clustering_1/{sample}/{sample}_cluster_representatives.fasta.clstr'
    resources: 
        mem_mb = lambda wildcards, attempt: attempt * 2500,
        cpus_per_task = 16
    benchmark: 
        'benchmarks/clustering_1_{sample}.tsv'
    retries: 5
    container: os.environ['apptainer_container']
    params:
        clusteringidentity = IDENTITY,
        wordlen = WORDLEN,
        clustprog = '1' #accurate but slow mode
    log:
        cluster = 'logs/{sample}/{sample}_clustering1-report.log',
    shell:
       """
            # This is the actual clustering command
            cd-hit-est -T {resources.cpus_per_task} -i {input.fasta} -o {output.clusterfasta} -c {params.clusteringidentity} -n {params.wordlen} -d 0 -M {resources.mem_mb} -g {params.clustprog} > {log.cluster}
       """

Logs

host: [NODENAME]
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided remote nodes: 1
Provided resources: mem_mb=2500, mem_mib=2385, disk_mb=1000, disk_mib=954, cpus_per_task=16
Select jobs to execute...
Execute 1 jobs...

[Tue Mar 11 14:06:39 2025]
Job 0: Clustering reads to make a list of most abundant, representative reads. Rule ont-amplicon_assembly.clustering_1.
Reason: Forced execution


         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.0 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted
[Tue Mar 11 14:06:39 2025]
Error in rule ont-amplicon_assembly.clustering_1:
    jobid: 0
    input: filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta
    output: clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta, clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta.clstr
    log: logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log (check log file(s) for error details)
    shell:
        
         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Mar 11 14:06:39 2025]
Job 0: Clustering reads to make a list of most abundant, representative reads. Rule ont-amplicon_assembly.clustering_1.
Reason: Forced execution


         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.1 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted
[Tue Mar 11 14:06:39 2025]
Error in rule ont-amplicon_assembly.clustering_1:
    jobid: 0
    input: filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta
    output: clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta, clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta.clstr
    log: logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log (check log file(s) for error details)
    shell:
        
         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Mar 11 14:06:39 2025]
Job 0: Clustering reads to make a list of most abundant, representative reads. Rule ont-amplicon_assembly.clustering_1.
Reason: Forced execution


         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.2 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted
[Tue Mar 11 14:06:39 2025]
Error in rule ont-amplicon_assembly.clustering_1:
    jobid: 0
    input: filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta
    output: clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta, clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta.clstr
    log: logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log (check log file(s) for error details)
    shell:
        
         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Mar 11 14:06:39 2025]
Job 0: Clustering reads to make a list of most abundant, representative reads. Rule ont-amplicon_assembly.clustering_1.
Reason: Forced execution


         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.3 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted
[Tue Mar 11 14:06:39 2025]
Error in rule ont-amplicon_assembly.clustering_1:
    jobid: 0
    input: filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta
    output: clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta, clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta.clstr
    log: logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log (check log file(s) for error details)
    shell:
        
         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Mar 11 14:06:39 2025]
Job 0: Clustering reads to make a list of most abundant, representative reads. Rule ont-amplicon_assembly.clustering_1.
Reason: Forced execution


         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.4 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted
[Tue Mar 11 14:06:39 2025]
Error in rule ont-amplicon_assembly.clustering_1:
    jobid: 0
    input: filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta
    output: clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta, clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta.clstr
    log: logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log (check log file(s) for error details)
    shell:
        
         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Mar 11 14:06:39 2025]
Job 0: Clustering reads to make a list of most abundant, representative reads. Rule ont-amplicon_assembly.clustering_1.
Reason: Forced execution


         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=125452.5 failed on node [NODENAME]: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted
[Tue Mar 11 14:06:39 2025]
Error in rule ont-amplicon_assembly.clustering_1:
    jobid: 0
    input: filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta
    output: clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta, clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta.clstr
    log: logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log (check log file(s) for error details)
    shell:
        
         # This is the actual clustering command
         cd-hit-est -T 16 -i filtered/2000169_EB20S0G1077-0153_[string]_5401_5800_coverage_quality_size_filtered.fasta -o clustering_1/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_cluster_representatives.fasta -c 0.99 -n 10,11 -d 0 -M 2500 -g 1 > logs/2000169_EB20S0G1077-0153_[string]_5401_5800/2000169_EB20S0G1077-0153_[string]_5401_5800_clustering1-report.log
       
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Storing output in storage.
WorkflowError:
At least one job did not complete successfully.

Minimal example

Additional context
We have looked into the submission scripts and found that removing --executor slurm-jobstep --jobs 1 in

snakemake-executor-plugin-slurm/snakemake_executor_plugin_slurm/__init__.py

Line 168 in 66dcdcf

def additional_general_args(self):

and returning an empty string solves the problem.

Why is slurm-jobstep used? Would it be possibile to have an option/parameter to switch that behavior off?

We are currently onboarding slurm and we are still a little bit inexperienced. It is also possible that our SLURM cluster is not properly configured yet. In a couple weeks we will upgrade slurm to v24.11.

[cmeesters]

Thank you for your bug report!

Why is slurm-jobstep used?

Snakemake essentially submits itself. This is to ensure the portability of workflows with all features (i.g. group jobs and more to come). So, no, you will not be able to switch it off.

The error message is weird:

srun: error: JobId=125452 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen

Apparently, srun is ill-behaving. I have never seen this error message.

What happens if you run sbatch hello.sh with the following test script:

#SBATCH -A <your slurm account>
#SBATCH -p <your partition>

srun echo "Hello world from $(hostname)"

if saved as hello.sh? Does srun work?

[jd3234]

Hi,

thanks for your quick answer. I believe that srun does work. Here is what I did

#!/bin/sh

#SBATCH -A default
#SBATCH -p general

srun echo "Hello world from $(hostname)"

Executing

sbatch hello.sh
Submitted batch job 126978

Checking

cat slurm-126978.out

Hello world from [here was the hostname]

[cmeesters]

sorry, I forgot the shebang ...

Anyway, can you please update the jobstep executor (to version 0.3) and try again? You might want to update the actual slurm executor too (packages snakemake-executor-plugin-slurm and snakemake-executor-plugin-slurm-jobstep), because recent SLURM versions require an indication of the number of CPUs per task to get the c-group right. Otherwise, there is no well threaded parallel execution possible. However, the introduction of better gpu support might have thrown the plugins out of sync.

[jd3234]

I tried to upgrade but there seem to be conflicting dependencies

Successfully installed snakemake-executor-plugin-slurm-jobstep-0.3.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snakemake-executor-plugin-slurm 0.15.1 requires snakemake-executor-plugin-slurm-jobstep<0.3.0,>=0.2.0, but you have snakemake-executor-plugin-slurm-jobstep 0.3.0 which is incompatible.

[jd3234]

Getting the versions this happens for snakemake ImportError: cannot import name 'get_cpus_per_task' from 'snakemake_executor_plugin_slurm_jobstep'. Gets into the direction you are indication.

mamba list | grep "snakemake-executor-plugin-slurm"
snakemake-executor-plugin-slurm 0.15.1                   pypi_0    pypi
snakemake-executor-plugin-slurm-jobstep 0.3.0                    pypi_0    pypi

snakemake --version
Traceback (most recent call last):
  File "[PATH]/env/bin/snakemake", line 6, in <module>
    from snakemake.cli import main
  File "[PATH]/env/lib/python3.12/site-packages/snakemake/cli.py", line 23, in <module>
    from snakemake.api import (
  File "[PATH]/env/lib/python3.12/site-packages/snakemake/api.py", line 49, in <module>
    from snakemake.workflow import Workflow
  File "[PATH]/env/lib/python3.12/site-packages/snakemake/workflow.py", line 66, in <module>
    from snakemake.scheduler import JobScheduler
  File "[PATH]/env/lib/python3.12/site-packages/snakemake/scheduler.py", line 27, in <module>
    registry = ExecutorPluginRegistry()
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH]/env/lib/python3.12/site-packages/snakemake_interface_common/plugin_registry/__init__.py", line 31, in __init__
    self.collect_plugins()
  File "[PATH]/env/lib/python3.12/site-packages/snakemake_interface_executor_plugins/registry/__init__.py", line 63, in collect_plugins
    super().collect_plugins()
  File "[PATH]/env/lib/python3.12/site-packages/snakemake_interface_common/plugin_registry/__init__.py", line 77, in collect_plugins
    module = importlib.import_module(moduleinfo.name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH]/env/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH]/env/lib/python3.12/site-packages/snakemake_executor_plugin_slurm/__init__.py", line 29, in <module>
    from snakemake_executor_plugin_slurm_jobstep import get_cpus_per_task
ImportError: cannot import name 'get_cpus_per_task' from 'snakemake_executor_plugin_slurm_jobstep'

[cmeesters]

Apparently the bioconda update lags behind: the plugin version is 1.0.0, and I hope to release another bug fix tomorrow.

[jd3234]

Thanks for pointing that out! I had not specified a version and assumed that the latest versions would be installed - and have to admit I forgot to check if that is true. I am installing everything again and will test it.

[jd3234]

Unfortunately, the problem persists... But it seems to be better somehow. Instead of all jobs dying immediately the majority seems to run through now.

[jd3234]

Taking my last comment back... Not better :(

[cmeesters]

Version 1.0.1 is on bioconda. That (as well as version 1.0.0) should work with the current jobstep plugin.

[jd3234]

I upgraded everything but problem persists.

snakemake --version
8.30.0

mamba list | grep "snakemake-executor-plugin-slurm"
snakemake-executor-plugin-slurm 1.0.1                    pypi_0    pypi
snakemake-executor-plugin-slurm-jobstep 0.3.0                    pypi_0    pypi

[cmeesters]

Please indicate your command line when starting the workflow, the snakemake profile and the workflow profile. And can you please alter your hello.sh to contain:

#!/bin/sh

#SBATCH -A default
#SBATCH -p general
#SBATCH -c 16

srun -n1 --cpu-bind=q --cpus-per-task=1 echo "Hello world from $(hostname)"

and show the slurm output?

[jd3234]

I am not using anything extra (snakemake or workflow profile) intentionally. I define resources in the rules or in the command.

This is my command:

snakemake --snakefile pipeline.smk --config samples=samples.tsv target=bla quality=12 identity=0.99   --executor slurm --default-resources slurm_account=default slurm_partition=general --slurm-keep-successful-logs --benchmark-extended --software-deployment-method apptainer --apptainer-args '-B /mnt:/mnt' --jobs 150 --latency-wait 90

Running the change script returns the host name. I can provide info that slurm is recording. Which command is the most meaningful for you?

sacct -j 136598
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
136598         hello.sh    general    default         16  COMPLETED      0:0 
136598.batch      batch               default         16  COMPLETED      0:0 
136598.exte+     extern               default         16  COMPLETED      0:0 
136598.0           echo               default          1  COMPLETED      0:0

[jd3234]

Here is the command from one of the failing samples when I use snakemake with --verbose

sbatch call: sbatch --parsable --job-name 1121be98-ef99-4b6a-a296-07975216f864 --output '[PATH]/.snakemake/slurm_logs/rule_pipeline.clustering_1/FOLDER/%j.log' --export=ALL --comment 'rule_pipeline.clustering_1_wildcards_[SAMPLE]' -A 'default' -p general --mem 2500 --ntasks=1 --cpus-per-task=16 -D [PATH] --wrap="[OTHER_PATH]/env/bin/python3.12 -m snakemake --snakefile '[PATH]/pipeline.smk' --target-jobs 'pipeline.clustering_1:sample=[SAMPLE]' --allowed-rules pipeline.clustering_1 --cores 64 --attempt 1 --force-use-threads  --resources 'mem_mb=2500' 'mem_mib=2385' 'disk_mb=1000' 'disk_mib=954' 'cpus_per_task=16' --wait-for-files [PATH]/.snakemake/tmp.e69tsuqs' 'filtered/[SAMPLE]_filtered.fasta' --force --target-files-omit-workdir-adjustment --keep-storage-local-copies --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --verbose  --rerun-triggers software-env input mtime params code --deployment-method apptainer --conda-frontend 'conda' --apptainer-args base64//LUIgL21udDovbW50 --shared-fs-usage persistence input-output sources source-cache storage-local-copies software-deployment --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --config 'samples=samples.tsv' 'target=bla' 'quality=12' 'identity=0.99' --benchmark-extended  --latency-wait 90 --scheduler 'ilp' --local-storage-prefix .snakemake/storage --scheduler-solver-path '[PATH]/env/bin' --default-resources base64//bWVtX21iPW1pbihtYXgoMippbnB1dC5zaXplX21iLCAxMDAwKSwgODAwMCk= base64//ZGlza19tYj1tYXgoMippbnB1dC5zaXplX21iLCAxMDAwKQ== base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= base64//c2x1cm1fYWNjb3VudD1kZWZhdWx0 base64//c2x1cm1fcGFydGl0aW9uPWdlbmVyYWw= --executor slurm-jobstep --jobs 1 --mode 'remote'"

[cmeesters]

ok, this is a wild guess, but: line 119 of the jobstep executor contains --cpu-bind=q. I want to keep that on for the most parts, but if you would be so kind to out-comment this in your installation and test your workflow? If it then works, I would code you a flag with which you can turn this off, such that no future tinkering is needed.

My guess is, that your container or compute node or OS does not appreciate bindings of CPUs. It should be a SLURM default, but I cannot count on defaults, hence the presence of this flag.

You can find the code at <your conda prefix>/lib/python3.12/site-packages/snakemake-executor-plugin-slurm-jobstep/__init__.py.

Yet, this is really a wild guess. (And I cannot test right now, because my cluster is broken. But, the error message, you will agree ;-), is really weird and seems to indicate there is something wrong with the CPU binding in this particular context.)