BenchExec and SLURM

[PhilippWendler]

Let's use this discussion as a place for connecting people who combine SLURM and BenchExec in any way, e.g., by submitting jobs from BenchExec to SLURM, or by using runexec in SLURM.

Please feel invited to share reports of what you are doing, knowledge, use cases, ideas, help, etc. here and ask questions. Contributions of documentation and code are also invited - for example it would be great if we have documentation on how to configure SLURM such that runexec works.

What we already have is a module for submitting jobs from BenchExec to SLRUM by @leventeBajczi.

ping @mpreiner @leventeBajczi

[leventeBajczi]

This is surreal, as (after quite some time) I was just working on the SLURM module in BenchExec.

We recently lost cgroups-support at our cluster (they still use v1 on our cluster, and upstream slurm decided in version 23 that root shall own the cgroup and therefore we no longer have the capability of running tools like runexec inside a job to limit resources), and I want to update the existing module to be more SLURM-aligned in the way it limits/measures/uses resources.

So soon (hopefully) I'll have something ready to use which should, in theory, be usable without explicit cgroups access, as it would rely on srun --exclusive invocations inside the jobs instead of the current behavior of delegating to runexec. Of course, the major problem I'm trying to tackle is that the two worlds (BenchExec and SLURM) have quite diverging views on resource consumption and limiting.

[PhilippWendler]

Thanks for the reply. It is unfortunate, however. What would be really good is if using BenchExec would always mean using runexec under the hood for the measurements, or at least some equivalent implementation. The phrase "we used BenchExec" in a paper should be a signal for reviewers that the measurements were as reliable as they can get.

[mpreiner]

We use SLURM for job scheduling and runexec for limiting/measuring resources. We recently migrated to SLURM 23.11 and had an issue similar to @leventeBajczi's. User authentication in our setup is through munge, which does not provide a full systemd user session and as a result runexec has issues setting up cgroups. I believe the main issue is still the root permissions for cgroups. What we did to work around this is ensure we always have a valid user session available on the compute node. We do this via a prolog script with the command loginctl enable-linger $SLURM_JOB_USER. Luckily this works, but I'd love to have a less hacky solution.