Add internal page about job resource limits

[gkaf89]

There in broken link in /slurm/qos/ to the internal page jobs/limits.md. It seems that the page has never been created.

This seems like an important notion to understand job limits. We should probably create the page.

Relevant presentations:

• https://jhpce.jhu.edu/slurm/environment/

[gkaf89]

The limits are discussed in detail in Slurm documentation.

[gkaf89]

We should probably add some details about how constraints and tractable resources are used to determine resource limits, and how can users find out the available constraint and tractable resource options.

Constraints and tractable resources limit the selection of nodes, and are not listed in the summary command sinfo that prints information about Slurm nodes and partitions. The constraints for each nodes can be viewed with the command

scontrol show nodes

in the fields

• AvailableFeatures and
• ActiveFeatures.

Similarly, the available tractable resources are listed in the field CfgTRES, and the resources that are currently be used are listed in the field AllocTRES.

I could not find a way to provide a summary view of these fields for the whole cluster similarly to who sinfo summarizes partition job resource limits.

This is not relevant to our current setup, but constraints can also be combined with a set of operations.

[gkaf89]

Add some details on how memory is allocated to jobs

In our systems we set the MaxMemPerCPU parameter in /etc/slurm/slurm.conf on a per partition basis. This means that there is a hard cap in the memory available per core. You cannot get more memory per core by setting --mem or --mem-per-cpu, if this option exceeds MaxMemPerCPU they simply trigger an adjustment feature that add more cores (with MaxMemPerCPU). No more memory per core is available, just more cores per job. So if --cpus-per-task is set to 1, the srun will only use 1 core and wont have access to the extra memory.

Display details with:

scontrol show config
scontrol show partitions

In summary, if you lunch a job with 1 cpu per process, it won't have enough resources to finish. If you need more memory per process you need to allocate more CPUs per process.

---

Source: man slurm.conf

MaxMemPerCPU

Maximum real memory size available per allocated CPU in megabytes. Used to avoid over-subscribing memory and causing paging. MaxMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_tres). The default value is 0 (unlimited). Also see DefMemPerCPU, DefMemPerGPU and MaxMemPerNode. MaxMemPerCPU and MaxMemPerNode are mutually exclusive.

NOTE: If a job specifies a memory per CPU limit that exceeds this system limit, that job's count of CPUs per task will try to automatically increase. This may result in the job failing due to CPU count limits. This auto-adjustment feature is a best-effort one and optimal assignment is not guaranteed due to the possibility of having heterogeneous configurations and multi-partition/qos jobs. If this is a concern it is advised to use a job submit LUA plugin instead to enforce auto-adjustments to your specific needs.