Add support for autodetection of gres resources #181

jovial · 2025-04-23T17:07:54Z

Adds support for setting the AutoDetection property on gres resources. This prevents the need to manually specify File in the gres dictionary. You can only use one auto-detection mechanism per node, otherwise slurm will complain (hence why it is a per partition option and not a per gres option).

Example:

# group_vars/all/openhpc.yml

openhpc_nodegroups:
    - name: cpu
    - name: gpu
      gres_autodetect: nvml
      gres:
        - conf: "gpu:nvidia_h100_80gb_hbm3:2"
        - conf: "gpu:nvidia_h100_80gb_hbm3_4g.40gb:2"
        - conf: "gpu:nvidia_h100_80gb_hbm3_1g.10gb:6"

templates/gres.conf.j2

README.md

sjpb

Have some concerns

README.md

sjpb · 2025-04-24T10:05:57Z

README.md

      - `conf`: A string with the [resource specification](https://slurm.schedmd.com/slurm.conf.html#OPT_Gres_1) but requiring the format `<name>:<type>:<number>`, e.g. `gpu:A100:2`. Note the `type` is an arbitrary string.
-      - `file`: A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.
+      - `file`: Omit if `gres_autodetect` is set, A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.


Suggested change

- `file`: Omit if `gres_autodetect` is set, A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.

- `file`: Omit if `gres_autodetect` is set. A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.

or move the addition to the end of the item 🤷 ?

Done. I've left it at the beginning, as I felt it was the most important bit of information.

README.md

jovial · 2025-05-08T16:02:03Z

Ready for review but merge #183 first (this PR targets that branch to avoid noise in diff)

This reverts commit ed717da.

sjpb

Few comments, but looks pretty good to me.

sjpb · 2025-05-16T13:41:45Z

README.md

In the initial PR comment you say:

Adds support for setting the AutoDetection property on gres resources. This prevents the need to manually specify File in the gres dictionary. You can only use one auto-detection mechanism per node, otherwise slurm will complain (hence why it is a per partition option and not a per gres option).

Can you change that to

... (hence why it is a per nodegroup option and not a per gres option).

And add something like:

NB: autodetection requires rebuild of the OpenHPC packages - this is not provided by this role

sjpb · 2025-05-16T13:43:09Z

README.md

@@ -59,9 +59,10 @@ unique set of homogenous nodes:
  `free --mebi` total * `openhpc_ram_multiplier`.
  * `ram_multiplier`: Optional.  An override for the top-level definition
  `openhpc_ram_multiplier`. Has no effect if `ram_mb` is set.
-  * `gres`: Optional. List of dicts defining [generic resources](https://slurm.schedmd.com/gres.html). Each dict must define:
+  * `gres_autodetect`: Optional. The [auto detection mechanism](https://slurm.schedmd.com/gres.conf.html#OPT_AutoDetect) to use for the  generic resources. Note: you must still define the `gres` dictionary (see below) but you only need the define the `conf` key.


Suggested change

* `gres_autodetect`: Optional. The [auto detection mechanism](https://slurm.schedmd.com/gres.conf.html#OPT_AutoDetect) to use for the generic resources. Note: you must still define the `gres` dictionary (see below) but you only need the define the `conf` key.

* `gres_autodetect`: Optional. The [auto detection mechanism](https://slurm.schedmd.com/gres.conf.html#OPT_AutoDetect) to use for the generic resources. NB: The `gres` dictionary below is still required but only requires the `conf` key.

Should we also mention the requirement for recompliation here? Rather than just in the example?

Oh or maybe given you've got a whole section on it, maybe just mention that section ("see section below") or something?

sjpb · 2025-05-16T13:44:10Z

README.md

      - `conf`: A string with the [resource specification](https://slurm.schedmd.com/slurm.conf.html#OPT_Gres_1) but requiring the format `<name>:<type>:<number>`, e.g. `gpu:A100:2`. Note the `type` is an arbitrary string.
-      - `file`: A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.
+      - `file`: Omit if `gres_autodetect` is set. A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.
    Note [GresTypes](https://slurm.schedmd.com/slurm.conf.html#OPT_GresTypes) must be set in `openhpc_config` if this is used.


This Note [GresTypes] ... bit is in the wrong place I think isn't it - it applies to the entire gres dict, not to the file key? If you look at the rendered page?

sjpb · 2025-05-16T13:50:26Z

templates/gres.conf.j2

+{%      else %}
+{%          for gres in gres_list %}
+{%              set gres_name, gres_type, _ = gres.conf.split(':') %}
+{%              for hostlist in (inventory_group_hosts | hostlist_expression) %}


Do we want to do something similar to slurm.conf now and provide a comma-sep string, and skip the loops?

{% set hostlist_string = inventory_group_hosts | hostlist_expression | join(',') %} ... NodeName={{ hostlist_string }}

instead of the (2x) loops for this?

sjpb · 2025-05-16T13:55:17Z

templates/gres.conf.j2

+{%      set gres_autodetect = nodegroup.gres_autodetect | default('off') %}
+{%      set inventory_group_name = openhpc_cluster_name ~ '_' ~ nodegroup.name %}
+{%      set inventory_group_hosts = groups.get(inventory_group_name, []) %}
+{%      if gres_autodetect | default('off') != 'off'  %}


Why do we need the default here? Its in l4, or am I missing something?

jovial requested a review from a team as a code owner April 23, 2025 17:07

jovial marked this pull request as draft April 23, 2025 20:20

jovial commented Apr 24, 2025

View reviewed changes

templates/gres.conf.j2 Outdated Show resolved Hide resolved

jovial commented Apr 24, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

jovial marked this pull request as ready for review April 24, 2025 09:04

jovial commented Apr 24, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

sjpb requested changes Apr 24, 2025

View reviewed changes

jovial commented Apr 28, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

jovial requested a review from sjpb April 28, 2025 08:39

sjpb mentioned this pull request May 7, 2025

Add group.node_params to partitions/groups. #182

Merged

jovial marked this pull request as draft May 8, 2025 14:18

jovial changed the base branch from master to feat/nodegroups May 8, 2025 14:27

jovial marked this pull request as ready for review May 8, 2025 15:59

Base automatically changed from feat/nodegroups to master May 13, 2025 08:19

sjpb added 2 commits May 15, 2025 12:29

re-add "Separate node and partition configuration (#183)"

1c8f96f

This reverts commit ed717da.

readme tweak for clarity

5ceb9e1

jovial changed the base branch from master to feat/nodegroups-v2 May 16, 2025 12:48

jovial force-pushed the feature/gres-autodetect branch from 3608f48 to e3f58ad Compare May 16, 2025 12:54

Add support for autodetection of gres resources

1ca4a4e

jovial force-pushed the feature/gres-autodetect branch from e3f58ad to 1ca4a4e Compare May 16, 2025 13:24

sjpb requested changes May 16, 2025

View reviewed changes

Base automatically changed from feat/nodegroups-v2 to master May 16, 2025 14:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for autodetection of gres resources #181

Add support for autodetection of gres resources #181

jovial commented Apr 23, 2025 •

edited

Loading

sjpb left a comment

sjpb Apr 24, 2025

jovial Apr 28, 2025

jovial commented May 8, 2025

sjpb left a comment

sjpb May 16, 2025

sjpb May 16, 2025

sjpb May 16, 2025

sjpb May 16, 2025

sjpb May 16, 2025

sjpb May 16, 2025

sjpb May 16, 2025

	- `file`: Omit if `gres_autodetect` is set, A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.
	- `file`: Omit if `gres_autodetect` is set. A string with the [File](https://slurm.schedmd.com/gres.conf.html#OPT_File) (path to device(s)) for this resource, e.g. `/dev/nvidia[0-1]` for the above example.

	* `gres_autodetect`: Optional. The [auto detection mechanism](https://slurm.schedmd.com/gres.conf.html#OPT_AutoDetect) to use for the generic resources. Note: you must still define the `gres` dictionary (see below) but you only need the define the `conf` key.
	* `gres_autodetect`: Optional. The [auto detection mechanism](https://slurm.schedmd.com/gres.conf.html#OPT_AutoDetect) to use for the generic resources. NB: The `gres` dictionary below is still required but only requires the `conf` key.

Add support for autodetection of gres resources #181

Are you sure you want to change the base?

Add support for autodetection of gres resources #181

Conversation

jovial commented Apr 23, 2025 • edited Loading

sjpb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jovial commented May 8, 2025

sjpb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jovial commented Apr 23, 2025 •

edited

Loading