Bug Report: flux-pam prolog crashes due to comma-separated DeviceAllow string

[DESD-04]

The Problem

When a user submits a job to a node that has multiple GPUs or Infiniband devices, the node immediately goes down
because the flux-pam-prolog python script crashes with:
failed to apply constraints: Command '['/usr/bin/systemctl', 'set-property', '--runtime', 'user-@.slice',
'DeviceAllow=/dev/dri/card1 rw,/dev/dri/renderD128 rw,...']' returned non-zero exit status 1.

The Root Cause

This issue occurs at the intersection of flux-core generating the systemd properties, and flux-pam trying to apply
them via the command line:

1. The Origin (flux-core): In flux-core PR #7566, commit 04bce0b3d (python: add flux.sdexec.map resource mapper), the
   HwlocMapper was introduced to map Flux resource IDs into systemd unit properties. Inside this commit, the map_gpus()
   method intentionally joins all hardware paths into a single comma-separated string: return {"DeviceAllow": ",".
   join(unique_devices)}.
2. The Crash (flux-pam): When the flux-pam integration retrieves this dictionary, the modify_slice() function in
   flux/pam.py blindly loops over the keys and passes them as arguments to systemctl set-property.
3. Why systemd rejects it: While a comma-separated string is perfectly valid inside a raw systemd .service text file,
   the systemctl set-property command-line tool explicitly does not accept a single string with commas for DeviceAllow.
   It treats the entire string as one literal path, realizes that a device named card1 rw,/dev/dri/render does not
   physically exist, and rejects the command with Invalid argument (exit 1).

The Fix

The easiest and most robust fix is to patch flux/pam.py (in the flux-pam repository) so that it splits the
DeviceAllow string and passes each device as its own distinct command-line argument to systemctl.

File: /usr/lib64/python3.12/site-packages/flux/pam.py (around line 238)

Before:

        for key, value in properties.items():
            args.append(f"{key}={value}")

After:

        for key, value in properties.items():
            if key == "DeviceAllow" and "," in str(value):
                for dev in str(value).split(","):
                    args.append(f"{key}={dev}")
            else:
                args.append(f"{key}={value}")

This forces the Python subprocess to pass "DeviceAllow=/dev/dri/card1 rw" "DeviceAllow=/dev/dri/renderD128 rw" as
separate arguments, which systemd parses correctly.

[grondo]

Thanks for the report! We did not catch this due to an unfortunate testing gap:

1. The in-tree tests only use a single device in DeviceAllow 🤦
2. The system on which we're testing this version of flux-pam on-site doesn't have GPUs, so the DeviceAllow path didn't get real-world testing