ras/slurm treats PMIX_ALLOC_EXTEND as PMIX_ALLOC_NEW

[rhc54]

ras/slurm's modify() maps PMIX_ALLOC_EXTEND onto an implementation that
is, by PMIx's own definition, PMIX_ALLOC_NEW.

From pmix_common.h:

#define PMIX_ALLOC_NEW          1   // new allocation is being requested. The resulting allocation will be
                                    // disjoint (i.e., not connected in a job sense) from the requesting allocation
#define PMIX_ALLOC_EXTEND       2   // extend the existing allocation, either in time or as additional resources

What the component does today

prte_ras_slurm_serve_extend_req() → prte_ras_slurm_launch_expander_job()
(src/mca/ras/slurm/ras_slurm_modify_extend.c) runs

salloc --no-shell --exclusive --job-name=prrte --nodes=<N> \
       [--account=…] [--partition=…] [--qos=…] [--chdir=…] \
       [--mem-per-cpu=… | --mem=…] [--time=…] [--threads-per-core=…]

copying the optional fields from the parent job's scontrol show job --json.

The result is a second, independent Slurm job. Verified on a live Slurm
24.11.6 (contrib/slurmswarm), inside a 2-node allocation, after
elastic extend 1:

 JOBID       NAME  NODES   NODELIST  STATE
     5      prrte      1      node3  RUNNING
     4   prte-dvm      2  node[1-2]  RUNNING

job 5:  Dependency=(null)   NumNodes=1
job 4:  Dependency=(null)   NumNodes=2

Two jobs, no dependency between them, nothing merged. That is precisely
PMIX_ALLOC_NEW's contract — "disjoint, not connected in a job sense" — and
it is wired to PMIX_ALLOC_EXTEND. The internal name launch_expander_job
suggests Slurm's job-expansion feature was the intent, but the argv carries
no --dependency=expand: and no merge step ever runs.

This is not cosmetic. The two directives mean different things to the caller
and to the site:

• Each grow is a separate accounting record, against whatever per-user job
  and QoS limits the site enforces.
• --time is copied from the parent's time_limit, not its remaining
  time, so an expander created an hour into a 2-hour job gets a fresh
  2 hours. With no dependency in either direction, it can outlive the
  allocation it was grown from, and the parent can hit its wall while the
  expander keeps running.
• --exclusive is unconditional, however the parent was allocated.
• The DVM ends up spanning N Slurm jobs, which is why the release path has to
  work out which job holds a node before it can resize it.

What a genuine PMIX_ALLOC_EXTEND looks like on Slurm

EXTEND is defined as "either in time or as additional resources", so there
are two things to attempt, and they have very different prospects. Measured
on Slurm 24.11.6 with SelectType=select/cons_tres:

Time — works, and is entirely unimplemented today.

$ scontrol update JobId=$P TimeLimit=30
before: TimeLimit=00:10:00
after:  TimeLimit=00:30:00

This is the clearest reading of "extend the existing allocation ... in time",
it is a one-command implementation, and ras/slurm does nothing with it.
(Increases are privileged; an unprivileged request is refused by Slurm, which
is a fine answer to hand back.)

Additional resources — the documented recipe, which modern Slurm refuses.
Slurm's way to grow a running allocation is to submit a new job that expands
the original and then merge it in:

salloc --no-shell --dependency=expand:<parent_jobid> --nodes=1 …
scontrol update JobId=<new_jobid> NumNodes=0     # merges into the parent, ends the new job

Slurm then writes slurm_job_<jobid>_resize.{sh,csh} for the environment
update — the very files prte_ras_slurm_cleanup_resize_scripts() already
removes on the release path, so the tree has met this machinery from the
shrink side.

On this cluster both routes are refused outright:

$ salloc --no-shell --dependency=expand:6 --nodes=1
salloc: error: Job submit/allocate failed: Requested operation not supported on this system

$ scontrol update JobId=6 NumNodes=3
Requested operation not supported on this system for job 6

while an in-place shrink — what the release path already does — works fine:

$ scontrol update job $P ReqNodeList=node1
before: NodeList=node[1-2] NumNodes=2
after:  NodeList=node1     NumNodes=1

So on a current select/cons_tres Slurm you can extend time and shrink nodes
in place, but you cannot grow nodes in place. A disjoint new allocation is
the only way to get more nodes — which is exactly why the existing expander
code is valuable, and exactly why it belongs under PMIX_ALLOC_NEW.

Suggested shape

1. Move the expander-job implementation to serve PMIX_ALLOC_NEW. It is a
   correct NEW and the component does not handle NEW at all today.
2. Implement PMIX_ALLOC_EXTEND as a real extension: scontrol update JobId=<id> TimeLimit=… for the time case, and the
   --dependency=expand: + NumNodes=0 merge for the resource case, with a
   clean PMIX_ERR_NOT_SUPPORTED where the configuration refuses it — rather
   than silently substituting a disjoint allocation.
3. A caller that genuinely wants a second allocation asks for NEW and gets
   today's behavior, knowingly.

Why this surfaced

ras/slurm not handling PMIX_ALLOC_NEW is a live bug on its own. modify()
returns PMIX_ERR_NOT_SUPPORTED for it, which prte_ras_base_modify() reads
as "ask the next module" — and ras/hosts (priority 1, last) claims
NEW/EXTEND/RELEASE on the directive alone, regardless of whether it
found any hosts. So under Slurm a PMIX_ALLOC_NEW naming hostnames is served
by ras/hosts, and the named nodes are added to the DVM without the scheduler
ever being asked:

>>> PHASE 1: allocation request returned PMIX_SUCCESS
>>> ALLOC_ID prte-node1-282@1.1        <-- we minted an allocation id
squeue:  3 node[1-3]                   <-- Slurm allocated nothing

PRRTE then tried to start a prted on the un-allocated node, failed, and took
the DVM down. Dropping ras/hosts from the module set gives the right answer
(PMIX_ERR_NOT_SUPPORTED, DVM survives), which isolates the fall-through as
the cause. That half is being addressed separately, but a ras/slurm that
handled NEW would also close it.

[Petter-Programs]

While I agree that PMIX_ALLOC_NEW should be supported, I would point out these two things:

• To the best of my knowledge, support for --dependency=expand: was silently dropped by newer Slurm releases (if in fact it ever worked in most production environments); so building support for it at this stage seems redundant. However, I would argue that the difference is largely cosmetic; from our POV, it doesn't really matter whether Slurm thinks of the allocations as actually belonging together, or not, since we bridge that gap with the DVM.

• The implementation in ras/slurm is intended to integrate the new resources into the existing ones. They are intended to live together (you are right to say that the time limit should be aligned with the original job), die together, and seamlessly shrink together (for example, PRRTE understands requests to remove nodes by count across multiple allocations like: alloc1[abc,def],alloc2[geh,ijk] --> shrink by 3 --> alloc1[abc])

If you feel like classifying the current PMIX_ALLOC_EXTEND as PMIX_ALLOC_NEW is more appropriate given the underlying disjointness of the allocations, then that is a fair change; from my perspective, the important part is that the current functionality remains supported.

Notes:

• I would also verify that extending an allocation in time is something actually supported in realistic settings without elevated privileges. I'm inclined to say that it isn't Edit: realized you already discussed this point
• I am aware that in newer releases of Slurm, more dynamic adjustment requests are supported: but they tend to only be supported when reducing a resource.

[Petter-Programs]

Interesting findings by Claude pasted below. This does strengthen the argument for supporting it:

--dependency=expand:<jobid> is not removed — it is disabled by default and undocumented.

Verified against 24.05.8 (SchedMD 23139d3cb6); line numbers are from that tag.

The code is fully intact

Piece	Location
Token parse	src/slurmctld/job_scheduler.c:3961 — !xstrncasecmp(tok, "expand:", 7)
Enum	src/common/job_record.h:500 — SLURM_DEPEND_EXPAND
Validation + TRES/GRES inheritance	src/slurmctld/job_scheduler.c:3532, :3562, :3619
Execution (merge + kill donor)	src/slurmctld/job_mgr.c:13532 — select_g_job_expand() + _merge_job_licenses()
Plugin implementations	select/cons_tres:443, select/linear:2401 — both are real, non-stub
Env fixup script	src/scontrol/update_job.c:1235 — still writes slurm_job_<id>_resize.{sh,csh}

The gate

permit_job_expansion() (src/slurmctld/job_mgr.c:11413) is an xstrcasestr over
SchedulerParameters, defaulting to false. With it unset the parser returns
ESLURM_NOT_SUPPORTED before a dependency is ever built, so submission fails with
"Requested operation not supported on this system". The same gate blocks growth via
scontrol update JobId=N NumNodes=<larger> at job_mgr.c:12291.

Enable with:

SchedulerParameters=...,permit_job_expansion

Flipped to off-by-default in 19.05.0pre2. From NEWS:

-- Disable running job expansion by default. A new SchedulerParameter of
   permit_job_expansion has been added for sites that wish to re-enable it.

What actually disappeared is the documentation

• expand is absent from the --dependency type lists in sbatch.1, srun.1,
  salloc.1 and scontrol.1. Only after, afterany, afterburstbuffer,
  aftercorr, afternotok, afterok, singleton are listed.
• permit_job_expansion appears nowhere in slurm.conf.5.
• FAQ #job_size now opens "Slurm supports the ability to decrease the size of
  jobs" and documents only the shrink flow. The only surviving mention of the
  option anywhere in the tree is a 2011-era NEWS line.

So the feature is unadvertised rather than deleted — which is why it reads as gone.

Constraints still enforced (job_scheduler.c:3532)

Exactly one expand dependency per job; target must be RUNNING; same QOS and same
partition as the target; neither job suspended; the expanding job must have no active
steps; federated/remote targets rejected. The donor job is killed and its resources
fold into the target.

Related removals (vs. 17.02)

• select_p_job_expand_allow() is gone — the per-plugin veto was replaced by the
  single SchedulerParameters gate.
• slurmctld/nonstop is gone. It was the only in-tree component that generated
  expand dependencies itself (xstrfmtcat(job_alloc_req.dependency, "expand:%u", ...)).
  Nothing in 24.05 creates one automatically; it is submit-side only now.

Caveat: this is a 24.05.8 audit. Worth a re-check on current head before asserting
the same holds for 25.x.

[rhc54]

I think it's largely a question of perspective. From a Slurm perspective, your "extend" looks like "new" - but from the PRRTE and user's perspective, we are treating it like "extend" because we hide the disjoint nature of the two allocations.

This only came up in the course of investigating an issue you filed - when we tested with a "new" allocation request, we fell through and unexpectedly added hosts as if we were in an unmanaged environment. We fixed that bug, but it left open the question of why ras/slurm wasn't servicing a "new" request.

Given how you are using it, I'd suggest to have the component treat a "new" alloc directive as a synonym for "extend". As you note, there is no way to really "extend" a Slurm allocation, but PRRTE hides that distinction for the user - creating the synonym completes that process.

Up to you - just a suggestion. 🤷‍♂️

[Petter-Programs]

I think the synonym approach makes sense. I'm working on fixing that and the time limit adjustment.