Scheduler stops cycling while eligible nodes remain idle

[jamesETsmith]

SPUR 0.9.1 scheduler appears stalled despite idle eligible nodes

Summary

The SPUR controller responds to queries and its server clock advances, but the scheduler cycle counter does not advance. At the same time, 148 jobs are pending, 51 nodes are idle, and 50 idle nodes have eight MI355X GPUs. Multiple eligible jobs remain pending with Reason=None.

Environment

• Controller: controller1
• SPUR version: 0.9.1
• Partition: <gpu-partition>
• Observation date: 2026-08-30 UTC

All cluster-specific identifiers in this report have been replaced with descriptive placeholders. Resource types, resource counts, scheduler metrics, states, and relative timing are unchanged.

Direct scheduler evidence

Snapshots over 38 seconds showed an advancing server clock but an unchanged scheduler cycle count:

UTC T+00s
  Version           : 0.9.1
  Server Time       : T+00s
  PENDING           : 148
  RUNNING           : 115
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0

UTC T+11s
  Server Time       : T+11s
  PENDING           : 148
  RUNNING           : 115
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0

UTC T+21s
  Server Time       : T+21s
  PENDING           : 148
  RUNNING           : 115
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0

UTC T+38s
  Server Time       : T+38s
  PENDING           : 148
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0

The unchanged Cycles=206540 over at least 38 seconds is strong evidence that scheduling cycles were not running, while the advancing Server Time shows that the controller itself was responsive.

Controlled submission test

A minimal one-CPU, one-node batch job was submitted between scheduler snapshots:

Pre-submission, T-10s
  Server Time       : T-10s
  PENDING           : 149
  RUNNING           : 115
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0

Submitted batch job <probe-job> at T+00s

Post-submission, T+06s
  Server Time       : T+06s
  PENDING           : 150
  RUNNING           : 115
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0
  <probe-job>|scheduler-cycle-probe|PD|None|T+00s|N/A|

Post-submission, T+11s
  Server Time       : T+11s
  Cycles              : 206540
  Jobs started (last) : 0
  <probe-job>|scheduler-cycle-probe|PD|None|T+00s|N/A|

Post-submission, T+26s
  Server Time       : T+26s
  Cycles              : 206540
  Jobs started (last) : 0
  <probe-job>|scheduler-cycle-probe|PD|None|T+00s|N/A|

Post-submission, T+56s
  Server Time       : T+56s
  PENDING           : 150
  RUNNING           : 115
  IDLE              : 51
  Cycles              : 206540
  Jobs started (last) : 0
  <probe-job>|scheduler-cycle-probe|PD|None|T+00s|N/A|

The submission was accepted and increased the pending count from 149 to 150, proving that controller state was updating. However, Cycles remained exactly 206540 from 10 seconds before submission through 56 seconds after submission. The probe remained pending with Reason=None despite 51 idle nodes. It was then manually cancelled to avoid leaving a stale diagnostic job.

Probe request:

#!/bin/bash
#SBATCH --job-name=scheduler-cycle-probe
#SBATCH --account=<account>
#SBATCH --qos=<qos>
#SBATCH --partition=<gpu-partition>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:01:00

hostname -f

Idle capacity

UTC <observation-time>
PARTITION|AVAIL|TIMELIMIT|NODES|STATE
<gpu-partition>*|up|1-00:00:00|111|alloc
<gpu-partition>*|up|1-00:00:00|7|down
<gpu-partition>*|up|1-00:00:00|9|drain
<gpu-partition>*|up|1-00:00:00|51|idle
<gpu-partition>*|up|1-00:00:00|4|mix

Eligible full idle nodes: 50

Example eligible nodes:

NodeName=node1
   State=IDLE Reason=
   Partitions=<gpu-partition>
   CPUTot=236 CPUAlloc=0
   Gres=gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1

NodeName=node2
   State=IDLE Reason=
   Partitions=<gpu-partition>
   CPUTot=236 CPUAlloc=0
   Gres=gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1,gpu:mi355x:1

Affected jobs

Two interactive allocations each requested one node, 32 CPUs, and eight MI355X GPUs. Neither received a start time or scheduling reason. Both were cancelled after exactly five minutes while still unstarted:

JobId=<interactive-job-1> JobName=interactive
   UserId=<user> Account=<account>
   Partition=<gpu-partition> QOS=<qos>
   JobState=CANCELLED Reason=None
   NumNodes=1 NumTasks=1 CPUs/Task=32
   TresPerNode=gpu:mi355x:8/node ReqGPUs=8
   SubmitTime=T+00s StartTime=N/A EndTime=T+05m01s
   ExitCode=-1:0

JobId=<interactive-job-2> JobName=interactive
   UserId=<user> Account=<account>
   Partition=<gpu-partition> QOS=<qos>
   JobState=CANCELLED Reason=None
   NumNodes=1 NumTasks=1 CPUs/Task=32
   TresPerNode=gpu:mi355x:8/node ReqGPUs=8
   SubmitTime=T+06m02s StartTime=N/A EndTime=T+11m02s
   ExitCode=-1:0

Other eligible eight-GPU jobs across several accounts and QOSes were also pending with Reason=None. This indicates the symptom was cluster-wide rather than isolated to one user or QOS.

QOS is not the limiting factor

<qos> ... GrpTRES=node=64
QOS running node count: 27

The affected QOS was using 27 of its 64-node group limit. The affected jobs reported Reason=None, not QOSGrpNodeLimit.

Expected behavior

The backfill scheduler should continue incrementing its cycle count and assign eligible pending jobs to idle matching nodes, or provide a concrete pending reason.

Actual behavior

The controller remained responsive, but Cycles stayed fixed at 206540, no jobs started, and eligible jobs remained pending with Reason=None despite 50 matching idle eight-GPU nodes.

Commands used

spur diag
sinfo -p <gpu-partition> -o '%P|%a|%l|%D|%t|%G'
sinfo -N -p <gpu-partition> -t idle -o '%N|%t|%G'
spur show node node1
spur show node node2
spur show job <interactive-job-1>
spur show job <interactive-job-2>
squeue -t PD -o '%i|%u|%a|%q|%t|%r|%V|%b|%C|%D'
spur accounts show qos