Halve proactive capacity, add HUD failure floor (#721)

jeanschmidt · web-flow · commit e0565535a509 · 2026-06-10T20:34:52.000Z
- Cut proactive_capacity by 50% across all 40 runner defs to reduce
steady-state idle placeholder cost
- Add hud_failure_base_capacity (3x the old proactive value) so HUD API
outages still surge to the previous warm-pool level
- Hardcode CAPACITY_AWARE_HUD_FAILURE_MULTIPLIER=1 in the template (was
using code default of 3)
- Update arc-fork-build-deploy.md to reflect the new multiplier default

The old formula on HUD failure was: proactive * 3 (multiplier default).
With multiplier pinned to 1 and the new base_capacity set to 3x the old
proactive value, HUD-failure behavior is preserved while normal
steady-state capacity is halved, cutting idle node spend.

Signed-off-by: Jean Schmidt &lt;contato@jschmidt.me&gt;
diff --git a/osdc/docs/arc-fork-build-deploy.md b/osdc/docs/arc-fork-build-deploy.md
@@ -176,8 +176,8 @@ The capacity monitor is configured via env vars on the listener pod, set in `mod
 | `CAPACITY_AWARE_RUNNER_MEMORY` | `512Mi` | `1Gi` | Runner placeholder memory request — **must match the runner container's actual memory request/limit**. The template value (`1Gi`) is the operative value; the code default (`512Mi`) would under-reserve and defeat the topology guarantee. |
 | `CAPACITY_AWARE_NODE_FLEET` | _(empty)_ | `{{NODE_FLEET}}` (from runner def) | Node fleet for placeholder-workflow scheduling (workflow pool, per-scale-set) |
 | `CAPACITY_AWARE_RUNNER_NODE_FLEET` | _(empty)_ | `c7i-runner` | **Required when `CAPACITY_AWARE_ENABLED=true`** — the cluster-wide runner-pool fleet for placeholder-runner scheduling (currently `c7i-runner`). `Validate()` returns an error if missing. Distinct from `CAPACITY_AWARE_NODE_FLEET` above: runner placeholders always land on the cluster-wide runner pool, while workflow placeholders land on the per-scale-set workflow pool. |
-| `CAPACITY_AWARE_HUD_FAILURE_MULTIPLIER` | `3` | _(unset — uses code default)_ | When the HUD API is unreachable, the capacity monitor over-provisions placeholders to `ProactiveCapacity * multiplier + HUDFailureBaseCapacity`. Outer caps (`MaxRunners` headroom, `MaxBurstCapacity`) bound the absolute blast radius. Clamped to a minimum of `1`. |
-| `CAPACITY_AWARE_HUD_FAILURE_BASE_CAPACITY` | `0` | `{{HUD_FAILURE_BASE_CAPACITY}}` (from runner def; defaults to `0` — no def currently sets a non-zero value) | Additive baseline added to the HUD-failure fallback formula: `desiredPairs = ProactiveCapacity * HUDFailureMultiplier + HUDFailureBaseCapacity`. Lets operators run with `proactive_capacity: 0` and still surge a fixed floor of pairs when the HUD API is unreachable. Clamped to `[0, 1000]`; warning logged above `100`. |
+| `CAPACITY_AWARE_HUD_FAILURE_MULTIPLIER` | `3` | `1` (hardcoded in `runner.yaml.tpl`) | When the HUD API is unreachable, the capacity monitor over-provisions placeholders to `ProactiveCapacity * multiplier + HUDFailureBaseCapacity`. Outer caps (`MaxRunners` headroom, `MaxBurstCapacity`) bound the absolute blast radius. Clamped to a minimum of `1`. |
+| `CAPACITY_AWARE_HUD_FAILURE_BASE_CAPACITY` | `0` | `{{HUD_FAILURE_BASE_CAPACITY}}` (from runner def; defaults to `0`) | Additive baseline added to the HUD-failure fallback formula: `desiredPairs = ProactiveCapacity * HUDFailureMultiplier + HUDFailureBaseCapacity`. Lets operators run with `proactive_capacity: 0` and still surge a fixed floor of pairs when the HUD API is unreachable. Clamped to `[0, 1000]`; warning logged above `100`. |
 | `CAPACITY_AWARE_RUNNER_CLASS` | _(empty)_ | `{{RUNNER_CLASS}}` (from runner def) | Runner class for placeholder node selector |
 | `CAPACITY_AWARE_HUD_API_URL` | _(built-in default URL)_ | hardcoded PyTorch HUD `queued_jobs_aggregate` URL | HUD endpoint for queued job counts |
 | `CAPACITY_AWARE_HUD_API_TOKEN` | _(empty)_ | from K8s secret `pytorch-hud-token` (optional mount) | PyTorch HUD API token for queued job counts |
diff --git a/osdc/modules/arc-runners/defs/l-arm64g2-6-25.yaml b/osdc/modules/arc-runners/defs/l-arm64g2-6-25.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 6
   memory: 25Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-arm64g3-16-62.yaml b/osdc/modules/arc-runners/defs/l-arm64g3-16-62.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 16
   memory: 62Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-arm64g3-61-463.yaml b/osdc/modules/arc-runners/defs/l-arm64g3-61-463.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 61
   memory: 463Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-arm64g4-16-62.yaml b/osdc/modules/arc-runners/defs/l-arm64g4-16-62.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 16
   memory: 62Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-barm64g3-62-226.yaml b/osdc/modules/arc-runners/defs/l-barm64g3-62-226.yaml
@@ -9,5 +9,6 @@ runner:
   vcpu: 62
   memory: 226Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-barm64g4-62-226.yaml b/osdc/modules/arc-runners/defs/l-barm64g4-62-226.yaml
@@ -9,5 +9,6 @@ runner:
   vcpu: 62
   memory: 226Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-bx86iamx-92-167.yaml b/osdc/modules/arc-runners/defs/l-bx86iamx-92-167.yaml
@@ -9,5 +9,6 @@ runner:
   vcpu: 92
   memory: 167Gi
   gpu: 0
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-bx86iavx512-94-344-t4-8.yaml b/osdc/modules/arc-runners/defs/l-bx86iavx512-94-344-t4-8.yaml
@@ -9,5 +9,6 @@ runner:
   vcpu: 94
   memory: 344Gi
   gpu: 8
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86aavx2-189-704-a10g-8.yaml b/osdc/modules/arc-runners/defs/l-x86aavx2-189-704-a10g-8.yaml
@@ -9,5 +9,6 @@ runner:
   vcpu: 189
   memory: 704Gi
   gpu: 8
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86aavx2-29-113-a10g.yaml b/osdc/modules/arc-runners/defs/l-x86aavx2-29-113-a10g.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 29
   memory: 113Gi
   gpu: 1
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 60
diff --git a/osdc/modules/arc-runners/defs/l-x86aavx2-29-113-l4.yaml b/osdc/modules/arc-runners/defs/l-x86aavx2-29-113-l4.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 29
   memory: 113Gi
   gpu: 1
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 60
diff --git a/osdc/modules/arc-runners/defs/l-x86aavx2-45-167-a10g-4.yaml b/osdc/modules/arc-runners/defs/l-x86aavx2-45-167-a10g-4.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 45
   memory: 167Gi
   gpu: 4
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86aavx2-45-172-l4-4.yaml b/osdc/modules/arc-runners/defs/l-x86aavx2-45-172-l4-4.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 45
   memory: 172Gi
   gpu: 4
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86aavx512-125-463.yaml b/osdc/modules/arc-runners/defs/l-x86aavx512-125-463.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 125
   memory: 463Gi
   gpu: 0
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-14-27.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-14-27.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 14
   memory: 27Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-16-128.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-16-128.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 16
   memory: 128Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-22-41.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-22-41.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 22
   memory: 41Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-32-128.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-32-128.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 32
   memory: 128Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-46-84.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-46-84.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 46
   memory: 84Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-8-16.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-8-16.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 8
   memory: 16Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-8-32.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-8-32.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 8
   memory: 32Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iamx-8-64.yaml b/osdc/modules/arc-runners/defs/l-x86iamx-8-64.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 8
   memory: 64Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx2-40-160.yaml b/osdc/modules/arc-runners/defs/l-x86iavx2-40-160.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 40
   memory: 160Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx2-8-32.yaml b/osdc/modules/arc-runners/defs/l-x86iavx2-8-32.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 8
   memory: 32Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-16-128.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-16-128.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 16
   memory: 128Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-16-32.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-16-32.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 16
   memory: 32Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-2-4.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-2-4.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 2
   memory: 4Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-29-115-t4.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-29-115-t4.yaml
@@ -10,5 +10,6 @@ runner:
   vcpu: 29
   memory: 115Gi
   gpu: 1
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 60
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-32-256.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-32-256.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 32
   memory: 256Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-37-68.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-37-68.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 37
   memory: 68Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-45-172-t4-4.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-45-172-t4-4.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 45
   memory: 172Gi
   gpu: 4
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-46-85.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-46-85.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 46
   memory: 85Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-48-384.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-48-384.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 48
   memory: 384Gi
   gpu: 0
-  proactive_capacity: 10
+  proactive_capacity: 5
+  hud_failure_base_capacity: 30
   max_burst_capacity: 250
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-8-16.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-8-16.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 8
   memory: 16Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-8-64.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-8-64.yaml
@@ -7,5 +7,6 @@ runner:
   vcpu: 8
   memory: 64Gi
   gpu: 0
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-94-192.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-94-192.yaml
@@ -9,5 +9,6 @@ runner:
   vcpu: 94
   memory: 189Gi
   gpu: 0
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/l-x86iavx512-94-768.yaml b/osdc/modules/arc-runners/defs/l-x86iavx512-94-768.yaml
@@ -8,5 +8,6 @@ runner:
   vcpu: 94
   memory: 740Gi
   gpu: 0
-  proactive_capacity: 5
+  proactive_capacity: 2
+  hud_failure_base_capacity: 15
   max_burst_capacity: 30
diff --git a/osdc/modules/arc-runners/defs/rel-l-arm64g4-16-62.yaml b/osdc/modules/arc-runners/defs/rel-l-arm64g4-16-62.yaml
@@ -10,5 +10,6 @@ runner:
   gpu: 0
   runner_group: release-runners
   runner_class: release
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/defs/rel-l-x86iavx512-8-64.yaml b/osdc/modules/arc-runners/defs/rel-l-x86iavx512-8-64.yaml
@@ -10,5 +10,6 @@ runner:
   gpu: 0
   runner_group: release-runners
   runner_class: release
-  proactive_capacity: 30
+  proactive_capacity: 15
+  hud_failure_base_capacity: 90
   max_burst_capacity: 2000
diff --git a/osdc/modules/arc-runners/templates/runner.yaml.tpl b/osdc/modules/arc-runners/templates/runner.yaml.tpl
@@ -163,6 +163,8 @@ listenerTemplate:
             value: "{{PROACTIVE_CAPACITY}}"
           - name: CAPACITY_AWARE_MAX_BURST_CAPACITY
             value: "{{MAX_BURST_CAPACITY}}"
+          - name: CAPACITY_AWARE_HUD_FAILURE_MULTIPLIER
+            value: "1"
           - name: CAPACITY_AWARE_HUD_FAILURE_BASE_CAPACITY
             value: "{{HUD_FAILURE_BASE_CAPACITY}}"
           - name: CAPACITY_AWARE_RECALCULATE_INTERVAL
