Right-size -opt runners and consolidate node fleets (#889)

**Impact:** CI only — arc-runners-opt / nodepools-opt pilot
(meta-prod-aws-ue1)
**Risk:** medium

## TLDR
Should drop total vCPU utilization * hours by 9.8% (tied directly to
cost) and improve utilization by 5.2% for the simulated period

## What
Trims per-runner vCPU/memory in the `-opt` runner defs to fit within
actual K8s-allocatable capacity, and reshapes the `-opt` Karpenter node
fleets: GPU families (g4dn, g5, g6) collapse from GPU-count-locked
sub-nodepools into single unified fleets, and CPU/memory fleets
consolidate onto smaller ideal instance sizes (c7a→4xl, r7a/r7i→12xl,
m7g/m8g→8xl) with fallback-up chains.

## Why
The previous runner sizings claimed more vCPU/memory than the nodes
actually make schedulable after system overhead, and the fleet layout
had proliferated into many narrow sub-nodepools. Right-sizing the
requests and consolidating fleets improves node packing and reduces the
number of distinct nodepools Karpenter has to manage, while keeping
fallback to larger sizes when the ideal instance is unavailable.

# Notes
- GPU runner defs drop the explicit `node_fleet` field; they now derive
the fleet from the instance family (`g5.8xlarge` → `g5`), so scheduling
relies on `nvidia.com/gpu` requests rather than fleet-level GPU-count
isolation. See `fleet_naming.derive_fleet_name`.
- `instance_specs.py` gains `c7a.4xlarge` (specs + ENI max-pods) to back
the new `c7a-4xl` fleet.
- Scope is the `-opt` pilot only; baseline runner/nodepool defs are
untouched.

script on: #870

```
  Nodepool fleets

  ┌───────────────┬──────────────────────────┬───────────────────────────────────────────────────┬───────────────────────────────────────┐
  │    Family     │     Base (monolith)      │                Opt (sub-nodepools)                │              Net change               │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ c7a           │ c7a → ideal c7a.48xlarge │ c7a-2xl, c7a-4xl, c7a-12xl (+c7a-large unchanged) │ 1 → 3 size pools                      │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ c7i           │ c7i → ideal c7i.48xlarge │ c7i-12xl (+c7i-large unchanged)                   │ 1 → 1 sub-pool                        │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ m7g           │ m7g → ideal m7g.metal    │ m7g-8xl (+m7g-metal unchanged)                    │ ideal off baremetal → 8xl             │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ m7i           │ m7i → ideal m7i.48xlarge │ m7i-12xl                                          │ 1 → 1 sub-pool                        │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ m8g           │ m8g monolith+release     │ m8g-8xl + m8g (now release-only)                  │ packing → 8xl; shell kept for release │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ r7a           │ r7a monolith+release     │ r7a-12xl, r7a-16xl + r7a (release-only)           │ 1 → 2 pools; shell for release        │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ r7i           │ r7i → ideal r7i.48xlarge │ r7i-12xl, r7i-16xl                                │ 1 → 2 size pools                      │
  ├───────────────┼──────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────┤
  │ c7i-runner ⚠️ │ Intel c7i (amd64)        │ Graviton m7g/m8g (arm64)                          │ control-plane pool re-pointed         │
  └───────────────┴──────────────────────────┴───────────────────────────────────────────────────┴───────────────────────────────────────┘

  Runner defs (21 changed)

  ┌─────────────────────┬──────────────────────────┬─────────────────────────┬────────────────────┐
  │       Runner        │ Base (inst / vcpu / mem) │ Opt (inst / vcpu / mem) │       Change       │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-2-4    │ c7a.48xl / 2 / 4Gi       │ c7a.2xl / 2 / 4Gi       │ node→2xl           │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-8-16   │ c7a.48xl / 8 / 16Gi      │ c7a.4xl / 8 / 16Gi      │ node→4xl           │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-16-32  │ c7a.48xl / 16 / 32Gi     │ c7a.12xl / 16 / 32Gi    │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-37-68  │ c7a.48xl / 37 / 68Gi     │ c7a.12xl / 37 / 68Gi    │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-46-85  │ c7a.48xl / 46 / 85Gi     │ c7a.12xl / 46 / 83Gi    │ node→12xl; −2Gi    │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-8-16      │ c7i.12xl / 8 / 16Gi      │ c7i.12xl / 8 / 16Gi     │ fleet only         │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-14-27     │ c7i.12xl / 14 / 27Gi     │ c7i.12xl / 14 / 27Gi    │ fleet only         │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-22-41     │ c7i.12xl / 22 / 41Gi     │ c7i.12xl / 22 / 41Gi    │ fleet only         │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-46-84     │ c7i.12xl / 46 / 84Gi     │ c7i.12xl / 46 / 83Gi    │ fleet only; −1Gi   │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-arm64g3-16-62     │ m7g.8xl / 16 / 62Gi      │ m7g.8xl / 15 / 56Gi     │ −1 vcpu, −6Gi      │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-8-32      │ m7i.48xl / 8 / 32Gi      │ m7i.12xl / 8 / 32Gi     │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx2-8-32     │ m7i.48xl / 8 / 32Gi      │ m7i.12xl / 8 / 32Gi     │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-32-128    │ m7i.48xl / 32 / 128Gi    │ m7i.12xl / 32 / 128Gi   │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx2-40-160   │ m7i.48xl / 40 / 160Gi    │ m7i.12xl / 40 / 160Gi   │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-arm64g4-16-62     │ m8g.48xl / 16 / 62Gi     │ m8g.8xl / 15 / 56Gi     │ node→8xl; trimmed  │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-8-64   │ r7a.48xl / 8 / 64Gi      │ r7a.16xl / 8 / 64Gi     │ node→16xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-16-128 │ r7a.48xl / 16 / 128Gi    │ r7a.12xl / 15 / 116Gi   │ node→12xl; trimmed │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-32-256 │ r7a.48xl / 32 / 256Gi    │ r7a.12xl / 32 / 256Gi   │ node→12xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iavx512-48-384 │ r7a.48xl / 48 / 384Gi    │ r7a.12xl / 46 / 350Gi   │ node→12xl; trimmed │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-8-64      │ r7i.48xl / 8 / 64Gi      │ r7i.16xl / 8 / 64Gi     │ node→16xl          │
  ├─────────────────────┼──────────────────────────┼─────────────────────────┼────────────────────┤
  │ l-x86iamx-16-128    │ r7i.48xl / 16 / 128Gi    │ r7i.12xl / 15 / 116Gi   │ node→12xl; trimmed │
  └─────────────────────┴──────────────────────────┴─────────────────────────┴────────────────────┘
```

```
 $  rm -rf scripts/node-size-sweep/output ; uv run scripts/node-size-sweep/optimize_search.py \
                        --last-days 35 --drop-provider lf --keep-fraction 0.5 \
                        --num-workers $(sysctl -n hw.ncpu) --num-restarts 20 \
                        --search-mode auto
2026-07-09 00:29:43,745 [global] INFO output_dir=/Users/jschmidt/meta/ci-infra/osdc/scripts/node-size-sweep/output/20260709T072943Z-f1e1abe
2026-07-09 00:29:43,781 [global] INFO hashing sim source files
2026-07-09 00:29:44,645 [global] INFO hashing CSV
2026-07-09 00:29:44,732 [global] INFO loading CSV /Users/jschmidt/meta/ci-infra/osdc/scripts/node-size-sweep/pytorch_60d.csv (last_days=35)
  dropped by downsample:      1,316,846
  filtered to last 35 days: 1,795,132 jobs kept, 833,236 dropped
2026-07-09 00:29:49,272 [global] INFO loaded 1795132 jobs
2026-07-09 00:29:51,037 [global] INFO sim_flags prod-parity: daemonsets_in_metric=True phantom_pods_enabled=True empty_ttl_buckets=1
2026-07-09 00:29:51,042 [global] INFO dispatching 8 families across 16 workers
[07/09/26 00:30:11] INFO     global heartbeat: 1/8 families done, elapsed=21s, eta=144s
[07/09/26 00:30:14] INFO     global heartbeat: 2/8 families done, elapsed=23s, eta=69s
[07/09/26 00:30:29] INFO     global heartbeat: 3/8 families done, elapsed=38s, eta=63s
[07/09/26 00:30:43] INFO     global heartbeat: 4/8 families done, elapsed=53s, eta=53s
[07/09/26 00:33:45] INFO     global heartbeat: 5/8 families done, elapsed=235s, eta=141s
[07/09/26 00:38:24] INFO     global heartbeat: 6/8 families done, elapsed=513s, eta=171s
[07/09/26 00:40:22] INFO     global heartbeat: 7/8 families done, elapsed=631s, eta=90s
[07/09/26 10:49:52] INFO     global heartbeat: 8/8 families done, elapsed=12190s, eta=0s
c7a     DONE     18,480 vcpu-min (baseline 95,040 vcpu-min, +80.6% reduction) peak 8 util 63.5%  [improved]
c7i     DONE     11,549,280 vcpu-min (baseline 16,223,040 vcpu-min, +28.8% reduction) peak 53 util 49.1%  [improved]
m6i     DONE     1,431,040 vcpu-min (baseline 1,431,040 vcpu-min, +0.0% reduction) peak 23 util 84.5%  [no_change]
m7g     DONE     26,248,320 vcpu-min (baseline 36,723,520 vcpu-min, +28.5% reduction) peak 76 util 82.4%  [improved]
m7i     DONE     15,069,120 vcpu-min (baseline 19,300,800 vcpu-min, +21.9% reduction) peak 115 util 52.9%  [improved]
m8g     DONE     31,373,920 vcpu-min (baseline 43,333,440 vcpu-min, +27.6% reduction) peak 81 util 82.2%  [improved]
r7a     DONE     378,480,400 vcpu-min (baseline 445,836,480 vcpu-min, +15.1% reduction) peak 503 util 82.4%  [improved]
c7a     DONE     18,480 vcpu-min (baseline 95,040 vcpu-min, +80.6% reduction) peak 8 util 63.5%  [improved]
c7i     DONE     11,549,280 vcpu-min (baseline 16,223,040 vcpu-min, +28.8% reduction) peak 53 util 49.1%  [improved]
m6i     DONE     1,431,040 vcpu-min (baseline 1,431,040 vcpu-min, +0.0% reduction) peak 23 util 84.5%  [no_change]
m7g     DONE     26,248,320 vcpu-min (baseline 36,723,520 vcpu-min, +28.5% reduction) peak 76 util 82.4%  [improved]
m7i     DONE     15,069,120 vcpu-min (baseline 19,300,800 vcpu-min, +21.9% reduction) peak 115 util 52.9%  [improved]
m8g     DONE     31,373,920 vcpu-min (baseline 43,333,440 vcpu-min, +27.6% reduction) peak 81 util 82.2%  [improved]
r7a     DONE     378,480,400 vcpu-min (baseline 445,836,480 vcpu-min, +15.1% reduction) peak 503 util 82.4%  [improved]
r7i     DONE     32,190,080 vcpu-min (baseline 37,583,040 vcpu-min, +14.3% reduction) peak 32 util 72.2%  [improved]
2026-07-09 10:49:52,423 [global] INFO cluster validation: loading full dataset (no --last-days filter) from /Users/jschmidt/meta/ci-infra/osdc/scripts/node-size-sweep/pytorch_60d.csv
  dropped by downsample:      1,316,846
2026-07-09 10:49:56,762 [global] INFO cluster validation: loaded 2628368 jobs (vs 35 in search window)
2026-07-09 10:49:57,552 [global] INFO cluster validation: dispatching 2 sims (baseline + recommendation) across 2 workers
2026-07-09 10:50:02,059 [global] INFO cluster sim baseline: starting (2628368 jobs, 0 extra fleets)
2026-07-09 10:50:05,425 [global] INFO cluster sim recommendation: starting (2628368 jobs, 11 extra fleets)
2026-07-09 10:50:32,159 [global] INFO cluster sim baseline: done vcpu_min=1644986400 peak_nodes=350 util=0.7039 vcpu_hours=27336410 node_hours=164121 elapsed=30.1s
2026-07-09 10:50:41,248 [global] INFO cluster sim recommendation: done vcpu_min=1483283960 peak_nodes=807 util=0.7559 vcpu_hours=24636708 node_hours=326974 elapsed=35.8s
2026-07-09 10:50:41,487 [global] INFO cluster validation: baseline vcpu_min=1644986400 rec vcpu_min=1483283960 delta=+9.83pp (baseline util=70.4% rec util=75.6%) elapsed=43.9s
2026-07-09 10:50:41,618 [global] INFO Phase 2.5: runner-fleet host search (arch=('amd64', 'arm64'))
2026-07-09 10:50:42,939 [global] INFO Phase 2.5: best amd64=c7i.8xlarge (-7.0% vs baseline); best arm64=m7g.8xlarge
2026-07-09 10:50:43,052 [global] INFO all done. Reports in /Users/jschmidt/meta/ci-infra/osdc/scripts/node-size-sweep/output/20260709T072943Z-f1e1abe/reports
```

---------

Signed-off-by: Jean Schmidt <contato@jschmidt.me>

Author: jeanschmidt

osdc/modules/arc-runners-opt/defs/l-arm64g3-16-62.yaml

@@ -1,12 +1,12 @@
 # ARC runner definition: l-arm64g3-16-62
-# Instance type: m7g.16xlarge (pool: m7g-16xl)
+# Instance type: m7g.8xlarge (pool: m7g-8xl)
 runner:
   name: l-arm64g3-16-62
-  instance_type: m7g.16xlarge
-  node_fleet: m7g-16xl
+  instance_type: m7g.8xlarge
+  node_fleet: m7g-8xl
   disk_size: 256
-  vcpu: 20
-  memory: 75Gi
+  vcpu: 15
+  memory: 56Gi
   gpu: 0
   proactive_capacity: 0
   hud_failure_base_capacity: 90

osdc/modules/arc-runners-opt/defs/l-arm64g4-16-62.yaml

@@ -1,12 +1,12 @@
 # ARC runner definition: l-arm64g4-16-62
-# Instance type: m8g.16xlarge (pool: m8g-16xl)
+# Instance type: m8g.8xlarge (pool: m8g-8xl)
 runner:
   name: l-arm64g4-16-62
-  instance_type: m8g.16xlarge
-  node_fleet: m8g-16xl
+  instance_type: m8g.8xlarge
+  node_fleet: m8g-8xl
   disk_size: 256
-  vcpu: 20
-  memory: 75Gi
+  vcpu: 15
+  memory: 56Gi
   gpu: 0
   proactive_capacity: 0
   hud_failure_base_capacity: 90

osdc/modules/arc-runners-opt/defs/l-x86aavx2-29-113-a10g.yaml

@@ -4,9 +4,8 @@
 runner:
   name: l-x86aavx2-29-113-a10g
   instance_type: g5.8xlarge
-  node_fleet: g5-8xl
   disk_size: 150
-  vcpu: 31
+  vcpu: 29
   memory: 113Gi
   gpu: 1
   proactive_capacity: 0

osdc/modules/arc-runners-opt/defs/l-x86aavx2-29-113-l4.yaml

@@ -4,9 +4,8 @@
 runner:
   name: l-x86aavx2-29-113-l4
   instance_type: g6.8xlarge
-  node_fleet: g6-8xl
   disk_size: 150
-  vcpu: 31
+  vcpu: 29
   memory: 113Gi
   gpu: 1
   proactive_capacity: 0

osdc/modules/arc-runners-opt/defs/l-x86aavx2-45-167-a10g-4.yaml

@@ -4,9 +4,8 @@
 runner:
   name: l-x86aavx2-45-167-a10g-4
   instance_type: g5.12xlarge
-  node_fleet: g5-12xl
   disk_size: 150
-  vcpu: 47
+  vcpu: 45
   memory: 165Gi
   gpu: 4
   proactive_capacity: 0

osdc/modules/arc-runners-opt/defs/l-x86aavx2-45-172-l4-4.yaml

@@ -1,13 +1,12 @@
 # ARC runner definition: l-x86aavx2-45-172-l4-4
 # Instance type: g6.12xlarge — 4-GPU L4, 48c/192Gi node
-# Actual K8s capacity ~177.6Gi. Job 171Gi after overhead incl. hf-cache 2Gi reserve; label keeps -172-.
+# Actual K8s capacity ~177.6Gi. Job 170Gi after overhead incl. hf-cache 2Gi reserve; label keeps -172-.
 runner:
   name: l-x86aavx2-45-172-l4-4
   instance_type: g6.12xlarge
-  node_fleet: g6-12xl
   disk_size: 150
-  vcpu: 47
-  memory: 171Gi
+  vcpu: 45
+  memory: 170Gi
   gpu: 4
   proactive_capacity: 0
   hud_failure_base_capacity: 15

osdc/modules/arc-runners-opt/defs/l-x86iamx-14-27.yaml

@@ -5,7 +5,7 @@ runner:
   instance_type: c7i.12xlarge
   node_fleet: c7i-12xl
   disk_size: 150
-  vcpu: 15
+  vcpu: 14
   memory: 27Gi
   gpu: 0
   proactive_capacity: 0

osdc/modules/arc-runners-opt/defs/l-x86iamx-16-128.yaml

@@ -1,12 +1,12 @@
 # ARC runner definition: l-x86iamx-16-128
-# Instance type: r7i.16xlarge (pool: r7i-16xl)
+# Instance type: r7i.12xlarge (pool: r7i-12xl)
 runner:
   name: l-x86iamx-16-128
-  instance_type: r7i.16xlarge
-  node_fleet: r7i-16xl
+  instance_type: r7i.12xlarge
+  node_fleet: r7i-12xl
   disk_size: 150
-  vcpu: 20
-  memory: 154Gi
+  vcpu: 15
+  memory: 116Gi
   gpu: 0
   proactive_capacity: 0
   hud_failure_base_capacity: 90

osdc/modules/arc-runners-opt/defs/l-x86iamx-22-41.yaml

@@ -5,8 +5,8 @@ runner:
   instance_type: c7i.12xlarge
   node_fleet: c7i-12xl
   disk_size: 200
-  vcpu: 23
-  memory: 42Gi
+  vcpu: 22
+  memory: 41Gi
   gpu: 0
   proactive_capacity: 0
   hud_failure_base_capacity: 30

osdc/modules/arc-runners-opt/defs/l-x86iamx-32-128.yaml

@@ -1,12 +1,12 @@
 # ARC runner definition: l-x86iamx-32-128
-# Instance type: m7i.48xlarge (pool: m7i-48xl)
+# Instance type: m7i.12xlarge (pool: m7i-12xl)
 runner:
   name: l-x86iamx-32-128
-  instance_type: m7i.48xlarge
-  node_fleet: m7i-48xl
+  instance_type: m7i.12xlarge
+  node_fleet: m7i-12xl
   disk_size: 200
-  vcpu: 37
-  memory: 139Gi
+  vcpu: 32
+  memory: 128Gi
   gpu: 0
   proactive_capacity: 0
   hud_failure_base_capacity: 30