Adds a simulation + optimization script to find ideal node/runner sizes based on historical data#870
Draft
jeanschmidt wants to merge 18 commits into
Draft
Adds a simulation + optimization script to find ideal node/runner sizes based on historical data#870jeanschmidt wants to merge 18 commits into
jeanschmidt wants to merge 18 commits into
Conversation
Real 2D CPU+memory+GPU bin-packing simulator that replays HUD workflow_job data against a Karpenter-style cluster. Warming-node model, ARC placeholder emulation, DaemonSet + phantom-pod prod-parity toggles, and per-fleet / cluster-wide utilization reports.
- Add optimize.md: design for a per-fleet-family node/pod sizing optimizer targeting max(CPU, mem) allocatable-weighted utilization on HUD data - Define objective split: opt metric (workload-only over post-kubelet capacity) for ranking, cal metric matching prod PromQL for calibration - Specify shape catalog + eligibility (data-driven N, not power-of-2) and a checkpointed, resumable multi-restart hill-climb search - Document 8 design decisions (family-locked, virtual sub-fleets, GPU/ baremetal/reserved scope-out) and 11 risks with mitigations - Lay out phased plan (Phase 0 calibration -> catalog -> search -> sensitivity -> git-apply-able patch deliverables) and file layout Notes: design only, no executable code yet. Scope excludes p4d/p5/p6, -large/-metal variants, reserved-capacity fleets, and c7i-runner (fixed ARC pod shape). Coupling across families is treated as zero on the workload side, so each family is searched independently. Absolute util numbers stay ungrounded until the Phase 0 sim-vs-prod delta is measured; if it exceeds 5pp, recs are reframed as deltas vs baseline. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- add benchmark.py: phase 0 harness (single-run wall-clock, noise floor across seeds, sim-vs-prod calibration incl. per-fleet cal) - add optimize_catalog.py: phase 1 analytical shape catalog + per-def eligibility + theoretical util ceiling, with JSON dump - simulate.py: expose per-pool workload/ds/raw-alloc breakdowns so opt/cal metrics can be derived; snapshot per-bucket arrivals to keep simulate() safe to re-run in-process - instance_specs.py: fill in missing instance sizes (2xl/4xl and metal variants) across CPU/GPU families for catalog enumeration - lint/style cleanups across build_csv, pull_hud, sim_load, sim_nodes, sim_report (imports, S311 noqa, formatting) Notes: The optimizer separates two utilization numbers: opt (workload over allocatable+ds, the ranking metric) and cal (used/alloc from per_pool, matching the prod PromQL/Grafana dashboard). The benchmark harness measures both plus the seed noise floor so later phases know what signal-vs-noise threshold their changes must clear. Phase 1 is purely analytical (no sim runs) — it enumerates every (instance, N) split per fleet family, filters to shapes that fit each runner def, and reports the best fit and a uniform-weighted ceiling. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add optimize_search.py: per-family multi-restart hill-climb over (instance_type, N) shapes, ranked on max(opt_cpu, opt_mem) with node-hours tie-break; SQLite sim cache + resumable checkpoint state - Extract optimize_config.py as single source for scope constants, def_totals/load_defs_by_family, and PROD_PARITY_SIM_FLAGS; rewire optimize_catalog.py to it (dropping the duplicated definitions) - Add last_days job filtering to sim_load.load_jobs, wired through benchmark.py and simulate.py CLIs - Let ClusterModel take a fleets_override so the search can swap in virtual per-(family, instance) sub-fleets - Regenerate optimize_catalog.json Notes: The search runs families as independent spawn subprocesses (D3 independence), keeping c7i-runner injected so runner pods still schedule. The sim cache key pins hashes of every sim/loader/analyzer source file plus the discovered DaemonSet set, so any change that affects results invalidates stale entries. Empty-window families are skipped but a prior best under a wider window is preserved rather than overwritten with a "skipped" verdict. Phase 4 (real patch generation) is still stubbed. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add ProgressDisplay class rendering single-family live terminal progress via rich.live.Live, with a plain carriage-return fallback - Swap loggers' stderr handlers for RichHandler (or bump them to WARNING) so log records don't shred the progress line - Thread progress hooks through _search_family (baseline, restart, step, neighbor advance, best updates, end-of-family) - Add --no-progress opt-out; auto-disable for non-TTY stderr or multi-worker runs; add rich>=13 dependency Notes: Display only runs in the single-process path (workers == 1) because multiprocessing workers can't share a terminal cleanly and ANSI/CR output would corrupt non-TTY logs. All state mutations are lock-guarded so the rich background-refresh thread never races the search thread. When disabled, ProgressDisplay is a null object whose methods are safe no-ops. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add QueueProgressDisplay: worker-side ProgressDisplay stand-in that forwards state changes to the parent over a multiprocessing.Queue - Add MultiFamilyProgressDisplay: parent aggregator rendering one row per family in a rich.Live panel, with stalled-worker detection - Wire the queue into _family_worker and main() so parallel searches report progress instead of falling back to plain heartbeat logs - Bump worker stderr StreamHandlers to WARNING+ when queued so INFO chatter doesn't interleave into the parent panel; file logs stay full Previously live progress only worked with a single worker; multi-worker runs disabled the display entirely. The queue-fed aggregator lets each family stream restart/step/candidate/best-score updates to a shared panel. Non-TTY or missing-rich falls back to a no-op display while still suppressing INFO noise. Stalled rows surface worker crashes (segfault/OOM) instead of letting them vanish silently. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Reformulate the search from per-def (instance, N) shapes into a partition of a family's defs across sub-nodepools plus a per-subpool instance choice; pod cpu/mem are now derived deterministically via the D4 tight-fit rule instead of being a free search variable - Rewrite optimize_catalog.py as a (def, instance) eligibility oracle and drop the pre-generated optimize_catalog.json - Split optimize_search.py into optimize_engine (search + sim wrapper), optimize_progress (live display), optimize_report (reports + patches), and optimize_storage (SimCache + StateStore) - Add exhaustive/hillclimb/auto modes and a full-dataset validation phase; replace the node-hours tie-breaker with size-invariant vCPU-hours - Add runner_hooks.py as the single source of hooks/runner overhead shared by sim_load and optimize_config so catalog feasibility matches what the sim schedules The prior formulation let the optimizer arbitrarily upsize pod requests to whatever slot maximized utilization, which is not a deployable change — labels encode shape and requests are bounded by what a def needs. The new formulation makes pod adjustment a mechanical byproduct of the (def, instance) pair, bounded by D4 tolerances; any pair needing a larger adjustment is infeasible and pruned before any sim runs. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Replace per-family full-dataset validation with two full-cluster sims (baseline as-is + all improved families' best_configs merged in) - Add cluster-sim helpers to optimize_engine: apply_recommendations_to_jobs, build_cluster_fleets_extra, run_cluster_sim, and cluster/per-family metric extraction - Add ClusterModel.fleets_extra to steer named fleets onto specific instances while unchanged fleets keep their YAML shape - Add ClusterValidationResult and rework reports: cluster-wide validation table plus a per-family "cluster contribution" section - Extract cluster + per-family metrics inside each sim worker so the large sim_out never crosses the process boundary A single full-cluster sim captures cross-family packing and interaction that isolated per-family sims miss. Each family's share is derived by pool-filtering the same two sim outputs (baseline uses original nodepool names, rec uses best_config sub_nodepool_ids), so per-family contribution numbers come for free without running extra sims. The validation phase now runs exactly two sims (2-worker spawn pool) instead of two per family. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- D4 now adjusts only the main pod's integer vcpu/memory, treating the runner-container-hooks sidecar (320 mcpu / 522 MiB) as a fixed tax that rides on top; subtract sidecar before flooring to whole vCPU/GiB - Replace mcpu-based CPU tolerance with integer main-vcpu bounds [min(orig-1, ceil(orig*0.95)), max(orig+1, ceil(orig*1.35))] - def_totals returns main_vcpu/main_memory_gib; catalog, report, and patch emit adjustments in operator YAML units (vcpu:/memory:) with new_main_* - Baseline now uses prod-reality (unadjusted) pod shapes routed to real nodepool names; add is_baseline_feasible physical-fit gate and thread baseline_defs through rebuild_jobs_for_config/run_sim_for_config/cached_sim - Add generated optimize_catalog.json eligibility output Notes: The old baseline was gated against the recommendation catalog, which enforces D4 bounds — so a perfectly-fine prod config could be rejected when its tight-fit on the largest instance overshot the upper bound. The baseline must reflect what prod actually runs (original shapes, real weighted fleet from YAML), so it is now checked only for physical fit and cached separately from recommendation sims to avoid shape collisions. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Check baseline fit against every instance in the real fleet's weighted list, not just the single reference instance - Load real fleets via ClusterModel and thread them into is_baseline_feasible - Drop GPU sign-parity gate; physical fit on any fleet member is the only invariant - Pick the reference instance as the largest prod choice by vCPU from INSTANCE_SPECS, keeping spec["instance"] display-only Baseline routing mirrors prod: pods go to the real nodepool and Karpenter picks a fitting instance per pod at scheduling time. Gating feasibility on a single family-largest instance wrongly skipped families whose pods fit some (but not the biggest) fleet member, and enforcing alloc_gpu == 0 for rare 0-GPU defs in a GPU family's GPU-only fleet was never satisfiable. The correct check is "does the original pod shape fit on at least one fleet member?". Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- add optimize_pricing: static on-demand price table (per-region + blended, reserved GPU families surface as unpriced) joined to instance specs - add optimize_cost: node-hours and USD post-processing of sim node-count buckets, with relative savings and family/cluster cost report blocks - add Phase 2.5 optimize_runner_fleet + report: closed-form cheapest instance to host the fixed 750m/1Gi ARC runner pods, dual amd64/arm64 winners - thread cost through the engine (cost_for_config), search (--skip-runner-fleet, --runner-fleet-arch), cluster validation, and family/global reports - emit per-type node_counts_by_type from simulate; label vcpu_hours a compute proxy vs authoritative node-hours x price - add unit tests for cost, pricing, engine, report, runner_fleet, search, storage, and simulate; scope coverage omit to untested sweep scripts Notes: Absolute dollars are approximate (on-demand list price, sim node-hours are a lower bound); relative percentage/ratio deltas are the trustworthy figures. arm64 runner-fleet winners are gated behind a multi-arch image + warmer DaemonSet, so amd64 is the actionable pick today. Reserved Capacity-Block families (p5, p6-b200) are intentionally absent from the price table. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
Capacity reportcommit ✅ simulate-cluster✅ analyze-utilization |
tofu plan — meta-prod-aws-ue2✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-uw1✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue2✅ Plan succeeded · commit Plan output |
jeanschmidt
marked this pull request as draft
July 6, 2026 02:41
- Reword per-family rec cost caveat: priced-$ tracks vcpu_hours x $/vcpu; node_hours is a size-blind count, not the cost signal - Update test_optimize_cost assertion to match new caveat text - Update test_optimize_report assertion to match new caveat text The old "node-hours x price = authoritative cost" phrasing was misleading — node_hours is size-blind and does not reflect the priced dollar figure, which is driven by vcpu_hours x $/vcpu. The new wording steers readers to the actual cost signal. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Correct Helm/EKS addon DaemonSet request constants to real values (node-exporter 0/0, alloy-logging 510/1074, kube-proxy 100/0, vpc-cni 50/0, ebs-csi-node 30/104) and add efs-csi-node - Discover .yaml.tpl templates and derive names from the module dir for placeholder names via _daemonset_name - Add tests for template discovery, _daemonset_name, and pinned addon/Helm constants - Update runner-slots expectation for c7i.2xlarge (10 -> 9) to reflect the higher per-node overhead Notes: Prior constants were rough guesses that undercounted per-node overhead. Values now match the rendered chart output and AWS-managed addon specs (including config-reloader/sidecar containers). hf-cache and other templated DaemonSets live in .yaml.tpl with a __DS_NAME__ placeholder, so they were previously invisible to the sweep; deriving the key from the owning module dir gives each a unique, stable name. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add a "Running" section to optimize.md with the canonical optimize_search.py command - Document --drop-provider lf, --last-days, and --keep-fraction flags and their meaning Notes: Records the exact command used to produce the canonical before/after reports, including the non-default --keep-fraction 0.5 downsample (CLI default is 1.0), so future runs reproduce the same results. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
…chmidt/gpu_fleet_score # Conflicts: # osdc/scripts/python/test_daemonset_overhead.py
- Split hf-cache overhead into rclone base + fixed taint-remover sidecar (10m/32Mi), raising the flat per-node base to 110m/288Mi - Keep rclone's per-GPU-count top-up separate from the flat sidecar memory in hf_cache_gpu_topup_mib() - Add the per-GPU-count hf-cache reserve to sim_nodes daemonset_totals so it mirrors compute_allocatable's daemonset basis - Update tests for the new base and add a reconstruction invariant test (allocatable + daemonset == capacity net of kubelet) The node-size-sweep simulator previously omitted the per-GPU-count hf-cache reserve that compute_allocatable applies, so daemonset_totals under-counted memory on GPU nodes and allocatable + daemonset no longer reconstructed the node's true capacity. Also folds the always-running taint-remover sidecar into the base overhead, which the raw-YAML scan can't see since hf-cache is .tpl-rendered. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Replace opt_max (max util) objective with minimizing total physical vCPU-minutes; tie-break on fewest peak concurrent nodes (rank_key, primary, improvement_pp in optimize_engine) - Add total_vcpu_minutes/peak_nodes to SimMetrics, SimCache schema, and restart_state (best_opt_max -> best_objective), threading them through search, resume, progress, and reports - Widen the CPU down-floor from 5% to 10% (MEM_LOWER_PCT 0.90) and pack pods against the shrink floor so a pod can cross an integer packing boundary; fill the slot back up but never above the operator's size - Move g5/g6/g4dn from in-scope to excluded families - Reframe progress/report output as objective + reduction %, keeping opt_max/cal_* as reported-only proxies; update tests accordingly The prior objective maximized allocatable utilization, which does not directly track cost: a family can look "less utilized" yet provision fewer real vCPU-hours. Ranking on physical vCPU-minutes (instance vCPU x minutes alive) targets the quantity proportional to spend, with peak node count as the tie-break for burst pressure. Packing against the down-floor lets the search find configs where a modest CPU shrink fits one more pod per node, then hands back the freed headroom capped at the original size. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full run and optimal values: