Skip to content

SKILL.md

Latest commit

 

History

History
359 lines (268 loc) · 30.9 KB

SKILL.md

File metadata and controls

359 lines (268 loc) · 30.9 KB
Raw
name osdc-deployment
description OSDC deployment workflow, just recipes, base deploy order, module deploy order, clusters.yaml configuration, Terraform architecture, smoke tests, justfile conventions. Applies to ~/meta/ci-infra/osdc. Load when deploying, adding modules, or modifying deploy scripts.

OSDC Deployment & Operations

How Deployment Works

clusters.yaml defines every cluster and its modules. The justfile reads it via scripts/cluster-config.py.

just setup                                       # Install Python dependencies (uv sync); mise tools are auto-installed from mise.toml on first `mise exec`
just clean                                       # Remove caches, init artifacts, .venv
just list                                        # Show all clusters and modules
just show <cluster>                              # Inspect cluster config
just kubeconfig <cluster>                        # Configure kubectl (uses scripts/kubeconfig-lock.sh)
just bootstrap <cluster>                         # Create S3 state bucket + DynamoDB
just bootstrap-all                               # Bootstrap all clusters
just deploy <cluster>                            # Full deploy (base + modules)
just deploy-base <cluster>                       # Base only
just deploy-module <cluster> <module> [force]    # Single module; force="" sets HELM_FORCE_UPGRADE=1
just plan <cluster>                              # Read-only `tofu plan` for base + every TF-backed module (CI-safe)
just deploy-history <cluster>                    # List deploy audit ConfigMaps in osdc-system namespace
just deploy-status <cluster> [name]              # Human-readable status of recent deploys
# ^ Both read the deploy-audit ConfigMaps written by every just deploy/deploy-base/deploy-module
#   into osdc-system. For the ConfigMap schema, raw kubectl queries, cross-cluster version diffs,
#   stuck-deploy detection, and failure triage, see the "Deploy-Audit ConfigMaps" section of the
#   osdc-cli-debugging skill.
just recycle-nodes <cluster>                     # Terminate Karpenter nodes (reprovisioned on demand)
just test                                        # Run all unit tests (95% per-file coverage required)
just test-compactor <cluster>                    # Run node-compactor e2e tests (base/node-compactor/tests/e2e/)
just test-janitor <cluster>                      # Run image-cache-janitor e2e tests (base/kubernetes/image-cache-janitor/tests/e2e/)
just smoke <cluster>                             # Run smoke tests against a deployed cluster
just integration-test <cluster> [args...]        # Run full integration test via integration-tests/scripts/python/run.py. CI passes --skip-drain --skip-smoke --skip-compactor
just generate-arc-runners <cluster>              # Generate ARC scale-set YAMLs (no apply)
just lint                                        # Lint all code (shell, Python, Docker, k8s, tf)
just lint-fix                                    # Auto-fix lint issues (ruff, shfmt, tofu fmt, taplo)
just analyze-utilization                         # Analyze runner-to-node packing efficiency
just simulate-cluster                            # Monte Carlo cluster simulation for CI load
just taint-nodes <cluster>                       # Taint ARC runner nodes with NoSchedule for graceful refresh
just untaint-nodes <cluster>                     # Reverse taint-nodes
just drain-runners <cluster>                     # Patch all ARC AutoscalingRunnerSets to maxRunners=0, taint runner nodes, wait for in-flight pods to finish (default 1h; tune with OSDC_DRAIN_TIMEOUT_SECS)
just resume-runners <cluster>                    # Restore maxRunners from defs/ files and untaint nodes (out-of-band recovery; normal cutover path is `just deploy`)
just remove-module <cluster> <module>            # Uninstall all osdc.io/module=<module> resources + matching arc-runners helm releases. Run `just drain-runners` first for runner modules.
just heal-arc-check <cluster> [name]             # Read-only: detect ARC scale-set-id drift (controller, ARS, listener, ERS). Exit 1 if drift found.
just heal-arc <cluster> [name]                   # Detect + recover ARC scale-set-id drift (controller restart, ARS re-registration, listener/ERS cascade).
just load-test <cluster>                         # Run load test against a cluster
just workload-test <cluster>                     # Run production workload test

Internal recipes (prefixed _, called from other recipes; documented here because they expose load-bearing behavior):

  • _deploy-harbor <cluster> — called from deploy-base. Reads harbor_role_arn, harbor_s3_bucket, harbor_s3_access_key_id, harbor_s3_secret_access_key from base tofu outputs; installs the Harbor Helm chart; applies the standalone Harbor PDB manifest (chart has no native PDB support); then calls _configure-harbor-projects.
  • _configure-harbor-projects — bootstraps Harbor proxy-cache projects via the Harbor REST API. Optionally consumes harbor-dockerhub-credentials and harbor-github-credentials secrets in harbor-system to authenticate proxy-cache pulls against Docker Hub / ghcr.io (avoids anonymous rate limits). Create those secrets out-of-band before deploy if needed.

Environment variables that control deploy behavior

  • OSDC_CONFIRM=yes|no|ask (default ask) — non-interactive confirmation for just deploy and just deploy-base plans.
  • OSDC_TAINT_NODES=yes|no|ask (default ask) — auto-taint ARC runner nodes after just deploy. The deploy recipe consults this only when recycle_karpenter_nodes != true; under recycle, tainting is skipped unconditionally (nodes are already being destroyed).
  • OSDC_SMOKE=yes|no|ask (default ask) — auto-run smoke tests at end of just deploy.
  • OSDC_DRAIN_TIMEOUT_SECS=<seconds> (default 3600) — wait budget for just drain-runners while in-flight runner pods finish before declaring stragglers and exiting non-zero.
  • OSDC_UNTAINT_NODES=yes|no (default yes) — controls whether just resume-runners removes the refresh taint from runner nodes at the end. Set no to leave the taint in place for a separate just untaint-nodes step. Mirrors OSDC_TAINT_NODES on the drain-runners side.
  • OSDC_CONFIRM_BASE=destroy — required in addition to OSDC_CONFIRM=yes for scripts/destroy-cluster.sh to non-interactively destroy base infrastructure. Not consumed by just deploy; included here because it sits on the same operational surface.
  • OSDC_RESOLVER_READONLY=1 — set by the smoke recipe around its generate-arc-runners preflight so the runner-image resolver reads the newest cached entry from the lock ConfigMap instead of mutating it or hitting GitHub. Anyone running generate-arc-runners standalone for testing should NOT set this unless they explicitly want the cached behavior.
  • HELM_FORCE_UPGRADE=1 — passed by deploy-module when invoked with the force arg; modules' deploy.sh use this to force Helm upgrades.
  • AGENT_ENVIRONMENT=true — agent mode for just lint, just test, just smoke, just generate-arc-runners: suppresses output on success, prints only on failure. Useful for sub-agents.
  • OSDC_ROOT, OSDC_UPSTREAM, CLUSTERS_YAML — exported by deploy-module, deploy-base, drain-runners, resume-runners, smoke, etc., into the environment of every module deploy.sh. OSDC_ROOT is the consumer repo root (often equal to OSDC_UPSTREAM); OSDC_UPSTREAM is the upstream OSDC checkout providing shared scripts and modules. Module scripts that source shared helpers (mise-activate.sh, state-config.sh, kubectl-apply.sh, etc.) MUST use OSDC_UPSTREAM so they work both in standalone and consumer-override setups.

Base deploy order (always)

  1. Terraform — VPC, EKS, Harbor S3 bucket + IAM role + S3 access keys (all produced by modules/eks/terraform, parameterized, no per-env dirs)
  2. Mirror images — Harbor bootstrap images to ECR (Harbor can't cache itself)
  3. Base k8s — StorageClass, NVIDIA plugin, performance tuning (kubectl apply -k)
  4. node-taint-remover — Generates the node-taint-remover-lib ConfigMap in kube-system from lib/taint_remover.py (generated at apply time to prevent drift). Mounted into DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning) so they can remove their own startup taint after init.
  5. ENIConfigs — One AZ-named ENIConfig CR per AZ, rendered from private_subnets_by_az (base terraform output). Inert by design under IPv6-only EKS — VPC CNI Custom Networking is unsupported in IPv6 mode. Retained on the deploy path as the rollback fallback for IPv4 + Custom Networking. Has its own deploy.sh because it must read the terraform output at apply time.
  6. Harbor — Pull-through cache (caches docker.io, ghcr.io, nvcr.io, registry.k8s.io, quay.io, public.ecr.aws). _deploy-harbor reads harbor_role_arn, harbor_s3_bucket, harbor_s3_access_key_id, harbor_s3_secret_access_key from base outputs.
  7. Node compactor — Taints underutilized Karpenter nodes for consolidation (if enabled)
  8. Image Cache Janitor — Cleans up unused cached images on nodes
  9. NodeLocal DNSCache — Per-node CoreDNS DaemonSet (iptables-mode interception). Has its own deploy.sh (NOT in base/kubernetes/kustomization.yaml — same pattern as image-cache-janitor) because it must resolve the live kube-dns Service ClusterIP at apply time and substitute it into Corefile + DaemonSet args.

Module deploy order (per clusters.yaml list order)

Each module may have:

File Purpose Applied by justfile
terraform/main.tf AWS resources (optional) tofu apply with cluster vars
kubernetes/kustomization.yaml K8s manifests (optional) kubectl apply -k
deploy.sh Custom deploy logic (optional) Called with (cluster-id, cluster-name, region)

The justfile auto-detects which of these exist and runs them in order: terraform -> kubernetes -> deploy.sh.

deploy-module exports three env vars consumed by every module deploy.sh: OSDC_ROOT (consumer repo), OSDC_UPSTREAM (upstream OSDC checkout), and CLUSTERS_YAML (path to the active clusters.yaml). It also resolves the module directory with consumer-first precedence: if $OSDC_ROOT/modules/$MODULE exists, it wins over $OSDC_UPSTREAM/modules/$MODULE. This is how downstream consumers override upstream modules without forking the repo. See docs/modules.md for the full module contract.

clusters.yaml drives everything

There is no hardcoded staging/production concept. Each cluster is an independent installation with its own config. clusters.yaml controls everything: infrastructure sizing, component tuning (Harbor replicas, Karpenter PDB, ARC log level), module-specific settings (BuildKit replicas, runner GitHub config, max runner counts), and module list.

Adding a new cluster: add an entry to clusters.yaml with region, cluster_name, state_bucket, base config, component tuning, module config, and module list. Then just bootstrap <cluster-id> + just deploy <cluster-id>.

Adding a module to a cluster: append the module name to the cluster's modules list.

To create a new module: make a directory under modules/, add any of the files above, and list the module name in clusters.yaml for the target cluster.

Reading cluster config in scripts: uv run scripts/cluster-config.py <cluster-id> <dot.path> [default]. Supports nested paths (e.g., harbor.core_replicas). Falls back to defaults: section, then optional third argument.

recycle_karpenter_nodes: true (cluster-level config, e.g. meta-staging-aws-uw1, meta-staging-aws-ue1): when set, just deploy calls just recycle-nodes "$CLUSTER" automatically after modules deploy and skips the ARC node-tainting step (tainting is pointless when nodes are being destroyed).

proactive_capacity_max: <int> (cluster-level): caps the warm-pool placeholder pairs the runner-set scaler will provision ahead of demand. Staging clusters set this to 0 to disable pre-provisioning entirely. See docs/arc-fork-build-deploy.md for how this is plumbed into the controller (CAPACITY_AWARE_PROACTIVE_CAPACITY).

Justfile Kubeconfig Convention (MANDATORY)

Every just recipe that interacts with a live cluster MUST call just kubeconfig <cluster> before doing any kubectl/helm work. This prevents operators from accidentally running commands against the wrong cluster.

The kubeconfig recipe wraps aws eks update-kubeconfig via scripts/kubeconfig-lock.sh — a directory-based mutex (atomic mkdir) that serializes concurrent kubeconfig updates and prevents races between parallel deploys. The wrapper enforces a 90s acquisition timeout and breaks stale locks older than 45s.

Note: just plan does NOT call just kubeconfig — it is read-only tofu and does no k8s/helm work, so it is intentionally exempt from the kubeconfig contract.

The pattern (in shebang recipes):

export CLUSTERS_YAML="{{CLUSTERS_YAML}}"
just kubeconfig "$CLUSTER"

This applies to: deploy-base, deploy-module, recycle-nodes, taint-nodes, untaint-nodes, drain-runners, resume-runners, _deploy-harbor, smoke, test-compactor, test-janitor, integration-test, deploy-history, deploy-status, and any future recipe that calls kubectl, helm, or deploy scripts. When adding a new recipe that touches a cluster, always include the just kubeconfig step. Do NOT call aws eks update-kubeconfig directly — the locking wrapper is required.

Justfile + Mise Gotcha

Tool versions for the entire repo are pinned in mise.toml (tofu, kubectl, helm, crane, awscli, ruff, taplo, shellcheck, shfmt, hadolint, kube-linter, kubeconform, tflint, trivy, uv, python, etc.). The [settings] auto_install = true setting means mise exec installs missing pinned tools on first use — there is no separate "install tools" command; just setup only runs uv sync for Python deps.

The justfile uses set shell := ["mise", "exec", "--", "bash", "-euo", "pipefail", "-c"] so that mise-managed tools are on PATH for every recipe line.

Shebang recipes bypass this. A recipe starting with #!/usr/bin/env bash is executed as a standalone script by just, NOT through the configured shell. This means mise-managed tools may not be on PATH inside shebang recipes or scripts they call. Shebang recipes in this repo work around it by sourcing scripts/mise-activate.sh as their first line.

For recipes that need mise tools, use non-shebang style (line-by-line with @ prefix) so they run through mise exec. If a shebang recipe is required, source mise-activate.sh (or call subscripts via mise exec -- ./script.sh explicitly).

Terraform Architecture

Single parameterized root at modules/eks/terraform/. No per-environment directories.

  • Variables (cluster_name, aws_region, vpc_cidr, etc.) come from clusters.yaml -> justfile -> tofu -var=
  • Backend configured at tofu init via -backend-config (bucket, key, region)
  • Each cluster gets its own state: s3://<state_bucket>/<cluster-id>/base/terraform.tfstate
  • Modules with their own terraform get: s3://<state_bucket>/<cluster-id>/<module>/terraform.tfstate. Module terraform also receives state_bucket and cluster_id as vars.

Smoke Tests (MANDATORY)

Post-deploy validation that a cluster is actually healthy. Run via just smoke <cluster> or automatically at the end of just deploy (controlled by OSDC_SMOKE=yes|no|ask).

Smoke tests verify that what was deployed is actually working: Helm releases are deployed, pods are Running, DaemonSets have all pods ready, required secrets and service accounts exist, AWS resources (IAM roles, SQS queues) are present, and Karpenter NodePools / ARC runner scale sets match their definition files. They do NOT test application-level behavior — just that the deployment landed correctly.

Architecture: Each module owns its own smoke tests, co-located at <module>/tests/smoke/. A root tests/smoke/conftest.py provides shared fixtures (batch-fetched cluster state, cluster config, CLI helpers) that all module tests inherit via pytest's conftest discovery. just smoke <cluster> reads clusters.yaml, finds enabled modules, collects their tests/smoke/ directories, and runs pytest across all of them. The recipe also unconditionally pulls smoke tests from base/**/tests/smoke AND modules/eks/tests/smoke (EKS is always deployed as part of base, regardless of clusters.yaml.modules). Adding a module with smoke tests = they're discovered automatically. Moving a module = its tests move with it.

Preflight: ARC runner YAML pre-generation. Before any pytest worker starts, the smoke recipe iterates the cluster's enabled modules and runs just generate-arc-runners "$CLUSTER" for every arc-runners / arc-runners-* variant (canonical + per-GPU-arch), parameterized via ARC_RUNNERS_DEFS_DIR / ARC_RUNNERS_OUTPUT_DIR / ARC_RUNNERS_MODULE_NAME. This regeneration MUST happen once up front — previously a test fixture shelled out per-test and raced under pytest-xdist. Anyone debugging "why does smoke regenerate before tests run" is looking at this step.

When changing ANY module or base component, you MUST update its smoke tests to cover the change. If the component doesn't have smoke tests yet, add them. Smoke tests are the safety net that catches deployment regressions — they only work if they stay current.

  • Smoke test files live in <component>/tests/smoke/test_*.py
  • Shared helpers live in the tests/smoke/helpers/ package:
    • helpers/cli.py — run_kubectl, run_helm, run_aws (proxy-bypass-aware)
    • helpers/filters.py — pure in-memory filters over batch fixtures
    • helpers/nodes.py — node-churn helpers (unstable nodes, disruption taints)
    • helpers/retry.py — retry_with_backoff primitive
    • helpers/k8s_asserts.py — DaemonSet / Deployment readiness assertions
    • helpers/remote.py — Mimir / Loki query helpers with skip-on-network-failure semantics
    • helpers/__init__.py re-exports everything (from helpers import X works without naming the submodule)
  • Each module's tests/smoke/conftest.py re-exports the root conftest: from smoke_conftest import *
  • DaemonSet checks use assert_daemonset_healthy() which tolerates mismatches caused by nodes in transition (new, NotReady, or being deleted). Deployment checks use assert_deployment_ready() which has a 90-second rollout-completion window followed by the standard retry_with_backoff schedule once the rollout finishes but replicas still aren't ready
  • Retry primitive: retry_with_backoff(check, refresh=None, delays=None) is the canonical retry abstraction for smoke tests. Retries only on AssertionError — other exceptions propagate immediately. Optional refresh callable runs between attempts to re-fetch live state. Re-raises the LAST AssertionError after exhaustion.
  • Backoff schedule (resolved at module load time from os.environ.get("CI")):
    • Local (default): 6 attempts total — 1 initial + 5 retries. Sleeps [10, 17, 28.9, 49.1, 83.4] seconds. Worst-case sleep budget per assertion: ~188s.
    • CI (CI=true, set by Buildkite/GHA/etc.): 8 attempts total — extended to [10, 17, 28.9, 49.1, 83.4, 141.78, 241.03] to give slow recoveries more time. Worst-case sleep budget: ~571s per assertion.
  • When writing new smoke tests: use retry_with_backoff for any "wait for X to be ready" pattern — do NOT hand-roll for _ in range(N): time.sleep(D) loops.

Unit Tests & Coverage

just test discovers test_*.py files across the repo, runs them with pytest-xdist (-n auto), and enforces a per-file coverage threshold of 95%. Any source file under the test dirs whose coverage drops below 95% will fail the run, even if pytest itself passed. Add or extend tests when you change source files — bumping global coverage is not enough.

Exclusions and gotcha around smoke tests:

  • Excluded entirely: tests/e2e/, */base/*/tests/smoke/, */modules/*/tests/smoke/ (live-cluster suites — run via just smoke <cluster>).
  • Included: the project-root tests/smoke/ directory, because it hosts unit tests for the smoke-helpers package itself (e.g. tests/smoke/helpers/test_retry.py). The recipe runs those tests but exempts them from the 95% per-file coverage gate (the helpers' full coverage comes from live-cluster smoke runs).

Module-Specific Deployment Notes

EKS Terraform State Key Gotcha

The eks module uses <cluster-id>/base/terraform.tfstate as the state key (historical, predates module extraction). Don't assume the state key matches the module name eks. Modules reading eks outputs via terraform_remote_state (e.g., karpenter reads OIDC provider, node role, SG, subnets) depend on this key.

Base Node Group — Hidden Behaviors

The aws_eks_node_group "base" resource at osdc/modules/eks/terraform/modules/eks/main.tf has two non-obvious behaviors that matter when changing base_node_count or base_node_instance_type in clusters.yaml.

base_node_count is half-applied by tofu

lifecycle.ignore_changes = [scaling_config[0].desired_size] means tofu updates min and max but NOT desired. After bumping base_node_count, you MUST manually run:

NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \
aws eks update-nodegroup-config \
    --cluster-name <cluster_name> \
    --nodegroup-name <cluster_name>-base-nodes \
    --region <region> \
    --scaling-config minSize=N,maxSize=N,desiredSize=N

Skipping this leaves the ASG at the OLD desired_size. Symptom: pods go Pending despite the new count being "applied".

base_node_instance_type change triggers an in-place rolling replacement

EKS treats an instance_types change as a managed node group update: every node is cordoned, drained, and replaced.

  • Honors PDBs for 15 minutes (the update_config timeout)
  • Then forces eviction because force_update_version = true
  • Blast radius controlled by base_node_max_unavailable_percentage in clusters.yaml (default 33%, staging 100%)

Monitor the rolling update with:

aws eks list-updates --cluster-name <cluster_name> --nodegroup-name <cluster_name>-base-nodes --region <region>
aws eks describe-update --cluster-name <cluster_name> --nodegroup-name <cluster_name>-base-nodes --region <region> --update-id <ID>

For one-time migrations (e.g. switching to a larger instance family), follow the cluster-specific playbook if one exists at the repo root — disruption coordination, manual desired_size bump, and rollback steps live there, not here. If no playbook exists for your migration, write one before starting.

Cluster recreation runbook:

  • docs/cluster-recreation.md — operator-facing runbook for destroying and recreating an existing OSDC cluster (used for any change that touches an immutable aws_eks_cluster property: IP family, VPC, CIDR, encryption_config, etc.). Cluster recreates are required because EKS does not support in-place IPv4↔IPv6 conversion. Automation: scripts/destroy-cluster.sh <cluster-id> (requires OSDC_CONFIRM=yes + OSDC_CONFIRM_BASE=destroy for non-interactive destroy).

CRD Ordering — Monitoring Module

Monitoring deploy order is strict: (1) namespace + DCGM DaemonSet via kustomization, (2) kube-prometheus-stack Helm (installs CRDs + exporters), (3) monitors/ applied AFTER CRDs exist, (4) Alloy conditionally. Applying ServiceMonitor/PodMonitor resources before kube-prometheus-stack installs the CRDs will fail.

Cache-Enforcer Dependency

cache-enforcer depends on Harbor (base) and pypi-cache module. Without pypi-cache deployed, pip installs in runner jobs fail entirely (traffic blocked but no cache). Deploy pypi-cache before or alongside cache-enforcer.

clusters.yaml Config Keys Reference

clusters.yaml itself is the source of truth — the section below is a curated subset highlighting load-bearing or non-obvious keys. For the full schema (every defaults: and every per-cluster override), read clusters.yaml directly.

Cluster-level keys (set under clusters.<id>:, no fallback to defaults:):

  • region — AWS region for the cluster (e.g. us-west-2).
  • cluster_name — EKS cluster name (used in IAM roles, ECR repo names, kubeconfig alias).
  • state_bucket — S3 bucket holding terraform state for THIS cluster (s3://<state_bucket>/<cluster-id>/...).
  • recycle_karpenter_nodes: true|false — when true, just deploy runs just recycle-nodes after modules and skips ARC taint-nodes.
  • access_config: — EKS authentication mode + cluster-admin access entries: 
    access_config:
  authentication_mode: API_AND_CONFIG_MAP
  cluster_admin_role_names:
    - osdc_gha_prod
    Consumed by scripts/cluster-config.py into tofu vars. Without this, no IAM role gets cluster-admin via access entries.

Karpenter:

karpenter:
  replicas: 4           # Controller replica count
  log_level: info
  pdb_enabled: true
  pdb_min_available: 1

Harbor:

harbor:
  core_replicas: 4
  registry_replicas: 4
  nginx_replicas: 6
  pdb_max_unavailable: 1     # Default: conservative production protection for core/registry/nginx
  # staging clusters (meta-staging-aws-uw1, meta-staging-aws-ue1) override to "100%" (string, quoted)
  # to effectively disable — staging runs 1 replica each

Standalone PDB manifests are applied by _deploy-harbor after the Helm install (Harbor chart has no native PDB support). See osdc-harbor skill for selector details and known limits.

ARC controller:

arc:
  chart_version: "0.14.1-jeanschmidt.17"   # Must match controller + runner charts (minor mismatch deletes ARS). Fork chart from jeanschmidt/actions-runner-controller. Check clusters.yaml for the current value.
  runner_image_tag: "2.334.0"
  replica_count: 4
  log_level: info
  controller_cpu_request: "1"
  controller_cpu_limit: "4"
  controller_memory_request: "2Gi"
  controller_memory_limit: "4Gi"

BuildKit: Defaults provide only replicas_per_arch: 12. All other knobs are per-cluster overrides:

defaults:
  buildkit:
    replicas_per_arch: 12     # Used when a cluster omits per-arch replica counts
clusters:
  my-cluster:
    buildkit:
      amd64_instance_type: m6id.24xlarge
      amd64_replicas: 32
      amd64_pods_per_node: 2
      arm64_instance_type: m7gd.16xlarge
      arm64_replicas: 8
      arm64_pods_per_node: 4

Per-arch override keys are separate (amd64_* and arm64_*) — there is no flat pods_per_node key.

Node Compactor: All config under node_compactor: key. Knobs configurable via clusters.yaml: enabled, interval_seconds (20), dry_run, min_nodes (1), min_node_age_seconds (900), capacity_reservation_nodes (0), max_uptime_hours (48). Other knobs (taint_rate, fleet_cooldown, spare_capacity_nodes, spare_capacity_ratio, spare_capacity_threshold) are internal Python defaults in the compactor source, NOT configurable via clusters.yaml.

Other defaults blocks (curated — see clusters.yaml defaults: for the full list and current values):

  • coredns.replicas — CoreDNS replica count.
  • nodepools.{gpu_consolidate_after,cpu_consolidate_after,baremetal_consolidate_after,gpu_disruption_budget,cpu_disruption_budget} — Karpenter NodePool churn knobs.
  • monitoring.{grafana_cloud_url,grafana_cloud_read_url} — Grafana Cloud Mimir push/read endpoints.
  • logging.grafana_cloud_loki_url — Grafana Cloud Loki push endpoint.
  • alloy_chart_version — Alloy Helm chart version (shared by monitoring + logging Alloy DaemonSets).
  • keda.chart_version — KEDA operator Helm chart version (used by the keda module; consumed by buildkit autoscaling).
  • zombie_cleanup.{enabled,pending_max_age_hours,running_max_age_hours,dry_run} — orphan-pod GC tuning.
  • pypi_cache.{instance_type,cuda_versions,python_versions,target_architectures,target_manylinux} — PyPI wheel cache matrix (see osdc-pypi-cache for slug semantics; cuda_versions and python_versions are mandatory for this module).

ARC Module — No Terraform

Unlike most modules, modules/arc/ has no terraform phase — it is pure k8s/helm. Don't look for terraform state or try just tf-* recipes for ARC.

Pypi-Cache — Two Terraform Roots

The pypi-cache module has TWO terraform roots; only one runs as part of just deploy:

  • modules/pypi-cache/terraform/main.tf — per-cluster IRSA role, IAM policies, etc. Auto-applied by just deploy-module pypi-cache (the standard deploy-module flow detects terraform/main.tf and runs tofu init && tofu apply on it).
  • modules/pypi-cache/terraform/wheel-cache-bucket/ — standalone tofu root for the shared S3 bucket (pytorch-pypi-wheel-cache). This is a one-time, account-wide resource, NOT part of just deploy. Run tofu init && tofu apply manually in this sub-directory if the bucket needs to be created or updated.

Pypi-Cache — Required clusters.yaml Keys

cuda_versions and python_versions are NOT module defaults. They must be configured in clusters.yaml under defaults.pypi_cache or per-cluster clusters.<id>.pypi_cache. Omitting them produces zero CUDA slugs and zero deployments. cuda_slug() normalizes versions to major.minor only: "12.8.1" → cu128 (matches PyTorch convention).

CI Workflows

OSDC is driven from CI via GitHub Actions workflows living in the parent repo (ci-infra/.github/workflows/), NOT inside osdc/. There is no osdc/.github/ directory.

Key workflows (all callable as reusable workflows via workflow_call from the orgs that ship from this repo):

Workflow What it runs
osdc-lint.yml just lint
osdc-test.yml just test
osdc-pre-merge.yml Lint + test on PR
osdc-plan-prod.yml just plan <cluster> (read-only tofu plan; CI-safe)
osdc-deploy-prod.yml Per-cluster prod deploy. Calls _osdc-deploy.yml
_osdc-deploy.yml Reusable: just lint && just test → just deploy <cluster> → just smoke <cluster> → just integration-test <cluster> --skip-drain --skip-smoke --skip-compactor. Pre-flight checks can be skipped with the skip_lint_test input (firefighting only).
_osdc-plan.yml Reusable: just plan <cluster> and tee to plan.txt
_osdc-slow-tests.yml Reusable: just load-test, just test-compactor, just test-janitor
osdc-capacity-report.yml just simulate-cluster + just analyze-utilization, periodic

When adding a new just recipe that CI should run, plumb it through the corresponding reusable workflow.

The OSDC-rooted CI jobs — the deploy/smoke/integration jobs in _osdc-deploy.yml, plus osdc-drain.yml and osdc-undrain.yml — set up their toolchain via the local composite action ./.github/actions/osdc-setup (installs just + mise + uv sync). Its setup steps are retried on transient failures by retry-by-duplication (continue-on-error on the first attempt plus a copy gated on steps.<id>.outcome == 'failure'), chosen over a retry action deliberately: GitHub Actions has no native step retry, and a retry action would run unpinnable third-party code on the deploy / merge-queue path. actions/checkout and aws-actions/configure-aws-credentials are intentionally NOT wrapped because they already retry internally.