Skip to content

SKILL.md

Latest commit

 

History

History
186 lines (159 loc) · 17.4 KB

SKILL.md

File metadata and controls

186 lines (159 loc) · 17.4 KB
Raw
name osdc-project-structure
description OSDC platform architecture, project organization, directory tree, key design decisions, external knowledge base, key files, and docs index. Applies to ~/meta/ci-infra/osdc. Load when working on OSDC to understand project layout and architecture.

OSDC Project Structure & Architecture

What This Is

Modular Kubernetes infrastructure platform on AWS EKS. A shared base/ provides the cluster (VPC, EKS, Harbor, GPU plugins, NodeLocal DNS), and optional modules/ layer services on top (ARC, runners, BuildKit, future projects). One codebase drives multiple clusters across regions via clusters.yaml.

Working directory: osdc/. Run all commands from here.

Project Organization — Critical

Before making ANY changes, understand how this project is organized. This project MUST be well-organized at all times:

  1. Base vs modules: Shared infrastructure goes in base/. Optional services go in modules/<name>/. If every cluster needs it, it's base.
  2. Folder separation is mandatory: Separate directories for terraform/, kubernetes/, helm/, scripts/, docker/ within each component.
  3. Technology separation: Don't mix Python, Terraform, Bash, Helm, and Kubernetes manifests in the same directory.
  4. Cloud separation: Separate cloud-specific code (AWS/Kubernetes) from cloud-agnostic definitions (runner defs).

Before planning changes:

  • FIRST: Examine the current project structure
  • THEN: Plan changes to maintain or improve the organization
  • NEVER: Mix unrelated files or technologies in the same directory
osdc/
├── clusters.yaml           # THE source of truth — clusters + module lists
├── justfile                # All operations (deploy, lint, test, setup)
├── mise.toml               # Tool versions (tofu, kubectl, helm, crane, ruff, etc.)
├── pyproject.toml          # Python deps + dev deps (managed by uv)
├── scripts/                # Orchestration helpers
│   ├── cluster-config.py   # Reads clusters.yaml, outputs values for just/shell
│   ├── bootstrap-state.sh  # Creates S3 + DynamoDB for tofu state
│   ├── mise-activate.sh    # Sourceable helper for shebang recipes needing mise tools
│   └── python/
│       └── configure_harbor_projects.py  # Harbor proxy cache project setup
├── base/                   # Deployed to EVERY cluster
│   ├── kubernetes/         # StorageClass, NVIDIA plugin, Harbor NS, perf tuning, image-cache-janitor, nodelocaldns, node-taint-remover, registry-mirror config, git-cache, algif-mitigation (CVE-2026-31431, temporary), dirtyfrag-mitigation (CVE-2026-43284 + CVE-2026-43500, temporary)
│   │   ├── eniconfigs/     # AZ-named ENIConfig CRs (one per AZ from private_subnets_by_az terraform output); inert until VPC CNI Custom Networking is enabled. Not used under IPv6-only mode (Custom Networking is IPv4-only); retained as IPv4 rollback safety net
│   │   ├── harbor/         # PDB template (sed-substituted, applied by _deploy-harbor)
│   │   ├── image-cache-janitor/ # Cleans up unused cached container images on nodes
│   │   ├── nodelocaldns/   # NodeLocal DNSCache — per-node CoreDNS DaemonSet (iptables-mode interception); deployed via own deploy.sh (resolves kube-dns ClusterIP at apply time)
│   │   ├── node-taint-remover/ # Operator that strips startup taints once their gating DaemonSets are Ready; ConfigMap registry generated by lib/taint_remover.py, consumed by cache-enforcer, registry-mirror-config, node-performance-tuning
│   │   └── tests/          # Base-level Kubernetes test fixtures
│   ├── helm/               # Harbor values
│   ├── docker/             # Container images (runner-base)
│   ├── scripts/            # Bootstrap (EKS node setup)
│   └── node-compactor/     # Node consolidation controller (taints underutilized nodes)
├── modules/                # Optional, per clusters.yaml
│   ├── eks/                # AWS infrastructure (VPC, EKS, Harbor S3/IAM, image mirroring)
│   ├── karpenter/          # Karpenter controller + AWS infra (IAM, SQS, EventBridge)
│   ├── keda/               # KEDA operator (provides ScaledObject/TriggerAuthentication CRDs used by buildkit autoscaling)
│   ├── arc/                # ARC controller (GitHub Actions)
│   ├── nodepools/          # Karpenter NodePools (pure compute provisioning)
│   ├── nodepools-b200/     # B200 (Blackwell) GPU nodepools
│   ├── nodepools-h100/     # H100 (Hopper) GPU nodepools
│   ├── arc-runners/        # ARC runner scale sets (requires arc + nodepools)
│   ├── arc-runners-b200/   # B200 GPU runner scale sets
│   ├── arc-runners-h100/   # H100 GPU runner scale sets
│   ├── buildkit/           # BuildKit build service (arm64 + amd64, HAProxy LB)
│   ├── cache-enforcer/     # DaemonSet blocking direct registry/PyPI access (forces Harbor + pypi-cache)
│   ├── harbor-cache-recovery/ # Recovers Harbor cache state after pod restarts/failures
│   ├── logging/            # Centralized log collection (Alloy DaemonSet for logs + Deployment for events -> Grafana Cloud Loki)
│   ├── monitoring/         # Metrics pipeline: kube-prometheus-stack CRDs/exporters + Alloy -> Grafana Cloud
│   ├── pypi-cache/         # Self-learning PyPI wheel cache (nginx + pypiserver)
│   └── zombie-cleanup/     # Periodic cleanup of orphaned/zombie pods and resources
├── integration-tests/      # Top-level integration suites (load-test, workflows, workload-test, scripts, docker)
├── tests/smoke/            # Top-level smoke tests
└── docs/                   # Architecture and operations docs

Key Design Decisions

  • Harbor is always-on — baked into base, not a module. Every cluster needs a pull-through cache.
  • NVIDIA device plugin in base — foundational infrastructure, not optional.
  • Karpenter is a module — allows clusters to opt-in to Karpenter. The controller, IAM, SQS, and EventBridge are all in modules/karpenter/. NodePools are separate in modules/nodepools/.
  • Compute split: nodepools + arc-runners — nodepools deploys pure Karpenter NodePools (compute provisioning). arc-runners deploys ARC runner scale sets (requires both arc and nodepools). Non-ARC clusters can use nodepools alone for compute.
  • One terraform root, many clusters — no code duplication per environment.
  • Modules are independent — each has its own terraform state, k8s resources, deploy script. No cross-module imports.
  • Monitoring and logging are both modules — metrics collection (Alloy Deployment -> Grafana Cloud Mimir) is opt-in via the monitoring module. Log collection (Alloy DaemonSet for pod/journal logs + Alloy Deployment for Kubernetes events -> Grafana Cloud Loki) is in modules/logging/. Both are secret-gated — no credentials, no Alloy.
  • Three Alloy installations — separate Helm releases, namespaces, and RBAC for: monitoring metrics (Deployment, modules/monitoring/helm/alloy-values.yaml), logging pod/journal logs (DaemonSet, modules/logging/helm/alloy-logging-values.yaml), and logging Kubernetes events (Deployment, modules/logging/helm/alloy-events-values.yaml). This avoids config/permission conflicts and lets each scale independently. See osdc-observability skill for details.

External Knowledge Base

The actions-knowledge-base/ directory contains source code, documentation, and detailed reference material for many of the open-source projects and tools used by this project — including Actions Runner Controller, Harbor, the Harbor Helm chart, the GitHub Actions runner, runner images, cloud-provider credential actions, and more. Each project lives as a git submodule under repos/ and the knowledge base's own AGENTS.md indexes every included repository with summaries and key paths.

Finding it: The directory is named actions-knowledge-base and lives somewhere above or beside this project. Walk upward from the current working directory, checking each ancestor and its children, until you find a directory named actions-knowledge-base. It is typically a sibling of the top-level repo (e.g., beside ci-infra/), but the exact location depends on the checkout layout. Do not hardcode a relative path — search for it.

The knowledge base has two key directories:

  • actions-knowledge-base/repos/ — Read-only upstream source code and docs (git submodules). Managed via sync.py.
  • actions-knowledge-base/docs/ — Our own findings, workarounds, and gotchas discovered during development and operations (e.g., BuildKit OTEL crash, Grafana Alloy setup, CRD ordering issues, deploy phase ordering).

You should consult this knowledge base whenever you need to:

  • Understand how a dependency works (e.g., ARC Helm chart values, Harbor configuration, runner lifecycle)
  • Clarify configuration quirks, edge cases, or undocumented behavior
  • Look up default values, API surfaces, or internal implementation details
  • Verify correct usage of upstream Helm charts, container images, or action inputs
  • Review operational learnings and previously discovered issues (docs/)

In most cases, reading the relevant source or docs in the knowledge base first will produce more accurate results than guessing or relying on general knowledge alone. Treat it as a first-class resource.

Key Files

File What it does
clusters.yaml Defines all clusters, modules, and per-installation config (replicas, log levels, runner limits, etc.)
justfile All operations (deploy, lint, test, setup, kubeconfig)
mise.toml Tool versions (tofu, kubectl, helm, crane, ruff, shellcheck, etc.) + env vars
pyproject.toml Python dependencies + dev deps, managed by uv
scripts/cluster-config.py Reads clusters.yaml for justfile/shell consumption
scripts/bootstrap-state.sh Creates S3 bucket + DynamoDB for tofu state
scripts/destroy-cluster.sh Operator-facing helper for tearing down a cluster's tofu-managed infrastructure
scripts/cleanup-git-cache.sh Transitional operator helper — removes orphaned in-cluster git-cache resources left over from #703 code removal; delete once all clusters are cleaned
scripts/mise-activate.sh Sourceable helper — adds mise tools to PATH for shebang recipes
scripts/deploy-log.sh, scripts/deploy-status.py Deploy logging + status reporting helpers (called by justfile)
scripts/helm-upgrade.sh, scripts/kubectl-apply.sh, scripts/yaml-diff.py, scripts/kubeconfig-lock.sh Wrappers used by module deploy.sh scripts
scripts/python/configure_harbor_projects.py Configures Harbor proxy cache projects via API
scripts/python/ (analysis) analyze_node_utilization.py, simulate_cluster.py, instance_specs.py (two dicts: INSTANCE_SPECS compute/memory/arch keyed by AWS instance type, ENI_MAX_PODS AWS-stock max-pods from awslabs/amazon-eks-ami eni-max-pods.txt, used by simulators and analyzers), pytorch_workload_data.py, daemonset_overhead.py, collect_instance_memory.py, simulate_cluster_cli.py — capacity / utilization analysis tooling
scripts/python/cni_constants.py Single source of truth for the VPC CNI Custom Networking node label key (ENI_CONFIG_LABEL = "ipam.osdc.internal/eni-config"); imported by smoke tests and the nodepool generator, referenced as a string literal by bootstrap and tofu addon config
scripts/state-config.sh Sourceable shell helper — exports STATE_REGION (AWS region of the shared tofu state bucket + DynamoDB lock table, independent of each cluster's own region). Sourced by every deploy script and module deploy.sh
modules/eks/terraform/main.tf Parameterized infra (VPC, EKS, Harbor)
modules/eks/terraform/variables.tf All variables driven from clusters.yaml
modules/eks/images.yaml Harbor bootstrap images to mirror to ECR
base/kubernetes/node-taint-remover/ Operator that removes startup taints from nodes once their gating DaemonSets are Ready; lib/taint_remover.py generates the ConfigMap registry used by cache-enforcer, registry-mirror-config, and node-performance-tuning
base/kubernetes/harbor/pdb.yaml.tpl Harbor PDB template (multi-doc, sed-substituted by _deploy-harbor)
base/node-compactor/ Node consolidation controller (taints underutilized Karpenter nodes)
modules/karpenter/deploy.sh Karpenter controller + AWS infra (IAM, SQS, Helm install)
modules/keda/deploy.sh KEDA operator install (provides ScaledObject CRDs used by buildkit autoscaling)
modules/nodepools/defs/*.yaml NodePool definitions (instance type, arch, disk, gpu flag)
modules/nodepools/deploy.sh Generate + apply Karpenter NodePools
modules/arc-runners/defs/*.yaml Runner definitions (instance type, CPU, memory, GPU, max runners)
modules/arc-runners/deploy.sh Generate + install ARC runner scale sets (requires arc module)
modules/buildkit/deploy.sh Generate + deploy BuildKit (Deployments, HAProxy, NodePools)
modules/monitoring/CLAUDE.md Per-module rules for monitoring (read first when editing this module)
modules/monitoring/deploy.sh Deploy kube-prometheus-stack + monitors + conditionally Alloy
modules/monitoring/helm/values.yaml kube-prometheus-stack Helm values (node placement, storage, auto-discovery)
modules/monitoring/helm/alloy-values.yaml Grafana Alloy Deployment values for metrics (ServiceMonitor/PodMonitor discovery, remote_write to Mimir)
modules/monitoring/kubernetes/monitors/ ServiceMonitors + PodMonitors for OSDC components (ARC, Harbor, Karpenter, etc.)
modules/monitoring/kubernetes/dcgm-exporter/ DCGM GPU exporter DaemonSet + headless Service (GPU nodes only)
modules/logging/deploy.sh Secret-gated Alloy install for log collection -> Grafana Cloud Loki (DaemonSet for journal logs, Deployment for Kubernetes events)
modules/logging/pipelines/base.alloy Alloy DaemonSet config — journal source + loki.write (container stdout/stderr intentionally not shipped)
modules/logging/helm/alloy-logging-values.yaml Alloy DaemonSet Helm values for journal logs (tolerates all taints, journal mount, positions hostPath)
modules/logging/helm/alloy-events-values.yaml Alloy Deployment Helm values for Kubernetes events stream
modules/logging/scripts/python/assemble_config.py Wraps base.alloy in a ConfigMap YAML
integration-tests/, tests/smoke/ Top-level integration suites and smoke tests (per-module tests/ subdirs cover module-local checks)

Base Component Notes

Node Taint Remover

base/kubernetes/node-taint-remover/ runs an operator that strips startup taints once their gating DaemonSets are Ready on a node. The taint registry is generated by lib/taint_remover.py and consumed by base DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning). When adding a new startup-taint-bearing DaemonSet to base or a module, register the taint here.

Node Compactor

base/node-compactor/ is a node consolidation controller that taints underutilized Karpenter nodes to compact workloads WITHOUT eviction. It relies on Karpenter's WhenEmpty consolidation to clean up empty nodes. See osdc-runners-nodepools skill for configuration details.

Documentation (docs/)

Keep this list in sync. When adding, removing, or significantly editing a document in docs/, update this table (description, filename, or entry presence).

Document Description
arc-fork-build-deploy.md Building and deploying the ARC controller fork (image build, push, version pinning)
architecture.md Modular platform architecture — base vs modules, deployment layers
ci-pre-merge.md Pre-merge CI workflow and gating
cluster-recreation.md Operator runbook for destroying and recreating an OSDC cluster (per-cluster cutover, accepted data losses for Harbor S3 + EFS pypi-cache, pre-flight checks)
current_runner_load_distribution.md 30-day job counts and peak concurrency analysis by runner type
loki_query.md CLI guide for querying Grafana Cloud Loki logs (credentials, LogQL patterns)
mimir_query.md CLI guide for querying Grafana Cloud Mimir metrics (credentials, PromQL patterns)
modules.md Module contract and deployment order (Terraform -> Kubernetes -> deploy.sh)
node-utilization-optimization.md Runner-to-node packing efficiency analysis and instance type recommendations
node-warmup-and-scheduling-gates.md Node startup flow — startup taints, gating DaemonSets (hooks, registry mirrors, perf tuning), init container gates, taint summary
observability.md Three-Alloy observability architecture (metrics Deployment + logs DaemonSet + events Deployment -> Grafana Cloud)
observability-estimates.md Per-unit cost estimates for metrics cardinality and log volume (Grafana Cloud billing)
operations.md Operational procedures — bootstrap, deploy, and manage clusters
prod-cluster-ha-us-west-1.md Production cluster HA design notes for us-west-1
pypi-package-cache.md PyPI wheel cache architecture and operational guide
runner-image-autoresolve.md Deploy-time auto-resolution of the runner image to a digest-pinned reference; ConfigMap shape, rollback paths, failure modes
runner_naming_convention.md Compact runner label naming scheme (42-char limit, abbreviation rules)