|
| 1 | +# ARC Fork: Build, Deploy, and Configuration |
| 2 | + |
| 3 | +## The ARC Fork |
| 4 | + |
| 5 | +**Repo**: https://github.com/jeanschmidt/actions-runner-controller.git |
| 6 | +**Branch**: `jeanschmidt/placeholder_run_poc` (based on upstream `actions/actions-runner-controller` master) |
| 7 | + |
| 8 | +**Why**: Adds capacity-aware autoscaling (proactive capacity) to the `ghalistener` binary. Stock ARC is count-based and capacity-unaware -- it scales runners without checking whether the cluster can actually fit the runner + workflow pod pair. The fork adds a CapacityMonitor goroutine that dynamically adjusts `maxRunners` reported to GitHub via the `X-ScaleSetMaxCapacity` header, backed by placeholder pod reservations. |
| 9 | + |
| 10 | +**What's changed**: |
| 11 | +- New package: `cmd/ghalistener/capacity/` (config.go, monitor.go, placeholder.go, hud_client.go) |
| 12 | +- Modified: `cmd/ghalistener/main.go` -- adds CapacityMonitor to the listener's errgroup when `CAPACITY_AWARE_ENABLED=true` |
| 13 | +- Everything else (controllers, CRDs, runner charts) runs stock |
| 14 | + |
| 15 | +## Helm Chart |
| 16 | + |
| 17 | +Published from the fork via the `gha-publish-chart.yaml` workflow (manual `workflow_dispatch`). The workflow builds the controller image and publishes the chart to GHCR. |
| 18 | + |
| 19 | +- **OCI registry**: `oci://ghcr.io/jeanschmidt/actions-runner-controller-charts/gha-runner-scale-set-controller` |
| 20 | +- **Chart version**: configured in `clusters.yaml` at `arc.chart_version` (currently `0.14.1-jeanschmidt.5`). Format is `<upstream-base>-jeanschmidt.<N>`; bump `<N>` for each fork publish. Valid as both Helm chart version and OCI image tag. |
| 21 | +- **Image tags**: `ghcr.io/jeanschmidt/gha-runner-scale-set-controller:<release_tag_name>` (set during workflow dispatch — pass `release_tag_name=0.14.1-jeanschmidt.5` to match the chart) |
| 22 | + |
| 23 | +To publish a new chart version, trigger `gha-publish-chart.yaml` from the fork's GitHub Actions UI with `publish_gha_runner_scale_set_controller_chart: true`. |
| 24 | + |
| 25 | +## Using the Local Chart (dev workflow) |
| 26 | + |
| 27 | +For local development, you can skip publishing the chart to GHCR and point `deploy.sh` directly at the local chart path: |
| 28 | + |
| 29 | +In `modules/arc/deploy.sh`, replace the OCI chart reference: |
| 30 | + |
| 31 | +```bash |
| 32 | +# Replace this: |
| 33 | + oci://ghcr.io/jeanschmidt/actions-runner-controller-charts/gha-runner-scale-set-controller \ |
| 34 | + --version "${ARC_CHART_VERSION}" \ |
| 35 | + |
| 36 | +# With the local path (no --version needed): |
| 37 | + /Users/jschmidt/meta/actions-runner-controller/charts/gha-runner-scale-set-controller \ |
| 38 | +``` |
| 39 | + |
| 40 | +Keep the original lines commented out with a `TODO: restore before committing` so they aren't accidentally merged. |
| 41 | + |
| 42 | +This skips the chart publish step entirely — Helm reads the chart directly from disk. You still need to build and push the controller image to Harbor (see below). |
| 43 | + |
| 44 | +## Building the Forked Image |
| 45 | + |
| 46 | +The `Dockerfile` builds all binaries (manager, ghalistener, webhook server, metrics server, sleep) from Go source into a distroless image. |
| 47 | + |
| 48 | +### Versioning |
| 49 | + |
| 50 | +The image has two version components that serve different purposes: |
| 51 | + |
| 52 | +- **`VERSION` build arg** (compatibility): must match the chart's `appVersion` (e.g., `0.14.1`). See the version check section below for why this is critical. |
| 53 | +- **Docker tag** (identity): tracks fork iterations using the format `<chart_version>-capacity.<N>` (e.g., `0.14.1-capacity.1`). Bump `<N>` for every new build. This is what goes in `deploy.sh`'s `image.tag` and what Harbor stores. |
| 54 | + |
| 55 | +When rebasing the fork onto a new upstream release (e.g., `0.15.0`), bump both the `VERSION` build arg and reset the capacity suffix (e.g., `0.15.0-capacity.1`). |
| 56 | + |
| 57 | +### Controller Version Check (CRITICAL — read before building) |
| 58 | + |
| 59 | +The ARC controller has a **hardcoded version reconciliation loop** that runs on every AutoscalingRunnerSet. On each reconcile, it compares: |
| 60 | + |
| 61 | +- `buildVersion`: compiled into the controller binary via `-ldflags -X build.Version=<VERSION>` (from the `VERSION` build arg) |
| 62 | +- `autoscalingRunnerSetVersion`: the `app.kubernetes.io/version` label on the CR, stamped by the runner scale set Helm chart from `Chart.AppVersion` |
| 63 | + |
| 64 | +If these don't match, **the controller deletes the AutoscalingRunnerSet CR**. This happens silently — Helm shows a successful deploy, but the controller immediately nukes every runner scale set. All listeners, runners, and placeholders disappear. |
| 65 | + |
| 66 | +**Source**: `controllers/actions.github.com/autoscalingrunnerset_controller.go:139-153` |
| 67 | + |
| 68 | +The comparison logic (`apis/actions.github.com/v1alpha1/version.go:IsVersionAllowed()`) allows the following: |
| 69 | + |
| 70 | +| Condition | Result | Example | |
| 71 | +|-----------|--------|---------| |
| 72 | +| `buildVersion == "dev"` | Always allowed | Default if `VERSION` build arg is not set | |
| 73 | +| `buildVersion` starts with `"canary-"` | Always allowed | `canary-test` | |
| 74 | +| Exact string match | Allowed | Both `0.14.1` | |
| 75 | +| Semver major.minor match | Allowed | `0.14.0` controller + `0.14.1` chart | |
| 76 | +| Any other mismatch | **CR deleted** | `v0.0.2` controller + `0.14.1` chart | |
| 77 | + |
| 78 | +**Common pitfalls**: |
| 79 | + |
| 80 | +- Using a `v` prefix (e.g., `v0.14.1`) — the semver parser does not strip the `v`, so parsing fails and the version is rejected |
| 81 | +- Using an unrelated version (e.g., `v0.0.2`) — no match, all runner scale sets deleted |
| 82 | +- There is **no flag, env var, or config option** to disable this check |
| 83 | + |
| 84 | +### Local build and push to Harbor |
| 85 | + |
| 86 | +```bash |
| 87 | +cd /Users/jschmidt/meta/actions-runner-controller |
| 88 | + |
| 89 | +# Build for linux/amd64 (what the cluster runs) |
| 90 | +# VERSION must match the chart's appVersion — DO NOT use an unrelated version |
| 91 | +docker buildx build \ |
| 92 | + --platform linux/amd64 \ |
| 93 | + --build-arg VERSION=0.14.1 \ |
| 94 | + --build-arg COMMIT_SHA=$(git rev-parse HEAD) \ |
| 95 | + -t localhost:30002/osdc/gha-runner-scale-set-controller:0.14.1-capacity.1 \ |
| 96 | + -f Dockerfile \ |
| 97 | + --load . |
| 98 | + |
| 99 | +# Save to tarball for crane push (faster than docker push over port-forward) |
| 100 | +docker save localhost:30002/osdc/gha-runner-scale-set-controller:0.14.1-capacity.1 -o /tmp/arc-controller.tar |
| 101 | + |
| 102 | +# Port-forward to Harbor (if not already running) |
| 103 | +kubectl port-forward -n harbor-system svc/harbor 30002:80 & |
| 104 | + |
| 105 | +# Authenticate crane with Harbor |
| 106 | +HARBOR_PASS=$(kubectl get secret -n harbor-system harbor-admin-password -o jsonpath='{.data.password}' | base64 -d) |
| 107 | +mise exec -- crane auth login localhost:30002 -u admin -p "$HARBOR_PASS" --insecure |
| 108 | + |
| 109 | +# Push via crane (much faster than docker push — deduplicates existing blobs) |
| 110 | +mise exec -- crane push --insecure /tmp/arc-controller.tar localhost:30002/osdc/gha-runner-scale-set-controller:0.14.1-capacity.1 |
| 111 | +``` |
| 112 | + |
| 113 | +**Tagging**: always use the `<chart_version>-capacity.<N>` format. Bump `<N>` for every new build — `IfNotPresent` pull policy means the same tag won't be re-pulled. Never use a static mutable tag like `latest` or `proactive-capacity` in production. Never use a `v` prefix on `VERSION` — the controller's semver parser does not strip it and will fail the version check. |
| 114 | + |
| 115 | +**Why crane instead of docker push**: `docker push` over `kubectl port-forward` is extremely slow and often times out. `crane` is a lightweight registry client that deduplicates blobs and works reliably over the port-forward. It's available via `mise` in the project. |
| 116 | + |
| 117 | +**Note**: The Harbor `osdc` project is auto-created by `modules/arc/deploy.sh` if it does not exist. |
| 118 | + |
| 119 | +## Deploying |
| 120 | + |
| 121 | +```bash |
| 122 | +just deploy-module <cluster> arc |
| 123 | +``` |
| 124 | + |
| 125 | +This runs `modules/arc/deploy.sh`, which: |
| 126 | + |
| 127 | +1. **Ensures Harbor project** `osdc` exists (port-forwards to Harbor, creates via API, 409 = already exists) |
| 128 | +2. **Applies PriorityClasses** from `modules/arc/kubernetes/priority-classes.yaml` (placeholder-runner, arc-runner, placeholder-workflow, arc-workflow) |
| 129 | +3. **Applies RBAC** from `modules/arc/kubernetes/capacity-monitor-rbac.yaml` |
| 130 | +4. **Helm upgrade** of the fork chart: |
| 131 | + - Chart: `oci://ghcr.io/jeanschmidt/actions-runner-controller-charts/gha-runner-scale-set-controller` |
| 132 | + - Version: from `clusters.yaml` `arc.chart_version` (default `0.14.1-jeanschmidt.5`) |
| 133 | + - Image: defaults to `ghcr.io/jeanschmidt/gha-runner-scale-set-controller:<chart_version>`. Override with `arc.image_repository` / `arc.image_tag` in `clusters.yaml` for local Harbor builds. |
| 134 | + |
| 135 | +Other deploy.sh config knobs (all from `clusters.yaml`): |
| 136 | + |
| 137 | +| Key | Default | What | |
| 138 | +|-----|---------|------| |
| 139 | +| `arc.chart_version` | `0.14.1-jeanschmidt.5` | Helm chart version (fork) | |
| 140 | +| `arc.image_repository` | `ghcr.io/jeanschmidt/gha-runner-scale-set-controller` | Controller image repo (override for local Harbor builds) | |
| 141 | +| `arc.image_tag` | _(chart_version)_ | Controller image tag (override for local Harbor builds) | |
| 142 | +| `arc.replica_count` | `2` | Controller replicas | |
| 143 | +| `arc.log_level` | `info` | Log level | |
| 144 | +| `arc.controller_cpu_request` | `1` | CPU request | |
| 145 | +| `arc.controller_cpu_limit` | `4` | CPU limit | |
| 146 | +| `arc.controller_memory_request` | `2Gi` | Memory request | |
| 147 | +| `arc.controller_memory_limit` | `4Gi` | Memory limit | |
| 148 | + |
| 149 | +## Configuration |
| 150 | + |
| 151 | +The capacity monitor is configured via env vars on the listener pod, set in `modules/arc-runners/templates/runner.yaml.tpl`: |
| 152 | + |
| 153 | +| Env Var | Default | Description | |
| 154 | +|---------|---------|-------------| |
| 155 | +| `CAPACITY_AWARE_ENABLED` | `true` | Enable the capacity monitor goroutine | |
| 156 | +| `CAPACITY_AWARE_PROACTIVE_CAPACITY` | `0` | Number of placeholder pairs to maintain ahead of demand | |
| 157 | +| `CAPACITY_AWARE_RECALCULATE_INTERVAL` | `30s` | Fallback reconciliation interval (event-driven is primary) | |
| 158 | +| `CAPACITY_AWARE_PLACEHOLDER_TIMEOUT` | `20m` | How long a placeholder can stay Pending before being deleted | |
| 159 | +| `CAPACITY_AWARE_WORKFLOW_CPU` | _(from runner def)_ | Workflow placeholder CPU request (template: `{{VCPU}}`) | |
| 160 | +| `CAPACITY_AWARE_WORKFLOW_MEMORY` | _(from runner def)_ | Workflow placeholder memory request (template: `{{MEMORY}}`) | |
| 161 | +| `CAPACITY_AWARE_WORKFLOW_GPU` | _(from runner def)_ | Workflow placeholder GPU count (template: `{{GPU_COUNT}}`) | |
| 162 | +| `CAPACITY_AWARE_WORKFLOW_DISK` | _(from runner def)_ | Workflow placeholder disk (template: `{{DISK_SIZE}}`) | |
| 163 | +| `CAPACITY_AWARE_RUNNER_CPU` | `750m` | Runner placeholder CPU request | |
| 164 | +| `CAPACITY_AWARE_RUNNER_MEMORY` | `512Mi` | Runner placeholder memory request | |
| 165 | +| `CAPACITY_AWARE_NODE_FLEET` | _(from runner def)_ | Node fleet for placeholder scheduling (template: `{{NODE_FLEET}}`) | |
| 166 | +| `CAPACITY_AWARE_RUNNER_CLASS` | _(from runner def)_ | Runner class for placeholder node selector (template: `{{RUNNER_CLASS}}`) | |
| 167 | +| `CAPACITY_AWARE_HUD_API_TOKEN` | _(from K8s secret)_ | PyTorch HUD API token for queued job counts | |
| 168 | + |
| 169 | +Currently enabled for all runners (`CAPACITY_AWARE_ENABLED=true` is hardcoded in the template). |
| 170 | + |
| 171 | +## Creating the HUD API Secret |
| 172 | + |
| 173 | +```bash |
| 174 | +kubectl create secret generic pytorch-hud-token \ |
| 175 | + --namespace arc-systems \ |
| 176 | + --from-literal=token='<hud-internal-bot-secret>' |
| 177 | +``` |
| 178 | + |
| 179 | +The template mounts this as optional (`optional: true`), so missing the secret does not prevent pod startup. |
| 180 | + |
| 181 | +## Adding maxRunners to a Runner Definition |
| 182 | + |
| 183 | +Add `max_runners: <value>` to the runner def YAML. Example: |
| 184 | + |
| 185 | +```yaml |
| 186 | +runner: |
| 187 | + name: l-x86iavx512-8-16 |
| 188 | + instance_type: c7a.48xlarge |
| 189 | + vcpu: 8 |
| 190 | + memory: 16Gi |
| 191 | + gpu: 0 |
| 192 | + disk_size: 150 |
| 193 | + max_runners: 100 |
| 194 | +``` |
| 195 | +
|
| 196 | +This flows through the template as `gha_max_runners:` in the generated Helm values. The capacity monitor reads `config.MaxRunners` (set from the listener config) and uses it as the ceiling for `X-ScaleSetMaxCapacity`. Without `max_runners`, the value is empty/unlimited. |
| 197 | + |
| 198 | +Currently no runner definition has `max_runners` set. |
| 199 | + |
| 200 | +## Load Testing the Capacity Monitor |
| 201 | + |
| 202 | +To verify the capacity monitor and HUD integration on arc-staging: |
| 203 | + |
| 204 | +```bash |
| 205 | +just load-test arc-staging --label l-x86iamx-8-16:400 |
| 206 | +``` |
| 207 | + |
| 208 | +**Why `--label l-x86iamx-8-16:400`**: arc-staging runs in us-west-1 where `c7a` instances are not available. The default distribution assigns most jobs to `l-x86iavx512-8-16` (node-fleet `c7a`), which will never schedule. `l-x86iamx-8-16` uses node-fleet `c7i`, which is available in us-west-1. |
| 209 | + |
| 210 | +To exercise multiple runner types in parallel (e.g., CPU + GPU), repeat `--label`: |
| 211 | + |
| 212 | +```bash |
| 213 | +just load-test arc-staging --label l-x86iamx-8-16:400 --label l-x86iavx512-29-115-t4:200 |
| 214 | +``` |
| 215 | + |
| 216 | +**GPU labels in us-west-1**: g5 (A10G) and g6 (L4) fleets have `exclude_regions: [us-west-1]`. Only g4dn (T4) is available — pick from `l-x86iavx512-29-115-t4` (1×T4), `l-x86iavx512-45-172-t4-4` (4×T4), or `l-bx86iavx512-94-344-t4-8` (8×T4, bare-metal). |
| 217 | + |
| 218 | +`--label` and `--jobs` are mutually exclusive. Without `--label`, `--jobs N` distributes proportionally across all available runner types. |
| 219 | + |
| 220 | +**Verifying the capacity monitor is working**: |
| 221 | + |
| 222 | +```bash |
| 223 | +# Check listener startup logs for "Capacity monitor enabled" and "Starting capacity monitor" |
| 224 | +kubectl logs -n arc-systems <listener-pod> | head -20 |
| 225 | +
|
| 226 | +# Check reconciliation logs (every 30s) |
| 227 | +kubectl logs -n arc-systems <listener-pod> | grep "capacity reconciled" |
| 228 | +
|
| 229 | +# Test HUD API directly |
| 230 | +curl -s 'https://hud.pytorch.org/api/clickhouse/queued_jobs_aggregate?parameters=%7B%22queuedThresholdMinutes%22%3A0%2C%22maxAgeDays%22%3A3%2C%22orgs%22%3A%5B%22pytorch%22%5D%2C%22repo%22%3A%22%22%7D' | python3 -m json.tool | head -20 |
| 231 | +``` |
| 232 | + |
| 233 | +If the capacity monitor is NOT starting despite `CAPACITY_AWARE_ENABLED=true`, the controller image likely needs to be rebuilt — the running binary may predate the capacity monitor code. |
| 234 | + |
| 235 | +## Maintenance |
| 236 | + |
| 237 | +On ARC upgrades: |
| 238 | +1. Check if `cmd/ghalistener/main.go` changed (the entry point wiring -- our changes are in the capacity monitor errgroup block) |
| 239 | +2. Check if `github.com/actions/scaleset` changed (specifically `listener.SetMaxRunners()` and `listener.Config`) |
| 240 | +3. Rebase the fork branch `jeanschmidt/placeholder_run_poc` onto upstream master |
| 241 | +4. Rebuild and push the image to Harbor |
| 242 | + |
| 243 | +The `capacity/` package is entirely ours -- no upstream merge conflicts possible. The fork surface is one modified file (`main.go`) plus the new `capacity/` package (4 files). |
0 commit comments