Skip to content

Commit a960b38

Browse files
authored
Pin CoreDNS replica count and topology spread constraints (#547)
**Impact:** All EKS clusters (production and staging) — CoreDNS pod placement and scaling **Risk:** medium ## What Pins CoreDNS replica count, topology spread constraints, and PDB configuration via the EKS addon `configuration_values`, and explicitly disables the cluster-proportional-autoscaler. Replica count is driven per-cluster from `clusters.yaml` (default 6, staging 2). ## Why Without explicit topology constraints, CoreDNS pods can end up pinned to a single AZ (e.g. 6-0-0) after node rolls or AMI upgrades, creating a DNS single-point-of-failure. The EKS addon's built-in autoscaler can also silently re-enable across addon version bumps, overriding the intended replica count. This change makes the CoreDNS topology deterministic and auditable through infrastructure-as-code. ## How - Replica count is a new `coredns_replicas` Terraform variable threaded from `clusters.yaml` through `cluster-config.py` → tofu vars → the EKS module, with a `>= 2` validation gate (single-replica deadlocks the PDB; zero means no DNS). - Autoscaling is explicitly disabled (`autoScaling.enabled = false`) as a defensive measure against addon-version flips re-enabling the cluster-proportional-autoscaler. - Two topology spread constraints are set: a **hard** zone spread (`DoNotSchedule`, `maxSkew=2`) to prevent catastrophic single-AZ pinning while tolerating AWS subnet placement drift, and a **soft** hostname spread (`ScheduleAnyway`, `maxSkew=1`) to prefer distributing across nodes. - PDB is pinned to `maxUnavailable=1` (matches the addon default, but now explicit). - The existing `CriticalAddonsOnly` toleration is preserved — `configuration_values` replaces (not merges) the addon defaults, so it must be re-stated. ## Changes - **`clusters.yaml`**: Add `coredns.replicas` default (6) and staging override (2) - **`modules/eks/terraform/modules/eks/main.tf`**: Expand CoreDNS addon `configuration_values` with `replicaCount`, `autoScaling`, `topologySpreadConstraints`, and `podDisruptionBudget` - **`modules/eks/terraform/modules/eks/variables.tf`**: Add `coredns_replicas` variable with `>= 2` validation - **`modules/eks/terraform/variables.tf`**: Mirror `coredns_replicas` variable at root module level - **`modules/eks/terraform/main.tf`**: Thread `coredns_replicas` from root to inner EKS module - **`scripts/cluster-config.py`**: Resolve `coredns.replicas` from clusters.yaml and emit as `-var="coredns_replicas=..."` in tfvars output - **`scripts/test_cluster_config.py`**: Add unit tests for coredns_replicas tfvar resolution (default fallback, defaults section, cluster override) - **`modules/eks/tests/smoke/test_eks.py`**: Add `TestCoreDNSTopology` smoke test class verifying replica count, hard zone spread, soft hostname spread, no CPA deployment, and PDB maxUnavailable=1 ## Testing ``` ============================================================ OSDC Integration Test Results ============================================================ Cluster: arc-staging (pytorch-arc-staging) Date: 2026-05-09 00:58 UTC PR Workflow Jobs: ✓ test-pypi-cache-action-cuda success ✓ test-cpu-x86-avx512 success ✓ test-pypi-cache-action-cpu success ✓ test-pypi-cache-defaults success ✓ test-cpu-arm64 success ✓ test-cpu-x86-amx success ✓ test-git-cache success ✓ test-gpu-t4-multi success ✓ test-release-arm64 success ✓ test-gpu-t4 success ✓ test-harbor success ✓ test-cache-enforcer success ✓ build-arm64 / build success ✓ build-amd64 / build success Smoke ⊘ SKIPPED Compactor ⊘ SKIPPED Overall: PASSED ============================================================ ``` ``` ============================================================================================================================ test session starts ============================================================================================================================ platform darwin -- Python 3.13.12, pytest-9.0.2, pluggy-1.6.0 rootdir: /Users/jschmidt/meta/ci-infra/osdc configfile: pyproject.toml plugins: anyio-4.12.1, xdist-3.8.0, cov-7.0.0 16 workers [237 items] .......................................................................s..........................................................................................................................s............s............................. [100%] ========================================================================================================================== short test summary info ========================================================================================================================== SKIPPED [1] modules/arc-runners/tests/smoke/test_capacity_parity.py:300: no scale sets with proactive_capacity > 0 SKIPPED [1] modules/cache-enforcer/tests/smoke/test_cache_enforcer.py:107: No cache-enforcer pods desired (no runner nodes in cluster) SKIPPED [1] modules/monitoring/tests/smoke/test_monitoring.py:173: No dcgm-exporter pods found (no GPU nodes) ====================================================================================================================== 234 passed, 3 skipped in 34.00s ====================================================================================================================== Smoke tests completed in 35s ``` Signed-off-by: Jean Schmidt <contato@jschmidt.me>
1 parent e2867e1 commit a960b38

8 files changed

Lines changed: 204 additions & 1 deletion

File tree

osdc/clusters.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,10 @@ defaults:
3131
eks_version: "1.35"
3232
single_nat_gateway: false
3333

34+
# --- CoreDNS sizing ---
35+
coredns:
36+
replicas: 6
37+
3438
# --- Git cache central ---
3539
git_cache:
3640
central_replicas: 2
@@ -123,6 +127,8 @@ clusters:
123127
base_node_max_unavailable_percentage: 100 # all-at-once for staging
124128
single_nat_gateway: true # cost optimization for staging
125129
vpc_cidr: "10.0.0.0/16"
130+
coredns:
131+
replicas: 2
126132
harbor:
127133
core_replicas: 1
128134
registry_replicas: 1

osdc/modules/eks/terraform/main.tf

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ module "eks" {
7878
base_node_max_unavailable_percentage = var.base_node_max_unavailable_percentage
7979
base_node_ami_version = var.base_node_ami_version
8080

81+
coredns_replicas = var.coredns_replicas
82+
8183
authentication_mode = var.authentication_mode
8284
cluster_admin_role_names = var.cluster_admin_role_names
8385

osdc/modules/eks/terraform/modules/eks/main.tf

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,19 @@ resource "aws_eks_addon" "coredns" {
178178
addon_version = "v1.13.2-eksbuild.1"
179179
resolve_conflicts_on_update = "PRESERVE"
180180

181-
# CoreDNS must tolerate base node taints to run on infrastructure nodes
181+
# CoreDNS pinned topology:
182+
# - replicaCount: per-cluster from clusters.yaml (default 6, staging 2)
183+
# - autoScaling explicitly disabled (defensive against addon-version flips)
184+
# - Hard zone spread (DoNotSchedule, maxSkew=2) + soft hostname spread (ScheduleAnyway, maxSkew=1)
185+
# - PDB maxUnavailable=1 (matches addon default)
186+
# - Required so CoreDNS pods land on the CriticalAddonsOnly-tainted base nodes.
187+
# Setting tolerations REPLACES (does not merge with) the addon default —
188+
# the addon's default tolerations include this one, so we preserve compat.
182189
configuration_values = jsonencode({
190+
replicaCount = var.coredns_replicas
191+
autoScaling = {
192+
enabled = false
193+
}
183194
tolerations = [
184195
{
185196
key = "CriticalAddonsOnly"
@@ -188,6 +199,33 @@ resource "aws_eks_addon" "coredns" {
188199
effect = "NoSchedule"
189200
}
190201
]
202+
topologySpreadConstraints = [
203+
{
204+
# maxSkew=2 tolerates AWS subnet placement drift (e.g. 4-2-0 across AZs after
205+
# AMI rolls) while still preventing catastrophic 6-0-0 single-AZ pinning.
206+
maxSkew = 2
207+
topologyKey = "topology.kubernetes.io/zone"
208+
whenUnsatisfiable = "DoNotSchedule"
209+
labelSelector = {
210+
matchLabels = {
211+
"k8s-app" = "kube-dns"
212+
}
213+
}
214+
},
215+
{
216+
maxSkew = 1
217+
topologyKey = "kubernetes.io/hostname"
218+
whenUnsatisfiable = "ScheduleAnyway"
219+
labelSelector = {
220+
matchLabels = {
221+
"k8s-app" = "kube-dns"
222+
}
223+
}
224+
}
225+
]
226+
podDisruptionBudget = {
227+
maxUnavailable = 1
228+
}
191229
})
192230

193231
tags = var.tags

osdc/modules/eks/terraform/modules/eks/variables.tf

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,17 @@ variable "base_node_count" {
5454
default = 5
5555
}
5656

57+
variable "coredns_replicas" {
58+
description = "Number of CoreDNS replicas (pinned; autoscaling disabled). Per-cluster via clusters.yaml."
59+
type = number
60+
default = 6
61+
62+
validation {
63+
condition = var.coredns_replicas >= 2
64+
error_message = "coredns_replicas must be >= 2 (single-replica CoreDNS deadlocks the PDB; zero replicas means no DNS)."
65+
}
66+
}
67+
5768
variable "base_node_instance_type" {
5869
description = "Instance type for base infrastructure nodes"
5970
type = string

osdc/modules/eks/terraform/variables.tf

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,17 @@ variable "base_node_count" {
2929
default = 3
3030
}
3131

32+
variable "coredns_replicas" {
33+
description = "Number of CoreDNS replicas (pinned; autoscaling disabled). Per-cluster via clusters.yaml."
34+
type = number
35+
default = 6
36+
37+
validation {
38+
condition = var.coredns_replicas >= 2
39+
error_message = "coredns_replicas must be >= 2 (single-replica CoreDNS deadlocks the PDB; zero replicas means no DNS)."
40+
}
41+
}
42+
3243
variable "base_node_instance_type" {
3344
description = "Instance type for base infrastructure nodes"
3445
type = string

osdc/modules/eks/tests/smoke/test_eks.py

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
"""Smoke tests for EKS cluster and AWS infrastructure."""
22

3+
import subprocess
4+
35
import pytest
46
import yaml
57
from helpers import get_unstable_node_names, pod_is_on_unstable_node, run_aws, run_kubectl
@@ -158,3 +160,107 @@ def test_ecr_repos_exist(self, cluster_config, upstream_dir):
158160

159161
missing = [repo for repo in expected_repos if repo not in ecr_repo_names]
160162
assert not missing, f"ECR repositories missing for bootstrap images: {missing}"
163+
164+
165+
# ============================================================================
166+
# CoreDNS Topology Pinning
167+
# ============================================================================
168+
169+
170+
class TestCoreDNSTopology:
171+
"""Verify CoreDNS replicaCount, topology spread, autoscaling, and PDB.
172+
173+
Pinned via aws_eks_addon.coredns configuration_values in
174+
modules/eks/terraform/modules/eks/main.tf. Replica count is per-cluster
175+
via clusters.yaml (coredns.replicas — default 6, staging 2).
176+
"""
177+
178+
def test_replica_count_matches_clusters_yaml(self, resolve_config):
179+
expected = resolve_config("coredns.replicas", 6)
180+
deploy = run_kubectl(["get", "deployment", "coredns"], namespace="kube-system")
181+
actual = deploy.get("spec", {}).get("replicas")
182+
assert actual == expected, (
183+
f"CoreDNS Deployment.spec.replicas={actual} does not match clusters.yaml expected={expected}"
184+
)
185+
186+
def test_zone_topology_spread_is_hard(self, resolve_config):
187+
deploy = run_kubectl(["get", "deployment", "coredns"], namespace="kube-system")
188+
constraints = deploy.get("spec", {}).get("template", {}).get("spec", {}).get("topologySpreadConstraints", [])
189+
zone_rules = [
190+
c
191+
for c in constraints
192+
if c.get("topologyKey") == "topology.kubernetes.io/zone" and c.get("whenUnsatisfiable") == "DoNotSchedule"
193+
]
194+
assert zone_rules, (
195+
f"CoreDNS Deployment missing hard zone spread (topology.kubernetes.io/zone, DoNotSchedule). "
196+
f"Found constraints: {constraints}"
197+
)
198+
# maxSkew=2 tolerates AWS subnet placement drift (e.g. 4-2-0 across AZs after
199+
# AMI rolls) while still preventing catastrophic 6-0-0 single-AZ pinning.
200+
for rule in zone_rules:
201+
assert rule.get("maxSkew") == 2, (
202+
f"CoreDNS zone spread maxSkew={rule.get('maxSkew')}, expected 2. Constraint: {rule}"
203+
)
204+
assert rule.get("labelSelector", {}).get("matchLabels", {}).get("k8s-app") == "kube-dns", (
205+
f"CoreDNS zone spread labelSelector mismatch. Constraint: {rule}"
206+
)
207+
208+
def test_hostname_topology_spread_is_soft(self, resolve_config):
209+
"""Soft hostname spread (ScheduleAnyway) keeps replicas off the same node when possible."""
210+
deploy = run_kubectl(["get", "deployment", "coredns"], namespace="kube-system")
211+
constraints = deploy.get("spec", {}).get("template", {}).get("spec", {}).get("topologySpreadConstraints", [])
212+
host_rules = [
213+
c
214+
for c in constraints
215+
if c.get("topologyKey") == "kubernetes.io/hostname" and c.get("whenUnsatisfiable") == "ScheduleAnyway"
216+
]
217+
assert host_rules, (
218+
f"CoreDNS Deployment missing soft hostname spread (kubernetes.io/hostname, ScheduleAnyway). "
219+
f"Found constraints: {constraints}"
220+
)
221+
for rule in host_rules:
222+
assert rule.get("maxSkew") == 1, (
223+
f"CoreDNS hostname spread maxSkew={rule.get('maxSkew')}, expected 1. Constraint: {rule}"
224+
)
225+
assert rule.get("labelSelector", {}).get("matchLabels", {}).get("k8s-app") == "kube-dns", (
226+
f"CoreDNS hostname spread labelSelector mismatch. Constraint: {rule}"
227+
)
228+
229+
def test_no_cluster_proportional_autoscaler(self, cluster_config):
230+
"""Addon-managed autoscaling must be off — no CPA Deployment should exist for coredns.
231+
232+
EKS CoreDNS uses the cluster-proportional-autoscaler (CPA), which runs as a
233+
Deployment named 'coredns-autoscaler' (newer addon versions) or 'eks-coredns-autoscaler'
234+
(older spelling). It is NOT a HorizontalPodAutoscaler. With autoScaling.enabled=false
235+
in the addon configuration_values, neither Deployment should be present.
236+
"""
237+
cluster_name = cluster_config["cluster"]["cluster_name"]
238+
for cpa_name in ("coredns-autoscaler", "eks-coredns-autoscaler"):
239+
try:
240+
deploy = run_kubectl(["get", "deployment", cpa_name], namespace="kube-system")
241+
except subprocess.CalledProcessError:
242+
# 'NotFound' surfaces as kubectl exit 1 — that's the desired state.
243+
continue
244+
# If we got JSON back, the CPA Deployment exists — that's a regression.
245+
pytest.fail(
246+
f"Unexpected cluster-proportional-autoscaler Deployment '{cpa_name}' in kube-system "
247+
f"on {cluster_name} (autoScaling.enabled=false should prevent this): {deploy}"
248+
)
249+
250+
def test_pdb_max_unavailable_is_one(self):
251+
pdbs = run_kubectl(["get", "poddisruptionbudgets"], namespace="kube-system")
252+
items = pdbs.get("items", [])
253+
coredns_pdbs = [
254+
p
255+
for p in items
256+
if p.get("spec", {}).get("selector", {}).get("matchLabels", {}).get("k8s-app") == "kube-dns"
257+
]
258+
assert coredns_pdbs, f"No PodDisruptionBudget found targeting CoreDNS (k8s-app=kube-dns). Items: {items}"
259+
# EKS addon picks one PDB resource name (typically 'coredns'); assert all
260+
# found PDBs have maxUnavailable=1 (defensive against multiple stale PDBs).
261+
for pdb in coredns_pdbs:
262+
spec = pdb.get("spec", {})
263+
max_unavail = spec.get("maxUnavailable")
264+
assert max_unavail == 1, (
265+
f"CoreDNS PDB {pdb['metadata']['name']} has maxUnavailable={max_unavail}, expected 1"
266+
)

osdc/scripts/cluster-config.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,11 @@ def resolve(cluster_cfg, defaults, dotpath):
5252
def tfvars(cluster_id, cluster_cfg, defaults):
5353
"""Produce -var flags for tofu from cluster config."""
5454
base = {**defaults, **(cluster_cfg.get("base") or {})}
55+
# Resolve nested config (e.g. coredns.replicas) using the same lookup
56+
# rules as resolve(): cluster value wins; otherwise defaults.
57+
coredns_replicas = resolve(cluster_cfg, defaults, "coredns.replicas")
58+
if coredns_replicas is None:
59+
coredns_replicas = 6
5560
pairs = {
5661
"cluster_name": cluster_cfg["cluster_name"],
5762
"aws_region": cluster_cfg["region"],
@@ -62,6 +67,7 @@ def tfvars(cluster_id, cluster_cfg, defaults):
6267
"base_node_max_unavailable_percentage": base.get("base_node_max_unavailable_percentage", 33),
6368
"base_node_ami_version": base.get("base_node_ami_version", defaults.get("base_node_ami_version", "v*")),
6469
"eks_version": base.get("eks_version", defaults.get("eks_version", "1.35")),
70+
"coredns_replicas": coredns_replicas,
6571
}
6672
# Optional fields — only emit if explicitly set
6773
access_config = cluster_cfg.get("access_config") or base.get("access_config") or {}

osdc/scripts/test_cluster_config.py

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,29 @@ def test_all_defaults(self, capsys):
166166
assert '-var="base_node_max_unavailable_percentage=33"' in out
167167
assert '-var="base_node_ami_version=v*"' in out
168168
assert '-var="eks_version=1.35"' in out
169+
# coredns.replicas not set anywhere — falls back to hardcoded default 6
170+
assert '-var="coredns_replicas=6"' in out
171+
172+
def test_coredns_replicas_from_defaults(self, capsys):
173+
cluster_cfg = {
174+
"cluster_name": "c",
175+
"region": "r",
176+
}
177+
defaults = {"coredns": {"replicas": 4}}
178+
cluster_config.tfvars("c", cluster_cfg, defaults)
179+
out = capsys.readouterr().out.strip()
180+
assert '-var="coredns_replicas=4"' in out
181+
182+
def test_coredns_replicas_cluster_override(self, capsys):
183+
cluster_cfg = {
184+
"cluster_name": "c",
185+
"region": "r",
186+
"coredns": {"replicas": 2},
187+
}
188+
defaults = {"coredns": {"replicas": 6}}
189+
cluster_config.tfvars("c", cluster_cfg, defaults)
190+
out = capsys.readouterr().out.strip()
191+
assert '-var="coredns_replicas=2"' in out
169192

170193
def test_cluster_overrides(self, capsys):
171194
cluster_cfg = {

0 commit comments

Comments
 (0)