Skip to content

Commit 9b084c3

Browse files
jeanschmidtjeanschmidt
authored
Add NodeLocal DNSCache as a base component (#548)
**Impact:** All clusters — every node gets a per-node DNS cache DaemonSet **Risk:** medium Part of the migration: #544 ## What Deploys [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) (NLD) as a base Kubernetes component on every OSDC cluster. Each node runs a local CoreDNS instance that intercepts pod DNS queries via iptables NOTRACK rules, caching responses and forwarding cache misses to cluster CoreDNS over persistent TCP. ## Why High-concurrency CI runners generate significant DNS traffic (pip installs, container pulls, API calls). All of it currently funnels through the cluster-wide kube-dns CoreDNS pods, creating a centralized bottleneck. NLD absorbs the bulk of repeated lookups at the node level, reducing cluster CoreDNS QPS, eliminating conntrack overhead for DNS, and lowering per-pod DNS latency. ## How - **iptables-mode (not `--skip-teardown`)**: NLD binds the kube-dns ClusterIP on a dummy `nodelocaldns` interface and installs NOTRACK iptables rules so pods' DNS traffic (destined for the kube-dns ClusterIP) is silently intercepted by the local cache. No pod or kubelet config changes needed. - **`deploy.sh` with runtime substitution**: The kube-dns ClusterIP varies per cluster (dynamic service CIDR). `deploy.sh` resolves it at apply time and sed-substitutes the `__KUBE_DNS_CLUSTER_IP__` placeholder in rendered manifests. A Python splitter applies Services before the DaemonSet to ensure kubelet injects `KUBE_DNS_UPSTREAM_SERVICE_HOST` env into node-cache pods. - **`kube-dns-upstream` Service**: A second Service fronting the same kube-dns pods, giving NLD a separate ClusterIP to forward cache misses to without creating an iptables loop through itself. - **No memory/CPU limits**: OOMKill orphans iptables rules (cluster-wide DNS degradation); CPU throttling adds DNS latency. `system-node-critical` priorityClass prevents eviction. - **`automountServiceAccountToken: false`**: The binary makes no API calls; disabling the token reduces credential-leak surface on a privileged container. - **Two metrics ports**: `:9253` (CoreDNS plugin metrics — cache hits, forward latency) and `:9353` (binary-emitted `coredns_nodecache_*` counters including `setup_errors_total`). Both scraped via a single PodMonitor with cardinality-reducing `metricRelabelings` (drop `go_*`, `process_*`, histogram buckets). - **First-deploy idempotency**: `deploy.sh` detects whether `kube-dns-upstream` already exists. On first deploy it triggers a rollout restart so pods pick up the upstream Service env var; on re-runs it skips the restart. ## Changes **Base component (`base/kubernetes/nodelocaldns/`)** - `configmap.yaml` — Corefile with four server blocks (cluster.local, in-addr.arpa, ip6.arpa, catch-all), `force_tcp` for cluster zones, external queries forwarded to `/etc/resolv.conf` - `daemonset.yaml` — DaemonSet with hostNetwork, privileged, system-node-critical, tolerations for all OSDC node taints (GPU, node-fleet, instance-type, git-cache-not-ready), image `k8s-dns-node-cache:1.26.8` - `upstream-service.yaml` — `kube-dns-upstream` Service selecting kube-dns pods (NLD forward target) - `metrics-service.yaml` — headless `node-local-dns-metrics` Service for PodMonitor discovery (ports 9253 + 9353) - `serviceaccount.yaml` — dedicated SA with `automountServiceAccountToken: false` - `kustomization.yaml` — Kustomize overlay for the component - `deploy.sh` — orchestrates: precondition check → ClusterIP resolution → kustomize render → sed substitution → Services-first apply → DaemonSet apply → conditional rollout restart **Deployment integration** - `justfile` — `deploy-base` recipe calls `nodelocaldns/deploy.sh` after image-cache-janitor - `base/kubernetes/kustomization.yaml` — comment noting nodelocaldns uses its own deploy.sh (not inline kustomize) **Monitoring** - `podmonitors/nodelocaldns.yaml` — PodMonitor scraping both metrics ports at 60s, dropping `go_*`, `process_*`, and histogram buckets - `nodelocaldns-alerts.yaml` — three PrometheusRule alerts: `NodeLocalDNSSetupErrors` (critical, iptables failures), `NodeLocalDNSPodRestarting` (warning), `NodeLocalDNSDaemonSetDegraded` (warning, unavailable pods >15m) - Alert and monitor kustomizations updated **Tests** - `test_base_kubernetes.py` — three new smoke tests: DaemonSet healthy on all nodes, metrics service is headless, upstream service selects kube-dns pods **Documentation** - `architecture.md` — base resources list and deploy sequence updated - `observability.md` — PodMonitor table and alert table updated - `CLAUDE.md` — `osdc-nodelocaldns` skill reference added ## Notes - **Soak gate**: After initial deploy, monitor `coredns_nodecache_setup_errors_total` and cache hit rate for 24-48h before considering NLD stable. The `NodeLocalDNSSetupErrors` alert (critical) will fire immediately if iptables rule installation fails on any node. - **Rolling update at 1% maxUnavailable**: Conservative to avoid many nodes simultaneously losing DNS interception during upgrades. - **External DNS bypass**: The catch-all `.:53` block forwards to `/etc/resolv.conf`, so external lookups (e.g., `download.pytorch.org`) bypass cluster CoreDNS entirely — they go through the node cache but miss → VPC resolver, not kube-dns. ## Testing ``` ============================================================ OSDC Integration Test Results ============================================================ Cluster: arc-staging (pytorch-arc-staging) Date: 2026-05-09 01:27 UTC PR Workflow Jobs: ✓ test-pypi-cache-defaults success ✓ test-cpu-arm64 success ✓ test-pypi-cache-action-cuda success ✓ test-cpu-x86-amx success ✓ test-pypi-cache-action-cpu success ✓ test-gpu-t4-multi success ✓ test-cpu-x86-avx512 success ✓ test-cache-enforcer success ✓ test-release-arm64 success ✓ test-gpu-t4 success ✓ test-harbor success ✓ test-git-cache success ✓ build-amd64 / build success ✓ build-arm64 / build success Smoke ⊘ SKIPPED Compactor ⊘ SKIPPED Overall: PASSED ============================================================ ``` ``` ============================================================================================================================ test session starts ============================================================================================================================ platform darwin -- Python 3.13.12, pytest-9.0.2, pluggy-1.6.0 rootdir: /Users/jschmidt/meta/ci-infra-code-review/osdc configfile: pyproject.toml plugins: anyio-4.12.1, xdist-3.8.0, cov-7.0.0 16 workers [235 items] ......................................................s.................................................................................................s............................................s..................................... [100%] ========================================================================================================================== short test summary info ========================================================================================================================== SKIPPED [1] modules/arc-runners/tests/smoke/test_capacity_parity.py:300: no scale sets with proactive_capacity > 0 SKIPPED [1] modules/cache-enforcer/tests/smoke/test_cache_enforcer.py:107: No cache-enforcer pods desired (no runner nodes in cluster) SKIPPED [1] modules/monitoring/tests/smoke/test_monitoring.py:173: No dcgm-exporter pods found (no GPU nodes) ================================================================================================================= 232 passed, 3 skipped in 92.41s (0:01:32) ================================================================================================================= Smoke tests completed in 1m34s ``` ``` 18:19  aws:308535385114  12  ~  meta  ci-infra-code-review  osdc  jeanschmidt/dnscache_daemonset_2  ? ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── $  kubectl get ds node-local-dns -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE node-local-dns 2 2 2 2 2 <none> 13m 18:19  aws:308535385114  12  ~  meta  ci-infra-code-review  osdc  jeanschmidt/dnscache_daemonset_2  ? ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── $  kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES node-local-dns-8g5km 1/1 Running 0 13m 10.0.60.247 ip-10-0-60-247.us-west-1.compute.internal <none> <none> node-local-dns-r8pbv 1/1 Running 0 13m 10.0.89.138 ip-10-0-89-138.us-west-1.compute.internal <none> <none> ``` --------- Signed-off-by: Jean Schmidt <contato@jschmidt.me>
1 parent 5cadb9c commit 9b084c3

22 files changed

Lines changed: 614 additions & 14 deletions

‎osdc/CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ Detailed instructions are broken into on-demand skills. Load the relevant skill
5555
| `osdc-cli-debugging` | Read-only kubectl, aws, helm, tofu commands and safety boundaries | Investigating cluster state, debugging pods |
5656
| `osdc-harbor` | Harbor Helm chart gotchas, image mirroring, proxy cache configuration | Working on Harbor or container registry config |
5757
| `osdc-pypi-cache` | PyPI wheel cache module — per-CUDA nginx+pypiserver fanout, EFS wheelhouse, S3 wheel pipeline, slug naming, prebuilt-cache.txt, NetworkPolicy, IRSA | Working on the pypi-cache module or debugging pip install failures on runners |
58+
| `osdc-nodelocaldns` | NodeLocal DNSCache base component — iptables-mode rationale, kube-dns ClusterIP substitution, Service-before-DaemonSet ordering, two metrics ports (9253/9353), `coredns_nodecache_*` prefix, soak gate | Working on the nodelocaldns base component or debugging DNS issues on runner nodes |
5859

5960
Load the relevant `osdc-*` skill when you need detailed instructions on any specific topic.
6061

‎osdc/base/kubernetes/kustomization.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,5 +15,6 @@ resources:
1515
- registry-mirror-config.yaml
1616
- algif-mitigation.yaml
1717
# image-cache-janitor is deployed via its own deploy.sh (builds custom image)
18+
# nodelocaldns is deployed via its own deploy.sh (resolves kube-dns ClusterIP at apply time)
1819
# NOTE: Namespaces for modules (arc-runners, arc-systems, buildkit, etc.)
1920
# are created by the module's own kubernetes/ directory, not here.

‎osdc/base/kubernetes/nodelocaldns/configmap.yaml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Corefile for NodeLocal DNSCache.
2+
# - __PILLAR__CLUSTER__DNS__ is substituted at runtime by the
3+
# k8s-dns-node-cache binary from the kube-dns-upstream Service ClusterIP
4+
# (resolved via the -upstreamsvc flag in the DaemonSet args).
5+
# - __KUBE_DNS_CLUSTER_IP__ is OUR placeholder; deploy.sh sed-substitutes
6+
# the live kube-dns Service ClusterIP at apply time (cluster service CIDR
7+
# is dynamic, cannot be hardcoded).
8+
# - force_tcp on cluster.local/in-addr.arpa/ip6.arpa: persistent TCP back to
9+
# cluster CoreDNS reduces conntrack churn and improves head-of-line behavior.
10+
# - .:53 (catch-all) forwards to /etc/resolv.conf so external lookups bypass
11+
# cluster CoreDNS entirely (upstream default).
12+
# - health 169.254.20.10:8080 is declared ONLY on cluster.local:53. The health
13+
# plugin opens a singleton listener; declaring it in a second server block
14+
# would race for the same address and risk "address already in use" at
15+
# CoreDNS startup. Matches upstream nodelocaldns.yaml.
16+
apiVersion: v1
17+
kind: ConfigMap
18+
metadata:
19+
name: node-local-dns
20+
namespace: kube-system
21+
labels:
22+
k8s-app: node-local-dns
23+
app.kubernetes.io/managed-by: osdc-base
24+
data:
25+
Corefile: |
26+
cluster.local:53 {
27+
errors
28+
cache {
29+
success 9984 30
30+
denial 9984 5
31+
}
32+
reload
33+
loop
34+
bind 169.254.20.10 __KUBE_DNS_CLUSTER_IP__
35+
forward . __PILLAR__CLUSTER__DNS__ {
36+
force_tcp
37+
}
38+
prometheus :9253
39+
health 169.254.20.10:8080
40+
}
41+
in-addr.arpa:53 {
42+
errors
43+
cache 30
44+
reload
45+
loop
46+
bind 169.254.20.10 __KUBE_DNS_CLUSTER_IP__
47+
forward . __PILLAR__CLUSTER__DNS__ {
48+
force_tcp
49+
}
50+
prometheus :9253
51+
}
52+
ip6.arpa:53 {
53+
errors
54+
cache 30
55+
reload
56+
loop
57+
bind 169.254.20.10 __KUBE_DNS_CLUSTER_IP__
58+
forward . __PILLAR__CLUSTER__DNS__ {
59+
force_tcp
60+
}
61+
prometheus :9253
62+
}
63+
.:53 {
64+
errors
65+
cache 30
66+
reload
67+
loop
68+
bind 169.254.20.10 __KUBE_DNS_CLUSTER_IP__
69+
forward . /etc/resolv.conf
70+
prometheus :9253
71+
}

‎osdc/base/kubernetes/nodelocaldns/daemonset.yaml‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# NodeLocal DNSCache DaemonSet: per-node CoreDNS instance that absorbs
2+
# DNS queries from local pods via iptables NOTRACK interception.
3+
#
4+
# Design notes (see ~/meta/actions-knowledge-base/docs/osdc/ for full rationale):
5+
# - hostNetwork + privileged: NLD manipulates iptables, the dummy nodelocaldns
6+
# interface, and binds the link-local IP. NET_ADMIN alone does not cover
7+
# all of these on AL2023.
8+
# - dnsPolicy: Default (NOT ClusterFirstWithHostNet): NLD must NOT depend on
9+
# itself for DNS resolution. Default uses the host's resolv.conf.
10+
# - No memory limit: OOMKill orphans iptables → cluster-wide DNS degradation.
11+
# system-node-critical priorityClass already prevents eviction.
12+
# - No CPU limit: throttling = DNS latency.
13+
# - terminationGracePeriodSeconds: 30 matches upstream NLD default; gives the
14+
# binary's signal-handler TeardownNetworking() time to delete iptables NOTRACK
15+
# rules and the dummy nodelocaldns interface before kubelet kills the pod.
16+
# Spec §4.9 had 5s but that risks orphaned rules on slow nodes during rolling
17+
# restart.
18+
# - __KUBE_DNS_CLUSTER_IP__ is substituted by deploy.sh at apply time.
19+
apiVersion: apps/v1
20+
kind: DaemonSet
21+
metadata:
22+
name: node-local-dns
23+
namespace: kube-system
24+
labels:
25+
k8s-app: node-local-dns
26+
app.kubernetes.io/managed-by: osdc-base
27+
spec:
28+
selector:
29+
matchLabels:
30+
k8s-app: node-local-dns
31+
updateStrategy:
32+
type: RollingUpdate
33+
rollingUpdate:
34+
maxUnavailable: 1%
35+
template:
36+
metadata:
37+
labels:
38+
k8s-app: node-local-dns
39+
spec:
40+
priorityClassName: system-node-critical
41+
serviceAccountName: node-local-dns
42+
hostNetwork: true
43+
dnsPolicy: Default
44+
terminationGracePeriodSeconds: 30
45+
tolerations:
46+
- key: CriticalAddonsOnly
47+
operator: Equal
48+
value: "true"
49+
effect: NoSchedule
50+
- key: nvidia.com/gpu
51+
operator: Exists
52+
effect: NoSchedule
53+
- key: node-fleet
54+
operator: Exists
55+
effect: NoSchedule
56+
- key: instance-type
57+
operator: Exists
58+
effect: NoSchedule
59+
- key: git-cache-not-ready
60+
operator: Exists
61+
effect: NoSchedule
62+
containers:
63+
- name: node-cache
64+
image: registry.k8s.io/dns/k8s-dns-node-cache:1.26.8
65+
imagePullPolicy: IfNotPresent
66+
args:
67+
- "-localip"
68+
- "169.254.20.10,__KUBE_DNS_CLUSTER_IP__"
69+
- "-conf"
70+
- "/etc/Corefile"
71+
- "-upstreamsvc"
72+
- "kube-dns-upstream"
73+
securityContext:
74+
privileged: true
75+
resources:
76+
requests:
77+
cpu: 25m
78+
memory: 100Mi
79+
ports:
80+
- name: dns
81+
containerPort: 53
82+
protocol: UDP
83+
- name: dns-tcp
84+
containerPort: 53
85+
protocol: TCP
86+
- name: metrics
87+
containerPort: 9253
88+
protocol: TCP
89+
- name: metrics-binary
90+
containerPort: 9353
91+
protocol: TCP
92+
livenessProbe:
93+
httpGet:
94+
host: 169.254.20.10
95+
path: /health
96+
port: 8080
97+
initialDelaySeconds: 60
98+
periodSeconds: 15
99+
timeoutSeconds: 5
100+
failureThreshold: 5
101+
volumeMounts:
102+
- name: xtables-lock
103+
mountPath: /run/xtables.lock
104+
readOnly: false
105+
- name: kube-dns-config
106+
mountPath: /etc/kube-dns
107+
- name: config-volume
108+
mountPath: /etc/coredns
109+
volumes:
110+
- name: xtables-lock
111+
hostPath:
112+
path: /run/xtables.lock
113+
type: FileOrCreate
114+
- name: kube-dns-config
115+
configMap:
116+
name: kube-dns
117+
optional: true
118+
- name: config-volume
119+
configMap:
120+
name: node-local-dns
121+
items:
122+
- key: Corefile
123+
path: Corefile.base

‎osdc/base/kubernetes/nodelocaldns/deploy.sh‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
#
4+
# Deploy NodeLocal DNSCache (NLD) as a per-node DaemonSet.
5+
# Called from the justfile's deploy-base recipe.
6+
#
7+
# Args: $1=cluster-id
8+
#
9+
# NLD requires the kube-dns ClusterIP at config-render time and the
10+
# kube-dns-upstream Service to exist BEFORE the DaemonSet pods are
11+
# scheduled (the binary reads KUBE_DNS_UPSTREAM_SERVICE_HOST, which
12+
# kubelet only injects for Services existing at pod-create time).
13+
14+
# shellcheck disable=SC2034 # CLUSTER is part of the deploy.sh interface
15+
CLUSTER="$1"
16+
if [[ -z "$CLUSTER" ]]; then
17+
echo "ERROR: cluster-id arg is required" >&2
18+
exit 1
19+
fi
20+
21+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
22+
OSDC_UPSTREAM="${OSDC_UPSTREAM:-$(cd "$SCRIPT_DIR/../../.." && pwd)}"
23+
# shellcheck source=/dev/null
24+
source "$OSDC_UPSTREAM/scripts/mise-activate.sh"
25+
# shellcheck source=/dev/null
26+
source "$OSDC_UPSTREAM/scripts/kubectl-apply.sh"
27+
28+
# --- Cleanup trap ---
29+
WORK_DIR=""
30+
cleanup() {
31+
[[ -n "$WORK_DIR" ]] && rm -rf "$WORK_DIR" 2>/dev/null || true
32+
}
33+
trap cleanup EXIT
34+
35+
# --- Step 1: Precondition — verify kube-dns-upstream Service is ours (or absent) ---
36+
echo "→ NodeLocal DNSCache: checking kube-dns-upstream Service precondition..."
37+
UPSTREAM_FIRST_DEPLOY=true
38+
EXISTING_SELECTOR=""
39+
if kubectl get svc kube-dns-upstream -n kube-system &>/dev/null; then
40+
UPSTREAM_FIRST_DEPLOY=false
41+
EXISTING_SELECTOR=$(kubectl get svc kube-dns-upstream -n kube-system \
42+
-o jsonpath='{.spec.selector.k8s-app}' 2>/dev/null)
43+
if [[ "$EXISTING_SELECTOR" != "kube-dns" ]]; then
44+
echo "ERROR: kube-dns-upstream Service exists with selector k8s-app=${EXISTING_SELECTOR:-<empty>}." >&2
45+
echo " Expected selector k8s-app=kube-dns. Refusing to overwrite." >&2
46+
exit 1
47+
fi
48+
echo " kube-dns-upstream already present with correct selector — no rollout restart needed."
49+
else
50+
echo " kube-dns-upstream not present — will create (rollout restart will follow)."
51+
fi
52+
53+
# --- Step 2: Resolve kube-dns ClusterIP ---
54+
echo "→ NodeLocal DNSCache: resolving kube-dns ClusterIP..."
55+
KUBE_DNS_CLUSTER_IP=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}' 2>/dev/null)
56+
if [[ -z "$KUBE_DNS_CLUSTER_IP" ]]; then
57+
echo "ERROR: failed to resolve kube-dns ClusterIP (empty result)." >&2
58+
exit 1
59+
fi
60+
if [[ ! "${KUBE_DNS_CLUSTER_IP}" =~ ^[0-9a-fA-F:.]+$ ]]; then
61+
echo "ERROR: KUBE_DNS_CLUSTER_IP value is not a valid IPv4/IPv6 address: ${KUBE_DNS_CLUSTER_IP}" >&2
62+
exit 1
63+
fi
64+
echo " kube-dns ClusterIP: $KUBE_DNS_CLUSTER_IP"
65+
66+
# --- Step 3: Render manifests via kustomize, then sed-substitute placeholders ---
67+
echo "→ NodeLocal DNSCache: rendering manifests..."
68+
WORK_DIR=$(mktemp -d)
69+
RENDERED="$WORK_DIR/all.yaml"
70+
SERVICES_FILE="$WORK_DIR/services.yaml"
71+
REST_FILE="$WORK_DIR/rest.yaml"
72+
73+
kubectl kustomize "$SCRIPT_DIR" >"$RENDERED"
74+
75+
# .bak suffix for macOS/BSD sed compatibility, then drop the backup.
76+
sed -i.bak "s|__KUBE_DNS_CLUSTER_IP__|${KUBE_DNS_CLUSTER_IP}|g" "$RENDERED"
77+
rm -f "${RENDERED}.bak"
78+
79+
# --- Step 4: Split rendered output — Services first, everything else after ---
80+
# Services must exist before DaemonSet pods are scheduled so kubelet injects
81+
# KUBE_DNS_UPSTREAM_SERVICE_HOST/PORT into the node-cache container env.
82+
uv run --with pyyaml python3 - "$RENDERED" "$SERVICES_FILE" "$REST_FILE" <<'PY'
83+
import sys
84+
import yaml
85+
86+
src, services_path, rest_path = sys.argv[1], sys.argv[2], sys.argv[3]
87+
with open(src) as fh:
88+
docs = [d for d in yaml.safe_load_all(fh) if d]
89+
90+
services = [d for d in docs if d.get("kind") == "Service"]
91+
rest = [d for d in docs if d.get("kind") != "Service"]
92+
93+
with open(services_path, "w") as fh:
94+
yaml.safe_dump_all(services, fh)
95+
with open(rest_path, "w") as fh:
96+
yaml.safe_dump_all(rest, fh)
97+
PY
98+
99+
# --- Step 5: Apply Services first ---
100+
echo "→ NodeLocal DNSCache: applying Services (kube-dns-upstream + metrics)..."
101+
kubectl_apply_if_changed -f "$SERVICES_FILE"
102+
103+
# Brief settle so the Service ClusterIP is allocated before the DaemonSet
104+
# pods are created (kubelet snapshots Service env at pod-create time).
105+
sleep 3
106+
107+
# --- Step 6: Apply ConfigMap, ServiceAccount, DaemonSet ---
108+
echo "→ NodeLocal DNSCache: applying ConfigMap, ServiceAccount, DaemonSet..."
109+
kubectl_apply_if_changed -f "$REST_FILE"
110+
111+
# --- Step 7: Idempotency safety net ---
112+
# Only restart on first-time deploy: the DaemonSet pods created above (or
113+
# any pre-existing ones from an earlier failed install) won't have the
114+
# KUBE_DNS_UPSTREAM_SERVICE_HOST env var unless the Service existed before
115+
# they were scheduled. On steady-state re-runs, env is already populated.
116+
if [[ "$UPSTREAM_FIRST_DEPLOY" == "true" ]]; then
117+
echo "→ NodeLocal DNSCache: first-time deploy — rolling DaemonSet to inject upstream Service env..."
118+
kubectl rollout restart ds node-local-dns -n kube-system
119+
else
120+
echo "→ NodeLocal DNSCache: not a first-time deploy — skipping rollout restart."
121+
fi
122+
123+
echo "NodeLocal DNSCache deployed."

‎osdc/base/kubernetes/nodelocaldns/kustomization.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# NodeLocal DNSCache (NLD): per-node CoreDNS DaemonSet that absorbs DNS
2+
# queries from local pods via iptables NOTRACK interception.
3+
# kube-dns ClusterIP is dynamic per cluster — deploy.sh resolves it at
4+
# apply time and substitutes the __KUBE_DNS_CLUSTER_IP__ placeholder.
5+
apiVersion: kustomize.config.k8s.io/v1beta1
6+
kind: Kustomization
7+
8+
namespace: kube-system
9+
10+
resources:
11+
- serviceaccount.yaml
12+
- upstream-service.yaml
13+
- metrics-service.yaml
14+
- configmap.yaml
15+
- daemonset.yaml

‎osdc/base/kubernetes/nodelocaldns/metrics-service.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Headless Service for PodMonitor discovery. Two ports:
2+
# - metrics (9253): CoreDNS plugin metrics (cache hits, forward latency, etc.)
3+
# - metrics-binary (9353): k8s-dns-node-cache binary metrics (coredns_nodecache_*)
4+
# Both must be scraped — setup-error counter lives on the binary endpoint.
5+
apiVersion: v1
6+
kind: Service
7+
metadata:
8+
name: node-local-dns-metrics
9+
namespace: kube-system
10+
labels:
11+
k8s-app: node-local-dns
12+
app.kubernetes.io/managed-by: osdc-base
13+
spec:
14+
clusterIP: None
15+
selector:
16+
k8s-app: node-local-dns
17+
ports:
18+
- name: metrics
19+
port: 9253
20+
protocol: TCP
21+
targetPort: 9253
22+
- name: metrics-binary
23+
port: 9353
24+
protocol: TCP
25+
targetPort: 9353

‎osdc/base/kubernetes/nodelocaldns/serviceaccount.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# ServiceAccount for the node-local-dns DaemonSet.
2+
# No RBAC: the binary does not talk to the API server. The upstream
3+
# Service IP is resolved by env var (KUBE_DNS_UPSTREAM_SERVICE_HOST)
4+
# injected by kubelet from the kube-dns-upstream Service.
5+
# automountServiceAccountToken: false — the binary makes no API calls,
6+
# so the auto-mounted SA token is unused dead weight. Disabling it
7+
# removes a credential-leak vector if the privileged container is
8+
# compromised.
9+
apiVersion: v1
10+
kind: ServiceAccount
11+
metadata:
12+
name: node-local-dns
13+
namespace: kube-system
14+
labels:
15+
k8s-app: node-local-dns
16+
app.kubernetes.io/managed-by: osdc-base
17+
automountServiceAccountToken: false

0 commit comments

Comments
 (0)