Prepare GKE node pool for bootstrap certificate rotation (expires 2026-12-05)

[curtismchale]

Background

Google notified us that the GKE node pool in proudcity-1184 (cluster proudcity, zone us-central1-a) has a node bootstrap leaf certificate expiring 2026-12-05 17:05:08 UTC.

If the cert expires:

• New nodes can no longer register to the cluster.
• Existing kubelet client/server certs (rotated every 10 days using the bootstrap cert) will eventually expire — within 1 year for shielded nodes, 5 for non-shielded.

GKE will automatically recreate the node pool ~30 days before expiry (~2026-11-05) if we do nothing — but at their timing, not ours.

Plan

Schedule a manual node pool upgrade (same version → same version is enough to regenerate the leaf cert) during a low-traffic maintenance window before 2026-11-05.

Before the upgrade, add disruption protection so the rolling node drain doesn't take services offline:

1. Apply per-tenant PodDisruptionBudgets for prod WP tenants — config/prod-pdbs.yml (already drafted; 103 PDBs, maxUnavailable: 1, mirrors elasticsearch-pdbs.yml).
2. Scale api/proudcitycityapi from 1 → 2 replicas with a PDB for the upgrade window — config/api-proudcitycityapi-scale-patch.yml (already drafted). The public city API is currently a single replica.
3. Confirm api/proudcityfeeds (replicas=0) is intentionally off before the upgrade — easy time to notice if it shouldn't be.
4. Leave alone: the 22 single-replica *redis pods in prod (WP falls back to DB on a brief cache cold start), kube-system/* (Google manages), cert-manager/*, jenkins.

Diagnostic + upgrade commands

# Identify affected node pools
gcloud container clusters describe proudcity \
  --zone us-central1-a --project proudcity-1184

# Apply PDBs + temporary scale-up
kubectl apply -f config/prod-pdbs.yml
kubectl apply -f config/api-proudcitycityapi-scale-patch.yml

# Verify PDBs in place
kubectl get pdb -A

# Trigger the node pool upgrade (use current cluster version)
gcloud container node-pools upgrade <pool-name> \
  --cluster=proudcity --zone=us-central1-a \
  --cluster-version=<current-version>

# After upgrade completes, revert proudcitycityapi if desired
kubectl scale deployment proudcitycityapi -n api --replicas=1

Acceptance

• Confirm proudcityfeeds replicas=0 is intentional.
• Apply config/prod-pdbs.yml and verify with kubectl get pdb -n prod.
• Apply config/api-proudcitycityapi-scale-patch.yml.
• Run node pool upgrade in maintenance window before 2026-11-05.
• Confirm new cert expiry pushed forward (check MIG instance template).
• Decide whether to leave proudcitycityapi at 2 replicas permanently or revert to 1.

References

• Google email (2026-05-27) — bootstrap leaf cert expiry notice.
• GKE node leaf cert rotation docs
• Existing PDB pattern: proudcity-kubernetes/config/elasticsearch-pdbs.yml (added in PCD265).

[curtismchale]

Dry-run results + procedure correction

Ran kubectl apply --dry-run=server against the GKE cluster:

• config/prod-pdbs.yml — 103 PDBs validate clean, no errors.
• config/api-proudcitycityapi-scale-patch.yml — original version errored because kubectl apply rejects a partial Deployment spec (missing selector/template/containers).

Fix: split the proudcitycityapi work. The manifest now contains only the PodDisruptionBudget. The replica bump is a separate kubectl scale command, since apply can't do partial Deployment updates.

Corrected apply sequence

# 1. Prod tenant PDBs (103 of them)
kubectl apply -f config/prod-pdbs.yml

# 2. Scale proudcitycityapi up + add its PDB
kubectl scale deployment proudcitycityapi -n api --replicas=2
kubectl apply -f config/api-proudcitycityapi-scale-patch.yml

# 3. Verify
kubectl get pdb -A
kubectl get deploy proudcitycityapi -n api

Revert after node pool upgrade (optional)

kubectl scale deployment proudcitycityapi -n api --replicas=1
kubectl delete pdb proudcitycityapi -n api

Or leave proudcitycityapi at 2 permanently as a reliability improvement — decision flagged for post-upgrade review.

[curtismchale]

Diagnostic results (2026-05-27)

Ran gcloud container clusters describe proudcity --zone us-central1-a --project proudcity-1184.

Cluster version: 1.34.6-gke.1307000 (master + nodes match)
Zone: us-central1-a (zonal — no zone redundancy)

Node pools

Pool	Machine	Nodes	What runs there
cgroupv2	e2-standard-4	50	All prod WP tenants, api/, test/, jenkins, kube-system, cert-manager
elasticsearch-data	n1-highmem-2	3	ES data nodes
elasticdata-cgroupv2	n1-highmem-2	3	Additional ES data nodes

Maintenance window (already configured)

FREQ=WEEKLY;BYDAY=SA,SU, 08:00 UTC → 08:00 UTC. Roughly all weekend in Pacific time. Natural target for the manual upgrade.

Affected pool unknown

Google's email didn't name which of the 3 pools holds the expiring leaf cert. Two options:

1. Run Google's node-cert diagnostic utility per pool to identify.
2. Skip the chase — same-version upgrade is cheap and regenerates the cert regardless. Upgrade all 3.

Concrete upgrade commands (do in this order)

Smallest blast radius first to validate, biggest pool last:

# 1. Validate on a small ES pool first
gcloud container node-pools upgrade elasticsearch-data \
  --cluster=proudcity --zone=us-central1-a \
  --project=proudcity-1184 \
  --cluster-version=1.34.6-gke.1307000

# 2. Second ES pool
gcloud container node-pools upgrade elasticdata-cgroupv2 \
  --cluster=proudcity --zone=us-central1-a \
  --project=proudcity-1184 \
  --cluster-version=1.34.6-gke.1307000

# 3. Main pool (50 nodes — paced by new PDBs, will take 1–3 hours)
gcloud container node-pools upgrade cgroupv2 \
  --cluster=proudcity --zone=us-central1-a \
  --project=proudcity-1184 \
  --cluster-version=1.34.6-gke.1307000

ES pools are quick (3 nodes, paced by elasticsearch-pdbs.yml). The cgroupv2 pool will be the long one — 50 nodes × 15–26 pods each, paced by the new prod PDBs (maxUnavailable: 1 per tenant).