Clean up leaked GKE LoadBalancer firewall rules in setup-gke teardown#1367
Open
lsierant wants to merge 10 commits into
Open
Clean up leaked GKE LoadBalancer firewall rules in setup-gke teardown#1367lsierant wants to merge 10 commits into
lsierant wants to merge 10 commits into
Conversation
lsierant
added
the
skip-changelog
Contributor
MCK 1.9.2 Release NotesBug Fixes
|
GKE auto-deletes its managed firewall rules on cluster delete, but the LoadBalancer Service firewall rules (k8s-fw-*, k8s-*-hc) are left behind and accumulate across CI runs until they exhaust the project FIREWALLS quota, failing 'gcloud container clusters create' with a 429. Delete each run's leftover rules in ra-01-setup-gke/teardown.sh, matched by the node network tag.
lsierant
force-pushed
the
lsierant-gke-firewall-reaper
branch
from
July 10, 2026 14:14
52e1bc8 to
c83ece6
Compare
lsierant
changed the title
The automation agent's StartFresh step races with the mongod container's image pull: the agent runs 'mongod -f ...' inside the agent container (where mongod is not installed), the process exits immediately, and the agent enters a retry loop checking /data/mongod.lock. On the central cluster the mongod container pulls its image quickly (~25s) and creates the lock file before the agent gives up on that check pattern. On member clusters the image pull takes ~54s, so the agent stops checking the lock file before mongod starts and never recovers — the AppDB pods stay not-ready forever, Ops Manager never reaches Pending, and the test times out. Fix: pre-pull the mongod image via a temporary DaemonSet on all three GKE clusters before applying the OpsManager CR. Once the image is cached on every node, the mongod container starts immediately and the agent detects it via the lock file.
The automation agent detects mongod via /data/mongod.lock, but with separate PID namespaces (the default for pods with init containers), the PID written by the mongod container is unverifiable from the agent container: PID 1 in the mongod container becomes PID 1 in the lock file, but PID 1 in the agent container is the agent itself. The agent falls back to connecting via the external domain, which never resolves on member clusters where external-dns doesn't watch. Setting shareProcessNamespace: true unconditionally makes the agent and mongod share a PID namespace, so the agent can verify mongod's PID from the lock file regardless of DNS state. This was already done for static architectures; the normal path with init containers was missed.
Snippet generator wraps each file as 'function X() {' + contents + '}'.
Without a trailing newline the final 'done' glued to '}' producing 'done}',
a syntax error that broke GKE teardown ('.generated/...: syntax error:
unexpected end of file').
The temporary pre-pull DaemonSet was a workaround for the AppDB agent timing race. It is redundant and is removed.
…ectures" This reverts commit 34cbc60.
…s render - Extract the inline LoadBalancer firewall-rule cleanup from teardown.sh into ra-01_9030_delete_firewall_rules.sh, run alongside the other ra-01_90xx teardown snippets. - Render the external-dns manifest once into a variable and apply it per context, restoring the three symmetric kubectl apply lines.
The mesh GKE snippet test failed at ra-06_0322 (wait_for_running after adding the second member cluster): the operator reached Running ~2.5 min after the 600s wait expired. Adding a member cluster triggers a large reconcile (AppDB scale 3->5 + OM redeploy across clusters) that regularly exceeds 600s on GKE. Standardize all AppDB/OM wait_for_running timeouts at 1200s (0312 was 900s, 0322/0522 were 600s).
…llisions The no-mesh OM reference architecture created its global GCP load balancer resources (ssl-certificate, forwarding-rule, target-https-proxy, url-map, backend-service, health-check, firewall-rule) under fixed names. Two runs sharing a project (e.g. public and private GKE snippet variants) collided on these project-global names: the second create hit 'already exists' and one run's teardown deleted the other's resources. Thread K8S_CLUSTER_SUFFIX through the names via OM_LB_* env vars, matching how cluster names are made unique. The suffix is empty by default, so documentation names are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[skip-ci] Clean up leaked GKE firewall rules + pin external-dns DNS zones in no-mesh GKE snippets
Summary
The
private_gke_code_snippetstasks were failing at cluster creation withgcloud container clusters createreturning429 Insufficient project quota ... resource "FIREWALLS": ... quota of '500.0' with '0.0' availablein thescratch-kubernetes-teamproject.Root cause: when a GKE cluster is deleted, GKE removes its own managed firewall rules (
gke-<cluster>-<hash>-*) but leaves behind the LoadBalancer Service firewall rules (k8s-fw-*,k8s-*-hc) created forServiceobjects of typeLoadBalancer. Across many CI runs these accumulate and eventually exhaust the project-wide FIREWALLS quota, blocking all new GKE clusters.This change:
ra-01_9030_delete_firewall_rules.sh(run fromra-01-setup-gke/teardown.sh) to delete each run's leftover LB firewall rules, scoped to the run's own clusters via the node network tag so it never touches concurrent runs.ra-01_9020_delete_disks.shto delete leftover persistent disks from PVCs withreclaimPolicy: Retain.--zone-id-filter,--domain-filter,--google-zone-visibility=private) inra-09-setup-externaldns. Without this, leakedmongodb.custom.zones in the shared project caused external-dns to scatter AppDB A records into stale zones, making cluster 1/2 AppDB members unreachable and timing out the no-mesh snippet test.wait_for_runningtimeouts at 1200s (ra-06 mesh + ra-10 no-mesh). The mesh test was failing atra-06_0322because the reconcile after adding the second member cluster (AppDB scale 3->5 + OM redeploy) regularly finished ~2-3 min after the old 600s wait expired.Proof of Work
unit_tests, and all 4 GKE snippet tests (public + private × mesh + no-mesh) passing concurrently.om-certificateglobal-name collision by suffixing OM load balancer GCP resource names withK8S_CLUSTER_SUFFIX(commit 7fe5ffe), letting public and private no-mesh runs co-exist in one project.shareProcessNamespacerevert both confirmed safe (AppDB + OM reached Running in no-mesh).Checklist