From 859126ad8c354bf6629e11fad88be321984f2027 Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 13:40:41 -0600 Subject: [PATCH 1/9] fix(mongodb): let MongoDB container resources be configured The operator owns the pod template of the StatefulSet it manages and injects its own resource defaults of 500m CPU and 500M memory per container and per init container. mongo-rs.yaml overrode serviceAccountName and affinity but never containers or initContainers, so nothing in the chart - no value, no --set - could change what a MongoDB pod requested. The init containers matter as much as the containers: a pod reserves max(max(initContainer), sum(containers)), so each merely copying a binary into a shared volume at 500m sets the pod's effective floor whenever the containers ask for less. Overriding only the containers is not enough. Adds mongodb.resources, mongodb.agentResources and mongodb.initResources, patched onto the operator-owned template by container name, with defaults sized for a production replica set rather than inherited from the operator. mongod gets 2Gi: WiredTiger sizes its cache from the container limit and an idle 3-member replica set already sits at ~340Mi, so the operator's 500M limit left almost nothing before an OOMKill. CPU goes the other way, since the same replica set idles under 50m per member. A MongoDB pod's reservation therefore moves from 1000m/800M to 300m/1152Mi, and ci/ci-values.yaml shrinks all three blocks to 150m/640Mi for CI. Existing installs get a rolling restart of the replica set on upgrade. --- charts/graylog/README.md | 33 ++++ charts/graylog/ci/ci-values.yaml | 27 +++ charts/graylog/templates/custom/mongo-rs.yaml | 34 ++++ charts/graylog/tests/mongodb_test.yaml | 159 ++++++++++++++++++ charts/graylog/values.schema.json | 60 +++++++ charts/graylog/values.yaml | 45 +++++ 6 files changed, 358 insertions(+) diff --git a/charts/graylog/README.md b/charts/graylog/README.md index 7ce7b74..d14e9dc 100644 --- a/charts/graylog/README.md +++ b/charts/graylog/README.md @@ -1511,6 +1511,27 @@ configuration. MongoDB Community Resource configuration. Requires the MCK Operator: https://github.com/mongodb/mongodb-kubernetes/tree/master/docs/mongodbcommunity +The operator creates the MongoDB containers itself and injects its own resource +defaults, which the three resource blocks below patch by container name — the +only way to size MongoDB from the chart. + +The defaults are sized for a production replica set. `mongod` gets 2Gi of +headroom because WiredTiger sizes its cache from the container limit and Graylog +keeps its whole configuration in MongoDB; an idle 3-member replica set already +sits at ~340Mi. CPU goes the other way: the same replica set idles under 50m per +member, so the requests are modest and the limits leave room to burst. + +`initResources` covers the two init containers the operator injects. Because a +pod reserves `max(max(initContainer), sum(containers))`, an init container +request larger than the containers' combined request becomes the pod's floor, so +these are kept deliberately small — each only copies a binary into a shared +volume. + +Overrides deep-merge, so setting only `requests` keeps the default `limits`; set +a block to `null` to drop the override and defer to the operator. For a +small-footprint install, see [`ci/ci-values.yaml`](ci/ci-values.yaml), which +shrinks all three blocks together. + | Key Path | Description | Default | |---------------------------------------|-------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `mongodb.communityResource.enabled` | Enables creation of the `MongoDBCommunity` custom resource. | `true` | @@ -1519,6 +1540,18 @@ Requires the MCK Operator: https://github.com/mongodb/mongodb-kubernetes/tree/ma | `mongodb.arbiters` | Number of arbiter nodes to deploy. | `0` | | `mongodb.annotations` | Annotations for the `MongoDBCommunity` object. | `{}` | | `mongodb.labels` | Labels for the `MongoDBCommunity` object. | `{}` | +| `mongodb.resources.limits.cpu` | CPU limit for the `mongod` container. | `"2"` | +| `mongodb.resources.limits.memory` | Memory limit for the `mongod` container. | `"2Gi"` | +| `mongodb.resources.requests.cpu` | CPU request for the `mongod` container. | `"250m"` | +| `mongodb.resources.requests.memory` | Memory request for the `mongod` container. | `"1Gi"` | +| `mongodb.agentResources.limits.cpu` | CPU limit for the `mongodb-agent` container. | `"500m"` | +| `mongodb.agentResources.limits.memory` | Memory limit for the `mongodb-agent` container. | `"512Mi"` | +| `mongodb.agentResources.requests.cpu` | CPU request for the `mongodb-agent` container. | `"50m"` | +| `mongodb.agentResources.requests.memory` | Memory request for the `mongodb-agent` container. | `"128Mi"` | +| `mongodb.initResources.limits.cpu` | CPU limit for both operator-injected init containers. | `"500m"` | +| `mongodb.initResources.limits.memory` | Memory limit for both operator-injected init containers. | `"512Mi"` | +| `mongodb.initResources.requests.cpu` | CPU request for both operator-injected init containers. | `"50m"` | +| `mongodb.initResources.requests.memory` | Memory request for both operator-injected init containers. | `"128Mi"` | | `mongodb.persistence.storageClass` | StorageClass to use for persistent volumes. | `""` | | `mongodb.persistence.size.data` | Persistent volume size for data storage. | `"10G"` | | `mongodb.persistence.size.logs` | Persistent volume size for MongoDB logs. | `"2G"` | diff --git a/charts/graylog/ci/ci-values.yaml b/charts/graylog/ci/ci-values.yaml index 7459bbf..81eeaed 100644 --- a/charts/graylog/ci/ci-values.yaml +++ b/charts/graylog/ci/ci-values.yaml @@ -31,3 +31,30 @@ mongodb: replicas: 1 arbiters: 0 version: "8.0.23" + # The chart defaults are sized for a production replica set (2Gi of headroom + # for mongod). A single-member CI replica set holds almost no data, so all + # three blocks are shrunk here. All three are needed: a pod reserves + # max(max(initContainer), sum(containers)), so leaving initResources alone + # would keep the pod's floor at the operator's 500m no matter how small the + # containers ask to be. + resources: + requests: + cpu: "100m" + memory: "512Mi" + limits: + cpu: "1000m" + memory: "1Gi" + agentResources: + requests: + cpu: "50m" + memory: "128Mi" + limits: + cpu: "500m" + memory: "256Mi" + initResources: + requests: + cpu: "50m" + memory: "128Mi" + limits: + cpu: "200m" + memory: "256Mi" diff --git a/charts/graylog/templates/custom/mongo-rs.yaml b/charts/graylog/templates/custom/mongo-rs.yaml index b1a7461..2473569 100644 --- a/charts/graylog/templates/custom/mongo-rs.yaml +++ b/charts/graylog/templates/custom/mongo-rs.yaml @@ -87,6 +87,40 @@ spec: # The chart's own labels are not present here. app: {{ include "graylog.mongodb.crName" . | printf "%s-svc" }} topologyKey: kubernetes.io/hostname + {{- with .Values.mongodb.initResources }} + {{/* + Both init containers only copy a binary into a shared volume, but the + operator requests as much CPU for them as for mongod. A pod reserves + max(max(initContainer), sum(containers)), so these set the pod's + effective request whenever the containers below ask for less. + */}} + initContainers: + - name: mongod-posthook + resources: {{ . | toYaml | nindent 16 }} + - name: mongodb-agent-readinessprobe + resources: {{ . | toYaml | nindent 16 }} + {{- end }} + {{- if or .Values.mongodb.resources .Values.mongodb.agentResources }} + {{/* + Container resources are patched in by name. The operator creates both + containers itself and applies its own defaults (500m CPU each), which + nothing in the chart could override before this: the pod always asked + for ~1 CPU regardless of how MongoDB was sized. + + Names must match what the operator generates - `mongod` and + `mongodb-agent` - or the entry is appended as an extra container + rather than merged into the existing one. + */}} + containers: + {{- with .Values.mongodb.resources }} + - name: mongod + resources: {{ . | toYaml | nindent 16 }} + {{- end }} + {{- with .Values.mongodb.agentResources }} + - name: mongodb-agent + resources: {{ . | toYaml | nindent 16 }} + {{- end }} + {{- end }} volumeClaimTemplates: - metadata: name: data-volume diff --git a/charts/graylog/tests/mongodb_test.yaml b/charts/graylog/tests/mongodb_test.yaml index b853da4..6219fb6 100644 --- a/charts/graylog/tests/mongodb_test.yaml +++ b/charts/graylog/tests/mongodb_test.yaml @@ -94,3 +94,162 @@ tests: - equal: path: spec.statefulSet.spec.volumeClaimTemplates[1].spec.resources.requests.storage value: "5G" + + # The operator injects its own defaults (500m CPU and 500M memory per + # container) and nothing in the chart could override them before these values + # existed. The chart's defaults deliberately differ: mongod gets real memory + # headroom, because an idle 3-member replica set already sits at ~340Mi + # against the operator's 500M limit, and the CPU request drops because the + # same replica set idles under 50m per member. + - it: patches the chart's production defaults onto the mongod and mongodb-agent containers + set: + mongodb.communityResource.enabled: true + asserts: + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].name + value: mongod + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].resources + value: + limits: + cpu: "2" + memory: 2Gi + requests: + cpu: 250m + memory: 1Gi + - equal: + path: spec.statefulSet.spec.template.spec.containers[1].name + value: mongodb-agent + - equal: + path: spec.statefulSet.spec.template.spec.containers[1].resources + value: + limits: + cpu: 500m + memory: 512Mi + requests: + cpu: 50m + memory: 128Mi + + - it: honors mongodb.resources and mongodb.agentResources independently + set: + mongodb.communityResource.enabled: true + mongodb.resources: + requests: + cpu: 100m + memory: 300M + mongodb.agentResources: + requests: + cpu: 50m + memory: 100M + asserts: + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].resources.requests.cpu + value: 100m + - equal: + path: spec.statefulSet.spec.template.spec.containers[1].resources.requests.cpu + value: 50m + # Values deep-merge, so overriding only `requests` keeps the default + # `limits`. Worth pinning: it means a small `requests` override does not + # silently leave the container unlimited. + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].resources.limits.cpu + value: "2" + + # Only `null` drops a map; `{}` would merge and leave the defaults in place. + - it: omits the container override when both resource blocks are null + set: + mongodb.communityResource.enabled: true + mongodb.resources: null + mongodb.agentResources: null + asserts: + - notExists: + path: spec.statefulSet.spec.template.spec.containers + + - it: patches only mongod when agentResources is null + set: + mongodb.communityResource.enabled: true + mongodb.agentResources: null + asserts: + - lengthEqual: + path: spec.statefulSet.spec.template.spec.containers + count: 1 + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].name + value: mongod + + # The mirror of the case above, and not redundant with it: without the inner + # `with` guard on mongodb.resources, this is the only input that renders + # `resources: null` onto the mongod container -- patching the field to empty + # instead of leaving it to the operator. Mutation-tested: dropping that guard + # keeps every other test in this file green. + - it: patches only mongodb-agent when resources is null + set: + mongodb.communityResource.enabled: true + mongodb.resources: null + asserts: + - lengthEqual: + path: spec.statefulSet.spec.template.spec.containers + count: 1 + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].name + value: mongodb-agent + # Nothing may carry a null resources block: that would clear the + # operator's defaults rather than defer to them. + - notExists: + path: spec.statefulSet.spec.template.spec.containers[1] + + # A pod reserves max(max(initContainer), sum(containers)), so these two set the + # pod's effective CPU request whenever the containers ask for less. Left at the + # operator's 500m they were the pod's floor no matter how small the containers + # asked to be; both init containers only copy a binary into a shared volume. + - it: keeps both init containers small by default + set: + mongodb.communityResource.enabled: true + asserts: + - lengthEqual: + path: spec.statefulSet.spec.template.spec.initContainers + count: 2 + - equal: + path: spec.statefulSet.spec.template.spec.initContainers[0].name + value: mongod-posthook + - equal: + path: spec.statefulSet.spec.template.spec.initContainers[1].name + value: mongodb-agent-readinessprobe + - equal: + path: spec.statefulSet.spec.template.spec.initContainers[0].resources.requests.cpu + value: 50m + - equal: + path: spec.statefulSet.spec.template.spec.initContainers[1].resources.requests.cpu + value: 50m + # The init containers must not set the pod's floor: their request stays + # below what the two containers ask for together. + - equal: + path: spec.statefulSet.spec.template.spec.containers[0].resources.requests.cpu + value: 250m + + - it: honors mongodb.initResources on both init containers + set: + mongodb.communityResource.enabled: true + mongodb.initResources: + requests: + cpu: 25m + memory: 64Mi + asserts: + - equal: + path: spec.statefulSet.spec.template.spec.initContainers[0].resources.requests.cpu + value: 25m + - equal: + path: spec.statefulSet.spec.template.spec.initContainers[1].resources.requests.cpu + value: 25m + + - it: omits the init container override when initResources is null + set: + mongodb.communityResource.enabled: true + mongodb.initResources: null + asserts: + - notExists: + path: spec.statefulSet.spec.template.spec.initContainers + # The container override is independent and must survive. + - lengthEqual: + path: spec.statefulSet.spec.template.spec.containers + count: 2 diff --git a/charts/graylog/values.schema.json b/charts/graylog/values.schema.json index 8eab53d..4f13923 100644 --- a/charts/graylog/values.schema.json +++ b/charts/graylog/values.schema.json @@ -954,6 +954,66 @@ "arbiters": { "type": "integer" }, "annotations": { "type": "object", "description": "Annotations for the MongoDBCommunity object" }, "labels": { "type": "object", "description": "Labels for the MongoDBCommunity object" }, + "resources": { + "type": "object", + "description": "Compute resources for the mongod container", + "properties": { + "limits": { + "type": "object", + "properties": { + "cpu": { "type": "string" }, + "memory": { "type": "string" } + } + }, + "requests": { + "type": "object", + "properties": { + "cpu": { "type": "string" }, + "memory": { "type": "string" } + } + } + } + }, + "agentResources": { + "type": "object", + "description": "Compute resources for the mongodb-agent container", + "properties": { + "limits": { + "type": "object", + "properties": { + "cpu": { "type": "string" }, + "memory": { "type": "string" } + } + }, + "requests": { + "type": "object", + "properties": { + "cpu": { "type": "string" }, + "memory": { "type": "string" } + } + } + } + }, + "initResources": { + "type": "object", + "description": "Compute resources for the mongod-posthook and mongodb-agent-readinessprobe init containers", + "properties": { + "limits": { + "type": "object", + "properties": { + "cpu": { "type": "string" }, + "memory": { "type": "string" } + } + }, + "requests": { + "type": "object", + "properties": { + "cpu": { "type": "string" }, + "memory": { "type": "string" } + } + } + } + }, "persistence": { "type": "object", "properties": { diff --git a/charts/graylog/values.yaml b/charts/graylog/values.yaml index 8eb9ea5..d252487 100644 --- a/charts/graylog/values.yaml +++ b/charts/graylog/values.yaml @@ -894,6 +894,51 @@ mongodb: # pod template of the StatefulSet it manages and discards anything the chart # puts there, so the keys would silently do nothing. annotations/labels above # apply to the MongoDBCommunity object itself and do work. + # + # resources are the compute requests/limits of the mongod container, and + # agentResources those of the mongodb-agent sidecar the operator injects. + # The operator creates both containers itself and applies its own defaults; + # these blocks patch them by container name, which is the only way to size + # MongoDB from the chart. + # + # The defaults below are sized for production, not for the operator's + # convenience. mongod gets 2Gi of headroom because WiredTiger sizes its cache + # from the container limit and Graylog holds its whole configuration here: an + # idle 3-member replica set already sits at ~340Mi, so the operator's own 500M + # limit leaves almost no room before an OOMKill. CPU goes the other way -- the + # same replica set idles under 50m per member, so reserving a full core per + # pod, as the operator does, only makes MongoDB hard to schedule. + # + # Overrides merge with the defaults below, so setting only `requests` keeps + # these `limits`. Set a block to `null` to drop the container override + # entirely and let the operator apply whatever it wants. + resources: + limits: + cpu: "2" + memory: "2Gi" + requests: + cpu: "250m" + memory: "1Gi" + agentResources: + limits: + cpu: "500m" + memory: "512Mi" + requests: + cpu: "50m" + memory: "128Mi" + # initResources applies to both init containers the operator injects, + # mongod-posthook and mongodb-agent-readinessprobe. Each only copies a binary + # into a shared volume, yet the operator gives them the same 500m request as + # mongod. Because a pod reserves max(max(initContainer), sum(containers)), + # that 500m was the pod's effective CPU floor no matter how small the + # containers asked to be -- so these are kept deliberately small. + initResources: + limits: + cpu: "500m" + memory: "512Mi" + requests: + cpu: "50m" + memory: "128Mi" persistence: # storageClass overrides global.storageClass for the MongoDB volumes. storageClass: "" From 026b51bb77c97c14c3e3356009d7f0bf9a5ae184 Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 14:45:39 -0600 Subject: [PATCH 2/9] ci: make the chart install job diagnosable and bounded helm-ct-install had no timeout, so a wedged install could burn the default 360-minute ceiling on every leg of the matrix. helm-extra-args --timeout=900s bounds the install itself, not namespace termination hanging on a finalizer. The step was also silent: helm --wait prints nothing until it succeeds or times out, so a stuck install looked exactly like a slow one for fifteen minutes. A watcher now streams not-yet-Running pods and recent warning events every 20s, and node capacity plus reservations are printed after the cluster comes up, which turns "Insufficient cpu" into a one-line diagnosis. Sizing moves entirely into charts/graylog/ci/ci-values.yaml, which ct discovers on its own. It was duplicated as --set flags, and those take precedence over the values file: the graylog memory request in ci-values.yaml was silently overridden and the two disagreed. Only the generated rootPassword is passed as a flag now, because it cannot be committed. The MongoDB operator is shrunk to 100m: its chart asks for 500m to reconcile a single one-member replica set. ci-values.yaml also sets graylog.terminationGracePeriodSeconds to 30. The chart default of 300s exists to let the preStop drain flush the journal, and ct pays it on every leg when it deletes the namespace, but nothing in CI holds data worth draining. The chart default is unchanged. --- .github/workflows/lint-and-test.yaml | 63 ++++++++++++++++++++++------ charts/graylog/ci/ci-values.yaml | 6 +++ 2 files changed, 56 insertions(+), 13 deletions(-) diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index bf90edf..e344dc8 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -108,6 +108,12 @@ jobs: helm-ct-install: runs-on: ubuntu-latest needs: [helm-ct-lint, helm-unittest] + # Backstop only. A healthy leg spends ~2 minutes on setup and a few more + # bringing the stack up; the default job ceiling is 360 minutes, so without + # this a wedged install burns hours per leg before anyone notices. Namespace + # termination can genuinely hang on finalizers, and that is not covered by + # `helm-extra-args: --timeout=900s`, which bounds only the install itself. + timeout-minutes: 30 strategy: fail-fast: false # Asymmetric K8s × Helm matrix: full K8s coverage on Helm 3 (the chart's @@ -143,16 +149,32 @@ jobs: node_image: kindest/node:${{ matrix.k8s-version }} - name: Install MongoDB Kubernetes Operator + # The operator's own chart requests 500m CPU by default — an eighth of + # the runner for a controller reconciling a single one-member replica + # set. Shrinking it leaves that CPU for the workloads under test. run: | helm upgrade --install mongodb-kubernetes-operator mongodb-kubernetes \ --repo https://mongodb.github.io/helm-charts \ --version "1.6.1" \ --set operator.watchNamespace="*" \ + --set operator.resources.requests.cpu=100m \ + --set operator.resources.requests.memory=200Mi \ + --set operator.resources.limits.cpu=500m \ + --set operator.resources.limits.memory=500Mi \ --namespace operators \ --create-namespace \ --wait \ --timeout 5m + # The CPU budget on this node decides whether the stack can be scheduled at + # all. Printing it once makes "Insufficient cpu" a one-line diagnosis + # instead of an inference from pod descriptions. + - name: Show node capacity and reservations + run: | + kubectl get nodes -o custom-columns=\ + 'NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory' + kubectl describe node | sed -n '/Allocated resources/,/Events/p' + - name: Generate ephemeral CI rootPassword and mask in workflow logs id: ci-root-password run: | @@ -160,20 +182,35 @@ jobs: echo "::add-mask::${password}" echo "value=${password}" >> "$GITHUB_OUTPUT" + # All sizing lives in charts/graylog/ci/ci-values.yaml, which ct discovers + # automatically. It used to be duplicated here as --set flags, which take + # precedence over the values file — so ci-values.yaml's graylog memory + # request was silently overridden and the two disagreed. Only the generated + # password is passed here, because it cannot be committed. + # + # `helm --wait` prints nothing until it succeeds or times out, so this step + # used to sit silent for the full 15 minutes and a stuck install looked + # exactly like a slow one. The watcher below streams anything not yet + # Running plus recent warning events, so a pod that cannot be scheduled is + # visible within seconds instead of in a post-mortem dump. - name: Run ct install run: | + watch_cluster() { + while true; do + echo "::group::cluster state $(date -u +%H:%M:%S)" + kubectl get pods -A \ + --field-selector=status.phase!=Running,status.phase!=Succeeded \ + -o wide 2>/dev/null || true + kubectl get events -A --field-selector type=Warning \ + --sort-by=.lastTimestamp 2>/dev/null | tail -8 || true + echo "::endgroup::" + sleep 20 + done + } + watch_cluster & + watcher=$! + trap 'kill "$watcher" 2>/dev/null || true' EXIT + ct install --config .github/ct.yaml --all \ --helm-extra-set-args "\ - --set graylog.config.rootPassword=${{ steps.ci-root-password.outputs.value }} \ - --set graylog.replicas=1 \ - --set datanode.replicas=1 \ - --set mongodb.replicas=1 \ - --set mongodb.arbiters=0 \ - --set graylog.resources.requests.cpu=250m \ - --set graylog.resources.requests.memory=512Mi \ - --set-string graylog.resources.limits.cpu=1 \ - --set graylog.resources.limits.memory=1Gi \ - --set datanode.resources.requests.cpu=250m \ - --set datanode.resources.requests.memory=1Gi \ - --set-string datanode.resources.limits.cpu=1 \ - --set datanode.resources.limits.memory=2Gi" + --set graylog.config.rootPassword=${{ steps.ci-root-password.outputs.value }}" diff --git a/charts/graylog/ci/ci-values.yaml b/charts/graylog/ci/ci-values.yaml index 81eeaed..13a68e2 100644 --- a/charts/graylog/ci/ci-values.yaml +++ b/charts/graylog/ci/ci-values.yaml @@ -4,6 +4,12 @@ graylog: replicas: 1 + # The chart default is 300s, to give the preStop drain room to flush the + # journal before SIGKILL. ct deletes the namespace after every install and + # waits for termination, so that budget is paid on every leg of the matrix. + # Nothing here holds data worth draining, and the drain hook is off by + # default anyway. Do not change the chart default. + terminationGracePeriodSeconds: 30 config: serverJavaOpts: "-Xms512m -Xmx768m" resources: From 71d1eb6b768e31dcdc5113962248bf16c21a40e2 Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 14:54:44 -0600 Subject: [PATCH 3/9] ci: narrowing down scope of test execution to specific pathing --- .github/workflows/lint-and-test.yaml | 85 +++++++++++++++++++++++++++- 1 file changed, 84 insertions(+), 1 deletion(-) diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index e344dc8..b6e3d47 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -7,8 +7,83 @@ on: branches: ["main"] jobs: + # Decides which test jobs are worth running for this change. + # + # Deliberately not `on..paths`: a workflow skipped by a path filter + # creates no check runs at all, so if these checks are ever made required, a + # docs-only PR waits forever on checks that will never report. Jobs skipped by + # an `if:` condition still report - as skipped, which branch protection + # accepts - and cost a few seconds rather than a runner. + changes: + runs-on: ubuntu-latest + outputs: + chart: ${{ steps.filter.outputs.chart }} + examples: ${{ steps.filter.outputs.examples }} + steps: + - name: Checkout + uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Classify changed paths + id: filter + env: + BASE_REF: ${{ github.base_ref }} + BEFORE: ${{ github.event.before }} + run: | + set -u + # On anything unexpected, fall through to running everything: a false + # negative here silently ships an untested chart. + if [[ "${{ github.event_name }}" == "pull_request" ]]; then + git fetch --quiet origin "$BASE_REF" + range="origin/${BASE_REF}...HEAD" + elif [[ -n "${BEFORE:-}" ]] && git cat-file -e "${BEFORE}^{commit}" 2>/dev/null; then + range="${BEFORE}..HEAD" + else + echo "no usable diff range; running every test job" + { + echo "chart=true" + echo "examples=true" + } >> "$GITHUB_OUTPUT" + exit 0 + fi + + files="$(git diff --name-only "$range")" + echo "::group::changed files ($range)" + printf '%s\n' "$files" + echo "::endgroup::" + + chart=false + examples=false + while IFS= read -r f; do + [[ -n "$f" ]] || continue + case "$f" in + # Markdown inside the chart is documentation - the values + # reference and the chart README. It cannot change a rendered + # manifest, so it does not earn a four-leg install matrix. + charts/graylog/*) + if [[ "$f" != *.md ]]; then chart=true; fi + ;; + # Shipped examples are values overlays, rendered by the lint job. + examples/*) + examples=true + ;; + # The CI definition itself: run everything, or a change to how the + # chart is tested would never be tested. + .github/workflows/lint-and-test.yaml|.github/ct.yaml) + chart=true + examples=true + ;; + esac + done <<< "$files" + + echo "chart=$chart" | tee -a "$GITHUB_OUTPUT" + echo "examples=$examples" | tee -a "$GITHUB_OUTPUT" + helm-ct-lint: runs-on: ubuntu-latest + needs: changes + if: needs.changes.outputs.chart == 'true' || needs.changes.outputs.examples == 'true' strategy: fail-fast: false matrix: @@ -60,6 +135,10 @@ jobs: helm-unittest: runs-on: ubuntu-latest + needs: changes + # Templates, values and the suites themselves all live under + # charts/graylog/, so a chart change is the only thing this can regress. + if: needs.changes.outputs.chart == 'true' strategy: fail-fast: false matrix: @@ -107,7 +186,11 @@ jobs: helm-ct-install: runs-on: ubuntu-latest - needs: [helm-ct-lint, helm-unittest] + needs: [changes, helm-ct-lint, helm-unittest] + # The expensive job: four legs, each standing up Graylog, a datanode and a + # MongoDB replica set. Only a change that alters what the chart renders can + # justify it. + if: needs.changes.outputs.chart == 'true' # Backstop only. A healthy leg spends ~2 minutes on setup and a few more # bringing the stack up; the default job ceiling is 360 minutes, so without # this a wedged install burns hours per leg before anyone notices. Namespace From 013b8a984d06ec28b50c9996c9fe4bb04858a83b Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 15:11:24 -0600 Subject: [PATCH 4/9] ci: report test results in the workflow run summary Answering "what happened in CI" meant opening a job and reading log output. Every job now writes what it learned to the run summary instead. - changes: which jobs run, which are skipped and why, plus the changed file list. A skipped run is now self-explanatory rather than suspicious. - lint: the ct lint verdict and a per-example table - rendered object count for each values overlay, and for a failure the offending key rather than Helm's generic schema error. Runs even when ct lint fails, so the table is not lost with it. - unittest: suite and test counts lifted from helm-unittest's own output, the two behaviour-script verdicts, and failing suites when something breaks. - install: per matrix leg, the result, how long the install took, node allocatable CPU and memory, and the pod requests as last observed. On failure it adds the matching error lines from the ct log, the pods that never reached Running, and recent warning events. The pod-request snapshot is taken by the watcher loop rather than at summary time: ct deletes its namespace when it finishes, so by then the workloads under test are gone and cannot be queried. --- .github/workflows/lint-and-test.yaml | 176 ++++++++++++++++++++++++++- 1 file changed, 173 insertions(+), 3 deletions(-) diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index b6e3d47..f25b628 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -80,6 +80,28 @@ jobs: echo "chart=$chart" | tee -a "$GITHUB_OUTPUT" echo "examples=$examples" | tee -a "$GITHUB_OUTPUT" + mark() { [[ "$1" == "true" ]] && echo "run" || echo "skipped"; } + { + echo "## What this run tests" + echo + echo "| job | verdict | why |" + echo "|-----|---------|-----|" + echo "| \`helm-ct-lint\` | $( [[ "$chart" == "true" || "$examples" == "true" ]] && echo run || echo skipped ) | chart or examples changed |" + echo "| \`helm-unittest\` | $(mark "$chart") | chart templates/values/tests changed |" + echo "| \`helm-ct-install\` | $(mark "$chart") | chart changed (4 legs, full stack each) |" + echo + echo "Markdown under \`charts/graylog/\` and everything in \`docs/\` cannot" + echo "change a rendered manifest, so they test nothing." + echo + echo "
Changed files ($(printf '%s\n' "$files" | grep -c . || true))" + echo + echo '```' + printf '%s\n' "$files" + echo '```' + echo + echo "
" + } >> "$GITHUB_STEP_SUMMARY" + helm-ct-lint: runs-on: ubuntu-latest needs: changes @@ -105,7 +127,10 @@ jobs: uses: helm/chart-testing-action@v2.8.0 - name: Run ct lint - run: ct lint --config .github/ct.yaml --all + id: ct-lint + run: | + set -o pipefail + ct lint --config .github/ct.yaml --all 2>&1 | tee /tmp/ct-lint.log # `ct lint` never looks at examples/, so a shipped example could reference a # removed key, miss a required one, or trip a template-time guard and no one @@ -115,22 +140,51 @@ jobs: # Raw manifests in examples/ (Secrets, Pods) are not values overlays and are # skipped by looking for a top-level `apiVersion:`. - name: Render shipped examples + id: examples + # always() so a ct lint failure still yields the examples table and a + # summary for this leg, rather than an empty run page. + if: always() run: | set -u rc=0 + rows="" + ok=0; skipped=0; failed=0 while IFS= read -r f; do if grep -q '^apiVersion:' "$f"; then echo "skip $f (raw manifest, not a values overlay)" + rows="${rows}| \`${f}\` | – | raw manifest, not a values overlay |"$'\n' + skipped=$((skipped + 1)) continue fi if out=$(helm template ci-examples charts/graylog -f "$f" 2>&1); then echo "ok $f" + objects=$(printf '%s\n' "$out" | grep -c '^kind:' || true) + rows="${rows}| \`${f}\` | ok | ${objects} objects |"$'\n' + ok=$((ok + 1)) else echo "FAILED $f" printf '%s\n' "$out" | sed 's/^/ /' + # The first `Error:` line is generic ("values don't meet the + # specifications of the schema"); the offending key is on the + # lines after it, so keep a few and flatten them into the cell. + reason=$(printf '%s\n' "$out" | grep -i -m1 -A3 'error' | tr '\n' ' ' | tr -s ' ' | cut -c1-220) + rows="${rows}| \`${f}\` | **FAILED** | ${reason//|/\\|} |"$'\n' + failed=$((failed + 1)) rc=1 fi done < <(find examples -name '*.yaml' | sort) + + { + echo "## Lint — Helm ${{ matrix.helm-version }}" + echo + echo "\`ct lint\`: **$( [[ "${{ steps.ct-lint.outcome }}" == "success" ]] && echo passed || echo failed )**" + echo + echo "Shipped examples: **${ok} rendered**, ${skipped} skipped, ${failed} failed" + echo + echo "| example | result | detail |" + echo "|---------|--------|--------|" + printf '%s' "$rows" + } >> "$GITHUB_STEP_SUMMARY" exit $rc helm-unittest: @@ -170,20 +224,62 @@ jobs: helm plugin install /tmp/helm-unittest - name: Run helm unittest - run: helm unittest charts/graylog + id: unittest + run: | + set -o pipefail + helm unittest charts/graylog 2>&1 | tee /tmp/unittest.log # helm-unittest can only assert on rendered text. These two run the shipped # shell scripts for real: the preStop drain against a stubbed metrics # endpoint, and the init script against fixture volumes (including the # upgrade path, where an existing volume must not be clobbered). - name: preStop drain behaviour tests + id: drain + if: always() run: sh charts/graylog/tests/scripts/prestop-drain-behavior-test.sh - name: init-script behaviour tests + id: init-script # always() so a drain failure above does not hide an init failure here. if: always() run: sh charts/graylog/tests/scripts/init-graylog-behavior-test.sh + - name: Summary + if: always() + run: | + set -u + verdict() { [[ "$1" == "success" ]] && echo "passed" || echo "**FAILED**"; } + + { + echo "## Unit tests — Helm ${{ matrix.helm-version }}" + echo + if [[ -f /tmp/unittest.log ]]; then + # helm-unittest prints "Test Suites: 28 passed, 28 total" and the + # same shape for Tests; lift those two lines verbatim. + suites=$(grep -m1 '^Test Suites:' /tmp/unittest.log | sed 's/Test Suites:[[:space:]]*//') + tests=$(grep -m1 '^Tests:' /tmp/unittest.log | sed 's/Tests:[[:space:]]*//') + echo "| suite group | result |" + echo "|-------------|--------|" + echo "| \`helm unittest\` suites | ${suites:-unknown} |" + echo "| \`helm unittest\` tests | ${tests:-unknown} |" + echo "| preStop drain behaviour | $(verdict "${{ steps.drain.outcome }}") |" + echo "| init-script behaviour | $(verdict "${{ steps.init-script.outcome }}") |" + else + echo "\`helm unittest\` did not run." + fi + + if [[ "${{ steps.unittest.outcome }}" != "success" && -f /tmp/unittest.log ]]; then + echo + echo "
Failing suites" + echo + echo '```' + grep -A3 '^ FAIL' /tmp/unittest.log | head -60 || true + echo '```' + echo + echo "
" + fi + } >> "$GITHUB_STEP_SUMMARY" + helm-ct-install: runs-on: ubuntu-latest needs: [changes, helm-ct-lint, helm-unittest] @@ -258,6 +354,15 @@ jobs: 'NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory' kubectl describe node | sed -n '/Allocated resources/,/Events/p' + # Carried into the job summary: whether the stack fits is the single + # most common reason this job fails, and CPU is the binding constraint. + cpu=$(kubectl get nodes -o jsonpath='{.items[0].status.allocatable.cpu}') + mem=$(kubectl get nodes -o jsonpath='{.items[0].status.allocatable.memory}') + { + echo "NODE_CPU=$cpu" + echo "NODE_MEM=$mem" + } >> "$GITHUB_ENV" + - name: Generate ephemeral CI rootPassword and mask in workflow logs id: ci-root-password run: | @@ -277,7 +382,11 @@ jobs: # Running plus recent warning events, so a pod that cannot be scheduled is # visible within seconds instead of in a post-mortem dump. - name: Run ct install + id: ct-install run: | + started=$(date +%s) + echo "INSTALL_STARTED=$started" >> "$GITHUB_ENV" + watch_cluster() { while true; do echo "::group::cluster state $(date -u +%H:%M:%S)" @@ -287,6 +396,14 @@ jobs: kubectl get events -A --field-selector type=Warning \ --sort-by=.lastTimestamp 2>/dev/null | tail -8 || true echo "::endgroup::" + + # Latest full pod state, kept for the job summary: ct deletes its + # namespace when it finishes, so by summary time the workloads + # under test are gone and cannot be queried any more. + kubectl get pods -A -o custom-columns=\ + 'NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory,STATUS:.status.phase' \ + > /tmp/pods-latest.txt 2>/dev/null || true + sleep 20 done } @@ -294,6 +411,59 @@ jobs: watcher=$! trap 'kill "$watcher" 2>/dev/null || true' EXIT + set -o pipefail ct install --config .github/ct.yaml --all \ --helm-extra-set-args "\ - --set graylog.config.rootPassword=${{ steps.ci-root-password.outputs.value }}" + --set graylog.config.rootPassword=${{ steps.ci-root-password.outputs.value }}" \ + 2>&1 | tee /tmp/ct-install.log + + - name: Summary + if: always() + run: | + set -u + elapsed="unknown" + if [[ -n "${INSTALL_STARTED:-}" ]]; then + elapsed="$(( ( $(date +%s) - INSTALL_STARTED ) / 60 ))m $(( ( $(date +%s) - INSTALL_STARTED ) % 60 ))s" + fi + passed="${{ steps.ct-install.outcome }}" + + { + echo "## Install — Helm ${{ matrix.helm-version }} on Kubernetes ${{ matrix.k8s-version }}" + echo + echo "| | |" + echo "|--|--|" + echo "| result | $( [[ "$passed" == "success" ]] && echo "passed" || echo "**FAILED**" ) |" + echo "| install duration | ${elapsed} |" + echo "| node allocatable | ${NODE_CPU:-?} CPU / ${NODE_MEM:-?} |" + echo "| sizing | \`charts/graylog/ci/ci-values.yaml\` |" + echo + + # What was actually admitted to the node, which is what "does it + # fit" means in practice, snapshotted by the watcher before ct tore + # the namespace down. + echo "
Pod requests, as last observed during the install" + echo + echo '```' + cat /tmp/pods-latest.txt 2>/dev/null || echo "no snapshot captured" + echo '```' + echo + echo "
" + + if [[ "$passed" != "success" ]]; then + echo + echo "### Why it failed" + echo + echo '```' + grep -iE 'error|failed|timed out|unhealthy' /tmp/ct-install.log 2>/dev/null | tail -20 || echo "no log" + echo '```' + echo + echo "
Pods not Running, and recent warnings" + echo + echo '```' + kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide 2>/dev/null || true + kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp 2>/dev/null | tail -25 || true + echo '```' + echo + echo "
" + fi + } >> "$GITHUB_STEP_SUMMARY" From 2acaaa93d830fc3c235798505c0268ac441bc206 Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 15:22:00 -0600 Subject: [PATCH 5/9] ci: report the values each ct job used ct discovers charts/graylog/ci/ci-values.yaml on its own and the workflow passes no sizing flags, so nothing in a run said what the chart was linted or installed with. Both ct jobs now report it. The report comes from rendering the chart with that overlay, not from printing the overlay file. The overlay mentions only a handful of keys, so only the rendered result accounts for chart defaults - MongoDB most of all, where the container names being patched belong to the operator and the pod's reservation is max(max(initContainer), sum(containers)) rather than a number written down anywhere. Per workload it reports replicas, the pod's request, per-container limits and the grace period, then the whole stack's total request, which is the figure that decides whether the install fits on the runner at all. It lives in .github/scripts/ci_values_summary.py because two jobs need it and the quantity arithmetic is worth having in one testable place. In the install job it runs before ct install, so the summary says what was attempted even when the install times out. Both calls are continue-on-error: a step that only describes the run must never fail it. --- .github/scripts/ci_values_summary.py | 178 +++++++++++++++++++++++++++ .github/workflows/lint-and-test.yaml | 29 +++++ 2 files changed, 207 insertions(+) create mode 100644 .github/scripts/ci_values_summary.py diff --git a/.github/scripts/ci_values_summary.py b/.github/scripts/ci_values_summary.py new file mode 100644 index 0000000..d123cc6 --- /dev/null +++ b/.github/scripts/ci_values_summary.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Report the values a ct run actually used, as Markdown for a job summary. + +`ct` discovers charts/graylog/ci/*-values.yaml on its own and the workflow passes +no sizing flags, so nothing in the log says what the chart was installed with. +This renders the chart with the same overlay and reports what came out, which is +the only view that accounts for chart defaults the overlay does not mention - +MongoDB's containers being the obvious case. + +Reads the rendered manifests as JSON on stdin: + + helm template ci charts/graylog -f | yq ea -o=json '[.]' - \ + | ci_values_summary.py --overlay [--extra-set "flag, flag"] +""" + +from __future__ import annotations + +import argparse +import json +import sys + +CPU_SUFFIX = {"m": 0.001, "": 1.0} +MEM_SUFFIX = { + "Ki": 1 / 1024, + "Mi": 1.0, + "Gi": 1024.0, + "K": 1000 / 1024 / 1024, + "M": 1000 * 1000 / 1024 / 1024, + "G": 1000 * 1000 * 1000 / 1024 / 1024, + "": 1 / 1024 / 1024, +} + + +def cpu_cores(value: str | None) -> float: + """Parse a Kubernetes CPU quantity into cores.""" + if not value: + return 0.0 + text = str(value) + if text.endswith("m"): + return float(text[:-1]) * CPU_SUFFIX["m"] + return float(text) + + +def mem_mib(value: str | None) -> float: + """Parse a Kubernetes memory quantity into MiB.""" + if not value: + return 0.0 + text = str(value) + for suffix in ("Ki", "Mi", "Gi", "K", "M", "G"): + if text.endswith(suffix): + return float(text[: -len(suffix)]) * MEM_SUFFIX[suffix] + return float(text) * MEM_SUFFIX[""] + + +def fmt_cpu(cores: float) -> str: + return f"{cores:g}" if cores >= 1 else f"{round(cores * 1000)}m" + + +def fmt_mem(mib: float) -> str: + return f"{mib / 1024:g}Gi" if mib >= 1024 else f"{mib:g}Mi" + + +def requests_of(container: dict, key: str) -> str | None: + return (container.get("resources") or {}).get(key, {}).get("cpu"), ( + container.get("resources") or {} + ).get(key, {}).get("memory") + + +def pod_reservation(containers: list[dict], init: list[dict]) -> tuple[float, float]: + """A pod reserves max(max(initContainer), sum(containers)) on each axis.""" + run_cpu = sum(cpu_cores(requests_of(c, "requests")[0]) for c in containers) + run_mem = sum(mem_mib(requests_of(c, "requests")[1]) for c in containers) + init_cpu = max( + (cpu_cores(requests_of(c, "requests")[0]) for c in init), default=0.0 + ) + init_mem = max((mem_mib(requests_of(c, "requests")[1]) for c in init), default=0.0) + return max(run_cpu, init_cpu), max(run_mem, init_mem) + + +def workload_rows(docs: list[dict]) -> tuple[list[list[str]], float, float]: + """One row per workload, plus the cluster-wide request totals.""" + rows: list[list[str]] = [] + total_cpu = total_mem = 0.0 + + for doc in docs: + if not isinstance(doc, dict): + continue + + if doc.get("kind") == "StatefulSet": + spec = doc["spec"]["template"]["spec"] + replicas = int(doc["spec"].get("replicas", 1)) + cpu, mem = pod_reservation( + spec.get("containers") or [], spec.get("initContainers") or [] + ) + limits = [ + ( + (c.get("resources") or {}).get("limits", {}).get("cpu"), + (c.get("resources") or {}).get("limits", {}).get("memory"), + ) + for c in spec.get("containers") or [] + ] + limit_text = ", ".join( + f"{l[0] or '–'} / {l[1] or '–'}" for l in limits + ) + grace = spec.get("terminationGracePeriodSeconds", "cluster default") + rows.append( + [ + f"`{doc['metadata']['name']}`", + str(replicas), + f"{fmt_cpu(cpu)} / {fmt_mem(mem)}", + limit_text or "–", + f"{grace}s" if isinstance(grace, int) else str(grace), + ] + ) + total_cpu += cpu * replicas + total_mem += mem * replicas + + elif doc.get("kind") == "MongoDBCommunity": + spec = doc["spec"]["statefulSet"]["spec"]["template"]["spec"] + members = int(doc["spec"].get("members", 1)) + int( + doc["spec"].get("arbiters", 0) or 0 + ) + cpu, mem = pod_reservation( + spec.get("containers") or [], spec.get("initContainers") or [] + ) + limit_text = ", ".join( + f"{(c.get('resources') or {}).get('limits', {}).get('cpu') or '–'}" + f" / {(c.get('resources') or {}).get('limits', {}).get('memory') or '–'}" + for c in spec.get("containers") or [] + ) + rows.append( + [ + f"`{doc['metadata']['name']}` (MongoDB {doc['spec'].get('version', '?')})", + str(members), + f"{fmt_cpu(cpu)} / {fmt_mem(mem)}", + limit_text or "operator defaults", + "operator-owned", + ] + ) + total_cpu += cpu * members + total_mem += mem * members + + return rows, total_cpu, total_mem + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--overlay", required=True, help="values file ct discovered") + ap.add_argument("--extra-set", default="", help="flags the workflow adds") + ap.add_argument("--heading", default="Values used") + args = ap.parse_args() + + docs = json.load(sys.stdin) + rows, total_cpu, total_mem = workload_rows(docs) + + out = [f"### {args.heading}", ""] + out.append(f"Overlay: `{args.overlay}` (discovered by `ct`)") + out.append("") + if args.extra_set: + out.append(f"Extra flags: `{args.extra_set}`") + out.append("") + out.append("| workload | replicas | request / pod | limits per container | grace |") + out.append("|---|---|---|---|---|") + for row in rows: + out.append("| " + " | ".join(row) + " |") + out.append("") + out.append( + f"**Whole stack requests {fmt_cpu(total_cpu)} CPU and {fmt_mem(total_mem)}**, " + "counting every replica. A pod reserves " + "`max(max(initContainer), sum(containers))`, so init containers are " + "included where they set the floor." + ) + print("\n".join(out)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index f25b628..3bc5756 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -132,6 +132,21 @@ jobs: set -o pipefail ct lint --config .github/ct.yaml --all 2>&1 | tee /tmp/ct-lint.log + # `ct` discovers ci/*-values.yaml on its own and the workflow passes no + # sizing flags, so without this nothing in the run says what the chart was + # linted against. + - name: Report values used + if: always() + # Advisory only: this step describes the run, it must never fail it. + continue-on-error: true + run: | + helm template ci charts/graylog -f charts/graylog/ci/ci-values.yaml \ + | yq ea -o=json '[.]' - \ + | python3 .github/scripts/ci_values_summary.py \ + --overlay charts/graylog/ci/ci-values.yaml \ + --heading "Values linted — Helm ${{ matrix.helm-version }}" \ + >> "$GITHUB_STEP_SUMMARY" + # `ct lint` never looks at examples/, so a shipped example could reference a # removed key, miss a required one, or trip a template-time guard and no one # would find out until a user copied it. `helm template` also enforces @@ -370,6 +385,20 @@ jobs: echo "::add-mask::${password}" echo "value=${password}" >> "$GITHUB_OUTPUT" + # Written before the install rather than after, so the summary says what + # was attempted even when the install times out or the job is cancelled. + - name: Report values used + # Advisory only: this step describes the run, it must never fail it. + continue-on-error: true + run: | + helm template ci charts/graylog -f charts/graylog/ci/ci-values.yaml \ + | yq ea -o=json '[.]' - \ + | python3 .github/scripts/ci_values_summary.py \ + --overlay charts/graylog/ci/ci-values.yaml \ + --extra-set "--set graylog.config.rootPassword=" \ + --heading "Values installed — Helm ${{ matrix.helm-version }} / Kubernetes ${{ matrix.k8s-version }}" \ + >> "$GITHUB_STEP_SUMMARY" + # All sizing lives in charts/graylog/ci/ci-values.yaml, which ct discovers # automatically. It used to be duplicated here as --set flags, which take # precedence over the values file — so ci-values.yaml's graylog memory From 54b43516b948b512dbbc61b5f1ad972272268fb0 Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 15:34:45 -0600 Subject: [PATCH 6/9] ci: publish the full rendered manifests as a run artifact Reviewing a chart change meant rendering it locally to see what actually changed. The lint job now renders the chart six ways - chart defaults, the CI overlay, and each values-overlay example - and uploads them as rendered-manifests-helm-, kept for 14 days. Reviewing a change becomes downloading that and the same artifact from main, and diffing the two. It lives in the lint job because rendering needs no cluster, so the artifact exists even for a change that skips the install matrix. The bundle is 232K. Secret values are redacted before upload. helm template generates a root password, the password_secret pepper and both MongoDB passwords for every value left empty, which is the default, and artifacts on a public repository are world-downloadable: those would read as leaked credentials to anyone who found them, and to a secret scanner. Keys are kept so a diff still shows one appearing or disappearing, and the placeholder is valid base64 so the manifests stay loadable. The step then proves the redaction worked and discards the bundle rather than uploading it if any value survived. A failed redaction must not become a published credential. --- .github/workflows/lint-and-test.yaml | 90 ++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index 3bc5756..e1e17ef 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -147,6 +147,96 @@ jobs: --heading "Values linted — Helm ${{ matrix.helm-version }}" \ >> "$GITHUB_STEP_SUMMARY" + # The rendered manifests are the chart's actual output, and reading them is + # how you answer "what does this PR change" without installing anything. + # Kept as an artifact rather than printed: a full render is thousands of + # lines, too much for a log and far too much for a job summary. + - name: Render full manifests + if: always() + continue-on-error: true + run: | + set -u + mkdir -p rendered + + render() { + local name="$1"; shift + if helm template graylog charts/graylog "$@" > "rendered/${name}.yaml" 2>"rendered/${name}.err"; then + rm -f "rendered/${name}.err" + printf '%-46s %6s lines, %4s objects\n' "$name" \ + "$(wc -l < "rendered/${name}.yaml" | tr -d ' ')" \ + "$(grep -c '^kind:' "rendered/${name}.yaml" || true)" + else + mv "rendered/${name}.yaml" "rendered/${name}.failed.yaml" 2>/dev/null || true + printf '%-46s FAILED (see %s.err)\n' "$name" "$name" + fi + } + + # Chart defaults are the production shape, and the one people actually + # deploy; the CI overlay is what this workflow tests. + render "00-chart-defaults" + render "01-ci-values" -f charts/graylog/ci/ci-values.yaml + + while IFS= read -r f; do + grep -q '^apiVersion:' "$f" && continue # raw manifest, not an overlay + render "example-$(basename "${f%.yaml}")" -f "$f" + done < <(find examples -name '*.yaml' | sort) + + # `helm template` generates credentials for every key left empty - the + # root password, the password_secret pepper, both MongoDB passwords. + # They are throwaway values that never reach a cluster, but artifacts on + # a public repository are world-downloadable and these would read as + # leaked credentials to anyone (and to a secret scanner). Redact the + # values, keep the keys, so a diff still shows a key appearing or + # disappearing. The placeholder is valid base64, so the manifests stay + # loadable. + for f in rendered/*.yaml; do + yq -i '(select(.kind == "Secret" and has("data")) | .data) |= with_entries(.value = "UkVEQUNURUQ=")' "$f" + yq -i '(select(.kind == "Secret" and has("stringData")) | .stringData) |= with_entries(.value = "REDACTED")' "$f" + done + + # A failed redaction must not become a published credential, so prove + # it worked and throw the bundle away if it did not. + leaked="$(yq ea 'select(.kind == "Secret") | .data // {} | to_entries | .[] | select(.value != "UkVEQUNURUQ=") | .key' rendered/*.yaml | grep -c . || true)" + if [[ "$leaked" != "0" ]]; then + echo "::error::redaction left ${leaked} secret value(s); discarding renders instead of uploading" + rm -rf rendered + exit 1 + fi + echo "all Secret values redacted" + + ls -l rendered/ + + - name: Upload rendered manifests + id: upload-rendered + if: always() + continue-on-error: true + uses: actions/upload-artifact@v4 + with: + name: rendered-manifests-helm-${{ matrix.helm-version }} + path: rendered/ + # Long enough to review a PR and bisect a regression, short enough not + # to accumulate a render of every push forever. + retention-days: 14 + if-no-files-found: warn + + - name: Note the artifact in the summary + if: always() + continue-on-error: true + run: | + { + echo "### Rendered manifests" + echo + echo "Full \`helm template\` output for chart defaults, the CI overlay and every" + echo "shipped example: [\`rendered-manifests-helm-${{ matrix.helm-version }}\`](${{ steps.upload-rendered.outputs.artifact-url }})" + echo "(kept 14 days). Download and diff against the same artifact on \`main\` to" + echo "see exactly what a change does to the output." + echo + echo "\`Secret\` values are redacted — keys are kept, so a diff still shows one" + echo "appearing or disappearing. Everything Helm generates for an empty value" + echo "(root password, \`password_secret\` pepper, MongoDB passwords) would" + echo "otherwise be world-downloadable from a public repository." + } >> "$GITHUB_STEP_SUMMARY" + # `ct lint` never looks at examples/, so a shipped example could reference a # removed key, miss a required one, or trip a template-time guard and no one # would find out until a user copied it. `helm template` also enforces From 79248034a8ffd66f05905809b7c463e6f6e52403 Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 15:58:22 -0600 Subject: [PATCH 7/9] wip --- .github/workflows/lint-and-test.yaml | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index e1e17ef..2cd0a0b 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -6,6 +6,16 @@ on: push: branches: ["main"] +# One run per ref. Pushing three times to a branch used to leave three full runs +# in flight - up to 27 jobs, of which only the last mattered - and they compete +# for the same concurrency slots as everyone else's runs. +# +# Pushes to `main` are never cancelled: each merge commit's result is a record of +# whether that commit is good, and nothing else re-establishes it. +concurrency: + group: tests-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + jobs: # Decides which test jobs are worth running for this change. # @@ -21,6 +31,8 @@ jobs: examples: ${{ steps.filter.outputs.examples }} steps: - name: Checkout + # The only job that needs history: the diff below is computed against the + # merge base, which a shallow clone does not contain. uses: actions/checkout@v6 with: fetch-depth: 0 @@ -114,9 +126,9 @@ jobs: - v4.2.0 steps: - name: Checkout + # Shallow on purpose: `ct` is invoked with --all and never diffs against a + # target branch, so no job past `changes` needs history. uses: actions/checkout@v6 - with: - fetch-depth: 0 - name: Set up Helm uses: azure/setup-helm@v5 @@ -306,9 +318,9 @@ jobs: - v4.2.0 steps: - name: Checkout + # Shallow on purpose: `ct` is invoked with --all and never diffs against a + # target branch, so no job past `changes` needs history. uses: actions/checkout@v6 - with: - fetch-depth: 0 - name: Set up Helm uses: azure/setup-helm@v5 @@ -415,9 +427,9 @@ jobs: k8s-version: v1.34.3 steps: - name: Checkout + # Shallow on purpose: `ct` is invoked with --all and never diffs against a + # target branch, so no job past `changes` needs history. uses: actions/checkout@v6 - with: - fetch-depth: 0 - name: Set up Helm uses: azure/setup-helm@v5 From f416cc2820c7f6b8078fc203990b06b51770474d Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Fri, 21 Aug 2026 16:19:05 -0600 Subject: [PATCH 8/9] ci: Pinning actions by sha --- .github/dependabot.yml | 28 +++++++++++++++++++++++++++ .github/workflows/lint-and-test.yaml | 25 +++++++++++++----------- .github/workflows/release-please.yaml | 2 +- 3 files changed, 43 insertions(+), 12 deletions(-) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..4691974 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,28 @@ +# Keeps the SHA-pinned actions in .github/workflows current. +# +# Pinning by commit SHA is what stops a moved tag from changing what runs in CI, +# but a pin never updates itself: without this, the workflows would silently +# freeze on whatever was current the day they were written, security fixes +# included. Dependabot reads the `# vX.Y.Z` comment beside each SHA, so its pull +# requests bump both together and stay readable. +version: 2 +updates: + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + # Repository tooling must not use fix:/feat:, or it would land in the chart's + # changelog and could trigger a release of the chart. `ci:` is a hidden + # changelog section, so these never reach a release. + commit-message: + prefix: ci + # One pull request for the routine bumps rather than five. Majors stay + # separate: those are the ones that change behaviour and deserve their own + # review. + groups: + actions-minor-patch: + patterns: ["*"] + update-types: ["minor", "patch"] + open-pull-requests-limit: 5 + labels: + - dependencies diff --git a/.github/workflows/lint-and-test.yaml b/.github/workflows/lint-and-test.yaml index 2cd0a0b..860d3cc 100644 --- a/.github/workflows/lint-and-test.yaml +++ b/.github/workflows/lint-and-test.yaml @@ -33,7 +33,7 @@ jobs: - name: Checkout # The only job that needs history: the diff below is computed against the # merge base, which a shallow clone does not contain. - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: fetch-depth: 0 @@ -128,15 +128,15 @@ jobs: - name: Checkout # Shallow on purpose: `ct` is invoked with --all and never diffs against a # target branch, so no job past `changes` needs history. - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Set up Helm - uses: azure/setup-helm@v5 + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1 with: version: ${{ matrix.helm-version }} - name: Set up chart-testing - uses: helm/chart-testing-action@v2.8.0 + uses: helm/chart-testing-action@6ec842c01de15ebb84c8627d2744a0c2f2755c9f # v2.8.0 - name: Run ct lint id: ct-lint @@ -222,7 +222,10 @@ jobs: id: upload-rendered if: always() continue-on-error: true - uses: actions/upload-artifact@v4 + # v7 rather than v4: v4 runs on Node 20, which the runners now force onto + # Node 24 with a deprecation warning on every run. Inputs and the + # artifact-url output the summary step reads are unchanged. + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: rendered-manifests-helm-${{ matrix.helm-version }} path: rendered/ @@ -320,10 +323,10 @@ jobs: - name: Checkout # Shallow on purpose: `ct` is invoked with --all and never diffs against a # target branch, so no job past `changes` needs history. - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Set up Helm - uses: azure/setup-helm@v5 + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1 with: version: ${{ matrix.helm-version }} @@ -429,18 +432,18 @@ jobs: - name: Checkout # Shallow on purpose: `ct` is invoked with --all and never diffs against a # target branch, so no job past `changes` needs history. - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Set up Helm - uses: azure/setup-helm@v5 + uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1 with: version: ${{ matrix.helm-version }} - name: Set up chart-testing - uses: helm/chart-testing-action@v2.8.0 + uses: helm/chart-testing-action@6ec842c01de15ebb84c8627d2744a0c2f2755c9f # v2.8.0 - name: Create kind cluster - uses: helm/kind-action@v1.14.0 + uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0 with: node_image: kindest/node:${{ matrix.k8s-version }} diff --git a/.github/workflows/release-please.yaml b/.github/workflows/release-please.yaml index 0c5d263..96a447e 100644 --- a/.github/workflows/release-please.yaml +++ b/.github/workflows/release-please.yaml @@ -18,7 +18,7 @@ jobs: release-please: runs-on: ubuntu-latest steps: - - uses: googleapis/release-please-action@v5 + - uses: googleapis/release-please-action@45996ed1f6d02564a971a2fa1b5860e934307cf7 # v5 id: release with: token: ${{ secrets.GITHUB_TOKEN }} From 24e717cdb9e330e348f41a917c0d8a73422ed05f Mon Sep 17 00:00:00 2001 From: alix-graylog Date: Thu, 27 Aug 2026 10:06:42 -0600 Subject: [PATCH 9/9] ci(gha): Removing unused import --- .github/scripts/ci_values_summary.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/scripts/ci_values_summary.py b/.github/scripts/ci_values_summary.py index d123cc6..0248b3d 100644 --- a/.github/scripts/ci_values_summary.py +++ b/.github/scripts/ci_values_summary.py @@ -13,8 +13,6 @@ | ci_values_summary.py --overlay [--extra-set "flag, flag"] """ -from __future__ import annotations - import argparse import json import sys