Skip to content

fix: don't render a null pod template annotations key in the chart - #9460

Open
oreonl wants to merge 1 commit into
aws:mainfrom
oreonl:fix/chart-null-pod-annotations
Open

fix: don't render a null pod template annotations key in the chart#9460
oreonl wants to merge 1 commit into
aws:mainfrom
oreonl:fix/chart-null-pod-annotations

Conversation

@oreonl

@oreonl oreonl commented Jul 30, 2026

Copy link
Copy Markdown

Fixes #N/A

Description

The pod template's annotations: key is emitted unconditionally, with its only
content behind {{- with .Values.podAnnotations }}. Since podAnnotations
defaults to {}, the chart renders a mapping key with no children:

$ helm template karpenter charts/karpenter | yq 'select(.kind == "Deployment") | .spec.template.metadata'
labels:
  app.kubernetes.io/name: karpenter
  app.kubernetes.io/instance: karpenter
annotations:        # <-- explicit null, not an omitted field

annotations: null is not equivalent to omitting the key, and the difference is
load-bearing for anything that computes a strategic merge patch from the
rendered manifest.

helm upgrade builds its patch with
strategicpatch.CreateThreeWayMergePatch(old_release, rendered, live).
That function protects fields written by other actors by only emitting deletions
found between the old release manifest and the rendered manifest — live-only
fields fall into the deletion path and are skipped via IgnoreDeletions. But a
key that is present and null in the rendered manifest never reaches that
path. It hits the type-mismatch branch in
diffMaps,
which is gated only on IgnoreChangesAndAdditions:

// If the types don't match, replace
if reflect.TypeOf(originalValue) != reflect.TypeOf(modifiedValue) {
    if !diffOptions.IgnoreChangesAndAdditions {
        patch[key] = modifiedValue   // -> annotations: null
    }
    continue
}

map vs nil are different types, so the computed patch contains
spec.template.metadata.annotations: null, and in a strategic merge patch a null
value deletes the entire map.

The practical effect is that every annotation any other actor has put on
karpenter's pod template is destroyed on upgrade, and shows up as permanent
phantom drift in the meantime. Easiest reproduction is kubectl rollout restart,
which sets kubectl.kubernetes.io/restartedAt:

$ kubectl -n karpenter rollout restart deployment/karpenter
$ helm diff upgrade --three-way-merge karpenter charts/karpenter -n karpenter --output dyff

spec.template.metadata
- one map entry removed:
annotations:
  kubectl.kubernetes.io/restartedAt: "2026-07-29T13:41:28Z"

A server-side dry run confirms the API server drops the whole map. The same shows
up as a never-reconciling diff in Argo CD and helmfile. Charts that guard the key
are unaffected: the annotation is preserved and no drift is reported.

Worse, the deletion is invisible in advance unless the diff consults the cluster.
helm diff upgrade without --three-way-merge compares the stored release
manifest against the newly rendered one; both contain annotations: null, so it
reports no changes — and the upgrade then rolls the Deployment anyway, because
the patch is only computed once the live object is the third input. A chart that
guards the key does not have this discrepancy: there, "no changes" is accurate,
since a live-only annotation is classified as a deletion and skipped. So the
rendered null turns an unrequested restart of the karpenter controller into one
that no pre-flight diff will warn about.

This is not limited to Helm 3. Helm 4 defaults new installs to server-side
apply, but helm upgrade defaults --server-side to auto, and
auto is releaseApplyMethod == "ssa"
— it preserves whatever the release already used rather than migrating it. A
release created before Helm 4 has an empty ApplyMethod, which is
treated as client-side,
so its first Helm 4 upgrade resolves to client-side and records "csa", and
every upgrade after that reads it back and stays there. Nothing migrates a
release to server-side apply short of an explicit --server-side=true.

$ helm get metadata karpenter -n karpenter -o json | jq .applyMethod
"csa"

Server-side apply resolves annotation conflicts by field ownership instead and is
not affected the same way, but the rendered null is wrong regardless.

Why the key is currently outside the guard

It was correct until the checksum it was sharing a block with went away:

So the chart has rendered annotations: null from v0.33.0 through v1.14.0 and
main.

Change

Restores the guard, which is also how this chart already handles the
Deployment's own object-level annotations
({{- with .Values.additionalAnnotations }}) and the adjacent podLabels:

-      annotations:
-        {{- with .Values.podAnnotations }}
-          {{- toYaml . | nindent 8 }}
-        {{- end }}
+      {{- with .Values.podAnnotations }}
+      annotations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}

No values changed, so no values.yaml or generated README.md update is needed.

How was this change tested?

helm template charts/karpenter, inspecting .spec.template.metadata:

  • default values — the annotations key is now absent, where before it rendered
    as an explicit null.
  • --set podAnnotations.foo=bar — renders annotations: {foo: bar}, unchanged
    from before.
  • --set-json 'podAnnotations={}' — key absent, as with the default.

End to end on a live cluster: kubectl rollout restart deployment/karpenter,
then helm diff upgrade --three-way-merge. Before the change it reports the
whole annotations map being removed; after it reports no diff, and the
restartedAt annotation survives a subsequent helm upgrade.

Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: #
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

🤖 Generated with Claude Code

The pod template's `annotations:` key was emitted unconditionally with its
only content behind `{{- with .Values.podAnnotations }}`. Since
`podAnnotations` defaults to `{}`, the chart rendered a bare mapping key,
i.e. `annotations: null`.

That is not equivalent to omitting the key. `helm upgrade` computes a
three-way strategic merge patch, and a present-but-null key hits the
type-mismatch branch of `diffMaps` rather than the deletion path that
`IgnoreDeletions` protects. The resulting patch carries
`spec.template.metadata.annotations: null`, which deletes the whole map --
destroying any annotation written by another actor (e.g. `kubectl rollout
restart`) and showing up as permanent phantom drift in the meantime.

The key was correctly guarded through v0.23.0. aws#3345 moved it out of the
guard to make room for an unconditional `checksum/settings`, which was fine
while the map always had a child; aws#5159 later dropped that checksum and left
the bare key behind. This restores the guard, matching how the chart already
handles `additionalAnnotations` and the adjacent `podLabels`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@oreonl
oreonl requested a review from a team as a code owner July 30, 2026 00:48
@oreonl
oreonl requested a review from jmdeal July 30, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant