fix(controller): Keep pod-template-hash stable across k8s library upgrade by aligning with the upstream Deployment controller.

[zachaller]

Fixes #4696

Summary

Upgrading the controller across a Kubernetes library bump (v1.8.x → v1.9.0, i.e. k8s v0.29 → v0.34) causes every Rollout to recompute a different rollouts-pod-template-hash. That spuriously creates a new revision which progresses through the canary steps and parks at the first indefinite pause, even though nothing in the user's spec.template actually changed.

This PR removes the correctness dependency on byte-stable hashing (the hash-label lookup remains as a fast path) and aligns Argo Rollouts with how the upstream Kubernetes Deployment controller decides ReplicaSet identity:

1. Compute pod-template-hash with the upstream controller.ComputeHash (spew/reflect based) instead of json.Marshal, so the hash is immune to JSON-serialization changes.
2. Fall back to semantic pod-template equality (PodTemplateEqualIgnoreHash) when no ReplicaSet carries the computed hash label, exactly like deploymentutil.FindNewReplicaSet. The hash-label lookup remains the first (fast) tier: the label was written by this controller from the same desired template, so a match is authoritative even when the live template has since diverged (apiserver defaulting from a newer server, or mutating admission on ReplicaSets).

The net effect: a controller/library upgrade is a no-op for existing Rollouts — the existing ReplicaSet is adopted (keeping its original hash label), so no new revision and no redeploy.

Background / root cause

hash.ComputePodTemplateHash previously computed an FNV-32a sum over json.Marshal(template). The hash function itself didn't change between releases — the Kubernetes libraries did:

	k8s.io/api	k8s.io/apimachinery	k8s.io/kubernetes
v1.8.3	v0.29.3	v0.29.3	v1.29.3
v1.9.0	v0.34.x	v0.34.x	v1.34.x

In apimachinery v0.34, the JSON struct tag on metav1.ObjectMeta.CreationTimestamp gained omitzero (kubernetes/kubernetes#130989):

// apimachinery v0.29.3
CreationTimestamp Time `json:"creationTimestamp,omitempty" ...`
// apimachinery v0.34.x
CreationTimestamp Time `json:"creationTimestamp,omitempty,omitzero" ...`

A pod template's creationTimestamp is always the zero value. Previously it serialized as "creationTimestamp":null; with omitzero (honored by Go 1.24+) it is omitted entirely. That single byte change flips the JSON-based hash for 100% of Rollouts on upgrade. This is a recurrence of the long-standing class of bug in #88.

How the upstream Deployment controller avoids this

The Deployment controller never had this problem because:

• It does not hash via JSON. controller.ComputeHash feeds the template through hashutil.DeepHashObject (the spew library, reflect-based). JSON struct tags such as omitzero are irrelevant to it.
• It does not identify the current ReplicaSet by hash label. deploymentutil.FindNewReplicaSet iterates ReplicaSets and compares with EqualIgnoreHash (a semantic apiequality.Semantic.DeepEqual of the pod templates). Even if the hashing algorithm changes, an existing ReplicaSet is still recognized as current.

This PR brings Argo Rollouts in line with both behaviors, but deliberately keeps the hash-label lookup as a fast first tier. Unlike the in-tree Deployment controller, Argo Rollouts is not version-locked to the apiserver: a newer apiserver can default pod-template fields our vendored libraries don't know, and mutating admission can rewrite ReplicaSet templates. In both cases semantic comparison would fail forever; the hash label — written by this controller from the same desired template — still matches and prevents an unwarranted redeploy (and collisionCount churn).

Changes

Production code

utils/hash/hash.go — ComputePodTemplateHash now delegates to the upstream Deployment controller's hash:

func ComputePodTemplateHash(template *corev1.PodTemplateSpec, collisionCount *int32) string {
	return controller.ComputeHash(template, collisionCount)
}

This makes the hash spew-based (matching Deployment/ReplicaSet labeling going forward) and immune to JSON-tag changes like omitzero.

utils/replicaset/replicaset.go

• FindNewReplicaSet is reduced from three tiers to two: (1) lookup by pod-template-hash label computed with the current algorithm, then (2) semantic pod-template equality, mirroring deploymentutil.FindNewReplicaSet. The "deprecated hash" tier is gone (our hash is controller.ComputeHash now, so the two label searches collapsed into one). When the fallback adopts a ReplicaSet whose label doesn't match the computed hash, an Infof is logged so operators can observe the one-time transition.
• Within the semantic tier, the ReplicaSet recorded in status.currentPodHash is preferred when it still matches the desired template; otherwise the oldest match is reused (upstream behavior). This matters when several retained ReplicaSets carry identical templates with different hash labels — the fleet state v1.9.0 created by spuriously duplicating revisions (Upgrading controller to v1.9.0 triggers spurious new revisions and Canary pause-for-promotion on all Rollouts due to pod template hash change from k8s library bump (#4497) #4696). Without the preference, oldest-first would resurrect the scaled-down pre-1.9.0 twin and trigger a second redeploy for rollouts whose spurious revision was already promoted. On a genuine template change the current ReplicaSet no longer matches semantically, so the preference is a no-op and rollback-reuse behaves exactly like the upstream Deployment controller.
• New exported helper ReplicaSetTemplateMatchesRollout encapsulates the rollouts-specific normalization before the semantic compare: strips injected ephemeral canary/stable metadata and the injected anti-affinity rule from the live template. Used by both FindNewReplicaSet and the hash-collision check (see rollout/sync.go below).
• PodTemplateEqualIgnoreHash now runs both the live and desired templates through library defaulting (defaultPodTemplateSpec) before Semantic.DeepEqual. Previously only the desired side was defaulted; defaulting both keeps the comparison stable when a live ReplicaSet was created by an older library that defaulted fewer fields. The spec.serviceAccount carve-out from #1356 is unchanged.
• CheckPodSpecChange is unchanged in behavior. Its control flow is identical to before (compare status.CurrentPodHash against the adopted RS's label, or against the freshly computed hash when no RS is found); only the underlying hash implementation changed. Doc comment expanded to explain why adopting via label keeps detection stable across the upgrade.
• GenerateReplicaSetAffinity and IfInjectedAntiAffinityRuleNeedsUpdate now take the ReplicaSet's pod-template-hash label (podHash) and compare it against status.StableRS, instead of recomputing a hash from the desired template. Both are stored label values of the same provenance, so the "are we mid-rollout?" decision is not skewed when the stored stable hash predates the hashing change. See "Edge cases handled" below.

rollout/sync.go

• Call sites pass the live hash to the anti-affinity helpers:
  • update path (adopted RS): replicasetutil.GetPodTemplateHash(c.newRS)
  • create path: the newly computed podTemplateSpecHash (computation reordered to occur before the call)
• The AlreadyExists hash-collision check in createDesiredReplicaSet now uses ReplicaSetTemplateMatchesRollout instead of comparing the raw live template against the rollout template. Canary/blue-green ReplicaSets are created with ephemeral metadata injected, so the raw comparison could misread the controller's own ReplicaSet as a hash collision and bump collisionCount (a pre-existing bug, fixed here because this path is adjacent to the matching changes). Hash matching is deliberately not used in this path — a genuine collision has a matching hash by definition, so only the semantic check is valid there.

rollout/experiment.go — GetExperimentFromTemplate now prefers the new ReplicaSet's stored pod-template-hash label for the experiment's name and hash label, falling back to the computed hash only when there is no ReplicaSet yet. This keeps experiments created mid-transition consistent with the adopted ReplicaSet they run against (matching how rollout/analysis.go already derives its labels).

Edge cases handled

• Upgrade with unchanged template (the bug): the old json-era label doesn't match the new computed hash, so FindNewReplicaSet adopts the existing ReplicaSet via the semantic fallback; the status keeps the adopted RS's original (json-era) label; CheckPodSpecChange sees status.CurrentPodHash == newRS.label → no change → no new revision.
• Genuine template change: no ReplicaSet matches by label or semantically → FindNewReplicaSet returns nil → CheckPodSpecChange reports a change → a new ReplicaSet is created and labeled with the new spew hash.
• Diverged live template (version skew / mutating admission): a ReplicaSet whose live template was altered after creation — a newer apiserver defaulting fields unknown to our vendored libraries, or a webhook injecting metadata — no longer matches semantically, but is still found by its hash label (tier 1), so no redeploy and no collisionCount churn.
• Twin ReplicaSets left behind by v1.9.0: when the spurious duplicate revision was already promoted, the running ReplicaSet (recorded in status.currentPodHash) is adopted rather than its older identical-template twin, so upgrading to this version is a no-op instead of a second redeploy. When status.currentPodHash matches no retained ReplicaSet, the oldest semantic match is reused, as upstream does.
• Anti-affinity + upgrade: a fully-stable Rollout using anti-affinity will not have a spurious anti-affinity rule injected into its adopted stable ReplicaSet just because the stored stable hash predates the hashing change. The decision compares the RS's own stored label to status.StableRS (equal for the stable RS) rather than a recomputed hash.

Tests

• utils/hash/hash_test.go — replaced the brittle golden-value assertion (which had to be updated on every k8s bump) with MatchesDeploymentControllerComputeHash, asserting our hash equals controller.ComputeHash.
• utils/replicaset/replicaset_test.go
  • TestFindNewReplicaSet now covers both tiers: lookup by current hash label, and semantic adoption when the label is stale.
  • TestFindNewReplicaSetAcrossHashChange (new) — an unchanged template whose RS carries a stale hash label is adopted; a real spec change is not; with twin identical-template ReplicaSets, the one recorded in status.currentPodHash wins over an older twin, with oldest-first as the fallback. Stable across future k8s upgrades (no hashes to update).
  • TestFindNewReplicaSetWithDivergedLiveTemplate (new) — pins the tier-1 protection: an RS whose live template was mutated after creation (simulated webhook-injected annotation) is still found via its matching hash label.
  • TestPodTemplateEqualIgnoreHashAcrossLibraryDefaulting (new) — an under-defaulted live template (older library) still matches the semantically identical desired template.
  • TestGenerateReplicaSetAffinity / TestIfInjectedAntiAffinityRuleNeedsUpdate / TestRemoveInjectedAntiAffinityRule updated for the new podHash parameter.
  • TestGenerateReplicaSetAffinityStableAcrossHashChange (new) — pins the anti-affinity upgrade edge case: a stable Rollout whose StableRS is an old-format label (deliberately != the recomputed hash) gets no injection.
• rollout/controller_test.go — TestPodTemplateHashEquivalence rewritten to assert the runtime-relevant property (two equivalent quantity formats compare equal via PodTemplateEqualIgnoreHash, so they are adopted, not redeployed) plus a self-consistent RS-name sanity check, instead of a hardcoded hash.
• rollout/ephemeralmetadata_test.go — the "ReplicaSet already patched" fixtures now set the ephemeral-metadata annotation and preserve realistic template labels, so semantic adoption works (these previously relied on label-based lookup).
• utils/conditions/rollouts_test.go and utils/experiment/experiment_test.go — compute expected hashes dynamically instead of pinning literal values that drift on every k8s bump.
• test/e2e/canary_test.go — the three hardcoded pod-template-hash selector assertions (which carried // NOTE: This must be updated for every k8s version upgrade) are replaced with assertCanarySelectorPointsToStable, a helper that reads the expected hash from the live stable (revision 1) ReplicaSet. Never needs updating on a hashing/library change.

Behavior / compatibility

• No redeploy on upgrade for Rollouts whose spec.template is unchanged. Existing ReplicaSets keep their original hash labels; services, selectors, status.stableRS, and status.currentPodHash continue to use those values. The new spew hash is only used when a genuinely new ReplicaSet is created (a real template change). Old and new ReplicaSets can coexist with different hash-label formats until the old ones are scaled down — adoption is by template, not by label format.
• CheckPodSpecChange mid-rollout step-reset semantics are preserved.
• Equivalent resource-quantity formatting (e.g. 2000m vs 2) is treated as equal by Semantic.DeepEqual (quantities compare by value), so it does not cause churn. Note the spew-based hash itself is not format-stable for quantities — adoption relies on the label/semantic matching above, not on hash stability.

Upgrading from v1.9.0 specifically

v1.9.0 already shipped the broken hash, so fleets are not in a clean pre-bug state — the spurious revisions it created leave twin ReplicaSets (identical templates, different hash labels) in revision history. What this version does for each state:

• From v1.8.x (never ran v1.9.0, or rolled the controller back): the current ReplicaSet is adopted semantically — no redeploy. The rollback case also works because after flipping back under v1.8.x the active ReplicaSet is the older twin, and it is also the one status.currentPodHash records.
• From v1.9.0, spurious revision still paused/progressing: status.currentPodHash points at the spurious ReplicaSet, which is adopted — the rollout stays exactly as v1.9.0 left it (paused awaiting promote/abort). No automatic state change on controller upgrade.
• From v1.9.0, spurious revision already promoted: status.currentPodHash points at the running ReplicaSet, which is adopted — the upgrade is a no-op. The scaled-down pre-v1.9.0 twin remains in revision history until it ages out via revisionHistoryLimit.
• Created entirely under v1.9.0: single ReplicaSet, adopted semantically — no redeploy.

In all cases adopted ReplicaSets keep their existing (json-era) hash labels permanently; only genuinely new revisions get spew-format names/labels. Services, traffic routing, and analysis key off stored labels, so mixed label formats coexist safely — but external tooling that pattern-matches hash values (dashboards, log queries) will see both formats during the transition.

Notes / tradeoffs

• The semantic DeepEqual fallback (defaulting + deep compare per ReplicaSet) only runs when the O(1) hash-label lookup misses — i.e. once per genuine template change or hashing transition — so steady-state reconcile cost is unchanged.
• GenerateReplicaSetAffinity and IfInjectedAntiAffinityRuleNeedsUpdate gained a podHash parameter — a signature change to two exported functions in utils/replicaset. All in-tree callers are updated; external importers (unlikely) would need to pass the ReplicaSet's hash.
• Because the hash source changed, hardcoded pod-template-hash values in tests no longer match; affected unit fixtures now compute hashes dynamically and the e2e selector assertions read the live value. There are intentionally no new golden-hash assertions to maintain.

Validation

• go build ./... — clean.
• go vet -tags e2e ./test/e2e/ — clean.
• Unit suites pass: utils/hash, utils/replicaset, utils/conditions, utils/experiment, rollout, experiments.
• Manual reasoning + regression tests confirm an upgrade with an unchanged template adopts the existing ReplicaSet (no new revision), while a real template change still creates one.

Checklist

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• My builds are green. Try syncing with master if they are not.
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• I have run all tests locally (including the flaky ones) and they pass on my workstation.
• I have used LLM/AI/Agent tools for this PR but I am responsible for all code of this PR.
• I understand what the code does and WHY/HOW it works in several scenarios.
• I know if my code is just adding new functionality or changing old functionality for existing users.
• My organization is added to USERS.md.

[github-actions]

Published E2E Test Results

  4 files    4 suites   3h 53m 51s ⏱️
120 tests 104 ✅  7 💤  9 ❌
498 runs  453 ✅ 28 💤 17 ❌

For more details on these failures, see this check.

Results for commit dd62288.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 482 tests   2 482 ✅  3m 20s ⏱️
  129 suites      0 💤
    1 files        0 ❌

Results for commit dd62288.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 90.32258% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.11%. Comparing base (9dd0811) to head (dd62288).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
rollout/experiment.go	33.33%	1 Missing and 1 partial ⚠️
rollout/sync.go	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4755   +/-   ##
=======================================
  Coverage   85.10%   85.11%           
=======================================
  Files         164      164           
  Lines       18998    18987   -11     
=======================================
- Hits        16169    16160    -9     
+ Misses       1987     1986    -1     
+ Partials      842      841    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[kostis-codefresh]

It seems that Flagger has a similar problem fluxcd/flagger#1673

[kostis-codefresh]

It would be great if we also include testing scenarios. At the very least

1. What happens when somebody has already updated to 1.9.0 and now updates to this (next version)
2. What happens if somebody never updated to 1.9.x and went from 1.8.x straight to this version