HelmRelease retries forever on StatefulSet immutable-field Forbidden with no Stalled remediation

[RaviTharuma]

Describe the bug

When a HelmRelease upgrade fails because Kubernetes rejects a StatefulSet immutable-field change, helm-controller retries forever on every spec.interval. The HelmRelease stays Ready=False, and notification-controller (when wired) emits an error event on every reconcile. There is no Stalled (or equivalent) remediation path that tells the operator “this will never succeed without human intervention.”

Classic Kubernetes API error (exact class):

StatefulSet.apps is invalid: spec: Forbidden: updates to statefulset spec for fields other than
replicas, ordinals, template, updateStrategy, persistentVolumeClaimRetentionPolicy and minReadySeconds

This is a well-known GitOps friction class for charts that render StatefulSets (official Loki-class Helm charts under a Flux HelmRelease are a common example). Once the live STS and the newly rendered STS diverge on an immutable field, no amount of helm-controller retry will fix it. Continuing to re-apply the same upgrade every interval only burns API load and alert budget.

Steps to reproduce

1. Deploy a HelmRelease for a chart that creates a StatefulSet (e.g. official Loki Helm chart class).
2. Change values (or chart version) such that the rendered STS differs on an immutable field (volumeClaimTemplates, serviceName, selector, etc. — whatever the chart mutation produces).
3. Observe helm-controller attempt the upgrade; the Kubernetes API returns Forbidden as above.
4. Wait successive spec.interval periods:
   • HelmRelease remains Ready=False with the same failure class
   • Controller re-attempts the doomed upgrade every interval
   • If an Alert covers HelmRelease errors, notification-controller re-notifies each time

Expected behavior

Detect this API error class and surface a terminal-ish condition instead of infinite silent hope:

1. Set a condition such as Stalled=True (or keep Ready=False with a dedicated reason) with:
   • reason: ImmutableField (or similar)
   • message: point operators at remediation options, e.g.:
     • delete the StatefulSet with cascade orphan and let Helm recreate, or
     • pin / revert values so the rendered STS matches live immutable fields, or
     • use a chart/release process that intentionally recreates the STS
2. Optionally back off re-attempts (longer requeue) once the failure is classified as immutable/Forbidden rather than transient.
3. Optionally coordinate with notification-controller so identical Forbidden messages are not re-notified every interval (link to rate-limit / transition-only discussion on notification-controller).

Non-goals

• Not asking Flux to mutate StatefulSets in ways Kubernetes forbids.
• Not asking helm-controller to automatically delete STS objects without explicit operator action (orphan-delete is a documented manual remediation, not a silent default).

Environment

• Kubernetes: single-node (generic)
• Flux CLI: v2.9.4
• Distribution: flux-v2.9.1
• helm-controller: v1.6.2
• notification-controller: v1.9.2 (error events on each failed reconcile)
• Multiple HelmReleases otherwise Ready; failure class is STS immutable Forbidden during upgrade

Additional context

Today the operator experience is: “HelmRelease is red forever, chat pings on interval, and the message is a raw apiserver Forbidden with no Flux guidance.” Classifying immutable STS failures as Stalled + remediation hint would match how other controllers already treat non-retryable configuration dead-ends.

Code of Conduct

• I agree to follow this project's Code of Conduct