Add support for helm --force-conflicts to address field ownership conflicts

doc/helmchart.md

@@ -18,7 +18,7 @@ _Underlying type:_ _string_
 
 
 _Validation:_
-- Enum: [abort reinstall]
+- Enum: [abort reinstall retry]
 
 _Appears in:_
 - [HelmChartConfigSpec](#helmchartconfigspec)
@@ -118,7 +118,8 @@ _Appears in:_
 | `values` _[JSON](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#json-v1-apiextensions-k8s-io)_ | Override complex Chart values via structured YAML. Takes precedence over options set via valuesContent.<br />Helm CLI positional argument/flag: `--values` |  |  |
 | `valuesContent` _string_ | Override complex Chart values via inline YAML content.<br />Helm CLI positional argument/flag: `--values` |  |  |
 | `valuesSecrets` _[SecretSpec](#secretspec) array_ | Override complex Chart values via references to external Secrets.<br />Helm CLI positional argument/flag: `--values` |  |  |
-| `failurePolicy` _[FailurePolicy](#failurepolicy)_ | Configures handling of failed chart installation or upgrades.<br />- `reinstall` will perform a clean uninstall and reinstall of the chart.<br />- `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error. | reinstall | Enum: [abort reinstall] <br /> |
+| `failurePolicy` _[FailurePolicy](#failurepolicy)_ | Configures handling of failed chart installation or upgrades.<br />- `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error.<br />- `reinstall` will perform a clean uninstall and reinstall of the chart; this is the default behavior.<br />- `retry` will attempt to retry the install or upgrade whenever chart configuration changes. | reinstall | Enum: [abort reinstall retry] <br /> |
+| `forceConflicts` _boolean_ | Set to True if helm should configure server-side apply to force changes when conflicts arise in ownership of managed fields.<br />Helm CLI positional argument/flag: `--force-conflicts` |  |  |
 
 
 
@@ -150,11 +151,12 @@ _Appears in:_
 | `helmVersion` _string_ | DEPRECATED. Helm version to use. Only v3 is currently supported. |  |  |
 | `bootstrap` _boolean_ | Set to True if this chart is needed to bootstrap the cluster (Cloud Controller Manager, CNI, etc). |  |  |
 | `takeOwnership` _boolean_ | Set to True if helm should take ownership of existing resources when installing/upgrading the chart.<br />Helm CLI positional argument/flag: `--take-ownership` |  |  |
+| `forceConflicts` _boolean_ | Set to True if helm should configure server-side apply to force changes when conflicts arise in ownership of managed fields.<br />Helm CLI positional argument/flag: `--force-conflicts` |  |  |
 | `chartContent` _string_ | Base64-encoded chart archive .tgz; overides `.spec.chart` and `.spec.version`.<br />Helm CLI positional argument/flag: `CHART` |  |  |
 | `jobImage` _string_ | Specify the image to use for tht helm job pod when installing or upgrading the helm chart. |  |  |
 | `backOffLimit` _integer_ | Specify the number of retries before considering the helm job failed. |  |  |
 | `timeout` _[Duration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#duration-v1-meta)_ | Timeout for Helm operations.<br />Helm CLI positional argument/flag: `--timeout` |  |  |
-| `failurePolicy` _[FailurePolicy](#failurepolicy)_ | Configures handling of failed chart installation or upgrades.<br />- `reinstall` will perform a clean uninstall and reinstall of the chart.<br />- `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error. | reinstall | Enum: [abort reinstall] <br /> |
+| `failurePolicy` _[FailurePolicy](#failurepolicy)_ | Configures handling of failed chart installation or upgrades.<br />- `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error.<br />- `reinstall` will perform a clean uninstall and reinstall of the chart; this is the default behavior.<br />- `retry` will attempt to retry the install or upgrade whenever chart configuration changes. | reinstall | Enum: [abort reinstall retry] <br /> |
 | `authSecret` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#localobjectreference-v1-core)_ | Reference to Secret of type kubernetes.io/basic-auth holding Basic auth credentials for the Chart repo. |  |  |
 | `authPassCredentials` _boolean_ | Pass Basic auth credentials to all domains.<br />Helm CLI positional argument/flag: `--pass-credentials` |  |  |
 | `insecureSkipTLSVerify` _boolean_ | Skip TLS certificate checks for the chart download.<br />Helm CLI positional argument/flag: `--insecure-skip-tls-verify` |  |  |

pkg/apis/helm.cattle.io/v1/types.go

@@ -10,7 +10,7 @@ import (
 	"k8s.io/apimachinery/pkg/util/intstr"
 )
 
-// +kubebuilder:validation:Enum={"abort","reinstall"}
+// +kubebuilder:validation:Enum={"abort","reinstall","retry"}
 type FailurePolicy string
 
 // +kubebuilder:validation:Enum={"secret","configmap"}
@@ -79,6 +79,9 @@ type HelmChartSpec struct {
 	// Set to True if helm should take ownership of existing resources when installing/upgrading the chart.
 	// Helm CLI positional argument/flag: `--take-ownership`
 	TakeOwnership bool `json:"takeOwnership,omitempty"`
+	// Set to True if helm should configure server-side apply to force changes when conflicts arise in ownership of managed fields.
+	// Helm CLI positional argument/flag: `--force-conflicts`
+	ForceConflicts bool `json:"forceConflicts,omitempty"`
 	// Base64-encoded chart archive .tgz; overides `.spec.chart` and `.spec.version`.
 	// Helm CLI positional argument/flag: `CHART`
 	ChartContent string `json:"chartContent,omitempty"`
@@ -90,8 +93,9 @@ type HelmChartSpec struct {
 	// Helm CLI positional argument/flag: `--timeout`
 	Timeout *metav1.Duration `json:"timeout,omitempty"`
 	// Configures handling of failed chart installation or upgrades.
-	// - `reinstall` will perform a clean uninstall and reinstall of the chart.
 	// - `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error.
+	// - `reinstall` will perform a clean uninstall and reinstall of the chart; this is the default behavior.
+	// - `retry` will attempt to retry the install or upgrade whenever chart configuration changes.
 	// +kubebuilder:default=reinstall
 	FailurePolicy FailurePolicy `json:"failurePolicy,omitempty"`
 	// Reference to Secret of type kubernetes.io/basic-auth holding Basic auth credentials for the Chart repo.
@@ -162,10 +166,14 @@ type HelmChartConfigSpec struct {
 	// Helm CLI positional argument/flag: `--values`
 	ValuesSecrets []SecretSpec `json:"valuesSecrets,omitempty"`
 	// Configures handling of failed chart installation or upgrades.
-	// - `reinstall` will perform a clean uninstall and reinstall of the chart.
 	// - `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error.
+	// - `reinstall` will perform a clean uninstall and reinstall of the chart; this is the default behavior.
+	// - `retry` will attempt to retry the install or upgrade whenever chart configuration changes.
 	// +kubebuilder:default=reinstall
 	FailurePolicy FailurePolicy `json:"failurePolicy,omitempty"`
+	// Set to True if helm should configure server-side apply to force changes when conflicts arise in ownership of managed fields.
+	// Helm CLI positional argument/flag: `--force-conflicts`
+	ForceConflicts *bool `json:"forceConflicts,omitempty"`
 }
 
 type HelmChartConditionType string

pkg/controllers/chart/chart.go

@@ -63,8 +63,9 @@ const (
 	LabelControlPlaneSuffix = "control-plane"
 	LabelEtcdSuffix         = "etcd"
 
-	FailurePolicyReinstall = "reinstall"
 	FailurePolicyAbort     = "abort"
+	FailurePolicyReinstall = "reinstall"
+	FailurePolicyRetry     = "retry"
 
 	chartBySecretIndex       = "helmcharts.helm.cattle.io/chart-by-secret"
 	chartConfigBySecretIndex = "helmcharts.helm.cattle.io/chartconfig-by-secret"
@@ -339,6 +340,28 @@ func (c *Controller) OnChange(chart *v1.HelmChart, chartStatus v1.HelmChartStatu
 		return nil, chartStatus, nil
 	}
 
+	if c.jobFailed(chart) {
+		c.recorder.Eventf(chart, corev1.EventTypeWarning, "JobFailed", "Job has reached configured number of retries without succeeding")
+		chartCopy := chart.DeepCopy()
+		chartCopy.Status.Conditions = []v1.HelmChartCondition{
+			{
+				Type:    v1.HelmChartJobCreated,
+				Status:  corev1.ConditionTrue,
+				Reason:  "Job created",
+				Message: fmt.Sprintf("Applying HelmChart using Job %s/%s", chart.Namespace, jobName(chart)),
+			},
+			{
+				Type:    v1.HelmChartFailed,
+				Status:  corev1.ConditionTrue,
+				Reason:  "Job failed",
+				Message: "Job has reached configured number of retries without succeeding",
+			},
+		}
+		if _, err := c.helms.UpdateStatus(chartCopy); err != nil {
+			return nil, chartStatus, fmt.Errorf("unable to update status of helm chart to set failed condition: %w", err)
+		}
+	}
+
 	// Jobs are created as suspended. Once the job controller syncs it and adds the
 	// Suspended condition, we know that is has been observed by the job controller,
 	// and we can safely manage it without triggering race conditions. If the job is
@@ -378,7 +401,7 @@ func (c *Controller) OnChange(chart *v1.HelmChart, chartStatus v1.HelmChartStatu
 			Type:    v1.HelmChartJobCreated,
 			Status:  corev1.ConditionTrue,
 			Reason:  "Job created",
-			Message: fmt.Sprintf("Applying HelmChart using Job %s/%s", job.Namespace, job.Name),
+			Message: fmt.Sprintf("Applying HelmChart using Job %s/%s", chart.Namespace, jobName(chart)),
 		},
 		{
 			Type:   v1.HelmChartFailed,
@@ -458,7 +481,7 @@ func (c *Controller) OnRemove(key string, chart *v1.HelmChart) (*v1.HelmChart, e
 		chartCopy.Status.JobName = job.Name
 		chart, err = c.helms.UpdateStatus(chartCopy)
 		if err != nil {
-			return chart, fmt.Errorf("unable to update status of helm chart to add uninstall job name %s", chartCopy.Status.JobName)
+			return chart, fmt.Errorf("unable to update status of helm chart to add uninstall job name %s: %w", chartCopy.Status.JobName, err)
 		}
 	}
 
@@ -531,6 +554,9 @@ func (c *Controller) getJobAndRelatedResources(chart *v1.HelmChart) (*batch.Job,
 		failurePolicy = fp
 	}
 
+	// set default for SSA force-conflicts
+	forceConflicts := chart.Spec.ForceConflicts
+
 	// override default backOffLimit if specified
 	backOffLimit := defaultBackOffLimit
 	if chart.Spec.BackOffLimit != nil {
@@ -568,6 +594,11 @@ func (c *Controller) getJobAndRelatedResources(chart *v1.HelmChart) (*batch.Job,
 				failurePolicy = fp
 			}
 
+			// Override the force-conflict setting to what is provided in the HelmChartConfig
+			if config.Spec.ForceConflicts != nil {
+				forceConflicts = *config.Spec.ForceConflicts
+			}
+
 			// make sure that changes to HelmChart ValuesSecrets triger change to hash
 			for _, secret := range config.Spec.ValuesSecrets {
 				if !secret.IgnoreUpdates && secret.Name != "chart-values-"+config.Name {
@@ -583,6 +614,7 @@ func (c *Controller) getJobAndRelatedResources(chart *v1.HelmChart) (*batch.Job,
 	// note: the purpose of the additional annotation is to cause the job to be destroyed
 	// and recreated if the hash of the HelmChartConfig changes while it is being processed
 	setFailurePolicy(job, failurePolicy)
+	setForceConflicts(job, forceConflicts)
 	setBackOffLimit(job, backOffLimit)
 	hashObjects(job, objects...)
 
@@ -698,6 +730,19 @@ func (c *Controller) jobComplete(chart *v1.HelmChart) bool {
 	return false
 }
 
+// jobFailed returns true if the job controller has added a True Failed
+// condition to the job for the given chart.
+func (c *Controller) jobFailed(chart *v1.HelmChart) bool {
+	if job, _ := c.jobs.Cache().Get(chart.Namespace, jobName(chart)); job != nil {
+		for _, condition := range job.Status.Conditions {
+			if condition.Type == batch.JobFailed {
+				return condition.Status == corev1.ConditionTrue
+			}
+		}
+	}
+	return false
+}
+
 // jobReady returns true if the job is suspended, has never been started, and
 // the Job controller has added a True Suspended condition to the job for the
 // given chart. The addition of this condition indicates that the controller
@@ -1404,8 +1449,17 @@ func setFailurePolicy(job *batch.Job, failurePolicy string) {
 	})
 }
 
+func setForceConflicts(job *batch.Job, forceConflicts bool) {
+	if forceConflicts {
+		job.Spec.Template.Spec.Containers[0].Args = slices.Insert(job.Spec.Template.Spec.Containers[0].Args, 1, "--force-conflicts")
+	}
+}
+
 func hashObjects(job *batch.Job, objs ...metav1.Object) {
 	hash := sha256.New()
+	if backoffLimit := job.Spec.BackoffLimit; backoffLimit != nil {
+		hash.Write(fmt.Append(nil, *backoffLimit))
+	}
 	if obj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&job.Spec.Template); err == nil {
 		ust := unstructured.Unstructured{Object: obj}
 		if b, err := ust.MarshalJSON(); err == nil {

pkg/crds/yaml/generated/helm.cattle.io_helmchartconfigs.yaml

@@ -50,12 +50,19 @@ spec:
                 default: reinstall
                 description: |-
                   Configures handling of failed chart installation or upgrades.
-                  - `reinstall` will perform a clean uninstall and reinstall of the chart.
                   - `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error.
+                  - `reinstall` will perform a clean uninstall and reinstall of the chart; this is the default behavior.
+                  - `retry` will attempt to retry the install or upgrade whenever chart configuration changes.
                 enum:
                 - abort
                 - reinstall
+                - retry
                 type: string
+              forceConflicts:
+                description: |-
+                  Set to True if helm should configure server-side apply to force changes when conflicts arise in ownership of managed fields.
+                  Helm CLI positional argument/flag: `--force-conflicts`
+                type: boolean
               values:
                 description: |-
                   Override complex Chart values via structured YAML. Takes precedence over options set via valuesContent.

pkg/crds/yaml/generated/helm.cattle.io_helmcharts.yaml

@@ -146,12 +146,19 @@ spec:
                 default: reinstall
                 description: |-
                   Configures handling of failed chart installation or upgrades.
-                  - `reinstall` will perform a clean uninstall and reinstall of the chart.
                   - `abort` will take no action and leave the chart in a failed state so that the administrator can manually resolve the error.
+                  - `reinstall` will perform a clean uninstall and reinstall of the chart; this is the default behavior.
+                  - `retry` will attempt to retry the install or upgrade whenever chart configuration changes.
                 enum:
                 - abort
                 - reinstall
+                - retry
                 type: string
+              forceConflicts:
+                description: |-
+                  Set to True if helm should configure server-side apply to force changes when conflicts arise in ownership of managed fields.
+                  Helm CLI positional argument/flag: `--force-conflicts`
+                type: boolean
               helmVersion:
                 description: DEPRECATED. Helm version to use. Only v3 is currently
                   supported.