ssa: add DriftIgnoreRules to ApplyOptions for field-level ignore on apply

[dipti-pai]

Changes include

• Add []jsondiff.IgnoreRule to ssa ApplyOptions
• Change has no effect in create code paths. In Apply and ApplyAll code paths that correspond to update, remove ignored fields matching the ignored rules from the object.

Includes tests for various scenarios for optional and mandatory fields, claiming shared/forceful ownership and orphaning owned fields without a manager.

1. When drift ignore rules are specified for optional fields:

• flux manager drops ownership of the ignored fields.
• If another manager co-owned or claimed force ownership, the field values are set accordingly.
• If no field manager is present, the field is dropped and garbage collected by Kubernetes.

2. When drift ignore rules are specified for mandatory fields:

• If another manager co-owned or claimed ownership of the fields forcefully, Flux drops ownership of the field. Once Flux drops ownership, the other manager cannot drop ownership of the mandatory field, they get an error.
• If no other manager claims ownership of the fields, flux cannot drop ownership of the field and returns an error.

Associated kustomize controller draft PR testing the changes from controller perspective - fluxcd/kustomize-controller#1627

Fixes: #696

[matheuscscp]

The code is looking great to me, but I'd like to better understand how applying the ignore rules after dry-run works. I'm not saying it's not correct at all, just saying that it's probably correct and I need to catch up and understand the whole flow. It would be great if we could talk about this in the next dev meeting 🙏

[dipti-pai]

Thanks @matheuscscp for reviewing.

Adding a few notes below regarding dry run behavior with ssa and potential improvements in the current version of the code to make it easier to discuss during the dev meeting tomorrow.

Dry-run behavior with DriftIgnoreRules

dryRunApply always sends the complete desired object (including fields targeted by DriftIgnoreRules) to the API server with ForceOwnership. This is intentional - The API server validates the full payload — missing required fields like spec.selector would be rejected. Stripping ignored fields before dry-run would cause validation failures on every reconciliation.

The ignored fields are only stripped later — from the apply payload (not the dry-run), and only for existing objects (not creates).

Summarizing the full behavior for different scenarios and couple of improvements -

1. With this feature, there is no change in behavior during the initial create of a resource.

2. If no fields of the resource have drifted, hasDrifted returns false and returns UnchangedAction

3. If driftIgnoreRules are specified and there is drift in one of the ignored fields where the other manager takes force ownership OR co-owns the field, Flux gives up the ownership correctly.

4. If driftIgnoreRules are specified and there is drift in one of the ignored fields where there is no other manager, i.e for example if client-side apply was used to trigger the drift, Flux gives up the ownership dropping the field in apply. If the ignored field is a mandatory field, Kubernetes API server throws an error since when Flux gives up ownership, this is interpreted as a delete and mandatory field cannot be deleted. If the ignored field is not a mandatory field, the field is deleted OR default value is applied. This is by-design since the feature is intended to work only with fields managed via server-side apply. Our design : "Ignored field + drift = flux drops ownership".

5. If driftIgnoreRules are specified and there is drift only in ignored field, on every reconcile, dryRunApply detects a drift in hasDrifted, the apply operation applies the ignore rules and ConfiguredAction is returned. Though from ssa perspective, the subsequent apply is a no-op. A possible improvement here could be to have a hasDriftedWithIgnoredFields that does the drift detection stripping the ignored fields from both the dryRunObject and the object to be applied. After the driftIgnoreRules are applied the first time, on subsequent reconciliation, hasDriftedWithIgnoredFields would detect no drift and return UnchangedAction. This optimization saves an unnecessary round trip to Kubernetes API server to do no-op ssa apply.

6. Drift ignore rules are applied even if the drift ignored field has not changed -- In the current version, if a field not specified in the driftIgnoreRules drifts without any drift to fields specified in the driftIgnoreRules, the drift is corrected in apply. While correcting the drift, the driftIgnoreRules are applied as well. This can have unintended consequences when the user has specified a field in driftIgnoreRules but a field manager has not yet taken ownership of the field. A possible improvement could be if a drift ignored field has not drifted, do not apply the corresponding driftIgnoreRule.

My opinion is to take the improvements in 5. Improvement 6 is debatable - if a field is in driftIgnoreRule we should respect the user's intent and drop the ownership during apply is one way to think. Checking if the field has really drifted involves some more JSON pointer fetches and comparison of JSON pointers that complicate the logic. We could keep 6 as-is for now and iterate if users complain about it as well.

[stefanprodan]

I think we should strive to avoid applying objects if only the ignored fields have drifted, and always return UnchangedAction without making extra API calls.

[matheuscscp]

5. If driftIgnoreRules are specified and there is drift only in ignored field, on every reconcile, dryRunApply detects a drift in hasDrifted, the apply operation applies the ignore rules and ConfiguredAction is returned. Though from ssa perspective, the subsequent apply is a no-op. A possible improvement here could be to have a hasDriftedWithIgnoredFields that does the drift detection stripping the ignored fields from both the dryRunObject and the object to be applied. After the driftIgnoreRules are applied the first time, on subsequent reconciliation, hasDriftedWithIgnoredFields would detect no drift and return UnchangedAction. This optimization saves an unnecessary round trip to Kubernetes API server to do no-op ssa apply.

If I understand this correctly, this is what @stefanprodan commented about above. I agree, we should make this improvement and do not apply if only ignored fields have drifted.

6. Drift ignore rules are applied even if the drift ignored field has not changed -- In the current version, if a field not specified in the driftIgnoreRules drifts without any drift to fields specified in the driftIgnoreRules, the drift is corrected in apply. While correcting the drift, the driftIgnoreRules are applied as well. This can have unintended consequences when the user has specified a field in driftIgnoreRules but a field manager has not yet taken ownership of the field. A possible improvement could be if a drift ignored field has not drifted, do not apply the corresponding driftIgnoreRule.

This improvement also seems quite important.

[dipti-pai]

@stefanprodan @matheuscscp

With the latest iteration, the improvements 5. and 6. in this comment are implemented. Summarizing the new behavior

1. Drift detection now ignores specified fields

Before comparing the existing object with the dry-run result, hasDriftedWithIgnore() strips the ignored paths from both objects. If the only differences are in ignored fields, the object is treated as Unchanged — no apply happens. This prevents unnecessary apply when only externally-managed fields (e.g. HPA replicas, VPA resource limits) differ and incorrect Configured action from being returned when ignored rules drift.

2. Selective field stripping on apply

When an apply does happen (because non-ignored fields changed), we compute which ignored paths have actually drifted between live object and dry-run. Only those drifted ignored paths are removed from the apply payload via JSON patch. This means --

• Drifted ignored fields → stripped from payload → external controller's values are preserved
• Non-drifted ignored fields → kept in payload → Flux retains field ownership

Because of these changes, a new scenario arises which I want to call out - Flux does not update an ignored field even when its own source changes. For example,

.spec.replicas = 2 (from Flux's last apply)

existing.spec.replicas = 5 (from HPA's apply)
dryRunObject.spec.replicas = 4 (Flux's new desired from new source change)

With the current changes, if .spec.replicas is ignored, it means "this field is fully delegated to another controller" — Flux doesn't touch it or try to reclaim ownership regardless of source changes.

[matheuscscp]

With the current changes, if .spec.replicas is ignored, it means "this field is fully delegated to another controller" — Flux doesn't touch it or try to reclaim ownership regardless of source changes.

Small but important correction here (if I understood the current iteration correctly; please correct me if I'm wrong, @dipti-pai): when values match, Flux does reclaim ownership even if another manager already owns the field. Stripping only on real drift keeps things simple (no need to parse managedFields.fieldsV1 to detect ownership), at the cost of occasional ownership ping-pong when the external controller's value converges with Flux's desired. Worth calling out in the DriftIgnoreRules docs.

[dipti-pai]

With the current changes, if .spec.replicas is ignored, it means "this field is fully delegated to another controller" — Flux doesn't touch it or try to reclaim ownership regardless of source changes.

Small but important correction here (if I understood the current iteration correctly; please correct me if I'm wrong, @dipti-pai): when values match, Flux does reclaim ownership even if another manager already owns the field. Stripping only on real drift keeps things simple (no need to parse managedFields.fieldsV1 to detect ownership), at the cost of occasional ownership ping-pong when the external controller's value converges with Flux's desired. Worth calling out in the DriftIgnoreRules docs.

@matheuscscp, yes this is true because we only strip the fields when they drift. So, if they converge, Flux reclaims ownership. Note this happens only when Apply is triggered due to another non-ignored field drifting. If no other field has drifted, then on every reconcile, UnchangedAction is returned (as per the latest iteration code changes).

[matheuscscp]

With the current changes, if .spec.replicas is ignored, it means "this field is fully delegated to another controller" — Flux doesn't touch it or try to reclaim ownership regardless of source changes.

Small but important correction here (if I understood the current iteration correctly; please correct me if I'm wrong, @dipti-pai): when values match, Flux does reclaim ownership even if another manager already owns the field. Stripping only on real drift keeps things simple (no need to parse managedFields.fieldsV1 to detect ownership), at the cost of occasional ownership ping-pong when the external controller's value converges with Flux's desired. Worth calling out in the DriftIgnoreRules docs.

@matheuscscp, yes this is true because we only strip the fields when they drift. So, if they converge, Flux reclaims ownership. Note this happens only when Apply is triggered due to another non-ignored field drifting. If no other field has drifted, then on every reconcile, UnchangedAction is returned (as per the latest iteration code changes).

LGTM 👍

Are there any outstanding items left for us to address from your initial list?

[dipti-pai]

Are there any outstanding items left for us to address from your initial list?

The things we discussed last dev meeting are addressed in the latest iteration. Made the changes for the AI-generated review comments, they look solid and rebased on main. Thanks so much for reviewing!