Better support for non-deterministic external-names by updating critical annotations #850
Conversation
twobiers
force-pushed
the
update-metadata
branch
from
f4fb5c6 to
15df832
Compare
|
@twobiers, we are interested in progressing the solution you propose in this PR. Are you currently able to give this attention? If so, would you mind resolving the conflicts and pushing up an updated version? If you aren't currently able to give this attention would you be happy for us to take it over and get it over the line (you'll be credited with the contribution)? |
Signed-off-by: twobiers <[email protected]>
As the UpdateCriticialAnnotations function is now not exclusively called in the creation process, we have to ensure no other fields like the spec are updated, so we don't interfer with the normal LateInitialize logic Signed-off-by: twobiers <[email protected]>
Signed-off-by: twobiers <[email protected]>
Signed-off-by: twobiers <[email protected]>
twobiers
force-pushed
the
update-metadata
branch
from
15df832 to
c14e6da
Compare
|
@jeanduplessis Thanks for your kind comment. I'm able to work on this and have rebased the branch onto master. |
|
Thanks, @twobiers. @erhancagirici will be doing a thorough review in the next few days, with a specific focus on making sure it's compatible with Upjet's async approach. |
|
Closing and reopening to kick the CI |
Signed-off-by: twobiers <[email protected]>
Signed-off-by: twobiers <[email protected]>
erhancagirici
left a comment
erhancagirici
left a comment
There was a problem hiding this comment.
Thanks for the PR! I've left some comments regarding the mechanics, otherwise LGTM conceptually. As you mentioned in the doc comments, this will be sort of a band-aid until a more async-aware implementation.
| // after the creation, but later as part of an asynchronous process. | ||
| // When Crossplane supports asynchronous creation of resources natively, this logic | ||
| // might not be needed anymore and can be revisited. | ||
| if err := r.managed.UpdateCriticalAnnotations(ctx, managed); err != nil { |
There was a problem hiding this comment.
a thing to consider (possible nit):
most reconciliation loops will enter this codepath regardless of a need to update critical annotations. I wonder if this one is no-op in terms of k8s API access or brings some extra load on the apiserver.
Especially in async creations with long-running creation times, currently (with no native async support) the external clients return observation.ResourceExists as true to avoid further actions for the observations during creation of the resource.
There was a problem hiding this comment.
Perhaps we should make writing these annotations optional. I'd suggest making them opt-out since that's the safest path. The option would be specified by the provider author - so they can disable these annotations if they know for sure they're not needed (i.e. naming is deterministic, API is strongly consistent). I remember discussing this with someone recently, but can't find a tracking issue.
(If we do make them optional, I think we could do it pretty easily by injecting a no-op implementation of the annotation updater.)
| return err | ||
| } | ||
|
|
||
| err = u.client.Patch(ctx, o, client.RawPatch(types.MergePatchType, patchData), client.FieldOwner(fieldOwnerAPISimpleRefResolver), client.ForceOwnership) |
There was a problem hiding this comment.
Looks like we patch all annotations here.
Just wondering how does this interact with the field ownerships of annotations that XP does not manage (like a custom annotation set by a user or some other controllers, tooling etc), and whether it is a thing to worry about regarding server-side apply etc. Would we cause a race regarding field ownerships?
| err := retry.OnError(retry.DefaultRetry, func(err error) bool { | ||
| return !errors.Is(err, context.Canceled) | ||
| }, func() error { | ||
| err := u.client.Update(ctx, o) |
There was a problem hiding this comment.
I like this update operation is being replaced by SSA.
| return err | ||
| } | ||
|
|
||
| err = u.client.Patch(ctx, o, client.RawPatch(types.MergePatchType, patchData), client.FieldOwner(fieldOwnerAPISimpleRefResolver), client.ForceOwnership) |
There was a problem hiding this comment.
We had better use a different manager name than managed.crossplane.io/api-simple-reference-resolver as the annotations have nothing to do with the API resolver. Maybe something like: managed.crossplane.io/critical-annotation-updater?
There was a problem hiding this comment.
Would it make sense for the MR reconciler to just use one manager name, regardless of operation?
Possibly too late if we're already using api-simple-reference-resolver for some.
| err := u.client.Update(ctx, o) | ||
| patchMap := map[string]interface{}{ | ||
| "metadata": map[string]any{ | ||
| "annotations": a, |
There was a problem hiding this comment.
Should we be more selective in what's being patched here, i.e., not all the annotations on the MR are the critical ones and we would only like to manage the critical annotations by this manager? This will probably not be an issue as this manager will not have an opinion on "non-critical" annotations and their respective managers will dictate their values. But one potential issue is when the other manager (who should really be owning the annotation) actually wants to delete the annotation. The managed.crossplane.io/critical-annotation-updater will still be owning the annotation. So the critical-annotation-updater had better not own non-critical ones...
| if observation.ResourceExists { | ||
| // When a resource exists or is just created, it might have received | ||
| // a non-deterministic external name after its creation, which we need to persist. | ||
| // We do this by updating the critical annotations. | ||
| // This is needed because some resources might not receive an external-name directly | ||
| // after the creation, but later as part of an asynchronous process. | ||
| // When Crossplane supports asynchronous creation of resources natively, this logic | ||
| // might not be needed anymore and can be revisited. | ||
| if err := r.managed.UpdateCriticalAnnotations(ctx, managed); err != nil { | ||
| log.Debug(errUpdateManagedAnnotations, "error", err) | ||
| record.Event(managed, event.Warning(reasonCannotUpdateManaged, errors.Wrap(err, errUpdateManagedAnnotations))) | ||
| return reconcile.Result{Requeue: true}, errors.Wrap(r.client.Status().Update(ctx, managed), errUpdateManagedAnnotations) | ||
| } | ||
| } |
There was a problem hiding this comment.
As @lsviben explains here, upjet should not be relying on setting ResourceLateInitialized to get the critical annotations updated in the first place. The async mode implemented by upjet breaks the assumptions of the managed reconciler. I believe we need to first address this discrepancy between upjet and the managed reconciler...
7 participants
Description of your changes
This PR ensures that critical annotations are stored when a resource is created. Some resources have non-deterministic external-names, which are never updated when the
mangementPoliciesdon't containLateInitialize.So far, there is an implicit contract that an Observation updating the external-name will be eventually stored as part of the LateInitialize process. However, that should only affect updates to the
specand not the annotations like external-name.More Context can be found here: crossplane/crossplane#5918
I'm not sure how a unit test can be expressed, as I'm not really familiar with the setup. If the change is approved, I think it would make sense to backport it aswell.
Fixes:
Maybe fixes aswell:
I have:
earthly +reviewableto ensure this PR is ready for review.Added or updated unit tests.Linked a PR or a docs tracking issue to document this change.backport release-x.ylabels to auto-backport this PR.Need help with this checklist? See the cheat sheet.