🐛 e2e: only retry creating objects that failed

[nojnhuh]

What this PR does / why we need it:

The "get + update" sequence that CreateOrUpdate implements is too blunt when retries kick in. If a controller or webhook makes some monotonic change after an object was created by CreateOrUpdate, then blindly updating the object in a retry to match literally was initially created will fail.

e.g. Creating a template consisting of a Cluster with spec.topology will prompt the topology controller to add a controlPlaneRef and infrastructureRef. Retrying that creation prompts an update which tries to remove those fields which is invalid:

admission webhook "validation.cluster.cluster.x-k8s.io" denied the request: Cluster.cluster.x-k8s.io "capz-e2e-fxe4ui-cc" is invalid: [spec.infrastructureRef: Forbidden: cannot be removed, spec.controlPlaneRef: Forbidden: cannot be removed

This PR moves the retry loop for CreateOrUpdate from over all objects to each individual object. Objects which succeed will not be retried.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #13264

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelspeed for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nojnhuh]

/retitle 🐛 Use server-side apply for e2e CreateOrUpdate
/area e2e-testing

[nojnhuh]

/test ?

[k8s-ci-robot]

@nojnhuh: The following commands are available to trigger required jobs:

/test pull-cluster-api-build-main

/test pull-cluster-api-e2e-blocking-main

/test pull-cluster-api-e2e-conformance-ci-latest-main

/test pull-cluster-api-e2e-conformance-main

/test pull-cluster-api-e2e-latestk8s-main

/test pull-cluster-api-e2e-main

/test pull-cluster-api-e2e-main-gke

/test pull-cluster-api-e2e-mink8s-main

/test pull-cluster-api-e2e-upgrade-1-35-1-36-main

/test pull-cluster-api-test-main

/test pull-cluster-api-test-mink8s-main

/test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-apidiff-main

pull-cluster-api-build-main

pull-cluster-api-e2e-blocking-main

pull-cluster-api-test-main

pull-cluster-api-verify-main

Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nojnhuh]

/test pull-cluster-api-e2e-upgrade-1-35-1-36-main

[sbueringer]

/hold

Let's discuss on the issue

[nojnhuh]

I've updated this per our discussion in the issue:

/retitle 🐛 e2e: only retry creating objects that failed

[sbueringer]

/hold cancel
/test pull-cluster-api-e2e-main-gke

[sbueringer]

Not sure what happened there:
/test pull-cluster-api-e2e-main-gke

@nojnhuh PTAL at the linter findings