Do not requeue when finalizing managed service instance #3500
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A couple of flakes on the CI showed that a managed service might be deprovisioned with the broker twice: ``` https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-tests-periodic/builds/19402 https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-tests-periodic/builds/19396 https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-tests-periodic/builds/19358 ``` Analysis showed that when the controller [requeues the reconcile event on finalization](https://github.com/cloudfoundry/korifi/blob/2dfdb3848bc8690e3e5be14a535ae52c46152d46/controllers/controllers/services/instances/managed/controller.go#L265), the very same object (same generation, same resource version) could be reconciled twice thus request deprovision the instance from the broker twice. In order to address this, this change does not requeue the event after deprovision is requested from the broker. Instead, upon successful deprovision from the broker, the controller removes the finalizer, sets the `DeprovisionRequested` condition and returns an empty result (no requeue). The drawback of this approach is that we need to remove the finalizers on two places - whenever deprovision from the broker is requested, and when the status has the `DeprovisionRequested` condition. Not removing the finalizer after the `deprovisionRequested` check seems to make the issue much harder to reproduce, however still possible. I could speculate that the service instance CR and its status are different resources ("main resource" and subresource) and therefore could go out of sync for a while.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Is there a related GitHub Issue?
Flakes
What is this change about?
A couple of flakes on the CI showed that a managed service might be
deprovisioned with the broker twice:
Analysis showed that when the controller requeues the reconcile event
on
finalization,
the very same object (same generation, same resource version) could be
reconciled twice thus request deprovision the instance from the broker
twice.
In order to address this, this change does not requeue the event after
deprovision is requested from the broker. Instead, upon successful
deprovision from the broker, the controller removes the finalizer, sets
the
DeprovisionRequestedcondition and returns an empty result (norequeue).
The drawback of this approach is that we need to remove the finalizers
on two places - whenever deprovision from the broker is requested, and
when the status has the
DeprovisionRequestedcondition. Notremoving the finalizer after the
deprovisionRequestedcheck seems tomake the issue much harder to reproduce, however still possible. I could
speculate that the service instance CR and its status are different
resources ("main resource" and subresource) and therefore could go out
of sync for a while.
Tag your pair, your PM, and/or team
@georgethebeatle