fix: Prevent excessive reconciliation when timeout disabled #9202
Conversation
tekton-robot
added
release-note
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
tekton-robot
added
the
size/L
vdemeester
force-pushed
the
fix-8495-excessive-reconciliation
branch
from
c99df33 to
70e919d
Compare
|
/hold |
tekton-robot
added
the
do-not-merge/hold
twoGiants
left a comment
twoGiants
left a comment
There was a problem hiding this comment.
Great job for the fix 😸 👍. My brain got challenged understanding the timeout order... 🧠 ⚡. It looks good though! I would just keep the code a bit more consistent, will aid in future understanding.
See my comments below.
| if timeout == config.NoTimeoutDuration { | ||
| waitTime = time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) * time.Minute | ||
| // Check if we have a specific task timeout to wait for | ||
| if pr.Status.FinallyStartTime == nil && taskTimeout != nil && taskTimeout.Duration != config.NoTimeoutDuration { |
There was a problem hiding this comment.
The introduced checks here and the existing below are quite similar. I would keep the style consistent in both of them. The early returns is the way to go. Then this code is a bit easier to understand. I will add comments in each line where I would change up a bit.
| waitTime := pr.FinallyTimeout().Duration - c.Clock.Since(pr.Status.FinallyStartTime.Time) | ||
| return controller.NewRequeueAfter(waitTime) |
There was a problem hiding this comment.
Lets use the same variable naming as below. Is clearer.
| waitTime := pr.FinallyTimeout().Duration - c.Clock.Since(pr.Status.FinallyStartTime.Time) | |
| return controller.NewRequeueAfter(waitTime) | |
| finallyWaitTime := pr.FinallyTimeout().Duration - c.Clock.Since(pr.Status.FinallyStartTime.Time) | |
| return controller.NewRequeueAfter(finallyWaitTime) |
| if taskTimeout.Duration == config.NoTimeoutDuration { | ||
| waitTime = time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) * time.Minute | ||
| waitTime = timeout - elapsed | ||
| } |
There was a problem hiding this comment.
Add an early return here like above and break the if-else-if apart. Like you have above.
| } | |
| return controller.NewRequeueAfter(waitTime) | |
| } |
| waitTime = time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) * time.Minute | ||
| waitTime = timeout - elapsed | ||
| } | ||
| } else if pr.Status.FinallyStartTime != nil && pr.FinallyTimeout() != nil && |
There was a problem hiding this comment.
Break the else-if and move return controller.NewRequeueAfter(waitTime) into the if.
}
if pr.Status.FinallyStartTime != nil && pr.FinallyTimeout() != nil &&
pr.FinallyTimeout().Duration != config.NoTimeoutDuration {
finallyWaitTime := pr.FinallyTimeout().Duration - c.Clock.Since(pr.Status.FinallyStartTime.Time)
if finallyWaitTime < waitTime {
waitTime = finallyWaitTime
}
return controller.NewRequeueAfter(waitTime)
}
}
return nil
}Can't do a better suggestion on the hidden lines. This is how it should look like
| return controller.NewRequeueAfter(waitTime) | ||
| } | ||
| // Check if we have a finally timeout to wait for | ||
| if pr.Status.FinallyStartTime != nil && pr.FinallyTimeout() != nil && pr.FinallyTimeout().Duration != config.NoTimeoutDuration { |
There was a problem hiding this comment.
This check is identical to the check in 311-316. Don't you need this here too?:
if finallyWaitTime < waitTime {
waitTime = finallyWaitTime
}
- Stop immediate requeue loops when default-timeout-minutes is "0" - Remove redundant hasCondition checks (Knative already deduplicates) This adds an e2e tests that looks at pipeline logs to see how much reconciler loop there is. If you run it before the fix, it would count more than 1500 reconciler loop, whereas with the fix, only about 10. Signed-off-by: Vincent Demeester <[email protected]> Co-Authored-By: Claude <[email protected]>
vdemeester
force-pushed
the
fix-8495-excessive-reconciliation
branch
from
661ed4e to
9d13377
Compare
|
@twoGiants I changed a bit the if/else chains to make it.. less weird.. hopefully it's more readable. |
|
/hold cancel |
tekton-robot
removed
the
do-not-merge/hold
|
/retest |
3 participants
Changes
This adds an e2e tests that looks at pipeline logs to see how much reconciler loop there is. If you run it before the fix, it would count more than 1500 reconciler loop, whereas with the fix, only about 10.
/cc @afrittoli @pritidesai @tektoncd/core-maintainers
It took me a while to figure out, and I got some help from Claude (AI) to write the tests. The previous behavior seemed very weird as well, with no timeout, we would, re-queue instantly, which is.. madness 🙃
/kind bug
Fixes #8495
This could be a good candidate to be backported.
Submitter Checklist
As the author of this PR, please check off the items in this checklist:
/kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tepRelease Notes