Add opt-in retry-with-backoff for rejected pushes - #1090
Open
monotek wants to merge 2 commits into
Open
Conversation
When many independent ImageUpdateAutomation objects push to the same branch of one GitOps repo, a rejected push (another writer already advanced the branch) previously just failed the reconciliation, with the next attempt only happening on the next scheduled reconcile or via controller-runtime's exponential-backoff requeue. Under heavy write contention this turns a sub-minute git operation into a delay of tens of minutes. Add commitAndPushWithRetry, which on a push rejected specifically due to a conflict (source.IsPushConflict) fetches and hard-resets onto the new remote tip (SourceManager.RefreshToRemote, no full re-clone), re-applies policies against the refreshed tree, and retries the commit and push, up to 5 attempts with exponential backoff (2s/4s/8s/16s). Any other error, or exhaustion of all attempts, is returned unchanged. The retry loop is bounded by min(.spec.interval / 2, 2 minutes) so contention on one branch cannot block a reconcile worker indefinitely. Gated behind the new GitPushRetryOnConflict feature gate, disabled by default. Depends on FetchAndReset and ErrPushRejected from a companion fluxcd/pkg change; go.mod currently points at that change's fork branch pending a tagged release. Assisted-by: Claude Sonnet 5/claude-sonnet-5 Signed-off-by: André Bauer <monotek23@gmail.com>
Signed-off-by: André Bauer <monotek@users.noreply.github.com>
monotek
marked this pull request as ready for review
September 1, 2026 16:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When many independent
ImageUpdateAutomationobjects push to the same branch of one GitOps repo (e.g. one IUA per service/region, all on independent 5-minute reconcile timers), a multi-service release can cause many of them to try pushing around the same time. A concrete example observed in production: five regions all resolved a new image tag within 1-2 minutes of each other, but the resulting git commits landed 13-29 minutes apart, because several IUAs lost a non-fast-forward push race against others pushing to the same branch:There is no in-process retry today — the next attempt only happens on the next scheduled reconcile, or via controller-runtime's per-item exponential-backoff requeue (750ms doubling, capped at 15min). Under heavy contention, a single IUA can lose several races in a row, each costing a full backoff cycle, which is how a sub-minute git operation turns into a 29-minute delay. This reproduces on every release with enough concurrent writers to one branch, and gets worse as more services onboard per-instance IUAs.
A retry-with-backoff loop that reuses the existing local clone (fetch + reset instead of a full re-clone) resolves a lost race in seconds instead of minutes, following the same shape as a standard
retry_with_backoff(5, 2, ...)wrapper around agit pull --rebase && git pushcycle.What
commitAndPushWithRetry(newinternal/controller/push_retry.go) wrapsCommitAndPush: on a push rejected specifically because another writer already advanced the branch (source.IsPushConflict), it waits (2s/4s/8s/16s backoff, 5 attempts total), fetches and hard-resets onto the new remote tip (SourceManager.RefreshToRemote, no full re-clone), re-applies policies against the refreshed tree, and retries the commit and push. Any other error, or exhaustion of all attempts, is returned unchanged — existing condition/error handling inreconcile()is untouched.min(.spec.interval / 2, 2 minutes)so contention on one branch can't block a reconcile worker indefinitely and starve otherImageUpdateAutomationobjects sharing the pool.(*gogit.Client).FetchAndResetandgit.ErrPushRejectedfrom git/gogit: add FetchAndReset and ErrPushRejected for push-conflict recovery pkg#1289 (companion PR) — go-git has no rebase API, so this fetches + hard-resets instead of a real rebase, which reaches the same end state here since policies are re-applied fresh against the reset tree rather than blindly replaying a stale diff.GitPushRetryOnConflict, disabled by default — opt-in, so this ships as an explicit choice rather than a behavior change for existing installs.Testing
internal/source:TestSourceManager_RefreshToRemote_RecoversFromPushConflict— forces a real non-fast-forward rejection via a competing out-of-band push, then proves recovery lands the new commit on top of the competing one (parent-hash check), not an overwrite.internal/controller: five tests covering recovery within N attempts, exhausting all attempts against a real conflict, a non-conflict error (connection failure) returning after exactly one attempt, retry-budget enforcement under a short.spec.interval, and the feature gate's off-by-default fallback path.go test ./...passes (envtest viamake install-envtest),go vet/gofmtclean.Notes for reviewers
go.modcurrently pointsgithub.com/fluxcd/pkg/gitat git/gogit: add FetchAndReset and ErrPushRejected for push-conflict recovery pkg#1289's fork branch (via a resolved pseudo-version, not a local path) so this branch builds standalone. Will be switched to a tagged release version once git/gogit: add FetchAndReset and ErrPushRejected for push-conflict recovery pkg#1289 merges and releases, before this is marked ready for review.