Skip to content

Support for resource patching with retries#877

Merged
Felix-Stakater merged 2 commits into
stakater:masterfrom
videlov:master
May 12, 2025
Merged

Support for resource patching with retries#877
Felix-Stakater merged 2 commits into
stakater:masterfrom
videlov:master

Conversation

@videlov
Copy link
Copy Markdown
Contributor

@videlov videlov commented Mar 27, 2025

For our scenario we need to scale Reloader to multiple Pods and the current HA implementation causes problems while we need zero downtime and leader election can not provide that with the current implementation.

Solution for us to increase resilience is to include patch on resources which will reduce conflicts possibility when multiple clients are trying. Also it solves the case for multiple restarts due to patching based on CM or Secret hash value, which will be same for all clients patch requests and one restart should be triggered.

For non-patch possible workloads, we introduce retries on conflicts, which will help to avoid missed updates.

PR introduces:

  • Introduce retry.RetryOnConflict in PerformAction for the upgrade handler
  • Add support for patching resources in rolling upgrade callbacks
  • Unit tests and manual testing to confirm our use case is solved

We would appreciate your comments and review. Thanks in advance.

@Nuckal777
Copy link
Copy Markdown

Ref #766.

@Felix-Stakater
Copy link
Copy Markdown
Contributor

Hi! We looked through this and are satisfied so far, nice job!
Would you mind rebasing your branch on our latest master? (requirement for the pipelines to run)

You can ping me here or in slack and I'll make sure to run them as soon as i see it. :)

* Initial

* State

* State

* Revert unintentional change

* Update

* Update

* Fix tests

* Support patching for both reload strategies

* rolling_upgrade tests

* Update

* More tests

* Remove unnecessary stuff
@videlov
Copy link
Copy Markdown
Contributor Author

videlov commented May 7, 2025

@Felix-Stakater thank you for looking into it. I did rebase to current master, and fixed lint issues.

@Felix-Stakater Felix-Stakater merged commit 5499911 into stakater:master May 12, 2025
5 checks passed
@Felix-Stakater
Copy link
Copy Markdown
Contributor

Thank you! :)

@msafwankarim msafwankarim mentioned this pull request Jul 2, 2025
pratikbin pushed a commit to NodeOps-app/Reloader that referenced this pull request Nov 13, 2025
Support for resource patching with retries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants