feat(client): make Kubernetes API client QPS and burst configurable#342
Open
oneubauer wants to merge 1 commit into
Open
feat(client): make Kubernetes API client QPS and burst configurable#342oneubauer wants to merge 1 commit into
oneubauer wants to merge 1 commit into
Conversation
b5accc3 to
0887022
Compare
Follow-up to openebs#321. The provisioner's workers share a single clientset whose rate limiter was pinned to the client-go defaults (QPS 5 / burst 10), so raising OPENEBS_IO_WORKER_THREADS alone left workers contending for the same client-side throttle. Add OPENEBS_IO_CLIENT_QPS and OPENEBS_IO_CLIENT_BURST (and matching localpv.controller.clientQPS / clientBurst Helm values) to override the client's rate limiter. Both are opt-in: when unset, client-go's defaults are left untouched and rendered manifests are unchanged. Signed-off-by: Oliver Neubauer <o.neubauer@gmail.com>
0887022 to
48c2e9e
Compare
niladrih
reviewed
Jun 18, 2026
Comment on lines
+188
to
+198
| func getClientQPS() (float32, bool) { | ||
| val, present := menv.Lookup(ProvisionerClientQPS) | ||
| if !present || val == "" { | ||
| return 0, false | ||
| } | ||
| qps, err := strconv.ParseFloat(val, 32) | ||
| if err != nil || qps <= 0 { | ||
| return 0, false | ||
| } | ||
| return float32(qps), true | ||
| } |
Member
There was a problem hiding this comment.
nit:
Suggested change
| func getClientQPS() (float32, bool) { | |
| val, present := menv.Lookup(ProvisionerClientQPS) | |
| if !present || val == "" { | |
| return 0, false | |
| } | |
| qps, err := strconv.ParseFloat(val, 32) | |
| if err != nil || qps <= 0 { | |
| return 0, false | |
| } | |
| return float32(qps), true | |
| } | |
| func getClientQPS() (float32, bool) { | |
| val, present := k8sEnv.GetFloat64(ProvisionerClientQPS, 0.0) | |
| if !present || val <= 0.0 { | |
| return 0.0, false | |
| } | |
| return float32(val), true | |
| } |
Comment on lines
+205
to
+213
| val, present := menv.Lookup(ProvisionerClientBurst) | ||
| if !present || val == "" { | ||
| return 0, false | ||
| } | ||
| burst, err := strconv.Atoi(val) | ||
| if err != nil || burst <= 0 { | ||
| return 0, false | ||
| } | ||
| return burst, true |
Member
There was a problem hiding this comment.
nit:
Suggested change
| val, present := menv.Lookup(ProvisionerClientBurst) | |
| if !present || val == "" { | |
| return 0, false | |
| } | |
| burst, err := strconv.Atoi(val) | |
| if err != nil || burst <= 0 { | |
| return 0, false | |
| } | |
| return burst, true | |
| val, present := k8sEnv.GetInt(ProvisionerClientBurst, 0) | |
| if !present || val <= 0 { | |
| return 0, false | |
| } | |
| return val, true |
| // Apply client-side API rate-limit overrides only when explicitly set, | ||
| // so an unconfigured provisioner keeps the client-go defaults (QPS 5, | ||
| // Burst 10). | ||
| var clientOpts []mKube.OptionFn |
Member
There was a problem hiding this comment.
nit:
Suggested change
| var clientOpts []mKube.OptionFn | |
| clientOpts := make([]mKube.OptionFn, 0, 2) |
Comment on lines
+253
to
+266
| if qps > 0 { | ||
| c.qps = qps | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // WithBurst sets the maximum burst of queries above the QPS limit the client | ||
| // may send to the Kubernetes API server. A non-positive value is ignored, | ||
| // leaving the client-go default in place. | ||
| func WithBurst(burst int) OptionFn { | ||
| return func(c *Client) { | ||
| if burst > 0 { | ||
| c.burst = burst | ||
| } |
Member
There was a problem hiding this comment.
Not required, as we're already doing this in applyRateLimits
Suggested change
| if qps > 0 { | |
| c.qps = qps | |
| } | |
| } | |
| } | |
| // WithBurst sets the maximum burst of queries above the QPS limit the client | |
| // may send to the Kubernetes API server. A non-positive value is ignored, | |
| // leaving the client-go default in place. | |
| func WithBurst(burst int) OptionFn { | |
| return func(c *Client) { | |
| if burst > 0 { | |
| c.burst = burst | |
| } | |
| c.qps = qps | |
| } | |
| } | |
| // WithBurst sets the maximum burst of queries above the QPS limit the client | |
| // may send to the Kubernetes API server. A non-positive value is ignored, | |
| // leaving the client-go default in place. | |
| func WithBurst(burst int) OptionFn { | |
| return func(c *Client) { | |
| c.burst = burst |
niladrih
reviewed
Jun 18, 2026
Comment on lines
+52
to
+55
| clientQPS: "" | ||
| # Maximum burst of queries above clientQPS the provisioner allows to the Kubernetes API | ||
| # server. Leave empty to use the client-go default (10). | ||
| clientBurst: "" |
Member
There was a problem hiding this comment.
Suggested change
| clientQPS: "" | |
| # Maximum burst of queries above clientQPS the provisioner allows to the Kubernetes API | |
| # server. Leave empty to use the client-go default (10). | |
| clientBurst: "" | |
| client: | |
| qps: "" | |
| # Maximum burst of queries above clientQPS the provisioner allows to the Kubernetes API | |
| # server. Leave empty to use the client-go default (10). | |
| burst: "" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this PR required? What issue does it fix?:
Follow-up to #321 (configurable worker threads). #321 made the provisioner's
worker count tunable via
OPENEBS_IO_WORKER_THREADS, but all workers share asingle Kubernetes clientset, and that clientset's rate limiter is still pinned
to the client-go defaults (QPS 5 / burst 10). Raising the worker count alone
therefore just creates more goroutines that contend for the same client-side
throttle — under bursty PVC create/delete load the provisioner self-throttles
before the API server's own API Priority & Fairness ever comes into play.
This PR exposes the client's QPS and burst so operators running at scale can
raise them alongside
workers.What this PR does?:
Adds two new, optional environment variables (and matching Helm values) to tune
the rate limiter on the shared Kubernetes API client:
OPENEBS_IO_CLIENT_QPS: maximum sustained queries/sec to the API server.OPENEBS_IO_CLIENT_BURST: maximum burst above QPS.Wiring:
cmd/provisioner-localpv/app/env.go: env-var names +getClientQPS()/getClientBurst()getters (functionalmenvlookup, same pattern as theexisting worker-thread/timeout getters).
pkg/kubernetes/client/client.go:WithQPS()/WithBurst()functionaloptions applied to the
rest.Configbefore the clientset is built.cmd/provisioner-localpv/app/start.go: reads the env vars and passes theoptions through when constructing the client.
localpv.controller.clientQPS/localpv.controller.clientBurstvalues, conditionally injected into the Deployment and DaemonSet env, plus
README parameter docs.
Default behavior is unchanged. When the variables are unset (or empty), no
override is applied and client-go's own defaults (QPS 5 / burst 10) remain in
effect. The env vars are only injected by the chart when explicitly set, so
existing deployments render identically.
Does this PR require any upgrade changes?:
No. The feature is opt-in and defaults preserve prior behavior.
If the changes in this PR are manually verified, list down the scenarios covered::
helm templaterenders noOPENEBS_IO_CLIENT_QPS/OPENEBS_IO_CLIENT_BURSTenv entries; client uses client-go defaults.localpv.controller.clientQPS/clientBurst→ both env varsrendered into the pod spec (verified for both Deployment and DaemonSet/node
modes), and the provisioner logs the overridden QPS/burst at startup.
Any additional information for your reviewer?:
Continuation of #321 — that PR made the worker pool configurable; this PR makes
the shared client's rate limiter configurable so the added workers aren't
bottlenecked by the default client-side throttle. QPS is a
float32and burstan
intto match therest.Config.QPS/rest.Config.Burstfield types inclient-go.
Checklist:
<type>(<scope>): <subject>