Skip to content

feat(client): make Kubernetes API client QPS and burst configurable#342

Open
oneubauer wants to merge 1 commit into
openebs:developfrom
oneubauer:add-configurable-client-rate-limits
Open

feat(client): make Kubernetes API client QPS and burst configurable#342
oneubauer wants to merge 1 commit into
openebs:developfrom
oneubauer:add-configurable-client-rate-limits

Conversation

@oneubauer

@oneubauer oneubauer commented Jun 17, 2026

Copy link
Copy Markdown

Why is this PR required? What issue does it fix?:

Follow-up to #321 (configurable worker threads). #321 made the provisioner's
worker count tunable via OPENEBS_IO_WORKER_THREADS, but all workers share a
single Kubernetes clientset, and that clientset's rate limiter is still pinned
to the client-go defaults (QPS 5 / burst 10). Raising the worker count alone
therefore just creates more goroutines that contend for the same client-side
throttle — under bursty PVC create/delete load the provisioner self-throttles
before the API server's own API Priority & Fairness ever comes into play.

This PR exposes the client's QPS and burst so operators running at scale can
raise them alongside workers.

What this PR does?:

Adds two new, optional environment variables (and matching Helm values) to tune
the rate limiter on the shared Kubernetes API client:

  • OPENEBS_IO_CLIENT_QPS: maximum sustained queries/sec to the API server.
  • OPENEBS_IO_CLIENT_BURST: maximum burst above QPS.

Wiring:

  • cmd/provisioner-localpv/app/env.go: env-var names + getClientQPS() /
    getClientBurst() getters (functional menv lookup, same pattern as the
    existing worker-thread/timeout getters).
  • pkg/kubernetes/client/client.go: WithQPS() / WithBurst() functional
    options applied to the rest.Config before the clientset is built.
  • cmd/provisioner-localpv/app/start.go: reads the env vars and passes the
    options through when constructing the client.
  • Helm chart: localpv.controller.clientQPS / localpv.controller.clientBurst
    values, conditionally injected into the Deployment and DaemonSet env, plus
    README parameter docs.

Default behavior is unchanged. When the variables are unset (or empty), no
override is applied and client-go's own defaults (QPS 5 / burst 10) remain in
effect. The env vars are only injected by the chart when explicitly set, so
existing deployments render identically.

Does this PR require any upgrade changes?:

No. The feature is opt-in and defaults preserve prior behavior.

If the changes in this PR are manually verified, list down the scenarios covered::

  • Unset vars → helm template renders no OPENEBS_IO_CLIENT_QPS /
    OPENEBS_IO_CLIENT_BURST env entries; client uses client-go defaults.
  • Vars set via localpv.controller.clientQPS / clientBurst → both env vars
    rendered into the pod spec (verified for both Deployment and DaemonSet/node
    modes), and the provisioner logs the overridden QPS/burst at startup.
  • Invalid / non-positive values → ignored, falls back to client-go defaults.

Any additional information for your reviewer?:

Continuation of #321 — that PR made the worker pool configurable; this PR makes
the shared client's rate limiter configurable so the added workers aren't
bottlenecked by the default client-side throttle. QPS is a float32 and burst
an int to match the rest.Config.QPS / rest.Config.Burst field types in
client-go.

Checklist:

  • Fixes #
  • PR Title follows the convention of <type>(<scope>): <subject>
  • Has the change log section been updated?
  • Commit has unit tests
  • Commit has integration tests
  • (Optional) Are upgrade changes included in this PR? If not, mention the issue/PR to track:
  • (Optional) If documentation changes are required, which issue on https://github.com/openebs/openebs-docs is used to track them:

@oneubauer oneubauer requested a review from a team as a code owner June 17, 2026 04:29
@oneubauer oneubauer force-pushed the add-configurable-client-rate-limits branch from b5accc3 to 0887022 Compare June 17, 2026 04:31
Follow-up to openebs#321. The provisioner's workers share a single clientset
whose rate limiter was pinned to the client-go defaults (QPS 5 /
burst 10), so raising OPENEBS_IO_WORKER_THREADS alone left workers
contending for the same client-side throttle.

Add OPENEBS_IO_CLIENT_QPS and OPENEBS_IO_CLIENT_BURST (and matching
localpv.controller.clientQPS / clientBurst Helm values) to override the
client's rate limiter. Both are opt-in: when unset, client-go's defaults
are left untouched and rendered manifests are unchanged.

Signed-off-by: Oliver Neubauer <o.neubauer@gmail.com>
@oneubauer oneubauer force-pushed the add-configurable-client-rate-limits branch from 0887022 to 48c2e9e Compare June 18, 2026 03:55
Comment on lines +188 to +198
func getClientQPS() (float32, bool) {
val, present := menv.Lookup(ProvisionerClientQPS)
if !present || val == "" {
return 0, false
}
qps, err := strconv.ParseFloat(val, 32)
if err != nil || qps <= 0 {
return 0, false
}
return float32(qps), true
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
func getClientQPS() (float32, bool) {
val, present := menv.Lookup(ProvisionerClientQPS)
if !present || val == "" {
return 0, false
}
qps, err := strconv.ParseFloat(val, 32)
if err != nil || qps <= 0 {
return 0, false
}
return float32(qps), true
}
func getClientQPS() (float32, bool) {
val, present := k8sEnv.GetFloat64(ProvisionerClientQPS, 0.0)
if !present || val <= 0.0 {
return 0.0, false
}
return float32(val), true
}

Comment on lines +205 to +213
val, present := menv.Lookup(ProvisionerClientBurst)
if !present || val == "" {
return 0, false
}
burst, err := strconv.Atoi(val)
if err != nil || burst <= 0 {
return 0, false
}
return burst, true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
val, present := menv.Lookup(ProvisionerClientBurst)
if !present || val == "" {
return 0, false
}
burst, err := strconv.Atoi(val)
if err != nil || burst <= 0 {
return 0, false
}
return burst, true
val, present := k8sEnv.GetInt(ProvisionerClientBurst, 0)
if !present || val <= 0 {
return 0, false
}
return val, true

// Apply client-side API rate-limit overrides only when explicitly set,
// so an unconfigured provisioner keeps the client-go defaults (QPS 5,
// Burst 10).
var clientOpts []mKube.OptionFn

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
var clientOpts []mKube.OptionFn
clientOpts := make([]mKube.OptionFn, 0, 2)

Comment on lines +253 to +266
if qps > 0 {
c.qps = qps
}
}
}

// WithBurst sets the maximum burst of queries above the QPS limit the client
// may send to the Kubernetes API server. A non-positive value is ignored,
// leaving the client-go default in place.
func WithBurst(burst int) OptionFn {
return func(c *Client) {
if burst > 0 {
c.burst = burst
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not required, as we're already doing this in applyRateLimits

Suggested change
if qps > 0 {
c.qps = qps
}
}
}
// WithBurst sets the maximum burst of queries above the QPS limit the client
// may send to the Kubernetes API server. A non-positive value is ignored,
// leaving the client-go default in place.
func WithBurst(burst int) OptionFn {
return func(c *Client) {
if burst > 0 {
c.burst = burst
}
c.qps = qps
}
}
// WithBurst sets the maximum burst of queries above the QPS limit the client
// may send to the Kubernetes API server. A non-positive value is ignored,
// leaving the client-go default in place.
func WithBurst(burst int) OptionFn {
return func(c *Client) {
c.burst = burst

Comment on lines +52 to +55
clientQPS: ""
# Maximum burst of queries above clientQPS the provisioner allows to the Kubernetes API
# server. Leave empty to use the client-go default (10).
clientBurst: ""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
clientQPS: ""
# Maximum burst of queries above clientQPS the provisioner allows to the Kubernetes API
# server. Leave empty to use the client-go default (10).
clientBurst: ""
client:
qps: ""
# Maximum burst of queries above clientQPS the provisioner allows to the Kubernetes API
# server. Leave empty to use the client-go default (10).
burst: ""

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants