Skip to content

fix: add option to disable consumption billing metrics worker to prevent OOM on large KV stores (#32059) - #32062

Open
waterWang wants to merge 3 commits into
hashicorp:mainfrom
waterWang:fix/consumption-billing-disable-32059
Open

fix: add option to disable consumption billing metrics worker to prevent OOM on large KV stores (#32059)#32062
waterWang wants to merge 3 commits into
hashicorp:mainfrom
waterWang:fix/consumption-billing-disable-32059

Conversation

@waterWang

Copy link
Copy Markdown

Summary

Fixes #32059 by adding a way to disable the consumption billing metrics worker that walks the entire KV store every 10 minutes, causing OOM kills on large installations.

Problem

After upgrading to Vault 2.0.x, the active node in a 3-node HA cluster runs out of memory every ~10 minutes because the consumption billing worker enumerates the entire KV store. On a large store this consumes hundreds of MB of cache memory and causes a restart loop. There is no configuration option or environment variable to disable it.

Changes

1. vault/billing/billing_counts.go

  • Added DisableConsumptionBilling bool field to BillingConfig struct

2. vault/consumption_billing.go

  • Added check for VAULT_DISABLE_CONSUMPTION_BILLING environment variable at the start of setupConsumptionBilling() — if set, the worker is not registered and a warning is logged
  • Added check for c.billingConfig.DisableConsumptionBilling config field — if true, the worker is not registered

3. sdk/helper/consts/consts.go

  • Added VaultDisableConsumptionBilling = "VAULT_DISABLE_CONSUMPTION_BILLING" constant

Usage

Environment variable

VAULT_DISABLE_CONSUMPTION_BILLING=true vault server -config=config.hcl

Config file (via enterprise config or future server config support)

The DisableConsumptionBilling flag on BillingConfig can be set through the config path.

Notes

  • The OIDC token duration counter, data protection call counts, and other in-memory billing metrics continue to accumulate correctly — they simply are not flushed to storage every 10 minutes
  • This is a minimal fix that addresses the immediate OOM issue. A future improvement could maintain incremental counters on write instead of enumerating the whole store.

@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 31, 2026
@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown

Deployment failed with the following error:

The `vercel.json` schema validation failed with the following message: should NOT have additional property `public`

Learn More: https://vercel.com/docs/concepts/projects/project-configuration

@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
vault-ui Error Error Jul 31, 2026 11:34am

Request Review

@hashicorp-cla-app

hashicorp-cla-app Bot commented Jul 31, 2026

Copy link
Copy Markdown

CLA assistant check

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes


water seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

Have you signed the CLA already but the status is still pending? Recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core/metric size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consumption billing metrics walk the entire KV store every 10 minutes on the active node, with no way to disable it (2.0.x)

1 participant