fix: add option to disable consumption billing metrics worker to prevent OOM on large KV stores (#32059) - #32062
Open
waterWang wants to merge 3 commits into
Open
Conversation
|
Deployment failed with the following error: Learn More: https://vercel.com/docs/concepts/projects/project-configuration |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement Learn more about why HashiCorp requires a CLA and what the CLA includes water seems not to be a GitHub user. Have you signed the CLA already but the status is still pending? Recheck it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #32059 by adding a way to disable the consumption billing metrics worker that walks the entire KV store every 10 minutes, causing OOM kills on large installations.
Problem
After upgrading to Vault 2.0.x, the active node in a 3-node HA cluster runs out of memory every ~10 minutes because the consumption billing worker enumerates the entire KV store. On a large store this consumes hundreds of MB of cache memory and causes a restart loop. There is no configuration option or environment variable to disable it.
Changes
1.
vault/billing/billing_counts.goDisableConsumptionBilling boolfield toBillingConfigstruct2.
vault/consumption_billing.goVAULT_DISABLE_CONSUMPTION_BILLINGenvironment variable at the start ofsetupConsumptionBilling()— if set, the worker is not registered and a warning is loggedc.billingConfig.DisableConsumptionBillingconfig field — if true, the worker is not registered3.
sdk/helper/consts/consts.goVaultDisableConsumptionBilling = "VAULT_DISABLE_CONSUMPTION_BILLING"constantUsage
Environment variable
Config file (via enterprise config or future server config support)
The
DisableConsumptionBillingflag onBillingConfigcan be set through the config path.Notes