Skip to content

Configurable OM JVM heap percentage and opt-in native auto-heap#1355

Draft
nammn wants to merge 3 commits into
masterfrom
om-jvm
Draft

Configurable OM JVM heap percentage and opt-in native auto-heap#1355
nammn wants to merge 3 commits into
masterfrom
om-jvm

Conversation

@nammn

@nammn nammn commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

The OM JVM heap was hardcoded to 90% of the container memory limit — an arbitrary value with no design rationale (traced to CLOUDP-82600, Feb 2021). This left insufficient headroom for non-heap JVM memory (metaspace, thread stacks, direct buffers), causing real customer OOM kills:

  • HELP-90788: Customer with 20Gi limit got -Xmx18432m (90%), leaving only 2GB headroom. Support confirmed "the 2gb headroom is definitely not sufficient" and recommended increasing to 24Gi. At 75%, the same pod would have 5GB headroom.

This PR:

  • Makes the percentage configurable via MDB_OM_JVM_HEAP_PERCENTAGE env var (default changed from 90% to 75%, the MongoDB internal standard per CLOUDP-373541)
  • Adds opt-in support for OM's native cgroup-aware heap sizing via MDB_OM_AUTO_HEAP env var — when enabled, the operator skips -Xmx/-Xms and delegates to OM's built-in logic

Before: OM pods got -Xmx4291m -Xms4291m (90% of default 5Gi). After: OM pods get -Xmx3576m -Xms3576m (75% of 5Gi), unless overridden.

Proof of Work

  • controllers/operator/construct/jvm_test.goTestGetOpsManagerPodMemPercentage (5 subtests), TestIsOmAutoHeapEnabled (3 subtests)
  • controllers/operator/construct/opsmanager_construction_test.goTestOmAutoHeapEnvVars, TestBuildJvmParamsEnvVars_WithAutoHeap, TestBuildJvmParamsEnvVars_WithCustomPercentage, updated existing tests for new 75% default
  • docker/mongodb-kubernetes-tests/tests/opsmanager/om_jvm_params.py — e2e test updated for new 75% expectations
  • All unit tests pass: go test ./controllers/operator/construct/ -run "Jvm|AutoHeap|CustomPercentage|OmAutoHeap|GetOpsManager" -v -count=1

Checklist

  • Have you linked a jira ticket and/or is the ticket in the title?
  • Have you checked whether your jira ticket required DOCSP changes?
  • Have you added changelog file?

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.10.0 Release Notes

New Features

  • OpsManager: The JVM heap percentage for Ops Manager pods is now configurable via the MDB_OM_JVM_HEAP_PERCENTAGE environment variable. The default has changed from 90% to 75% (the MongoDB internal standard), reducing the risk of container OOM kills. Users who need the previous value can set MDB_OM_JVM_HEAP_PERCENTAGE=90.
  • OpsManager: Added opt-in support for Ops Manager's native cgroup-aware heap sizing via the MDB_OM_AUTO_HEAP environment variable. When enabled, the operator skips setting -Xmx/-Xms and delegates heap sizing to Ops Manager's built-in logic. This requires OM pods with at least 16Gi of memory.

Bug Fixes

  • MongoDBUser: Fixed a bug where the connection string Secret created in the central cluster did not carry a controller owner reference to the MongoDBUser CR. The missing reference prevented Kubernetes garbage collection from cleaning up the Secret when the MongoDBUser was deleted.
  • MongoDBUser: Fixed a bug where the ownership guard on the connection string Secret was always satisfied regardless of the actual owner, allowing the operator to silently overwrite a Secret controlled by a different resource.
  • Fixed a bug where all MongoDBCommunity deployment telemetry rows incorrectly reported IsRunningEnterpriseImage = true. The field was being evaluated against the operator-level enterprise image rather than the image configured in each CR's spec, causing a misleading ~100% enterprise rate for the Community deployment type.
  • IsRunningEnterpriseImage for Community deployments is now derived from the mongod container image override in spec.statefulSet.spec.template.spec.containers, if present. When no override is set, the field correctly defaults to false, reflecting that Community CRs use the community MongoDB server image by default.
  • MongoDBSearch: The default JVM heap size (half of the memory request) is now capped at 30GB, following the mongot sizing guidance. Heap sizes above ~30GB prevent the JVM from using compressed object pointers and degrade performance. User-provided heap flags are not affected, if more than 30GB heap is required, we recommend using jvmFlags.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant