KUBE-141: Remove stale mms.https.PEMKeyFile from Ops Manager config when TLS is disabled

[Julien-Ben]

Summary

Disabling TLS on a running Ops Manager (removing spec.security) left the instance unable to start whenever the conf directory outlives the container — most commonly a user PVC mounted over /mongodb-ops-manager/conf, which the operator supports as a StatefulSet override (HELP-84463). The operator builds the new StatefulSet correctly, but the mmsconfiguration binary in the init-ops-manager image only ever overwrote or appended OM_PROP_* properties in conf-mms.properties, never deleting one it stopped setting. The stale mms.https.PEMKeyFile then points at a certificate secret that is no longer mounted and Ops Manager crash-loops in preflight. With the default emptyDir layout the conf directory is recreated on every rollout, which is why this never reproduced on kind.

Solution: Operator-set properties not already present in the file are now written inside a BEGIN/END-delimited block that is stripped and rewritten on every container start, the same way custom JVM params are already handled in this binary. A property the operator stops setting disappears from the file instead of lingering forever.

A note for reviewers: this is admittedly a bit manual — operator-owned properties are tracked with custom BEGIN/END comment markers and rewritten by string manipulation, mirroring the existing JVM-params handling. The approach deserves a careful look; happy to discuss alternatives.

Deployments already broken by an older image have the stale property as a plain line outside any block; those still need the existing workaround (remove the line, or delete the conf PVC). The fix prevents the state from the first start on a fixed image.

Proof of Work

• New unit test reproducing the stale-property bug (fails without the fix), plus idempotency coverage.
• The om_ops_manager_https e2e now runs with a PVC mounted over the conf directory and ends with a disable-TLS stage:
  • without the fix it fails on exactly that stage, with the customer's crash-loop signature in the OM pod log: patch
  • with the fix the task is green: patch

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • use skip-changelog label if not needed
  • refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.10.0 Release Notes

New Features

• MongoDBOpsManager, AppDB: The mongodb-agent-monitoring sidecar container has been merged into the main
  mongodb-agent container. Both automation and monitoring now run as a single process in a single container. The
  spec.appDB.monitoringAgent field is now deprecated and has no effect; Monitoring Agent options for the AppDB —
  including log level and rotation — are configured via spec.applicationDatabase.agent and
  spec.applicationDatabase.agent.monitoringAgent, the same fields used by MongoDB resources. Enabling AppDB
  monitoring still results in rolling restarts of AppDB pods.
• MongoDBSearch: Added support to show cluster level status of Search, Managed LoadBalancer and MetricsForwarder in the MongoDBSearch resource's status.clusters field.
• MongoDBSearch: Added support to configure the node affinity of the MongoDB Search (mongot) pods using the MongoDBSearch CR fields spec.clusters[].nodeAffinity or spec.clusters[].shardOverrides[].nodeAffinity.

Bug Fixes

• MongoDBUser: Fixed a bug where the connection string Secret created in the central cluster did not carry a controller owner reference to the MongoDBUser CR. The missing reference prevented Kubernetes garbage collection from cleaning up the Secret when the MongoDBUser was deleted.
• MongoDBUser: Fixed a bug where the ownership guard on the connection string Secret was always satisfied regardless of the actual owner, allowing the operator to silently overwrite a Secret controlled by a different resource.
• Fixed a bug where all MongoDBCommunity deployment telemetry rows incorrectly reported IsRunningEnterpriseImage = true. The field was being evaluated against the operator-level enterprise image rather than the image configured in each CR's spec, causing a misleading ~100% enterprise rate for the Community deployment type.
• IsRunningEnterpriseImage for Community deployments is now derived from the mongod container image override in spec.statefulSet.spec.template.spec.containers, if present. When no override is set, the field correctly defaults to false, reflecting that Community CRs use the community MongoDB server image by default.
• MongoDBOpsManager, AppDB: Fix for spec.applicationDatabase.agent.monitoringAgent.logRotate field not being
  handled properly. Additionally, added defaults for
  spec.applicationDatabase.agent.monitoringAgent.logRotate.sizeThresholdMB to 1000 and
  spec.applicationDatabase.agent.monitoringAgent.logRotate.timeThresholdHrs to 24, which are same defaults Ops Manager
  would set.
• MongoDBSearch: The default JVM heap size (half of the memory request) is now capped at 30GB, following the mongot sizing guidance. Heap sizes above ~30GB prevent the JVM from using compressed object pointers and degrade performance. User-provided heap flags are not affected, if more than 30GB heap is required, we recommend using jvmFlags.
• MongoDBOpsManager: Fixed Ops Manager failing to start after TLS is disabled when the configuration directory is persisted (for example a PVC mounted over /mongodb-ops-manager/conf). Properties the operator no longer sets, such as mms.https.PEMKeyFile, are now removed from conf-mms.properties on container start instead of lingering forever.