Getting errors UserNotFound , Read Concern Majority Not Available Yet, livness fails

[naorsisense]

Description:
I am have a multi-node cluster of amazon-linux 2023 with storage fsx, the psmdb-db storageClass is gp3, using ebs-csi-driver i configure my own user for mongodb authentication, getting failures such as UserNotFound , ReadConcernMajorityNotAvailableYet and the livness is failing.

Steps to Reproduce:
environemnt - multi-node (3) of amazon-linux-2023, psmdb-db storageClass is gp3
installing psmbd with custom-user, pod is crashing with error of UserNotFound, ReadConcernMajorityNotAvailableYet

my helm values configuration:

psmdb-db:
backup:
enabled: false
enabled: true
users:
- name: sisense
db: admin
passwordSecretRef:
name: sisense-mongodb-secrets
key: mongo-password
roles:
- db: admin
name: root
- db: admin
name: userAdminAnyDatabase
- db: admin
name: readWriteAnyDatabase
- db: admin
name: dbAdminAnyDatabase
- db: admin
name: clusterAdmin
unsafeFlags:
replsetSize: true
sharding:
enabled: false
replsets:
rs0:
podSecurityContext:
fsGroup: 1000
containerSecurityContext:
runAsUser: 1000
runAsGroup: 1000
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-Application"
operator: In
values:
- "true"
- matchExpressions:
- key: "node-Build"
operator: In
values:
- "true"
volumeSpec:
pvc:
resources:
requests:
storage: "20Gi"
storageClassName: gp3

Version:
psmdb-operator 1.19.0
psmdb-db charts 1.19
mongodb image 6.0.20-17

Logs:

{"t":{"$date":"2025-03-20T14:43:20.204+00:00"},"s":"I",  "c":"ACCESS",   "id":20251,   "ctx":"conn64","msg":"Supported SASL mechanisms requested for unknown user","attr":{"user":{"user":"<USER>","db":"admin"}}}
{"t":{"$date":"2025-03-20T14:43:20.204+00:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn64","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-256","speculative":true,"principalName":"<USER>","authenticationDatabase":"admin","remote":"10.42.120.227:39198","extraInfo":{},"error":"UserNotFound: Could not find user \"<USER>\" for db \"admin\""}}
{"t":{"$date":"2025-03-20T14:43:20.204+00:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn64","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"<USER>","authenticationDatabase":"admin","remote":"10.42.120.227:39198","extraInfo":{},"error":"UserNotFound: Could not find user \"<USER>\" for db \"admin\""}}
{"t":{"$date":"2025-03-20T14:43:20.205+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn62","msg":"Interrupted operation as its client disconnected","attr":{"opId":15236}}
{"t":{"$date":"2025-03-20T14:43:23.853+00:00"},"s":"W",  "c":"NETWORK",  "id":23235,   "ctx":"conn67","msg":"SSL peer certificate validation failed","attr":{"reason":"certificate signature failure"}}
{"t":{"$date":"2025-03-20T14:43:23.853+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn67","msg":"client metadata","attr":{"remote":"127.0.0.1:54138","client":"conn67","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.4","env":{"container":{"orchestrator":"kubernetes"}}}}}
{"t":{"$date":"2025-03-20T14:43:23.858+00:00"},"s":"W",  "c":"NETWORK",  "id":23235,   "ctx":"conn68","msg":"SSL peer certificate validation failed","attr":{"reason":"certificate signature failure"}}
{"t":{"$date":"2025-03-20T14:43:23.858+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn68","msg":"client metadata","attr":{"remote":"127.0.0.1:54150","client":"conn68","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.4","env":{"container":{"orchestrator":"kubernetes"}}}}}
{"t":{"$date":"2025-03-20T14:43:27.117+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":7600}}
{"t":{"$date":"2025-03-20T14:43:33.829+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn67","msg":"Interrupted operation as its client disconnected","attr":{"opId":15653}}
{"t":{"$date":"2025-03-20T14:43:34.722+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":7800}}
{"t":{"$date":"2025-03-20T14:43:36.230+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn73","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":73,"remote":"10.42.61.22:42006"}}
{"t":{"$date":"2025-03-20T14:43:36.230+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn73","msg":"client metadata","attr":{"remote":"10.42.61.22:42006","client":"conn73","negotiatedCompressors":[],"doc":{"application":{"name":"configuration"},"driver":{"name":"nodejs","version":"5.9.2"},"platform":"Node.js v22.14.0, LE","os":{"name":"linux","architecture":"x64","version":"6.1.129-138.220.amzn2023.x86_64","type":"Linux"}}}}
{"t":{"$date":"2025-03-20T14:43:42.529+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":8000}}
{"t":{"$date":"2025-03-20T14:43:46.735+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn77","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":77,"remote":"10.42.61.22:47042"}}
{"t":{"$date":"2025-03-20T14:43:46.735+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn77","msg":"client metadata","attr":{"remote":"10.42.61.22:47042","client":"conn77","negotiatedCompressors":[],"doc":{"application":{"name":"configuration"},"driver":{"name":"nodejs","version":"5.9.2"},"platform":"Node.js v22.14.0, LE","os":{"name":"linux","architecture":"x64","version":"6.1.129-138.220.amzn2023.x86_64","type":"Linux"}}}}
{"t":{"$date":"2025-03-20T14:43:50.201+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn79","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":79,"remote":"10.42.120.227:57668"}}
{"t":{"$date":"2025-03-20T14:43:50.201+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn80","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":80,"remote":"10.42.120.227:57670"}}
{"t":{"$date":"2025-03-20T14:43:50.201+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn79","msg":"client metadata","attr":{"remote":"10.42.120.227:57668","client":"conn79","negotiatedCompressors":[],"doc":{"application":{"name":"mongodb_exporter"},"driver":{"name":"mongo-go-driver","version":"v1.12.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.21.3"}}}
{"t":{"$date":"2025-03-20T14:43:50.201+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn80","msg":"client metadata","attr":{"remote":"10.42.120.227:57670","client":"conn80","negotiatedCompressors":[],"doc":{"application":{"name":"mongodb_exporter"},"driver":{"name":"mongo-go-driver","version":"v1.12.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.21.3"}}}
{"t":{"$date":"2025-03-20T14:43:50.203+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn81","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":81,"remote":"10.42.120.227:57686"}}
{"t":{"$date":"2025-03-20T14:43:50.203+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn81","msg":"client metadata","attr":{"remote":"10.42.120.227:57686","client":"conn81","negotiatedCompressors":[],"doc":{"application":{"name":"mongodb_exporter"},"driver":{"name":"mongo-go-driver","version":"v1.12.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.21.3"}}}
{"t":{"$date":"2025-03-20T14:43:50.204+00:00"},"s":"I",  "c":"ACCESS",   "id":20251,   "ctx":"conn81","msg":"Supported SASL mechanisms requested for unknown user","attr":{"user":{"user":"<USER>","db":"admin"}}}
{"t":{"$date":"2025-03-20T14:43:50.204+00:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn81","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-256","speculative":true,"principalName":"<USER>","authenticationDatabase":"admin","remote":"10.42.120.227:57686","extraInfo":{},"error":"UserNotFound: Could not find user \"<USER>\" for db \"admin\""}}
{"t":{"$date":"2025-03-20T14:43:50.204+00:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn81","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"<USER>","authenticationDatabase":"admin","remote":"10.42.120.227:57686","extraInfo":{},"error":"UserNotFound: Could not find user \"<USER>\" for db \"admin\""}}
{"t":{"$date":"2025-03-20T14:43:50.205+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn79","msg":"Interrupted operation as its client disconnected","attr":{"opId":18632}}
{"t":{"$date":"2025-03-20T14:43:50.534+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":8200}}
{"t":{"$date":"2025-03-20T14:43:53.250+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn83","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":83,"remote":"10.42.120.199:51396"}}
{"t":{"$date":"2025-03-20T14:43:53.250+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn83","msg":"client metadata","attr":{"remote":"10.42.120.199:51396","client":"conn83","negotiatedCompressors":[],"doc":{"application":{"name":"configuration"},"driver":{"name":"nodejs","version":"5.9.2"},"platform":"Node.js v22.14.0, LE","os":{"name":"linux","architecture":"x64","version":"6.1.129-138.220.amzn2023.x86_64","type":"Linux"}}}}
{"t":{"$date":"2025-03-20T14:43:53.882+00:00"},"s":"W",  "c":"NETWORK",  "id":23235,   "ctx":"conn85","msg":"SSL peer certificate validation failed","attr":{"reason":"certificate signature failure"}}
{"t":{"$date":"2025-03-20T14:43:53.883+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn85","msg":"client metadata","attr":{"remote":"127.0.0.1:60266","client":"conn85","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.4","env":{"container":{"orchestrator":"kubernetes"}}}}}
{"t":{"$date":"2025-03-20T14:43:53.887+00:00"},"s":"W",  "c":"NETWORK",  "id":23235,   "ctx":"conn86","msg":"SSL peer certificate validation failed","attr":{"reason":"certificate signature failure"}}
{"t":{"$date":"2025-03-20T14:43:53.888+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn86","msg":"client metadata","attr":{"remote":"127.0.0.1:60268","client":"conn86","negotiatedCompressors":[],"doc":{"driver":{"name":"mongo-go-driver","version":"1.17.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.23.4","env":{"container":{"orchestrator":"kubernetes"}}}}}
{"t":{"$date":"2025-03-20T14:43:58.739+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":8400}}
{"t":{"$date":"2025-03-20T14:44:03.756+00:00"},"s":"I",  "c":"NETWORK",  "id":23838,   "ctx":"conn90","msg":"SSL mode is set to 'preferred' and connection to remote is not using SSL.","attr":{"connectionId":90,"remote":"10.42.120.199:46188"}}
{"t":{"$date":"2025-03-20T14:44:03.756+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn90","msg":"client metadata","attr":{"remote":"10.42.120.199:46188","client":"conn90","negotiatedCompressors":[],"doc":{"application":{"name":"configuration"},"driver":{"name":"nodejs","version":"5.9.2"},"platform":"Node.js v22.14.0, LE","os":{"name":"linux","architecture":"x64","version":"6.1.129-138.220.amzn2023.x86_64","type":"Linux"}}}}
{"t":{"$date":"2025-03-20T14:44:03.860+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn85","msg":"Interrupted operation as its client disconnected","attr":{"opId":19057}}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"CONTROL",  "id":23377,   "ctx":"SignalHandler","msg":"Received signal","attr":{"signal":15,"error":"Terminated"}}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"CONTROL",  "id":23378,   "ctx":"SignalHandler","msg":"Signal was sent by kill(2)","attr":{"pid":0,"uid":0}}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"CONTROL",  "id":23381,   "ctx":"SignalHandler","msg":"will terminate after current cmd ends"}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"SignalHandler","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":15000}}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"REPL",     "id":4794602, "ctx":"SignalHandler","msg":"Attempting to enter quiesce mode"}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"-",        "id":6371601, "ctx":"SignalHandler","msg":"Shutting down the FLE Crud thread pool"}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"FLECrudNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"SignalHandler","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"REPL",     "id":40441,   "ctx":"SignalHandler","msg":"Stopping TopologyVersionObserver"}
{"t":{"$date":"2025-03-20T14:44:03.879+00:00"},"s":"I",  "c":"REPL",     "id":40447,   "ctx":"TopologyVersionObserver","msg":"Stopped TopologyVersionObserver"}
{"t":{"$date":"2025-03-20T14:44:03.880+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"MirrorMaestro","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.880+00:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"SignalHandler","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2025-03-20T14:44:03.880+00:00"},"s":"I",  "c":"CONTROL",  "id":4784903, "ctx":"SignalHandler","msg":"Shutting down the LogicalSessionCache"}
{"t":{"$date":"2025-03-20T14:44:03.880+00:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"SignalHandler","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2025-03-20T14:44:03.880+00:00"},"s":"I",  "c":"NETWORK",  "id":23017,   "ctx":"listener","msg":"removing socket file","attr":{"path":"/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"SignalHandler","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"CONTROL",  "id":4784906, "ctx":"SignalHandler","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"SignalHandler","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"REPL",     "id":4784907, "ctx":"SignalHandler","msg":"Shutting down the replica set node executor"}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"ReplNodeDbWorkerNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"CONTROL",  "id":4784908, "ctx":"SignalHandler","msg":"Shutting down the PeriodicThreadToAbortExpiredTransactions"}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"REPL",     "id":4784909, "ctx":"SignalHandler","msg":"Shutting down the ReplicationCoordinator"}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"REPL",     "id":5074000, "ctx":"SignalHandler","msg":"Shutting down the replica set aware services."}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"REPL",     "id":5123006, "ctx":"SignalHandler","msg":"Shutting down PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","numInstances":0,"numOperationContexts":0}}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"TenantMigrationRecipientServiceNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"REPL",     "id":5123006, "ctx":"SignalHandler","msg":"Shutting down PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","numInstances":0,"numOperationContexts":0}}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"ShardSplitDonorServiceNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.881+00:00"},"s":"I",  "c":"REPL",     "id":5123006, "ctx":"SignalHandler","msg":"Shutting down PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","numInstances":0,"numOperationContexts":0}}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"TenantMigrationDonorServiceNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"REPL",     "id":21328,   "ctx":"SignalHandler","msg":"Shutting down replication subsystems"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"ReplNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"SHARDING", "id":4784910, "ctx":"SignalHandler","msg":"Shutting down the ShardingInitializationMongoD"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"REPL",     "id":4784911, "ctx":"SignalHandler","msg":"Enqueuing the ReplicationStateTransitionLock for shutdown"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"-",        "id":4784912, "ctx":"SignalHandler","msg":"Killing all operations for shutdown"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"-",        "id":4695300, "ctx":"SignalHandler","msg":"Interrupted all currently running operations","attr":{"opsKilled":6}}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"TENANT_M", "id":5093807, "ctx":"SignalHandler","msg":"Shutting down all TenantMigrationAccessBlockers on global shutdown"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"COMMAND",  "id":4784913, "ctx":"SignalHandler","msg":"Shutting down all open transactions"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"REPL",     "id":4784914, "ctx":"SignalHandler","msg":"Acquiring the ReplicationStateTransitionLock for shutdown"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"INDEX",    "id":4784915, "ctx":"SignalHandler","msg":"Shutting down the IndexBuildsCoordinator"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"SignalHandler","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2025-03-20T14:44:03.882+00:00"},"s":"I",  "c":"REPL",     "id":4784920, "ctx":"SignalHandler","msg":"Shutting down the LogicalTimeValidator"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"SignalHandler","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"MigrationUtil-TaskExecutor","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"COMMAND",  "id":4784923, "ctx":"SignalHandler","msg":"Shutting down the ServiceEntryPoint"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"CONTROL",  "id":4784927, "ctx":"SignalHandler","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"CONTROL",  "id":4784928, "ctx":"SignalHandler","msg":"Shutting down the TTL monitor"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"INDEX",    "id":3684100, "ctx":"SignalHandler","msg":"Shutting down TTL collection monitor thread"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"INDEX",    "id":3684101, "ctx":"SignalHandler","msg":"Finished shutting down TTL collection monitor thread"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"CONTROL",  "id":6278511, "ctx":"SignalHandler","msg":"Shutting down the Change Stream Expired Pre-images Remover"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"QUERY",    "id":6278515, "ctx":"SignalHandler","msg":"Shutting down Change Stream Expired Pre-images Remover thread"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"QUERY",    "id":6278516, "ctx":"SignalHandler","msg":"Finished shutting down Change Stream Expired Pre-images Remover thread"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"CONTROL",  "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"CONTROL",  "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engine"}
{"t":{"$date":"2025-03-20T14:44:03.883+00:00"},"s":"I",  "c":"STORAGE",  "id":22320,   "ctx":"SignalHandler","msg":"Shutting down journal flusher thread"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22321,   "ctx":"SignalHandler","msg":"Finished shutting down journal flusher thread"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22322,   "ctx":"SignalHandler","msg":"Shutting down checkpoint thread"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22323,   "ctx":"SignalHandler","msg":"Finished shutting down checkpoint thread"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22261,   "ctx":"SignalHandler","msg":"Timestamp monitor shutting down"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":20282,   "ctx":"SignalHandler","msg":"Deregistering all the collections"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22317,   "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting down"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22318,   "ctx":"SignalHandler","msg":"Shutting down session sweeper thread"}
{"t":{"$date":"2025-03-20T14:44:03.884+00:00"},"s":"I",  "c":"STORAGE",  "id":22319,   "ctx":"SignalHandler","msg":"Finished shutting down session sweeper thread"}
{"t":{"$date":"2025-03-20T14:44:03.885+00:00"},"s":"I",  "c":"STORAGE",  "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,"}}
{"t":{"$date":"2025-03-20T14:44:03.885+00:00"},"s":"I",  "c":"WTCHKPT",  "id":22430,   "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1742481843,"ts_usec":885873,"thread":"1:0x7ffb48a70640","session_name":"close_ckpt","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 8, snapshot max: 8 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 234"}}}
{"t":{"$date":"2025-03-20T14:44:03.894+00:00"},"s":"I",  "c":"WTRECOV",  "id":22430,   "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1742481843,"ts_usec":894469,"thread":"1:0x7ffb48a70640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"shutdown checkpoint has successfully finished and ran for 8 milliseconds"}}}
{"t":{"$date":"2025-03-20T14:44:03.894+00:00"},"s":"I",  "c":"WTRECOV",  "id":22430,   "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1742481843,"ts_usec":894571,"thread":"1:0x7ffb48a70640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"shutdown was completed successfully and took 9ms, including 0ms for the rollback to stable, and 8ms for the checkpoint."}}}
{"t":{"$date":"2025-03-20T14:44:03.909+00:00"},"s":"I",  "c":"STORAGE",  "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":{"durationMillis":24}}
{"t":{"$date":"2025-03-20T14:44:03.935+00:00"},"s":"I",  "c":"STORAGE",  "id":22279,   "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."}
{"t":{"$date":"2025-03-20T14:44:03.935+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2025-03-20T14:44:03.935+00:00"},"s":"I",  "c":"FTDC",     "id":20626,   "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"}
{"t":{"$date":"2025-03-20T14:44:03.940+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"SignalHandler","msg":"Now exiting"}
{"t":{"$date":"2025-03-20T14:44:03.940+00:00"},"s":"I",  "c":"CONTROL",  "id":8423404, "ctx":"SignalHandler","msg":"mongod shutdown complete","attr":{"Summary of time elapsed":{"Statistics":{"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"0 ms","Time spent in quiesce mode":"0 ms","Shut down FLE Crud subsystem":"0 ms","Shut down MirrorMaestro":"1 ms","Shut down WaitForMajorityService":"0 ms","Shut down the logical session cache":"0 ms","Shut down the transport layer":"0 ms","Shut down the global connection pool":"1 ms","Shut down the flow control ticket holder":"0 ms","Shut down the replica set node executor":"0 ms","Shut down the replica set aware services":"0 ms","Shut down the replica set aware services":"1 ms","Shut down replication":"0 ms","Shut down replication":"0 ms","Shut down external state":"0 ms","Shut down external state":"0 ms","Shut down replication executor":"0 ms","Shut down replication executor":"0 ms","Join replication executor":"0 ms","Join replication executor":"0 ms","Kill all operations for shutdown":"0 ms","Shut down all tenant migration access blockers on global shutdown":"0 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"0 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"0 ms","Shut down the logical time validator":"0 ms","Shut down the migration util executor":"0 ms","Shut down the health log":"0 ms","Shut down the TTL monitor":"0 ms","Shut down expired pre-images remover":"0 ms","Shut down the storage engine":"52 ms","Shut down full-time data capture":"0 ms","shutdownTask total elapsed time":"61 ms"}}}}
{"t":{"$date":"2025-03-20T14:44:03.940+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCode":0}

operator-logs

2025-03-20T13:53:02.345Z	INFO	server version	{"platform": "kubernetes", "version": "v1.32.1+rke2r1"}
2025-03-20T13:53:02.349Z	INFO	controller-runtime.metrics	Starting metrics server
2025-03-20T13:53:02.349Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
2025-03-20T13:53:02.349Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
I0320 13:53:02.349524       1 leaderelection.go:254] attempting to acquire leader lease sisense/08db0feb.percona.com...
I0320 13:53:02.355067       1 leaderelection.go:268] successfully acquired lease sisense/08db0feb.percona.com
2025-03-20T13:53:02.355Z	INFO	Starting EventSource	{"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2025-03-20T13:53:02.355Z	INFO	Starting Controller	{"controller": "psmdb-controller"}
2025-03-20T13:53:02.355Z	INFO	Starting EventSource	{"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2025-03-20T13:53:02.355Z	INFO	Starting EventSource	{"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2025-03-20T13:53:02.355Z	INFO	Starting Controller	{"controller": "psmdbrestore-controller"}
2025-03-20T13:53:02.355Z	INFO	Starting EventSource	{"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2025-03-20T13:53:02.355Z	INFO	Starting EventSource	{"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2025-03-20T13:53:02.355Z	INFO	Starting Controller	{"controller": "psmdbbackup-controller"}
2025-03-20T13:53:02.556Z	INFO	Starting workers	{"controller": "psmdb-controller", "worker count": 1}
2025-03-20T13:53:02.560Z	INFO	Starting workers	{"controller": "psmdbrestore-controller", "worker count": 1}
2025-03-20T13:53:02.560Z	INFO	Starting workers	{"controller": "psmdbbackup-controller", "worker count": 1}
2025-03-20T13:53:24.977Z	INFO	Created a new mongo key	{"controller": "psmdb-controller", "object": {"name":"sisense-psmdb-db","namespace":"sisense"}, "namespace": "sisense", "name": "sisense-psmdb-db", "reconcileID": "73525246-72a3-48c6-8faa-605a24548e15", "KeyName": "sisense-psmdb-db-mongodb-keyfile"}
2025-03-20T13:53:24.983Z	INFO	Created a new mongo key	{"controller": "psmdb-controller", "object": {"name":"sisense-psmdb-db","namespace":"sisense"}, "namespace": "sisense", "name": "sisense-psmdb-db", "reconcileID": "73525246-72a3-48c6-8faa-605a24548e15", "KeyName": "sisense-psmdb-db-mongodb-encryption-key"}
2025-03-20T13:53:25.084Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"sisense-psmdb-db","namespace":"sisense"}, "namespace": "sisense", "name": "sisense-psmdb-db", "reconcileID": "73525246-72a3-48c6-8faa-605a24548e15", "replset": "rs0", "size": 3, "pods": 1}
2025-03-20T13:53:25.258Z	INFO	add new job	{"controller": "psmdb-controller", "object": {"name":"sisense-psmdb-db","namespace":"sisense"}, "namespace": "sisense", "name": "sisense-psmdb-db", "reconcileID": "73525246-72a3-48c6-8faa-605a24548e15", "name": "ensure-version/sisense/sisense-psmdb-db", "schedule": "0 2 * * *"}

Expected Result:
other deployment and clusters i have works fine such as eks, ubuntu cluster and so on

Additional Information:
I Really dont understand how to debug this issue, same configuration works for me on other deployments, i cannot log into the mongodb while the users seems to be missing on admin db.

[naorsisense]

in addition that some times i also get Authentication failed and UserNotFound on single node and it happens from time to time.

[naorsisense]

Still getting this issue on multiplay deployment even on customers,
please give me how to debug this I can not see what the operator is doing regarding writing the User in the admin db

[rojinebrahimi]

Same issue here.

[Yossizz]

same for me - anyone found solution for that ?
seems like the psmdb is stuck on error state and the operator is not able to connect it using the admin user

handleReplsetInit: exec add admin user: command terminated with exit code 1
MongoServerError: command createUser requires authentication
Warning: Could not access file: EACCES: permission denied, mkdir '/.mongodb'

test-psmdb-db error 8m10s

it may be related to
canonical/mongodb-k8s-operator#397

[mr-anderson86]

Having same issue, I though I did a mistake, but now I'm seeing this happens to more...
This is part of my values:

users:
- name: my-user
  db: admin
  passwordSecretRef:
    name: my-user-password
    key: my-user-password-key
  roles:
    - name: clusterAdmin
      db: admin
    - name: userAdminAnyDatabase
      db: admin
replsets:
  rs0:
    name: mongodb
    size: 3

And yes, the k8s secret my-user-password is pre-created before installing Percona MongoDB (and contains the key my-user-password-key).

The issue: seems that somehow, this user my-user isn't being created.
(I also try to login to the mongo with $MONGODB_USER_ADMIN_USER and $MONGODB_USER_ADMIN_PASSWORD from the secret it creates - but always getting authentication error, like also this admin user isn't being created 😕)

Hope someone out there have a solution or WA... 🙏🏼